[kafka monitoring] Prometheus 서비스 기동

Prometheus는 등록된 target에 대해 주기적으로 metric을 수집하는 역할을 합니다. PromQL(쿼리 언어)을 사용하여 수집된 데이터를 활용 가능합니다. Default listen port: 9090

 

2-1. Prometheus Download

• 최신 버전 확인: https://prometheus.io/download/

**$ wget <https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz>
$ tar xzf prometheus-2.45.0.linux-amd64.tar.gz
$ ln -s prometheus-2.45.0.linux-amd64 prometheus**

2-2. prometheus.yml

• jmx-monitoring-stacks/jmxexporter-prometheus-grafana/assets/prometheus/prometheus-config 아래 파일을 prometheus binary 파일과 같은 디렉토리에 위치시킵니다.

**$ cp /home/confluent/jmx-monitoring-stacks/jmxexporter-prometheus-grafana/assets/prometheus/prometheus-config/* prometheus**

• prometheus.yml을 수정: 각 컴포넌트 서버의 hostname 및 jmx metric 노출용 port 를 설정합니다.

---
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 15s

rule_files:

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "zookeeper"
    static_configs:
      - targets:
          - "zookeeper11:1234"
        labels:
          instance: "zookeeper11"
          env: "clusterlocal"
      - targets:
          - "zookeeper12:1234"
        labels:
          instance: "zookeeper12"
          env: "clusterlocal"
      - targets:
          - "zookeeper13:1234"
        labels:
          instance: "zookeeper13"
          env: "clusterlocal"
    relabel_configs:
      - source_labels: [__address__]
        target_label: hostname
        regex: "([^:]+)(:[0-9]+)?"
        replacement: "${1}"

  - job_name: "kafka-broker"
    static_configs:
      - targets:
          - "kafka11:1234"
        labels:
          instance: "kafka11"
          env: "clusterlocal"
      - targets:
          - "kafka12:1234"
        labels:
          instance: "kafka12"
          env: "clusterlocal"
      - targets:
          - "kafka13:1234"
        labels:
          instance: "kafka13"
          env: "clusterlocal"
    relabel_configs:
      - source_labels: [__address__]
        target_label: hostname
        regex: "([^:]+)(:[0-9]+)?"
        replacement: "${1}"

  - job_name: "schema-registry"
    static_configs:
      - targets:
          - "schema-registry11:1234"
        labels:
          instance: "schema-registry11"
          env: "clusterlocal"
      - targets:
          - "schema-registry12:1234"
        labels:
          instance: "schema-registry12"
          env: "clusterlocal"
    relabel_configs:
      - source_labels: [__address__]
        target_label: hostname
        regex: "([^:]+)(:[0-9]+)?"
        replacement: "${1}"

  - job_name: "kafka-connect"
    static_configs:
      - targets:
          - "kafka-connect11:1234"
        labels:
          instance: "kafka-connect11"
          env: "clusterlocal"
          kafka_connect_cluster_id: "connect-cluster"
      - targets:
          - "kafka-connect12:1234"
        labels:
          instance: "kafka-connect12"
          env: "clusterlocal"
          kafka_connect_cluster_id: "connect-cluster"
    relabel_configs:
      - source_labels: [__address__]
        target_label: hostname
        regex: "([^:]+)(:[0-9]+)?"
        replacement: "${1}"

  - job_name: "ksqldb"
    static_configs:
      - targets:
          - "ksqldb11:1234"
        labels:
          instance: "ksqldb11"
          env: "clusterlocal"
    relabel_configs:
      - source_labels: [__address__]
        target_label: hostname
        regex: "([^:]+)(:[0-9]+)?"
        replacement: "${1}"

 

✅ [ Prometheus 주요 설정 사항 ] 

• scrape_interval: metric을 수집하는 시간 간격
• evaluation_interval: 규칙을 평가하는 시간 간격
• scrape_timeout: 수집 요청 시간이 초과되기 전 대기할 시간
• rule_files: 규칙을 로딩할 소스 파일, evaluation_interval 값에 따라 주기적으로 평가
• scrape_configs: 실제 수집 대상을 지정하는 설정
• job_name: 인스턴스 집합
• static_configs: 데이터를 수집하는 방식에 대한 정의
• targets: prometheus 서버가 접근해서 metric 데이터를 가져올 Exporter의 HTTP

2-3. 기동 및 확인

1. 스크립트를 통한 기동: start-prom.sh / stop-prom.sh

### start-prometheus.sh

#!/bin/bash

PROMETHEUS_HOME="/opt/prometheus"
PROMETHEUS_CONFIG_FILE="${PROMETHEUS_HOME}/prometheus.yml"
LOG_DIR="${PROMETHEUS_HOME}"

### create log directory
if [ ! -d "${LOG_DIR}/backup" ]; then
    mkdir -p ${LOG_DIR}/backup
fi

### backup stdout log
DATETIME="$(date +'%Y%m%d_%H%M%S')"
if [ -f "${LOG_DIR}/nohup.prometheus.out" ]; then
    mv ${LOG_DIR}/nohup.prometheus.out ${LOG_DIR}/backup/nohup.prometheus.${DATETIME}.out
fi

### process start
nohup ${PROMETHEUS_HOME}/prometheus \\
    --config.file ${PROMETHEUS_CONFIG_FILE} \\
    --web.enable-admin-api > ${LOG_DIR}/nohup.prometheus.out 2>&1 &

### stop-prometheus.sh

#!/bin/bash

# PROMETHEUS_HOME="/usr/local/prometheus"
# PROMETHEUS_CONFIG_FILE="${PROMETHEUS_HOME}/prometheus.yml"

pkill -9 -ecx prometheus

# 기동
$ ./start-prometheus.sh

1. service등록 후 기동: /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus Server
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Restart=on-failure
ExecStart=/app/monitoring/prometheus/prometheus \\
  --config.file=/app/monitoring/prometheus/prometheus.yml \\
  --storage.tsdb.path=/app/monitoring/prometheus/data \\
  --storage.tsdb.retention.time=30d

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl start prometheus