OpenTelemetry를 통한 자체 호스팅 HAProxy 모니터링

아직 HAProxy stats 엔드포인트를 활성화하지 않았다면 haproxy.cfg에 다음을 추가하십시오:

frontend stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s

설정을 적용하려면 HAProxy를 다시 로드하십시오:

$
sudo systemctl reload haproxy

stats 엔드포인트에 액세스할 수 있는지 확인하십시오:

$
curl -s http://localhost:8404/stats

아직 OpenTelemetry Collector Contrib을 설치하지 않은 경우 OpenTelemetry Collector 릴리스에서 최신 릴리스를 다운로드하여 설치합니다:

Debian/Ubuntu의 경우:

$
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/latest/download/otelcol-contrib_<VERSION>_linux_amd64.deb
$
sudo dpkg -i otelcol-contrib_<VERSION>_linux_amd64.deb

RHEL/Amazon Linux의 경우:

$
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/latest/download/otelcol-contrib_<VERSION>_linux_amd64.rpm
$
sudo rpm -U otelcol-contrib_<VERSION>_linux_amd64.rpm

수집기 설정 파일(/etc/otelcol-contrib/config.yaml)의 내용을 다음으로 교체하십시오:

receivers:
  haproxy:
    endpoint: http://localhost:8404/stats
    collection_interval: 15s

processors:
  resourcedetection:
    detectors: [system]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true

  batch:

exporters:
  otlphttp:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEW_RELIC_LICENSE_KEY}

service:
  pipelines:
    metrics:
      receivers: [haproxy]
      processors: [resourcedetection, batch]
      exporters: [otlphttp]

이 최소 설정은 17개의 기본 메트릭을 모두 수집합니다. haproxyreceiver는 각 메트릭에 haproxy.proxy_name, haproxy.service_name 및 haproxy.addr 속성을 자동으로 추가합니다.

더 깊은 가시성을 위해 추가 메트릭(총 27개: 기본 17개+선택 10개)을 활성화하려면 metrics 섹션을 추가하십시오:

receivers:
  haproxy:
    endpoint: http://localhost:8404/stats
    collection_interval: 15s
    metrics:
      haproxy.connections.average_time:
        enabled: true
      haproxy.requests.average_time:
        enabled: true
      haproxy.responses.average_time:
        enabled: true
      haproxy.active:
        enabled: true
      haproxy.backup:
        enabled: true
      haproxy.downtime:
        enabled: true
      haproxy.failed_checks:
        enabled: true
      haproxy.weight:
        enabled: true
      haproxy.sessions.total:
        enabled: true
      haproxy.connections.total:
        enabled: true

구성 매개변수

필요한 환경 변수를 설정하세요:

$
# Set your New Relic license key
$
export NEW_RELIC_LICENSE_KEY="<YOUR_LICENSE_KEY>"
$

$
# Set the OTLP endpoint. For more information, refer to New Relic OTLP endpoint: https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-otlp
$
export OTEL_EXPORTER_OTLP_ENDPOINT="<YOUR_NEWRELIC_OTLP_ENDPOINT>"

systemd 서비스의 경우, 서비스 환경 파일에 다음을 추가하십시오:

$
echo "NEW_RELIC_LICENSE_KEY=<YOUR_LICENSE_KEY>" | sudo tee -a /etc/otelcol-contrib/otelcol-contrib.conf
$
echo "OTEL_EXPORTER_OTLP_ENDPOINT=<YOUR_NEWRELIC_OTLP_ENDPOINT>" | sudo tee -a /etc/otelcol-contrib/otelcol-contrib.conf

<YOUR_LICENSE_KEY> 라이선스 키로 바꾸세요.

수집기 서비스를 시작(또는 다시 시작)하십시오:

$
sudo systemctl daemon-reload
$
sudo systemctl restart otelcol-contrib

서비스 상태를 확인하세요:

$
sudo systemctl status otelcol-contrib

설정 오류가 있는지 로그를 검토하십시오:

$
sudo journalctl -u otelcol-contrib -n 50 --no-pager

데이터가 유입되도록 2～3분 정도 기다린 후, 뉴렐릭에서 확인하십시오:

FROM Metric SELECT uniques(metricName)
WHERE metricName LIKE 'haproxy.%'
SINCE 10 minutes ago

HAProxy 메트릭이 나타나는 것을 볼 수 있습니다. 차원 속성을 확인하십시오:

FROM Metric SELECT latest(haproxy.sessions.count)
WHERE metricName = 'haproxy.sessions.count'
FACET haproxy.proxy_name, haproxy.service_name
SINCE 5 minutes ago

haproxy.cfg의 통계 프런트엔드에 Prometheus 익스포터를 추가하십시오:

frontend stats
    bind *:8404
    http-request use-service prometheus-exporter if { path /metrics }
    stats enable
    stats uri /stats
    stats refresh 10s

핵심 줄은 http-request use-service prometheus-exporter if { path /metrics } 입니다. ― 이는 /metrics에 대한 requests를 가로채고 HAProxy의 내장 promex 모듈을 통해 Prometheus 형식의 메트릭을 제공합니다.

HAProxy 다시 로드:

$
sudo systemctl reload haproxy

Prometheus 엔드포인트를 확인하십시오:

$
curl -s http://localhost:8404/metrics | head -20

haproxy_frontend_current_sessions 및 haproxy_backend_http_responses_total과(와) 같은 Prometheus 형식의 메트릭을 확인하실 수 있습니다. HAProxy는 이 엔드포인트를 통해 약 204개의 메트릭을 노출합니다.

아직 OpenTelemetry Collector Contrib을 설치하지 않은 경우 OpenTelemetry Collector 릴리스에서 최신 릴리스를 다운로드하여 설치합니다:

Debian/Ubuntu의 경우:

$
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/latest/download/otelcol-contrib_<VERSION>_linux_amd64.deb
$
sudo dpkg -i otelcol-contrib_<VERSION>_linux_amd64.deb

RHEL/Amazon Linux의 경우:

$
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/latest/download/otelcol-contrib_<VERSION>_linux_amd64.rpm
$
sudo rpm -U otelcol-contrib_<VERSION>_linux_amd64.rpm

이 설정은 HAProxy의 Prometheus 엔드포인트를 스크랩하고, 관련 메트릭으로 필터링하며, haproxyreceiver 속성 체계(haproxy.proxy_name, haproxy.service_name)와 일치하도록 레이블을 변환하고, 메트릭의 이름을 OTel haproxy.* 명명 규칙으로 바꿉니다.

이 설정은 NRDOT 및 OTel Collector Contrib 모두에서 작동합니다. 적절한 설정 위치에 배치하십시오:

• NRDOT: /etc/nrdot-collector/config.yaml

• OTel Collector Contrib: /etc/otelcol-contrib/config.yaml

  중요

  편집하기 전에: 기존 설정을 백업하세요.

  bash

  카피

  $
  # NRDOT
  $
  sudo cp /etc/nrdot-collector/config.yaml /etc/nrdot-collector/config.yaml.backup
  $
  # OTel Collector Contrib
  $
  sudo cp /etc/otelcol-contrib/config.yaml /etc/otelcol-contrib/config.yaml.backup

  수집기 설정 파일의 내용을 다음으로 교체하십시오:

  receivers:
    prometheus/haproxy_transform:
      config:
        scrape_configs:
          - job_name: haproxy_transform
            scrape_interval: 30s
            static_configs:
              - targets: ["localhost:8404"]
            metrics_path: /metrics
  
  processors:
    # Filter to keep only the metrics we need to transform
    filter/haproxy:
      metrics:
        include:
          match_type: regexp
          metric_names:
            - "haproxy_frontend_bytes_in_total"
            - "haproxy_frontend_bytes_out_total"
            - "haproxy_frontend_requests_denied_total"
            - "haproxy_frontend_request_errors_total"
            - "haproxy_frontend_responses_denied_total"
            - "haproxy_frontend_http_requests_total"
            - "haproxy_frontend_connections_rate_max"
            - "haproxy_frontend_http_requests_rate_max"
            - "haproxy_frontend_current_sessions"
            - "haproxy_frontend_max_session_rate"
            - "haproxy_backend_bytes_in_total"
            - "haproxy_backend_bytes_out_total"
            - "haproxy_backend_connection_errors_total"
            - "haproxy_backend_retry_warnings_total"
            - "haproxy_backend_requests_denied_total"
            - "haproxy_backend_responses_denied_total"
            - "haproxy_backend_redispatch_warnings_total"
            - "haproxy_backend_http_requests_total"
            - "haproxy_backend_response_errors_total"
            - "haproxy_backend_loadbalanced_total"
            - "haproxy_backend_current_queue"
            - "haproxy_backend_total_time_average_seconds"
            - "haproxy_backend_current_sessions"
            - "haproxy_backend_max_session_rate"
            - "haproxy_server_bytes_in_total"
            - "haproxy_server_bytes_out_total"
            - "haproxy_server_connection_errors_total"
            - "haproxy_server_retry_warnings_total"
            - "haproxy_server_responses_denied_total"
            - "haproxy_server_redispatch_warnings_total"
            - "haproxy_server_response_errors_total"
            - "haproxy_server_loadbalanced_total"
            - "haproxy_server_current_queue"
            - "haproxy_server_total_time_average_seconds"
            - "haproxy_server_current_sessions"
            - "haproxy_server_max_session_rate"
  
    # Add haproxy.service_name and haproxy.proxy_name attributes using OTTL
    transform/add_labels:
      metric_statements:
        - context: datapoint
          statements:
            # Frontend metrics: haproxy.service_name = "FRONTEND"
            - set(attributes["haproxy.service_name"], "FRONTEND") where metric.name == "haproxy_frontend_bytes_in_total"
            - set(attributes["haproxy.service_name"], "FRONTEND") where metric.name == "haproxy_frontend_bytes_out_total"
            - set(attributes["haproxy.service_name"], "FRONTEND") where metric.name == "haproxy_frontend_requests_denied_total"
            - set(attributes["haproxy.service_name"], "FRONTEND") where metric.name == "haproxy_frontend_request_errors_total"
            - set(attributes["haproxy.service_name"], "FRONTEND") where metric.name == "haproxy_frontend_responses_denied_total"
            - set(attributes["haproxy.service_name"], "FRONTEND") where metric.name == "haproxy_frontend_http_requests_total"
            - set(attributes["haproxy.service_name"], "FRONTEND") where metric.name == "haproxy_frontend_connections_rate_max"
            - set(attributes["haproxy.service_name"], "FRONTEND") where metric.name == "haproxy_frontend_http_requests_rate_max"
            - set(attributes["haproxy.service_name"], "FRONTEND") where metric.name == "haproxy_frontend_current_sessions"
            - set(attributes["haproxy.service_name"], "FRONTEND") where metric.name == "haproxy_frontend_max_session_rate"
            # Backend metrics: haproxy.service_name = "BACKEND"
            - set(attributes["haproxy.service_name"], "BACKEND") where metric.name == "haproxy_backend_bytes_in_total"
            - set(attributes["haproxy.service_name"], "BACKEND") where metric.name == "haproxy_backend_bytes_out_total"
            - set(attributes["haproxy.service_name"], "BACKEND") where metric.name == "haproxy_backend_connection_errors_total"
            - set(attributes["haproxy.service_name"], "BACKEND") where metric.name == "haproxy_backend_retry_warnings_total"
            - set(attributes["haproxy.service_name"], "BACKEND") where metric.name == "haproxy_backend_requests_denied_total"
            - set(attributes["haproxy.service_name"], "BACKEND") where metric.name == "haproxy_backend_responses_denied_total"
            - set(attributes["haproxy.service_name"], "BACKEND") where metric.name == "haproxy_backend_redispatch_warnings_total"
            - set(attributes["haproxy.service_name"], "BACKEND") where metric.name == "haproxy_backend_http_requests_total"
            - set(attributes["haproxy.service_name"], "BACKEND") where metric.name == "haproxy_backend_response_errors_total"
            - set(attributes["haproxy.service_name"], "BACKEND") where metric.name == "haproxy_backend_loadbalanced_total"
            - set(attributes["haproxy.service_name"], "BACKEND") where metric.name == "haproxy_backend_current_queue"
            - set(attributes["haproxy.service_name"], "BACKEND") where metric.name == "haproxy_backend_total_time_average_seconds"
            - set(attributes["haproxy.service_name"], "BACKEND") where metric.name == "haproxy_backend_current_sessions"
            - set(attributes["haproxy.service_name"], "BACKEND") where metric.name == "haproxy_backend_max_session_rate"
            # Server metrics: haproxy.service_name = server label value
            - set(attributes["haproxy.service_name"], attributes["server"]) where metric.name == "haproxy_server_bytes_in_total"
            - set(attributes["haproxy.service_name"], attributes["server"]) where metric.name == "haproxy_server_bytes_out_total"
            - set(attributes["haproxy.service_name"], attributes["server"]) where metric.name == "haproxy_server_connection_errors_total"
            - set(attributes["haproxy.service_name"], attributes["server"]) where metric.name == "haproxy_server_retry_warnings_total"
            - set(attributes["haproxy.service_name"], attributes["server"]) where metric.name == "haproxy_server_responses_denied_total"
            - set(attributes["haproxy.service_name"], attributes["server"]) where metric.name == "haproxy_server_redispatch_warnings_total"
            - set(attributes["haproxy.service_name"], attributes["server"]) where metric.name == "haproxy_server_response_errors_total"
            - set(attributes["haproxy.service_name"], attributes["server"]) where metric.name == "haproxy_server_loadbalanced_total"
            - set(attributes["haproxy.service_name"], attributes["server"]) where metric.name == "haproxy_server_current_queue"
            - set(attributes["haproxy.service_name"], attributes["server"]) where metric.name == "haproxy_server_total_time_average_seconds"
            - set(attributes["haproxy.service_name"], attributes["server"]) where metric.name == "haproxy_server_current_sessions"
            - set(attributes["haproxy.service_name"], attributes["server"]) where metric.name == "haproxy_server_max_session_rate"
            # Rename proxy label to haproxy.proxy_name for ALL metrics
            - set(attributes["haproxy.proxy_name"], attributes["proxy"]) where attributes["proxy"] != nil
            # Remove old Prometheus labels
            - delete_key(attributes, "proxy") where attributes["proxy"] != nil
            - delete_key(attributes, "server") where attributes["server"] != nil
  
    # Rename metrics to match haproxyreceiver naming convention
    metricstransform/rename:
      transforms:
        # haproxy.bytes.input
        - include: haproxy_frontend_bytes_in_total
          action: update
          new_name: haproxy.bytes.input
        - include: haproxy_backend_bytes_in_total
          action: update
          new_name: haproxy.bytes.input
        - include: haproxy_server_bytes_in_total
          action: update
          new_name: haproxy.bytes.input
        # haproxy.bytes.output
        - include: haproxy_frontend_bytes_out_total
          action: update
          new_name: haproxy.bytes.output
        - include: haproxy_backend_bytes_out_total
          action: update
          new_name: haproxy.bytes.output
        - include: haproxy_server_bytes_out_total
          action: update
          new_name: haproxy.bytes.output
        # haproxy.connections.errors
        - include: haproxy_backend_connection_errors_total
          action: update
          new_name: haproxy.connections.errors
        - include: haproxy_server_connection_errors_total
          action: update
          new_name: haproxy.connections.errors
        # haproxy.connections.retries
        - include: haproxy_backend_retry_warnings_total
          action: update
          new_name: haproxy.connections.retries
        - include: haproxy_server_retry_warnings_total
          action: update
          new_name: haproxy.connections.retries
        # haproxy.requests.denied
        - include: haproxy_frontend_requests_denied_total
          action: update
          new_name: haproxy.requests.denied
        - include: haproxy_backend_requests_denied_total
          action: update
          new_name: haproxy.requests.denied
        # haproxy.requests.errors
        - include: haproxy_frontend_request_errors_total
          action: update
          new_name: haproxy.requests.errors
        # haproxy.requests.redispatched
        - include: haproxy_backend_redispatch_warnings_total
          action: update
          new_name: haproxy.requests.redispatched
        - include: haproxy_server_redispatch_warnings_total
          action: update
          new_name: haproxy.requests.redispatched
        # haproxy.requests.total
        - include: haproxy_frontend_http_requests_total
          action: update
          new_name: haproxy.requests.total
        - include: haproxy_backend_http_requests_total
          action: update
          new_name: haproxy.requests.total
        # haproxy.responses.denied
        - include: haproxy_frontend_responses_denied_total
          action: update
          new_name: haproxy.responses.denied
        - include: haproxy_backend_responses_denied_total
          action: update
          new_name: haproxy.responses.denied
        - include: haproxy_server_responses_denied_total
          action: update
          new_name: haproxy.responses.denied
        # haproxy.responses.errors
        - include: haproxy_backend_response_errors_total
          action: update
          new_name: haproxy.responses.errors
        - include: haproxy_server_response_errors_total
          action: update
          new_name: haproxy.responses.errors
        # haproxy.server_selected.total
        - include: haproxy_backend_loadbalanced_total
          action: update
          new_name: haproxy.server_selected.total
        - include: haproxy_server_loadbalanced_total
          action: update
          new_name: haproxy.server_selected.total
        # haproxy.connections.rate
        - include: haproxy_frontend_connections_rate_max
          action: update
          new_name: haproxy.connections.rate
        # haproxy.requests.queued
        - include: haproxy_backend_current_queue
          action: update
          new_name: haproxy.requests.queued
        - include: haproxy_server_current_queue
          action: update
          new_name: haproxy.requests.queued
        # haproxy.requests.rate
        - include: haproxy_frontend_http_requests_rate_max
          action: update
          new_name: haproxy.requests.rate
        # haproxy.sessions.average
        - include: haproxy_backend_total_time_average_seconds
          action: update
          new_name: haproxy.sessions.average
        - include: haproxy_server_total_time_average_seconds
          action: update
          new_name: haproxy.sessions.average
        # haproxy.sessions.count
        - include: haproxy_frontend_current_sessions
          action: update
          new_name: haproxy.sessions.count
        - include: haproxy_backend_current_sessions
          action: update
          new_name: haproxy.sessions.count
        - include: haproxy_server_current_sessions
          action: update
          new_name: haproxy.sessions.count
        # haproxy.sessions.rate
        - include: haproxy_frontend_max_session_rate
          action: update
          new_name: haproxy.sessions.rate
        - include: haproxy_backend_max_session_rate
          action: update
          new_name: haproxy.sessions.rate
        - include: haproxy_server_max_session_rate
          action: update
          new_name: haproxy.sessions.rate
  
    # Add resource attributes and strip Prometheus auto-labels
    resource/haproxy_transform:
      attributes:
        - key: haproxy.addr
          value: "http://localhost:8404/stats"
          action: upsert
        - key: service.name
          action: delete
        - key: service.instance.id
          action: delete
        - key: net.host.name
          action: delete
        - key: net.host.port
          action: delete
        - key: server.port
          action: delete
        - key: url.scheme
          action: delete
  
    resourcedetection:
      detectors: [system]
      system:
        resource_attributes:
          host.name:
            enabled: true
          host.id:
            enabled: true
          os.type:
            enabled: true
  
    transform/metadata_nullify:
      metric_statements:
        - context: metric
          statements:
            - set(description, "")
            - set(unit, "")
  
    batch:
  
  exporters:
    otlphttp:
      endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
      headers:
        api-key: ${env:NEW_RELIC_LICENSE_KEY}
  
  service:
    pipelines:
      metrics/haproxy_transform:
        receivers:
          - prometheus/haproxy_transform
        processors:
          - filter/haproxy
          - transform/add_labels
          - metricstransform/rename
          - resourcedetection
          - resource/haproxy_transform
          - transform/metadata_nullify
          - batch
        exporters:
          - otlphttp

  카피

  중요

  이 설정은 세 가지 주요 변환을 수행합니다.

  1. 17개의 OTel 기본 메트릭(프런트엔드/백앤드/서버 티어 전체)에 해당하는 36개의 Prometheus 메트릭으로만 필터링 합니다.
  2. haproxyreceiver의 속성 체계와 일치하는 차원 속성 (haproxy.proxy_name, haproxy.service_name)을 추가합니다.
  3. 일관된 대시보드 및 알림을 위해 Prometheus 명명(haproxy_frontend_*)에서 OTel 명명(haproxy.*)으로 메트릭의 이름을 변경합니다.

  구성 매개변수

  매개변수	설명
  targets	HAProxy Prometheus 엔드포인트 주소(기본값 localhost:8404 )
  haproxy.addr 값	HAProxy 통계 엔드포인트 URL로 교체하십시오.
  scrape_interval	Prometheus 엔드포인트를 스크랩하는 빈도입니다. 기본: 30s

뉴렐릭 자격 증명을 사용하여 systemd 재정의 파일을 생성하십시오:

$
sudo mkdir -p /etc/systemd/system/<collector-service>.service.d

$
cat <<EOF | sudo tee /etc/systemd/system/<collector-service>.service.d/environment.conf
$
[Service]
$
Environment="NEW_RELIC_LICENSE_KEY=<YOUR_LICENSE_KEY>"
$
Environment="OTEL_EXPORTER_OTLP_ENDPOINT=<YOUR_NEWRELIC_OTLP_ENDPOINT>"
$
EOF

<YOUR_LICENSE_KEY>을(를) 라이선스 키(으)로, <collector-service>을(를) 수집기 서비스 이름(예: otelcol-contrib 또는 nrdot-collector)(으)로 바꾸십시오.

$
sudo systemctl daemon-reload
$
sudo systemctl restart <collector-service>

서비스 상태 및 로그를 확인하십시오:

$
sudo systemctl status <collector-service>
$
sudo journalctl -u <collector-service> -n 20

오류 없이 Active: active (running)이(가) 표시되어야 합니다.

몇 분 후, 쿼리 빌더에서 이 NRQL 쿼리를 실행하십시오:

FROM Metric SELECT *
WHERE metricName LIKE 'haproxy.%'
SINCE 10 minutes ago