--- title: Monitor Kubernetes-hosted RabbitMQ with OpenTelemetry source: https://docs.newrelic.com/docs/opentelemetry/integrations/rabbitmq/kubernetes --- Monitor RabbitMQ running in Kubernetes clusters using the OpenTelemetry Collector with automatic pod discovery. This guide walks you through deploying the collector via Helm with dynamic broker detection powered by the Kubernetes observer. > #### 💡 TIP > > Running RabbitMQ on Linux hosts? See the [self-hosted installation guide](https://docs.newrelic.com/docs/opentelemetry/integrations/rabbitmq/self-hosted) for virtual machine or bare metal deployments. ## How the integration works [#architecture] This Kubernetes-native deployment provides automatic discovery and monitoring: **Key components:** - **Kubernetes observer**: Continuously watches for RabbitMQ pods based on label selectors - **Receiver creator**: Dynamically creates RabbitMQ receivers for each discovered pod - **Resource attribution**: Automatically enriches metrics with Kubernetes metadata: - `k8s.cluster.name` - Your cluster identifier - `k8s.namespace.name` - Pod namespace - `k8s.pod.name` - Individual pod name - `rabbitmq.deployment.name` - Derived from pod name **Deployment model:** The collector runs as a Kubernetes deployment (single replica) with RBAC permissions to list and watch pods across namespaces. When new RabbitMQ pods are created or existing ones are removed, the observer automatically updates the collector's configuration. > #### ⚠️ IMPORTANT > > The collector requires `get`, `list`, and `watch` permissions on pods to discover RabbitMQ instances automatically. ## Installation steps [#install] Follow these steps to deploy the OpenTelemetry Collector with RabbitMQ monitoring. ### Before you begin [#requirements] Ensure your environment meets these requirements: **Kubernetes cluster requirements** - **Kubernetes version**: 1.19 or higher - **kubectl access**: Configured with appropriate RBAC permissions to create deployments, services, and RBAC resources - **Helm version**: Helm 3.x installed locally **RabbitMQ requirements** - **Management plugin**: Must be enabled on all RabbitMQ pods - **Service exposure**: RabbitMQ Service must expose the management API port (default: 15672) - **Pod labels**: RabbitMQ pods should have consistent labels (e.g., `app=rabbitmq`) for discovery - **Credentials**: Admin user with permissions to access the management API **New Relic requirements** - **Account**: Active New Relic account - **License key**: [New Relic license key](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key) for your account ### Verify the management endpoint [#verify-endpoint] Ensure RabbitMQ pods have the management plugin enabled: ```bash kubectl exec deploy/ -n -- curl -I -u admin:password http://localhost:15672/api/overview ``` You should see `HTTP/1.1 200 OK` in the response. ### Create Helm values file [#helm-values] Create a file named `otel-collector-values.yaml`. This configuration uses the `receiver_creator` with `k8s_observer` to automatically discover RabbitMQ pods. Update these placeholders: - `my-rabbitmq-cluster`: Your Kubernetes cluster name - `rabbitmq`: The value of the `app` label for your RabbitMQ pods - `admin`/`password`: Your RabbitMQ credentials - `15672`: The management API port ```yaml opentelemetry-collector: mode: deployment image: repository: otel/opentelemetry-collector-contrib pullPolicy: IfNotPresent command: name: otelcol-contrib resources: limits: cpu: 500m memory: 300Mi requests: cpu: 100m memory: 100Mi extraEnvs: - name: NEWRELIC_LICENSE_KEY valueFrom: secretKeyRef: name: newrelic-licenses key: NEWRELIC_LICENSE_KEY - name: NEWRELIC_OTLP_ENDPOINT valueFrom: secretKeyRef: name: newrelic-licenses key: NEWRELIC_OTLP_ENDPOINT - name: K8S_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: K8S_CLUSTER_NAME value: my-rabbitmq-cluster - name: RABBITMQ_USERNAME value: admin - name: RABBITMQ_PASSWORD value: password clusterRole: create: true rules: - apiGroups: [""] resources: ["pods", "nodes", "nodes/stats", "nodes/proxy"] verbs: ["get", "list", "watch"] - apiGroups: ["apps"] resources: ["replicasets", "deployments", "statefulsets"] verbs: ["get", "list", "watch"] clusterRoleBinding: name: "" config: extensions: health_check: endpoint: 0.0.0.0:13133 k8s_observer: auth_type: serviceAccount observe_pods: true observe_nodes: false receivers: receiver_creator/rabbitmq: watch_observers: [k8s_observer] receivers: rabbitmq: # Discover pods with label "app=rabbitmq" # Adjust the label selector to match your RabbitMQ pods rule: type == "pod" && labels["app"] == "rabbitmq" config: endpoint: 'http://`endpoint`:15672' username: ${env:RABBITMQ_USERNAME} password: ${env:RABBITMQ_PASSWORD} collection_interval: 30s metrics: # Queue Metrics (essential for message flow and backlog) rabbitmq.consumer.count: enabled: true rabbitmq.message.delivered: enabled: true rabbitmq.message.published: enabled: true rabbitmq.message.acknowledged: enabled: true rabbitmq.message.dropped: enabled: true rabbitmq.message.current: enabled: true # Crucial for monitoring queue backlog, includes 'ready' and 'unacknowledged' states # Node Health Metrics (critical for server resource monitoring) rabbitmq.node.disk_free: enabled: true rabbitmq.node.disk_free_limit: enabled: true rabbitmq.node.disk_free_alarm: enabled: true rabbitmq.node.mem_used: enabled: true rabbitmq.node.mem_limit: enabled: true rabbitmq.node.mem_alarm: enabled: true rabbitmq.node.mem_used_details.rate: enabled: true rabbitmq.node.fd_used: enabled: true rabbitmq.node.fd_total: enabled: true rabbitmq.node.sockets_used: enabled: true rabbitmq.node.sockets_total: enabled: true rabbitmq.node.proc_used: enabled: true rabbitmq.node.proc_total: enabled: true rabbitmq.node.uptime: enabled: true rabbitmq.node.run_queue: enabled: true rabbitmq.node.processors: enabled: true rabbitmq.node.context_switches_details.rate: enabled: true rabbitmq.node.gc_num_details.rate: enabled: true rabbitmq.node.gc_bytes_reclaimed_details.rate: enabled: true # I/O Metrics (important for understanding disk and network activity) rabbitmq.node.io_read_count_details.rate: enabled: true rabbitmq.node.io_read_bytes_details.rate: enabled: true rabbitmq.node.io_read_avg_time_details.rate: enabled: true rabbitmq.node.io_write_count_details.rate: enabled: true rabbitmq.node.io_write_bytes_details.rate: enabled: true rabbitmq.node.io_write_avg_time_details.rate: enabled: true rabbitmq.node.io_sync_count_details.rate: enabled: true rabbitmq.node.io_sync_avg_time_details.rate: enabled: true rabbitmq.node.io_seek_count_details.rate: enabled: true rabbitmq.node.io_seek_avg_time_details.rate: enabled: true rabbitmq.node.io_reopen_count_details.rate: enabled: true # Mnesia and Store Metrics (for internal database and message storage) rabbitmq.node.mnesia_ram_tx_count_details.rate: enabled: true rabbitmq.node.mnesia_disk_tx_count_details.rate: enabled: true rabbitmq.node.msg_store_read_count_details.rate: enabled: true rabbitmq.node.msg_store_write_count_details.rate: enabled: true rabbitmq.node.queue_index_write_count_details.rate: enabled: true rabbitmq.node.queue_index_read_count_details.rate: enabled: true # Connection/Channel/Queue Lifecycle Metrics rabbitmq.node.connection_created_details.rate: enabled: true rabbitmq.node.connection_closed_details.rate: enabled: true rabbitmq.node.channel_created_details.rate: enabled: true rabbitmq.node.channel_closed_details.rate: enabled: true rabbitmq.node.queue_declared_details.rate: enabled: true rabbitmq.node.queue_created_details.rate: enabled: true rabbitmq.node.queue_deleted_details.rate: enabled: true resource_attributes: rabbitmq.server.endpoint: 'http://`endpoint`:15672' rabbitmq.port: '15672' processors: batch: send_batch_size: 1024 timeout: 30s resource/cluster: attributes: - key: k8s.cluster.name value: ${env:K8S_CLUSTER_NAME} action: upsert transform/rabbitmq: metric_statements: - context: resource statements: # Create a display name combining Kubernetes metadata - set(attributes["rabbitmq.display.name"], Concat([ "server", "k8s", attributes["k8s.cluster.name"], attributes["k8s.namespace.name"], "pod", attributes["k8s.pod.name"], "rabbitmq", attributes["rabbitmq.port"] ], ":")) # Use pod name as deployment name - set(attributes["rabbitmq.deployment.name"], attributes["k8s.pod.name"]) exporters: otlphttp: endpoint: "${NEWRELIC_OTLP_ENDPOINT}" headers: api-key: "${NEWRELIC_LICENSE_KEY}" compression: gzip service: extensions: [health_check, k8s_observer] pipelines: metrics/rabbitmq: receivers: [receiver_creator/rabbitmq] processors: [batch, resource/cluster, transform/rabbitmq] exporters: [otlphttp] ``` > #### 💡 TIP > > **Label selector customization:** > If your RabbitMQ pods use different labels, update the `rule` line. For example: > > - StatefulSet pods: `type == "pod" && labels["app.kubernetes.io/name"] == "rabbitmq"` > - Operator deployments: `type == "pod" && labels["app.kubernetes.io/component"] == "rabbitmq"` > > Run `kubectl get pods --show-labels -n ` to view your pod labels. ### Create Kubernetes secret [#create-secret] Store your New Relic credentials in a Kubernetes secret. Select the region that matches your New Relic account (check your browser URL when logged into New Relic): #### US Region (default) ```bash kubectl create secret generic newrelic-licenses \ --from-literal=NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY \ --from-literal=NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318 \ --namespace newrelic ``` Replace `YOUR_LICENSE_KEY` with your [New Relic license key](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key). #### EU Region ```bash kubectl create secret generic newrelic-licenses \ --from-literal=NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY \ --from-literal=NEWRELIC_OTLP_ENDPOINT=https://otlp.eu01.nr-data.net:4318 \ --namespace newrelic ``` Replace `YOUR_LICENSE_KEY` with your [New Relic license key](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key). #### JP Region ```bash kubectl create secret generic newrelic-licenses \ --from-literal=NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY \ --from-literal=NEWRELIC_OTLP_ENDPOINT=https://otlp.jp.nr-data.net:4318 \ --namespace newrelic ``` Replace `YOUR_LICENSE_KEY` with your [New Relic license key](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key). ### Deploy with Helm [#deploy-helm] Add the OpenTelemetry Helm repository: ```bash helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm repo update ``` Install or upgrade the collector: ```bash helm upgrade --install rabbitmq-otel-collector \ open-telemetry/opentelemetry-collector \ --namespace newrelic \ --create-namespace \ --values otel-collector-values.yaml ``` This creates: - A deployment named `rabbitmq-otel-collector` in the `newrelic` namespace - A Serviceaccount with RBAC permissions to watch pods - ConfigMaps containing the collector configuration ### Verify the deployment and data flow [#verify] Check that the collector pod is running: ```bash kubectl get pods -n newrelic -l app.kubernetes.io/name=opentelemetry-collector ``` Expected output: ``` NAME READY STATUS RESTARTS AGE rabbitmq-otel-collector-6d8c5c5d8d-abc12 1/1 Running 0 2m ``` View collector logs to verify pod discovery: ```bash kubectl logs deploy/rabbitmq-otel-collector -n newrelic --tail=50 ``` Look for messages indicating successful pod discovery and metric collection: ``` INFO k8sobserver/extension.go:150 Discovered pod {"kind": "pod", "name": "rabbitmq-0", "namespace": "default"} INFO RabbitmqReceiver Successfully scraped rabbitmq metrics from pod rabbitmq-0 ``` Verify your data in New Relic: Wait 2-3 minutes for data to appear, then run this query in the [query builder](https://docs.newrelic.com/docs/query-your-data/explore-query-data/query-builder/introduction-query-builder): ```sql SELECT count(*) FROM Metric WHERE metricName LIKE 'rabbitmq.%' AND instrumentation.provider = 'opentelemetry' AND k8s.cluster.name = 'my-rabbitmq-cluster' FACET k8s.pod.name, metricName SINCE 10 minutes ago ``` You should see metrics from each RabbitMQ pod with Kubernetes attributes: - `k8s.cluster.name` - Your cluster identifier - `k8s.namespace.name` - Pod namespace - `k8s.pod.name` - Individual pod name - `rabbitmq.deployment.name` - Derived from pod name > #### 💡 TIP > > If you don't see data after 5 minutes, check the [troubleshooting section](#troubleshoot) below. ## Troubleshooting [#troubleshoot] **No data appearing in New Relic** If metrics aren't appearing in New Relic after deployment: **1. Verify RabbitMQ management plugin is enabled:** ````bash kubectl exec -n -- rabbitmq-plugins list | grep management ``` You should see `[E*] rabbitmq_management` indicating it's enabled. **2. Check collector pod logs:** ```bash kubectl logs deploy/rabbitmq-otel-collector -n newrelic --tail=100 ``` Look for: * Pod discovery messages * Authentication errors (incorrect credentials) * Connection errors (network issues) * Parsing errors (configuration problems) **3. Verify RabbitMQ service is accessible:** ```bash kubectl get svc -n ``` Ensure there's a service exposing port 15672. **4. Test management API from within the cluster:** ```bash kubectl run test-pod --image=curlimages/curl:latest --rm -it --restart=Never -- \ curl -I -u admin:password http://..svc.cluster.local:15672/api/overview ``` Should return `HTTP/1.1 200 OK`. **5. Check RBAC permissions:** ```bash kubectl auth can-i list pods --as=system:serviceaccount:newrelic:rabbitmq-otel-collector -n ``` Should return `yes`. If not, verify the ClusterRole and ClusterRoleBinding. ```` **Pod discovery not working** If the collector isn't discovering RabbitMQ pods automatically: **1. Verify label selector matches your pods:** ````bash kubectl get pods --show-labels -n ``` Compare the labels with the `rule` in your Helm values file. **2. Check k8s_observer logs:** ```bash kubectl logs deploy/rabbitmq-otel-collector -n newrelic | grep k8sobserver ``` Look for messages about discovered pods. **3. Test the label selector:** ```bash kubectl get pods -l app=rabbitmq --all-namespaces ``` This should list your RabbitMQ pods. If empty, your label selector is incorrect. **4. Verify ClusterRole permissions:** ```bash kubectl describe clusterrole rabbitmq-otel-collector ``` Ensure it has `get`, `list`, and `watch` permissions on pods. **Common label patterns:** * Standard deployments: `app=rabbitmq` * StatefulSets: `app.kubernetes.io/name=rabbitmq` * Operators: `app.kubernetes.io/component=rabbitmq` ```` **Connection errors** If you see connection errors in the collector logs: **Check service endpoints:** ````bash kubectl get endpoints -n ``` Verify that endpoints exist for your RabbitMQ service. **Verify network policies:** Ensure network policies allow traffic from the `newrelic` namespace to your RabbitMQ namespace. **Test connectivity:** ```bash kubectl run test-pod --image=curlimages/curl:latest --rm -it --restart=Never -n newrelic -- \ curl http://..svc.cluster.local:15672/api/overview ``` **Check DNS resolution:** ```bash kubectl run test-pod --image=busybox:latest --rm -it --restart=Never -n newrelic -- \ nslookup ..svc.cluster.local ``` ```` **High cardinality warnings** If you see high cardinality warnings in New Relic: **Filter to specific queues:** If monitoring many queues, consider filtering to only critical ones. **Increase collection interval:** ````yaml receivers: receiver_creator/rabbitmq: receivers: rabbitmq: config: collection_interval: 60s # Increased from 30s ``` **Use metric aggregation:** Configure the batch processor to aggregate metrics before export. **Disable less critical metrics:** Comment out metrics you don't need in the receiver configuration. **Additional optimization tips:** * Use more specific label selectors to monitor only production pods * Filter specific queues by name instead of monitoring all queues * Increase the `collection_interval` to reduce data points * Consider using metric aggregation in the collector ```` **Collector pod crashing or restarting** If the collector pod is crashing: **Check resource limits:** ````bash kubectl describe pod -n newrelic -l app.kubernetes.io/name=opentelemetry-collector ``` Look for OOMKilled status. If so, increase memory limits: ```yaml resources: limits: memory: 512Mi # Increased from 300Mi ``` **Review crash logs:** ```bash kubectl logs -n newrelic -l app.kubernetes.io/name=opentelemetry-collector --previous ``` **Check configuration syntax:** Ensure your YAML configuration is valid and properly indented. ```` **Authentication failures** If you see `401 Unauthorized` errors: **Verify credentials in the Helm values match your RabbitMQ setup:** ````bash kubectl exec -n -- rabbitmqctl list_users ``` **Check that the RabbitMQ user has administrator privileges** and ensure credentials don't contain special characters that need escaping. ```` ## What's next? [#whats-next] Now that you have RabbitMQ monitoring set up, you can enhance your observability: **Explore your data:** - [Explore RabbitMQ metrics](https://docs.newrelic.com/docs/opentelemetry/integrations/rabbitmq/metrics-reference) - Complete list of metrics with NRQL query examples **Enhance monitoring:** - [Create alerts](https://docs.newrelic.com/docs/alerts-applied-intelligence/new-relic-alerts/get-started/introduction-alerts) - Set up alerts for queue depths and pod health - [Build dashboards](https://docs.newrelic.com/docs/query-your-data/explore-query-data/dashboards/introduction-dashboards) - Create custom dashboards to visualize your RabbitMQ metrics