---
title: Monitor self-hosted Elasticsearch with OpenTelemetry
source: https://docs.newrelic.com/docs/opentelemetry/integrations/elasticsearch/elasticsearch-otel-integration-install
---

Monitor your self-hosted Elasticsearch cluster by installing the OpenTelemetry Collector directly on servers or virtual machines. New Relic provides flexible deployment options to match your infrastructure setup and monitoring requirements.

You can choose between three collector options:

-   **NRDOT:** New Relic Distribution of OpenTelemetry
-   **OTel Collector Contrib:** Standard OpenTelemetry Collector with community-contributed components
-   **Prometheus Receiver:** For environments already running a [Prometheus Elasticsearch exporter](https://github.com/prometheus-community/elasticsearch_exporter)

## Installation options [#installation-options]

Choose the collector distribution that matches your needs:

### NRDOT Collector

#### Before you begin [#prereq-nrdot]

Before configuring Elasticsearch monitoring with NRDOT, ensure you have:

**NRDOT collector installed:**

-   You must have NRDOT collector installed and running as a systemd service on your host
-   If not installed, follow the [official NRDOT installation guide](https://docs.newrelic.com/docs/opentelemetry/nrdot/nrdot-collector/) to install the collector using the package manager (DEB/RPM) which sets up the systemd service automatically

**Required access privileges:**

-   Your New Relic license key
-   Root or sudo privileges on the host machine
-   Elasticsearch cluster admin privileges with `monitor` or `manage` cluster privilege (see [Elasticsearch security privileges documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html) for details)

**System requirements:**

-   **Elasticsearch version 7.16 or higher** - This integration requires a modern Elasticsearch cluster
-   **Network connectivity** - Outbound HTTPS (port 443) to New Relic's [OTLP ingest endpoint](https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-otlp)

**Configuration values ready:**

-   **Elasticsearch endpoint** - Your Elasticsearch cluster URL (e.g., `http://localhost:9200`)
-   **Cluster name** - A unique identifier for your cluster in New Relic

#### Configure Elasticsearch monitoring [#configure-es-nrdot]

Once the NRDOT collector is installed, replace the collector's default configuration file with the Elasticsearch monitoring configuration. This will enable Elasticsearch metrics collection. Host metrics and logs are optional and can be added separately.

The configuration file is located at: `/etc/nrdot-collector/config.yaml`

> #### 💡 TIP
>
> **Backup your default configuration:** Before modifying the configuration file, create a backup copy to preserve the default settings:
>
> ```bash
> sudo cp /etc/nrdot-collector/config.yaml /etc/nrdot-collector/config.yaml.backup
> ```

To configure the collector:

1.  Open the configuration file with a text editor using root or sudo privileges:

    ```bash
    sudo nano /etc/nrdot-collector/config.yaml
    ```

2.  Delete all existing content and replace it with the following configuration for Elasticsearch monitoring:

> #### ⚠️ IMPORTANT
>
> Replace the `endpoint` value with your Elasticsearch cluster endpoint and update `elasticsearch.cluster.name` in the processor block with a unique name to identify your cluster in New Relic.

```yaml
receivers:
  elasticsearch:
    endpoint: "http://localhost:9200"
    collection_interval: 15s
    metrics:
      elasticsearch.os.cpu.usage:
        enabled: true
      elasticsearch.cluster.data_nodes:
        enabled: true
      elasticsearch.cluster.health:
        enabled: true
      elasticsearch.cluster.in_flight_fetch:
        enabled: true
      elasticsearch.cluster.nodes:
        enabled: true
      elasticsearch.cluster.pending_tasks:
        enabled: true
      elasticsearch.cluster.shards:
        enabled: true
      elasticsearch.cluster.state_update.time:
        enabled: true
      elasticsearch.index.documents:
        enabled: true
      elasticsearch.index.operations.merge.current:
        enabled: true
      elasticsearch.index.operations.time:
        enabled: true
      elasticsearch.node.cache.count:
        enabled: true
      elasticsearch.node.cache.evictions:
        enabled: true
      elasticsearch.node.cache.memory.usage:
        enabled: true
      elasticsearch.node.shards.size:
        enabled: true
      elasticsearch.node.cluster.io:
        enabled: true
      elasticsearch.node.documents:
        enabled: true
      elasticsearch.node.disk.io.read:
        enabled: true
      elasticsearch.node.disk.io.write:
        enabled: true
      elasticsearch.node.fs.disk.available:
        enabled: true
      elasticsearch.node.fs.disk.total:
        enabled: true
      elasticsearch.node.http.connections:
        enabled: true
      elasticsearch.node.ingest.documents.current:
        enabled: true
      elasticsearch.node.ingest.operations.failed:
        enabled: true
      elasticsearch.node.open_files:
        enabled: true
      elasticsearch.node.operations.completed:
        enabled: true
      elasticsearch.node.operations.current:
        enabled: true
      elasticsearch.node.operations.get.completed:
        enabled: true
      elasticsearch.node.operations.time:
        enabled: true
      elasticsearch.node.shards.reserved.size:
        enabled: true
      elasticsearch.index.shards.size:
        enabled: true
      elasticsearch.os.cpu.load_avg.1m:
        enabled: true
      elasticsearch.os.cpu.load_avg.5m:
        enabled: true
      elasticsearch.os.cpu.load_avg.15m:
        enabled: true
      elasticsearch.os.memory:
        enabled: true
      jvm.gc.collections.count:
        enabled: true
      jvm.gc.collections.elapsed:
        enabled: true
      jvm.memory.heap.max:
        enabled: true
      jvm.memory.heap.used:
        enabled: true
      jvm.memory.heap.utilization:
        enabled: true
      jvm.threads.count:
        enabled: true
      elasticsearch.index.segments.count:
        enabled: true
      elasticsearch.index.operations.completed:
        enabled: true
      elasticsearch.node.script.cache_evictions:
        enabled: false
      elasticsearch.node.cluster.connections:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.preprocessed:
        enabled: false
      elasticsearch.node.thread_pool.tasks.queued:
        enabled: false
      elasticsearch.cluster.published_states.full:
        enabled: false
      jvm.memory.pool.max:
        enabled: false
      elasticsearch.node.script.compilation_limit_triggered:
        enabled: false
      elasticsearch.node.shards.data_set.size:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.current:
        enabled: false
      elasticsearch.cluster.state_update.count:
        enabled: false
      elasticsearch.node.fs.disk.free:
        enabled: false
      jvm.memory.nonheap.used:
        enabled: false
      jvm.memory.pool.used:
        enabled: false
      elasticsearch.node.translog.size:
        enabled: false
      elasticsearch.node.thread_pool.threads:
        enabled: false
      elasticsearch.cluster.state_queue:
        enabled: false
      elasticsearch.node.translog.operations:
        enabled: false
      elasticsearch.memory.indexing_pressure:
        enabled: false
      elasticsearch.node.ingest.documents:
        enabled: false
      jvm.classes.loaded:
        enabled: false
      jvm.memory.heap.committed:
        enabled: false
      elasticsearch.breaker.memory.limit:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.replica_rejections:
        enabled: false
      elasticsearch.breaker.memory.estimated:
        enabled: false
      elasticsearch.cluster.published_states.differences:
        enabled: false
      jvm.memory.nonheap.committed:
        enabled: false
      elasticsearch.node.translog.uncommitted.size:
        enabled: false
      elasticsearch.node.script.compilations:
        enabled: false
      elasticsearch.node.pipeline.ingest.operations.failed:
        enabled: false
      elasticsearch.indexing_pressure.memory.limit:
        enabled: false
      elasticsearch.breaker.tripped:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.primary_rejections:
        enabled: false
      elasticsearch.node.thread_pool.tasks.finished:
        enabled: false
processors:
  memory_limiter:
    check_interval: 60s
    limit_mib: ${env:NEW_RELIC_MEMORY_LIMIT_MIB:-100}
  cumulativetodelta: {}
  attributes/cluster_state_aggregate:
    include:
      match_type: strict
      metric_names:
        - elasticsearch.cluster.state_update.time
    actions:
      - key: type
        action: delete
      - key: state
        action: delete
  filter/critical_operations:
    metrics:
      datapoint:
        # Filters to keep only: query, index, get, merge
        # Affects only 4 metrics: *.operations.completed and *.operations.time
        # All other metrics pass through unchanged
        - 'attributes["operation"] == "query" or attributes["operation"] == "index" or attributes["operation"] == "get" or attributes["operation"] == "merge" or attributes["operation"] == nil'
  resource/cluster_name_override:
    attributes:
      - key: elasticsearch.cluster.name
        value: "<elasticsearch-cluster-name>"
        action: upsert
  resourcedetection:
    detectors: [ system ]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
        os.type:
          enabled: true
  batch:
    timeout: 30s
    send_batch_size: 2048
    send_batch_max_size: 4096
  attributes/cardinality_reduction:
    actions:
      - key: process.pid
        action: delete
      - key: process.parent_pid
        action: delete
  transform/metadata_nullify:
    metric_statements:
      - context: metric
        statements:
          - set(description, "")
          - set(unit, "")
exporters:
  otlphttp:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}
    compression: gzip
    timeout: 30s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [memory_limiter, resourcedetection, resource/cluster_name_override, attributes/cardinality_reduction, filter/critical_operations, attributes/cluster_state_aggregate, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]
```

3.  _(Optional)_ For secured Elasticsearch with authentication and SSL, modify the receiver configuration:

    ```yaml
    receivers:
      elasticsearch:
        endpoint: "https://localhost:9200"
        username: "your_elasticsearch_username"
        password: "your_elasticsearch_password"
        tls:
          ca_file: "/etc/elasticsearch/certs/http_ca.crt"
          insecure_skip_verify: false
        collection_interval: 15s
    ```

4.  _(Optional)_ To collect host metrics, add the hostmetrics receiver:

    ```yaml
    receivers:
      hostmetrics:
        collection_interval: 60s
        scrapers:
          cpu:
            metrics:
              system.cpu.utilization: {enabled: true}
              system.cpu.time: {enabled: true}
          load:
            metrics:
              system.cpu.load_average.1m: {enabled: true}
              system.cpu.load_average.5m: {enabled: true}
              system.cpu.load_average.15m: {enabled: true}
          memory:
            metrics:
              system.memory.usage: {enabled: true}
              system.memory.utilization: {enabled: true}
          disk:
            metrics:
              system.disk.io: {enabled: true}
              system.disk.operations: {enabled: true}
          filesystem:
            metrics:
              system.filesystem.usage: {enabled: true}
              system.filesystem.utilization: {enabled: true}
          network:
            metrics:
              system.network.io: {enabled: true}
              system.network.packets: {enabled: true}
          process:
            metrics:
              process.cpu.utilization:
                enabled: true
    ```

    And add to the service pipelines:

    ```yaml
    service:
      pipelines:
        metrics/host:
          receivers: [hostmetrics]
          processors: [memory_limiter, resourcedetection, batch]
          exporters: [otlphttp]
    ```

5.  _(Optional)_ To collect Elasticsearch logs, add the filelog receiver. Ensure the user running the collector service (nrdot-collector) has read access to your Elasticsearch log files:

    **If running Elasticsearch on Linux (Host):**

    ```yaml
    receivers:
      filelog:
        include:
          - /var/log/elasticsearch/elasticsearch.log
          - /var/log/elasticsearch/*.log
    ```

    **If running Elasticsearch in Docker:**

    ```yaml
    receivers:
      filelog:
        include:
          - /var/lib/docker/containers/*/*.log
        operators:
          - type: move
            from: attributes.log
            to: body
    ```

    And add to the service pipelines:

    ```yaml
    service:
      pipelines:
        logs:
          receivers: [filelog]
          processors: [resource/cluster_name_override]
          exporters: [otlphttp]
    ```

6.  _(Optional)_ To add custom metadata tags to your metrics, use the `resource/static_override` processor:

    ```yaml
    processors:
      resource/static_override:
        attributes:
          - key: env
            value: "production"
            action: upsert
    service:
      pipelines:
        metrics/elasticsearch:
          receivers: [elasticsearch]
          processors: [memory_limiter, resourcedetection, resource/cluster_name_override, resource/static_override, attributes/cardinality_reduction, cumulativetodelta, transform/metadata_nullify, batch]
          exporters: [otlphttp]
    ```

7.  Save the configuration file.

8.  Set the environment variables:

    Create a systemd override directory:

    ```bash
    sudo mkdir -p /etc/systemd/system/nrdot-collector.service.d
    ```

    Create the environment configuration file:

    ```bash
    cat <<EOF | sudo tee /etc/systemd/system/nrdot-collector.service.d/environment.conf
    [Service]
    Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318"
    Environment="NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY_HERE"
    Environment="NEW_RELIC_MEMORY_LIMIT_MIB=100"
    EOF
    ```

    Update the configuration with your values:

    -   Replace `https://otlp.nr-data.net:4318` with your region's endpoint
    -   Replace `YOUR_LICENSE_KEY_HERE` with your actual New Relic license key
    -   Replace `100` with your desired memory limit in MiB for the collector (default: 100 MiB). Adjust based on your environment's needs

9.  Restart the NRDOT collector to apply changes:

    ```bash
    sudo systemctl daemon-reload
    sudo systemctl restart nrdot-collector.service
    ```

#### Verify data collection [#verify-nrdot]

Verify that the NRDOT collector is running and collecting data without errors:

1.  Check the collector service status:

    ```bash
    sudo systemctl status nrdot-collector.service
    ```

2.  Monitor the collector logs for any errors:

    ```bash
    sudo journalctl -u nrdot-collector.service -f
    ```

    Look for successful connections to Elasticsearch and New Relic. If you see errors, refer to the [troubleshooting guide](https://docs.newrelic.com/docs/opentelemetry/integrations/elasticsearch/troubleshooting).

### OTel Collector Contrib

#### Before you begin [#prereq-otel]

Before configuring the OTel Collector Contrib, ensure you have:

**Required access privileges:**

-   Your New Relic license key
-   Root or sudo privileges on the host machine
-   Elasticsearch cluster admin privileges with `monitor` or `manage` cluster privilege (see [Elasticsearch security privileges documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html) for details)

**System requirements:**

-   **Elasticsearch version 7.16 or higher** - This integration requires a modern Elasticsearch cluster
-   **Network connectivity** - Outbound HTTPS (port 443) to New Relic's [OTLP ingest endpoint](https://docs.newrelic.com/docs/opentelemetry/best-practices/opentelemetry-otlp)

**Configuration values ready:**

-   **Elasticsearch endpoint** - Your Elasticsearch cluster URL (e.g., `http://localhost:9200`)
-   **Cluster name** - A unique identifier for your cluster in New Relic

> #### ⚠️ IMPORTANT
>
> You must have [OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/releases/latest) installed on your host before proceeding. We recommend installing via official packages (.deb or .rpm) to ensure the systemd service unit is created correctly.

#### Configure Elasticsearch monitoring [#configure-es-otel]

Once the OTel Collector Contrib is installed, replace the collector's default configuration file with the Elasticsearch monitoring configuration. This will enable Elasticsearch metrics collection. Host metrics and logs are optional and can be added separately.

The configuration file is located at: `/etc/otelcol-contrib/config.yaml`

> #### 💡 TIP
>
> **Backup your default configuration:** Before modifying the configuration file, create a backup copy to preserve the default settings:
>
> ```bash
> sudo cp /etc/otelcol-contrib/config.yaml /etc/otelcol-contrib/config.yaml.backup
> ```

To configure the collector:

1.  Open the configuration file with a text editor using root or sudo privileges:

    ```bash
    sudo nano /etc/otelcol-contrib/config.yaml
    ```

2.  Delete all existing content and replace it with the following configuration for Elasticsearch monitoring:

> #### ⚠️ IMPORTANT
>
> Replace the `endpoint` value with your Elasticsearch cluster endpoint and update `elasticsearch.cluster.name` in the processor block with a unique name to identify your cluster in New Relic.

```yaml
receivers:
  elasticsearch:
    endpoint: "http://localhost:9200"
    collection_interval: 15s
    metrics:
      elasticsearch.os.cpu.usage:
        enabled: true
      elasticsearch.cluster.data_nodes:
        enabled: true
      elasticsearch.cluster.health:
        enabled: true
      elasticsearch.cluster.in_flight_fetch:
        enabled: true
      elasticsearch.cluster.nodes:
        enabled: true
      elasticsearch.cluster.pending_tasks:
        enabled: true
      elasticsearch.cluster.shards:
        enabled: true
      elasticsearch.cluster.state_update.time:
        enabled: true
      elasticsearch.index.documents:
        enabled: true
      elasticsearch.index.operations.merge.current:
        enabled: true
      elasticsearch.index.operations.time:
        enabled: true
      elasticsearch.node.cache.count:
        enabled: true
      elasticsearch.node.cache.evictions:
        enabled: true
      elasticsearch.node.cache.memory.usage:
        enabled: true
      elasticsearch.node.shards.size:
        enabled: true
      elasticsearch.node.cluster.io:
        enabled: true
      elasticsearch.node.documents:
        enabled: true
      elasticsearch.node.disk.io.read:
        enabled: true
      elasticsearch.node.disk.io.write:
        enabled: true
      elasticsearch.node.fs.disk.available:
        enabled: true
      elasticsearch.node.fs.disk.total:
        enabled: true
      elasticsearch.node.http.connections:
        enabled: true
      elasticsearch.node.ingest.documents.current:
        enabled: true
      elasticsearch.node.ingest.operations.failed:
        enabled: true
      elasticsearch.node.open_files:
        enabled: true
      elasticsearch.node.operations.completed:
        enabled: true
      elasticsearch.node.operations.current:
        enabled: true
      elasticsearch.node.operations.get.completed:
        enabled: true
      elasticsearch.node.operations.time:
        enabled: true
      elasticsearch.node.shards.reserved.size:
        enabled: true
      elasticsearch.index.shards.size:
        enabled: true
      elasticsearch.os.cpu.load_avg.1m:
        enabled: true
      elasticsearch.os.cpu.load_avg.5m:
        enabled: true
      elasticsearch.os.cpu.load_avg.15m:
        enabled: true
      elasticsearch.os.memory:
        enabled: true
      jvm.gc.collections.count:
        enabled: true
      jvm.gc.collections.elapsed:
        enabled: true
      jvm.memory.heap.max:
        enabled: true
      jvm.memory.heap.used:
        enabled: true
      jvm.memory.heap.utilization:
        enabled: true
      jvm.threads.count:
        enabled: true
      elasticsearch.index.segments.count:
        enabled: true
      elasticsearch.index.operations.completed:
        enabled: true
      elasticsearch.node.script.cache_evictions:
        enabled: false
      elasticsearch.node.cluster.connections:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.preprocessed:
        enabled: false
      elasticsearch.node.thread_pool.tasks.queued:
        enabled: false
      elasticsearch.cluster.published_states.full:
        enabled: false
      jvm.memory.pool.max:
        enabled: false
      elasticsearch.node.script.compilation_limit_triggered:
        enabled: false
      elasticsearch.node.shards.data_set.size:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.current:
        enabled: false
      elasticsearch.cluster.state_update.count:
        enabled: false
      elasticsearch.node.fs.disk.free:
        enabled: false
      jvm.memory.nonheap.used:
        enabled: false
      jvm.memory.pool.used:
        enabled: false
      elasticsearch.node.translog.size:
        enabled: false
      elasticsearch.node.thread_pool.threads:
        enabled: false
      elasticsearch.cluster.state_queue:
        enabled: false
      elasticsearch.node.translog.operations:
        enabled: false
      elasticsearch.memory.indexing_pressure:
        enabled: false
      elasticsearch.node.ingest.documents:
        enabled: false
      jvm.classes.loaded:
        enabled: false
      jvm.memory.heap.committed:
        enabled: false
      elasticsearch.breaker.memory.limit:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.replica_rejections:
        enabled: false
      elasticsearch.breaker.memory.estimated:
        enabled: false
      elasticsearch.cluster.published_states.differences:
        enabled: false
      jvm.memory.nonheap.committed:
        enabled: false
      elasticsearch.node.translog.uncommitted.size:
        enabled: false
      elasticsearch.node.script.compilations:
        enabled: false
      elasticsearch.node.pipeline.ingest.operations.failed:
        enabled: false
      elasticsearch.indexing_pressure.memory.limit:
        enabled: false
      elasticsearch.breaker.tripped:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.primary_rejections:
        enabled: false
      elasticsearch.node.thread_pool.tasks.finished:
        enabled: false
processors:
  memory_limiter:
    check_interval: 60s
    limit_mib: ${env:NEW_RELIC_MEMORY_LIMIT_MIB:-100}
  cumulativetodelta: {}
  resource/cluster_name_override:
    attributes:
      - key: elasticsearch.cluster.name
        value: "<elasticsearch-cluster-name>"
        action: upsert
  resourcedetection:
    detectors: [ system ]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
        os.type:
          enabled: true
  batch:
    timeout: 10s
    send_batch_size: 1024
  attributes/cardinality_reduction:
    actions:
      - key: process.pid
        action: delete
      - key: process.parent_pid
        action: delete
  transform/metadata_nullify:
    metric_statements:
      - context: metric
        statements:
          - set(description, "")
          - set(unit, "")
exporters:
  otlphttp:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [memory_limiter, resourcedetection, resource/cluster_name_override, attributes/cardinality_reduction, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]
```

3.  _(Optional)_ For secured Elasticsearch with authentication and SSL, modify the receiver configuration:

    ```yaml
    receivers:
      elasticsearch:
        endpoint: "https://localhost:9200"
        username: "your_elasticsearch_username"
        password: "your_elasticsearch_password"
        tls:
          ca_file: "/etc/elasticsearch/certs/http_ca.crt"
          insecure_skip_verify: false
        collection_interval: 15s
    ```

4.  _(Optional)_ To collect host metrics, add the hostmetrics receiver:

    ```yaml
    receivers:
      hostmetrics:
        collection_interval: 60s
        scrapers:
          cpu:
            metrics:
              system.cpu.utilization: {enabled: true}
              system.cpu.time: {enabled: true}
          load:
            metrics:
              system.cpu.load_average.1m: {enabled: true}
              system.cpu.load_average.5m: {enabled: true}
              system.cpu.load_average.15m: {enabled: true}
          memory:
            metrics:
              system.memory.usage: {enabled: true}
              system.memory.utilization: {enabled: true}
          disk:
            metrics:
              system.disk.io: {enabled: true}
              system.disk.operations: {enabled: true}
          filesystem:
            metrics:
              system.filesystem.usage: {enabled: true}
              system.filesystem.utilization: {enabled: true}
          network:
            metrics:
              system.network.io: {enabled: true}
              system.network.packets: {enabled: true}
          process:
            metrics:
              process.cpu.utilization:
                enabled: true
    ```

    And add to the service pipelines:

    ```yaml
    service:
      pipelines:
        metrics/host:
          receivers: [hostmetrics]
          processors: [memory_limiter, resourcedetection, batch]
          exporters: [otlphttp]
    ```

5.  _(Optional)_ To collect Elasticsearch logs, add the filelog receiver. Ensure the user running the collector service (otelcol-contrib) has read access to your Elasticsearch log files:

    **If running Elasticsearch on Linux (Host):**

    ```yaml
    receivers:
      filelog:
        include:
          - /var/log/elasticsearch/elasticsearch.log
          - /var/log/elasticsearch/*.log
    ```

    **If running Elasticsearch in Docker:**

    ```yaml
    receivers:
      filelog:
        include:
          - /var/lib/docker/containers/*/*.log
        operators:
          - type: move
            from: attributes.log
            to: body
    ```

    And add to the service pipelines:

    ```yaml
    service:
      pipelines:
        logs:
          receivers: [filelog]
          processors: [resource/cluster_name_override]
          exporters: [otlphttp]
    ```

6.  _(Optional)_ To add custom metadata tags to your metrics, use the `resource/static_override` processor:

    ```yaml
    processors:
      resource/static_override:
        attributes:
          - key: env
            value: "production"
            action: upsert
    service:
      pipelines:
        metrics/elasticsearch:
          receivers: [elasticsearch]
          processors: [memory_limiter, resourcedetection, resource/cluster_name_override, resource/static_override, attributes/cardinality_reduction, cumulativetodelta, transform/metadata_nullify, batch]
          exporters: [otlphttp]
    ```

7.  Save the configuration file.

8.  Set the environment variables:

    Create a systemd override directory:

    ```bash
    sudo mkdir -p /etc/systemd/system/otelcol-contrib.service.d
    ```

    Create the environment configuration file:

    ```bash
    cat <<EOF | sudo tee /etc/systemd/system/otelcol-contrib.service.d/environment.conf
    [Service]
    Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318"
    Environment="NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY_HERE"
    Environment="NEW_RELIC_MEMORY_LIMIT_MIB=100"
    EOF
    ```

    Update the configuration with your values:

    -   Replace `https://otlp.nr-data.net:4318` with your region's endpoint
    -   Replace `YOUR_LICENSE_KEY_HERE` with your actual New Relic license key
    -   Replace `100` with your desired memory limit in MiB for the collector (default: 100 MiB). Adjust based on your environment's needs

9.  Restart the OTel Collector Contrib to apply changes:

    ```bash
    sudo systemctl daemon-reload
    sudo systemctl restart otelcol-contrib.service
    ```

#### Verify data collection [#verify-otel]

Verify that the OTel Collector Contrib is running and collecting data without errors:

1.  Check the collector service status:

    ```bash
    sudo systemctl status otelcol-contrib.service
    ```

2.  Monitor the collector logs for any errors:

    ```bash
    sudo journalctl -u otelcol-contrib.service -f
    ```

    Look for successful connections to Elasticsearch and New Relic. If you see errors, refer to the [troubleshooting guide](https://docs.newrelic.com/docs/opentelemetry/integrations/elasticsearch/troubleshooting).

### Prometheus Receiver

Use this approach if you already have a [Prometheus Elasticsearch exporter](https://github.com/prometheus-community/elasticsearch_exporter) running in your environment, or if you're migrating from a Prometheus-based monitoring stack.

> #### 💡 TIP
>
> **Recommended:** If you don't already have a Prometheus exporter running, use the **NRDOT Collector** or **OTel Collector Contrib** tabs instead. They connect directly to the Elasticsearch API without needing an additional exporter component.

#### Install and configure the Elasticsearch exporter [#install-exporter]

The [Prometheus Elasticsearch exporter](https://github.com/prometheus-community/elasticsearch_exporter) exposes Elasticsearch metrics in Prometheus format on port `9114`.

1.  Download the latest release:

    ```bash
    wget https://github.com/prometheus-community/elasticsearch_exporter/releases/download/v1.8.0/elasticsearch_exporter-1.8.0.linux-amd64.tar.gz
    tar xzf elasticsearch_exporter-*.tar.gz
    cd elasticsearch_exporter-*
    ```

2.  Start the exporter (point it to your Elasticsearch instance):

    ```bash
    ./elasticsearch_exporter --es.uri=http://localhost:9200
    ```

    > #### 💡 TIP
    >
    > For secured Elasticsearch, use: `--es.uri=https://username:password@localhost:9200 --es.ssl-skip-verify`

3.  Verify the exporter is running:

    ```bash
    curl http://localhost:9114/metrics | grep elasticsearch_cluster_health
    ```

    You should see Prometheus-formatted metrics like `elasticsearch_cluster_health_status`, `elasticsearch_cluster_health_number_of_nodes`, etc.

#### Configure the collector [#configure-prom]

This configuration works with both **NRDOT** and **OTel Collector Contrib**. Place it in the appropriate config location for your collector:

-   **NRDOT:** `/etc/nrdot-collector/config.yaml`
-   **OTel Collector Contrib:** `/etc/otelcol-contrib/config.yaml`

If you don't have a collector installed yet, follow the installation steps in the **NRDOT Collector** or **OTel Collector Contrib** tabs above first, then return here to apply this configuration.

This configuration scrapes metrics from the Elasticsearch exporter and translates them to OpenTelemetry-compatible metric names that power the New Relic Elasticsearch dashboard.

**Full Prometheus receiver configuration**

```yaml
receivers:
  prometheus/elasticsearch:
    config:
      scrape_configs:
        - job_name: 'elasticsearch'
          scrape_interval: 15s
          static_configs:
            - targets: ['localhost:9114']  # <-- elasticsearch_exporter address

processors:
  memory_limiter:
    check_interval: 60s
    limit_mib: ${env:NEW_RELIC_MEMORY_LIMIT_MIB:-100}

  metricstransform/prom_to_otel:
    transforms:
      - include: ^elasticsearch_cluster_health_number_of_data_nodes$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.data_nodes
      - include: ^elasticsearch_cluster_health_status$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.health
        operations:
          - action: update_label
            label: color
            new_label: status
      - include: ^elasticsearch_cluster_health_number_of_in_flight_fetch$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.in_flight_fetch
      - include: ^elasticsearch_cluster_health_number_of_nodes$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.nodes
      - include: ^elasticsearch_cluster_health_number_of_pending_tasks$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.pending_tasks
      - include: ^elasticsearch_cluster_health_active_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: active
      - include: ^elasticsearch_cluster_health_active_primary_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: active_primary
      - include: ^elasticsearch_cluster_health_initializing_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: initializing
      - include: ^elasticsearch_cluster_health_relocating_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: relocating
      - include: ^elasticsearch_cluster_health_unassigned_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: unassigned
      - include: ^elasticsearch_indices_docs$
        match_type: regexp
        action: insert
        new_name: elasticsearch.node.documents
        operations:
          - action: add_label
            new_label: state
            new_value: active
      - include: ^elasticsearch_indices_docs$
        match_type: regexp
        action: insert
        new_name: elasticsearch.index.documents
        operations:
          - action: add_label
            new_label: state
            new_value: active
          - action: add_label
            new_label: aggregation
            new_value: primary_shards
          - action: add_label
            new_label: elasticsearch.index.name
            new_value: _all
      - include: ^elasticsearch_indices_docs_deleted$
        match_type: regexp
        action: insert
        new_name: elasticsearch.node.documents
        operations:
          - action: add_label
            new_label: state
            new_value: deleted
      - include: ^elasticsearch_indices_fielddata_evictions$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.evictions
        operations:
          - action: add_label
            new_label: cache_name
            new_value: fielddata
      - include: ^elasticsearch_indices_query_cache_evictions$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.evictions
        operations:
          - action: add_label
            new_label: cache_name
            new_value: query
      - include: ^elasticsearch_indices_fielddata_memory_size_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.memory.usage
        operations:
          - action: add_label
            new_label: cache_name
            new_value: fielddata
      - include: ^elasticsearch_indices_query_cache_memory_size_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.memory.usage
        operations:
          - action: add_label
            new_label: cache_name
            new_value: query
      - include: ^elasticsearch_transport_rx_size_bytes_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cluster.io
        operations:
          - action: add_label
            new_label: direction
            new_value: received
      - include: ^elasticsearch_transport_tx_size_bytes_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cluster.io
        operations:
          - action: add_label
            new_label: direction
            new_value: sent
      - include: ^elasticsearch_filesystem_data_available_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.fs.disk.available
      - include: ^elasticsearch_filesystem_data_size_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.fs.disk.total
      - include: ^elasticsearch_process_open_files_count$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.open_files
      - include: ^elasticsearch_indices_indexing_index_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: index
      - include: ^elasticsearch_indices_search_query_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: query
      - include: ^elasticsearch_indices_get_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: get
      - include: ^elasticsearch_indices_merges_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: merge
      - include: ^elasticsearch_indices_indexing_index_time_seconds_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.time
        operations:
          - action: add_label
            new_label: operation
            new_value: index
          - action: experimental_scale_value
            experimental_scale: 1000
      - include: ^elasticsearch_indices_search_query_time_seconds$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.time
        operations:
          - action: add_label
            new_label: operation
            new_value: query
          - action: experimental_scale_value
            experimental_scale: 1000
      - include: ^elasticsearch_thread_pool_active_count$
        match_type: regexp
        action: insert
        new_name: elasticsearch.node.operations.current
        operations:
          - action: update_label
            label: type
            new_label: operation
      - include: ^elasticsearch_os_cpu_percent$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.usage
      - include: ^elasticsearch_os_load1$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.load_avg.1m
      - include: ^elasticsearch_os_load5$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.load_avg.5m
      - include: ^elasticsearch_os_load15$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.load_avg.15m
      - include: ^elasticsearch_jvm_gc_collection_seconds_count$
        match_type: regexp
        action: update
        new_name: jvm.gc.collections.count
      - include: ^elasticsearch_jvm_gc_collection_seconds_sum$
        match_type: regexp
        action: update
        new_name: jvm.gc.collections.elapsed
        operations:
          - action: experimental_scale_value
            experimental_scale: 1000
      - include: ^elasticsearch_jvm_memory_max_bytes$
        match_type: regexp
        action: update
        new_name: jvm.memory.heap.max
      - include: ^elasticsearch_jvm_memory_used_bytes$
        match_type: regexp
        action: update
        new_name: jvm.memory.heap.used
      - include: ^elasticsearch_thread_pool_threads_count$
        match_type: regexp
        action: update
        new_name: jvm.threads.count
        operations:
          - action: aggregate_labels
            label_set: [name, cluster]
            aggregation_type: sum
      - include: ^elasticsearch_indices_segments_count$
        match_type: regexp
        action: update
        new_name: elasticsearch.index.segments.count

  filter/allowed_metrics:
    metrics:
      include:
        match_type: strict
        metric_names:
          - elasticsearch.cluster.data_nodes
          - elasticsearch.cluster.health
          - elasticsearch.cluster.in_flight_fetch
          - elasticsearch.cluster.nodes
          - elasticsearch.cluster.pending_tasks
          - elasticsearch.cluster.shards
          - elasticsearch.index.documents
          - elasticsearch.index.segments.count
          - elasticsearch.node.cache.evictions
          - elasticsearch.node.cache.memory.usage
          - elasticsearch.node.cluster.io
          - elasticsearch.node.documents
          - elasticsearch.node.fs.disk.available
          - elasticsearch.node.fs.disk.total
          - elasticsearch.node.open_files
          - elasticsearch.node.operations.completed
          - elasticsearch.node.operations.current
          - elasticsearch.node.operations.time
          - elasticsearch.os.cpu.usage
          - elasticsearch.os.cpu.load_avg.1m
          - elasticsearch.os.cpu.load_avg.5m
          - elasticsearch.os.cpu.load_avg.15m
          - jvm.gc.collections.count
          - jvm.gc.collections.elapsed
          - jvm.memory.heap.max
          - jvm.memory.heap.used
          - jvm.threads.count

  transform/rename_node_name:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["elasticsearch.node.name"], attributes["name"]) where attributes["name"] != nil
          - delete_key(attributes, "name") where attributes["elasticsearch.node.name"] != nil

  groupbyattrs/node:
    keys:
      - elasticsearch.node.name
      - elasticsearch.cluster.name

  cumulativetodelta: {}

  resource/cluster_name_override:
    attributes:
      - key: elasticsearch.cluster.name
        value: "<elasticsearch-cluster-name>"
        action: upsert

  transform/scope_override:
    metric_statements:
      - context: scope
        statements:
          - set(name, "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/elasticsearchreceiver")
          - set(version, "")

  resourcedetection:
    detectors: [system]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
        os.type:
          enabled: true

  batch:
    timeout: 30s
    send_batch_size: 2048
    send_batch_max_size: 4096

  attributes/cardinality_reduction:
    actions:
      - key: process.pid
        action: delete
      - key: process.parent_pid
        action: delete

  transform/metadata_nullify:
    metric_statements:
      - context: metric
        statements:
          - set(description, "")
          - set(unit, "")

exporters:
  otlphttp:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}
    compression: gzip
    timeout: 30s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  pipelines:
    metrics/elasticsearch:
      receivers: [prometheus/elasticsearch]
      processors:
        - memory_limiter
        - metricstransform/prom_to_otel
        - filter/allowed_metrics
        - transform/rename_node_name
        - groupbyattrs/node
        - transform/scope_override
        - resourcedetection
        - resource/cluster_name_override
        - attributes/cardinality_reduction
        - cumulativetodelta
        - transform/metadata_nullify
        - batch
      exporters: [otlphttp]
```

Replace the following values in the configuration:

-   `<elasticsearch-cluster-name>`: Your Elasticsearch cluster name for identification in New Relic.
-   `localhost:9114`: The address of your `elasticsearch_exporter` if running on a different host or port.

#### Set up environment variables [#env-vars-prom]

Create a systemd override to inject the required environment variables. Replace `<collector-service>` with your collector service name (`nrdot-collector` or `otelcol-contrib`):

```bash
sudo mkdir -p /etc/systemd/system/<collector-service>.service.d
```

Create the file `/etc/systemd/system/<collector-service>.service.d/environment.conf`:

```ini
[Service]
Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318"
Environment="NEWRELIC_LICENSE_KEY=YOUR_NEWRELIC_LICENSE_KEY"
Environment="NEW_RELIC_MEMORY_LIMIT_MIB=100"
```

Replace `YOUR_NEWRELIC_LICENSE_KEY` with your license key.

> #### 💡 TIP
>
> For EU accounts, use `NEWRELIC_OTLP_ENDPOINT=https://otlp.eu01.nr-data.net:4318`

#### Start the collector [#start-prom]

Replace `<collector-service>` with your collector service name (`nrdot-collector` or `otelcol-contrib`):

```bash
sudo systemctl daemon-reload
sudo systemctl enable <collector-service>
sudo systemctl restart <collector-service>
```

Check the collector status:

```bash
sudo systemctl status <collector-service>
sudo journalctl -u <collector-service> -f
```

#### Verify data in New Relic [#verify-prom]

After a few minutes, verify that data is flowing to New Relic:

```sql
FROM Metric SELECT count(*)
WHERE metricName LIKE 'elasticsearch.%'
AND elasticsearch.cluster.name = '<elasticsearch-cluster-name>'
SINCE 10 minutes ago
```

> #### 💡 TIP
>
> **Correlate APM with Elasticsearch**: To connect your APM application and Elasticsearch cluster, include the resource attribute `es.cluster.name="your-cluster-name"` in your APM metrics. This enables cross-service visibility and faster troubleshooting within New Relic.

## View your Elasticsearch data [#find-and-use]

Once the collector is running and sending data, you can explore your Elasticsearch metrics, create custom queries, and set up monitoring dashboards in New Relic.

For detailed information on accessing your data, writing NRQL queries, and configuring alerts, see [Find and query Elasticsearch data](https://docs.newrelic.com/docs/opentelemetry/integrations/elasticsearch/find-and-query-data).

## Troubleshooting [#troubleshooting]

If you encounter issues during installation or don't see data in New Relic, see our comprehensive [troubleshooting guide](https://docs.newrelic.com/docs/opentelemetry/integrations/elasticsearch/troubleshooting) for step-by-step solutions to common problems.