--- title: Monitor ECS on EC2 with OpenTelemetry source: https://docs.newrelic.com/docs/opentelemetry/integrations/ecs-monitoring/ecs-ec2 --- Monitor Amazon ECS tasks running on EC2 instances by deploying OpenTelemetry Collector Contrib as a sidecar container. This comprehensive guide walks you through creating task definitions, configuring the collector, and setting up monitoring for your ECS on EC2 workloads. ## Installation steps [#install] Follow these steps in order to set up monitoring for your ECS on EC2 tasks. ### Before you begin [#requirements] Make sure your environment meets these requirements: **ECS requirements** - ECS cluster: Active ECS cluster with EC2 capacity providers - Task definitions: Existing task definitions you want to monitor - Launch type: EC2 launch type - Network mode: Supports `default`, `host` network modes **EC2 requirements** - Instance types: Any EC2 instance type supported by ECS - AMI: Amazon Linux 2 or other ECS-optimized AMI - Storage: Sufficient disk space for container images and logs - Security groups: Allow outbound HTTPS traffic for metrics export **AWS permissions** - Task execution role: `AmazonECSTaskExecutionRolePolicy` attached - SSM permissions: Access to Systems Manager Parameter Store for configuration - CloudWatch permissions: For logging (optional but recommended) **New Relic requirements** - Account: Active New Relic account - License key: [New Relic license key](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#ingest-license-key) for your account ### Store your New Relic license key [#store-license-key] Save your license key as a Systems Manager (SSM) parameter to securely store credentials for the OpenTelemetry Collector: ```bash aws ssm put-parameter \ --name "/newrelic-infra/ecs/license-key" \ --type SecureString \ --description 'New Relic license key for ECS monitoring' \ --value "YOUR_NEW_RELIC_LICENSE_KEY" ``` ### Create IAM policy and execution role [#create-role] 1. Create an IAM policy so your ECS containers can securely retrieve the New Relic license key: ```bash aws iam create-policy \ --policy-name "NewRelicSSMLicenseKeyReadAccess" \ --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ssm:GetParameters"],"Resource":["arn:aws:ssm:*:*:parameter/newrelic-infra/ecs/license-key"]}]}' \ --description "Provides read access to the New Relic SSM license key parameter" ``` 2. Create an IAM role to be used as the task execution role: ```bash aws iam create-role \ --role-name "NewRelicECSTaskExecutionRole" \ --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ecs-tasks.amazonaws.com"},"Action":"sts:AssumeRole"}]}' \ --description "ECS task execution role for New Relic infrastructure" ``` 3. Attach the required managed policies to the role: ```bash # Attach the standard ECS task execution policy aws iam attach-role-policy \ --role-name "NewRelicECSTaskExecutionRole" \ --policy-arn "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy" # Attach the New Relic SSM license key read access policy aws iam attach-role-policy \ --role-name "NewRelicECSTaskExecutionRole" \ --policy-arn "arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):policy/NewRelicSSMLicenseKeyReadAccess" ``` ### Store collector configuration [#store-config] Store the OpenTelemetry Collector configuration in AWS Systems Manager Parameter Store so you can manage and update settings without rebuilding container images: ```bash aws ssm put-parameter \ --name "/ecs/otel-collector/ec2-config" \ --type "String" \ --value "$(cat < hostmetrics: collection_interval: scrapers: cpu: metrics: system.cpu.time: enabled: false system.cpu.utilization: enabled: true load: memory: metrics: system.memory.utilization: enabled: true paging: metrics: system.paging.utilization: enabled: false system.paging.faults: enabled: false filesystem: metrics: system.filesystem.utilization: enabled: true disk: metrics: system.disk.merged: enabled: false system.disk.pending_operations: enabled: false system.disk.weighted_io_time: enabled: false network: metrics: system.network.connections: enabled: false processors: metricstransform: transforms: - include: system.cpu.utilization action: update operations: - action: aggregate_labels label_set: [ state ] aggregation_type: mean - include: system.paging.operations action: update operations: - action: aggregate_labels label_set: [ direction ] aggregation_type: sum filter/exclude_cpu_utilization: metrics: datapoint: - 'metric.name == "system.cpu.utilization" and attributes["state"] == "interrupt"' - 'metric.name == "system.cpu.utilization" and attributes["state"] == "nice"' - 'metric.name == "system.cpu.utilization" and attributes["state"] == "softirq"' filter/exclude_memory_utilization: metrics: datapoint: - 'metric.name == "system.memory.utilization" and attributes["state"] == "slab_unreclaimable"' - 'metric.name == "system.memory.utilization" and attributes["state"] == "inactive"' - 'metric.name == "system.memory.utilization" and attributes["state"] == "cached"' - 'metric.name == "system.memory.utilization" and attributes["state"] == "buffered"' - 'metric.name == "system.memory.utilization" and attributes["state"] == "slab_reclaimable"' filter/exclude_memory_usage: metrics: datapoint: - 'metric.name == "system.memory.usage" and attributes["state"] == "slab_unreclaimable"' - 'metric.name == "system.memory.usage" and attributes["state"] == "inactive"' filter/exclude_filesystem_utilization: metrics: datapoint: - 'metric.name == "system.filesystem.utilization" and attributes["type"] == "squashfs"' filter/exclude_filesystem_usage: metrics: datapoint: - 'metric.name == "system.filesystem.usage" and attributes["type"] == "squashfs"' - 'metric.name == "system.filesystem.usage" and attributes["state"] == "reserved"' filter/exclude_filesystem_inodes_usage: metrics: datapoint: - 'metric.name == "system.filesystem.inodes.usage" and attributes["type"] == "squashfs"' - 'metric.name == "system.filesystem.inodes.usage" and attributes["state"] == "reserved"' filter/exclude_system_disk: metrics: datapoint: - 'metric.name == "system.disk.operations" and IsMatch(attributes["device"], "^loop.*") == true' - 'metric.name == "system.disk.merged" and IsMatch(attributes["device"], "^loop.*") == true' - 'metric.name == "system.disk.io" and IsMatch(attributes["device"], "^loop.*") == true' - 'metric.name == "system.disk.io_time" and IsMatch(attributes["device"], "^loop.*") == true' - 'metric.name == "system.disk.operation_time" and IsMatch(attributes["device"], "^loop.*") == true' filter/exclude_system_paging: metrics: datapoint: - 'metric.name == "system.paging.usage" and attributes["state"] == "cached"' - 'metric.name == "system.paging.operations" and attributes["type"] == "cached"' filter/exclude_network: metrics: datapoint: - 'IsMatch(metric.name, "^system.network.*") == true and attributes["device"] == "lo"' attributes/exclude_system_paging: include: match_type: strict metric_names: - system.paging.operations actions: - key: type action: delete cumulativetodelta: transform/host: metric_statements: - context: metric statements: - set(metric.description, "") - set(metric.unit, "") transform: trace_statements: - context: span statements: - truncate_all(span.attributes, ) - truncate_all(resource.attributes, ) log_statements: - context: log statements: - truncate_all(log.attributes, ) - truncate_all(resource.attributes, ) memory_limiter: check_interval: limit_mib: \${env:NEW_RELIC_MEMORY_LIMIT_MIB:-} metricstransform/base: transforms: - include: container.cpu.utilized action: insert new_name: container.cpu.utilization - include: container.memory.usage action: insert new_name: container.memory.usage.total - include: container.storage.read_bytes action: insert new_name: container.blockio.io_service_bytes_recursive operations: - action: add_label new_label: operation new_value: read - include: container.storage.write_bytes action: insert new_name: container.blockio.io_service_bytes_recursive operations: - action: add_label new_label: operation new_value: write transform/promote_cluster_keys: metric_statements: - context: datapoint statements: - set(attributes["aws.ecs.cluster.name"], resource.attributes["aws.ecs.cluster.name"]) where resource.attributes["aws.ecs.cluster.name"] != nil - set(attributes["cloud.account.id"], resource.attributes["cloud.account.id"]) where resource.attributes["cloud.account.id"] != nil - set(attributes["cloud.region"], resource.attributes["cloud.region"]) where resource.attributes["cloud.region"] != nil - set(attributes["aws.ecs.task.arn"], resource.attributes["aws.ecs.task.arn"]) where resource.attributes["aws.ecs.task.arn"] != nil metricstransform/cluster_aggregation: transforms: - include: container.cpu.utilization action: insert new_name: ecs.cluster.cpu.utilization operations: - action: aggregate_labels label_set: [aws.ecs.cluster.name, cloud.account.id, cloud.region] aggregation_type: mean - include: container.memory.usage.total action: insert new_name: ecs.cluster.memory.usage operations: - action: aggregate_labels label_set: [aws.ecs.cluster.name, cloud.account.id, cloud.region] aggregation_type: sum - include: container.network.io.usage.rx_bytes action: insert new_name: ecs.cluster.network.rx_bytes operations: - action: aggregate_labels label_set: [aws.ecs.cluster.name, cloud.account.id, cloud.region] aggregation_type: sum - include: container.network.io.usage.tx_bytes action: insert new_name: ecs.cluster.network.tx_bytes operations: - action: aggregate_labels label_set: [aws.ecs.cluster.name, cloud.account.id, cloud.region] aggregation_type: sum - include: container.cpu.utilization action: insert new_name: ecs.cluster.running_task_count operations: - action: aggregate_labels label_set: [aws.ecs.cluster.name, cloud.account.id, cloud.region, aws.ecs.task.arn] aggregation_type: mean - action: aggregate_labels label_set: [aws.ecs.cluster.name, cloud.account.id, cloud.region] aggregation_type: count - include: container.memory.usage.limit action: insert new_name: ecs.cluster.memory.limit operations: - action: aggregate_labels label_set: [aws.ecs.cluster.name, cloud.account.id, cloud.region] aggregation_type: sum transform/running_task_count_one: metric_statements: - context: datapoint statements: - set(value_double, 1.0) where metric.name == "ecs.cluster.running_task_count" metricstransform/running_task_count_sum: transforms: - include: ecs.cluster.running_task_count action: update operations: - action: aggregate_labels label_set: [aws.ecs.cluster.name, cloud.account.id, cloud.region] aggregation_type: sum filter/cluster_metrics: metrics: include: match_type: regexp metric_names: - ^ecs\.cluster\..* metricstransform/task_metrics: transforms: - include: ecs.task.cpu.utilized action: insert new_name: ecs.task.cpu.utilization - include: container.memory.usage.total action: insert new_name: ecs.task.memory.usage filter/task_metrics: metrics: include: match_type: strict metric_names: - "ecs.task.cpu.utilization" - "ecs.task.cpu.utilized" - "ecs.task.memory.utilized" - "ecs.task.memory.usage" - "ecs.task.memory.usage.limit" - "ecs.task.network.io.usage.rx_bytes" - "ecs.task.network.io.usage.tx_bytes" filter/container_metrics: metrics: include: match_type: regexp metric_names: - ^container\..* transform/cluster_identifier: metric_statements: - context: resource statements: - set(attributes["aws.arn"], resource.attributes["aws.ecs.task.arn"]) where resource.attributes["aws.ecs.task.arn"] != nil and resource.attributes["aws.ecs.cluster.name"] != nil - replace_pattern(attributes["aws.arn"], ":task/.*", "") where attributes["aws.arn"] != nil - set(attributes["aws.arn"], Concat([attributes["aws.arn"], ":cluster/", resource.attributes["aws.ecs.cluster.name"]], "")) where attributes["aws.arn"] != nil and resource.attributes["aws.ecs.cluster.name"] != nil transform/task_identifier: metric_statements: - context: resource statements: - set(attributes["TaskArn"], resource.attributes["aws.ecs.task.arn"]) where resource.attributes["aws.ecs.task.arn"] != nil batch: send_batch_size: timeout: resource: attributes: - key: ClusterName from_attribute: aws.ecs.cluster.name action: insert - key: ServiceName from_attribute: aws.ecs.service.name action: insert - key: TaskId from_attribute: aws.ecs.task.id action: insert - key: TaskDefinitionFamily from_attribute: aws.ecs.task.family action: insert - key: LaunchType from_attribute: aws.ecs.launch_type action: insert resource/cluster: attributes: - key: ClusterName from_attribute: aws.ecs.cluster.name action: insert - key: cloud.platform value: "aws_ecs" action: upsert resource/task: attributes: - key: ClusterName from_attribute: aws.ecs.cluster.name action: insert - key: ServiceName from_attribute: aws.ecs.service.name action: insert - key: TaskId from_attribute: aws.ecs.task.id action: insert - key: TaskArn from_attribute: aws.ecs.task.arn action: insert - key: TaskDefinitionFamily from_attribute: aws.ecs.task.family action: insert - key: LaunchType from_attribute: aws.ecs.launch_type action: insert - key: cloud.platform value: "aws_ecs" action: upsert - key: docker.host from_attribute: aws.ecs.task.id action: insert - key: docker.imageName from_attribute: container.image.name action: insert - key: docker.containerId from_attribute: container.id action: insert - key: docker.state from_attribute: aws.ecs.container.know_status action: insert resourcedetection: detectors: - env - ecs - ec2 - system timeout: override: false exporters: otlphttp: endpoint: https://otlp.nr-data.net:443 headers: api-key: \${NEW_RELIC_LICENSE_KEY} debug: verbosity: basic service: pipelines: metrics/clusters: receivers: [awsecscontainermetrics] processors: [ metricstransform/base, transform/promote_cluster_keys, metricstransform/cluster_aggregation, transform/running_task_count_one, metricstransform/running_task_count_sum, filter/cluster_metrics, transform/cluster_identifier, resource/cluster, batch ] exporters: [otlphttp, debug] metrics/containers: receivers: [awsecscontainermetrics] processors: [ metricstransform/base, filter/container_metrics, resource/task, batch ] exporters: [otlphttp, debug] metrics/tasks: receivers: [awsecscontainermetrics] processors: [ metricstransform/base, metricstransform/task_metrics, filter/task_metrics, transform/task_identifier, resource/task, batch ] exporters: [otlphttp, debug] metrics/host: receivers: [hostmetrics] processors: - memory_limiter - metricstransform - filter/exclude_cpu_utilization - filter/exclude_memory_utilization - filter/exclude_memory_usage - filter/exclude_filesystem_utilization - filter/exclude_filesystem_usage - filter/exclude_filesystem_inodes_usage - filter/exclude_system_disk - filter/exclude_network - attributes/exclude_system_paging - transform/host - resourcedetection - cumulativetodelta - batch exporters: [otlphttp, debug] EOF )" ``` #### Configuration parameters The following parameters can be customized in the OpenTelemetry Collector configuration: | Parameter | Description | | --------------------------------------- | --------------------------------------------------------------------------------- | | `` | Interval to collect metrics from ECS container and host metrics endpoints. | | `` | Memory limit for the OpenTelemetry Collector in MiB | | `` | Interval for the memory limiter to check current memory usage | | `` | Number of metrics to batch before sending to New Relic | | `` | Maximum time to wait before sending a batch | | `` | Timeout for resource detection processors | | `` | Maximum length for span and log attribute values before truncation. Default: 4095 | | `` | Maximum length for resource attribute values before truncation. Default: 4095 | ### Create task definition [#create-task-definition] Create a new ECS task definition that includes the collector sidecar container. Choose between NRDOT Collector (New Relic's distribution) or OpenTelemetry Collector, then select the task definition for your container platform: #### NRDOT Collector > #### 💡 TIP > > **NRDOT Collector** is New Relic's distribution of the OpenTelemetry Collector with New Relic support for assistance. It bundles the `awsecscontainermetrics` and `hostmetrics` receivers used by this configuration, so it runs with the same collector config you stored in the previous step. > #### ⚠️ IMPORTANT > > NRDOT Collector publishes Linux container images only. To monitor Windows containers, use the **OpenTelemetry Collector** tab. ```json { "family": "otel-ecs-ec2-sidecar-metrics", "executionRoleArn": "arn:aws:iam::YOUR_ACCOUNT:role/NewRelicECSTaskExecutionRole", "networkMode": "host", "requiresCompatibilities": ["EC2"], "cpu": "", "memory": "", "containerDefinitions": [ { "name": "your-application", "image": "your-app:latest", "cpu": , "memory": , "essential": true, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "", "awslogs-create-group": "true", "awslogs-region": "", "awslogs-stream-prefix": "" } } }, { "name": "otel-collector", "image": "newrelic/nrdot-collector:latest", "cpu": , "memory": , "essential": true, "command": ["--config=env:OTEL_CONFIG"], "mountPoints": [ { "sourceVolume": "proc", "containerPath": "/proc", "readOnly": true }, { "sourceVolume": "sys", "containerPath": "/sys", "readOnly": true } ], "secrets": [ { "name": "OTEL_CONFIG", "valueFrom": "arn:aws:ssm:us-east-1:YOUR_ACCOUNT:parameter/ecs/otel-collector/ec2-config" }, { "name": "NEW_RELIC_LICENSE_KEY", "valueFrom": "arn:aws:ssm:us-east-1:YOUR_ACCOUNT:parameter/newrelic-infra/ecs/license-key" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "", "awslogs-create-group": "true", "awslogs-region": "", "awslogs-stream-prefix": "" } } } ], "volumes": [ { "name": "proc", "host": { "sourcePath": "/proc" } }, { "name": "sys", "host": { "sourcePath": "/sys" } } ] } ``` #### OpenTelemetry Collector **Linux containers** ```json { "family": "otel-ecs-ec2-sidecar-metrics", "executionRoleArn": "arn:aws:iam::YOUR_ACCOUNT:role/NewRelicECSTaskExecutionRole", "networkMode": "host", "requiresCompatibilities": ["EC2"], "cpu": "", "memory": "", "containerDefinitions": [ { "name": "your-application", "image": "your-app:latest", "cpu": , "memory": , "essential": true, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "", "awslogs-create-group": "true", "awslogs-region": "", "awslogs-stream-prefix": "" } } }, { "name": "otel-collector", "image": "otel/opentelemetry-collector-contrib:latest", "cpu": , "memory": , "essential": true, "command": ["--config=env:OTEL_CONFIG"], "mountPoints": [ { "sourceVolume": "proc", "containerPath": "/proc", "readOnly": true }, { "sourceVolume": "sys", "containerPath": "/sys", "readOnly": true } ], "secrets": [ { "name": "OTEL_CONFIG", "valueFrom": "arn:aws:ssm:us-east-1:YOUR_ACCOUNT:parameter/ecs/otel-collector/ec2-config" }, { "name": "NEW_RELIC_LICENSE_KEY", "valueFrom": "arn:aws:ssm:us-east-1:YOUR_ACCOUNT:parameter/newrelic-infra/ecs/license-key" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "", "awslogs-create-group": "true", "awslogs-region": "", "awslogs-stream-prefix": "" } } } ], "volumes": [ { "name": "proc", "host": { "sourcePath": "/proc" } }, { "name": "sys", "host": { "sourcePath": "/sys" } } ] } ``` **Windows containers** ```json { "family": "otel-ecs-ec2-sidecar-metrics-windows", "executionRoleArn": "arn:aws:iam::YOUR_ACCOUNT:role/NewRelicECSTaskExecutionRole", "networkMode": "default", "requiresCompatibilities": ["EC2"], "cpu": "", "memory": "", "runtimePlatform": { "cpuArchitecture": "X86_64", "operatingSystemFamily": "WINDOWS_SERVER_2022_CORE" }, "containerDefinitions": [ { "name": "your-application", "image": "your-app:latest", "cpu": , "memory": , "essential": true, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "", "awslogs-create-group": "true", "awslogs-region": "", "awslogs-stream-prefix": "" } } }, { "name": "otel-collector", "image": "otel/opentelemetry-collector-contrib:latest-windows-2022-amd64", "cpu": , "memory": , "essential": true, "command": ["--config", "env:OTEL_CONFIG"], "secrets": [ { "name": "OTEL_CONFIG", "valueFrom": "arn:aws:ssm:us-east-1:YOUR_ACCOUNT:parameter/ecs/otel-collector/ec2-config-windows" }, { "name": "NEW_RELIC_LICENSE_KEY", "valueFrom": "arn:aws:ssm:us-east-1:YOUR_ACCOUNT:parameter/newrelic-infra/ecs/license-key" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "", "awslogs-create-group": "true", "awslogs-region": "", "awslogs-stream-prefix": "" } } } ], "volumes": [] } ``` #### Task definition parameters The following parameters can be customized in your ECS task definition: | Parameter | Description | | ------------------------------- | --------------------------------------------------------- | | `` | Total CPU units for the EC2 task | | `` | Total memory for the EC2 task in MiB | | `` | CPU units allocated to your application container | | `` | Memory allocated to your application container in MiB | | `` | CPU units allocated to the OpenTelemetry Collector | | `` | Memory allocated to the OpenTelemetry Collector in MiB | | `` | CloudWatch log group name for your application container | | `` | CloudWatch log group name for the OpenTelemetry Collector | | `` | AWS region for CloudWatch logs | | `` | Log stream prefix for your application container | | `` | Log stream prefix for the OpenTelemetry Collector | > #### 💡 TIP > > The `networkMode` is set to `"host"` for Linux containers and should be `"default"` for Windows containers. Host mode provides better access to system metrics on EC2 instances. > #### ⚠️ IMPORTANT > > Replace `YOUR_ACCOUNT` and region values with your actual AWS account ID and AWS region. ### Deploy and run the task [#deploy-task] Deploy your task definition to your ECS cluster: 1. Register the task definition: ```bash aws ecs register-task-definition --cli-input-json file://task-definition.json ``` 2. Create a service with daemon scheduling strategy: ```bash aws ecs create-service \ --cluster your-cluster-name \ --service-name otel-monitoring-service \ --task-definition otel-ecs-ec2-sidecar-metrics:1 \ --scheduling-strategy DAEMON \ --launch-type EC2 ``` > #### 💡 TIP > > DAEMON scheduling strategy ensures one monitoring task runs on every EC2 instance in your cluster, providing comprehensive infrastructure monitoring coverage. ### Verify data collection [#verify-data] Check that your data is flowing to New Relic: - Check OpenTelemetry Collector status: Review container logs to confirm the collector is running without errors and successfully connecting to New Relic: ```bash aws logs get-log-events \ --log-group-name "/ecs/otel-collector-ec2" \ --log-stream-name "otel/otel-collector/TASK_ID" ``` - Verify data in New Relic UI: Navigate to **[one.newrelic.com](https://one.newrelic.com) > All Capabilities > Infrastructure** to confirm your ECS hosts and containers appear with metrics. For detailed guidance on exploring your data, see [Find and query your ECS monitoring data](https://docs.newrelic.com/docs/opentelemetry/integrations/ecs-monitoring/find-and-query-your-data). /\* ## Key features \[#key-features] This configuration provides: \- \*\*Container metrics\*\*: CPU, memory, network, and storage metrics for each container in your tasks \- \*\*Host metrics\*\*: System-level metrics from the EC2 instances running your containers \- \*\*ECS metadata\*\*: Automatic tagging with cluster, service, task, and container information \- \*\*Filtered metrics\*\*: Optimized metric collection to reduce noise and costs \- \*\*Health monitoring\*\*: Built-in health checks for the collector sidecar \*/ ## Next steps [#next-steps] After setting up monitoring, you can: - [Create custom dashboards](https://docs.newrelic.com/docs/query-your-data/explore-query-data/dashboards/introduction-dashboards/) for your ECS metrics - [Set up alerts](https://docs.newrelic.com/docs/alerts/create-alert/create-alert-condition/alert-conditions/) for container and host-level issues - [Correlate ECS metrics with application traces and logs](https://docs.newrelic.com/docs/opentelemetry/integrations/ecs-monitoring/overview#how-it-works)