--- title: Set up the trace observer source: https://docs.newrelic.com/docs/distributed-tracing/infinite-tracing/set-trace-observer --- > #### ⚠️ FEATURE AVAILABILITY > > This feature isn't available in all regions. Refer to the [data center and regional availability](https://docs.newrelic.com/docs/accounts/accounts-billing/account-setup/choose-your-data-center#feature-limitations-by-region) page to check if it's supported in your region and explore alternative solutions. If you're following the [steps to set up distributed tracing](https://docs.newrelic.com/docs/distributed-tracing/enable-configure/quick-start) and are continuing with the steps for the advanced option called Infinite Tracing, you need to set up a trace observer. The trace observer is an AWS-based service for Infinite Tracing that provides you with a low-latency and low-cost way to send your telemetry data to New Relic. As it receives your tracing data, the trace observer uses [tail-based sampling](https://docs.newrelic.com/docs/distributed-tracing/concepts/how-new-relic-distributed-tracing-works/#tail-based) to decide what to keep and send to New Relic. Complete the trace observer setup up that fits your types of data sources (services, apps, or entities). You may need to do steps in more than one section if you have a variety of types: - [Services instrumented with APM agents and third-party telemetry tools](#apm-and-third-party) - [Services instrumented with New Relic browser, mobile, and Lambda](#browser-mobile-lambda) - [(Optional) Additional configuration for all service types](#follow-up-config) > #### 💡 TIP > > Do some of your requests communicate with services that are still using standard distributed tracing? To avoid [configuration conflicts](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/troubleshooting/infinite-tracing-trace-configuration-conflicts), you should ideally enable Infinite Tracing for all associated services. ## Services instrumented with APM agents and third-party telemetry tools [#apm-and-third-party] To create a new trace observer if you're using New Relic APM agents or third-party integrations: 1. Go to **[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Infinite Tracing settings**. 2. Select an account in the upper-left dropdown. If you have access to multiple accounts, make sure you're in the account where you want Infinite Tracing enabled. If you can't add an observer, it's likely because we allow only [one observer per region, per account family](https://docs.newrelic.com/docs/distributed-tracing/concepts/how-new-relic-distributed-tracing-works/#infinite-architecture). 3. If no trace observers are already present, click **New trace observer** to add one, fill out the information, and click **Create**. Note that we offer the following regions: - **us-east-1 (US)** - **us-west-2 (US)** - **eu-central-1 (EU):** Select this region if you are collecting data from EU requests and you have an EU-based New Relic account that reports in EU data centers. - **eu-west-1 (US):** Select this region if you are collecting data from EU requests and you have a US-based New Relic account that reports in US data centers. - **ap-southeast-1 (US)** - **ap-southeast-2 (US)** - **us-east-2 (US, FedRAMP)** 4. Under the **Endpoints** dropdown, copy the endpoint value under **For other integrations** and have it ready. This value is referenced in our code examples as `YOUR_TRACE_OBSERVER_URL`. This is used to test the [trace observer](#sample-payload), as well as to configure our [telemetry integrations](https://docs.newrelic.com/docs/distributed-tracing/enable-configure/quick-start) and for sending data via the [Trace API](https://docs.newrelic.com/docs/distributed-tracing/trace-api/introduction-trace-api/). 5. If you are setting up an APM agent, you should also copy the endpoint value under **For language agents** and have it ready. This value is referenced in our code examples as `YOUR_TRACE_OBSERVER_HOST`. This is used for [configuring our language agents to send data to the trace observer](https://docs.newrelic.com/docs/distributed-tracing/enable-configure/quick-start). 6. (Optional) To verify the trace observer is working, we recommend sending a sample trace payload. If you're using our [Trace API](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/trace-api/introduction-trace-api) directly, this will help you learn how the API works. **Send sample payload** > #### ⚠️ IMPORTANT > > If you're using Zipkin-format data, see [Send Zipkin payload](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/trace-api/report-zipkin-format-traces-trace-api#send-zipkin-curl). This test sends a sample trace payload with one trace and two spans from a service named `Test Service A`. To send this sample request: 1. Get the license key for the account you want to report data to and have it ready. 2. Copy the following `curl` request into a text editor: ```bash curl -i -H 'Content-Type: application/json' \ -H 'Api-Key: YOUR_LICENSE_KEY' \ -H 'Data-Format: newrelic' \ -H 'Data-Format-Version: 1' \ -X POST \ -d '[ { "common": { "attributes": { "environment": "staging" } }, "spans": [ { "trace.id": "123456", "id": "ABC", "attributes": { "duration.ms": 12.53, "host": "host123.example.com", "name": "/home", "service.name": "Test Service A" } }, { "trace.id": "123456", "id": "DEF", "attributes": { "error.message": "Invalid credentials", "service.name": "Test Service B", "host": "host456.example.com", "duration.ms": 2.97, "name": "/auth", "parent.id": "ABC" } } ] } ]' 'YOUR_TRACE_OBSERVER_URL' ``` 3. Insert your own values into the `curl` request: | Value | Description | | ------------------------- | ---------------------------------------------------------------------------------------------- | | `YOUR_LICENSE_KEY` | Replace this with your ingest license key. | | `YOUR_TRACE_OBSERVER_URL` | Replace this with the **For other integrations** endpoint value you copied in a previous step. | 4. Copy the curl request into a terminal and execute it. 5. The test should return `HTTP/1.1 202 Accepted`, indicating success. If it does not, check the following common issues: - Confirm that you used the [**For other integrations** endpoint](#endpoints) value. - Confirm you're using single quotes around `YOUR_TRACE_OBSERVER_URL`. - Check that you're using the correct API key. 6. If your test returned `HTTP/1.1 202 Accepted`, go to the New Relic UI to see a query of the sample payload data using the span attribute `service.name = Test Service A` ([here's a link for that query](https://one.newrelic.com/launcher/distributed-tracing-nerdlets.distributed-tracing?launcher=eyJ0aW1lUmFuZ2UiOnsiYmVnaW5fdGltZSI6bnVsbCwiZW5kX3RpbWUiOm51bGwsImR1cmF0aW9uIjoxODAwMDAwfSwiJGlzRmFsbGJhY2tUaW1lUmFuZ2UiOnRydWV9&pane=eyJuZXJkbGV0SWQiOiJkaXN0cmlidXRlZC10cmFjaW5nLW5lcmRsZXRzLmRpc3RyaWJ1dGVkLXRyYWNpbmctbGF1bmNoZXIiLCJzb3J0SW5kZXgiOjAsInNvcnREaXJlY3Rpb24iOiJERVNDIiwicXVlcnkiOnsib3BlcmF0b3IiOiJBTkQiLCJpbmRleFF1ZXJ5Ijp7ImNvbmRpdGlvblR5cGUiOiJJTkRFWCIsIm9wZXJhdG9yIjoiQU5EIiwiY29uZGl0aW9ucyI6W119LCJzcGFuUXVlcnkiOnsib3BlcmF0b3IiOiJBTkQiLCJjb25kaXRpb25TZXRzIjpbeyJjb25kaXRpb25UeXBlIjoiU1BBTiIsIm9wZXJhdG9yIjoiQU5EIiwiY29uZGl0aW9ucyI6W3siYXR0ciI6InNlcnZpY2UubmFtZSIsIm9wZXJhdG9yIjoiRVEiLCJ2YWx1ZSI6IlRlc3QgU2VydmljZSBBIn1dfV19fX0=)). Because the sample payload contains an error attribute, the error sampler will mark it for keeping. If you modify the payload to remove the error attributes, the random sampler may not choose to keep this particular trace. > #### 💡 TIP > > Traces may take up to one minute to show up in the UI. 7. Bookmark the [optional configurations](#follow-up-config) below so you can return to them after you experiment with Infinite Tracing. 8. Return to your original distributed tracing setup instructions for your agent or third-party telemetry tool and insert your new endpoint value in the configuration for each service. 9. If you also have [services instrumented with New Relic browser, mobile, and Lambda](#browser-mobile-lambda), follow the steps in that section to select your data sources (services, apps, or entities) in the trace observer. ## Services instrumented with New Relic browser, mobile, and Lambda [#browser-mobile-lambda] To create a new trace observer if you're using New Relic browser, mobile, or Lambda, you follow similar steps to those for [APM agents and third-party integrations](#apm-and-third-party). Where this setup differs is that you don't need to insert a new trace observer endpoint in each service's configuration. Instead, you complete a separate configuration in the trace observer that specifies which data sources (services, apps, or entities) should have their data sent to the Infinite Tracing trace observer. If you already created a trace observer while setting up another type of service (for example, an APM agent), you can skip the steps about creating the trace observer and go right to [Initial trace observer configuration for browser, mobile, and Lambda](#initial-config) to select your data sources. ### Create a trace observer [#create-for-mlb] If you haven't already set up a trace observer, complete the following: 1. Go to **[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Infinite Tracing settings**. 2. Select an account in the upper-left dropdown. If you have access to multiple accounts, make sure you're in the account where you want Infinite Tracing enabled. If you can't add an observer, it's likely because we allow only [one observer per region, per account family](https://docs.newrelic.com/docs/distributed-tracing/concepts/how-new-relic-distributed-tracing-works/#infinite-architecture). 3. If no trace observers are already present, click **New trace observer** to add one, fill out the information, and click **Create**. Note that we offer the following regions: - **us-east-1 (US)** - **us-west-2 (US)** - **eu-central-1 (EU):** Select this region if you are collecting data from EU requests and you have an EU-based New Relic account that reports in EU data centers. - **eu-west-1 (US):** Select this region if you are collecting data from EU requests and you have a US-based New Relic account that reports in US data centers. - **ap-southeast-1 (US)** - **ap-southeast-2 (US)** - **us-east-2 (US, FedRAMP)** 4. Under the **Endpoints** dropdown, copy the endpoint value under **For other integrations** and have it ready. This value is referenced in our code examples as `YOUR_TRACE_OBSERVER_URL`. This is used to test the [trace observer](#sample-payload). 5. (Optional) To verify the trace observer is working, we recommend sending a sample trace payload. **Send sample payload** > #### ⚠️ IMPORTANT > > If you're using Zipkin-format data, see [Send Zipkin payload](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/trace-api/report-zipkin-format-traces-trace-api#send-zipkin-curl). This test sends a sample trace payload with one trace and two spans from a service named `Test Service A`. To send this sample request: 1. Get the license key for the account you want to report data to and have it ready. 2. Copy the following `curl` request into a text editor: ```bash curl -i -H 'Content-Type: application/json' \ -H 'Api-Key: YOUR_LICENSE_KEY' \ -H 'Data-Format: newrelic' \ -H 'Data-Format-Version: 1' \ -X POST \ -d '[ { "common": { "attributes": { "environment": "staging" } }, "spans": [ { "trace.id": "123456", "id": "ABC", "attributes": { "duration.ms": 12.53, "host": "host123.example.com", "name": "/home", "service.name": "Test Service A" } }, { "trace.id": "123456", "id": "DEF", "attributes": { "error.message": "Invalid credentials", "service.name": "Test Service B", "host": "host456.example.com", "duration.ms": 2.97, "name": "/auth", "parent.id": "ABC" } } ] } ]' 'YOUR_TRACE_OBSERVER_URL' ``` 3. Insert your own values into the `curl` request: | Value | Description | | ------------------------- | ---------------------------------------------------------------------------------------------- | | `YOUR_LICENSE_KEY` | Replace this with your license key. | | `YOUR_TRACE_OBSERVER_URL` | Replace this with the **For other integrations** endpoint value you copied in a previous step. | 4. Copy the curl request into a terminal and execute it. 5. The test should return `HTTP/1.1 202 Accepted`, indicating success. If it does not, check the following common issues: - Confirm that you used the [**For other integrations** endpoint](#endpoints) value. - Confirm you're using single quotes around `YOUR_TRACE_OBSERVER_URL`. - Check that you're using the correct API key. 6. If your test returned `HTTP/1.1 202 Accepted`, go to the New Relic UI to see a query of the sample payload data using the span attribute `service.name = Test Service A` ([here's a link for that query](https://one.newrelic.com/launcher/distributed-tracing-nerdlets.distributed-tracing?launcher=eyJ0aW1lUmFuZ2UiOnsiYmVnaW5fdGltZSI6bnVsbCwiZW5kX3RpbWUiOm51bGwsImR1cmF0aW9uIjoxODAwMDAwfSwiJGlzRmFsbGJhY2tUaW1lUmFuZ2UiOnRydWV9&pane=eyJuZXJkbGV0SWQiOiJkaXN0cmlidXRlZC10cmFjaW5nLW5lcmRsZXRzLmRpc3RyaWJ1dGVkLXRyYWNpbmctbGF1bmNoZXIiLCJzb3J0SW5kZXgiOjAsInNvcnREaXJlY3Rpb24iOiJERVNDIiwicXVlcnkiOnsib3BlcmF0b3IiOiJBTkQiLCJpbmRleFF1ZXJ5Ijp7ImNvbmRpdGlvblR5cGUiOiJJTkRFWCIsIm9wZXJhdG9yIjoiQU5EIiwiY29uZGl0aW9ucyI6W119LCJzcGFuUXVlcnkiOnsib3BlcmF0b3IiOiJBTkQiLCJjb25kaXRpb25TZXRzIjpbeyJjb25kaXRpb25UeXBlIjoiU1BBTiIsIm9wZXJhdG9yIjoiQU5EIiwiY29uZGl0aW9ucyI6W3siYXR0ciI6InNlcnZpY2UubmFtZSIsIm9wZXJhdG9yIjoiRVEiLCJ2YWx1ZSI6IlRlc3QgU2VydmljZSBBIn1dfV19fX0=)). Because the sample payload contains an error attribute, the error sampler will mark it for keeping. If you modify the payload to remove the error attributes, the random sampler may not choose to keep this particular trace. > #### 💡 TIP > > Traces may take up to one minute to show up in the UI. ### Initial trace observer configuration for browser, mobile, and Lambda [#initial-config] Once you have a trace observer, you need to specify which data sources (services, apps, or entities) should have their data sent to the Infinite Tracing trace observer. Before you select your data sources, familiarize yourself with the following: **How long before I see my data in the New Relic UI?** When you select your browser, mobile, and Lambda data sources, you should expect a delay of up to three minutes before you see your results in the New Relic UI for that trace observer. This allows the trace observer to wait long enough for these high-latency data sources to arrive before making a sampling decision. For example, if you re-run the [sample curl test](#sample-request) test after you add browser, mobile, or Lambda data sources, you should expect a delay before the data appears in the UI. **What if my browser app is connected to an APM agent that has multiple names?** If your browser app is associated with an [APM application that has multiple names](https://docs.newrelic.com/docs/apm/agents/manage-apm-agents/app-naming/use-multiple-names-app/), you can only select one from the group of multi-named browser apps when you choose your data source. If you try to select the same application that goes by another name, the UI will prompt you that you need to choose only one browser app. **How does Lambda sampling work?** By default, when you set up Lambda layers, you automatically get head-based sampling. When you enable Infinite Tracing for Lambda, your head-based sampling data is additionally sampled by the trace observer. The main benefit to enabling Infinite Tracing for your Lambda functions is to ensure their spans are included with spans from other services tracked by Infinite Tracing. The procedure below is for using the New Relic UI, but if you prefer a programmatic approach, you can [specify data sources with GraphQL](https://docs.newrelic.com/docs/apis/nerdgraph/examples/configure-infinite-tracing-graphql/#updating-the-data-sources). If you use GraphQL, return to this procedure and continue with step 4 (optional configurations) below. To finish the configuration for browser, mobile, and Lambda in the UI: 1. Return to the trace observer app (**[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Infinite Tracing settings**). 2. For your trace observer, go to the end of the row, click the ellipsis icon **...**, and then click **Edit mobile, browser, lambda sources**. ![Screenshot showing menu when you click the three dots next to your trace observer](https://docs.newrelic.com/images/distributed-tracing_screenshot-crop_mobile-lambda-menu-.webp "Screenshot showing menu when you click the ellipsis next to your trace observer") 3. Select the data sources you want to include, keeping the following in mind: - You can use the filter to narrow your options. - If your browser app is connected to an APM agent that has multiple names, you can can only choose one name. - If an entry is gray, it is already being used by another trace observer. 4. Bookmark the [optional configurations](#follow-up-config) below so you can return to them after you experiment with Infinite Tracing. 5. View your Infinite Tracing data in the [New Relic distributed tracing UI](https://docs.newrelic.com/docs/distributed-tracing/ui-data/understand-use-distributed-tracing-ui/). 6. If you also have [services instrumented with APM agents or third-party telemetry tools](#apm-and-third-party) follow the steps in that section to ensure their data is reported to New Relic. ## (Optional) Additional configuration for all service types [#follow-up-config] > #### 💡 TIP > > You may want to postpone these optional configurations until after you've worked with this feature for a while. You can control the behavior of the trace observer with various configurations. Here are some of the features you can configure: - [Configure trace observer monitoring](https://docs.newrelic.com/docs/distributed-tracing/infinite-tracing/infinite-tracing-configure-trace-observer-monitoring/): Allows you to see the amount of traces seen and retained. - [Configure span attribute trace filter](https://docs.newrelic.com/docs/distributed-tracing/infinite-tracing/infinite-tracing-configure-span-attribute-trace-filter/): Make custom filter rules for spans. - [Infinite Tracing: Configure random trace filter](https://docs.newrelic.com/docs/distributed-tracing/infinite-tracing/infinite-tracing-configure-random-trace-filter): Adjust the percentage used by the random trace filter.