--- title: Distributed tracing for your Node.js services source: https://docs.newrelic.com/docs/apm/agents/nodejs-agent/installation-configuration/distributed-tracing-nodejs-agent --- Distributed tracing allows you to see the entire journey of your requests throughout a [distributed system](https://docs.newrelic.com/docs/distributed-tracing/concepts/introduction-distributed-tracing). For the Node.js agent, we offer two types of distributed tracing (for more details, see [How span sampling works](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/get-started/how-new-relic-distributed-tracing-works#sampling)): - Standard (head-based sampling): Before any traces arrive, we determine a set percentage of traces to accept and analyze. This gives you a solid starting point to see how tracing can help you. It is turned on by default in Node.js agents 8.3.0 and higher. By default, the Node.js agent samples 10 traces per minute, with a max span limit of 2000 spans per minute. - Infinite Tracing (tail-based sampling): Our cloud-based service accepts all your traces and then sorts through them to find the most important. Infinite Tracing analyzes all of your traces and gives you configuration options to sample the traces that matter most to you. Whether you just want to try out standard distributed tracing (head-based sampling) or also want to set up Infinite Tracing (tail-based sampling), you need to start by setting up standard tracing. ## Standard distributed tracing [#quick-start-apm] This is the best approach to set up [standard distributed tracing](https://docs.newrelic.com/docs/distributed-tracing/concepts/how-new-relic-distributed-tracing-works/#head-based) if you haven't installed any APM agents for your services yet. > #### 💡 TIP > > When you install the New Relic Node.js agent, standard distributed tracing is turned on by default. If you prefer to turn it off, see our [configuration guide](https://docs.newrelic.com/docs/apm/agents/nodejs-agent/installation-configuration/nodejs-agent-configuration). ### Identify services [#identify] Figure out which services you want to instrument so they each send trace data to New Relic. ### Instrument each service with an APM agent [#update-agents] For each service involved in your transactions, you'll perform separate installations of the agent. If some of your services use other languages, simply repeat the [installation steps](https://docs.newrelic.com/docs/distributed-tracing/enable-configure/quick-start) for those languages. To start the installation routine, click the tile below. When you're finished installing each agent, return here to see tips for [viewing your traces](#view-traces). 1. [Start installation](https://one.newrelic.com/launcher/nr1-core.settings?pane=eyJuZXJkbGV0SWQiOiJ0dWNzb24ucGxnLWluc3RydW1lbnQtZXZlcnl0aGluZyJ9&cards[0]=eyJuZXJkbGV0SWQiOiJzZXR1cC1uZXJkbGV0cy5zZXR1cC1ub2RlLWludGVncmF0aW9uIiwiYWNjb3VudElkIjoyNjQwNDA5fQ==&platform[accountId]=2498654&platform[timeRange][duration]=1800000&platform[$isFallbackTimeRange]=true) ### View traces [#view-traces] After you instrument each of your services with APM agents, generate some traffic in your application so we can capture some traces. Here are two ways to view your traces in the UI: **View traces that include a specific service** Here's one way you can see traces for a particular service: 1. Go to **[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > APM & services**. 2. Click your entity (service). 3. In the left pane's **Monitor** section, click **Distributed tracing**. 4. For details, click on an individual trace. If **Group similar traces** is on in the top menu, click on a trace group, and then click on an individual trace. 5. If you don't see the traces you want, you can filter by the `trace.id`. **View traces across accounts** To see traces that cross accounts: 1. Go to **[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Traces**. 2. Select your entity (service) in the left pane. 3. For details, click on an individual trace. If **Group similar traces** is on in the top menu, click on a trace group, and then click on an individual trace. 4. If you don't see the traces you want, you can filter by the `trace.id`. **Examine logs related to traces** You can bring your logs and tracing details together to make troubleshooting easier and faster. With [logs in context](https://docs.newrelic.com/docs/logs/logs-context/configure-logs-context-apm-agents/), you can see log messages alongside traces in the New Relic UI. After you've located an interesting trace using the steps in [View traces that include a specific service](#entity-explorer) or [View traces across accounts](#cross-acct-traces), do the following: 1. If you've enabled logs in context, click on the **Logs** tab (next to **Trace details**). 2. To view details related to an individual log, click directly on the message. For more help finding your traces in the UI: - [Understand and use the distributed tracing UI](https://docs.newrelic.com/docs/distributed-tracing/ui-data/understand-use-distributed-tracing-ui) - [Query distributed trace data](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/ui-data/query-distributed-trace-data) ## Infinite Tracing (optional) [#infinite-tracing] With Infinite Tracing (tail-based sampling) we can analyze all of your data and determine which traces are the most relevant. ### Complete the setup for standard distributed tracing [#install-agents-first] The Infinite Tracing setup builds on standard distributed tracing. So, make sure you've completed the [steps above](#quick-start-apm), and then continue with the trace observer setup. ### Set up the trace observer [#provision-trace-observer] The trace observer is a New Relic AWS-based service that collects and analyzes all your traces. Follow the instructions in [Set up trace observer](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/infinite-tracing/set-trace-observer). When you're done, return here with your trace observer information and continue with the next step to configure the agent. ### Configure the agent for Infinite Tracing [#configure-agent-inf] Infinite Tracing configuration settings include the standard distributed tracing plus information about the trace observer. Find the settings for your language agent below: **Node.js Infinite Tracing configuration** Here's an overview of the settings. For more help with configuration, see [Node.js agent configuration](https://docs.newrelic.com/docs/agents/nodejs-agent/installation-configuration/nodejs-agent-configuration#distributed-tracing). | Type | Required configuration | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Infinite Tracing | Configuration options: Configuration file (`newrelic.js`): ```` distributed_tracing: { enabled: true } infinite_tracing: { trace_observer: { host: 'YOUR_TRACE_OBSERVER_HOST' } } ``` Environment variables: ``` NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST="YOUR_TRACE_OBSERVER_HOST" ``` ```` | > #### 💡 TIP > > If you need help with proxy configuration, see [Proxy support](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/other-requirements/infinite-tracing-proxy-support). ### (Optional) Customize Infinite Tracing [#customize-inf] After you add the agent configuration settings, you should start seeing data in the New Relic UI. After you spend some time analyzing your data, you may want to adjust some of the features of Infinite Tracing: - [Configure trace observer monitoring](https://docs.newrelic.com/docs/distributed-tracing/infinite-tracing/infinite-tracing-configure-trace-observer-monitoring) - [Configure span attribute trace filter](https://docs.newrelic.com/docs/distributed-tracing/infinite-tracing/infinite-tracing-configure-span-attribute-trace-filter) - [Configure random trace filter](https://docs.newrelic.com/docs/distributed-tracing/infinite-tracing/infinite-tracing-configure-random-trace-filter) ## Manual instrumentation (If automatic instrumentation doesn't work) [#manual-instrumentation] Recommendation: Before performing any custom instrumentation, read: - [How distributed tracing works](https://docs.newrelic.com/docs/apm/distributed-tracing/getting-started/how-new-relic-distributed-tracing-works) - [Troubleshoot missing data](https://docs.newrelic.com/docs/apm/distributed-tracing/troubleshooting/troubleshooting-missing-trace-data) If a service is not passing the trace header to other services, you can use the distributed tracing payload APIs to instrument the [calling service](#calling-service) and the [called service](#called-service). The calling service uses an API call to generate a payload, which is accepted by the called service. **Instrument the calling service** To instrument the calling service: 1. Ensure the [version of the APM agent](#compatibility-requirements) that monitors the calling service supports distributed tracing. 2. Invoke the agent API call for generating a distributed trace payload (see the [Node.js agent API](https://newrelic.github.io/node-newrelic/TransactionHandle.html#insertDistributedTraceHeaders)). > #### ⚠️ IMPORTANT > > To maintain proper ordering of spans in a trace, ensure you **generate the payload in the context of the span that sends it**. 3. Add that payload to the call made to the destination service (for example, in a header). For more information, see the [distributed tracing example application](https://github.com/newrelic/newrelic-node-examples/tree/0a036fd669b4b47b8afbc7add8696980f799f0da/custom-instrumentation/distributed-tracing) on GitHub. **Instrument the called service** To instrument the called service: 1. Ensure the [version of the APM agent](#compatibility-requirements) that monitors the called service supports distributed tracing. 2. If the New Relic agent on the called service does not identify a New Relic transaction, use the agent API to declare a transaction. One way to tell that a transaction is not in progress is that the Node.js agent logs will report an error similar to this: ``` No transaction found when calling Transaction.acceptDistributedTracePayload. ``` Use [`startWebTransaction`](https://newrelic.github.io/node-newrelic/API.html#startWebTransaction) to create a web transaction or [`startBackgroundTransaction`](https://newrelic.github.io/node-newrelic/API.html#startBackgroundTransaction) to capture a [non-web transaction](https://docs.newrelic.com/docs/apm/transactions/intro-transactions/monitor-background-processes-other-non-web-transactions). For more information, check out the [startWebTransaction](https://github.com/newrelic/newrelic-node-examples/tree/0a036fd669b4b47b8afbc7add8696980f799f0da/custom-instrumentation/start-web-transaction) and [startBackgroundTransaction](https://github.com/newrelic/newrelic-node-examples/tree/0a036fd669b4b47b8afbc7add8696980f799f0da/custom-instrumentation/background-transactions) example applications on GitHub. 3. Extract the payload from the call that you received (for example, in a header). 4. Invoke the call for accepting the payload (see the [Node.js agent API](https://newrelic.github.io/node-newrelic/TransactionHandle.html#acceptDistributedTraceHeaders)) or check out the [distributed tracing example application](https://github.com/newrelic/newrelic-node-examples/tree/0a036fd669b4b47b8afbc7add8696980f799f0da/custom-instrumentation/distributed-tracing) on GitHub.