---
title: Enable distributed tracing for your Go applications
source: https://docs.newrelic.com/docs/apm/agents/go-agent/configuration/distributed-tracing-go-agent
---

Distributed tracing allows you to see the entire journey of your requests throughout a [distributed system](https://docs.newrelic.com/docs/distributed-tracing/concepts/introduction-distributed-tracing). For the Go agent, we offer two types of distributed tracing (for more details, see [How span sampling works](https://docs.newrelic.com/docs/distributed-tracing/concepts/how-new-relic-distributed-tracing-works/#sampling)):

-   Standard (head-based sampling): Before any traces arrive, we determine a set percentage of traces to accept and analyze. This gives you a solid starting point to see how tracing can help you. It is turned on by default in Go agents 3.16.0 and higher. By default, the Go agent samples 10 traces per minute, with a max span limit of 2000 spans per minute.

-   Infinite Tracing (tail-based sampling): Our cloud-based service accepts all your traces and then sorts through them to find the most important. Infinite Tracing analyzes all of your traces and gives you configuration options to sample the traces that matter most to you.

Whether you just want to try out standard distributed tracing (head-based sampling) or also want to set up Infinite Tracing (tail-based sampling), you need to start by setting up standard tracing.

## Standard distributed tracing [#quick-start-apm]

This is the best approach to set up [standard distributed tracing](https://docs.newrelic.com/docs/distributed-tracing/concepts/how-new-relic-distributed-tracing-works/#head-based) if you haven't installed any APM agents for your services yet.

> #### 💡 TIP
>
> When you install the New Relic Go agent, standard distributed tracing is turned on by default. If you prefer to turn it off, see our [configuration guide](https://docs.newrelic.com/docs/apm/agents/go-agent/configuration/go-agent-configuration).

### Identify services [#identify]

Figure out which services touch your request so you can instrument each of them to send trace data to New Relic.

### Instrument each service with an APM agent [#update-agents]

For each service involved in your transactions, you'll perform separate installations of the agent. If some of your services use other languages, simply repeat the [installation steps](https://docs.newrelic.com/docs/distributed-tracing/enable-configure/quick-start) for those languages.

> #### 💡 TIP
>
> The Go agent requires you to manually instrument your Go services, unlike the auto-instrumentation of the other New Relic agents. This means you need to add some lines to your code to use the Go agent. You'll learn about this when you start the installation below.

To start the installation routine, click the tile below. When you're finished installing each agent, return here to see tips for [viewing your traces](#view-traces).

1.  [Start installation](https://one.newrelic.com/marketplace/install-data-source?account=2498654&duration=1800000&state=4b9222ea-058a-c29d-b561-0bf33a98e412)

### View traces [#view-traces]

After you instrument each of your services with APM agents, generate some traffic in your application so we can capture some traces. Here are some ways to view your traces in the UI:

**View traces that include a specific service**

Here's one way you can see traces for a particular service:

1.  Go to **[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > APM & services**.
2.  Click your entity (service).
3.  In the left pane's **Monitor** section, click **Distributed tracing**.
4.  For details, click on an individual trace. If **Group similar traces** is on in the top menu, click on a trace group, and then click on an individual trace.
5.  If you don't see the traces you want, you can filter by the `trace.id`.

**View traces across accounts**

To see traces that cross accounts:

1.  Go to **[one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Traces**.
2.  Select your entity (service) in the left pane.
3.  For details, click on an individual trace. If **Group similar traces** is on in the top menu, click on a trace group, and then click on an individual trace.
4.  If you don't see the traces you want, you can filter by the `trace.id`.

**Examine logs related to traces**

You can bring your logs and tracing details together to make troubleshooting easier and faster. With [logs in context](https://docs.newrelic.com/docs/logs/logs-context/configure-logs-context-apm-agents/), you can see log messages alongside traces in the New Relic UI.

After you've located an interesting trace using the steps in [View traces that include a specific service](#entity-explorer) or [View traces across accounts](#cross-acct-traces), do the following:

1.  If you've enabled logs in context, click on the **Logs** tab (next to **Trace details**).
2.  To view details related to an individual log, click directly on the message.

For more help finding your traces in the UI:

-   [Understand and use the distributed tracing UI](https://docs.newrelic.com/docs/distributed-tracing/ui-data/understand-use-distributed-tracing-ui)
-   [Query distributed trace data](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/ui-data/query-distributed-trace-data)

## Infinite Tracing (optional) [#infinite-tracing]

With Infinite Tracing (tail-based sampling) we can analyze all of your data and determine which traces are the most relevant.

### Complete the setup for standard distributed tracing [#install-agents-first]

The Infinite Tracing setup builds on standard distributed tracing. So, make sure you've completed the [steps above](#quick-start-apm), and then continue with the trace observer setup.

### Set up the trace observer [#provision-trace-observer]

The trace observer is a New Relic AWS-based service that collects and analyzes all your traces. Follow the instructions in [Set up trace observer](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/infinite-tracing/set-trace-observer). When you're done, return here with your trace observer information and continue with the next step to configure the agent.

### Configure the agent for Infinite Tracing [#configure-agent-inf]

Infinite Tracing configuration settings include the standard distributed tracing plus information about the trace observer:

**Go Infinite Tracing configuration**

Here's an overview of the settings.

| Type             | Required configuration                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Infinite Tracing | Configuration options: - `newrelic.Config` structure: ```go app, err := newrelic.NewApplication(     newrelic.ConfigAppName(YOUR_APP_NAME),     newrelic.ConfigLicense(YOUR_LICENSE_KEY),     func(cfg *newrelic.Config) {         cfg.InfiniteTracing.TraceObserver.Host = YOUR_TRACE_OBSERVER_HOST     }, ) ``` - Environment variable: ``` NEW_RELIC_INFINITE_TRACING_TRACE_OBSERVER_HOST=<a href="/docs/understand-dependencies/distributed-tracing/infinite-tracing/set-trace-observer#ui-endpoints">YOUR_TRACE_OBSERVER_HOST</a> ``` |

> #### 💡 TIP
>
> If you need help with proxy configuration, see [Proxy support](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/other-requirements/infinite-tracing-proxy-support).

### (Optional) Customize Infinite Tracing [#customize-inf-tracing]

After you add the agent configuration settings, you should start seeing data in the New Relic UI. After you spend some time analyzing your data, you may want to adjust some of the features of Infinite Tracing:

-   [Configure trace observer monitoring](https://docs.newrelic.com/docs/distributed-tracing/infinite-tracing/infinite-tracing-configure-trace-observer-monitoring)
-   [Configure span attribute trace filter](https://docs.newrelic.com/docs/distributed-tracing/infinite-tracing/infinite-tracing-configure-span-attribute-trace-filter)
-   [Configure random trace filter](https://docs.newrelic.com/docs/distributed-tracing/infinite-tracing/infinite-tracing-configure-random-trace-filter)

## Additional manual instrumentation [#guidelines]

All installations of the Go agent and distributed tracing require some manual instrumentation using the settings listed in [Go agent configuration settings](https://docs.newrelic.com/docs/agents/go-agent/configuration/go-agent-configuration#distributed-tracing). Still, you may need to do some additional configuration to optize your setup. Here are some guidelines for instrumenting transactions and HTTP requests.

**Instrument transactions (if using ServeMux)**

If you are using Go's [`http.ServeMux`](https://golang.org/pkg/net/http/#ServeMux) and want to enable New Relic's distributed tracing, your Go application must be instrumented with New Relic's `WrapHandle` and `WrapHandleFunc` wrappers. These wrappers automatically start and end transactions with the request and response writer, which will automatically add the correct distributed tracing headers. For more on how header propagation works, see [How distributed tracing works](https://docs.newrelic.com/docs/apm/distributed-tracing/getting-started/how-new-relic-distributed-tracing-works).

**Before and after example**

Here is an example of code before instrumentation:

````go
http.HandleFunc("/users", usersHandler)
```

And here is an example of that same code after instrumentation:

```go
http.HandleFunc(newrelic.WrapHandleFunc(app, "/users", usersHandler))
```

````

Read more about using these wrappers in [Instrument Go transactions](https://docs.newrelic.com/docs/agents/go-agent/get-started/instrument-go-transactions#http-handler-txns).

**Instrument outbound HTTP requests as external segments**

In order to have your outbound HTTP requests eligible for distributed tracing, [create an external segment](https://docs.newrelic.com/docs/agents/go-agent/get-started/instrument-go-segments#go-external-segments).

The easiest way to create an external segment for your outbound HTTP request is to use the `newrelic.NewRoundTripper` method. Here is an example of making a request to `http://api.example.com` that includes the outgoing distributed tracing headers:

````go
func useNewRoundTripper(txn *newrelic.Transaction) (*http.Response, error) {
    client := &http.Client{}
    client.Transport = newrelic.NewRoundTripper(client.Transport)
    request, _ := http.NewRequest("GET", "http://example.com", nil)
    request = newrelic.RequestWithTransactionContext(request, txn)
    return client.Do(request)
}
```

If you have a more complex request that uses the Go standard library's `http.Request`, use the `newrelic.StartExternalSegment` method to ensure your outbound request is eligible for distributed tracing:

```go
func external(txn *newrelic.Transaction, req *http.Request) (*http.Response, error) { 
    s := newrelic.StartExternalSegment(txn, req) 
    response, err := http.DefaultClient.Do(req) 
    s.Response = response 
    s.End() 
    return response, err
}
```

An `ExternalSegment` created with a struct literal cannot be used for distributed tracing. Because of this, New Relic recommends using [`newrelic.NewRoundTripper` or `newrelic.StartExternalSegment`](/docs/agents/go-agent/get-started/instrument-go-segments#go-external-segments).

```go
func noGoodForDt(txn *newrelic.Transaction, url string) (*http.Response, error) {
    // Distributed tracing headers are not added to the outgoing request.
    // Use newrelic.NewRoundTripper or newrelicc.StartExternalSegment instead.
    defer newrelic.ExternalSegment{
        StartTime: txn.StartSegmentNow(),
        URL:       url,
    }.End()

    return http.Get(url)
}
```

````

**Manually create and accept distributed trace payload**

The distributed trace payload contains information that allows New Relic to stitch together transactions occurring in multiple services into a complete transaction trace. If New Relic-monitored services are not sending trace context to each other, it will result in incomplete trace details.

For general instructions on how to use the API calls below to implement distributed tracing, first see [Use distributed tracing APIs](https://docs.newrelic.com/docs/enable-distributed-tracing#agent-apis).

| If you want to...                                                                                | Use this                                                                                                                                                                         |
| ------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Create a payload to be sent to the called service.                                               | ````go <a href="https://godoc.org/github.com/newrelic/go-agent/v3/newrelic#Transaction.InsertDistributedTraceHeaders">InsertDistributedTraceHeaders(h http.Header)</a> ```  ```` |
| Accept a payload sent from the first service; this will link these services together in a trace. | ````go <a href="https://godoc.org/github.com/newrelic/go-agent/v3/newrelic#Transaction.AcceptDistributedTraceHeaders">AcceptDistributedTraceHeaders(h http.Header)</a> ```  ```` |

For more details on using these, see the [Go agent GitHub repo](https://github.com/newrelic/go-agent/blob/master/GUIDE.md#distributed-tracing).