---
title: Link APM-instrumented applications to Kubernetes
source: https://docs.newrelic.com/docs/kubernetes-pixie/kubernetes-integration/advanced-configuration/link-apm-applications-kubernetes
---

You can surface Kubernetes metadata and link it to your [APM agents](https://docs.newrelic.com/docs/integrations/kubernetes-integration/metadata-injection/kubernetes-apm-metadata-injection#agent-compatibility) as [distributed traces](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/get-started/introduction-distributed-tracing) to explore performance issues and troubleshoot transaction errors. For more information, see [this blog post about monitoring app performance via Kubernetes](https://blog.newrelic.com/engineering/monitoring-application-performance-in-kubernetes/).

The metadata injection product uses a `MutatingAdmissionWebhook` to add the following environment variables to pods:

```yaml
NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME
NEW_RELIC_METADATA_KUBERNETES_NODE_NAME
NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME
NEW_RELIC_METADATA_KUBERNETES_DEPLOYMENT_NAME
NEW_RELIC_METADATA_KUBERNETES_POD_NAME
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_IMAGE_NAME
```

> #### 💡 TIP
>
> Our Kubernetes metadata injection project is open source. Here's the [code to link APM and infrastructure data](https://github.com/newrelic/k8s-metadata-injection).

## Compatibility and requirements [#compatibility]

To connect your applications to Kubernetes, you need to be able to deploy \`MutatingWebhookConfiguration' to your Kubernetes cluster.

To verify that you have the required permissions, run this command:

```shell
kubectl auth can-i create mutatingwebhookconfigurations.admissionregistration.k8s.io -A
```

The output for the command above should be something similar to:

```shell
yes
```

If you see a different result, follow the Kubernetes documentation to [enable admission control in your cluster](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#how-do-i-turn-on-an-admission-controller).

### Network requirements [#network-req]

For Kubernetes to talk to our `MutatingAdmissionWebhook`, the control plane node (or API server container, depending on how the cluster is set up) should allow egress for HTTPS traffic on port 443 to pods in all other nodes in the cluster.

This may require specific configuration depending on how your infrastructure is set up (on-premises, AWS, Google Cloud, etc.).

### APM agent compatibility [#agent-compatibility]

The following New Relic agents collect Kubernetes metadata:

-   [Go 2.3.0 or higher](https://docs.newrelic.com/docs/agents/go-agent/installation/install-new-relic-go#get-new-relic)
-   [Java 4.10.0 or higher](https://docs.newrelic.com/docs/agents/java-agent/installation/upgrade-java-agent)
-   [Node.js 5.3.0 or higher](https://docs.newrelic.com/docs/agents/nodejs-agent/installation-configuration/upgrade-nodejs-agent)
-   [Python 4.14.0 or higher](https://docs.newrelic.com/docs/agents/python-agent/installation-configuration/upgrade-python-agent)
-   [Ruby 6.1.0 or higher](https://docs.newrelic.com/docs/agents/ruby-agent/installation/upgrade-ruby-agent-versions)
-   [.NET 8.17.438 or higher](https://docs.newrelic.com/docs/agents/net-agent/installation/update-net-agent)

## Set up the injection of metadata [#configure-injection]

When you [install our integration using Helm](/install/kubernetes/?dropdown1=helm), it includes metadata injection. When configuring the [`nri-bundle`](https://github.com/newrelic/helm-charts/tree/master/charts/nri-bundle) chart, make sure you enable the metadata injection webhook as follows.

```yaml
nri-metadata-injection:
    enabled: true
```

**Important:** Your application pods will need to be restarted after the nri-metadata-injection is deployed, so that they can pick up the necessary environment variables.

By default, all the pods you create that include APM agents have the correct environment variables set and the metadata injection applies to the entire cluster. To check that the environment variables have been set, any running container must be stopped and a new instance started. See [Validate the injection of metadata](https://docs.newrelic.com/docs/integrations/kubernetes-integration/link-your-applications/link-your-applications-kubernetes/#validate-injection) for more info.

This default configuration also uses the [Kubernetes certificates API](https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/) to automatically manage the certificates required for the injection. If needed, you can limit the injection of metadata to specific namespaces in your cluster or self-manage your certificates.

## Custom configuration [#custom-configuration]

### Limit namespaces subject to injection [#limit-namespaces]

You can limit the injection of metadata only to specific namespaces by using labels.

To enable this feature, add the following to `values-newrelic.yaml` file:

```yaml
nri-metadata-injection:
    injectOnlyLabeledNamespaces: true
```

With this option, injection is only applied to those namespaces that have the `newrelic-metadata-injection` label set to `enabled`:

```shell
kubectl label namespace YOUR_NAMESPACE newrelic-metadata-injection=enabled
```

### Use cert-manager to generate certs [#cert-manager]

By default, our chart uses [`kube-webhook-certgen`](https://github.com/kubernetes/ingress-nginx/tree/main/images/kube-webhook-certgen) to automatically generate the required certificates for the webhook to run.

However, if you've [cert-manager](https://cert-manager.io/) installed, you can configure our chart to use it instead, which can make deploying much easier:

```yaml
nri-metadata-injection:
  certManager:
    enabled: true
```

### Manage custom certificates [#option-activation]

> #### 💡 TIP
>
> Manually managing webhook certificates is recommended for advanced users only. New Relic support team might not be able to help troubleshooting this configuration.

To use custom certificates you need to disable the automatic installation of certificates when you are installing using Helm.

To disable the installation for certificates just modify nri-bundle Helm `values.yaml` like this:

```yaml
nri-metadata-injection:
  customTLSCertificate: true
```

Now you can proceed with the custom certificate management option. You need your certificate, server key, and Certification Authority (CA) bundle encoded in PEM format.

-   If you've them in the standard certificate format (X.509), install `openssl`, and run the following:

    ```shell
    openssl x509 -in YOUR_CERTIFICATE_FILENAME -outform PEM -out YOUR_CERTIFICATE_FILENAME.pem
    openssl x509 -in YOUR_SERVER_KEY_FILENAME -outform PEM -out YOUR_SERVER_KEY_FILENAME.pem
    openssl x509 -in YOUR_CA_BUNDLE_FILENAME -outform PEM -out YOUR_BUNDLE_FILENAME.pem
    ```

-   If your certificate and key pair are in another format, see the [Digicert knowledgebase](https://knowledge.digicert.com/solution/SO26449.html) for more help.

Create the TLS secret with the signed certificate/key pair, and patch the mutating webhook configuration with the CA using the following commands:

```shell
kubectl create secret tls YOUR_NEWRELIC_METADATA_INJECTION_ADMISSION \
  --key=YOUR_PEM_ENCODED_SERVER_KEY \
  --cert=YOUR_PEM_ENCODED_CERTIFICATE \
  --dry-run -o yaml |
kubectl -n newrelic apply -f -

caBundle=$(cat YOUR_PEM_ENCODED_CA_BUNDLE | base64 | td -d $'\n')
kubectl patch mutatingwebhookconfiguration newrelic-metadata-injection-cfg --type='json' -p "[{'op': 'replace', 'path': '/webhooks/0/clientConfig/caBundle', 'value':'${caBundle}'}]"
```

> #### ⚠️ IMPORTANT
>
> Certificates signed by Kubernetes have an expiration of one year. For more information, see the [Kubernetes source code in GitHub](https://github.com/kubernetes/kubernetes/blob/1b28775db1290a772967d192a19a8ec447053cd5/pkg/controller/apis/config/v1alpha1/defaults.go#L118).

## Validate the injection of metadata [#validate-injection]

Deploy a new pod and check for the New Relic environment variables to verify the correct installation of the webhook responsible for injecting the metadata.

1.  Create a dummy nginx pod by running:

    ```shell
    kubectl run test-nginx --image nginx -n newrelic
    ```

2.  Check if New Relic environment variables were injected:

    ```shell
    kubectl exec -n newrelic test-nginx -- env | grep NEW_RELIC_METADATA_KUBERNETES
    ```

The expected output would be something like the following:

```ini
NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME=THE_CLUSTER_NAME
NEW_RELIC_METADATA_KUBERNETES_NODE_NAME=nodea
NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME=newrelic
NEW_RELIC_METADATA_KUBERNETES_POD_NAME=test-nginx
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME=nginx
```

## Disable the injection of metadata [#uninstall]

To uninstall the injection of metadata, change your `values-newrelic.yaml` file as follows:

```yaml
webhook:
    enabled: false
```

After that, re-run the [installation command](/install/kubernetes/?dropdown1=helm).

## Troubleshooting [#metadata-troubleshooting]

Follow these troubleshooting tips as needed.

**No Kubernetes metadata in APM or distributed tracing transactions**

### Problem

The creation of the secret by the `k8s-webhook-cert-manager` job used to fail due to the `kubectl` version used by the image when running in Kubernetes version 1.19.x,
The new version [1.3.2](https://github.com/newrelic/k8s-webhook-cert-manager/releases/tag/v1.3.2) fixes this issue, therefore it is enough to run again the job using an update version of the image to fix the issue.

### Solution

1.  Update the image `k8s-webhook-cert-manager` to a version >= [1.3.2](https://github.com/newrelic/k8s-webhook-cert-manager/releases/tag/v1.3.2) and re-run the job.
2.  Check that the secret was created correctly and that the `k8s-metadata-injection` pod starts.
3.  Note that the new version of the manifest and of the [`nri-bundle`](https://github.com/newrelic/helm-charts/tree/master/charts/nri-bundle) are already updated with the correct version of the image.

**No Kubernetes metadata in the transaction attributes**

### Problem

There is no Kubernetes metadata included in the transaction attributes of your APM agent or in distributed tracing.

### Solution

1.  Verify that the environment variables have been correctly injected into new pods by following the instructions described in the [Validate the injection of metadata](https://docs.newrelic.com/docs/integrations/kubernetes-integration/metadata-injection/kubernetes-apm-metadata-injection#validate-injection) section.

2.  If they do not exist, firstly, tail the logs of the webhook service:

    ```shell
    kubectl logs -l app.kubernetes.io/name=nri-metadata-injection -f
    ```

3.  In another terminal, create a new pod and inspect the logs of the metadata injection deployment for errors. See [Validate the injection of metadata](https://docs.newrelic.com/docs/integrations/kubernetes-integration/metadata-injection/kubernetes-apm-metadata-injection#validate-injection) section to create a new pod. For each pod created, there should be log entry, assuming the [log level (logLevel)](https://github.com/newrelic/k8s-metadata-injection/blob/main/charts/nri-metadata-injection/README.md#values) is at INFO or lower:

    ```json
    {"level":"info","ts":"2026-02-16T09:58:05.934Z","caller":"server/webhook.go:160","msg":"admission review received","operation":"CREATE","kind":"Pod","namespace":"default","podGenerateName":"test-pod-2-55476b7f5c-","uid":"0b689368-5395-4cda-9b8c-61fa2705f6ee","mutate":true}
    ```

    If there are no new entries in the logs, it means that the API server can't communicate with the webhook service, this could be due to network rules or security groups rejecting the communication.

4.  To check if the API server is not being able to communicate with the webhook you should inspect the API server logs for errors like:

    ```shell
    failed calling webhook "metadata-injection.newrelic.com": THE_ERROR_REASON
    ```

    To get the API server logs:

5.  Start a proxy to the Kubernetes API server by the executing this command in a terminal window and keep it running.

    ```shell
    kubectl proxy --port=8001
    ```

6.  Create a new pod in your cluster, this will make the API server try to communicate with the webhook. This command will create a busybox.

    ```shell
    kubectl create -f https://git.io/vPieo
    ```

7.  Retrieve the API server logs.

    ```shell
    curl localhost:8001/logs/kube-apiserver.log > apiserver.log
    ```

8.  Delete the busybox container.

    ```shell
    kubectl delete -f https://git.io/vPieo
    ```

9.  Inspect the logs for errors.

    ```shell
    grep -E 'failed calling webhook' apiserver.log
    ```

    > #### 💡 TIP
    >
    > One of the [requirements for the metadata injection](#network-req) is that the API server must be allowed egress to the pods running on the cluster. If you encounter errors regarding connection timeouts or failed connections, make sure to check the cluster's security groups and firewall rules.

10. If there are no log entries in either the API server logs or the metadata injection deployment, it means that the webhook was not registered properly.

11. Ensure the metadata injection setup jobs ran successfully by inspecting the output of this command:

    ```shell
    kubectl get job nri-metadata-injection-admission-create
    kubectl get job nri-metadata-injection-admission-patch
    ```

12. If the job isn't completed, investigate the logs of the setup job:

    ```shell
    kubectl logs job/nri-metadata-injection-admission-create
    kubectl logs job/nri-metadata-injection-admission-patch
    ```

13. Ensure the `CertificateSigningRequest` is approved and issued by running this command:

    ```shell
    kubectl get csr newrelic-metadata-injection-svc.default
    ```

14. Ensure the TLS secret is present by running this command:

    ```shell
    kubectl get secret nri-metadata-injection-admission
    ```

15. Ensure the CA bundle is present in the mutating webhook configuration:

    ```shell
    kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io nri-metadata-injection -o json
    ```

16. Ensure the `TargetPort` of the **Service** resource matches the **Port** of the **Deployment**'s container:

    ```shell
    kubectl describe service/nri-metadata-injection
    kubectl describe deployment/nri-metadata-injection
    ```