---
title: NVIDIA Triton integration
source: https://docs.newrelic.com/docs/infrastructure/host-integrations/host-integrations-list/nvidia-triton-integration
---

Our NVIDIA Triton integration monitors the deployment and management of AI models in production environments. Triton provides a flexible and scalable solution for deploying deep learning models, enabling organizations to efficiently deploy AI applications across a variety of hardware platforms, including GPUs and CPUs.

![NVIDIA Triton dashboard](https://docs.newrelic.com/images/infrastructure_screenshot-crop_triton-dashboard.webp "NVIDIA Triton dashboard")

After setting up our NVIDIA Triton integration, we give you a dashboard for your NVIDIA Triton metrics.

## Install the infrastructure agent [#infra-install]

To use the NVIDIA Triton integration, you need to also [install the infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent/get-started/install-infrastructure-agent-new-relic/) on the same host. The infrastructure agent monitors the host itself, while the integration you'll install in the next step extends your monitoring with NVIDIA Triton-specific data.

## Enable the NVIDIA Triton integration with `nri-prometheus` [#enable-triton]

The Triton server metrics are displayed at the URL `http://localhost:8002/metrics`.

> #### 💡 TIP
>
> For additional details on collecting Triton server metrics, please refer to the [NVIDIA documentation](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/metrics.html)

To set up the NVIDIA Triton integration, follow these steps:

1.  Run this command to create a file named `nri-prometheus-config.yml` in the integrations directory:

    ```shell
    touch /etc/newrelic-infra/integrations.d/nri-prometheus-config.yml
    ```

2.  Add the following snippet to your `nri-prometheus-config.yml` file to enable the agent to capture NVIDIA Triton data:

    ```yml
    integrations:
      - name: nri-prometheus
        config:
          # When standalone is set to false nri-prometheus requires an infrastructure agent to work and send data. Defaults to true
          standalone: false

          # When running with infrastructure agent emitters will have to include infra-sdk
          emitters: infra-sdk

          # The name of your cluster. It's important to match other New Relic products to relate the data.
          cluster_name: "YOUR_DESIRED_CLUSTER_NAME"

          targets:
            - description: NVIDIA Triton metrics list
              urls: ["http://localhost:8002/metrics"]

          #     tls_config:
          #     ca_file_path: "/etc/etcd/etcd-client-ca.crt"
          #     cert_file_path: "/etc/etcd/etcd-client.crt"
          #      key_file_path: "/etc/etcd/etcd-client.key"

          # Whether the integration should run in verbose mode or not. Defaults to false
          verbose: false

          # Whether the integration should run in audit mode or not. Defaults to false.
          # Audit mode logs the uncompressed data sent to New Relic. Use this to log all data sent.
          # It does not include verbose mode. This can lead to a high log volume, use with care
          audit: false

          # The HTTP client timeout when fetching data from endpoints. Defaults to 30s.
          # scrape_timeout: "30s"

          # Length in time to distribute the scraping from the endpoints
          scrape_duration: "5s"

          # Number of worker threads used for scraping targets.
          # For large clusters with many (>400) endpoints, slowly increase until scrape
          # time falls between the desired `scrape_duration`.
          # Increasing this value too much will result in huge memory consumption if too
          # many metrics are being scraped.
          # Default: 4
          # worker_threads: 4

          # Whether the integration should skip TLS verification or not. Defaults to false
          insecure_skip_verify: true
        timeout: 10s
    ```

## NVIDIA Triton logs configuration [#nvidia-triton-logs-configuration]

To configure nvidia triton logs, follow the steps outlined below.

1.  Run this Docker command to check the status of running containers:

    ```shell
    sudo docker ps
    ```

2.  Copy the container ID for the **nvidia-triton** container and execute this command:

    ```shell
    sudo docker logs -f <container_id> &> /tmp/triton.log &
    ```

    Afterwards, verify there is a log file named `triton.log` located in the `/tmp/` directory.

## Forwarding NVIDIA Triton logs to New Relic [#nvidia-triton-logs-to-newrelic]

You can use our [log forwarding](https://docs.newrelic.com/docs/logs/forward-logs/forward-your-logs-using-infrastructure-agent/) to forward NVIDIA Triton logs to New Relic.
On Linux machines, your log file named `logging.yml` should be in this path:

````shell
cd /etc/newrelic-infra/logging.d/
```

Once you find the log file in the above path, include this script into the `logging.yml` file:

```yml
logs:
  - name: triton.log
    file: /tmp/triton.log
    attributes:
      logtype: triton_logs
```

````

## Restart the New Relic infrastructure agent [#restart-agent]

Run this command to [restart your infrastructure agent](https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent/manage-your-agent/start-stop-restart-infrastructure-agent):

````shell
sudo systemctl restart newrelic-infra.service
```

In a couple of minutes, your NVIDIA Triton server will send metrics to [one.newrelic.com](https://one.newrelic.com).

````

## Find your data [#find-data]

You can choose our pre-built dashboard template named `NVIDIA Triton` to monitor your NVIDIA Triton server metrics. Follow these steps to use our pre-built dashboard template:

1.  Go to **[one.newrelic.com](https://one.newrelic.com) > Integrations & Agents** and type
    **NVIDIA Triton**.
2.  Under **Dashboards**, click **NVIDIA Triton**.
3.  Click **Edit** if you want to change the account in the open popup window.
4.  Click **Setup NVIDIA Triton** or **Skip this step** if you already setup this datasource.
5.  Click **View dashboard**, and see your NVIDIA Triton data in New Relic.

    You can find your custom NVIDIA Triton dashboard in the **Dashboards** UI. See our [dashboard](https://docs.newrelic.com/docs/query-your-data/explore-query-data/dashboards/introduction-dashboards) section for more information.

    Here is a [NRQL query](https://docs.newrelic.com/docs/nrql/get-started/introduction-nrql-new-relics-query-language/) to check the NVIDIA Triton CPU memory:

    ```sql
    SELECT latest(nv_cpu_memory_total_bytes) / 1e+6 AS 'memory (MB)' FROM Metric
    ```

## What's next?

To learn more about building NRQL queries and generating dashboards, check out these docs:

-   [Introduction to the query builder](https://docs.newrelic.com/docs/query-your-data/explore-query-data/query-builder/introduction-query-builder) to create basic and advanced queries.
-   [Introduction to dashboards](https://docs.newrelic.com/docs/query-your-data/explore-query-data/dashboards/introduction-dashboards) to customize your dashboard and carry out different actions.
-   [Manage your dashboard](https://docs.newrelic.com/docs/query-your-data/explore-query-data/dashboards/manage-your-dashboard) to adjust kkukkyour dashboards display mode, or to add more content to your dashboard.