---
title: Error optimization: Improve your error tracking
source: https://docs.newrelic.com/docs/new-relic-solutions/observability-maturity/uptime-performance-reliability/error-optimization
---

This guide shows you how to optimize your errors so you can improve error rates, error detection, and customer experience. It's part of [our series on observability maturity](https://docs.newrelic.com/docs/new-relic-solutions/observability-maturity/introduction).

## Overview [#overview]

Error tracking is the practice of capturing application errors and error rates so you can address issues affecting your customers' experience of your software.

The aim of this guide is to enable a New Relic customer or team to:

-   Calibrate the way that errors are understood by New Relic, so that error related metrics reflect only errors that are meaningful to you
-   Lower the incidence of errors over time

## Desired outcome [#desired-outcome]

Improve customer experience and reliablity by reducing application error rates and mean time to resolution.

## Key performance indicators [#kpis]

### Business KPIs

Reducing errors that customers experience improves reliability. Organizations that improve reliability experience higher conversion rates (user journey completion rates) and higher user engagement. This brings organizations closer to meeting their revenue targets (commercial) or social impact goals (non-profit).

**Conversion rate**

Conversion rate is often used to indicate rate of purchases or ad clickthroughs.  In this case, the conversion rate is used to measure completed user journeys. Completed user journeys include things like raising a ticket, submitting a form, and watching a video, as well as following an ad to a site or making a purchase online.

**Goal:** Increase the ratio of user journey completions to user sessions.

**Best practices**

Errors occur in both frontend and backend applications but are typically measured on the frontend. [Funnel queries](https://docs.newrelic.com/docs/query-your-data/nrql-new-relic-query-language/nrql-query-tutorials/funnels-evaluate-data-series-related-events) are popular for measuring conversions, but you can do it even more simply by counting the number of conversions over the total number of sessions for a given time period.

If you provide an API service and conversion rates apply to your business, you can measure them by comparing the number of calls to the final service that gets call to the number of calls to the first service. For example:

````sql
FROM Transaction SELECT 
   (FROM Transaction SELECT count(*) WHERE request.uri = '/api/v1/lastStep') /
   (FROM Transaction SELECT count(*) WHERE request.uri = '/api/v1/firstStep') AS conversionRate
```

For more examples of improving conversions, check out the [Bottom-of-the-funnel analysis guide](/docs/new-relic-solutions/observability-maturity/customer-experience/bottom-funnel-analysis-customer-journey-guide), which explains how to improve conversion rates by starting with the final steps of the user journey.  


````

**Page view count**

Measure increased or decreased engagement by counting page views.

**Goal:** Increase page views

**Best practices**

Errors occur in both frontend and backend applications but are typically measured on the frontend. Measure the impact of improving errors on user engagement by tracking page views for the frontend applications related to where you're making improvements.

Your NRQL query will look something like the following:

````sql
FROM PageView SELECT count(*) WHERE appName IN ('CustomerApp1', 'CustomerApp2')
```

If you provide an API as a service and it's relevant to your business, you can track the equivalent of page views by tracking transaction counts:

```sql
FROM Transaction SELECT count(*) WHERE appName = 'apiService'
```

````

**User count**

Measure increased or decreased engagement by counting page views.

**Goal:** Increase the number of users accessing your site

**Best practices**

Errors occur in both frontend and backend applications but are typically measured on the frontend. Measure the impact of improving errors on user engagement by tracking the number of users accessing your site over a given time period.  If you have not added custom instrumentation to track users you can track sessions instead.

Your NRQL query will look something like the following:

````sql
FROM PageView SELECT uniqueCount(userId) WHERE appName IN ('CustomerApp1', 'CustomerApp2')
```

OR

```sql
FROM PageView SELECT uniqueCount(session) WHERE appName IN ('CustomerApp1', 'CustomerApp2')
```

If you provide an API as a service, capture customer IDs, and it's relevant to your business, you can track the equivalent of users by tracking customers in transactions:

```sql
FROM Transaction uniqueCount(customerId) WHERE appName = 'apiService'
```

````

The business KPIs above work on the assumption that you support your users by providing a frontend application. If you support your customers via API, it may be possible to fit the above KPIs to the Transaction entity type. Some organizations that provide APIs as a service use operational KPIs like the ones below to promote the quality of their offering.

### Operational KPIs

**Error rate**

The error rate is the ratio of errors to requests.

An error can be a 300+ HTTP response code, an unhandled exception, a mobile crash, or an event someone on your team configured to be an error.

**Goal:** Reduce the error rate across the applications you manage.

**Best practices**

This is the main KPI you will use to track your progress against improving error tracking. Steps to improve error rates include filtering out inconsequential errors as well as resolving impactful errors.

**Mean time to close errors**

Mean-time-to-close measures the time from when an alert notifies you of an issue until the issue is closed, just after it has been resolved.  This KPI is tracked as part of [Alert quality management](https://docs.newrelic.com/docs/new-relic-solutions/observability-maturity/uptime-performance-reliability/alert-quality-management-guide#key-perf-indicators)

**Goal:** Reduce the mean-time-to-close-issues by reducing the error rate.

**Best practices**

Build a strong error practice so that when an issue occurs, you will be able to more quickly detect:

-   Whether or not the issue is related to a spike in errors
-   Which error is responsible for the issue, if the issue is caused by an error

    The [Alert quality management guide](https://docs.newrelic.com/docs/new-relic-solutions/observability-maturity/uptime-performance-reliability/alert-quality-management-guide) shows you how to track this KPI.

## Prerequisites [#prerequisites]

### Required installation and configuration

-   Make sure that our APM, browser monitoring, mobile monitoring, serverless monitoring, or OpenTelemetry solutions are getting your errors.
-   Up-to-date [source maps](https://docs.newrelic.com/docs/browser/new-relic-browser/browser-pro-features/upload-source-maps-api) for web applications
-   Up-to-date [symbolification](https://docs.newrelic.com/docs/mobile-monitoring/new-relic-mobile-ios/configuration/ios-tvos-crash-reporting) for mobile applications

## Establish the current state [#establish-current-state]

-   [Create a workload for your applications](#create-workload)
-   [Create service levels for your workload](#create-service-level)

### Create a workload for your applications [#create-workload]

Define the list of applications and services for which you are trying to optimize errors. The team conducting the error optimization process should have full responsibility for and control of these apps and services. Once you have decided, set up a [workload](https://docs.newrelic.com/docs/new-relic-solutions/new-relic-one/workloads/workloads-isolate-resolve-alert-events-faster) for these entities.

Workloads are a group of entities (applications, instances, etc.) for which a specific team is responsible. They allow you to look at data for only the entities you can do something about. We'll be basing most of our work going forward around the workload you set up here.

It only takes a few minutes to set up a workload. See [the workload instructions](https://docs.newrelic.com/docs/new-relic-solutions/new-relic-one/workloads/workloads-isolate-resolve-alert-events-faster).

If you're already familiar with workloads and prefer to divide your applications and services into multiple workloads, you can. Just follow the steps below for each workload.

### Create service levels for your workload [#create-service-level]

[Service levels](https://docs.newrelic.com/docs/service-level-management/intro-slm) allow you to easily configure and view [Service level objectives (SLOs)](https://sre.google/sre-book/service-level-objectives) for a given group of entities. Using service levels is one way you can monitor and communicate the success of your error management project.

From your workload, navigate to the **Service levels** tab. Create a service level that measures error rates for each entity in the workload. This is configured in Step 2 in the service level UI. For each service level, use the WHERE clauses to filter out good requests or errors that shouldn't be factored in.

![Add a service level to your workload](https://docs.newrelic.com/images/oma-eto_screenshot-full_add-service-level.webp "Add a service level to your workload")

**Create an error rate service level for each application service**

For step 2 of service creation, choose the **Success** SLI.

![Set the service level type](https://docs.newrelic.com/images/oma-eto_screenshot-full_sl-type-apm.webp "Set the service level type")

Add a tag to the service level so you can see all the error rates in one view. You can use the default tag `category:success`

![Create a service level tag](https://docs.newrelic.com/images/oma-eto_screenshot-crop_sl-tag-apm.webp "Create a service level tag")

**Create an error rate service level for each browser application**

-   For step 2 of service creation, choose the **Success** SLI.

    ![Set the service level type](https://docs.newrelic.com/images/oma-eto_screenshot-crop_sl-type-browser.webp "Set the service level type")

-   For Step 4, add a tag to the service level. You can use the default tag `category:success`

    ![Create a service level tag](https://docs.newrelic.com/images/oma-eto_screenshot-crop_sl-tag-browser.webp "Create a service level tag")

**Create an error rate service level for each mobile application**

Create a service level for mobile crashes

-   For Step 2, choose `MobileSession` as the source of valid events. Choose `MobileCrash` as the source of bad responses.
    ![Set the service level type](https://docs.newrelic.com/images/oma-eto_screenshot-full_sl-type-mobile-crash.webp "Set the service level type")

-   For Step 4, add a tag to the service level. You can use the default tag `category:success`.

    Create a service level for mobile request errors

-   For Step 2, choose `MobileRequest` as the source of valid events. Choose `MobileRequestError` as the source of bad responses.
    ![Set the service level type](https://docs.newrelic.com/images/oma-eto_screenshot-full_sl-type-mobile-request-error.webp "Set the service level type")

-   For Step 4, add a tag to the service level. You can use the default tag `category:success`.

**Create an error rate service level for each serverless application**

Create an error rate service level for AWS Lambda function integrated with our [serverless monitoring](https://docs.newrelic.com/docs/serverless-function-monitoring/aws-lambda-monitoring/get-started/monitoring-aws-lambda-serverless-monitoring):

-   For Step 1, select `Lambda function` as the entity type

-   For Step 2, select `AWSLambdaInvocation` for valid events and `AwsLambdaInvocationError` for bad responses

    ![Set the service level type](https://docs.newrelic.com/images/oma-eto_screenshot-full_sl-type-serverless.webp "Set the service level type")

-   For Step 4, add a tag to the service level. You can use the default tag `category:success`

    ![Create a service level tag](https://docs.newrelic.com/images/oma-eto_screenshot-crop_sl-tag-browser.webp "Create a service level tag")

    Currently, service levels only support error rates for AWS Lambda functions using [New Relic serverless monitoring for AWS Lambda](https://docs.newrelic.com/docs/serverless-function-monitoring/aws-lambda-monitoring/get-started/monitoring-aws-lambda-serverless-monitoring). You can capture the error rate outside of service levels using the following query:

    ```sql
    SELECT sum(provider.errors.Sum) / sum(provider.invocations.Sum) * 100 
    FROM ServerlessSample
    ```

**Create an error rate service level for each open telemetry application**

-   For Step 1, select `Service - OpenTelemetry`.
-   For Step 2's valid events, use the `Span` entity type for good event types. Add the following to the WHERE clause:
    `(span.kind LIKE 'server' OR span.kind LIKE 'consumer' OR kind LIKE 'server' OR kind LIKE 'consumer')`
-   For Step 2's invalid events, use the `Span` entity type and `Repeat WHERE clause` option. Add the following to the WHERE clause to detect bad responses:
    `otel.status_code = 'ERROR'`
-   For Step 4, add a tag to the service level. You can use the default tag `category:success`.

### Use Service levels to track your progress against current error rates

Using the process documented above, you've created Service levels based on your current error rates.

-   The SLO column shows you the targeted error rate you selected using a baseline.
-   The Operational view mode shows you recent performance against your target.
-   The Period over period view mode shows you performance against your target over a longer period of time.
-   You can update error rate targets as improvements are made.

![Group service levels by category](https://docs.newrelic.com/images/oma-eto_screenshot-full_sl-filter.webp "Group service levels by category")

## Improvement process [#improvement-process]

-   [Identify inconsequential errors](#create-workload)
-   [Remove inconsequential errors from your error rate](#create-service-level)
-   [Set up error rate alerts](#error-rate-alerts)
-   [Establish an error-hero roster](#error-hero)
-   [Triage errors using errors inbox](#errors-inbox)
-   [Link errors to JIRA](#jira)
-   [Link errors to Slack](#slack)
-   [Use CodeStream](#codestream)

### Identify inconsequential errors [#inconsequential-errors]

Explore your errors however you are most comfortable. You can do this using:

-   Out-of-the-box views for APM, mobile monitoring, JavaScript errors, serverless monitoring, and OpenTelemetry
-   Errors inbox filtered for your workload
-   NRQL data types such as `TransactionError`, `JavaScriptError`, `MobileRequestError`, `AwsLambdaInvocationError`, `Span`

### Remove inconsequential errors from your error rate [#remove-inconsequential-errors]

You can remove errors that don't matter in one of two ways:

-   Stop them from being ingested using [configuration](https://docs.newrelic.com/docs/apm/agents/manage-apm-agents/agent-data/manage-errors-apm-collect-ignore-or-mark-expected/#ignore) (APM only) or using [drop rules](https://docs.newrelic.com/docs/data-apis/manage-data/drop-data-using-nerdgraph). This approach only works for errors that you are certain don't need capturing. The added advantage of this approach is reduction in ingest for noisy errors.
-   Use NRQL to filter the errors out of service level calculations. Do this by adding onto the WHERE clause filter of bad responses. Be sure to re-base the service level if this improves the error rate signficantly. Doing so will improve error alerting accuracy.

### Set up error rate alerts [#error-rate-alerts]

Review each of the service levels set up in [Create service levels for your workload](#create-service-level) and create an alert to notify the team when error rate increase beyond an acceptable rate.

### Establish an error-hero roster[#error-hero]

Alerting will let you know if you are meeting the current levels of error performance but won't help you improve them. To improve customer sentiment, create a process where errors are reviewed daily by a member of your team. The error hero should:

-   Initially focus on errors that happen above the fold. For a daily review process this means focusing on errors that happened only in the last 24 hours.
-   Triage errors using [Errors inbox](https://docs.newrelic.com/docs/errors-inbox/errors-inbox)

### Triage errors using errors inbox [#errors-inbox]

Errors inbox is a single place to proactively detect, triage, and take action on all errors before they impact customers. Similar errors will be grouped together to avoid duplication of work and to allow you to prioritize errors by number of occurrences.

When accessing errors inbox, be sure to select your workload so that you're only seeing errors relevant to your team.

Set aside a regular time to go through errors inbox as a team. To begin with, daily or a few times a week makes sense as you will have a lot of error groups to go through. Later, weekly or bi-weekly may be more appropriate.

Go through the errors one by one by clicking into the error details screen when necessary to get more information such as traces, logs, etc. This will either point to a cause of the error or provide a starting point for further investigation.

After brief discussion you may be in a position to mark the error group as one of the following:

-   Ignore: For use when the error is not problematic. This will hide the error group from the inbox view from that point on.
-   Resolved: For use when the error was the result of a known issue that has now been fixed. This will remove the error group from the list unless it recurs. If it does recur, the fix implemented previously should be re-thought.

Note: Ignoring and/or resolving errors via errors inbox will not stop them from counting towards error rate metrics.

If neither of the above statuses is suitable, assign the error to the appropriate team member for further investigation and resolution. That team member can conduct further investigation in their own time, updating the error group notes with their progress and/or asking for help from other team members via the notes section.

At the next triaging meeting you can revisit these error groups to see if they can now be marked as resolved. As time goes by, you should start to see fewer new error groups appearing, and positive movement in your KPIs.

### Link errors to JIRA [#jira]

As you get into more edge-case or complex errors, you might find you need to ask for help from other teams. Linking errors inbox to Jira may help with this. [Connect your Errors inbox to Jira](https://docs.newrelic.com/docs/errors-inbox/error-external-services#jira) to enable you to easily create tickets connected to error groups. You can control the information sent to Jira via Jira templates.

### Link errors to Slack [#slack]

As the velocity of errors coming into errors inbox goes down, a regular team session may no longer be a good use of time. An alternative is to [link Errors inbox to Slack](https://docs.newrelic.com/docs/errors-inbox/error-external-services#slack) and either a) designate a rotating person to monitor the channel and resolve/ignore/assign error groups as they come in or b) allow the team to proactively respond to error groups.

### Use CodeStream [#codestream]

Many of your error groups are going to require code changes to resolve. [Connect CodeStream to your New Relic account](https://docs.newrelic.com/docs/codestream/start-here/what-is-codestream) to open the offending code directly in your IDE to investigate the code directly. You can also leave notes and comments on specific lines of code for developers to review and vice versa.

New Relic with CodeStream gives you more context on your error groups, such as being able to see version numbers or commit SHAs. Additionally, using errors inbox as a centralized place to identify, discuss, and rectify code issues allows you to respond to code problems efficiently and avoid duplicating work.

## Value realization [#value-realization]

Review error rates weekly as you progress through the practice. As error rates decrease, you should see faster mean time to resolution and increased customer satisfaction.