---
title: Filter processor
source: https://docs.newrelic.com/docs/new-relic-control/pipeline-control/gateway/filter-processor
---

The filter processor drops telemetry records or specific attributes based on OTTL (OpenTelemetry Transformation Language) boolean expressions. Use it to remove test data, debug logs, health checks, or any low-value telemetry before it leaves your network.

## When to use filter processor

Use the filter processor when you need to:

-   **Drop PII or test environment data**: Remove data that shouldn't leave your network
-   **Remove debug-level logs from production**: Filter by severity to reduce noise
-   **Filter out health check requests**: Drop repetitive, low-value monitoring traffic
-   **Drop metrics with specific prefixes or patterns**: Remove unnecessary metric streams
-   **Remove low-value telemetry based on attributes**: Filter by service name, environment, or custom tags

## How filter processor works

The filter processor evaluates OTTL boolean expressions against each telemetry record. **When a condition evaluates to `true`, the record is dropped.**

This is the opposite of many query languages where `WHERE status = 'ERROR'` means "keep errors." In filter processor, `status == 'ERROR'` means "drop errors."

## Configuration

Add a filter processor to your pipeline:

```yaml
      filter/Logs:
        description: Apply drop rules and data processing for logs
        config:
          error_mode: ignore
          rules:
            - name: drop the log records
              description: drop all records which has severity text INFO
              conditions:
                - log.severity_text == "INFO"
```

**Config fields**:

-   `rules`: Array of filtering rules evaluated in order.
    -   `name`: Rule identifier.
    -   `context`: The type of data to evaluate. Supported values: `log`, `span`, `span_event`, `metric`, `datapoint`.
    -   `conditions`: A list of OTTL boolean expressions.

**Multiple conditions**: When you provide multiple expressions in the array, they are evaluated with OR logic. If **any** condition is true, the record is dropped.

## OTTL boolean operators

### Comparison operators

-   `==` - Equal to
-   `!=` - Not equal to
-   `<` - Less than
-   `<=` - Less than or equal to
-   `>` - Greater than
-   `>=` - Greater than or equal to

### Logical operators

-   `and` - Both conditions must be true
-   `or` - Either condition must be true
-   `not` - Negates a condition

### Pattern matching

-   `IsMatch` - Regex pattern matching

```yaml
rules:
  - name: match-health-logs
    conditions:
      - 'IsMatch(body, ".*health.*") or IsMatch(attributes["http.url"], ".*\\/api\\/v1\\/health.*")'
```

## Complete examples

### Example 1: Drop test environment data

Remove all telemetry from test and development environments:

```yaml
config:
  rules:
    - name: drop-test-environment
      description: Drop logs from test environment
      conditions: 
        - 'resource.attributes["environment"] == "test"'
      context: log
    - name: drop-dev-environment
      description: Drop logs from dev environment
      conditions: 
        - 'resource.attributes["environment"] == "dev"'
      context: log
```

### Example 2: Drop debug logs in production

Keep only meaningful log levels in production:

```yaml
config:
  rules:
    - name: drop-debug-logs
      description: Drop all DEBUG severity logs
      conditions: 
        - 'severity_text == "DEBUG"'
      context: log
    - name: drop-low-severity-logs
      description: Drop INFO and below severity logs
      conditions: 
        - "severity_number < 9"
      context: log
```

**Severity number reference**:

-   TRACE = 1-4
-   DEBUG = 5-8
-   INFO = 9-12
-   WARN = 13-16
-   ERROR = 17-20
-   FATAL = 21-24

### Example 3: Drop health check spans

Remove health check traffic that adds no diagnostic value:

```yaml
config:
  rules:
    - name: drop-health-endpoint
      description: Drop spans from /health endpoint
      conditions: 
        - 'attributes["http.path"] == "/health"'
      context: span
    - name: drop-health-check-spans
      description: Drop spans named health_check
      conditions: 
        - 'name == "health_check"'
      context: span
```

### Example 4: Drop by service name

Filter out specific services or service patterns:

```yaml
config:
  rules:
    - name: drop-legacy-api
      description: Drop logs from legacy API v1 service
      conditions: 
        - 'resource.attributes["service.name"] == "legacy-api-v1"'
      context: log
    - name: drop-canary-services
      description: Drop logs from canary deployment services
      conditions: 
        - 'IsMatch(resource.attributes["service.name"], ".*-canary")'
      context: log
```

### Example 5: Drop metrics with specific prefixes

Remove unnecessary metric streams:

```yaml
config:
  rules:
    - name: drop-internal-metrics
      description: Drop metrics with internal prefix
      conditions: 
        - 'IsMatch(name, "^internal\\.")'
      context: metric
    - name: drop-debug-datapoints
      description: Drop datapoints marked as debug type
      conditions: 
        - 'attributes["metric.type"] == "debug"'
      context: datapoint
```

### Example 6: Combined conditions with AND

Drop only when multiple conditions are true:

```yaml
config:
  rules:
    - name: drop-debug-logs-from-test
      description: Drop DEBUG logs from background-worker service in test environment
      conditions:
        - 'severity_text == "DEBUG"'
        - 'resource.attributes["service.name"] == "background-worker"'
        - 'resource.attributes["environment"] == "test"'
      context: log
```

### Example 7: Keep errors, drop everything else

Invert the logic to keep only valuable data:

```yaml
config:
  rules:
    - name: drop-non-error-logs
      description: Drop everything below ERROR severity level
      conditions: 
        - "severity_number < 17"
      context: log
```

Or use NOT logic:

```yaml
filter/Logs:
  description: "Drop non-errors"
  config:
    error_mode: ignore
    rules:
      - name: drop-non-error-logs
        description: Drop logs that are not ERROR or FATAL
        conditions:
          - 'not (severity_text == "ERROR" or severity_text == "FATAL")'
        context: log
```

### Example 8: Pattern matching in log body

Drop logs containing specific patterns:

```yaml
config:
  rules:
    - name: drop-health-check-logs
      description: Drop logs with health check in body
      conditions: 
        - 'IsMatch(body, ".*health check.*")'
      context: log
```

### Example 9: Drop high-volume, low-value spans

Remove spans that occur frequently but provide little value:

```yaml
config:
  rules:
    - name: drop-fast-cache-hits
      description: Drop cache hit operations faster than 1ms
      conditions:
        - 'attributes["db.operation"] == "get"'
        - 'end_time_unix_nano - start_time_unix_nano < 1000000'
        - 'attributes["cache.hit"] == true'
      context: span
```

### Example 10: Drop based on HTTP status

Filter successful requests, keep errors:

```yaml
config:
  rules:
    - name: drop-successful-requests
      description: Drop HTTP requests with status code less than 400
      conditions: 
        - 'attributes["http.status_code"] < 400'
      context: span
```

### Example 11: Multiple conditions with OR

Drop if any condition matches:

```yaml
config:
    rules:
      - name: drop-test-health-debug
        description: Drop logs from test environment, health checks, or debug severity
        conditions:
          - 'resource.attributes["environment"] == "test"'
          - 'IsMatch(body, ".*health.*")'
          - 'severity_text == "DEBUG"'
        context: log 
```

## Drop data vs drop attributes

The filter processor can drop entire records (as shown above) or drop specific attributes from records that are kept.

To drop attributes while keeping the record, you need to use the transform processor's `delete_key()` function, not the filter processor. The filter processor only drops entire records.

**Wrong approach** (this won't work):

```yaml
filter/Logs:
  config:
    rules:
      - name: wrong-attribute-drop
        description: Identify and drop logs containing specific sensitive attributes
        conditions:
          - 'delete attributes["sensitive_field"]'  # Filter conditions must be boolean, not actions
        context: log
```

**Correct approach** (use transform processor instead):

```yaml
transform/Logs:
  description: "Remove sensitive attribute"
  config:
    rules:
      - name: remove-sensitive-field
        description: "Redacts the 'sensitive_field' attribute from log records to ensure privacy compliance."
        statements:
          - 'delete_key(attributes, "sensitive_field")'
  output: 
    - nrexporter/newrelic
```

## Performance considerations

-   **Order matters**: Place filter processors early in your pipeline to drop unwanted data before expensive processing
-   **Combine conditions**: Use `and`/`or` logic in a single expression rather than chaining multiple filter processors
-   **Regex performance**: `IsMatch` is more expensive than exact equality checks. Use `==` when possible.

**Example of efficient ordering**:

```yaml
    steps:
      # ... receive steps ...
      # ... probabilistic sampler steps ...
      filter/Logs:
        description: Apply drop rules and data processing for logs
        output:
          - transform/Logs
        config:
          error_mode: ignore
          rules:
            - name: drop the log records
              description: drop all records which has severity text INFO
              conditions:
                - log.severity_text == "INFO"
              context: log
      filter/Metrics:
        description: Apply drop rules and data processing for metrics
        output:
          - transform/Metrics
        config:
          error_mode: ignore
          rules:
            - name: drop-internal-metrics
              description: drop internal metric
              conditions:
                - IsMatch(name, "^internal\\.")
              context: metric
            - name: drop-debug-datapoints
              description: drop-debug-datapoints
              conditions:
                - attributes["metric.type"] == "debug"
              context: datapoint
      filter/Traces:
        description: Apply drop rules and data processing for traces
        output:
          - transform/Traces
        config:
          error_mode: ignore
          rules:
            - name: drop-health-endpoint
              description: drop-health-endpoint
              conditions:
                - attributes["http.path"] == "/health"
              context: span
            - name: drop-debug-events
              description: drop-debug-events
              conditions:
                - name == "debug_event"
              context: span_event
        # ... transformer steps...
```

## OTTL boolean expression reference

For complete OTTL syntax and additional operators:

-   [OTTL boolean expressions](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md)
-   [Filter processor documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor)

## Next steps

-   Learn about [Transform processor](https://docs.newrelic.com/docs/new-relic-control/pipeline-control/gateway/transform-processor) for modifying data before filtering
-   See [Sampling processor](https://docs.newrelic.com/docs/new-relic-control/pipeline-control/gateway/sampling-processor) for probabilistic volume reduction
-   Review [YAML configuration reference](https://docs.newrelic.com/docs/new-relic-control/pipeline-control/gateway/yaml-overview) for complete syntax