---
title: Transform processor
source: https://docs.newrelic.com/docs/new-relic-control/pipeline-control/gateway/transform-processor
---

The transform processor modifies, enriches, or parses telemetry data using OTTL (OpenTelemetry Transformation Language). Use it to add context, normalize schemas, parse unstructured data, or obfuscate sensitive information before data leaves your network.

## When to use transform processor

Use the transform processor when you need to:

-   **Enrich telemetry with organizational metadata**: Add environment, region, team, or cost center tags
-   **Parse unstructured log messages**: Extract structured attributes using regex, Grok patterns, or JSON parsing
-   **Normalize attribute names and value schemas**: Standardize different naming conventions across services or agents (`level` → `severity.text`, `env` → `environment`)
-   **Hash or redact sensitive data**: Remove PII, credentials, or other sensitive information before it leaves your network
-   **Extract values from strings**: Pull HTTP status codes, durations, or other data from log messages
-   **Aggregate or scale metrics**: Modify metric values or combine multiple metrics

## OTTL contexts

OTTL operates in different contexts depending on the telemetry type:

-   **Logs**: `log` context - access log body, attributes, severity
-   **Traces**: `trace` context - access span attributes, duration, status
-   **Metrics**: `metric` and `datapoint` contexts - access metric name, value, attributes

## Configuration

Add a transform processor to your pipeline:

```yaml
      transform/Logs:
        description: Transform and process logs
        output:
          - nrexporter/newrelic
        config:
          rules:
            - name: add new field to attribute
              description: for otlp-test-service application add newrelic source type field
              conditions:
                - resource.attributes["service.name"] == "otlp-java-test-service"
              statements:
                - set(resource.attributes["source.type"],"otlp")
      transform/Metrics:
        description: Transform and process metrics
        output:
          - nrexporter/newrelic
        config:
          rules:
            - name: adding a new attributes
              description: adding a new field into a attributes
              conditions:
                - resource.attributes["service.name"] == "payments-api"
              statements:
                - set(resource.attributes["application.name"], "compute-application")
      transform/Traces:
        description: Transform and process traces
        output:
          - nrexporter/newrelic
        config:
          rules:
            - name: remove the attribute
              description: remove the attribute when service name is payment-service
              conditions:
                - resource.attributes["service.name"] == "payment-service"
              statements:
                - delete_key(resource.attributes, "service.version")
      nrexporter/newrelic:
        description: Export to New Relic
```

**Config fields**:

-   `rules`: An array of transformation rules.

    -   `name`: A unique identifier for the rule.

    -   `description`: A human-readable explanation of what the transformation does.

    -   `conditions`: A list of OTTL boolean expressions.

    -   `statements`: A list of OTTL actions (such as `set` or `delete_key`) to perform on the telemetry data.

## Key OTTL functions

### set()

Sets an attribute value.

```yaml
- set(attributes["environment"], "production")
- set(attributes["team"], "platform")
- set(severity.text, "ERROR") where severity.number >= 17
```

### delete_key()

Removes an attribute.

```yaml
- delete_key(attributes, "internal_debug_info")
- delete_key(attributes, "temp_field")
```

### replace_pattern()

Replaces text matching a regex pattern.

```yaml
  # Redact email addresses
- replace_pattern(attributes["user_email"], "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}", "[REDACTED_EMAIL]")

  # Mask passwords
- replace_pattern(attributes["password"], ".+", "password=***REDACTED***")

  # Obfuscate all non-whitespace (extreme)
- replace_pattern(body, "[^\\s]*(\\s?)", "****")
```

### Hash()

Hashes a value for pseudonymization.

```yaml
- set(attributes["user_id_hash"], Hash(attributes["user_id"]))
- delete_key(attributes, "user_id")
```

### ParseJSON()

Extracts attributes from JSON strings.

```yaml
  # Parse JSON body into attributes
- merge_maps(attributes, ParseJSON(body), "upsert") where IsString(body)
```

### ExtractGrokPatterns()

Parses structured data using Grok patterns.

```yaml
  # Parse JSON log format
- ExtractGrokPatterns(body, "\\{\"timestamp\":\\s*\"%{TIMESTAMP_ISO8601:extracted_timestamp}\",\\s*\"level\":\\s*\"%{WORD:extracted_level}\",\\s*\"message\":\\s*\"Elapsed time:\\s*%{NUMBER:elapsed_time}ms\"\\}")

  # Parse custom format with custom pattern
- ExtractGrokPatterns(attributes["custom_field"], "%{USERNAME:user.name}:%{PASSWORD:user.password}", true, ["PASSWORD=%{GREEDYDATA}"])
```

### flatten()

Flattens nested map attributes.

```yaml
  # Flatten nested map to top-level attributes
- flatten(attributes["map.attribute"])
```

### limit()

Limits number of attributes, keeping specified priority keys.

```yaml
  # Keep only 3 attributes, prioritizing "array.attribute"
- limit(attributes, 3, ["array.attribute"])
```

## Complete examples

### Example 1: Add environment metadata

```yaml
transform/Logs:
  description: "Enrich logs with environment context"
  config:
    rules:
      - name: enrich-with-environment-metadata
        description: Add environment, region, team, and cost center metadata to all logs
        statements:
          - set(attributes["environment"], "production")
          - set(attributes["region"], "us-east-1")
          - set(attributes["team"], "platform-engineering")
          - set(attributes["cost_center"], "eng-infra")
```

### Example 2: Normalize severity levels

Different services use different severity conventions. Standardize them:

```yaml
transform/Logs:
  description: "Normalize severity naming"
  config:
    rules:
      - name: convert-level-to-severity
        description: Convert custom level attribute to severity_text
        conditions:
          - attributes["level"] != nil
        statements:
          - set(severity_text, attributes["level"])
      
      - name: delete-level-attribute
        description: Remove the redundant level attribute after conversion
        statements:
          - delete_key(attributes, "level")
      
      - name: normalize-error-case
        description: Normalize error severity to uppercase ERROR
        conditions:
          - severity_text == "error"
        statements:
          - set(severity_text, "ERROR")
      
      - name: normalize-warning-case
        description: Normalize warning severity to uppercase WARN
        conditions:
          - severity_text == "warning"
        statements:
          - set(severity_text, "WARN")
      
      - name: normalize-info-case
        description: Normalize info severity to uppercase INFO
        conditions:
          - severity_text == "info"
        statements:
          - set(severity_text, "INFO")
```

### Example 3: Parse JSON log bodies

Extract structured attributes from JSON-formatted log messages:

```yaml
transform/Logs:
  description: "Parse JSON logs into attributes"
  config:
    rules:
      - name: parse-json-body-to-attributes
        description: Parse JSON log body and merge into attributes
        conditions:
          - IsString(body)
        statements:
          - merge_maps(attributes, ParseJSON(body), "upsert")
```

**Before**: `body = '{"timestamp": "2025-03-01T12:12:14Z", "level":"INFO", "message":"Elapsed time: 10ms"}'`

**After**: Attributes extracted: `timestamp`, `level`, `message`

### Example 4: Extract HTTP status codes

Pull status codes from log messages:

```yaml
transform/Logs:
  description: "Extract HTTP status from message"
  config:
    rules:
      - name: extract-http-status-code
        description: Extract HTTP status code from log body using regex pattern
        statements:
          - ExtractPatterns(body, "status=(\\d+)")
          - set(attributes["http.status_code"], body)
```

### Example 5: Redact PII

Remove sensitive information before data leaves your network:

```yaml
transform/Logs:
  description: "Redact PII for compliance"
  config:
    rules:
      - name: redact-email-addresses
        description: Redact email addresses from user_email attribute
        conditions:
          - attributes["user_email"] != nil
        statements:
          - replace_pattern(attributes["user_email"], "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}", "[REDACTED_EMAIL]")
      
      - name: mask-passwords
        description: Mask password attribute values
        conditions:
          - attributes["password"] != nil
        statements:
          - replace_pattern(attributes["password"], ".+", "***REDACTED***")
      
      - name: hash-user-ids
        description: Hash user IDs and remove original value
        conditions:
          - attributes["user_id"] != nil
        statements:
          - set(attributes["user_id_hash"], SHA256(attributes["user_id"]))
          - delete_key(attributes, "user_id")
      
      - name: mask-credit-cards-in-body
        description: Mask credit card numbers in log body
        statements:
          - replace_pattern(body, "\\d{4}[\\s-]?\\d{4}[\\s-]?\\d{4}[\\s-]?\\d{4}", "****-****-****-****")
```

### Example 6: Parse NGINX access logs

Extract structured fields from NGINX combined log format:

```yaml
transform/Logs:
  description: "Parse and enrich NGINX access logs"
  config:
    rules:
      - name: extract-nginx-fields
        description: Parse NGINX access log format into structured attributes
        statements:
          - ExtractGrokPatterns(body, "%{IPORHOST:client.ip} - %{USER:client.user} \\[%{HTTPDATE:timestamp}\\] \"%{WORD:http.method} %{URIPATHPARAM:http.path} HTTP/%{NUMBER:http.version}\" %{NUMBER:http.status_code} %{NUMBER:http.response_size}")
      
      - name: set-severity-for-server-errors
        description: Set severity to ERROR for 5xx server errors
        conditions:
          - attributes["http.status_code"] >= "500"
        statements:
          - set(severity_text, "ERROR")
      
      - name: set-severity-for-client-errors
        description: Set severity to WARN for 4xx client errors
        conditions:
          - attributes["http.status_code"] >= "400"
          - attributes["http.status_code"] < "500"
        statements:
          - set(severity_text, "WARN")
      
      - name: set-severity-for-success
        description: Set severity to INFO for successful requests
        conditions:
          - attributes["http.status_code"] >= "200"
          - attributes["http.status_code"] < "400"
        statements:
          - set(severity_text, "INFO")
```

### Example 7: Flatten nested attributes

Convert nested structures to flat attributes:

```yaml
transform/Logs:
  description: "Flatten nested map attributes"
  config:
    rules:
      - name: flatten-kubernetes-attributes
        description: Flatten nested kubernetes attributes into dot notation
        conditions:
          - attributes["kubernetes"] != nil
        statements:
          - flatten(attributes["kubernetes"])
      
      - name: flatten-cloud-provider-attributes
        description: Flatten nested cloud provider attributes into dot notation
        conditions:
          - attributes["cloud.provider"] != nil
        statements:
          - flatten(attributes["cloud.provider"])
```

**Before**: `attributes["kubernetes"] = {"pod": {"name": "my-app-123", "uid": "abc-xyz"},"namespace": {"name": "production"}}`

**After**: Attributes flattened: `kubernetes.pod.name`, `kubernetes.pod.uid`, `kubernetes.namespace.name`

### Example 8: Conditional transformations

Apply transformations only when conditions are met:

```yaml
transform/Logs:
  description: "Conditional enrichment"
  config:
    rules:
      - name: tag-critical-services
        description: Add business criticality tag for checkout and payment services
        conditions:
          - resource.attributes["service.name"] == "checkout" or resource.attributes["service.name"] == "payment"
        statements:
          - set(attributes["business_criticality"], "HIGH")
      
      - name: normalize-production-environment
        description: Normalize production environment names to standard format
        conditions:
          - attributes["env"] == "prod" or attributes["environment"] == "prd"
        statements:
          - set(attributes["deployment.environment"], "production")
      
      - name: normalize-staging-environment
        description: Normalize staging environment names to standard format
        conditions:
          - attributes["env"] == "stg" or attributes["environment"] == "stage"
        statements:
          - set(attributes["deployment.environment"], "staging")
      
      - name: cleanup-legacy-env-fields
        description: Remove old environment attribute fields after normalization
        statements:
          - delete_key(attributes, "env")
          - delete_key(attributes, "environment")
```

### Example 9: Data type conversion

Convert attributes to different types:

```yaml
transform/Logs:
  description: "Convert data types"
  config:
    rules:
      - name: convert-error-flag-to-boolean
        description: Convert string error_flag to boolean is_error attribute
        conditions:
          - attributes["error_flag"] != nil
        statements:
          - set(attributes["is_error"], Bool(attributes["error_flag"]))
      
      - name: set-success-boolean
        description: Set success attribute to boolean true
        statements:
          - set(attributes["success"], Bool("true"))
      
      - name: convert-retry-count-to-int
        description: Convert retry_count_string to integer retry_count
        conditions:
          - attributes["retry_count_string"] != nil
        statements:
          - set(attributes["retry_count"], Int(attributes["retry_count_string"]))
```

### Example 10: Limit cardinality

Reduce attribute cardinality to manage costs:

```yaml
transform/Logs:
  description: "Limit high-cardinality attributes"
  config:
    rules:
      - name: limit-attribute-cardinality
        description: Keep only the 5 most important attributes
        statements:
          - limit(attributes, 5, ["service.name", "environment", "severity_text"])
      
      - name: generalize-user-api-paths
        description: Replace user ID in path with wildcard to reduce cardinality
        conditions:
          - IsMatch(attributes["http.path"], "/api/users/\\d+")
        statements:
          - set(attributes["http.path"], "/api/users/*")
```

## OTTL function reference

For the complete list of OTTL functions, operators, and syntax:

-   [OTTL functions reference](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/ottlfuncs/README.md)
-   [Transform processor documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor)

## Next steps

-   Learn about [Filter processor](https://docs.newrelic.com/docs/new-relic-control/pipeline-control/gateway/filter-processor) for dropping unwanted data
-   See [Sampling processor](https://docs.newrelic.com/docs/new-relic-control/pipeline-control/gateway/sampling-processor) for volume reduction
-   Review [YAML configuration reference](https://docs.newrelic.com/docs/new-relic-control/pipeline-control/gateway/yaml-overview) for complete syntax