--- title: Gateway data schema differences source: https://docs.newrelic.com/docs/new-relic-control/pipeline-control/gateway/data-schema-differences --- Gateway processors operate on telemetry data before it reaches New Relic, which means some attributes available in NRDB are not yet available at the gateway. Understanding these differences is critical when writing OTTL expressions for filter, transform, and sampling processors. ## Why schemas differ ### Data flow and enrichment When telemetry data flows through New Relic's platform: 1. **Gateway processing** - Your gateway receives raw telemetry from agents and other sources 2. **Enrichment** - New Relic adds attributes (like `entity.guid`, `appName`) and renames some existing attributes 3. **Cloud rules processing** - NRQL-based cloud rules operate on enriched data 4. **Storage** - Data is stored in NRDB with all enrichments applied ### Impact on gateway processors **Gateway processors see pre-enrichment data**, which means: - Some attributes don't exist yet (like `entity.guid`, `appName`, `entityGuid`) - Attribute names may differ from what you see in NRDB - Filter and transform logic must account for this reduced attribute set **Cloud rules see post-enrichment data**, which means: - All enriched attributes are available - NRQL queries can reference attributes that don't exist at the gateway ## Data sources The gateway receives telemetry from: - [New Relic APM agents](https://docs.newrelic.com/docs/new-relic-control/pipeline-control/gateway/modify-agent-configuration) (multiple languages supported) - New Relic Infrastructure agent - OpenTelemetry collectors - New Relic APIs (Events API, Logs API, Traces API, Metrics API) - Other OTLP-compatible sources > #### ⚠️ IMPORTANT > > Refer to [agent configuration documentation](https://docs.newrelic.com/docs/new-relic-control/pipeline-control/gateway/modify-agent-configuration) to verify which agents and versions are supported for gateway deployment. All data arrives as complex, multi-nested JSON with numerous attributes. ## Writing OTTL expressions for gateway processors ### Attribute availability When writing OTTL filter conditions or transform statements: **Available attributes:** - Core telemetry attributes sent by agents/collectors - Attributes your instrumentation adds directly - Standard OTLP attributes (like `span_id`, `trace_id`, `severity.number`) **Unavailable attributes (added during enrichment):** - `entity.guid`, `entityGuid` - `appId`, `appName` - `host` (in most cases) - `realAgentId` - Various NR-specific metadata attributes See the [attribute reference table](#attribute-reference-by-data-type) below for complete details. ### Best practices **Test with actual data:** Use your gateway's monitoring data to verify which attributes exist in your telemetry before writing complex filters. **Use available attributes:** ```yaml # ✓ Works - span_id exists in raw telemetry filter/Spans: config: spans: - 'span_id.string == "abc123"' # ✗ May not work - entity.guid added during enrichment filter/Spans: config: spans: - 'attributes["entity.guid"] == "xyz789"' ``` **Consider cloud rules for enriched attributes:** If your filtering logic requires enriched attributes (like `appName` or `entity.guid`), use cloud rules instead of gateway processors. **Check the reference table:** Before using an attribute in a filter or transform, verify it's not listed as "unavailable at gateway" in the table below. ## Attribute reference by data type The following table shows which attributes are unavailable at the gateway level for each telemetry data type. If you need to filter or transform based on these attributes, consider using cloud rules instead. | Data type | Attributes unavailable at gateway | Example filter expression (OTTL) | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | Transaction (APM) | `appId`, `appName`, `containerId`, `entity.guid`, `entityGuid`, `host`, `realAgentId`, `transactionSubType`, `transactionType` | `attributes["guid"] == "c2906c2e8b9f11ff"` | | Custom Events | `appId`, `appName`, `containerId`, `entityGuid`, `host`, `realAgentId` | `attributes["myFloat"] == 0.603` | | ErrorTrace | `aggregateFacet`, `appId`, `appName`, `applicationIds`, `count`, `entity.guid`, `entityGuid`, `error.class`, `message`, `path`, `exceptionClass`, `fingerprint`, `id`, `message`, `realAgentId`, `storageId`, `timestamp`, `transactionName`, `transactionUiName` | `attributes["traceId"] == "b366efe772fa6d1c8e0852558026c40e"` | | TransactionError | `aggregateFacet`, `appId`, `appName`, `containerId`, `entity.guid`, `entityGuid`, `host`, `realAgentId`, `transactionUiName` | `attributes["error.message"] == "my expected error message"` | | Log | `entity.guids`, `messageId`, `newrelic.logPattern`, `newrelic.logs.batchIndex`, `newrelic.source` | `span_id == "8b583de97341d094"` | | Metric (timeslice) | `appId`, `appName`, `entity.guid`, `entityGuid`, `language`, `metricName`, `metricTimesliceName`, `newrelic.timeslice.value`, `scope`, `timestamp` | Use dimensional metrics or cloud rules | | Span (distributed tracing) | `appId`, `appName`, `containerId`, `duration.ms`, `entity.guid`, `entity.name`, `entityGuid`, `host`, `id`, `process.id`, `realAgentId`, `trace.id` | `name == "WebTransaction/Go/GET /log"` | | SqlTrace | `applicationIds`, `callCount`, `databaseMetricName`, `entity.guid`, `id`, `maxCallTime`, `minCallTime`, `path`, `realAgentId`, `sql`, `sqlId`, `storageId`, `timestamp`, `totalCallTime`, `uri` | `attributes["uri"] == "Custom/Simple/sqlTransaction"` | | TransactionTrace | `storageId`, `uri`, `path`, `agentRunId`, `applicationIds`, `duration`, `entity.guid`, `guid`, `id`, `protocolVersion`, `realAgentId`, `timestamp` | Use attributes available in raw trace data | | Metric (gauge) | `newrelic.source` (value: `metricAPI`), `metricName`: `{type, count, latest, max, min, sum}` | `name == "redis_aof_rewrite_in_progress" and value < 100` | | Metric (summary) | `newrelic.source` (value: `metricAPI`), `metricName`: `{type, count, max, min, sum}` | `attributes["scrapedTargetKind"] == "user_provided"` | | Metric (count) | `newrelic.source` (value: `metricAPI`), `metricName`: `{type, count}` | `attributes["instrumentation.name"] == "nri-prometheus"` | | SystemSample (Infrastructure) | None | `attributes["entityKey"] == "vagrant"` | | StorageSample (Infrastructure) | `entityAndMountPoint` | `attributes["inodesUsed"] == 161604` | | NetworkSample (Infrastructure) | `entityAndInterface` | `attributes["entityKey"] == "vagrant"` | | ProcessSample (Infrastructure) | `entityAndPid` | `attributes["entityKey"] == "vagrant"` | | ContainerSample (Infrastructure) | `entityGuid`, `entityType`, `entityId` | `attributes["agentName"] == "ContainerSampleAgent"` | ## Common scenarios ### Filtering by entity **Problem:** You want to filter spans by entity, but `entity.guid` doesn't exist at the gateway. **Solution:** Use service name or other identifying attributes that exist in the raw telemetry: ```yaml filter/Spans: config: spans: - 'attributes["service.name"] == "my-service"' ``` ### Filtering by application name **Problem:** APM transactions don't have `appName` at the gateway. **Solution:** Use attributes your agent sets directly, or apply filtering after enrichment with cloud rules. ### Adding entity information **Problem:** You want to add entity context to telemetry at the gateway. **Solution:** You can't access `entity.guid` at the gateway, but you can add your own identifying metadata: ```yaml transform/Logs: config: log_statements: - set(attributes["deployment"], "production-us-east") - set(attributes["cluster"], "k8s-prod-01") ``` ## Troubleshooting ### Filter not matching expected data If your filter processor isn't matching data you expect: 1. **Check attribute availability** - Verify the attribute exists at the gateway (not just in NRDB) 2. **Inspect actual telemetry** - Use gateway monitoring to see what attributes are actually present 3. **Test attribute access** - Try a simple filter on the attribute to see if it exists: ```yaml filter/Test: config: logs: - 'attributes["entity.guid"] != ""' # Will match nothing if attribute doesn't exist ``` ### Transform not setting expected values If attributes aren't being added or modified: 1. **Verify attribute names** - Pre-enrichment attribute names may differ from NRDB 2. **Check data type** - Ensure you're accessing attributes correctly (for example, `attributes["key"]` vs direct field access) 3. **Review processor order** - Ensure transforms run before filters that depend on them ## Next steps - [Filter processor reference](https://docs.newrelic.com/docs/new-relic-control/pipeline-control/gateway/filter-processor) - Learn OTTL filter syntax - [Transform processor reference](https://docs.newrelic.com/docs/new-relic-control/pipeline-control/gateway/transform-processor) - Learn OTTL transform statements - [Cloud rules documentation](https://docs.newrelic.com/docs/new-relic-control/pipeline-control/cloud-rules/create-pipeline-rules) - Use NRQL on enriched data