--- title: NerdGraph tutorial: Stream your data to an AWS Kinesis Firehose, Azure Event Hub, or GCP Pub/Sub source: https://docs.newrelic.com/docs/apis/nerdgraph/examples/nerdgraph-streaming-export --- With the streaming export feature available through [Data Plus](https://docs.newrelic.com/docs/accounts/accounts-billing/new-relic-one-pricing-billing/data-ingest-billing/#data-plus), you can send your data to AWS Kinesis Firehose, Azure Event Hub, or GCP Pub/Sub by creating custom rules using [NRQL](https://docs.newrelic.com/docs/query-your-data/nrql-new-relic-query-language/get-started/introduction-nrql-new-relics-query-language) to specify which data should be exported. This guide explains how to create and update streaming rules using [NerdGraph](https://docs.newrelic.com/docs/apis/nerdgraph/get-started/introduction-new-relic-nerdgraph) and view the existing rules. You can use the [NerdGraph explorer](https://docs.newrelic.com/docs/apis/nerdgraph/get-started/nerdgraph-explorer) to make these calls. Additionally, you have the option to compress the data before exporting with the [Export Compression](#compression) feature. Here are some examples of how you can use the streaming export feature: - To populate a data lake - Enhance AI/ML training - Ensure long-term retention for compliance, legal, or security reasons You can enable or disable streaming export rules at any time. However, be aware that streaming export only processes currently ingested data. If you disable and later re-enable the feature, any data ingested while it was off will not be exported. To export past data, you should use the [Historical data export](https://docs.newrelic.com/docs/apis/nerdgraph/examples/nerdgraph-historical-data-export) feature. ## Requirements and limits [#requirements] Limits on streamed data: The amount of data you can stream per month is limited by your total [ingested data](https://docs.newrelic.com/docs/accounts/accounts-billing/new-relic-one-pricing-billing/data-ingest-billing/#usage-calculation) per month. If your streaming data amount exceeds your ingested data amount, we may suspend your access to and use of streaming export. Permissions-related requirements: - Pro or Enterprise edition with [Data Plus](https://docs.newrelic.com/docs/accounts/accounts-billing/new-relic-one-pricing-billing/data-ingest-billing/#data-plus) option - User type: [core user or full platform user](https://docs.newrelic.com/docs/accounts/accounts-billing/new-relic-one-user-management/user-type) - The [streaming data permission](https://docs.newrelic.com/docs/accounts/accounts-billing/new-relic-one-user-management/user-permissions#data-platform) You must have an AWS Kinesis Firehose, Azure Event Hub, or GCP Pub/Sub set up to receive New Relic data. If you haven't already done this, you can follow our steps below for [AWS](#firehose-setup), [Azure](#event-hub-setup) or [GCP Pub/Sub](#gcp-pubsub-setup). NRQL requirements: - Must be flat queries, with no aggregation. For example, `SELECT *` or `SELECT column1, column2` forms are supported. - Applicable for anything in the `WHERE` clause, except subqueries. - Query cannot have a `FACET` clause, `COMPARE WITH`, or `LOOKUP`. - Nested queries are not supported. - Supports [data types stored in NRDB](https://docs.newrelic.com/docs/data-apis/understand-data/new-relic-data-types/#timeslice-data), and not [metric timeslice data](https://docs.newrelic.com/docs/data-apis/understand-data/new-relic-data-types/#timeslice-data). ## Prerequisites - A [New Relic account](https://newrelic.com/signup), and with that account, you can access your API user key that you need to include with [queries and mutations](https://docs.newrelic.com/docs/apis/nerdgraph/get-started/introduction-new-relic-nerdgraph/). - User type and assigned roles can affect your NerdGraph permissions. For more details, see [Factors affecting access](https://docs.newrelic.com/docs/accounts/accounts-billing/account-structure/factors-affecting-access-features-data/#user-permissions). ## Set up an AWS Kinesis Firehose [#firehose-setup] To set up streaming data export to AWS, you must first set up Amazon Kinesis Firehose. Follow these steps: ### Create a Firehose for streaming export [#create-firehose] Create a dedicated Firehose to stream your New Relic data to: 1. Go to [Amazon Kinesis Data Firehose](https://aws.amazon.com/firehose/). 2. Create a delivery stream. 3. Name the stream. You'll use this name later when registering the rule. 4. Use **Direct PUT or other sources** and specify a destination compatible with New Relic's JSON event format (for example, S3, Redshift, or OpenSearch). ### Create IAM Firehose write access policy [#create-policy] 1. Go to the IAM console and sign in with your user. 2. In the left navigation, click **Policies**, and then click **Create policy**. 3. Select the Firehose service, and then select `PutRecord` and `PutRecordBatch`. 4. For `Resources`, select the delivery stream, add ARN, and select the region of your stream. 5. Enter your AWS account number, and then enter your desired delivery stream name in the name box. 6. Create the policy. ### Create IAM role for granting New Relic write access [#iam-role] To set up the IAM role: 1. Navigate to the IAM and click **Roles**. 2. Create a role for an AWS account, and then select **for another AWS account**. 3. Enter the New Relic export account ID: `888632727556`. 4. Select **Require external ID** and enter the [account ID](https://docs.newrelic.com/docs/accounts/accounts-billing/account-structure/account-id) of the New Relic account you want to export from. 5. Click **Permissions**, and then select the policy you created above. 6. Add a role name, which will be used during export registration, and provide a description. 7. Create the role. When you're done with these steps, you can set up your export rules using [NerdGraph](https://docs.newrelic.com/docs/apis/nerdgraph/get-started/introduction-new-relic-nerdgraph/). **Create a stream** This API allows you to create a streaming export rule to send data to AWS Kinesis Firehose. #### Input parameters | Parameter | Data Type | Is it Required? | Description | | ----------------------------------- | --------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `accountId` | Integer | Yes | Your New Relic [account ID](https://docs.newrelic.com/docs/accounts/accounts-billing/account-structure/account-id/). | | `ruleParameters.description` | String | No | A brief description of the rule. Use this to provide any additional information or context about the export rule. | | `ruleParameters.name` | String | Yes | The name of the export rule. | | `ruleParameters.nrql` | String | Yes | An NRQL to select the telemetry data to export. | | `ruleParameters.payloadCompression` | String | No | [payloadCompression](#compression) accepts two values: `DISABLED`: By default, payloads will not be compressed before being exported. `GZIP`: Select this to compress payloads before being exported. | | `awsParameters.awsAccountId` | ID | Yes | Specifies the [AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/console-account-alias.html) that owns the target Firehose delivery stream. For example: `10000000000`. | | `awsParameters.deliveryStreamName` | String | Yes | Specifies the [name of the Kinesis stream](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_CreateStream.html) used for data delivery. For example: `firehose-test-stream`. | | `awsParameters.region` | String | Yes | Specifies the [AWS region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) where the resources are deployed. For example: `us-east-1`. | | `awsParameters.role` | String | Yes | Specifies the [AWS IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) configured for New Relic. This will always be `firehose-role`. | #### Sample request ```graphql mutation { streamingExportCreateRule( accountId: YOUR_NR_ACCOUNT_ID, ruleParameters: { description: "ADD_RULE_DESCRIPTION", name: "PROVIDE_RULE_NAME", nrql: "SELECT * FROM NodeStatus", payloadCompression: DISABLED }, awsParameters: { awsAccountId: "YOUR_AWS_ACCOUNT_ID", deliveryStreamName: "FIREHOSE_STREAM_NAME", region: "SPECIFY_AWS_REGION", role: "firehose-role" } ) { id status } } ``` **Response**: You receive a response with the rule ID and status. The status will be `CREATION_IN_PROGRESS` initially, and it can take up to six minutes for the rule to complete creation. **Points to note**: Before the rule finishes registering, you can't initiate another mutation action (like `Enable`, `Disable`, or `Update`) because the rule is locked for the creation process. If you try another mutation action before the rule completes the registration process, you'll get a message like, "The export rule is currently being updated by another request, please wait and try again later." You can use `Delete` at any time. The creation can finish and change the status at any time within the roughly six minutes required for rule creation. The status will change to `ENABLED`, `DISABLED`, or `CREATION_FAILED`. See these details on the values: - `ENABLED` means the rule is created successfully and data has started to be streamed. - `CREATION_FAILED` means the rule failed on creation. This can happen for several reasons, but is often due to AWS policy or Azure SAS validation failing. - `DISABLED` means the rule is created but is not enabled yet due to reasons such as filter stream limit being reached, or failing on filter stream rule creation. If the status still remains as `CREATION_IN_PROGRESS` after six minutes, that means the rule creation failed due to a system error on our service. You can delete the rule and try to create a new one again. **List all streams** This API allows you to retrieve a comprehensive list of all stream rules associated with your [account ID](https://docs.newrelic.com/docs/accounts/accounts-billing/account-structure/account-id/). #### Sample request ```graphql { actor { account(id: YOUR_NR_ACCOUNT_ID) { streamingExport { streamingRules { aws { awsAccountId region deliveryStreamName role } createdAt description id message name nrql status updatedAt payloadCompression } } } } } ``` **View stream rule details** This API allows you to query information about a specific stream rule by providing the account ID and rule ID. #### Sample request ```graphql { actor { account(id: YOUR_NR_ACCOUNT_ID) { streamingExport { streamingRule(id: "RULE_ID") { aws { awsAccountId deliveryStreamName region role } createdAt description id message name nrql status updatedAt payloadCompression } } } } } ``` **Update a stream** This API allows you to modify an existing streaming export rule. #### Input parameters | Parameter | Data Type | Is it Required? | Description | | ----------------------------------- | --------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | ID | Yes | The id of the streaming rule to update. | | `ruleParameters.description` | String | No | A brief description of the rule. Use this to provide any additional information or context about the export rule. | | `ruleParameters.name` | String | Yes | The name of the export rule. | | `ruleParameters.nrql` | String | Yes | An NRQL to select the telemetry data to export. | | `ruleParameters.payloadCompression` | String | No | [payloadCompression](#compression) accepts two values: `DISABLED`: By default, payloads will not be compressed before being exported. `GZIP`: Select this to compress payloads before being exported. | | `awsParameters.awsAccountId` | ID | Yes | Specifies the [AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/console-account-alias.html) that owns the target Firehose delivery stream. For example: `10000000000`. | | `awsParameters.deliveryStreamName` | String | Yes | Specifies the [name of the Kinesis stream](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_CreateStream.html) used for data delivery. For example: `firehose-test-stream`. | | `awsParameters.region` | String | Yes | Specifies the [AWS region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) where the resources are deployed. For example: `us-east-1`. | | `awsParameters.role` | String | Yes | Specifies the [AWS IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) configured for New Relic. This will always be `firehose-role`. | #### Sample request ```graphql mutation { streamingExportUpdateRule( id: RULE_ID, ruleParameters: { description: "ADD_RULE_DESCRIPTION", name: "PROVIDE_RULE_NAME", nrql: "YOUR_NRQL_QUERY", payloadCompression: DISABLED }, awsParameters: { awsAccountId: "YOUR_AWS_ACCOUNT_ID", deliveryStreamName: "FIREHOSE_STREAM_NAME", region: "SPECIFY_AWS_REGION", role: "firehose-role" } ) { id status } } ``` **Response**: You receive a response with the rule ID and status. **Points to note**: When updating, you'll get a message in the message field: “The export rule is being updated and the process may take a few minutes to complete. Please check again later.” It can take up to six minutes to be fully updated. You can check whether the rule is updated by calling [streamingRule](view-stream-rule-details) to retrieve the rule. During the period that the rule is under updating, the rule is locked, and no other mutation action can act on the rule. If you are trying to perform another mutation action on the same rule, you will get a message saying, “The export rule is currently being updated by another request, please wait and try again later.” A user can update a rule of any status except a deleted rule. **Enable a stream** This API allows you to enable an existing streaming rule, facilitating the continuous export of data for real-time processing and integration with other services. #### Input parameters | Parameter | Data Type | Is it Required? | Description | | --------- | --------- | --------------- | --------------------------------------- | | `id` | ID | Yes | The id of the streaming rule to enable. | #### Sample request ```graphql mutation { streamingExportEnableRule(id: RULE_ID) { id status message } } ``` **Response**: If you're trying to enable a rule that's in another state, it returns the error message like, "The export rule can't be enabled or disabled due to status not being allowed." You can't enable the rule if the rule is locked due to another mutation being done. > #### ⚠️ IMPORTANT > > You can only enable the rule when it has a status of DISABLED. **Disable a stream** This API allows you to disable an existing streaming rule, stopping the continuous export of data. #### Input parameters | Parameter | Data Type | Is it Required? | Description | | --------- | --------- | --------------- | ---------------------------------------- | | `id` | ID | Yes | The id of the streaming rule to disable. | #### Sample request ```graphql mutation { streamingExportDisableRule(id: RULE_ID) { id status message } } ``` **Response**: If you're trying to disable a rule that's in another state, it returns the error message like, "The export rule can't be enabled or disabled due to status not being allowed." You can't disable the rule if the rule is locked due to another mutation being done. > #### ⚠️ IMPORTANT > > You can only disable the rule when it has a status of ENABLED. **Delete a stream** This API allows you to delete a streaming rule and stop all data export related to that rule. #### Input parameters | Parameter | Data Type | Is it Required? | Description | | --------- | --------- | --------------- | --------------------------------------- | | `id` | ID | Yes | The id of the streaming rule to delete. | #### Sample request ```graphql mutation { streamingExportDeleteRule(id: RULE_ID) { id } } ``` > #### ⚠️ IMPORTANT > > Deleting can be performed on a rule of any status unless it's already deleted. Once a rule is deleted, it can't be reactivated again. The rule can be still viewed within the first 24 hours after deletion by calling the [steamingRule](#view-stream-rule-details) API with the rule ID. After 24 hours, the rule won't be searchable anymore through NerdGraph. ## Set up an Azure Event Hub [#event-hub-setup] To set up streaming data export to Azure, you must first set up an Event Hub. Follow these steps: Alternatively, you can follow the Azure guide [here](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-create). ### Create an Event Hubs namespace [#create-event-hubs-namespace] 1. From your Microsoft Azure account, navigate to Event Hubs. 2. Follow the steps to create an Event Hubs namespace. We recommend enabling auto-inflate to ensure you receive all of your data. 3. Ensure public access is enabled, as we will use a Shared Access Policy to securely authenticate with your Event Hub. 4. Once your Event Hubs namespace is deployed, click **Go to resource**. ### Create an Event Hub [#create-event-hub] 1. In the left column, click **Event Hubs**. 2. To create an Event Hub, click **+Event Hub**. 3. Enter the desired Event Hub Name. Save this, as you need it later to create the streaming export rule. 4. For **Retention**, select **Delete** `Cleanup policy` and desired `Retention time (hrs)`. > #### ⚠️ IMPORTANT > > Streaming export is currently not supported for Event Hubs with **Compact** retention policy. 5. Once the Event Hub is created, click **Event Hub**. ### Create and attach a shared access policy [#event-hub-policy] 1. In the left column, go to **Shared access policies**. 2. Click **+Add** near the top of the page. 3. Choose a name for your shared access policy. 4. Check **Send**, and click **Create**. 5. Click the created policy, and copy the **Connection string–primary key**. Save this, as you need it later to authenticate and send data to your Event Hub. When you're done with these steps, you can set up your export rules using [NerdGraph](https://docs.newrelic.com/docs/apis/nerdgraph/get-started/introduction-new-relic-nerdgraph/). **Create a stream** This API allows you to create a streaming export rule to send data to an Azure Event Hub. #### Input parameters | Parameter | Data Type | Is it Required? | Description | | ------------------------------------------ | --------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `accountId` | Integer | Yes | Your New Relic [account ID](https://docs.newrelic.com/docs/accounts/accounts-billing/account-structure/account-id/). | | `ruleParameters.description` | String | No | A brief description of the rule. Use this to provide any additional information or context about the export rule. | | `ruleParameters.name` | String | Yes | The name of the export rule. | | `ruleParameters.nrql` | String | Yes | An NRQL to select the telemetry data to export. | | `ruleParameters.payloadCompression` | String | No | [payloadCompression](#compression) accepts two values: `DISABLED`: By default, payloads will not be compressed before being exported. `GZIP`: Select this to compress payloads before being exported. | | `azureParameters.eventHubConnectionString` | String | Yes | This parameter is the [connection string](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string) which provides access to a specific Event Hub. Looks similar to: `Endpoint=sb://.servicebus.windows.net/;SharedAccessKeyName=;SharedAccessKey=;EntityPath=` | | `azureParameters.eventHubName` | String | Yes | This parameter specifies the name of the Event Hub where events will be sent and processed. For example: `my-event-hub.` | #### Sample request ```graphql mutation { streamingExportCreateRule( accountId: YOUR_NR_ACCOUNT_ID, ruleParameters: { description: "ADD_RULE_DESCRIPTION", name: "PROVIDE_RULE_NAME", nrql: "SELECT * FROM NodeStatus", payloadCompression: DISABLED }, azureParameters: { eventHubConnectionString: "YOUR_EVENT_HUB_SAS_CONNECTION_STRING", eventHubName: "YOUR_EVENT_HUB_NAME" } ) { id status } } ``` **Response**: You receive a response with the rule ID and status. The status will be `CREATION_IN_PROGRESS` initially, and it can take up to six minutes for the rule to complete creation. **List all streams** This API allows you to retrieve a comprehensive list of all stream rules associated with your [account ID](https://docs.newrelic.com/docs/accounts/accounts-billing/account-structure/account-id/). #### Sample request ```graphql { actor { account(id: YOUR_NR_ACCOUNT_ID) { streamingExport { streamingRules { azure { eventHubConnectionString eventHubName } createdAt description id message name nrql status updatedAt payloadCompression } } } } } ``` **View stream rule details** This API allows you to query information about a specific stream rule by providing the account ID and rule ID. #### Sample request ```graphql { actor { account(id: YOUR_NR_ACCOUNT_ID) { streamingExport { streamingRule(id: "RULE_ID") { azure { eventHubConnectionString eventHubName } createdAt description id message name nrql status updatedAt payloadCompression } } } } } ``` **Update a stream** This API allows you to modify an existing streaming export rule. #### Input parameters | Parameter | Data Type | Is it Required? | Description | | ------------------------------------------ | --------- | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | ID | Yes | The id of the streaming rule to update. | | `ruleParameters.description` | String | No | A brief description of the rule. Use this to provide any additional information or context about the export rule. | | `ruleParameters.name` | String | Yes | The name of the export rule. | | `ruleParameters.nrql` | String | Yes | An NRQL to select the telemetry data to export. | | `ruleParameters.payloadCompression` | String | No | [payloadCompression](#compression) accepts two values: `DISABLED`: By default, payloads will not be compressed before being exported. `GZIP`: Select this to compress payloads before being exported. | | `azureParameters.eventHubConnectionString` | ID | Yes | This parameter is the connection string which provides access to a specific Event Hub. Looks similar to: `Endpoint=sb://.servicebus.windows.net/;SharedAccessKeyName=;SharedAccessKey=;EntityPath=` | | `azureParameters.eventHubName` | String | Yes | This parameter specifies the name of the Event Hub where events will be sent and processed. For example: `my-event-hub.` | #### Sample request ```graphql mutation { streamingExportUpdateRule( id: RULE_ID, ruleParameters: { description: "ADD_RULE_DESCRIPTION", name: "PROVIDE_RULE_NAME", nrql: "YOUR_NRQL_QUERY", payloadCompression: DISABLED }, azureParameters: { eventHubConnectionString: "YOUR_EVENT_HUB_SAS_CONNECTION_STRING", eventHubName: "YOUR_EVENT_HUB_NAME" } ) { id status } } ``` **Response**: You receive a response with the rule ID and status. **Enable a stream** To enable an existing streaming rule, facilitating the continuous export of data for real-time processing and integration with other services, refer [here](#enable-stream). **Disable a stream** To disable an existing streaming rule, stopping the continuous export of data, refer [here](#disable-stream). **Delete a stream** To delete a streaming rule and stop all data export related to that rule, refer [here](#delete-stream). ## Set up a GCP Pub/Sub [#gcp-pubsub-setup] To set up streaming data export to GCP, you must first set up a Pub/Sub. Follow these steps: ### Create a Pub/Sub topic [#create-pubsub-topic] 1. Form your GCP Console, navigate to the Pub/Sub page. 2. Click **Create topic**. 3. Enter a topic ID and click **Create**. ### Set up permissions on Pub/Sub [#setup-permissions-pubsub] 1. In the right column of the created topic, click **More actions**. 2. Select **View permissions**. 3. Click **Add Principal**, and in the new principals box, enter the service account email provided by us: - US region (default): `us-prod-uds-streaming-export@h0c17c65df9291b526b433650e6a0a.iam.gserviceaccount.com` - EU region: `eu-prod-uds-streaming-export@h0c17c65df9291b526b433650e6a0a.iam.gserviceaccount.com` - JP region: `uds-jp-production@unifieddatastreams.iam.gserviceaccount.com` 4. Under **Assign roles** section, search for Pub/Sub Publisher and click **Save**. When you're done with these steps, you can set up your export rules using [NerdGraph](https://docs.newrelic.com/docs/apis/nerdgraph/get-started/introduction-new-relic-nerdgraph/). **Create a stream** This API allows you to create a streaming export rule to send data to GCP Pub/Sub. #### Input parameters | Parameter | Data Type | Is it Required? | Description | | ----------------------------------- | --------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `accountId` | Integer | Yes | Your New Relic [account ID](https://docs.newrelic.com/docs/accounts/accounts-billing/account-structure/account-id/). | | `ruleParameters.description` | String | No | A brief description of the rule. Use this to provide any additional information or context about the export rule. | | `ruleParameters.name` | String | Yes | The name of the export rule. | | `ruleParameters.nrql` | String | Yes | An NRQL to select the telemetry data to export. | | `ruleParameters.payloadCompression` | String | No | [payloadCompression](#compression) accepts two values: `DISABLED`: By default, payloads will not be compressed before being exported. `GZIP`: Select this to compress Payloads before being exported. | | `gcpParameters.gcpProjectId` | String | Yes | This parameter represents the unique identifier for your GCP project. For example: `google-project-id` | | `gcpParameters.pubsubTopicId` | String | Yes | This parameter specifies the topic in GCP Pub/Sub, used for sending and receiving messages within your cloud applications. For example: `pub-sub-topic-id.` | #### Sample request ```graphql mutation { streamingExportCreateRule( accountId: YOUR_NR_ACCOUNT_ID, ruleParameters: { description: "ADD_RULE_DESCRIPTION", name: "PROVIDE_RULE_NAME", nrql: "SELECT * FROM NodeStatus", payloadCompression: DISABLED }, gcpParameters: { gcpProjectId: "project-id", pubsubTopicId: "pub-sub-topic-id" } ) { id status } } ``` **Response**: You receive a response with the rule ID and status. The status will be `CREATION_IN_PROGRESS` initially, and it can take up to six minutes for the rule to complete creation. **List all streams** This API allows you to retrieve a comprehensive list of all stream rules associated with your [account ID](https://docs.newrelic.com/docs/accounts/accounts-billing/account-structure/account-id/). #### Sample request ```graphql { actor { account(id: YOUR_NR_ACCOUNT_ID) { streamingExport { streamingRules { gcp { gcpProjectId pubsubTopicId } createdAt description id message name nrql status updatedAt payloadCompression } } } } } ``` **View stream rule details** This API allows you to query information about a specific stream rule by providing the account ID and rule ID. #### Sample request ```graphql { actor { account(id: YOUR_NR_ACCOUNT_ID) { streamingExport { streamingRule(id: "RULE_ID") { gcp { gcpProjectId pubsubTopicId } createdAt description id message name nrql status updatedAt payloadCompression } } } } } ``` **Update a stream** This API allows you to modify an existing streaming export rule. #### Input parameters | Parameter | Data Type | Is it Required? | Description | | ----------------------------------- | --------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | ID | Yes | The id of the streaming rule to update. | | `ruleParameters.description` | String | No | A brief description of the rule. Use this to provide any additional information or context about the export rule. | | `ruleParameters.name` | String | Yes | The name of the export rule. | | `ruleParameters.nrql` | String | Yes | An NRQL to select the telemetry data to export. | | `ruleParameters.payloadCompression` | String | No | [payloadCompression](#compression) accepts two values: `DISABLED`: By default, payloads will not be compressed before being exported. `GZIP`: Select this to compress payloads before being exported. | | `gcpParameters.gcpProjectId` | String | Yes | This parameter represents the unique identifier for your Google Cloud Platform (GCP) project. For example: `google-project-id` | | `gcpParameters.pubsubTopicId` | String | Yes | This parameter specifies the topic in Google Cloud Pub/Sub, used for sending and receiving messages within your cloud applications. For example: `pub-sub-topic-id.` | #### Sample request ```graphql mutation { streamingExportUpdateRule( id: RULE_ID, ruleParameters: { description: "ADD_RULE_DESCRIPTION", name: "PROVIDE_RULE_NAME", nrql: "YOUR_NRQL_QUERY", payloadCompression: DISABLED }, gcpParameters: { gcpProjectId: "project-id", pubsubTopicId: "pub-sub-topic-id" } ) { id status } } ``` **Response**: You receive a response with the rule ID and status. **Enable a stream** To enable an existing streaming rule, facilitating the continuous export of data for real-time processing and integration with other services, refer [here](#enable-stream). **Disable a stream** To disable an existing streaming rule, stopping the continuous export of data, refer [here](#disable-stream). **Delete a stream** To delete a streaming rule and stop all data export related to that rule, refer [here](#delete-stream). ## Understand export compression [#compression] You can choose to compress data before exporting it. This feature is off by default. Compressing can help prevent exceeding your data limit and lower outbound data costs. You can enable compression using the `payloadCompression` field under `ruleParameters`. This field can be any of the following values: - `DISABLED`: Payloads will not be compressed before being exported. If unspecified, `payloadCompression` will default to this value. - `GZIP`: Compress payloads with the GZIP format before exporting. GZIP is the only compression format currently available, though we may choose to make more formats available in the future. When compression is enabled on an existing AWS export rule, the next message from Kinesis Firehose may contain both compressed and uncompressed data. This is due to buffering within Kinesis Firehose. To avoid this, you can temporarily disable the export rule before enabling compression, or create a new Kinesis Firehose stream for the compressed data alone to flow through. If you do encounter this issue and you're exporting to S3 or another file storage system, you can view the compressed part of the data by following these steps: 1. Manually download the object. 2. Separate the object into two separate files by copying the compressed data into a new file. 3. Decompress the new, compressed-only data file. Once you have the compressed data, you can re-upload it to S3 (or whatever other service you're using) and delete the old file. Please be aware that in S3 or another file storage system, objects may consist of multiple GZIP-encoded payloads that are appended consecutively. Therefore, your decompression library should have the capability to handle such concatenated GZIP payloads. ### Automatic decompression in AWS Once your data has arrived in AWS, you may want options to automatically decompress it. If you're streaming that data to an S3 bucket, there are two ways to enable automatic decompression: **Object Lambda access point** Access points function as separate methods by which objects in S3 buckets can be accessed and downloaded. AWS supplies a feature called [Object Lambda access points](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html), which perform a Lambda function on each S3 object accessed via the access point. Follow these steps to enable such an access point: 1. Navigate to [this page](https://docs.aws.amazon.com/AmazonS3/latest/userguide/olap-examples.html#olap-examples-3), click the link to the serverless repo. 2. Click **Deploy**. 3. [Set up an access point on your S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-access-points.html). 4. [Create an Object Lambda access point](https://docs.aws.amazon.com/AmazonS3/latest/userguide/olap-create.html). This access point must have these settings: - The **Supporting Access Point** on this Lambda access point will need to be set to the access point you set up on the S3 bucket. - Under **Transformation Configuration**: - Select **GetObject**. - Specify the DecompressGZFunction Lambda function, or another necessary function if using a different compression format. **Metadata-setting Lambda function** AWS will automatically decompress objects downloaded from S3, if those objects have the correct metadata set. We have written a function that automatically applies this metadata to every new object downloaded to a set S3 object. Here's how to set it up: 1. Navigate [here](https://github.com/newrelic/metadata-setting-lambda-function), clone the repository locally, and follow the provided steps in the README file to generate a ZIP file containing the lambda function. 2. Create an [IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) for the function. - When [creating the role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html#roles-creatingrole-service-console), be sure to set the trusted entity type as "AWS Service", with "Lambda" as your use case. - This role must have a policy with these permissions: `s3:PutObject` and `s3:GetObject`. 3. In AWS, navigate to the [Lambda functions page](https://console.aws.amazon.com/lambda/home#/functions). 4. Click **Create function**. 5. Select the Java 11 runtime environment. 6. Click **Change default execution role > Use an existing role**. Enter the role you created in step 2 here. 7. Scroll down and click **Create function**. 8. Once the function has been created, click **Upload from** and select **.zip or .jar file** from the dropdown. 9. Click **Upload** from the box that pops up, and select the ZIP file created in step 1. 10. Once the upload has finished, click **Save** to exit the pop-up box. 11. Edit **Runtime settings** by adding the handler. In our provided function the handler is `metadatasetter.App::handleRequest`. 12. All that's left to do now is enable this Lambda function to trigger on S3 object creation. Click **Add trigger** to start setting that up. 13. From the dropdown, select **S3** as your source. 14. Enter the name of the S3 bucket you'd like to apply the metadata to in the **Bucket** field. 15. Remove the default **All object create events** from the event types. From the Event types dropdown, select **PUT**. 16. Check the **Recursive invocation** box, then click **Add** in the bottom right. The Lambda function will now start automatically adding the compression metadata to all newly added S3 objects. ### Automatic decompression in Azure If you're exporting data to Azure, it's possible to view decompressed versions of the objects stored in your event hub using a [Stream Analytics Job](https://learn.microsoft.com/en-us/azure/event-hubs/process-data-azure-stream-analytics). To do so, follow these steps: 1. Follow [this guide](https://learn.microsoft.com/en-us/azure/event-hubs/process-data-azure-stream-analytics) up to step 16. - On step 13, you may choose to use the same event hub as the output without breaking anything, but this is not recommended if you intend to proceed to step 17 and start the job, as this approach has not been tested. 2. In the left pane of your streaming analytics job, click **Inputs**, then click the input you set up. 3. Scroll down to the bottom of the pane that appears on the right, and configure the input with these settings: - Event serialization format: JSON - Encoding: UTF-8 - Event compression type: GZip 4. Click **Save** at the bottom of the pane. 5. Click **Query** on the side of the screen. Using the **Input preview** tab, you should now be able to query the event hub from this screen. ### Automatic decompression in GCP In GCP Cloud Storage, objects will automatically decompress when downloaded if the metadata is set to Content-Encoding: gzip. For more details, check the [GCP documentation](https://cloud.google.com/storage/docs/transcoding).