--- title: Google BigQuery monitoring integration source: https://docs.newrelic.com/docs/infrastructure/google-cloud-platform-integrations/gcp-integrations-list/google-bigquery-monitoring-integration --- [New Relic integrations](https://docs.newrelic.com/docs/infrastructure/introduction-infra-monitoring) with the [Google Cloud Platform (GCP)](https://cloud.google.com/) include one that reports [Google BigQuery](https://cloud.google.com/bigquery/) data to New Relic. This document explains how to activate the GCP BigQuery integration and describes the data it reports. ## Features BigQuery is Google Cloud's serverless data warehouse. New Relic BigQuery integration collects query, execution, storage, and slot utilization metrics at both project and dataset scope. ## Activate integration [#activate] To enable the integration, follow standard procedures to [connect your GCP service to New Relic](https://docs.newrelic.com/docs/connect-google-cloud-platform-services-infrastructure): - [Connect with Workload Identity Federation (recommended)](https://docs.newrelic.com/docs/connect-google-cloud-platform-services-infrastructure) - [Connect with service account or user account](https://docs.newrelic.com/docs/connect-google-cloud-platform-services-infrastructure) ## Polling frequency [#polling] New Relic integrations query your GCP services according to a polling interval that varies by integration. The polling frequency for Google BigQuery is 5 minutes. The resolution is 1 data point every minute. > #### ⚠️ IMPORTANT > > BigQuery supports 1-minute polling intervals for WIF integrations in limited preview. To request access, contact [New Relic support](https://support.newrelic.com/). ## Activate Table metrics [#table-metrics] The collection of BigQuery table metrics is disabled by default. This affects the metrics listed under [GcpBigQueryTableSample](#table-sample). To collect those metrics, turn on the "Enable table metrics" setting under the configuration menu of the BigQuery integration. ## Workload Identity Federation [#wif] ### Find and use data [#find-data-wif] After you enable the integration, your BigQuery resources appear as entities in the New Relic entity explorer. To see dashboards and manage services, go to [one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Infrastructure > GCP. All BigQuery metrics available in GCP Cloud Monitoring are collected as dimensional metrics in the `Metric` event type. Additional metrics beyond this table are collected automatically. See [Google's BigQuery metrics documentation](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-bigquery) for the complete list. #### Entities **BigQuery entities** | Entity | Entity type | Resource type | | ------- | -------------------- | ------------------ | | Project | `GCPBIGQUERYPROJECT` | `bigquery_project` | | Dataset | `GCPBIGQUERYDATASET` | `bigquery_dataset` | | Table | `GCPBIGQUERYTABLE` | `bigquery_table` | ### Metric data [#metrics-wif] #### Key metrics — Project | Metric name | Unit | Description | | ------------------------------------------ | ------- | ------------------------------------------------------------------------------- | | `gcp.bigquery.query.count` | Count | Number of queries executed in the project, faceted by `query_type` and `state`. | | `gcp.bigquery.query.execution_times` | Seconds | Query execution time distribution across the project. | | `gcp.bigquery.slots.allocated_for_project` | Count | Number of slots currently allocated to the project. | | `gcp.bigquery.storage.uploaded_bytes` | Bytes | Bytes uploaded to BigQuery in the project. | | `gcp.bigquery.storage.uploaded_row_count` | Count | Number of rows uploaded to BigQuery in the project. | #### Key metrics — Dataset | Metric name | Unit | Description | | -------------------------------------------- | ----- | --------------------------------------------------------------- | | `gcp.bigquery.storage.stored_bytes` | Bytes | Total bytes stored in the dataset. | | `gcp.bigquery.storage.table_count` | Count | Number of tables in the dataset. | | `gcp.bigquery.storage.stored_logical_bytes` | Bytes | Logical bytes stored in the dataset (uncompressed size). | | `gcp.bigquery.storage.stored_physical_bytes` | Bytes | Physical bytes stored in the dataset (compressed on-disk size). | #### Key metrics — Table | Metric name | Unit | Description | | -------------------------------------------- | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `gcp.bigquery.storage.stored_bytes` | Bytes | Number of logical bytes stored in the table. GCP labels the top 100 tables per dataset (by size) with a `table` label on this metric; the rest are grouped together with no table label. | | `gcp.bigquery.storage.uploaded_bytes` | Bytes | Bytes uploaded to the table. | | `gcp.bigquery.storage.uploaded_bytes_billed` | Bytes | Bytes billed for data uploaded to the table. Currently only the streaming API is billed. | | `gcp.bigquery.storage.uploaded_row_count` | Count | Number of rows uploaded to the table. | For the complete list of BigQuery metrics, see [Google's BigQuery metrics documentation](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-bigquery). ### Metrics-only resource types [#metrics-only] New Relic also collects metrics for the following BigQuery resource types. These resource types don't create entities in the entity explorer, so query their metrics directly on the `Metric` event type. **Project-wide (Global) — 10 metrics** | Metric name | Unit | Description | | ------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `gcp.bigquery.query.count` | Count | In flight queries. | | `gcp.bigquery.query.execution_times` | Seconds | Distribution of execution times for queries that completed successfully within the last sampling interval. Incomplete and failed queries are not included. | | `gcp.bigquery.query.scanned_bytes` | Bytes | Scanned bytes. | | `gcp.bigquery.query.scanned_bytes_billed` | Bytes | Scanned bytes billed. | | `gcp.bigquery.slots.allocated` | Count | Number of BigQuery slots currently allocated for project. Slot allocation can be broken down based on reservation and job type. The accuracy of this metric can be poor, especially the break-down by reservation. Consider using resource charts instead. | | `gcp.bigquery.slots.allocated_for_project` | Count | Number of BigQuery slots currently allocated for query jobs in the project. | | `gcp.bigquery.slots.allocated_for_project_and_job_type` | Count | Number of BigQuery slots currently allocated for the project and job type. | | `gcp.bigquery.slots.allocated_for_reservation` | Count | Number of BigQuery slots currently allocated for project in the reservation. | | `gcp.bigquery.slots.total_allocated_for_reservation` | Count | Number of BigQuery slots currently allocated across projects in the reservation. Note that the metric data is only reported while at least one project has been assigned to the reservation and is consuming slots. As an alternative, consider querying reservations information from [`INFORMATION_SCHEMA`](https://cloud.google.com/bigquery/docs/information-schema-reservations), which doesn't have these limitations. | | `gcp.bigquery.slots.total_available` | Count | (Deprecated) Total number of BigQuery slots available for the project. If you are using the BigQuery Reservations API, consider using `bigquery.googleapis.com/slots/assigned` and `bigquery.googleapis.com/slots/max_assigned` instead. | **Continuous Job — 7 metrics** | Metric name | Unit | Description | | ------------------------------------------------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------- | | `gcp.bigquery.job.continuous.backlog_records` | Count | The estimated number of backlog records for each stage of the continuous job. | | `gcp.bigquery.job.continuous.estimated_backlog_logical_bytes` | Bytes | The number of bytes in the backlog for each stage of the continuous job. | | `gcp.bigquery.job.continuous.estimated_bytes_processed` | Bytes | The estimated number of bytes processed for each stage of the continuous job. | | `gcp.bigquery.job.continuous.output_watermark` | Microseconds | The most recent timestamp, in microseconds since the epoch, up to which all data has been processed by this stage of the continuous job. | | `gcp.bigquery.job.continuous.records_read` | Count | The number of input records read for each stage of the continuous job. | | `gcp.bigquery.job.continuous.records_written` | Count | The number of output records written for each stage of the continuous job. | | `gcp.bigquery.job.continuous.slots.used_time` | Milliseconds | The total slot milliseconds used by the continuous job. | **Storage API Project — 7 metrics** | Metric name | Unit | Description | | ---------------------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `gcp.bigquerystorage.write.concurrent_connections` | Count | Concurrent connection per project for BigQuery Write API. To be consistent with concurrent connection quota, the metric is by default on the project that initiated the AppendRowsRequests (associated with the API key or the service account). | | `gcp.bigquerystorage.write.connection_results_count` | Count | Total accumulation of connection results with BigQuery Write API. | | `gcp.bigquerystorage.write.server_side_latencies` | Count | The latency distribution (in milliseconds) in BigQuery Write API frontend from receiving a AppendRowsRequest to sending a AppendRowsResponse. | | `gcp.bigquerystorage.write.transcoding_latencies` | Count | The transcoding latency distribution (in milliseconds) in BigQuery Write API frontend. | | `gcp.bigquerystorage.write.uploaded_bytes_count` | Count | Total accumulation of BigQuery Write API ingested bytes. | | `gcp.bigquerystorage.write.uploaded_row_count` | Count | Total accumulation of BigQuery Write API ingested rows. | | `gcp.bigquerystorage.read.concurrent_streams` | Count | Number of concurrent ReadRows streams. | **Storage API Dataflow Write — 6 metrics** | Metric name | Unit | Description | | ------------------------------------------------------------- | ----- | --------------------------------------------------------------------------------------------------------------------------------------------- | | `gcp.bigquerystorage.dataflow_write.connection_results_count` | Count | Total accumulation of BigQuery Write API connection results. | | `gcp.bigquerystorage.dataflow_write.request_count` | Count | Total accumulation of BigQuery Write API request count. | | `gcp.bigquerystorage.dataflow_write.server_side_latencies` | Count | The latency distribution (in milliseconds) in BigQuery Write API frontend from receiving a AppendRowsRequest to sending a AppendRowsResponse. | | `gcp.bigquerystorage.dataflow_write.transcoding_latencies` | Count | The duration distribution (in milliseconds) in BigQuery Write API frontend taken by transcoding data. | | `gcp.bigquerystorage.dataflow_write.uploaded_bytes_count` | Count | Total accumulation of BigQuery Write API ingested bytes. | | `gcp.bigquerystorage.dataflow_write.uploaded_row_count` | Count | Total accumulation of BigQuery Write API ingested rows. | **Data Transfer Config — 4 metrics** | Metric name | Unit | Description | | ------------------------------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------ | | `gcp.bigquerydatatransfer.transfer_config.active_runs` | Count | Number of transfer runs that are running or pending of the transfer configuration. | | `gcp.bigquerydatatransfer.transfer_config.completed_runs` | Count | Number of completed DTS runs per transfer configuration. | | `gcp.bigquerydatatransfer.transfer_config.last_execution_job_rows` | Count | Number of returned rows from last executed job of transfer configuration. | | `gcp.bigquerydatatransfer.transfer_config.run_duration_seconds` | Seconds | Distribution of the execution time (in seconds) of each transfer run per transfer configuration. | **Managed Routine Invocation — 3 metrics** | Metric name | Unit | Description | | --------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------- | | `gcp.bigquery.managed_routine.python.cpu_utilizations` | Percent | CPU utilization per each instance. | | `gcp.bigquery.managed_routine.python.max_request_concurrencies` | Count | Distribution of maximum concurrent requests across all Python UDF instances. | | `gcp.bigquery.managed_routine.python.memory_utilizations` | Percent | Memory utilization per each instance. | **BI Engine Model — 3 metrics** | Metric name | Unit | Description | | ---------------------------------------------- | ------------ | ------------------------------------------------------------------------------------------------ | | `gcp.bigquerybiengine.model.inflight_requests` | Count | Inflight requests to a BI model. Only reported without LookerStudio native BigQuery integration. | | `gcp.bigquerybiengine.model.request_count` | Count | BI Engine request count. Only reported without LookerStudio native BigQuery integration. | | `gcp.bigquerybiengine.model.request_latencies` | Milliseconds | Distribution of request execution times. | **BI Engine Project — 3 metrics** | Metric name | Unit | Description | | ---------------------------------------------------------- | ----- | ----------------------------------------------------------------------------------------------- | | `gcp.bigquerybiengine.reservation.top_tables_cached_bytes` | Bytes | The amount of data stored in the BI Engine cache per table for the largest tables in the cache. | | `gcp.bigquerybiengine.reservation.total_bytes` | Bytes | Size of the BI engine reservation. | | `gcp.bigquerybiengine.reservation.used_bytes` | Bytes | Utilization of the BI engine reservation. | ## Service account or user account [#service-account] ### Find and use data [#find-data] After activating the integration and waiting a few minutes (based on the [polling frequency](#polling)), data will appear in the New Relic UI. To [find and use your data](https://docs.newrelic.com/docs/infrastructure/integrations/find-use-infrastructure-integration-data), including links to your dashboards and alert settings, go to [one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Infrastructure > GCP > (select an integration). To view [metric data](https://docs.newrelic.com/docs/telemetry-data-platform/understand-data/new-relic-data-types/#metrics) for your Google BigQuery integration in New Relic, [create NRQL queries](https://docs.newrelic.com/docs/integrations/new-relic-integrations/getting-started/use-integration-data-new-relic-insights#nrql) for `GcpBigQueryProjectSample`, `GcpBigQueryDatasetSample`, and `GcpBigQueryTableSample` [events](https://docs.newrelic.com/docs/using-new-relic/welcome-new-relic/getting-started/glossary#event) and their attributes. ### GcpBigQueryProjectSample [#project-sample] Query `GcpBigQueryProjectSample` events in New Relic to view data for the following metrics. Resolution is one data point per minute with an 8-minute delay. | Attribute | Description | | ------------------------------------- | ----------------------------------------------------------------------- | | `query.Count` | In-flight queries. | | `query.ExecutionTimes` | Distribution of query execution times. | | `slots.Allocated` | Number of slots currently allocated for project. | | `slots.AllocatedForProject` | Number of slots currently allocated for query jobs in the project. | | `slots.AllocatedForProjectAndJobType` | Number of slots currently allocated for the project and job type. | | `slots.AllocatedForReservation` | Number of slots currently allocated for project in the reservation. | | `slots.TotalAllocatedForReservation` | Number of slots currently allocated across projects in the reservation. | | `slots.TotalAvailable` | Total number of slots available for the project. | ### GcpBigQueryDatasetSample [#dataset-sample] Query `GcpBigQueryDatasetSample` events in New Relic to view data for the following metrics. Resolution is one data point every 30 minutes with a 3-hour delay. | Attribute | Description | | --------------------- | ----------------------- | | `storage.StoredBytes` | Amount of stored bytes. | | `storage.Tables` | Number of tables. | ### GcpBigQueryTableSample [#table-sample] Query `GcpBigQueryTableSample` events to view data for the following metrics. Resolution is one data point per minute with a 6-hour delay. | Attribute | Description | | ----------------------------- | -------------------------------------------------------------- | | `storage.UploadedBytes` | Uploaded bytes. | | `storage.UploadedBytesBilled` | Uploaded bytes billed. Currently only streaming API is billed. | | `storage.UploadedRowCount` | Uploaded rows. |