--- title: Google Cloud Dataproc monitoring integration source: https://docs.newrelic.com/docs/infrastructure/google-cloud-platform-integrations/gcp-integrations-list/google-cloud-dataproc-monitoring-integration --- [New Relic integrations](https://docs.newrelic.com/docs/infrastructure/introduction-infra-monitoring) with the [Google Cloud Platform (GCP)](https://cloud.google.com/) include one that reports [Google Cloud Dataproc](https://cloud.google.com/dataproc) data to New Relic. This document explains how to activate the GCP Dataproc integration and describes the data it reports. ## Features Cloud Dataproc is Google Cloud's managed service for running Apache Hadoop, Apache Spark, and other open-source data-processing workloads. New Relic Dataproc integration collects HDFS, YARN, job, and operation metrics for each Dataproc cluster. ## Activate integration [#activate] To enable the integration, follow standard procedures to [connect your GCP service to New Relic](https://docs.newrelic.com/docs/connect-google-cloud-platform-services-infrastructure): - [Connect with Workload Identity Federation (recommended)](https://docs.newrelic.com/docs/connect-google-cloud-platform-services-infrastructure) - [Connect with service account or user account](https://docs.newrelic.com/docs/connect-google-cloud-platform-services-infrastructure) ## Polling frequency [#polling] New Relic integrations query your GCP services according to a polling interval that varies by integration. The polling frequency for Google Cloud Dataproc is 5 minutes. The resolution is 1 data point every minute. > #### ⚠️ IMPORTANT > > Cloud Dataproc supports 1-minute polling intervals for WIF integrations in limited preview. To request access, contact [New Relic support](https://support.newrelic.com/). ## Workload Identity Federation [#wif] ### Find and use data [#find-data-wif] After you enable the integration, your Dataproc resources appear as entities in the New Relic entity explorer. To see dashboards and manage services, go to [one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Infrastructure > GCP. All Dataproc metrics available in GCP Cloud Monitoring are collected as dimensional metrics in the `Metric` event type. Additional metrics beyond this table are collected automatically. See [Google's Dataproc metrics documentation](https://cloud.google.com/monitoring/api/metrics_gcp_d_h#gcp-dataproc) for the complete list. #### Entities **Dataproc entities** | Entity | Entity type | Resource type | | ------- | -------------------- | ------------------------ | | Cluster | `GCPDATAPROCCLUSTER` | `cloud_dataproc_cluster` | ### Metric data [#metrics-wif] #### Key metrics — Cluster | Metric name | Unit | Description | | ------------------------------------------------------- | --------- | ----------------------------------------------------------------------- | | `gcp.dataproc.cluster.hdfs.datanodes` | Count | Number of HDFS DataNodes running on the cluster. | | `gcp.dataproc.cluster.hdfs.storage_capacity` | Gibibytes | HDFS storage capacity available on the cluster. | | `gcp.dataproc.cluster.hdfs.storage_utilization` | Percent | Fraction of HDFS storage currently in use on the cluster. | | `gcp.dataproc.cluster.hdfs.unhealthy_blocks` | Count | Number of unhealthy HDFS blocks on the cluster. | | `gcp.dataproc.cluster.job.submitted_count` | Count | Number of jobs submitted to the cluster. | | `gcp.dataproc.cluster.job.running_count` | Count | Number of jobs currently running on the cluster. | | `gcp.dataproc.cluster.job.failed_count` | Count | Number of jobs that have failed on the cluster. | | `gcp.dataproc.cluster.job.completion_time` | Seconds | Time from job submission to Dataproc reporting completion. | | `gcp.dataproc.cluster.job.duration` | Seconds | Time jobs have spent in a given state, faceted by state. | | `gcp.dataproc.cluster.operation.submitted_count` | Count | Number of cluster operations submitted. | | `gcp.dataproc.cluster.operation.running_count` | Count | Number of cluster operations currently running. | | `gcp.dataproc.cluster.operation.failed_count` | Count | Number of cluster operations that have failed. | | `gcp.dataproc.cluster.operation.completion_time` | Seconds | Time from operation submission to Dataproc reporting completion. | | `gcp.dataproc.cluster.yarn.allocated_memory_percentage` | Percent | Fraction of YARN memory currently allocated. | | `gcp.dataproc.cluster.yarn.memory_size` | Gibibytes | Total YARN memory available on the cluster. | | `gcp.dataproc.cluster.yarn.pending_memory_size` | Gibibytes | YARN memory currently requested but not yet allocated by the scheduler. | | `gcp.dataproc.cluster.yarn.containers` | Count | Number of YARN containers on the cluster. | | `gcp.dataproc.cluster.yarn.apps` | Count | Number of active YARN applications on the cluster. | | `gcp.dataproc.cluster.yarn.nodemanagers` | Count | Number of YARN NodeManagers running on the cluster. | | `gcp.dataproc.cluster.yarn.virtual_cores` | Count | Number of YARN virtual cores on the cluster. | For the complete list of Dataproc metrics, see [Google's Dataproc metrics documentation](https://cloud.google.com/monitoring/api/metrics_gcp_d_h#gcp-dataproc). ### Metrics-only resource types [#metrics-only] New Relic also collects metrics for the following Dataproc resource types. These resource types don't create entities in the entity explorer, so query their metrics directly on the `Metric` event type. **Job — 3 metrics** | Metric name | Unit | Description | | -------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------ | | `gcp.dataproc.job.state` | Count | Indicates whether job is currently in a particular state or not. True indicates in that state and False indicates exited that state. | | `gcp.dataproc.job.yarn.memory_seconds` | Count | Indicates the Memory Seconds consumed by the `job_id` job per yarn `application_id`. | | `gcp.dataproc.job.yarn.vcore_seconds` | Count | Indicates the VCore Seconds consumed by the `job_id` job per yarn `application_id`. | **Batch — 1 metric** | Metric name | Unit | Description | | ------------------------------------ | ----- | ---------------------------------------------- | | `gcp.dataproc.batch.spark.executors` | Count | Indicates the number of Batch Spark executors. | **Session — 1 metric** | Metric name | Unit | Description | | -------------------------------------- | ----- | ------------------------------------------------ | | `gcp.dataproc.session.spark.executors` | Count | Indicates the number of Session Spark executors. | ## Service account or user account [#service-account] ### Find and use data [#find-data] After activating the integration and waiting a few minutes (based on the [polling frequency](#polling)), data will appear in the New Relic UI. To [find and use your data](https://docs.newrelic.com/docs/infrastructure/integrations/find-use-infrastructure-integration-data), including links to your dashboards and alert settings, go to [one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Infrastructure > GCP > (select an integration). Data is attached to the following [event type](https://docs.newrelic.com/docs/data-apis/understand-data/new-relic-data-types/#event-data): | Entity | Event Type | Provider | | ------- | -------------------------- | -------------------- | | Cluster | `GcpDataprocClusterSample` | `GcpDataprocCluster` | For more on how to use your data, see [Understand and use integration data](https://docs.newrelic.com/docs/infrastructure/integrations/find-use-infrastructure-integration-data). ### Dataproc Cluster data | Metric | Unit | Description | | ---------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------- | | `cluster.hdfs.Datanodes` | Count | Indicates the number of HDFS DataNodes that are running inside a cluster. | | `cluster.hdfs.StorageCapacity` | Gibibytes | Indicates capacity of HDFS system running on cluster in GB. | | `cluster.hdfs.StorageUtilization` | Percent | The percentage of HDFS storage currently used. | | `cluster.hdfs.UnhealthyBlocks` | Count | Indicates the number of unhealthy blocks inside the cluster. | | `cluster.job.CompletionTime` | Seconds | The time jobs took to complete from the time the user submits a job to the time Dataproc reports it is completed. | | `cluster.job.Duration` | Seconds | The time jobs have spent in a given state. | | `cluster.job.Failures` | Count | Indicates the number of jobs that have failed on a cluster. | | `cluster.job.Running` | Count | Indicates the number of jobs that are running on a cluster. | | `cluster.job.Submitted` | Count | Indicates the number of jobs that have been submitted to a cluster. | | `cluster.operation.CompletionTime` | Seconds | The time operations took to complete from the time the user submits a operation to the time Dataproc reports it is completed. | | `cluster.operation.Duration` | Seconds | The time operations have spent in a given state. | | `cluster.operation.Failures` | Count | Indicates the number of operations that have failed on a cluster. | | `cluster.operation.Running` | Count | Indicates the number of operations that are running on a cluster. | | `cluster.operation.Submitted` | Count | Indicates the number of operations that have been submitted to a cluster. | | `cluster.yarn.AllocatedMemoryPercentage` | Percent | The percentage of YARN memory is allocated. | | `cluster.yarn.Apps` | Count | Indicates the number of active YARN applications. | | `cluster.yarn.Containers` | Count | Indicates the number of YARN containers. | | `cluster.yarn.MemorySize` | Gibibytes | Indicates the YARN memory size in GB. | | `cluster.yarn.Nodemanagers` | Count | Indicates the number of YARN NodeManagers running inside cluster. | | `cluster.yarn.PendingMemorySize` | Gibibytes | The current memory request, in GB, that is pending to be fulfilled by the scheduler. | | `cluster.yarn.VirtualCores` | Count | Indicates the number of virtual cores in YARN. |