DCGM Exporter Metrics#

Metric Selection Model#

DCGM Exporter exposes the fields selected by its collector CSV, ConfigMap, or YAML configuration. This page documents the shipped collector configuration and exporter-specific behavior. Use DCGM Field Identifiers for authoritative field meanings, units, entity support, and hardware requirements.

This page exhaustively lists the fields shipped with this DCGM Exporter release and all exporter-owned metric families. It does not list every DCGM field that a custom collector can select or guarantee that a configured field will be emitted on a particular system. Use the DCGM field reference for field semantics and the exporter’s /metrics endpoint for its effective runtime output.

An active CSV row is enabled by default. A commented row is shipped as an optional example. The configured Prometheus type is used as written; DCGM Exporter does not infer or semantically validate it against the DCGM field definition. A row with type label adds a label when DCGM returns a value for the selected entity instead of creating its own metric family.

Selecting a row does not guarantee runtime emission. DCGM, the GPU and other entities, the driver, permissions, and the deployment configuration determine whether a field is available.

Shipped Default Metrics#

These active rows are enabled by the shipped default collector CSV. The linked canonical field column makes legacy exporter names explicit and leads to the source of truth for what each field means.

Configured name	Prometheus type	Canonical DCGM field
`DCGM_FI_DEV_SM_CLOCK`	`gauge`	`DCGM_FI_DEV_SM_CLOCK`
`DCGM_FI_DEV_MEM_CLOCK`	`gauge`	`DCGM_FI_DEV_MEM_CLOCK`
`DCGM_FI_DEV_MEMORY_TEMP`	`gauge`	`DCGM_FI_DEV_MEMORY_TEMP_CELSIUS`
`DCGM_FI_DEV_GPU_TEMP`	`gauge`	`DCGM_FI_DEV_GPU_TEMP_CELSIUS`
`DCGM_FI_DEV_POWER_USAGE`	`gauge`	`DCGM_FI_DEV_BOARD_POWER_WATTS`
`DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION`	`counter`	`DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION`
`DCGM_FI_PROF_PCIE_TX_BYTES`	`gauge`	`DCGM_FI_PROF_PCIE_TX_BYTES`
`DCGM_FI_PROF_PCIE_RX_BYTES`	`gauge`	`DCGM_FI_PROF_PCIE_RX_BYTES`
`DCGM_FI_DEV_PCIE_REPLAY_COUNTER`	`counter`	`DCGM_FI_DEV_PCIE_REPLAY_TOTAL`
`DCGM_FI_DEV_GPU_UTIL`	`gauge`	`DCGM_FI_DEV_GPU_UTIL_RATIO`
`DCGM_FI_DEV_MEM_COPY_UTIL`	`gauge`	`DCGM_FI_DEV_MEM_COPY_UTIL`
`DCGM_FI_DEV_ENC_UTIL`	`gauge`	`DCGM_FI_DEV_ENC_UTIL`
`DCGM_FI_DEV_DEC_UTIL`	`gauge`	`DCGM_FI_DEV_DEC_UTIL`
`DCGM_FI_DEV_XID_ERRORS`	`gauge`	`DCGM_FI_DEV_XID_ERROR`
`DCGM_FI_DEV_FB_FREE`	`gauge`	`DCGM_FI_DEV_FB_FREE`
`DCGM_FI_DEV_FB_USED`	`gauge`	`DCGM_FI_DEV_FB_USED`
`DCGM_FI_DEV_FB_RESERVED`	`gauge`	`DCGM_FI_DEV_FB_RESERVED`
`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL`	`gauge`	`DCGM_FI_DEV_NVLINK_THROUGHPUT_TOTAL`
`DCGM_FI_DEV_VGPU_LICENSE_STATUS`	`gauge`	`DCGM_FI_DEV_VGPU_LICENSE_STATUS`
`DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS`	`counter`	`DCGM_FI_DEV_ROW_REMAP_UNCORRECTABLE_TOTAL`
`DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS`	`counter`	`DCGM_FI_DEV_ROW_REMAP_CORRECTABLE_TOTAL`
`DCGM_FI_DEV_ROW_REMAP_FAILURE`	`gauge`	`DCGM_FI_DEV_ROW_REMAP_FAILED`
`DCGM_FI_DRIVER_VERSION`	`label`	`DCGM_FI_SYSTEM_DRIVER_VERSION`
`DCGM_FI_PROF_GR_ENGINE_ACTIVE`	`gauge`	`DCGM_FI_PROF_GR_ENGINE_UTIL_RATIO`
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`	`gauge`	`DCGM_FI_PROF_TENSOR_UTIL_RATIO`
`DCGM_FI_PROF_DRAM_ACTIVE`	`gauge`	`DCGM_FI_PROF_DRAM_UTIL_RATIO`

Shipped Optional Metrics#

These rows are present but commented in the default collector CSV. Enable them with a custom CSV, YAML metrics.fields list, or Helm customMetrics value. Exporter-owned rows are documented separately below.

Configured name	Prometheus type	Canonical DCGM field
`DCGM_FI_DEV_POWER_VIOLATION`	`counter`	`DCGM_FI_DEV_POWER_VIOLATION`
`DCGM_FI_DEV_THERMAL_VIOLATION`	`counter`	`DCGM_FI_DEV_THERMAL_VIOLATION`
`DCGM_FI_DEV_SYNC_BOOST_VIOLATION`	`counter`	`DCGM_FI_DEV_SYNC_BOOST_VIOLATION`
`DCGM_FI_DEV_BOARD_LIMIT_VIOLATION`	`counter`	`DCGM_FI_DEV_BOARD_LIMIT_VIOLATION`
`DCGM_FI_DEV_LOW_UTIL_VIOLATION`	`counter`	`DCGM_FI_DEV_LOW_UTIL_VIOLATION`
`DCGM_FI_DEV_RELIABILITY_VIOLATION`	`counter`	`DCGM_FI_DEV_RELIABILITY_VIOLATION`
`DCGM_FI_DEV_ECC_SBE_VOL_TOTAL`	`counter`	`DCGM_FI_DEV_ECC_SBE_VOL_TOTAL`
`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL`	`counter`	`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL`
`DCGM_FI_DEV_ECC_SBE_AGG_TOTAL`	`counter`	`DCGM_FI_DEV_ECC_SBE_AGG_TOTAL`
`DCGM_FI_DEV_ECC_DBE_AGG_TOTAL`	`counter`	`DCGM_FI_DEV_ECC_DBE_AGG_TOTAL`
`DCGM_FI_DEV_RETIRED_SBE`	`counter`	`DCGM_FI_DEV_PAGE_RETIRED_SBE_TOTAL`
`DCGM_FI_DEV_RETIRED_DBE`	`counter`	`DCGM_FI_DEV_PAGE_RETIRED_DBE_TOTAL`
`DCGM_FI_DEV_RETIRED_PENDING`	`counter`	`DCGM_FI_DEV_PAGE_RETIRED_PENDING`
`DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL`	`counter`	`DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_TOTAL`
`DCGM_FI_DEV_NVLINK_CRC_DATA_ERROR_COUNT_TOTAL`	`counter`	`DCGM_FI_DEV_NVLINK_CRC_DATA_ERROR_TOTAL`
`DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL`	`counter`	`DCGM_FI_DEV_NVLINK_REPLAY_ERROR_TOTAL`
`DCGM_FI_DEV_NVLINK_RECOVERY_ERROR_COUNT_TOTAL`	`counter`	`DCGM_FI_DEV_NVLINK_RECOVERY_ERROR_TOTAL`
`DCGM_FI_DEV_NVLINK_BANDWIDTH_L0`	`counter`	`DCGM_FI_DEV_NVLINK_THROUGHPUT_L0`
`DCGM_FI_NVML_VERSION`	`label`	`DCGM_FI_SYSTEM_NVML_VERSION`
`DCGM_FI_DEV_BRAND`	`label`	`DCGM_FI_DEV_GPU_BRAND`
`DCGM_FI_DEV_SERIAL`	`label`	`DCGM_FI_DEV_BOARD_SERIAL`
`DCGM_FI_DEV_OEM_INFOROM_VER`	`label`	`DCGM_FI_DEV_INFOROM_OEM_VERSION`
`DCGM_FI_DEV_ECC_INFOROM_VER`	`label`	`DCGM_FI_DEV_INFOROM_ECC_VERSION`
`DCGM_FI_DEV_POWER_INFOROM_VER`	`label`	`DCGM_FI_DEV_INFOROM_POWER_VERSION`
`DCGM_FI_DEV_INFOROM_IMAGE_VER`	`label`	`DCGM_FI_DEV_INFOROM_IMAGE_VERSION`
`DCGM_FI_DEV_VBIOS_VERSION`	`label`	`DCGM_FI_DEV_VBIOS_VERSION`
`DCGM_FI_PROF_SM_ACTIVE`	`gauge`	`DCGM_FI_PROF_SM_UTIL_RATIO`
`DCGM_FI_PROF_SM_OCCUPANCY`	`gauge`	`DCGM_FI_PROF_SM_OCCUPANCY_RATIO`
`DCGM_FI_PROF_PIPE_FP64_ACTIVE`	`gauge`	`DCGM_FI_PROF_FP64_UTIL_RATIO`
`DCGM_FI_PROF_PIPE_FP32_ACTIVE`	`gauge`	`DCGM_FI_PROF_FP32_UTIL_RATIO`
`DCGM_FI_PROF_PIPE_FP16_ACTIVE`	`gauge`	`DCGM_FI_PROF_FP16_UTIL_RATIO`
`DCGM_FI_PROF_SM_CYCLES_ELAPSED_TOTAL`	`counter`	`DCGM_FI_PROF_SM_CYCLES_ELAPSED_TOTAL`
`DCGM_FI_PROF_SM_CYCLES_ACTIVE_TOTAL`	`counter`	`DCGM_FI_PROF_SM_CYCLES_ACTIVE_TOTAL`
`DCGM_FI_PROF_MMA_CYCLES_ACTIVE_TOTAL`	`counter`	`DCGM_FI_PROF_MMA_CYCLES_ACTIVE_TOTAL`
`DCGM_FI_PROF_DMMA_CYCLES_ACTIVE_TOTAL`	`counter`	`DCGM_FI_PROF_DMMA_CYCLES_ACTIVE_TOTAL`
`DCGM_FI_PROF_HMMA_CYCLES_ACTIVE_TOTAL`	`counter`	`DCGM_FI_PROF_HMMA_CYCLES_ACTIVE_TOTAL`
`DCGM_FI_PROF_IMMA_CYCLES_ACTIVE_TOTAL`	`counter`	`DCGM_FI_PROF_IMMA_CYCLES_ACTIVE_TOTAL`
`DCGM_FI_PROF_DFMA_CYCLES_ACTIVE_TOTAL`	`counter`	`DCGM_FI_PROF_DFMA_CYCLES_ACTIVE_TOTAL`
`DCGM_FI_PROF_PCIE_TX_BYTES_TOTAL`	`counter`	`DCGM_FI_PROF_PCIE_TX_BYTES_TOTAL`
`DCGM_FI_PROF_PCIE_RX_BYTES_TOTAL`	`counter`	`DCGM_FI_PROF_PCIE_RX_BYTES_TOTAL`
`DCGM_FI_PROF_INT_CYCLES_ACTIVE_TOTAL`	`counter`	`DCGM_FI_PROF_INT_CYCLES_ACTIVE_TOTAL`
`DCGM_FI_PROF_FP64_CYCLES_ACTIVE_TOTAL`	`counter`	`DCGM_FI_PROF_FP64_CYCLES_ACTIVE_TOTAL`
`DCGM_FI_PROF_FP32_CYCLES_ACTIVE_TOTAL`	`counter`	`DCGM_FI_PROF_FP32_CYCLES_ACTIVE_TOTAL`
`DCGM_FI_PROF_FP16_CYCLES_ACTIVE_TOTAL`	`counter`	`DCGM_FI_PROF_FP16_CYCLES_ACTIVE_TOTAL`

Custom Collector Fields#

A custom collector can select a field name recognized by the go-dcgm version built into DCGM Exporter, provided that the running DCGM recognizes the resolved field ID and the field is compatible with an entity that the exporter collects. Use DCGM Field Identifiers for upstream field definitions and availability requirements.

Each CSV row has exactly three fields: DCGM field name, Prometheus type, and help text. The help text becomes the Prometheus HELP description; it does not affect collection. Supported types are counter, gauge, untyped, and label.

Use gauge or untyped for general numeric fields and counter only for nonnegative cumulative values. A negative counter value is rejected. A string field configured as a metric cannot be rendered as a Prometheus scalar and should normally be configured as label. Integer, double, and string fields can supply label values. Timestamp and binary/blob field values are skipped.

A configured label is attached only when the field is compatible with the sample entity and DCGM returns a nonblank value. For the CSV format and loading options, see Select Metrics.

Exporter-Owned Metrics#

These DCGM_EXP_* metrics are calculated by DCGM Exporter from DCGM APIs or source fields. They are shipped as commented, opt-in rows in the default CSV.

Metric	Prometheus type	Source reference
`DCGM_EXP_CLOCK_EVENTS_COUNT`	`gauge`	`DCGM_FI_DEV_CLOCKS_EVENT_REASONS`
`DCGM_EXP_CLOCK_EVENTS_TOTAL`	`counter`	`DCGM_FI_DEV_CLOCKS_EVENT_REASONS`
`DCGM_EXP_XID_ERRORS_COUNT`	`gauge`	`DCGM_FI_DEV_XID_ERROR`; Xid errors
`DCGM_EXP_XID_ERRORS_TOTAL`	`counter`	`DCGM_FI_DEV_XID_ERROR`; Xid errors
`DCGM_EXP_GPU_HEALTH_STATUS`	`gauge`	DCGM health monitoring
`DCGM_EXP_P2P_STATUS`	`gauge`	DCGM topology; `DCGM_FI_DEV_NVLINK_P2P_STATUS`

Windowed sample counts#

DCGM_EXP_XID_ERRORS_COUNT counts XID samples returned by DCGM in the configured window and groups them by the numeric xid label. DCGM_EXP_CLOCK_EVENTS_COUNT counts samples in that window for which each clock-event bit is active and groups them by clock_event. These values count sample records, not unique incidents or clock-event transitions, so collection cadence affects the result.

Both families add window_size_in_ms. When DCGM returns no valid source sample for an entity, the exporter emits one zero-valued series with window_size_in_ms but without xid or clock_event. If a clock sample is present but has no recognized active reason, no clock-event series is emitted for that sample.

Cumulative totals#

DCGM_EXP_XID_ERRORS_TOTAL counts nonzero XID records observed after the collector starts and groups them by xid. DCGM_EXP_CLOCK_EVENTS_TOTAL counts inactive-to-active transitions for each clock_event. It primes the initial clock-event state without counting reasons that were already active.

A cumulative series is absent until its XID or clock event is observed. Totals reset when the exporter starts or the collectors are rebuilt by hot reload.

Clock-event label values#

The count and total clock-event families can use these clock_event values:

gpu_idle, clocks_setting, power_cap, hw_slowdown, sync_boost, sw_thermal, hw_thermal, hw_power_brake, display_clocks

GPU health status#

DCGM_EXP_GPU_HEALTH_STATUS emits one series for each exporter health watch on each monitored GPU. The metric value is the DCGM health result:

Value	Result
0	`PASS`
10	`WARN`
20	`FAIL`

Every series adds health_watch, health_error_code, health_error_severity, and health_error_category. A passing default uses DCGM_FR_OK with severity and category NONE. Unknown metadata is labeled UNKNOWN.

Health watch values:

PCIE, NVLINK, PMU, MCU, MEM, SM, INFOROM, THERMAL, POWER, DRIVER, ALL

Health severity values:

NONE, MONITOR, ISOLATE, UNKNOWN, TRIAGE, CONFIG, RESET

Health category values:

NONE, PERF_THRESHOLD, PERF_VIOLATION, SOFTWARE_CONFIG, SOFTWARE_LIBRARY, SOFTWARE_XID, SOFTWARE_CUDA, SOFTWARE_EUD, SOFTWARE_OTHER, HARDWARE_THERMAL, HARDWARE_MEMORY, HARDWARE_NVLINK, HARDWARE_NVSWITCH, HARDWARE_PCIE, HARDWARE_POWER, HARDWARE_OTHER, INTERNAL_OTHER, UNKNOWN

P2P status#

DCGM_EXP_P2P_STATUS emits the off-diagonal entries of the DCGM GPU P2P status matrix. Each series represents an ordered local-GPU and peer-GPU pair, not one physical NVLink. peer_gpu identifies the peer matrix index, and link_status provides the symbolic status.

Value	link_status
0	`OK`
1	`ChipsetNotSupported`
2	`GPUNotSupported`
3	`TopologyNotSupported`
4	`DisabledByRegKey`
5	`NotSupported`
6	`Unknown`

An unexpected numeric value is preserved as the metric value and uses link_status="Unknown".

Exporter Labels#

Entity labels#

Entity	Always present	Conditional
`GPU`	`gpu`, `UUID` or `uuid`, `pci_bus_id`, `device`, `modelName`	`GPU_I_PROFILE` and `GPU_I_ID` for MIG instances; `hostname` when enabled.
`NvLink`	`nvlink`	Parent labels: `nvswitch` or GPU labels `gpu`, `gpu_uuid`, `pci_bus_id`, `device`, and `model_name` when populated; MIG and hostname labels when populated.
`NvSwitch`	`nvswitch`	`hostname` when enabled.
`CPU`	`cpu`	`cpu_serial` when available; `hostname` when enabled.
`CPU core`	`cpucore`, `cpu`	`hostname` when enabled.

Configuration and mapping labels#

Source	Labels	Condition
UUID namespace	`UUID` by default; `uuid` with `--use-old-namespace`.	GPU entity samples.
Hostname	`hostname`	Added when hostname collection is enabled; omitted by `--no-hostname`.
CSV label rows	configured field name	Added only when DCGM returns a nonblank value for a compatible sample entity.
XID field	`err_code`, `err_msg`	Added to `DCGM_FI_DEV_XID_ERRORS` samples.
Kubernetes pod mapping	`pod`, `namespace`, `container`	Added by `--kubernetes` when pod-resources mapping succeeds.
Old Kubernetes names	`pod_name`, `pod_namespace`, `container_name`	Replace the current pod label names with `--use-old-namespace`.
Runtime container labels	`container`	Added in non-Kubernetes mode when `--container-labels` maps GPU or MIG assignments from the runtime socket; original unlabeled samples remain.
Pod UID	`pod_uid`	Added when pod UID collection is enabled.
Pod labels	`pod_label_*`	Added when pod label collection is enabled; colliding labels are prefixed.
Virtual GPUs	`vgpu`	Added for Kubernetes virtual GPU sharing paths.
DRA	`dra_claim_name`, `dra_claim_namespace`, `dra_driver_name`, `dra_pool_name`, `dra_device_name`, `dra_mig_profile`, `dra_mig_device_uuid`	Added when Kubernetes DRA mapping is enabled and available.
HPC job mapping	`hpc_job`	Added when `--hpc-job-mapping-dir` maps the GPU or MIG instance to one or more jobs.

Exporter-owned metric labels#

Metric	Labels	Presence
`DCGM_EXP_XID_ERRORS_COUNT`	`window_size_in_ms`; `xid`	The window label is always present. `xid` is omitted from the zero series when no XID sample is found.
`DCGM_EXP_CLOCK_EVENTS_COUNT`	`window_size_in_ms`; `clock_event`	The window label is always present. `clock_event` is omitted from the zero series when no active reason is found.
`DCGM_EXP_XID_ERRORS_TOTAL`	`xid`	A series and its label appear only after that XID is observed.
`DCGM_EXP_CLOCK_EVENTS_TOTAL`	`clock_event`	A series and its label appear only after that transition is observed.
`DCGM_EXP_GPU_HEALTH_STATUS`	`health_watch`, `health_error_code`, `health_error_severity`, `health_error_category`	Present on every emitted health series.
`DCGM_EXP_P2P_STATUS`	`peer_gpu`, `link_status`	Present on every emitted ordered GPU-pair series.

Runtime Considerations#

Profiling fields#

At startup, DCGM Exporter queries the supported profiling metric groups using GPU 0 and applies that capability set when it loads configured DCGM_FI_PROF_* fields. Unsupported profiling rows are skipped. The DCGM profiling documentation describes hardware support, concurrent metric-group limits, and blank values.

MIG and entity selection#

In MIG mode, exporter device selectors can choose full GPUs, GPU instances, or both. MIG instance samples add GPU_I_PROFILE and GPU_I_ID. Which fields DCGM supports on a GPU, GPU instance, compute instance, NvSwitch, NvLink, CPU, or CPU core is defined by DCGM rather than this inventory. See Metrics on Multi-Instance GPU and Fields and Watches.

Kubernetes, DRA, vGPU, runtime-container, and HPC mapping add labels; they do not make an unsupported DCGM field available. YAML watch groups change polling cadence for matching fields without changing field support.

DCGM connection and version#

Runtime results also depend on the entities and permissions visible through the DCGM host engine. Run DCGM Exporter with the DCGM version paired with that exporter release; mismatched combinations are untested and unsupported.