dcgm-exporter#

NAME#

dcgm-exporter - expose NVIDIA DCGM telemetry in Prometheus format

SYNOPSIS#

dcgm-exporter [<options>]
dcgm-exporter --help
dcgm-exporter --version

DESCRIPTION#

dcgm-exporter selects DCGM fields, watches them for the requested entities, adds entity and optional workload labels, and serves the resulting Prometheus metrics at /metrics. The default listener is :9400.

DCGM Exporter is supported on Linux.

Without --remote-hostengine-info, the exporter initializes an embedded DCGM host engine inside its process through libdcgm. Supplying that option connects to a separately managed DCGM host engine instead.

This page describes the program’s behavior and options. For package, container, Helm, and GPU Operator procedures, see Install DCGM Exporter. To configure Prometheus, see Configure Prometheus for DCGM Exporter. For metric families and labels, see DCGM Exporter Metrics.

CONFIGURATION SOURCES#

The setting with the highest priority wins. From highest to lowest:

Command-line flags explicitly supplied at startup.
Environment variables associated with those flags.
Values loaded through --config-file or DCGM_EXPORTER_CONFIG_FILE.
Built-in defaults.

The startup YAML configures only the metric source and collections. Explicit --collectors or --configmap-data settings replace a YAML metric source. If both are explicitly set, --configmap-data is applied last. The value none selects the collector file instead of a ConfigMap.

YAML is read at startup. The exporter watches a file-backed collector CSV for changes; SIGHUP requests the same metric reload. Other flags and environment variables require a process restart.

OPTIONS#

General Settings#

General dcgm-exporter settings#
Option	Environment variable	Default	Behavior
`-f`, `--collectors <path>`	`DCGM_EXPORTER_COLLECTORS`	`/etc/dcgm-exporter/default-counters.csv`	Select a three-column collector CSV.
`--config-file <path>`	`DCGM_EXPORTER_CONFIG_FILE`	Unset	Load a YAML startup configuration.
`-a`, `--address <host:port>`	`DCGM_EXPORTER_LISTEN`	`:9400`	Set the HTTP listen address. Bracket an IPv6 address.
`-c`, `--collect-interval <milliseconds>`	`DCGM_EXPORTER_INTERVAL`	`30000`	Set the default DCGM collection interval.
`-r`, `--remote-hostengine-info <endpoint>`	`DCGM_REMOTE_HOSTENGINE_INFO`	Embedded mode	Connect to a separate DCGM host engine. The option’s displayed value is `localhost:5555`, but remote mode starts only when you set the option or environment variable.
`-n`, `--no-hostname`	`DCGM_EXPORTER_NO_HOSTNAME`	`false`	Omit the `hostname` label.
`--replace-blanks-in-model-name`, `--rbmn`	`DCGM_EXPORTER_REPLACE_BLANKS_IN_MODEL_NAME`	`false`	Replace spaces in `modelName` label values with hyphens.
`--disable-startup-validate`	`DISABLE_STARTUP_VALIDATE`	`false`	Skip startup prerequisite checks. Use only in controlled tests.
`--web-read-timeout <duration>`	`DCGM_EXPORTER_WEB_READ_TIMEOUT`	`10s`	Limit the time used to read an HTTP request.
`--web-write-timeout <duration>`	`DCGM_EXPORTER_WEB_WRITE_TIMEOUT`	`30s`	Limit metric generation and response writing.

Device and Entity Selection#

Device and entity settings#
Option	Environment variable	Default	Behavior
`-d`, `--devices <selector>`	`DCGM_EXPORTER_DEVICES_STR`	`f`	Select GPUs and GPU instances.
`-s`, `--switch-devices <selector>`	`DCGM_EXPORTER_OTHER_DEVICES_STR`	`f`	Select NvSwitches and NvLinks.
`-p`, `--cpu-devices <selector>`	`DCGM_EXPORTER_CPU_DEVICES_STR`	`f`	Select CPUs and CPU cores.
`--fake-gpus`	`DCGM_EXPORTER_USE_FAKE_GPUS`	`false`	Accept fake GPU entities for tests.

Kubernetes Settings#

Kubernetes settings#
Option	Environment variable	Default	Behavior
`-k`, `--kubernetes`	`DCGM_EXPORTER_KUBERNETES`	`false`	Map metrics to Kubernetes pods.
`-o`, `--use-old-namespace`	`DCGM_EXPORTER_USE_OLD_NAMESPACE`	`false`	Use legacy 1.x pod label names when Kubernetes mapping is enabled.
`-m`, `--configmap-data <namespace:name>`	`DCGM_EXPORTER_CONFIGMAP_DATA`	`none`	Read the `metrics` key from a ConfigMap.
`--pod-resources-kubelet-socket <path>`	`DCGM_POD_RESOURCES_KUBELET_SOCKET`	`/var/lib/kubelet/pod-resources/kubelet.sock`	Set the kubelet pod-resources socket.
`--kubernetes-enable-pod-labels`	`DCGM_EXPORTER_KUBERNETES_ENABLE_POD_LABELS`	`false`	Add pod labels when Kubernetes mapping is enabled.
`--kubernetes-enable-pod-uid`	`DCGM_EXPORTER_KUBERNETES_ENABLE_POD_UID`	`false`	Add `pod_uid` when Kubernetes mapping is enabled.
`--kubernetes-gpu-id-type <type>`	`DCGM_EXPORTER_KUBERNETES_GPU_ID_TYPE`	`uid`	Match pod resources by `uid` or `device-name` when Kubernetes mapping is enabled.
`--kubernetes-pod-label-allowlist-regex <patterns>`	`DCGM_EXPORTER_KUBERNETES_POD_LABEL_ALLOWLIST_REGEX`	Empty	Filter added pod labels with comma-separated regular expressions. An empty value includes all labels when pod-label collection is enabled.
`--nvidia-resource-names <names>`	`NVIDIA_RESOURCE_NAMES`	Empty	Recognize additional comma-separated NVIDIA resource names during Kubernetes pod mapping.
`--kubernetes-virtual-gpus`	`KUBERNETES_VIRTUAL_GPUS`	`false`	Attribute supported time-sharing or MPS assignments in Kubernetes mode.
`--kubernetes-enable-dra`	`KUBERNETES_ENABLE_DRA`	`false`	Add metadata from NVIDIA GPU DRA ResourceSlices in Kubernetes mode.

Runtime and HPC Labels#

Runtime and HPC label settings#
Option	Environment variable	Default	Behavior
`--container-labels`	`DCGM_EXPORTER_CONTAINER_LABELS`	`false`	Add non-Kubernetes container labels. Requires a runtime socket and is ignored in Kubernetes mode.
`--container-runtime-socket <path>`	`DCGM_CONTAINER_RUNTIME_SOCKET`	Unset	Set the Docker-compatible runtime socket used for container mapping.
`--hpc-job-mapping-dir <path>`	`DCGM_HPC_JOB_MAPPING_DIR`	Unset	Read scheduler-created GPU-to-job mapping files.

Web, Logging, and Diagnostics#

Web, logging, and diagnostic settings#
Option	Environment variable	Default	Behavior
`--web-config-file <path>`	`DCGM_EXPORTER_WEB_CONFIG_FILE`	Unset	Load a Prometheus exporter-toolkit TLS and authentication file.
`--web-systemd-socket`	`DCGM_EXPORTER_SYSTEMD_SOCKET`	`false`	On Linux, use listeners inherited through systemd socket activation instead of `--address`.
`--enable-pprof`	`DCGM_EXPORTER_ENABLE_PPROF`	`false`	Expose `/debug/pprof/`. Requires `--web-config-file`.
`--debug`	`DCGM_EXPORTER_DEBUG`	`false`	Enable debug-level exporter logging.
`--log-format <format>`	`DCGM_EXPORTER_LOG_FORMAT`	`text`	Use `text` or `json` exporter logs.
`--enable-dcgm-log`	`DCGM_EXPORTER_ENABLE_DCGM_LOG`	`false`	Write DCGM logs to standard output.
`--dcgm-log-level <level>`	`DCGM_EXPORTER_DCGM_LOG_LEVEL`	`NONE`	Set `NONE`, `FATAL`, `ERROR`, `WARN`, `INFO`, `DEBUG`, or `VERB`. The value is validated even when DCGM logging is disabled.
`-x`, `--xid-count-window-size <milliseconds>`	`DCGM_EXPORTER_XID_COUNT_WINDOW_SIZE`	`300000`	Set the window used by `DCGM_EXP_XID_ERRORS_COUNT`.
`--clock-events-count-window-size <milliseconds>`	`DCGM_EXPORTER_CLOCK_EVENTS_COUNT_WINDOW_SIZE`	`300000`	Set the window used by `DCGM_EXP_CLOCK_EVENTS_COUNT`.
`--dump-enabled`	`DCGM_EXPORTER_DUMP_ENABLED`	`false`	Write file-based diagnostic dumps.
`--dump-directory <path>`	`DCGM_EXPORTER_DUMP_DIRECTORY`	`/tmp/dcgm-exporter-debug`	Select the debug-dump directory.
`--dump-retention <hours>`	`DCGM_EXPORTER_DUMP_RETENTION`	`24`	Set dump retention; `0` disables cleanup.
`--dump-compression[=<boolean>]`	`DCGM_EXPORTER_DUMP_COMPRESSION`	`true`	Enable gzip compression. Use `--dump-compression=false` to disable it.
`--enable-gpu-bind-unbind-watch`	`DCGM_EXPORTER_ENABLE_GPU_BIND_UNBIND_WATCH`	`false`	Rebuild exporter topology after GPU bind or unbind. Requires DCGM 4.5 or later.
`--gpu-bind-unbind-poll-interval <duration>`	`DCGM_EXPORTER_GPU_BIND_UNBIND_POLL_INTERVAL`	`1s`	Set the topology polling interval. A non-positive value is rejected when bind/unbind watching is enabled.

ENDPOINTS AND SELECTORS#

DCGM Host Engine Endpoints#

--remote-hostengine-info accepts:

host:port
tcp://host:port
unix:///path/to/socket
vsock://cid:port
[ipv6-address]:port

The URI forms require DCGM 4.5 or later. The exporter DCGM client library must be the same version as, or newer than, the separate DCGM host engine to connect. Use the tested versions described in the version pairing guidance.

Device Selectors#

Selectors have one of these forms:

f
g[:id1[,-id2...]]
i[:id1[,-id2...]]
a g selector and an i selector, in either order, joined with +

f selects each full GPU when MIG is disabled and each GPU instance when MIG is enabled. g selects top-level entities and i selects sub-level entities. Examples include g:0,2-4, i:0,2-4, g+i, g:0,2-4+i, and g+i:7.

f cannot be combined with g or i. GPU i selection requires MIG mode. Explicit IDs must exist, ranges must increase, and an expanded selector cannot exceed the applicable DCGM entity limit. i+g and g+i are equivalent.

YAML CONFIGURATION#

Use schema version 2 for YAML startup configurations:

version: 2
metrics:
  file: /etc/dcgm-exporter/default-counters.csv
collections:
  - name: scrape
    every: 30s
    metrics:
      include: ["*"]
  - name: fast-thermals
    every: 5s
    metrics:
      include:
        - DCGM_FI_DEV_GPU_TEMP_CELSIUS
        - DCGM_FI_DEV_BOARD_POWER_WATTS

Use metrics.file to select a collector CSV or metrics.fields to define fields inline. Omit metrics to use the default collector. Do not specify both a file and inline fields.

Each collections entry requires a unique name, a positive every duration, and a metrics.include list. The scrape collection normally includes "*" to set the default cadence. More specific collections can override that cadence for selected field patterns.

An inline field entry requires non-empty name, prometheusType, and help values. Field names must be unique. Collection durations use positive Go duration strings expressed in whole milliseconds or larger. Each field pattern must match a configured field, and two narrower collections cannot match the same field.

The parser rejects unknown YAML keys, duplicate YAML keys, multiple YAML documents, unsupported schema versions, and ambiguous metric sources.

FILES#

/etc/dcgm-exporter/default-counters.csv: Default collector installed by the package and included in the container.
/etc/dcgm-exporter/dcp-metrics-included.csv: Collector preset installed with the package and included in the container. It includes supported profiling fields.
/etc/dcgm-exporter/1.x-compatibility-metrics.csv: Collector preset installed with the package and included in the container for legacy metric names and metadata.
Collector CSV and ConfigMap data: A collector CSV contains exactly three comma-separated columns: a DCGM field name, one of counter, gauge, untyped, or label, and a help message. Lines beginning with # are comments. A ConfigMap selected with --configmap-data must use the key metrics and the same CSV format. See Select Metrics and DCGM Exporter Metrics for the field format rules.
Kubelet pod-resources socket: --pod-resources-kubelet-socket names the Unix socket used for Kubernetes pod attribution. The exporter process must be able to access the mounted socket.
Container runtime socket: --container-runtime-socket names a Unix socket that implements the Docker-compatible container API. Non-Kubernetes container mapping requires this socket. Mounting it gives the exporter access to privileged runtime metadata; protect it accordingly.
HPC job mapping directory: Each regular file is named <gpu-id> or <gpu-id>.<gpu-instance-id>. Each line contains one job ID. Matching samples gain an hpc_job label and are duplicated when a file contains multiple job IDs. Other filenames and subdirectories are ignored.
Debug dump directory: With --dump-enabled, the exporter writes JSON snapshots of runtime objects. Filenames include the object type, timestamp, and a random suffix; compression adds .json.gz instead of .json. The retention task removes files older than --dump-retention hours. A value of 0 disables cleanup.

The web configuration follows the Prometheus exporter-toolkit schema. It applies TLS or authentication to the root, health, metrics, and enabled pprof endpoints. Debug dumps can contain device and workload metadata; protect their directory as required by your retention policy.

On Linux, --web-systemd-socket uses listeners passed by the systemd socket-activation protocol. The packaged service does not enable this option; an administrator must provide the matching socket unit and service override. When enabled, those listeners replace --address.

SIGNALS AND LIFETIME#

SIGHUP: Reload the metric source, just as when a collector file changes. The exporter reuses the YAML, flag, and environment values from startup; it does not reread a changed YAML file without a process restart.
SIGINT, SIGTERM, SIGQUIT: Shut down the exporter.

A metric reload rebuilds watches and the exporter registry. Exporter-owned cumulative counters are stored only in process memory and reset on reload or restart. During a hot reload, health responses remain available and report X-Reload-In-Progress: true.

FAILURES#

Startup fails for malformed device selectors or ConfigMap identifiers, unsupported YAML, invalid exporter log formats or DCGM log levels, an enabled non-positive GPU polling interval, pprof without a web configuration, or container labels without a runtime socket outside Kubernetes.

Invalid duration strings log a warning and use the corresponding default. Non-positive HTTP timeouts also use their defaults. Collector CSV and collection errors are reported when the metric source and DCGM field groups are built. --disable-startup-validate skips prerequisite checks; it does not make an invalid runtime configuration valid.

EXAMPLES#

Connect to a separate DCGM host engine and use a custom collector:

$ dcgm-exporter \
    --remote-hostengine-info tcp://host123:5555 \
    --collectors /opt/dcgm-exporter/custom.csv

Listen on IPv6 and select full GPUs plus GPU instances:

$ dcgm-exporter --address "[::]:9400" --devices g+i

Enable protected profiling endpoints:

$ dcgm-exporter \
    --web-config-file /etc/dcgm-exporter/web-config.yaml \
    --enable-pprof