dcgm-exporter#

NAME#

dcgm-exporter - expose NVIDIA DCGM telemetry in Prometheus format

SYNOPSIS#

dcgm-exporter [<options>]
dcgm-exporter --help
dcgm-exporter --version

DESCRIPTION#

dcgm-exporter selects DCGM fields, watches them for the requested entities, adds entity and optional workload labels, and serves the resulting Prometheus metrics at /metrics. The default listener is :9400.

DCGM Exporter is supported on Linux.

Without --remote-hostengine-info, the exporter initializes an embedded DCGM host engine inside its process through libdcgm. Supplying that option connects to a separately managed DCGM host engine instead.

This page describes the executable contract. For package, container, Helm, and GPU Operator procedures, see Install DCGM Exporter. To configure Prometheus, see Configure Prometheus for DCGM Exporter. For metric families and labels, see DCGM Exporter Metrics.

CONFIGURATION SOURCES#

The effective precedence, from highest to lowest, is:

Command-line flags explicitly supplied at startup.
Environment variables associated with those flags.
Values loaded through --config-file or DCGM_EXPORTER_CONFIG_FILE.
Built-in defaults.

The startup YAML schema configures only the metric source, collection interval, and watch groups. Explicit --collectors or --configmap-data settings replace a YAML metric source. If both are explicitly set, --configmap-data is applied last. The value none selects the collector file instead of a ConfigMap.

YAML is read at startup. A file-backed collector CSV is watched for changes; SIGHUP requests the same metric-source reload. Other flags and environment variables require a process restart.

OPTIONS#

General Settings#

General dcgm-exporter settings#
Option	Environment variable	Default	Behavior
`-f`, `--collectors <path>`	`DCGM_EXPORTER_COLLECTORS`	`/etc/dcgm-exporter/default-counters.csv`	Select a three-column collector CSV.
`--config-file <path>`	`DCGM_EXPORTER_CONFIG_FILE`	Unset	Load the version 1 YAML startup configuration.
`-a`, `--address <host:port>`	`DCGM_EXPORTER_LISTEN`	`:9400`	Set the HTTP listen address. Bracket an IPv6 address.
`-c`, `--collect-interval <milliseconds>`	`DCGM_EXPORTER_INTERVAL`	`30000`	Set the default DCGM collection interval.
`-r`, `--remote-hostengine-info <endpoint>`	`DCGM_REMOTE_HOSTENGINE_INFO`	Embedded mode	Connect to a separate DCGM host engine. The option’s displayed value is `localhost:5555`, but remote mode is enabled only when the option or environment variable is explicitly set.
`-n`, `--no-hostname`	`DCGM_EXPORTER_NO_HOSTNAME`	`false`	Omit the `hostname` label.
`--replace-blanks-in-model-name`, `--rbmn`	`DCGM_EXPORTER_REPLACE_BLANKS_IN_MODEL_NAME`	`false`	Replace spaces in `modelName` label values with hyphens.
`--disable-startup-validate`	`DISABLE_STARTUP_VALIDATE`	`false`	Skip startup prerequisite checks. Use only in controlled tests.
`--web-read-timeout <duration>`	`DCGM_EXPORTER_WEB_READ_TIMEOUT`	`10s`	Limit the time used to read an HTTP request.
`--web-write-timeout <duration>`	`DCGM_EXPORTER_WEB_WRITE_TIMEOUT`	`30s`	Limit metric generation and response writing.

Device and Entity Selection#

Device and entity settings#
Option	Environment variable	Default	Behavior
`-d`, `--devices <selector>`	`DCGM_EXPORTER_DEVICES_STR`	`f`	Select GPUs and GPU instances.
`-s`, `--switch-devices <selector>`	`DCGM_EXPORTER_OTHER_DEVICES_STR`	`f`	Select NvSwitches and NvLinks.
`-p`, `--cpu-devices <selector>`	`DCGM_EXPORTER_CPU_DEVICES_STR`	`f`	Select CPUs and CPU cores.
`--fake-gpus`	`DCGM_EXPORTER_USE_FAKE_GPUS`	`false`	Accept fake GPU entities for tests.

Kubernetes Settings#

Kubernetes settings#
Option	Environment variable	Default	Behavior
`-k`, `--kubernetes`	`DCGM_EXPORTER_KUBERNETES`	`false`	Map metrics to Kubernetes pods.
`-o`, `--use-old-namespace`	`DCGM_EXPORTER_USE_OLD_NAMESPACE`	`false`	Use legacy 1.x pod label names when Kubernetes mapping is enabled.
`-m`, `--configmap-data <namespace:name>`	`DCGM_EXPORTER_CONFIGMAP_DATA`	`none`	Read the `metrics` key from a ConfigMap.
`--pod-resources-kubelet-socket <path>`	`DCGM_POD_RESOURCES_KUBELET_SOCKET`	`/var/lib/kubelet/pod-resources/kubelet.sock`	Set the kubelet pod-resources socket.
`--kubernetes-enable-pod-labels`	`DCGM_EXPORTER_KUBERNETES_ENABLE_POD_LABELS`	`false`	Add pod labels when Kubernetes mapping is enabled.
`--kubernetes-enable-pod-uid`	`DCGM_EXPORTER_KUBERNETES_ENABLE_POD_UID`	`false`	Add `pod_uid` when Kubernetes mapping is enabled.
`--kubernetes-gpu-id-type <type>`	`DCGM_EXPORTER_KUBERNETES_GPU_ID_TYPE`	`uid`	Match pod resources by `uid` or `device-name` when Kubernetes mapping is enabled.
`--kubernetes-pod-label-allowlist-regex <patterns>`	`DCGM_EXPORTER_KUBERNETES_POD_LABEL_ALLOWLIST_REGEX`	Empty	Filter added pod labels with comma-separated regular expressions. An empty value includes all labels when pod-label collection is enabled.
`--nvidia-resource-names <names>`	`NVIDIA_RESOURCE_NAMES`	Empty	Recognize additional comma-separated NVIDIA resource names during Kubernetes pod mapping.
`--kubernetes-virtual-gpus`	`KUBERNETES_VIRTUAL_GPUS`	`false`	Attribute supported time-sharing or MPS assignments in Kubernetes mode.
`--kubernetes-enable-dra`	`KUBERNETES_ENABLE_DRA`	`false`	Add metadata from NVIDIA GPU DRA ResourceSlices in Kubernetes mode.

Runtime and HPC Labels#

Runtime and HPC label settings#
Option	Environment variable	Default	Behavior
`--container-labels`	`DCGM_EXPORTER_CONTAINER_LABELS`	`false`	Add non-Kubernetes container labels. Requires a runtime socket and is ignored in Kubernetes mode.
`--container-runtime-socket <path>`	`DCGM_CONTAINER_RUNTIME_SOCKET`	Unset	Set the Docker-compatible runtime socket used for container mapping.
`--hpc-job-mapping-dir <path>`	`DCGM_HPC_JOB_MAPPING_DIR`	Unset	Read scheduler-created GPU-to-job mapping files.

Web, Logging, and Diagnostics#

Web, logging, and diagnostic settings#
Option	Environment variable	Default	Behavior
`--web-config-file <path>`	`DCGM_EXPORTER_WEB_CONFIG_FILE`	Unset	Load a Prometheus exporter-toolkit TLS and authentication file.
`--web-systemd-socket`	`DCGM_EXPORTER_SYSTEMD_SOCKET`	`false`	On Linux, use listeners inherited through systemd socket activation instead of `--address`.
`--enable-pprof`	`DCGM_EXPORTER_ENABLE_PPROF`	`false`	Expose `/debug/pprof/`. Requires `--web-config-file`.
`--debug`	`DCGM_EXPORTER_DEBUG`	`false`	Enable debug-level exporter logging.
`--log-format <format>`	`DCGM_EXPORTER_LOG_FORMAT`	`text`	Use `text` or `json` exporter logs.
`--enable-dcgm-log`	`DCGM_EXPORTER_ENABLE_DCGM_LOG`	`false`	Write DCGM logs to standard output.
`--dcgm-log-level <level>`	`DCGM_EXPORTER_DCGM_LOG_LEVEL`	`NONE`	Set `NONE`, `FATAL`, `ERROR`, `WARN`, `INFO`, `DEBUG`, or `VERB`. The value is validated even when DCGM logging is disabled.
`-x`, `--xid-count-window-size <milliseconds>`	`DCGM_EXPORTER_XID_COUNT_WINDOW_SIZE`	`300000`	Set the window used by `DCGM_EXP_XID_ERRORS_COUNT`.
`--clock-events-count-window-size <milliseconds>`	`DCGM_EXPORTER_CLOCK_EVENTS_COUNT_WINDOW_SIZE`	`300000`	Set the window used by `DCGM_EXP_CLOCK_EVENTS_COUNT`.
`--dump-enabled`	`DCGM_EXPORTER_DUMP_ENABLED`	`false`	Write file-based diagnostic dumps.
`--dump-directory <path>`	`DCGM_EXPORTER_DUMP_DIRECTORY`	`/tmp/dcgm-exporter-debug`	Select the debug-dump directory.
`--dump-retention <hours>`	`DCGM_EXPORTER_DUMP_RETENTION`	`24`	Set dump retention; `0` disables cleanup.
`--dump-compression[=<boolean>]`	`DCGM_EXPORTER_DUMP_COMPRESSION`	`true`	Enable gzip compression. Use `--dump-compression=false` to disable it.
`--enable-gpu-bind-unbind-watch`	`DCGM_EXPORTER_ENABLE_GPU_BIND_UNBIND_WATCH`	`false`	Rebuild exporter topology after GPU bind or unbind. Requires DCGM 4.5 or later.
`--gpu-bind-unbind-poll-interval <duration>`	`DCGM_EXPORTER_GPU_BIND_UNBIND_POLL_INTERVAL`	`1s`	Set the topology polling interval. A non-positive value is rejected when bind/unbind watching is enabled.

ENDPOINTS AND SELECTORS#

DCGM Host Engine Endpoints#

--remote-hostengine-info accepts:

host:port
tcp://host:port
unix:///path/to/socket
vsock://cid:port
[ipv6-address]:port

The URI forms require DCGM 4.5 or later. The exporter DCGM client library must be greater than or equal to the separate DCGM host engine version. Supported deployments still use the paired releases described in the version pairing guidance.

Device Selectors#

Selectors have one of these forms:

f
g[:id1[,-id2...]]
i[:id1[,-id2...]]
a g selector and an i selector, in either order, joined with +

f selects each full GPU when MIG is disabled and each GPU instance when MIG is enabled. g selects top-level entities and i selects sub-level entities. Examples include g:0,2-4, i:0,2-4, g+i, g:0,2-4+i, and g+i:7.

f cannot be combined with g or i. GPU i selection requires MIG mode. Explicit IDs must exist, ranges must increase, and an expanded selector cannot exceed the applicable DCGM entity limit. i+g and g+i are equivalent.

YAML CONFIGURATION#

The only supported top-level schema version is 1:

version: 1
metrics:
  file: /etc/dcgm-exporter/default-counters.csv
collection:
  interval: 30s
  watchGroups:
    - name: fast-thermals
      interval: 5s
      fields:
        - DCGM_FI_DEV_GPU_TEMP
        - DCGM_FI_DEV_POWER_USAGE

metrics must specify exactly one of:

file: A collector CSV path.
fields: A list of inline entries containing non-empty name, prometheusType, and help values. Field names must be unique.

collection.interval and watch-group intervals use positive Go duration strings expressed in whole milliseconds or larger. Watch-group names must be non-empty and unique, each group must contain at least one non-empty glob-style field pattern, each pattern must match a configured field, and one field cannot match multiple groups.

The parser rejects unknown YAML keys, duplicate YAML keys, multiple YAML documents, unsupported schema versions, and ambiguous metric sources.

FILES#

/etc/dcgm-exporter/default-counters.csv: Default collector installed by the package and included in the container.
/etc/dcgm-exporter/dcp-metrics-included.csv: Collector preset shipped in the container that includes supported profiling fields.
/etc/dcgm-exporter/1.x-compatibility-metrics.csv: Collector preset shipped in the container for legacy metric names and metadata.
Collector CSV and ConfigMap data: A collector CSV contains exactly three comma-separated columns: a DCGM field name, one of counter, gauge, untyped, or label, and a help message. Lines beginning with # are comments. A ConfigMap selected with --configmap-data must use the key metrics and the same CSV format. See Select Metrics and DCGM Exporter Metrics for the field contract.
Kubelet pod-resources socket: --pod-resources-kubelet-socket names the Unix socket used for Kubernetes pod attribution. The exporter process must be able to access the mounted socket.
Container runtime socket: --container-runtime-socket names a Unix socket that implements the Docker-compatible container API. Non-Kubernetes container mapping requires this socket. Mounting it gives the exporter access to privileged runtime metadata; protect it accordingly.
HPC job mapping directory: Each regular file is named <gpu-id> or <gpu-id>.<gpu-instance-id>. Each line contains one job ID. Matching samples gain an hpc_job label and are duplicated when a file contains multiple job IDs. Other filenames and subdirectories are ignored.
Debug dump directory: With --dump-enabled, the exporter writes JSON snapshots of runtime objects. Filenames include the object type, timestamp, and a random suffix; compression adds .json.gz instead of .json. The retention task removes files older than --dump-retention hours. A value of 0 disables cleanup.

The web configuration follows the Prometheus exporter-toolkit schema. It applies TLS or authentication to the root, health, metrics, and enabled pprof endpoints. Debug dumps can contain device and workload metadata; protect their directory according to site retention policy.

On Linux, --web-systemd-socket consumes listeners supplied through the systemd socket-activation protocol. The packaged service does not enable this option; an administrator must provide the matching socket unit and service override. When enabled, inherited listeners replace --address.

SIGNALS AND LIFETIME#

SIGHUP: Reload the effective metric source through the same hot-reload path used by collector file changes. YAML, flags, and environment variables are rebuilt from their startup values; changing YAML on disk does not load a new YAML file without a process restart.
SIGINT, SIGTERM, SIGQUIT: Shut down the exporter.

A metric reload rebuilds watches and the exporter registry. Exporter-owned cumulative counters are process memory and reset on reload or restart. During a hot reload, health responses remain available and report X-Reload-In-Progress: true.

FAILURES#

Startup fails for malformed device selectors or ConfigMap identifiers, unsupported YAML, invalid exporter log formats or DCGM log levels, an enabled non-positive GPU polling interval, pprof without a web configuration, or container labels without a runtime socket outside Kubernetes.

Invalid duration strings log a warning and use the corresponding default. Non-positive HTTP timeouts also use their defaults. Collector CSV and watch-group errors are reported when the metric source and DCGM field groups are built. --disable-startup-validate skips prerequisite checks; it does not make an invalid runtime configuration valid.

EXAMPLES#

Connect to a separate DCGM host engine and use a custom collector:

$ dcgm-exporter \
    --remote-hostengine-info tcp://host123:5555 \
    --collectors /opt/dcgm-exporter/custom.csv

Listen on IPv6 and select full GPUs plus GPU instances:

$ dcgm-exporter --address "[::]:9400" --devices g+i

Enable protected profiling endpoints:

$ dcgm-exporter \
    --web-config-file /etc/dcgm-exporter/web-config.yaml \
    --enable-pprof