Install DCGM Exporter

DCGM Exporter converts selected DCGM telemetry fields to the Prometheus exposition format. Install one exporter on each GPU node that Prometheus should monitor.

DCGM Exporter can run as a package-managed systemd service, an OCI container, or a Kubernetes DaemonSet managed by Helm or the NVIDIA GPU Operator. By default, DCGM Exporter initializes an embedded DCGM host engine in its own process using libdcgm. It can also connect to a separately managed DCGM host engine over TCP, a Unix socket, or VSOCK.

Requirements and Version Compatibility

Before installation, verify that the target system has:

• An NVIDIA GPU, driver, and DCGM combination listed in the DCGM product support matrix.

• A runtime that can access the NVIDIA GPUs. Container deployments require the NVIDIA Container Toolkit.

• Network access from Prometheus to the exporter HTTP endpoint when Prometheus runs on another host or cluster.

See Install DCGM for DCGM package installation, privileges, and verification.

DCGM and DCGM Exporter use separate version numbers. Each DCGM Exporter release note lists the DCGM version tested with that exporter release. DCGM Exporter container images record their bundled DCGM version in the com.nvidia.dcgm.version label. To check that label on a running Docker container, inspect its metadata from the container host:

$ docker container inspect --format '{{ index .Config.Labels "com.nvidia.dcgm.version" }}' <container-name>
<dcgm-version>

RPM and Debian package versions, and dcgm-exporter --version, identify the DCGM Exporter version. The release notes identify the tested DCGM version.

Important

Use the tested DCGM and DCGM Exporter version pair listed in the release notes. This is the supported combination.

If the exporter connects to a separate DCGM host engine, its DCGM client library must be the same version as, or newer than, the host engine to connect. This technical requirement does not make another version pair supported.

When troubleshooting, first update to the tested, current versions of DCGM and DCGM Exporter.

Container Images

DCGM Exporter container images are available from NGC and Docker Hub:

• nvcr.io/nvidia/k8s/dcgm-exporter:<release-tag>

• nvidia/dcgm-exporter:<release-tag>

Use the release tag from the DCGM Exporter release notes when you need a fixed version instead of latest.

Choose an Installation Method

DCGM Exporter installation methods

Method	Use it when	Lifecycle owner
Package	You manage DCGM and the exporter as services on individual hosts.	The operating system package manager and systemd.
Container	You manage a standalone container on each GPU host.	Your container runtime or service manager.
Helm	You manage the exporter directly in a Kubernetes cluster.	The DCGM Exporter Helm release.
GPU Operator	NVIDIA GPU Operator already manages the cluster’s GPU software stack.	The NVIDIA GPU Operator.

Install the Package

DCGM Exporter is distributed as the datacenter-gpu-manager-exporter package. It installs:

• /usr/bin/dcgm-exporter

• /etc/dcgm-exporter/default-counters.csv

• /etc/dcgm-exporter/dcp-metrics-included.csv

• /etc/dcgm-exporter/1.x-compatibility-metrics.csv

• /lib/systemd/system/nvidia-dcgm-exporter.service

Configure the NVIDIA CUDA network repository by following the package repository guidance, then install or update DCGM before installing the exporter.

To install the DCGM version tested with a specific exporter release, pin the DCGM runtime package. For example, DCGM Exporter 4.8.4 was tested with DCGM 4.6.1:

$ CUDA_VERSION=<CUDA user-mode driver major version>
$ sudo apt-get install --yes --install-recommends \
    "datacenter-gpu-manager-4-cuda${CUDA_VERSION}=1:4.6.1-1"

$ CUDA_VERSION=<CUDA user-mode driver major version>
$ sudo dnf install --assumeyes --setopt=install_weak_deps=True \
    datacenter-gpu-manager-4-cuda${CUDA_VERSION}-4.6.1-1

Use apt-cache madison or dnf --showduplicates list to find the exact package version available for another tested DCGM version.

On Ubuntu or Debian:

$ sudo apt-get install datacenter-gpu-manager-exporter

On Red Hat Enterprise Linux or a compatible distribution:

$ sudo dnf install datacenter-gpu-manager-exporter

Start the standalone DCGM host engine and the exporter:

$ sudo systemctl enable --now nvidia-dcgm
$ sudo systemctl enable --now nvidia-dcgm-exporter

Configure the systemd service

The package unit starts after nvidia-dcgm.service, starts the exporter in embedded mode with the default collector CSV, and restarts after a failure. Inspect the effective unit, including administrator overrides:

$ systemctl cat nvidia-dcgm-exporter.service

Do not edit the vendor unit under /lib because a package upgrade can replace it. Create an administrator drop-in instead:

$ sudo systemctl edit nvidia-dcgm-exporter.service

For example, replace the packaged command to connect to a separate DCGM host engine:

[Service]
ExecStart=
ExecStart=/usr/bin/dcgm-exporter \
    -f /etc/dcgm-exporter/default-counters.csv \
    -r tcp://host123:5555

The empty ExecStart= clears the vendor command before the replacement is defined. Restart and inspect the service:

$ sudo systemctl restart nvidia-dcgm-exporter.service
$ systemctl status nvidia-dcgm-exporter.service --no-pager
$ journalctl --unit nvidia-dcgm-exporter.service --since today

After an exporter package upgrade, compare a replaced ExecStart with the updated vendor command because the drop-in does not inherit vendor command changes. See systemd unit layering for drop-in precedence and directive behavior.

Run the Container

Choose one of these tag forms from the DCGM Exporter container catalog before you upgrade:

• latest follows the most recent DCGM Exporter release.

• <exporter-version> pins a DCGM Exporter release, such as 4.8.4.

• <dcgm-version>-<exporter-version> identifies the tested DCGM and DCGM Exporter pair, such as 4.6.1-4.8.4.

• <dcgm-version>-<exporter-version>-distroless selects the distroless image for that pair, such as 4.6.1-4.8.4-distroless.

Set DCGM_EXPORTER_TAG to the selected tag, then run:

$ export DCGM_EXPORTER_TAG=latest
$ docker run -d --rm \
    --name dcgm-exporter \
    --gpus all \
    --cap-add SYS_ADMIN \
    -p 9400:9400 \
    nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_TAG}

This command exposes every GPU and allows DCGM profiling fields such as DCGM_FI_PROF_*. Review Docker’s runtime privilege and Linux capabilities before adapting the command to a production host. Access to the Docker daemon also requires administrative authorization.

Install with Helm

The Helm chart deploys one exporter pod on each selected GPU node and enables Kubernetes pod mapping. Install Helm, then add the chart repository:

$ helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts
$ helm repo update

If the cluster has the Prometheus Operator custom resource definitions, install the chart with its default ServiceMonitor:

$ helm install dcgm-exporter gpu-helm-charts/dcgm-exporter \
    --namespace gpu-monitoring \
    --create-namespace

Otherwise, disable the ServiceMonitor until those definitions are installed:

$ helm install dcgm-exporter gpu-helm-charts/dcgm-exporter \
    --namespace gpu-monitoring \
    --create-namespace \
    --set serviceMonitor.enabled=false

Install with GPU Operator

The NVIDIA GPU Operator installs the driver, container toolkit, DCGM, DCGM Exporter, and related Kubernetes resources as one GPU software stack. Use it when the Operator already manages those components.

Follow the GPU Operator installation procedure, then verify that its exporter pods are running:

$ kubectl get pods --all-namespaces \
    --selector app=nvidia-dcgm-exporter

Configure the GPU Operator through its Helm values or ClusterPolicy. Do not edit the resources it creates.

To use custom fields, create a metrics ConfigMap in the same namespace as the GPU Operator release. The ConfigMap key must be dcgm-metrics.csv, then reference the ConfigMap from ClusterPolicy:

$ kubectl create configmap custom-dcgm-metrics \
    --namespace <operator-namespace> \
    --from-file=dcgm-metrics.csv=<custom-metrics.csv>
$ kubectl patch clusterpolicy <cluster-policy-name> --type merge \
    --patch '{"spec":{"dcgmExporter":{"config":{"name":"custom-dcgm-metrics"}}}}'

Under spec.dcgmExporter, you can enable or disable the exporter, add annotations, set host networking or service traffic policy, and add pod labels or pod UIDs. For example:

spec:
  dcgmExporter:
    enablePodLabels: true
    enablePodUID: true
    podLabelAllowlistRegex:
      - "^app$"
    hostNetwork: true
    service:
      internalTrafficPolicy: Local

Use a label allowlist to avoid creating too many Prometheus time series. Set hostNetwork only when Prometheus needs the exporter port on the node network.

Configure DCGM Exporter

Command-line flags and environment variables set at startup override values in an optional YAML file. A collector CSV or the YAML metrics section selects the fields to export. The dcgm-exporter command reference lists every flag, environment variable, default, and validation rule.

Connect to a DCGM Host Engine

Omitting --remote-hostengine-info uses an embedded DCGM host engine. To connect to a separately managed DCGM host engine, use one of these forms:

• host:port

• tcp://host:port

• unix:///path/to/socket

• vsock://cid:port

• [ipv6-address]:port

The URI forms require DCGM 4.5 or later. Examples:

$ dcgm-exporter -r tcp://host123:5555
$ dcgm-exporter -r unix:///tmp/nv-hostengine
$ dcgm-exporter -r vsock://3:5555
$ dcgm-exporter -r "[::1]:5555" -a "[::]:9400"

The endpoint must match the transport and listener configured for the DCGM host engine. A container also needs access to the corresponding host network, Unix socket, or VSOCK device.

Select Metrics

A collector CSV has exactly three columns:

DCGM field, Prometheus metric type, help message

Supported types are counter, gauge, untyped, and label. For a metric row, the help message becomes the Prometheus # HELP text and does not affect collection. A label row adds a returned string value to other samples instead of creating a metric family; its help message is not emitted.

Create a custom collector:

DCGM_FI_DEV_GPU_UTIL, gauge, GPU utilization (in %).
DCGM_FI_DEV_FB_USED, gauge, Framebuffer memory used (in MiB).
DCGM_FI_SYSTEM_DRIVER_VERSION, label, Driver version.

Pass it with --collectors. For a container, mount the file read-only and pass its in-container path:

$ docker run -d --rm \
    --name dcgm-exporter \
    --gpus all \
    --cap-add SYS_ADMIN \
    -p 9400:9400 \
    --mount type=bind,src=/opt/dcgm-exporter/custom.csv,dst=/etc/dcgm-exporter/custom.csv,readonly \
    nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_TAG} \
    --collectors /etc/dcgm-exporter/custom.csv

In Helm, customMetrics replaces rather than extends the shipped collector CSV. See DCGM Exporter Metrics for the shipped inventory, exporter-owned metrics, label behavior, and custom-field constraints.

Set Collection Cadence

--collect-interval sets the default DCGM watch cadence requested by the exporter, in milliseconds; the default is 30000. In YAML, named collections assign a cadence to selected fields. The all-fields scrape collection sets the default cadence, and narrower collections override it:

version: 2
metrics:
  file: /etc/dcgm-exporter/default-counters.csv
collections:
  - name: scrape
    every: 30s
    metrics:
      include: ["*"]
  - name: fast-thermals
    every: 5s
    metrics:
      include:
        - DCGM_FI_DEV_GPU_TEMP_CELSIUS
        - DCGM_FI_DEV_BOARD_POWER_WATTS

DCGM Exporter reads YAML only at startup. It watches the active file-backed collector CSV for changes and can also reload it with SIGHUP. See Fields and Watches for how DCGM combines watches and caches samples. The DCGM host engine refreshes that cache at the effective watch cadence. Each HTTP scrape reads the latest cached sample; the exporter does not have a separate periodic capture or file-storage interval.

Select Devices and Entities

Use --devices for GPUs and GPU instances, --switch-devices for NvSwitches and NvLinks, and --cpu-devices for CPUs and CPU cores. The default f selector monitors each full GPU without MIG and each GPU instance with MIG. The command reference documents the complete f, g, i, range, and combined-selector grammar.

Add Workload Labels

Enable only the metadata required by queries and alerts because each unique label set creates another Prometheus time series.

Optional workload mappings

Mapping	Primary setting	Result
Kubernetes pods	--kubernetes	Adds pod, namespace, and container labels from the kubelet pod-resources socket.
Pod labels and UID	--kubernetes-enable-pod-labels and --kubernetes-enable-pod-uid	Adds selected pod metadata. Restrict labels with the allowlist regex.
Shared or virtual GPUs	--kubernetes-virtual-gpus	Adds virtual-GPU attribution for supported sharing strategies.
Kubernetes DRA	--kubernetes-enable-dra	Adds claim, driver, pool, device, and applicable MIG labels.
Runtime containers	--container-labels	Maps explicit GPU assignments through a mounted Docker-compatible runtime socket outside Kubernetes.
HPC jobs	--hpc-job-mapping-dir	Adds job IDs from scheduler-managed GPU mapping files.

Verify the Installation

A successful metrics response contains names that begin with DCGM_:

$ curl --fail http://localhost:9400/metrics

For Kubernetes, forward the Service before running the same command:

$ kubectl port-forward \
    --namespace gpu-monitoring \
    service/dcgm-exporter 9400:9400

Administer and Troubleshoot the Exporter

Restart the package service with systemctl restart nvidia-dcgm-exporter.service. Replace a standalone container with the same configuration to upgrade it. Upgrade a Helm release after refreshing the repository:

$ helm repo update
$ helm upgrade dcgm-exporter gpu-helm-charts/dcgm-exporter \
    --namespace gpu-monitoring \
    --reuse-values

Changing a collector CSV reloads its metric selection. Changing YAML requires a restart. A restart or reload resets exporter-owned counters stored in process memory and can briefly interrupt the endpoint.

Use --web-config-file with a Prometheus exporter-toolkit configuration to enable TLS or basic authentication. Protect runtime sockets, debug dumps, and profiling endpoints because they can expose host, process, or workload data. --enable-pprof requires a web configuration so the profiling endpoints are protected.

When the endpoint or expected metrics are unavailable:

1. Run nvidia-smi on the GPU host to confirm that the driver can see the GPUs. Then confirm that the exporter process, container, or DaemonSet is running.

2. Inspect the exporter logs and verify that port 9400 is published and reachable.

3. For a separate DCGM host engine, verify the endpoint, transport, socket mount, and client/host-engine version relationship.

4. Confirm that the field is selected, then follow its link in the metrics reference for entity and runtime support.

5. For DCGM_FI_PROF_* fields, verify GPU support and the required SYS_ADMIN capability.

6. For Kubernetes labels, verify the pod-resources socket, device ID type, service account, and RBAC.

If the exporter log contains Cannot init NVML library; err: ERROR_LIBRARY_NOT_FOUND, the container cannot load libnvidia-ml.so.1 from the node driver. This is different from a missing DCGM library. Check that the NVIDIA runtime is configured, then inspect the DaemonSet’s configured RuntimeClass:

$ kubectl get runtimeclass
$ kubectl get daemonset dcgm-exporter \
    --namespace gpu-monitoring \
    --output=jsonpath='{.spec.template.spec.runtimeClassName}{"\\n"}'

If the default runtime does not provide NVIDIA support, set the chart’s runtimeClassName to a configured NVIDIA RuntimeClass and update the release:

$ helm upgrade dcgm-exporter gpu-helm-charts/dcgm-exporter \
    --namespace gpu-monitoring \
    --reuse-values \
    --set runtimeClassName=<runtime-class-name>

Do not copy NVML from the host into the image; the NVIDIA runtime injects the driver libraries that match the node.

Use --debug for more detailed logs. The command reference documents DCGM logging, debug dumps, and pprof. For broader DCGM failures, see Debugging and Troubleshooting.

Related Software

Software related to DCGM Exporter

Component	Relationship	Required for
NVIDIA driver and NVML	Provide GPU access and the management library used by DCGM. NVML is installed with the driver.	Every deployment.
nvidia-smi	Checks that the driver can discover the GPUs before you troubleshoot DCGM Exporter.	Host verification and troubleshooting.
DCGM	Discovers entities, owns field watches, and supplies cached telemetry.	Every deployment.
NVIDIA Container Toolkit	Exposes NVIDIA GPUs to containers.	Container deployments.
NVIDIA GPU Operator	Manages the Kubernetes GPU software stack and exporter lifecycle.	Operator-managed clusters.
Helm	Installs and upgrades the standalone DCGM Exporter Kubernetes chart.	Helm-managed clusters.
Prometheus Operator	Supplies the ServiceMonitor custom resource used by the chart’s default scrape configuration.	Chart-managed ServiceMonitor resources.
Prometheus	Scrapes, stores, and queries the exported time series.	Central metric collection.

Next Steps

• Configure Prometheus for DCGM Exporter configures Prometheus to scrape and verify DCGM Exporter targets.

• dcgm-exporter lists startup syntax, defaults, accepted values, and failure behavior.

• DCGM Exporter Metrics documents the shipped metric inventory, exporter-owned metrics, labels, and runtime considerations.

• Fields and Watches explains DCGM collection cadence, retention, timestamps, and cached values.

Image and Package Contents

Third-Party Notices

Third-party notices are installed at:

• /THIRD_PARTY_NOTICES in archives

• /usr/share/doc/datacenter-gpu-manager-exporter/THIRD_PARTY_NOTICES in Debian packages

• /usr/share/licenses/datacenter-gpu-manager-exporter/THIRD_PARTY_NOTICES in RPM packages

• /licenses/THIRD_PARTY_NOTICES in container images

Software Bill of Materials (SBOM)

NGC publishes an SBOM for each DCGM Exporter container image. Select the image version in the DCGM Exporter NGC catalog, then use the NGC API or ORAS retrieval instructions to retrieve the SBOM for an image digest.