Capture 1-second GPU and CPU telemetry with DCGM Exporter

One-off SRE playbook: capture GPU and CPU power/utilization with DCGM Exporter

Use this playbook to collect a short, shareable power and utilization trace from an NVIDIA GB200, B200, or H100 system. It uses the current public DCGM Exporter release and its CSV/CLI configuration interface only—no YAML configuration file is required.

This playbook is pinned to:

DCGM Exporter 4.8.2
Paired DCGM version 4.5.3
Container image nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless
Helm chart 4.8.2

What this captures

Requested signal	DCGM field used by this release	Scope	Unit
Device-reported requested GPU power limit	`DCGM_FI_DEV_POWER_MGMT_LIMIT`	GPU	W
Device-reported enforced GPU power limit	`DCGM_FI_DEV_ENFORCED_POWER_LIMIT`	GPU	W
GPU utilization	`DCGM_FI_DEV_GPU_UTIL`	GPU	%
GPU power draw	`DCGM_FI_DEV_POWER_USAGE`	GPU	W
CPU power draw	`DCGM_FI_DEV_CPU_POWER_UTIL_CURRENT`	Grace CPU	W
CPU utilization	`DCGM_FI_DEV_CPU_UTIL_TOTAL`	Grace CPU core	%

DCGM CPU telemetry is supported for NVIDIA Grace CPUs. Expect all six metric families on a GB200/Grace system. On a B200 or H100 node with a non-Grace host CPU, the four GPU metric families should be present and the two CPU metric families will normally be absent. That is expected: DCGM Exporter is not a general-purpose x86 or non-NVIDIA CPU exporter.

The two DCGM power-limit fields are device-side observations. DCGM_FI_DEV_POWER_MGMT_LIMIT does not show the requested target, and neither DCGM field shows whether a request was clamped before dispatch. Correlate this capture with the per-update debug evidence required by the parent pilot runbook.

Non-Grace host CPU utilization

On B200, H100, or another node with a non-Grace host CPU, collect host CPU utilization through node exporter or an equivalent host collector at the same 1-second interval. Retain both aggregate and per-core data for the complete workload window. node_cpu_seconds_total is a cumulative counter; calculate utilization from its non-idle rate rather than treating the raw value as a percentage.

If the host does not already expose CPU metrics, install or enable a supported host collector before the pilot. For a bounded diagnostic when a metrics pipeline is unavailable, mpstat -P ALL 1 from the sysstat package is an acceptable fallback. Record the collector, version, sampling interval, and exact command. Do not report non-Grace CPU power unless a supported platform-specific source provides it.

Before you begin

Run this procedure on the node you need to investigate, or target that node when using Kubernetes.

You need:

A working NVIDIA driver; nvidia-smi -L must list the target GPUs.
curl, tar, and Bash.
Root or sudo access for the package and Docker methods.
Exactly one of the three installation methods below.
For Docker, a working NVIDIA Container Toolkit installation.
For Helm, a Kubernetes cluster that can schedule GPU workloads and kubectl/Helm access.

Avoid running the package and Docker methods at the same time because both publish host port 9400 in this playbook.

Create a timestamped capture directory and the collector CSV:

export CAPTURE_DIR="${HOME}/dcgm-sre-capture-$(hostname -s)-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "${CAPTURE_DIR}"

export METRICS_CSV="${CAPTURE_DIR}/sre-power-metrics.csv"
tee "${METRICS_CSV}" >/dev/null <<'EOF'
DCGM_FI_DEV_POWER_MGMT_LIMIT, gauge, Requested GPU power limit in watts.
DCGM_FI_DEV_ENFORCED_POWER_LIMIT, gauge, Enforced GPU power limit in watts.
DCGM_FI_DEV_GPU_UTIL, gauge, GPU utilization in percent.
DCGM_FI_DEV_POWER_USAGE, gauge, GPU power draw in watts.
DCGM_FI_DEV_CPU_POWER_UTIL_CURRENT, gauge, Grace CPU power draw in watts.
DCGM_FI_DEV_CPU_UTIL_TOTAL, gauge, Grace CPU core utilization in percent.
EOF

sed -n '1,20p' "${METRICS_CSV}"

Each row has exactly three values: DCGM field, Prometheus type, and HELP text. Keep commas out of unquoted HELP text.

Install and start DCGM Exporter

Choose one method.

Method 1: Native DEB or RPM package

These commands assume that the appropriate NVIDIA CUDA network repository is already configured for the operating system and architecture. See Install DCGM if it is not.

Set CUDA_MAJOR to the CUDA user-mode driver major version installed on the node:

export CUDA_MAJOR=12  # Change to 13 when the installed user-mode driver is CUDA 13.

On Ubuntu or Debian, install the pinned, tested pair:

sudo apt-get update
apt-cache madison "datacenter-gpu-manager-4-cuda${CUDA_MAJOR}" datacenter-gpu-manager-exporter
sudo apt-get install --yes --install-recommends \
  "datacenter-gpu-manager-4-cuda${CUDA_MAJOR}=1:4.5.3-1" \
  "datacenter-gpu-manager-exporter=4.8.2-1"

On RHEL, Rocky Linux, or another supported RPM-based distribution, install the same pair:

sudo dnf --showduplicates list \
  "datacenter-gpu-manager-4-cuda${CUDA_MAJOR}" \
  datacenter-gpu-manager-exporter
sudo dnf install --assumeyes --setopt=install_weak_deps=True \
  "datacenter-gpu-manager-4-cuda${CUDA_MAJOR}-4.5.3-1" \
  "datacenter-gpu-manager-exporter-4.8.2-1"

Install the collector CSV and add a systemd override that selects it and changes the DCGM watch interval to 1,000 milliseconds:

sudo install -d -m 0755 /etc/dcgm-exporter
sudo install -m 0644 "${METRICS_CSV}" /etc/dcgm-exporter/sre-power-metrics.csv

sudo install -d -m 0755 /etc/systemd/system/nvidia-dcgm-exporter.service.d
sudo tee /etc/systemd/system/nvidia-dcgm-exporter.service.d/sre-capture.conf >/dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/sre-power-metrics.csv -c 1000 -p f
EOF

sudo systemctl enable --now nvidia-dcgm.service
sudo systemctl daemon-reload
sudo systemctl enable --now nvidia-dcgm-exporter.service
sudo systemctl --no-pager --full status nvidia-dcgm-exporter.service

Use this endpoint for the capture steps:

export EXPORTER_URL=http://127.0.0.1:9400

Method 2: Docker container

The container includes the paired DCGM version, so do not install a separate DCGM package for this method. Run the current release image with the collector CSV mounted read-only:

docker run --detach --rm \
  --name dcgm-exporter-sre \
  --gpus all \
  --cap-add SYS_ADMIN \
  --publish 9400:9400 \
  --volume "${METRICS_CSV}:/etc/dcgm-exporter/sre-power-metrics.csv:ro" \
  nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless \
  -f /etc/dcgm-exporter/sre-power-metrics.csv \
  -c 1000 \
  -p f

docker logs dcgm-exporter-sre
export EXPORTER_URL=http://127.0.0.1:9400

On an SELinux-enforcing host, use :ro,Z rather than :ro on the volume mount if the container cannot read the CSV.

Method 3: Helm on Kubernetes

This method uses --set-file and CLI arguments, so it does not require a values or manifest YAML file. It installs one exporter pod on each eligible node.

helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts
helm repo update

helm upgrade --install dcgm-exporter gpu-helm-charts/dcgm-exporter \
  --namespace gpu-monitoring \
  --create-namespace \
  --version 4.8.2 \
  --set-string image.tag=4.5.3-4.8.2-distroless \
  --set serviceMonitor.enabled=false \
  --set-file customMetrics="${METRICS_CSV}" \
  --set-string 'arguments[0]=-m' \
  --set-string 'arguments[1]=gpu-monitoring:exporter-metrics-config-map' \
  --set-string 'arguments[2]=-c' \
  --set-string 'arguments[3]=1000' \
  --set-string 'arguments[4]=-p' \
  --set-string 'arguments[5]=f'

kubectl rollout status --namespace gpu-monitoring daemonset/dcgm-exporter
kubectl get pods --namespace gpu-monitoring --output wide

Select the exporter pod on the node being investigated. Replace the placeholder with the Kubernetes node name shown by kubectl get nodes:

export TARGET_NODE='<target-kubernetes-node-name>'
export EXPORTER_POD="$(kubectl get pods \
  --namespace gpu-monitoring \
  --selector app.kubernetes.io/name=dcgm-exporter \
  --field-selector "spec.nodeName=${TARGET_NODE}" \
  --output jsonpath='{.items[0].metadata.name}')"

test -n "${EXPORTER_POD}" || { echo "No DCGM Exporter pod found on ${TARGET_NODE}" >&2; exit 1; }

kubectl port-forward \
  --namespace gpu-monitoring \
  "pod/${EXPORTER_POD}" \
  9400:9400 \
  >"${CAPTURE_DIR}/kubectl-port-forward.log" 2>&1 &
export PORT_FORWARD_PID=$!
export EXPORTER_URL=http://127.0.0.1:9400
sleep 2

Verify the requested metrics

Check the endpoint before starting the timed capture:

curl --fail --silent --show-error "${EXPORTER_URL}/metrics" \
  | grep -E '^DCGM_FI_DEV_(POWER_MGMT_LIMIT|ENFORCED_POWER_LIMIT|GPU_UTIL|POWER_USAGE|CPU_POWER_UTIL_CURRENT|CPU_UTIL_TOTAL)(\{|[[:space:]])'

Confirm that:

GPU sample lines exist for requested limit, enforced limit, utilization, and power draw.
CPU power and utilization lines exist on GB200/Grace.
CPU utilization has one series per Grace CPU core rather than one host-wide aggregate.
Missing CPU series on a non-Grace B200/H100 host are expected.

If a GPU series is absent, check the exporter log and confirm that the GPU, driver, and pinned DCGM/exporter pair are healthy before collecting data. For the package or Docker method, start with:

nvidia-smi -L

For the package method:

sudo journalctl --unit nvidia-dcgm-exporter.service --since '10 minutes ago' --no-pager

For Docker:

docker logs dcgm-exporter-sre

For Helm:

kubectl logs --namespace gpu-monitoring "pod/${EXPORTER_POD}" --tail=200

Capture one sample per second to a file

Choose the capture duration in seconds. The default below records five minutes. DCGM Exporter watches the fields every 1,000 ms because each installation method passed -c 1000; the loop also scrapes the exporter once per second.

export CAPTURE_SECONDS=300
export METRICS_FILE="${CAPTURE_DIR}/dcgm-exporter-power-utilization.prom"
export METRIC_PATTERN='^DCGM_FI_DEV_(POWER_MGMT_LIMIT|ENFORCED_POWER_LIMIT|GPU_UTIL|POWER_USAGE|CPU_POWER_UTIL_CURRENT|CPU_UTIL_TOTAL)(\{|[[:space:]])'

: >"${METRICS_FILE}"

On a non-Grace host with node exporter, start a matching host CPU capture before the DCGM loop. Repeat this on every target node or point HOST_EXPORTER_URL at the corresponding node-exporter endpoint:

export HOST_EXPORTER_URL=http://127.0.0.1:9100
export HOST_CPU_FILE="${CAPTURE_DIR}/host-cpu-utilization.prom"
: >"${HOST_CPU_FILE}"

(
  for ((sample = 1; sample <= CAPTURE_SECONDS; sample++)); do
    printf '# sample=%d timestamp_utc=%s\n' \
      "${sample}" \
      "$(date -u +%Y-%m-%dT%H:%M:%S.%NZ)" \
      >>"${HOST_CPU_FILE}"

    curl --fail --silent --show-error --max-time 5 "${HOST_EXPORTER_URL}/metrics" \
      | grep -E '^node_cpu_seconds_total(\{|[[:space:]])' \
      >>"${HOST_CPU_FILE}"

    sleep 1
  done
) &
export HOST_CPU_CAPTURE_PID=$!

Run the DCGM capture:

for ((sample = 1; sample <= CAPTURE_SECONDS; sample++)); do
  printf '# sample=%d timestamp_utc=%s\n' \
    "${sample}" \
    "$(date -u +%Y-%m-%dT%H:%M:%S.%NZ)" \
    >>"${METRICS_FILE}"

  curl --fail --silent --show-error --max-time 5 "${EXPORTER_URL}/metrics" \
    | grep -E "${METRIC_PATTERN}" \
    >>"${METRICS_FILE}"

  sleep 1
done

if test -n "${HOST_CPU_CAPTURE_PID:-}"; then
  wait "${HOST_CPU_CAPTURE_PID}"
fi

wc --lines "${METRICS_FILE}"
tail -30 "${METRICS_FILE}"

The resulting file is a sequence of raw Prometheus samples. Each block begins with a UTC timestamp comment and retains the exporter labels needed to distinguish GPUs, CPUs, and CPU cores. It is a diagnostic capture log, not a single valid Prometheus exposition response.

Record enough system context to interpret the trace:

{
  printf 'capture_start_utc=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  printf 'hostname=%s\n' "$(hostname -f 2>/dev/null || hostname)"
  printf 'dcgm_exporter_release=%s\n' '4.5.3-4.8.2'
  printf 'capture_seconds=%s\n' "${CAPTURE_SECONDS}"
  printf 'exporter_url=%s\n' "${EXPORTER_URL}"
  uname -a
  if command -v nvidia-smi >/dev/null 2>&1; then
    nvidia-smi -L
  fi
  if test -n "${TARGET_NODE:-}"; then
    printf 'kubernetes_target_node=%s\n' "${TARGET_NODE}"
    printf 'kubernetes_exporter_pod=%s\n' "${EXPORTER_POD}"
    kubectl get node "${TARGET_NODE}" --output wide
  fi
} >"${CAPTURE_DIR}/capture-info.txt" 2>&1

Create the shareable tarball

Package the raw samples, the exact collector CSV, capture metadata, and any port-forward log into one archive:

export ARCHIVE="${CAPTURE_DIR}.tar.gz"
tar --create --gzip \
  --file "${ARCHIVE}" \
  --directory "$(dirname "${CAPTURE_DIR}")" \
  "$(basename "${CAPTURE_DIR}")"

sha256sum "${ARCHIVE}" | tee "${ARCHIVE}.sha256"
tar --list --gzip --file "${ARCHIVE}"
printf 'Share this archive: %s\n' "${ARCHIVE}"

Share the .tar.gz file and, when useful, its .sha256 checksum. Review the captured labels before sending the archive outside the intended support channel; hostnames, GPU UUIDs, device identifiers, and Kubernetes workload labels can be operationally sensitive.

Optional cleanup

For the package method, stop the one-off exporter configuration and remove the override:

sudo systemctl disable --now nvidia-dcgm-exporter.service
sudo rm -f /etc/systemd/system/nvidia-dcgm-exporter.service.d/sre-capture.conf
sudo systemctl daemon-reload