Collect Pilot Telemetry: 1-Second GPU and CPU Data with DCGM Exporter#

Use this support guide to collect the power and utilization evidence for the Run the MaxLPS Power Pilot. Return to MaxLPS Power Pilot to choose another pilot step.

This guide collects a short, shareable power and utilization trace from an NVIDIA GB200, GB300, B200, or H100 system. It uses a pinned DCGM Exporter release and its CSV/CLI configuration interface only. No YAML configuration file is required.

This playbook is pinned to:

DCGM Exporter 4.8.2
Paired DCGM version 4.5.3
Container image nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless
Helm chart 4.8.2

What This Captures#

Requested signal	DCGM field used by this release	Scope	Unit
Device-reported GPU power management limit	`DCGM_FI_DEV_POWER_MGMT_LIMIT`	GPU	W
Device-reported enforced GPU power limit	`DCGM_FI_DEV_ENFORCED_POWER_LIMIT`	GPU	W
GPU utilization	`DCGM_FI_DEV_GPU_UTIL`	GPU	%
GPU power draw	`DCGM_FI_DEV_POWER_USAGE`	GPU	W
CPU power draw	`DCGM_FI_DEV_CPU_POWER_UTIL_CURRENT`	Grace CPU	W
CPU utilization	`DCGM_FI_DEV_CPU_UTIL_TOTAL`	Grace CPU core	%

DCGM CPU telemetry is supported for NVIDIA Grace CPUs. Expect all six metric families on a GB200/Grace or GB300/Grace system. On a B200 or H100 node with a non-Grace host CPU, the four GPU metric families should be present and the two CPU metric families will normally be absent. That is expected: DCGM Exporter is not a general-purpose x86 or non-NVIDIA CPU exporter.

The two DCGM power-limit fields are device-side observations. DCGM_FI_DEV_POWER_MGMT_LIMIT does not show the requested target, and neither DCGM field shows whether a request was clamped before dispatch. Correlate this capture with the per-update debug evidence required by the parent pilot runbook.

Non-Grace Host CPU Utilization#

On B200, H100, or another node with a non-Grace host CPU, collect host CPU utilization through node exporter or an equivalent host collector at the same 1-second interval. Retain both aggregate and per-core data for the complete workload window. node_cpu_seconds_total is a cumulative counter; calculate utilization from its non-idle rate rather than treating the raw value as a percentage.

If the host does not already expose CPU metrics, install or enable a supported host collector before the pilot. For a bounded diagnostic when a metrics pipeline is unavailable, mpstat -P ALL 1 from the sysstat package is an acceptable fallback. Record the collector, version, sampling interval, and exact command. Do not report non-Grace CPU power unless a supported platform-specific source provides it.

Before You Begin#

Run this procedure on the node you need to investigate, or target that node when using Kubernetes.

You need:

A working NVIDIA driver; nvidia-smi -L must list the target GPUs.
curl, tar, and Bash.
Root or sudo access for the package and Docker methods.
Exactly one of the three installation methods in this playbook.
For Docker, a working NVIDIA Container Toolkit installation.
For Helm, a Kubernetes cluster that can schedule GPU workloads and kubectl/Helm access.

Avoid running the package and Docker methods at the same time because both publish host port 9400 in this playbook.

Create a timestamped capture directory and the collector CSV:

export CAPTURE_DIR="${HOME}/dcgm-sre-capture-$(hostname -s)-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "${CAPTURE_DIR}"

export METRICS_CSV="${CAPTURE_DIR}/sre-power-metrics.csv"
tee "${METRICS_CSV}" >/dev/null <<'EOF'
DCGM_FI_DEV_POWER_MGMT_LIMIT, gauge, Device-reported GPU power management limit in watts.
DCGM_FI_DEV_ENFORCED_POWER_LIMIT, gauge, Enforced GPU power limit in watts.
DCGM_FI_DEV_GPU_UTIL, gauge, GPU utilization in percent.
DCGM_FI_DEV_POWER_USAGE, gauge, GPU power draw in watts.
DCGM_FI_DEV_CPU_POWER_UTIL_CURRENT, gauge, Grace CPU power draw in watts.
DCGM_FI_DEV_CPU_UTIL_TOTAL, gauge, Grace CPU core utilization in percent.
EOF

sed -n '1,20p' "${METRICS_CSV}"

Each row has exactly three values: DCGM field, Prometheus type, and HELP text. Keep commas out of unquoted HELP text.

Install and Start DCGM Exporter#

Choose one method.

Method 1: Native DEB or RPM Package#

These commands assume that the appropriate NVIDIA CUDA network repository is already configured for the operating system and architecture. Refer to Install DCGM if it is not.

Set CUDA_MAJOR to the CUDA user-mode driver major version installed on the node:

export CUDA_MAJOR=12  # Change to 13 when the installed user-mode driver is CUDA 13.

On Ubuntu or Debian, install the pinned, tested pair:

sudo apt-get update
apt-cache madison "datacenter-gpu-manager-4-cuda${CUDA_MAJOR}" datacenter-gpu-manager-exporter
sudo apt-get install --yes --install-recommends \
  "datacenter-gpu-manager-4-cuda${CUDA_MAJOR}=1:4.5.3-1" \
  "datacenter-gpu-manager-exporter=4.8.2-1"

On RHEL, Rocky Linux, or another supported RPM-based distribution, install the same pair:

sudo dnf --showduplicates list \
  "datacenter-gpu-manager-4-cuda${CUDA_MAJOR}" \
  datacenter-gpu-manager-exporter
sudo dnf install --assumeyes --setopt=install_weak_deps=True \
  "datacenter-gpu-manager-4-cuda${CUDA_MAJOR}-4.5.3-1" \
  "datacenter-gpu-manager-exporter-4.8.2-1"

Install the collector CSV and add a systemd override that selects it and changes the DCGM watch interval to 1,000 milliseconds:

sudo install -d -m 0755 /etc/dcgm-exporter
sudo install -m 0644 "${METRICS_CSV}" /etc/dcgm-exporter/sre-power-metrics.csv

sudo install -d -m 0755 /etc/systemd/system/nvidia-dcgm-exporter.service.d
sudo tee /etc/systemd/system/nvidia-dcgm-exporter.service.d/sre-capture.conf >/dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/sre-power-metrics.csv -c 1000 -p f
EOF

sudo systemctl enable --now nvidia-dcgm.service
sudo systemctl daemon-reload
sudo systemctl enable nvidia-dcgm-exporter.service
sudo systemctl restart nvidia-dcgm-exporter.service
sudo systemctl --no-pager --full status nvidia-dcgm-exporter.service

Use this endpoint for the capture steps:

export EXPORTER_URL=http://127.0.0.1:9400

Method 2: Docker Container#

The container includes the paired DCGM version, so do not install a separate DCGM package for this method. Run the current release image with the collector CSV mounted read-only:

docker run --detach --rm \
  --name dcgm-exporter-sre \
  --gpus all \
  --cap-add SYS_ADMIN \
  --publish 9400:9400 \
  --volume "${METRICS_CSV}:/etc/dcgm-exporter/sre-power-metrics.csv:ro" \
  nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless \
  -f /etc/dcgm-exporter/sre-power-metrics.csv \
  -c 1000 \
  -p f

docker logs dcgm-exporter-sre
export EXPORTER_URL=http://127.0.0.1:9400

On an SELinux-enforcing host, use :ro,Z rather than :ro on the volume mount if the container cannot read the CSV.

Method 3: Helm on Kubernetes#

This method uses --set-file and CLI arguments, so it does not require a values or manifest YAML file. It installs one exporter pod on each eligible node.

helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts
helm repo update

helm upgrade --install dcgm-exporter gpu-helm-charts/dcgm-exporter \
  --namespace gpu-monitoring \
  --create-namespace \
  --version 4.8.2 \
  --set-string image.tag=4.5.3-4.8.2-distroless \
  --set serviceMonitor.enabled=false \
  --set-file customMetrics="${METRICS_CSV}" \
  --set-string 'arguments[0]=-m' \
  --set-string 'arguments[1]=gpu-monitoring:exporter-metrics-config-map' \
  --set-string 'arguments[2]=-c' \
  --set-string 'arguments[3]=1000' \
  --set-string 'arguments[4]=-p' \
  --set-string 'arguments[5]=f'

kubectl rollout status --namespace gpu-monitoring daemonset/dcgm-exporter
kubectl get pods --namespace gpu-monitoring --output wide

Select the exporter pod on the node being investigated. Replace the placeholder with the Kubernetes node name shown by kubectl get nodes:

export TARGET_NODE='<target-kubernetes-node-name>'
export EXPORTER_POD="$(kubectl get pods \
  --namespace gpu-monitoring \
  --selector app.kubernetes.io/name=dcgm-exporter \
  --field-selector "spec.nodeName=${TARGET_NODE}" \
  --output jsonpath='{.items[0].metadata.name}')"

test -n "${EXPORTER_POD}" || { echo "No DCGM Exporter pod found on ${TARGET_NODE}" >&2; exit 1; }

kubectl port-forward \
  --namespace gpu-monitoring \
  "pod/${EXPORTER_POD}" \
  9400:9400 \
  >"${CAPTURE_DIR}/kubectl-port-forward.log" 2>&1 &
export PORT_FORWARD_PID=$!
export EXPORTER_URL=http://127.0.0.1:9400
sleep 2

Verify the Requested Metrics#

Check the endpoint before starting the timed capture:

curl --fail --silent --show-error --connect-timeout 5 --max-time 5 "${EXPORTER_URL}/metrics" \
  | grep -E '^DCGM_FI_DEV_(POWER_MGMT_LIMIT|ENFORCED_POWER_LIMIT|GPU_UTIL|POWER_USAGE|CPU_POWER_UTIL_CURRENT|CPU_UTIL_TOTAL)(\{|[[:space:]])'

Confirm that:

GPU sample lines exist for requested limit, enforced limit, utilization, and power draw.
CPU power and utilization lines exist on GB200/Grace or GB300/Grace.
CPU utilization has one series per Grace CPU core rather than one host-wide aggregate.
Missing CPU series on a non-Grace B200/H100 host are expected.

If a GPU series is absent, check the exporter log and confirm that the GPU, driver, and pinned DCGM/exporter pair are healthy before collecting data. For the package or Docker method, start with:

nvidia-smi -L

For the package method:

sudo journalctl --unit nvidia-dcgm-exporter.service --since '10 minutes ago' --no-pager

For Docker:

docker logs dcgm-exporter-sre

For Helm:

kubectl logs --namespace gpu-monitoring "pod/${EXPORTER_POD}" --tail=200

Capture One Sample per Second to a File#

Choose the capture duration in seconds. The default records five minutes. DCGM Exporter watches the fields every 1,000 ms because each installation method passed -c 1000; the loop also scrapes the exporter once per second.

export CAPTURE_SECONDS=300
export METRICS_FILE="${CAPTURE_DIR}/dcgm-exporter-power-utilization.prom"
export METRIC_PATTERN='^DCGM_FI_DEV_(POWER_MGMT_LIMIT|ENFORCED_POWER_LIMIT|GPU_UTIL|POWER_USAGE|CPU_POWER_UTIL_CURRENT|CPU_UTIL_TOTAL)(\{|[[:space:]])'

: >"${METRICS_FILE}"
export CAPTURE_START_UTC="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

On a non-Grace host with node exporter, start a matching host CPU capture before the DCGM loop. Repeat this on every target node or point HOST_EXPORTER_URL at the corresponding node-exporter endpoint:

export HOST_EXPORTER_URL=http://127.0.0.1:9100
export HOST_CPU_FILE="${CAPTURE_DIR}/host-cpu-utilization.prom"
: >"${HOST_CPU_FILE}"

(
  for ((sample = 1; sample <= CAPTURE_SECONDS; sample++)); do
    printf '# sample=%d timestamp_utc=%s\n' \
      "${sample}" \
      "$(date -u +%Y-%m-%dT%H:%M:%S.%NZ)" \
      >>"${HOST_CPU_FILE}"

    if host_metrics="$(curl --fail --silent --show-error --connect-timeout 5 --max-time 5 "${HOST_EXPORTER_URL}/metrics")"; then
      printf '%s\n' "${host_metrics}" \
        | grep -E '^node_cpu_seconds_total(\{|[[:space:]])' \
        >>"${HOST_CPU_FILE}"
    else
      printf '# sample=%d ERROR: host CPU scrape failed\n' "${sample}" \
        >>"${HOST_CPU_FILE}"
    fi

    sleep 1
  done
) &
export HOST_CPU_CAPTURE_PID=$!

Run the DCGM capture:

for ((sample = 1; sample <= CAPTURE_SECONDS; sample++)); do
  printf '# sample=%d timestamp_utc=%s\n' \
    "${sample}" \
    "$(date -u +%Y-%m-%dT%H:%M:%S.%NZ)" \
    >>"${METRICS_FILE}"

  if exporter_metrics="$(curl --fail --silent --show-error --connect-timeout 5 --max-time 5 "${EXPORTER_URL}/metrics")"; then
    printf '%s\n' "${exporter_metrics}" \
      | grep -E "${METRIC_PATTERN}" \
      >>"${METRICS_FILE}"
  else
    printf '# sample=%d ERROR: GPU and CPU scrape failed\n' "${sample}" \
      >>"${METRICS_FILE}"
  fi

  sleep 1
done

if test -n "${HOST_CPU_CAPTURE_PID:-}"; then
  wait "${HOST_CPU_CAPTURE_PID}"
fi

wc --lines "${METRICS_FILE}"
tail -30 "${METRICS_FILE}"

The resulting file is a sequence of raw Prometheus samples. Each block begins with a UTC timestamp comment and retains the exporter labels needed to distinguish GPUs, CPUs, and CPU cores. It is a diagnostic capture log, not a single valid Prometheus exposition response.

Record enough system context to interpret the trace:

{
  printf 'capture_start_utc=%s\n' "${CAPTURE_START_UTC}"
  printf 'hostname=%s\n' "$(hostname -f 2>/dev/null || hostname)"
  printf 'dcgm_exporter_release=%s\n' '4.5.3-4.8.2'
  printf 'capture_seconds=%s\n' "${CAPTURE_SECONDS}"
  printf 'exporter_url=%s\n' "${EXPORTER_URL}"
  uname -a
  if command -v nvidia-smi >/dev/null 2>&1; then
    nvidia-smi -L
  fi
  if test -n "${TARGET_NODE:-}"; then
    printf 'kubernetes_target_node=%s\n' "${TARGET_NODE}"
    printf 'kubernetes_exporter_pod=%s\n' "${EXPORTER_POD}"
    kubectl get node "${TARGET_NODE}" --output wide
  fi
} >"${CAPTURE_DIR}/capture-info.txt" 2>&1

Create the Shareable Tarball#

Package the raw samples, the exact collector CSV, capture metadata, and any port-forward log into one archive:

export ARCHIVE="${CAPTURE_DIR}.tar.gz"
tar --create --gzip \
  --file "${ARCHIVE}" \
  --directory "$(dirname "${CAPTURE_DIR}")" \
  "$(basename "${CAPTURE_DIR}")"

sha256sum "${ARCHIVE}" | tee "${ARCHIVE}.sha256"
tar --list --gzip --file "${ARCHIVE}"
printf 'Share this archive: %s\n' "${ARCHIVE}"

Share the .tar.gz file and, when useful, its .sha256 checksum. Review the captured labels before sending the archive outside the intended support channel; hostnames, GPU UUIDs, device identifiers, and Kubernetes workload labels can be operationally sensitive.

Optional Cleanup#

For the package method, stop the one-off exporter configuration and remove the override:

sudo systemctl disable --now nvidia-dcgm-exporter.service
sudo rm -f /etc/systemd/system/nvidia-dcgm-exporter.service.d/sre-capture.conf
sudo systemctl daemon-reload