---
name: dcgm-perf-dashboard
description: Use when producing or reviewing an interactive performance dashboard from DCGM Exporter power-pilot captures, especially 1 Hz raw Prometheus logs, GPU power-limit correlation, event annotations, or multi-capture comparisons.
---

# DCGM Performance Dashboard

Build a self-contained interactive HTML dashboard that preserves evidence from
NVIDIA Data Center GPU Manager (DCGM) capture logs. Retain every valid source
sample as a point in an individual series in the delivered chart; aggregation can add a trace but must
never replace, resample, or discard the source series. Reproduce useful
dashboard behaviors, including linked zoom, reset, filters, and event overlays,
without adopting the source system's scheduler, topology, hostname, workload, or path
conventions.

## Inputs

Read the current [Dynamic Power Software (DPS) DCGM pilot runbook](https://docs.nvidia.com/datacenter/dps/versions/latest/guides/runbooks/maxlps-simple-mode-pilot-dcgm-telemetry.html)
before interpreting a capture. The normal input is one or more capture
directories or tarballs containing raw scrape blocks such as:

```text
# sample=42 timestamp_utc=2026-04-21T16:26:10.123456789Z
DCGM_FI_DEV_POWER_USAGE{Hostname="node-a",gpu="0",UUID="..."} 512.3
```

Also read `capture-info.txt` and the collector CSV when present. Use an
explicit `capture_id` and display name for comparisons; do not infer a Slurm
job, role, rack, model, framework, or workload from a path or hostname.

Use optional sidecar files only when the user explicitly maps them to the capture:

| File | Required columns | Purpose |
| --- | --- | --- |
| `inventory.csv` | `hostname,group,role,domain` | Neutral filters and aggregate groups. |
| `events.csv` | `start_utc,end_utc,event_type,label,scope,color` | Point or interval annotations. |
| Auxiliary telemetry | timestamp, series identity, value, unit | Memory, temperature, throughput, rate, or latency. |

An empty `end_utc` is a point event; otherwise shade the interval in every
time-series panel. Event types are labels, not a fixed vocabulary.

For an experiment-wide dashboard, `inventory.csv` is required and its
`hostname` column is the expected node set. Collect a separate 1 Hz capture
and metadata for every expected node over the experiment window. Without an
expected-node inventory, label the output as a partial capture view; never
claim it covers the whole experiment.

## Normalize and Validate First

Create long-form records keyed by `timestamp_utc`, `capture_id`, metric, and
the complete label set. Give each trace a stable `series_id` derived from
`capture_id`, metric, and the complete label set, excluding the timestamp.
Prefer the GPU universally unique identifier (UUID) and retain the hostname
so equal GPU indexes on different hosts never merge. Parse Prometheus label
escaping. Preserve GPU UUID/index, hostname, CPU/core labels, and other
identity labels unless the user asks to redact them.

A failed scrape comment is a gap, never a zero, and lines must not bridge that
gap. Normalize timestamps to Coordinated Universal Time (UTC); flag non-finite
or collector-documented sentinel values; and report cadence, gaps, selected-GPU coverage, capture
windows, and exporter/DCGM versions before rendering. Detect duplicate series
samples but do not silently choose one: retain them in provenance and require
an explicit deduplication rule if a user asks to remove them.

Compare observed hostnames with the expected node inventory. List every
missing node, node with no valid capture, and node missing a requested metric
family. Report partial node coverage as a coverage failure, including in fleet
aggregates.

Use the runbook's DCGM fields exactly:

| Field | Dashboard metric | Unit | Meaning |
| --- | --- | --- | --- |
| `DCGM_FI_DEV_POWER_MGMT_LIMIT` | Power-management limit | W | Device-reported observation. |
| `DCGM_FI_DEV_ENFORCED_POWER_LIMIT` | Enforced power limit | W | Device-reported observation. |
| `DCGM_FI_DEV_GPU_UTIL` | GPU utilization | % | Per GPU. |
| `DCGM_FI_DEV_POWER_USAGE` | GPU power draw | W | Per GPU. |
| `DCGM_FI_DEV_CPU_POWER_UTIL_CURRENT` | Grace CPU power draw | W | Per Grace CPU. |
| `DCGM_FI_DEV_CPU_UTIL_TOTAL` | Grace CPU utilization | % | Per Grace CPU core. |

The two limit fields do not prove an external request, the requested target,
or whether a request was clamped. Correlate them with supplied policy/debug
events only, and label that correlation as such. Grace CPU fields are expected
on GB200/GB300 Grace systems; their absence on non-Grace B200/H100 hosts is
expected, not zero CPU telemetry. For non-Grace host CPU utilization, use a
separate 1 Hz host source and derive utilization from the non-idle rate of
`node_cpu_seconds_total`.

`DCGM_FI_DEV_FB_USED` (MiB) is an optional DCGM extension for a
memory panel; it is not in the runbook collector CSV. Never invent a memory,
temperature, CPU, or throughput series when it was not captured.

## Render the Dashboard

Produce one portable interactive HTML artifact. Select a rendering library
that can retain and inspect every source point at the required scale. Add a compact UTC summary with duration, selected
GPU count, average GPU utilization, peak GPU power, source versions, input
checksums, and data-coverage warnings. Charts must be interactive: mouseover
of every raw point must reveal its UTC timestamp, value, unit, `capture_id`,
metric, and available GPU/CPU/host identity labels. Synchronize x-axis navigation, hover, and the crosshair across the following
panels. Include each panel only when its data is available:

1. GPU utilization: aggregate mean by supplied group plus hidden-by-default
   individual GPU traces.
2. GPU memory used: only when an explicitly mapped memory series exists; for
   partial multi-capture coverage, name captures that did not collect it rather
   than drawing zero-valued traces.
3. Per-GPU power: individual GPU traces, group mean, and visually distinct
   management/enforced-limit overlays when present.
4. Total GPU power: group/host totals with contributing-GPU count in hover.
5. CPU telemetry: only when captured; keep core traces separate from an
   explicitly labeled core average.

Use supplied inventory for filters/tabs; otherwise provide `all`, capture,
host, and GPU views. Every individual trace must retain every valid sample in
the chart data, even if an initial view hides a trace behind a user-controlled
filter. Aggregate traces are additional context, never a substitute. Provide a
reset control, a "show individual GPU traces" control, and an export of the
currently selected raw records with their source/provenance identifiers. A
generic throughput, rate, or latency series can be an opt-in secondary-axis
overlay only when its unit and source are shown.

For an experiment-wide dashboard, include an accessible per-node chart or
filtered view for every expected node. Show nodes with no usable data as
`not collected` or `invalid capture`; do not omit them or substitute fleet
aggregates for their charts.

Before making a portable HTML file shareable, ask whether hostnames, GPU UUIDs,
device IDs, and workload labels can be shared. Pseudonymize selected labels
consistently when requested, and record the redaction choice in provenance.

At an exact common timestamp, utilization is the arithmetic mean of observed
GPUs and host/group power is the sum of observed `POWER_USAGE` values. Do not
combine non-coincident host samples as an instantaneous total. If clocks or
scrape times differ, require an explicit derived-aggregate window and rule for
multiple samples per `series_id`; label the result as windowed and show its
contributing-device count. Never interpolate, zero-fill, bin, or downsample
source series. If a browser cannot render the full capture, choose a renderer
or user-controlled partitioning (for example, time window or selected
entities) that preserves every point in each rendered view; never silently
decimate data.

## Verify the Result

Before delivery, use small fixtures to check label escaping, UTC boundaries,
scrape failures and line breaks, non-finite values, duplicate handling,
no-zero-fill power totals, non-coincident-host aggregate rules, absent
non-Grace CPU metrics, optional-memory suppression, and event rendering. Add
a known one-second spike and assert that each valid fixture sample appears
once in both the individual chart data and the raw-record export; aggregates
must not be the only trace. Verify that the observed node set equals the
experiment inventory or that every missing/invalid node is visibly reported
and has an explicit per-node state. Open the generated HTML and confirm
panels, legends, per-point mouseover tooltips, linked zoom, annotations,
provenance, and redaction state are visible. State the capture and coverage
limits instead of presenting the dashboard as proof of requested power-policy
actuation or a complete facility-power measurement.