dcgmi health#

NAME#

dcgmi-health - configure and query DCGM background health watches

SYNOPSIS#

dcgmi health --fetch [--group <group>] [--json] [--host <address>]
dcgmi health --set <systems> [--group <group>]
             [--update-interval <seconds>] [--max-keep-age <seconds>]
             [--host <address>]
dcgmi health --clear [--group <group>] [--host <address>]
dcgmi health --check [--group <group>] [--json] [--host <address>]

DESCRIPTION#

dcgmi health controls passive health monitoring. The set operation tells the host engine which systems to sample in the background. Fetch reports that configuration, and check evaluates the retained samples for warnings and failures.

Health watches remain active in the host engine after dcgmi exits. Some systems need approximately 60 seconds of samples before the first meaningful check.

OPTIONS#

Operation selection#

-f, --fetch

Display the systems currently watched for the selected group.

-s <systems>, --set <systems>

Replace the watched systems. <systems> is a string containing one or more of these letters, with no duplicate letter:

a: PCIe, memory, InfoROM, thermal, power, NVLink, driver, ConnectX, NVIDIA Internode Memory Exchange (IMEX), and NVSwitch fatal and nonfatal watches. Use a by itself. It does not select the SM watch shown by --fetch.
d: Driver health.
e: NVIDIA Internode Memory Exchange (IMEX) daemon and domain health.
i: InfoROM health.
m: GPU memory health.
n: NVLink health.
p: PCIe health.
t: Thermal and power health.
x: ConnectX health.

--clear

Disable every health watch for the selected group.

-c, --check

Evaluate watched systems and report incidents.

Sampling#

-u <seconds>, --update-interval <seconds>: With --set, set the driver sampling interval. The default is 30 seconds.
-m <seconds>, --max-keep-age <seconds>: With --set, retain samples for this many seconds. The default is 600 seconds.

Targets and output#

-g <group>, --group <group>: Select a numeric persistent group or one of the built-in group aliases: g for all GPUs, s for all NVSwitches, i for all GPU instances, c for all compute instances, or a for all entities. The default is g.
-j, --json: Format --fetch or successful --check results as JSON. The command writes a plain-text report heading before the JSON document. See JSON output.
--host <address>: Connect to the specified host engine.
-h, --help: Print generated usage information and exit.

WATCH AND RESULT REFERENCE#

--set replaces the complete watch mask for the selected group; letters are combined except that a must be used alone.

Health watch systems#
Selector	Systems enabled	Examples of incidents evaluated
`p`	PCIe	Excessive replay rate and bus- or fabric-related XIDs
`m`	GPU memory	Volatile double-bit ECC errors, faulty memory, pending or excessive page retirement, row-remap failures, and contained or uncontained errors
`i`	InfoROM	Reported InfoROM corruption
`d`	Driver	Recovery states that require traffic drain, reset, or reboot
`t`	Thermal and power	Thermal or power clock events, threshold violations, and unreadable GPU power
`n`	NVLink	Link errors and error rates, BER thresholds, and unhealthy fabric state
`e`	NVIDIA Internode Memory Exchange (IMEX)	Unhealthy IMEX daemon or domain state
`x`	ConnectX	Unhealthy device state and fatal or nonfatal PCIe AER errors
`a`	Every preceding system plus NVSwitch fatal and nonfatal; not SM	The preceding incidents plus NVSwitch errors and down links

The complete, version-specific incident and error-code table is in Background Health Checks. Programmatic watch constants and result structures are in Health Monitor.

The e selector evaluates IMEX independently of the NVLink selector. An unhealthy IMEX daemon or domain produces DCGM_FR_IMEX_UNHEALTHY. When the driver reports that recovery requires restoring the IMEX domain, DCGM reports DCGM_FR_GPU_RECOVERY_IMEX_DOMAIN. This recovery result is distinct from an IMEX health-watch incident: it tells the recovery workflow which domain action is required.

--check reports Healthy, Warning, or Failure overall, followed by affected entities and their incidents. It evaluates only retained data for the enabled watches; it does not run an active diagnostic. --fetch also shows an SM row, but dcgmi health --set has no selector for SM and a does not enable it. An SM watch set through another client can still be displayed.

SAMPLING AND LIFETIME#

PCIe, memory, NVLink, thermal/power, and ConnectX watches can require about 60 seconds before the first useful check. The update interval controls collection frequency; maximum keep age controls how long samples can contribute to a check. A shorter retention window can discard the evidence for an earlier incident.

The configured mask and watches remain after dcgmi disconnects. They are state in the running host engine and are not durable across a host-engine restart. Use --clear when the group no longer needs the watches.

EXAMPLES#

Watch PCIe, memory, and InfoROM health for group 1:

$ dcgmi health --set pmi --group 1

Display the configured watches:

$ dcgmi health --fetch --group 1

Check all GPU health and return JSON:

$ dcgmi health --check --group g --json

Disable health monitoring for group 1:

$ dcgmi health --clear --group 1