dcgmi diag#

NAME#

dcgmi-diag - run DCGM active health diagnostics

SYNOPSIS#

dcgmi diag --run <suite-or-tests>
           [--entity-id <entities> | --group <group-id>]
           [--parameters <test.parameter=value>]...
           [--configfile <path>] [--iterations <count>]
           [--timeout <seconds>] [--verbose] [--json]
           [<diagnostic-options>] [--host <address>]

DESCRIPTION#

dcgmi diag runs active tests that exercise deployment, hardware, and software components. A run can select a predefined suite level, one named test, or a comma-separated list of named tests.

Diagnostics can consume substantial GPU, CPU, memory, power, and fabric resources. Coordinate a run with workload owners.

DCGM supports diagnostics on datacenter and non-datacenter GPUs, including GeForce and Quadro GPUs. Non-datacenter GPUs support level 1; use higher levels only where support is explicitly documented for the product. See Feature Availability by GPU Class.

OPTIONS#

Test selection#

-r <suite-or-tests>, --run <suite-or-tests>

Select the diagnostics to run. A single digit selects a predefined suite:

1: Quick deployment checks, normally lasting seconds.
2: Medium validation, normally lasting approximately two minutes.
3: Long hardware diagnostics, normally lasting approximately 15 minutes.
4: Extended, longer-running hardware diagnostics.

The names quick and short, medium, long, and xlong are aliases for levels 1 through 4, respectively. A canonical test name selects one test, and commas select several named tests, for example pcie,diagnostic. See DIAGNOSTIC TESTS for the complete name and suite catalogue.

-p <test.parameter=value>, --parameters <test.parameter=value>

Override a diagnostic parameter. The option is repeatable, and one argument can contain semicolon-separated assignments. A key normally has the form <test-name>.<parameter-name>. A key may be defined only once, except that repeated passthrough_args definitions are combined. The test references in DIAGNOSTIC TESTS list supported parameters and value constraints.

-c <path>, --configfile <path>

Read diagnostic configuration from a local file and send its contents with the request. The file must fit within the DCGM diagnostic configuration size limit. See Diagnostic Configuration File for the complete YAML schema, GPU matching, merge behavior, and precedence rules; see Example configuration for a task-specific example.

Target selection#

-i <entities>, --entity-id <entities>: Run on a comma-separated entity list. See dcgmi for selector syntax. When no group or deprecated GPU list is supplied, the default is all GPUs and all NVIDIA CPUs (*,cpu:*).
-g <group-id>, --group <group-id>: Run on a persistent group. Do not combine with --entity-id or --gpuList.
--gpuList <gpu-id-list>: Run on a comma-separated list of numeric GPU IDs. This compatibility option is deprecated; use --entity-id.
-f <gpu-id-list>, --fakeGpuList <gpu-id-list>: Use fake GPU IDs. This option is for DCGM testing and cannot be combined with a group or normal GPU list.
-n <gpu:N>, --expectedNumEntities <gpu:N>: Fail before testing if the number of active GPUs is not N. This check is supported with the default entity selection or the built-in all-GPU group, not with a custom entity or GPU list.

Execution control#

--iterations <count>: Run the requested diagnostics consecutively count times. The default is 1, and the value must be greater than zero.
-t <seconds>, --timeout <seconds>: Set an overall execution timeout. Zero, the default, means unlimited. When explicit test_duration parameters are present, their sum must be less than the timeout.
--watch-frequency <microseconds>: Set the field-sampling interval. The default is 5,000,000 microseconds; the accepted range is 100,000 through 60,000,000 microseconds.
--enable-heartbeat: Enable the diagnostic heartbeat exchanged with the host engine. This option is available in DCGM 4.4.2 and later. If the initiating client disconnects or stops sending heartbeats, the host engine cancels its active diagnostic. The heartbeat is separate from diagnostic task hang detection, host-engine hang detection, and the overall --timeout limit.

Diagnostic task hang detection#

Diagnostic task hang detection is available in DCGM 4.4 and later. It is enabled by default and monitors registered diagnostic threads and processes for inactivity. NVVS_HANGDETECT_DISABLE disables this mechanism, and NVVS_HANGDETECT_EXPIRY_SEC configures its inactivity expiry in the host engine environment. See Environment for the configuration contract.

Host-engine hang detection is a separate mechanism that is available in DCGM 4.4.2 and later. See Host Engine Environment Variables Affecting Hang Detection.

Failure handling#

--fail-early: Check for failure while targeted-power, targeted-stress, SM-stress, and diagnostic tests are running instead of checking only at completion.
--check-interval <seconds>: Set the early-failure check interval. This option requires --fail-early. The default is 5 seconds and the accepted range is 1 through 300 seconds.
--clocksevent-mask <reasons-or-mask>: Ignore selected clock-event reasons. Supply a comma-separated list of HW_SLOWDOWN, SW_THERMAL, HW_THERMAL, and HW_POWER_BRAKE (case-insensitive), or a decimal mask made from values 8, 32, 64, and 128.
--throttle-mask <reasons-or-mask>: Deprecated alias for --clocksevent-mask. Do not specify both forms.
--ignoreErrorCodes <specification>: Ignore suppressible diagnostic error codes. A comma-separated code list such as 28,140 applies to all supported entities. Semicolon-separated GPU clauses such as gpu0:28;gpu1:140 apply codes to individual GPUs. *:* requests every suppressible code on all entities. DCGM 4.6 supports per-entity clauses for GPUs only. See Diagnostic Errors for the diagnostic error catalogue and response categories.

Output and logging#

-v, --verbose: Include informational and warning messages for each test.
-j, --json: Use JSON output. See JSON output.
--statsonfail: Write plugin statistics files only when a test fails.
--statspath <directory>: Write plugin statistics under the specified server-side directory instead of the diagnostic working directory. The path must be shorter than 128 characters.
--debugLogFile <path>: Set the encrypted server-side diagnostic debug-log path. When a debug level is set without this option, the default file is nvvs.log. The path must be shorter than 128 characters.
-d <level>, --debugLevel <level>: Set diagnostic logging to NONE, FATAL, ERROR, WARN, INFO, DEBUG, or VERB. The default diagnostic severity is DEBUG.

Common options#

--host <address>: Require a connection to the specified host engine.
-h, --help: Print generated usage information and exit.

Removed option#

--train: Training mode has been removed. DCGM 4.6 rejects this option even though it is present in the generated option list.

DIAGNOSTIC TESTS#

Test names#

Use the lowercase names in the following table as the documented canonical values for --run and as the test-name prefix for --parameters. Test-name matching is case-insensitive, and spaces in a requested name are normalized to underscores, but scripts should use the documented spelling.

An explicitly named run also performs the built-in software deployment checks before the requested GPU tests. software is the result name for those built-in checks, but DCGM 4.6 does not accept it as a standalone --run value; use the level-1 suite to run the deployment checks alone. A canonical name does not guarantee that a test is runnable: the host engine constructs the available set from the diagnostic plugins it can load, and a plugin can require particular hardware, entities, libraries, executables, environment variables, or privileges. An unavailable or unsupported requested test produces an error; dcgmi 4.6 does not provide a public operation that lists the tests available at runtime.

Documented diagnostic test names#
Name	Purpose	Included by suite	Availability and reference
`context_create`	Verify that a CUDA context can be created on the target GPU.	Named runs only	Requires the CUDA context-create plugin and a supported GPU. See Context Create Plugin.
`memory`	Test GPU memory allocation, access, integrity, and cache behavior.	2 through 4; `production_testing`	See GPU Memory Plugin.
`pcie`	Exercise PCIe and peer-to-peer paths and check bandwidth, latency, link state, and error counters.	2 through 4; `production_testing`	See PCIe - GPU Bandwidth Plugin.
`diagnostic`	Exercise GPU compute hardware with sustained matrix operations.	3, 4, and `production_testing`	See Diagnostic Plugin.
`memory_bandwidth`	Measure and validate local GPU memory bandwidth.	3, 4, and `production_testing`	See Memory Bandwidth Plugin.
`targeted_stress`	Maintain a requested compute-performance load.	3, 4, and `production_testing`	See Targeted Stress Plugin.
`targeted_power`	Drive a GPU toward a requested power level.	3, 4, and `production_testing`	See Targeted Power Plugin.
`nvbandwidth`	Validate GPU interconnect bandwidth with `nvbandwidth`.	3, 4, and `production_testing`	Requires the plugin and its matching executable. See NVBandwidth Plugin.
`memtest`	Exercise GPU memory with a set of memory-test patterns.	4	See Memtest Diagnostic.
`pulse_test`	Apply a rapidly changing compute and power load.	4	See Pulse Test Diagnostic.

Suite selectors#

Each numbered suite includes the preceding level. Tests marked as conditional in the table above are included only when their plugins advertise them for the running host engine and target entities.

Diagnostic suite selectors#
Accepted selector	Suite	Added tests
`1`, `quick`, `short`	Quick	Built-in `software` deployment checks. See Software Deployment Checks.
`2`, `medium`	Medium	`memory` and `pcie`.
`3`, `long`	Long	`diagnostic`, `memory_bandwidth`, `targeted_stress`, `targeted_power`, and the available conditional level-3 tests.
`4`, `xlong`	Extra long	`memtest` and `pulse_test` in addition to the level-3 set.
`production_testing`	Production meta-suite	The level-3 set with production-test behavior for supported plugins. This selector is not a numbered validation level and does not add the level-4 `memtest` or `pulse_test` tests.

EXIT STATUS#

The following shell statuses identify diagnostic-specific outcomes. They are the low eight bits of the corresponding negative dcgmReturn_t value; see EXIT STATUS for conversion details and for failures common to all dcgmi commands.

Diagnostic-specific shell statuses#
Status	DCGM status	Meaning
`0`	`DCGM_ST_OK`	The requested diagnostics completed without a reported error.
`226`	`DCGM_ST_NVVS_ERROR`	The diagnostic ran but reported an error.
`217`	`DCGM_ST_DIAG_ALREADY_RUNNING`	Another diagnostic is already running.
`215`	`DCGM_ST_DIAG_BAD_LAUNCH`	DCGM could not launch the diagnostic.
`205`	`DCGM_ST_NVVS_ISOLATE_ERROR`	The diagnostic reported a condition that requires isolation.
`204`	`DCGM_ST_NVVS_BINARY_NOT_FOUND`	The diagnostic executable was not found.
`203`	`DCGM_ST_NVVS_KILLED`	The diagnostic process was terminated by a signal.
`198`	`DCGM_ST_NVVS_NO_AVAILABLE_TEST`	No available diagnostic matched the request.

Parser, connection, resource, host-engine, and other DCGM failures can return additional nonzero statuses. Use the status name and message printed by dcgmi as the authoritative reason.

EXAMPLES#

Run the quick suite on all default entities:

$ dcgmi diag --run 1

Run two named tests on GPUs 0 and 1:

$ dcgmi diag --run pcie,diagnostic --entity-id gpu:0,gpu:1

Run the PCIe test three times and return JSON:

$ dcgmi diag --run pcie --iterations 3 --json

Set a test parameter and a ten-minute timeout:

$ dcgmi diag --run targeted_power \
    --parameters targeted_power.test_duration=300 \
    --timeout 600