Diagnostic Plugin#

The Diagnostic plugin seeks to expose GPU compute or framebuffer data-path instability under sustained load. It verifies matrix results and framebuffer data, reports monitored health errors observed during the workload, and can identify cross-GPU outliers in throughput, clocks, or board power when the corresponding tolerances are enabled.

Availability and prerequisites#

The canonical test name is diagnostic. It runs on selected physical GPUs. The plugin requires CUDA support and should run on GPUs that are not serving production workloads because it applies sustained compute, memory, and power load.

Behavior#

This process stresses the GPU by drawing substantial power and providing a high level of throughput for three minutes by default. During this process, the GPU will be monitored for all standard errors (XIDs, temperature violations, uncorrectable memory errors, etc.) as well as the correctness of data being written and read.

Parameters#

The following table lists the global parameters for the diagnostic plugin:

Parameter	Type	Default	Description
max_sbe_errors	Double	Blank	This is the threshold beyond which SBE’s are treated as errors.
test_duration	Double	180.0	This is the time in seconds that the test should run.
use_doubles	Bool	False	This indicates doubles should be used instead of floats.
temperature_max	Double	See description	Accepted for compatibility. In DCGM 4.6, common error monitoring uses the GPU’s reported maximum operating temperature rather than this parameter. Do not use it to impose a lower site-specific limit.
is_allowed	Bool	False	This is whether the specified test is allowed to run.
matrix_dim	Double	2048.0 (8192.0 for Blackwell and newer)	This is the starting dimension of the matrix used for S/Dgemm.
precision	String	Automatic	Selects one or more precisions. The value is matched case-insensitively for `h` (half), `s` (single), and `d` (double), so values such as `half,single` are accepted. When the parameter is omitted, DCGM selects the precisions supported by the GPU.
gflops_tolerance_pcnt	Double	0.0	Maximum permitted fractional shortfall from the mean GFLOPS of the tested GPUs. For example, `0.20` reports a GPU below 80% of the mean. `0.0` disables this comparison.
always_use_tensor	Bool	False (True for Blackwell and newer)	Instructs CUDA to use Tensor cores for calculations where possible.
tolerance_pcnt	Double	0.0	Default fractional tolerance for both clock and power comparisons. The field-specific parameters take precedence.
clocks_tolerance_pcnt	Double	0.0	Maximum permitted fractional shortfall from the mean SM clock of the tested GPUs. `0.0` disables this comparison.
power_tolerance_pcnt	Double	0.0	Maximum permitted fractional shortfall from the mean board power of the tested GPUs. `0.0` disables this comparison.

Examples#

Run the diagnostic for 5 minutes:

$ dcgmi diag -r 3 -p diagnostic.test_duration=300.0

Run the diagnostic, with a smaller starting dimension for matrix operations:

$ dcgmi diag -r 3 -p diagnostic.matrix_dim=1024.0

Run the diagnostic, reporting an error if a GPU is more than 60% below the mean GFLOPS across all tested GPUs:

$ dcgmi diag -r 3 -p diagnostic.gflops_tolerance_pcnt=0.60

Run the diagnostic, using double precision:

$ dcgmi diag -r 3 -p diagnostic.precision=double

Results and failure conditions#

Diagnostic plugin results#
Condition	Result or code	Interpretation
A computed value does not match the expected value.	`DCGM_FR_FAULTY_MEMORY`; fail	The compute or framebuffer data path produced incorrect data.
A computation produces a NaN where a valid value is expected.	`DCGM_FR_NAN_VALUE`; fail	The numerical workload produced an invalid result.
GFLOPS, average SM clock, or average board power falls below an enabled cross-GPU tolerance.	`DCGM_FR_GFLOPS_THRESHOLD_VIOLATION`; fail	The message names the metric and affected GPU. This is a comparative threshold, not an absolute performance guarantee.
A common monitored condition occurs, including an uncorrectable memory error, XID, failed or pending row remap, thermal violation, or an enabled SBE threshold violation.	Fail with the condition-specific `DCGM_FR_*` code	See Diagnostic Errors; the reported code identifies the monitored condition.
CUDA is not bound to the expected device.	`DCGM_FR_CUDA_UNBOUND`; fail	Check device visibility and the diagnostic process environment.
CUDA initialization, context creation, or another test operation cannot be completed.	`DCGM_FR_CUDA_API` or `DCGM_FR_INTERNAL`; fail	This is an execution failure and does not establish a hardware defect by itself.
The plugin is disabled.	`DCGM_FR_TEST_DISABLED`; skip	Enable it through the applicable diagnostic configuration or run a suite for which it is enabled.
The run is interrupted.	`DCGM_FR_ABORTED`; skip	The workload did not complete.