Diagnostic Plugin

Overview

The Diagnostic plugin is part of the level 3 tests. It performs large matrix multiplies while copying data to various addresses in the frame buffer and checking that the data can be written and read correctly.

This test performs large matrix multiplications; by default it will alternate running these multiplications at all available among 64, 32, and 16-bit precisions. It will also walk the frame buffer, writing values to different addresses and making sure that the values are written and read correctly.

Test Description

This process will stress the GPU by having it draw a large amount of power and provide a high-level of throughput for five minutes (by default). During this process, the GPU will be monitored for all standard errors (XIDs, temperature violations, uncorrectable memory errors, etc.) as well as the correctness of data being written and read.

Supported Parameters

The following table lists the global parameters for the diagnostic plugin:

Parameter Name	Type	Default	Description
max_sbe_errors	Double	Blank	This is the threshold beyond which SBE’s are treated as errors.
test_duration	Double	180.0	This is the time in seconds that the test should run.
use_doubles	String	False	This indicates doubles should be used instead of floats.
temperature_max	Double	30.0	This is the maximum temperature in degrees allowed during the test.
is_allowed	Bool	False	This is whether the specified test is allowed to run.
matrix_dim	Double	2048.0 (8192.0 for Blackwell and newer)	This is the starting dimension of the matrix used for S/Dgemm.
precision	String	Half Single Double	This is the precision to use: half, single, or double
gflops_tolerance_pcnt	Double	0.0	This is the percent of mean below which gflops are treated as errors. The mean is the mean for all GPUs on the system.
always_use_tensor	Bool	False (True for Blackwell and newer)	Instructs CUDA to use Tensor cores for calculations where possible.
tolerance_pcnt	Double	0.0	This is the percent of mean below which both clock speed and power usage would be treated as errors. If specified with clocks_tolerance_pcnt or power_tolerance_pcnt, those settings take precedence.
clocks_tolerance_pcnt	Double	0.0	This is the percent of mean below which clock speed would be treated as errors. The mean is the mean clock speed throughout the test for each GPU on the system.
power_tolerance_pcnt	Double	0.0	This is the percent of mean below which power usage would be treated as errors. The mean is the mean power usage throughout the test for each GPU on the system.

Sample Commands

Run a quick diagnostic:

Run the diagnostic for 5 minutes:

$ dcgmi diag -r 3 -p diagnostic.test_duration=300.0

Run the diagnostic, stopping if max temperature exceeds 28 degrees:

$ dcgmi diag -r 3 -p diagnostic.temperature_max=28.0

Run the diagnostic, with a smaller starting dimension for matrix operations:

$ dcgmi diag -r 3 -p diagnostic.matrix_dim=1024.0

Run the diagnostic, reporting an error if a GPU reports gflops not within 60% of the mean gflops across all GPUs:

$ dcgmi diag -r 3 -p diagnostic.gflops_tolerance_pcnt=0.60

Run the diagnostic, using double precision:

$ dcgmi diag -r 3 -p diagnostic.precision=double

Failure Conditions

• The test will fail if unrecoverable memory errors, temperature violations, or XIDs occur during the test.