Targeted Stress Plugin#

The Targeted Stress plugin detects a GPU that cannot achieve a configured compute-performance target under sustained matrix work. It also reports memory, thermal, PCIe, and other GPU errors observed under that load, so it is primarily a test of compute performance and stability.

Availability and prerequisites#

The canonical test name is targeted_stress. It runs on selected physical GPUs. The plugin requires CUDA support and should run on idle GPUs so that competing compute work does not distort the achieved-performance result.

Behavior#

The Targeted Stress plugin performs the following operations:

Matrix Multiplication Operations: Continuously runs GEMM (General Matrix Multiply) operations using either single precision (SGEMM) or double precision (DGEMM), according to the configured use_dgemm value.
Performance Targeting: Monitors actual GPU performance and adjusts the workload to maintain the specified target performance level in GFLOPS.
Multi-Stream Processing: Utilizes multiple CUDA streams per GPU to pipeline operations and maximize GPU utilization while maintaining the target stress level.
Performance Validation: Ensures that the achieved performance meets the minimum ratio threshold of the target performance, accounting for memory transfer overhead.
Health Monitoring: Continuously monitors GPU health metrics including temperature, memory errors, PCIe replays, and other standard error conditions.

Parameters#

The following table lists the global parameters for the targeted stress plugin:

Parameter	Type	Default	Description
test_duration	Double	30.0	Duration of the test in seconds.
target_stress	Double	Product-specific; plugin fallback is 100.0	Target performance level in GFLOPS. The installed SKU configuration normally supplies the calibrated target.
target_perf_min_ratio	Double	0.95	Minimum ratio of achieved performance to target performance (0.0 to 1.0).
temperature_max	Double	See description	Accepted and stored for compatibility, but neither Targeted Stress nor shared error checking uses this value to determine the thermal threshold. DCGM instead uses the GPU’s reported maximum operating temperature, falling back to its slowdown temperature when necessary. Do not use this parameter to impose a site-specific limit.
is_allowed	Bool	See description	Whether the targeted stress test is allowed to run. Must be “True” for the test to execute. Note: When unspecified, DCGM searches for configuration files for the specified GPU; if no configuration is found, the parameter defaults to “False”.
use_dgemm	Bool	True	Use double precision GEMM (DGEMM) instead of single precision (SGEMM).
cuda_streams_per_gpu	Double	8.0	Number of CUDA streams to use per GPU for pipelining operations.
ops_per_stream_queue	Double	100.0	Number of operations to queue per stream before waiting for completion.
max_pcie_replays	Double	160.0	Maximum allowed PCIe replay count during the test.
max_memory_clock	Double	0.0	Accepted for compatibility. The DCGM 4.6 Targeted Stress plugin does not use this parameter to set clocks or determine the result.
max_graphics_clock	Double	0.0	Accepted for compatibility. The DCGM 4.6 Targeted Stress plugin does not use this parameter to set clocks or determine the result.
max_sbe_errors	Double	Blank	Threshold beyond which SBE (Single Bit Error) count constitutes a failure.
always_use_tensor	Bool	False	Use Tensor Cores for GEMM operations when the selected GPU supports them.

Examples#

Run a quick targeted stress test for 2 minutes:

$ dcgmi diag -r targeted_stress -p targeted_stress.test_duration=120.0

Run the targeted stress test targeting 200 GFLOPS performance:

$ dcgmi diag -r targeted_stress -p targeted_stress.target_stress=200.0

Run the targeted stress test with single precision operations:

$ dcgmi diag -r targeted_stress -p targeted_stress.use_dgemm=False

Run the targeted stress test with stricter performance requirements (98% of target):

$ dcgmi diag -r targeted_stress -p targeted_stress.target_perf_min_ratio=0.98

Run the targeted stress test with custom stream configuration:

$ dcgmi diag -r targeted_stress -p "targeted_stress.cuda_streams_per_gpu=4.0;targeted_stress.ops_per_stream_queue=50.0"

Run the targeted stress test as part of level 3 diagnostics:

$ dcgmi diag -r 3 -p targeted_stress.test_duration=120.0

Results and failure conditions#

Targeted Stress results#
Condition	Result or code	Interpretation
Maximum achieved performance is below the effective `target_perf_min_ratio * target_stress` threshold.	`DCGM_FR_STRESS_LEVEL`; fail	The effective threshold accounts for time spent transferring data rather than performing GEMM work.
DCGM cannot obtain the performance statistic needed for evaluation.	`DCGM_FR_CANNOT_GET_STAT`; fail	The plugin cannot make its target-performance determination.
A common monitored condition occurs, including an uncorrectable memory error, XID, failed or pending row remap, thermal violation, an enabled SBE threshold, or the configured PCIe replay threshold.	Fail with the condition-specific `DCGM_FR_*` code	See Diagnostic Errors for the reported condition.
CUDA setup or another internal workload operation fails.	`DCGM_FR_CUDA_API` or `DCGM_FR_INTERNAL`; fail	The workload did not complete. This is distinct from failing the performance threshold.
The plugin is disabled.	`DCGM_FR_TEST_DISABLED`; skip	Numbered-suite or product configuration did not allow the workload.
The run is interrupted.	`DCGM_FR_ABORTED`; skip	No completed stress result was produced.