Memory Bandwidth Plugin#

The Memory Bandwidth plugin detects a GPU whose local framebuffer throughput falls below the configured minimum_bandwidth. Such a result identifies a local-memory performance shortfall for this workload, although competing traffic or an unsuitable threshold can produce the same symptom; the plugin does not assess memory data integrity.

Availability and prerequisites#

The canonical test name is memory_bandwidth. It runs on selected physical GPUs. The plugin requires CUDA support and enough free framebuffer memory for its working set. Run it on an idle GPU so that competing memory traffic does not distort the result.

Behavior#

The CUDA workload is derived from the Triad kernel of the STREAM benchmark [McCalpin1991]. DCGM runs only the Triad-derived workload; it does not run STREAM’s Copy, Scale, or Sum kernels.

For each GPU, the plugin allocates three integer arrays whose combined size is controlled by memory_size_mb. It searches kernel configurations that vary block size, stride, loads per thread, and load method. After selecting the fastest configuration, the plugin runs it repeatedly, discards the first timed iteration, and calculates the reported bandwidth from the shortest remaining elapsed time. The plugin therefore compares the best measured bandwidth, not every timed iteration, with minimum_bandwidth.

The workload consumes framebuffer capacity and memory bandwidth. Use the GPU Memory Plugin or Memtest Diagnostic plugin when the primary question is data integrity rather than bandwidth.

Parameters#

The following table lists the parameters for the Memory Bandwidth plugin.

Parameter	Type	Default	Description
minimum_bandwidth	Double	Product-specific; plugin fallback is 100.0	Minimum bandwidth in MB/s that the best timed iteration must achieve. The installed SKU configuration normally supplies a calibrated threshold.
is_allowed	Bool	See description	Whether this test is allowed to run. Must be “True” for the test to execute. Note: When unspecified, DCGM searches for configuration files for the specified GPU; if no configuration is found, the parameter defaults to “False”.
max_sbe_errors	Double	Unset	Maximum permitted increase in the GPU’s volatile single-bit ECC error (SBE) total during the test. When set, an increase strictly greater than this value fails with `DCGM_FR_SBE_THRESHOLD_VIOLATION`. When unset, no SBE threshold is applied.
memory_size_mb	Double	Automatic	Total working-set size. In DCGM 4.6 the automatic size is 768 MiB per GPU. Explicit values are clamped to the supported range of 6 through 768 MiB.

Examples#

Run a basic memory bandwidth test:

$ dcgmi diag -r memory_bandwidth -p "memory_bandwidth.is_allowed=True"

Run the test with a custom minimum bandwidth threshold:

$ dcgmi diag -r memory_bandwidth -p "memory_bandwidth.is_allowed=True;memory_bandwidth.minimum_bandwidth=500"

Run with a smaller working set when framebuffer capacity is constrained:

$ dcgmi diag -r memory_bandwidth -p "memory_bandwidth.is_allowed=True;memory_bandwidth.memory_size_mb=256"

Results and failure conditions#

Memory Bandwidth results#
Condition	Result or code	Interpretation
The best measured bandwidth is at least `minimum_bandwidth`.	Pass	The GPU met the configured local-memory bandwidth threshold for this workload and working-set size.
The best measured bandwidth is below `minimum_bandwidth`.	`DCGM_FR_MEMORY_BANDWIDTH`; fail	The message identifies the GPU, measured bandwidth, and threshold.
The increase in a GPU’s volatile SBE total is greater than `max_sbe_errors` when the threshold is set.	`DCGM_FR_SBE_THRESHOLD_VIOLATION`; fail	The diagnostic framework compares the counter increase during the test; a value equal to the threshold does not fail.
CUDA setup, allocation, workload execution, or another internal operation fails.	CUDA-specific error or `DCGM_FR_INTERNAL`; fail	The performance comparison did not complete. This is distinct from a measured value below the threshold.
No GPU was selected.	`DCGM_FR_EMPTY_GPU_LIST`; fail	Correct the entity selection and rerun the test.
The plugin is disabled.	`DCGM_FR_TEST_DISABLED`; skip	The workload was not run.
The run is interrupted.	`DCGM_FR_ABORTED`; skip	No completed bandwidth result was produced.

References#

[McCalpin1991]

John D. McCalpin, STREAM: Sustainable Memory Bandwidth in High Performance Computers, continually updated technical report (1991–2007).