GPU Memory Plugin#

The GPU Memory plugin detects data-integrity faults in framebuffer memory and, when enabled, L1 cache. It writes known values and verifies their readback, and it reports uncorrectable error-correcting code (ECC) conditions that prevent memory operations; it is not a memory-bandwidth test.

Availability and prerequisites#

The canonical test name is memory. The framebuffer test requires the expected ECC capability and state and enough free GPU memory for the requested allocation. The optional L1 cache test requires compute capability 7.0 or later and no more than 256 KB of L1 cache per streaming multiprocessor.

Behavior#

Framebuffer memory test#

The main memory test:

Allocates a significant portion of GPU memory, 75% by default.
Writes test patterns to memory using CUDA kernels.
Reads the data back and verifies its integrity.
Uses the patterns 0x00, 0xAA, 0x55, 0xFF, and 0x00.
Detects memory mismatches and ECC errors.

L1 cache test#

When l1_is_allowed is enabled, the plugin performs cache operations and validates L1 cache data integrity on supported GPUs.

Parameters#

Plugin control#

The following table lists the global parameter for the memory plugin.

Parameter	Type	Default	Description
is_allowed	Bool	When unspecified, DCGM will search configuration files for an appropriate default for the specified GPU; if no entry is found, the parameter defaults to false.	Specifies whether or not this test is allowed to run.

Framebuffer memory parameters#

The following table lists the parameters for the framebuffer memory test.

Parameter	Type	Default	Description
minimum_allocation_percentage	Double	75.0	Minimum percentage of GPU memory to allocate (0.0-100.0).
max_free_memory_mb	Double	0.0	Maximum amount of GPU memory which will be allocated for memory testing, specified in megabytes (MB). Supersedes minimum_allocation_percentage. If not specified, the test will allocate value of minimum_allocation_percentage for the GPU memory.

L1 cache parameters#

The following table lists the parameters for the L1 cache test.

Parameter	Type	Default	Description
l1_is_allowed	Bool	When unspecified, DCGM will search configuration files for an appropriate default for the specified GPU; if no entry is found, the parameter defaults to false.	Enables/disables the L1 cache subtest.
test_loops	Double	0	Number of test loops.
test_duration	Double	1.0	Duration of L1 cache test in seconds (0 = run for test_loops).
inner_iterations	Double	1024	Number of inner iterations per test.
log_len	Double	8192	Length of error log for L1 cache test.
dump_miscompares	Bool	True	Whether to dump miscompare details.
l1cache_size_kb_per_sm	Double	0	L1 cache size per SM in KB.

Examples#

Run the basic framebuffer memory test:

$ dcgmi diag -r memory -p "memory.is_allowed=True"

Enable the L1 cache test:

$ dcgmi diag -r memory -p "memory.is_allowed=True;memory.l1_is_allowed=True"

Set max_free_memory_mb to 0.1:

$ dcgmi diag -r memory -p "memory.is_allowed=True;memory.max_free_memory_mb=0.1"

Set max_free_memory_mb to 512:

$ dcgmi diag -r memory -p "memory.is_allowed=True;memory.max_free_memory_mb=512"

Set max_free_memory_mb to 100:

$ dcgmi diag -r memory -p "memory.is_allowed=True;memory.max_free_memory_mb=100"

Results and failure conditions#

Memory plugin results#
Condition	Result or code	Interpretation
Repeated allocation attempts cannot satisfy `minimum_allocation_percentage`.	`DCGM_FR_MEMORY_ALLOC`; fail	The requested amount was available initially but the plugin could not allocate enough memory to execute the test.
Data read from framebuffer memory differs from the value written.	`DCGM_FR_MEMORY_MISMATCH`; fail	The main memory test detected data corruption.
CUDA reports an uncorrectable double-bit ECC error.	`DCGM_FR_CUDA_DBE`; fail	CUDA could not complete the memory operation because of an uncorrectable ECC condition.
The enabled L1 cache subtest reads an unexpected value.	`DCGM_FR_L1TAG_MISCOMPARE`; fail	The L1 cache subtest detected a data miscompare.
DCGM cannot query ECC state, CUDA cannot initialize or resolve the GPU, or the plugin encounters an unexpected execution error.	`DCGM_FR_DCGM_API`, `DCGM_FR_CUDA_API`, `DCGM_FR_CUDA_DEVICE`, or `DCGM_FR_INTERNAL`; fail	The test could not complete. These codes do not by themselves identify faulty memory.
Less memory is free than `minimum_allocation_percentage` requires before testing begins.	Skip	Stop workloads using the GPU or lower the requirement if that is appropriate. No memory-integrity result was produced.
ECC is unsupported or disabled on the selected GPU.	`DCGM_FR_ECC_UNSUPPORTED` or `DCGM_FR_ECC_DISABLED`; skip	The plugin does not run its main memory check without the expected ECC capability and state.
The enabled L1 subtest is unsupported by the GPU.	`DCGM_FR_L1TAG_UNSUPPORTED`; subtest skipped	The main framebuffer-memory result remains independently meaningful.
The plugin is disabled or the run is interrupted.	`DCGM_FR_TEST_DISABLED` or a skipped result	The test did not complete and did not establish memory health.