Memtest Diagnostic

The memtest plugin detects framebuffer-memory corruption by writing and verifying configurable data and address patterns. Its patterns exercise address selection, stored bit values, block movement, retention, and sustained memory access to expose pattern-dependent memory faults.

Availability and prerequisites

The canonical test name is memtest. It runs on selected physical GPUs. The plugin requires enough free and allocatable framebuffer memory to satisfy minimum_allocation_percentage. Run it on idle GPUs so that competing allocations do not prevent testing.

Behavior

The pattern set is similar to the algorithms described by MemTest86.

Test patterns

Note

Runtimes are averages for one iteration on a single A100 40 GB GPU.

test0 — Walking 1 bit

This test changes one bit at a time in memory to determine whether it affects a different memory location. It is designed to test the address wires.

Average runtime:

Approximately 3 seconds.

test1 — Address check

Each memory location is filled with its own address, then checked to confirm that its value still matches the address.

Average runtime:

Less than 1 second.

test2 — Moving inversions, ones and zeros

This test uses the moving-inversions algorithm from MemTest86 with patterns of all ones and all zeros.

Average runtime:

Approximately 4 seconds.

test3 — Moving inversions, 8-bit pattern

This test uses the same algorithm as test1 with an 8-bit-wide pattern of walking ones and zeros.

Average runtime:

Approximately 4 seconds.

test4 — Moving inversions, random pattern

This test uses the same algorithm as test1, but the data pattern is a random number and its complement. It uses 60 patterns. The random-number sequence changes with each pass, so multiple passes can increase the test’s effectiveness.

Average runtime:

Approximately 2 seconds.

test5 — Block move, 64 moves

This test initializes memory with shifting patterns that are inverted every 8 bytes. It then moves the memory blocks and checks the data patterns after the moves are complete.

Average runtime:

Approximately 1 second.

test6 — Moving inversions, 32-bit pattern

This variation of the moving-inversions algorithm shifts the data pattern left by one bit for each successive address. The test makes 32 passes to use all possible data patterns.

Average runtime:

Approximately 155 seconds.

test7 — Random-number sequence

This test initializes a 1 MB block of memory with random patterns. It uses these patterns and their complements in moving-inversion tests across the rest of memory.

Average runtime:

Approximately 2 seconds.

test8 — Modulo 20, random pattern

This test generates a random pattern and writes it to every 20th memory location. It writes the pattern’s complement to the remaining locations. It repeats this process 20 times, shifting the locations that receive the pattern each time.

Average runtime:

Approximately 10 seconds.

test9 — Bit fade, two patterns

This test initializes all memory with a pattern, waits for one minute, and then examines memory for changed bits. It tests both all-one and all-zero patterns.

Average runtime:

Approximately 244 seconds.

test10 — Memory stress

This test generates a random pattern and launches a large kernel that sets all memory to that pattern. It then launches a read-and-write kernel to check for errors and set memory to the pattern’s complement. The process repeats 1,000 times for one pattern. The kernel is designed to maximize bandwidth to GPU global memory.

Average runtime:

Approximately 6 seconds.

Note

By default, test7 and test10 alternate for 10 minutes. The diagnostic fails if either test detects an error.

Parameters

Pattern selection

Memtest pattern parameters

Parameter	Type	Default	Description
test0	Boolean	false	Enable the walking-1-bit test.
test1	Boolean	false	Enable the address-check test.
test2	Boolean	false	Enable the moving-inversions ones-and-zeros test.
test3	Boolean	false	Enable the moving-inversions 8-bit-pattern test.
test4	Boolean	false	Enable the moving-inversions random-pattern test.
test5	Boolean	false	Enable the block-move test.
test6	Boolean	false	Enable the moving-inversions 32-bit-pattern test.
test7	Boolean	true	Enable the random-number-sequence test.
test8	Boolean	false	Enable the modulo-20 random-pattern test.
test9	Boolean	false	Enable the bit-fade test.
test10	Boolean	true	Enable the memory-stress test.

Execution and allocation

Memtest execution parameters

Parameter	Type	Default	Description
is_allowed	Boolean	true	Allow memtest to run. false produces a skipped result.
test_duration	Number	600	Number of seconds to run the enabled pattern sequence.
minimum_allocation_percentage	Number	75	Minimum percentage of total framebuffer memory that must be free and allocatable. The GPU is skipped when this requirement cannot be met.
num_chunks	Integer	1	Divide the tested allocation into this many chunks. This can help when one contiguous allocation is impractical.
use_mapped_mem	Boolean	false	Use CUDA mapped memory rather than native device-memory allocation.

Configuration

SKU map:

skus[].memtest

SKU-file fields:

is_allowed, test_duration, num_chunks, and minimum_allocation_percentage

Request-only fields:

test0 through test10, pattern, and use_mapped_mem

Common fields:

ignore_error_codes; see Diagnostic Configuration File

Set request-only fields with --parameters rather than placing them in a dcgm-diag-v1 SKU overlay.

Examples

Run test7 and test10 for 10 minutes (this is the default):

$ dcgmi diag --run memtest

Run each test serially for 1 hour then display results:

$ dcgmi diag --run memtest \
   -p memtest.test0=true\;memtest.test1=true\;memtest.test2=true\;memtest.test3=true\;memtest.test4=true\;memtest.test5=true\;memtest.test6=true\;memtest.test7=true\;memtest.test8=true\;memtest.test9=true\;memtest.test10=true\;memtest.test_duration=3600

Run test0 for one minute 10 times, displaying the results each minute:

$ dcgmi diag \
   --iterations 10 \
   --run memtest \
   -p memtest.test0=true\;memtest.test7=false\;memtest.test10=false\;memtest.test_duration=60

Results and failure conditions

Memtest results

Condition	Result or code	Interpretation
Any enabled pattern test reads a value different from the expected value.	DCGM_FR_MEMORY_MISMATCH; fail	Memtest detected one or more framebuffer-memory miscompares on the affected GPU.
The plugin cannot initialize or encounters an unexpected runtime exception.	DCGM_FR_INTERNAL or another execution-specific error; fail	The test did not complete; inspect the accompanying message and log before treating the result as a memory defect.
No GPU was selected.	DCGM_FR_EMPTY_GPU_LIST; fail	Correct the entity selection and rerun the test.
Total memory is too small, less than minimum_allocation_percentage is free, or no suitable test allocation can be made.	Skip for the affected GPU	No memory-integrity result was produced. Stop competing workloads or adjust the allocation requirement when appropriate.
is_allowed=false.	DCGM_FR_TEST_DISABLED; skip	Memtest was disabled by configuration.
The run is interrupted.	DCGM_FR_ABORTED; skip	The enabled pattern sequence did not complete.

See also

dcgmi diag, Diagnostics, Diagnostic Errors