mnubergemm#

Purpose and availability#

mnubergemm combines pulsed single-precision (FP32) matrix multiplication, GPU copy-engine work, and communication through the NVIDIA Collective Communications Library (NCCL) in one MPI job across the participating hosts. Use it to reproduce failures that appear only under combined compute and communication load and are not yet isolated to one GPU pair. It is the default multi-node diagnostic and is selected explicitly with:

$ dcgmi mndiag --hostList 'node01;node02' --run mnubergemm

It is available for GB200 NVL in DCGM 4.3 and later; GB300 NVL support was added in DCGM 4.4. The built-in eligible devices are GB200 NVL (PCI device ID 2941) and GB300 NVL (PCI device ID 31c2). Every participating host must have active GPUs, and all participating GPUs must use the same eligible SKU.

DCGM packages mnubergemm binaries for CUDA 12 and CUDA 13 on x86-64 and Arm64. It selects the CUDA 12 binary for a CUDA driver API major of 12 and the CUDA 13 binary for a major of 13 or later. A driver API major below 12 has no compatible packaged binary.

Resource and launch model#

mnubergemm is an active stress workload. The host list selects hosts and transports; it does not select GPU subsets. DCGM derives one common processes-per-node count from the active-GPU count visible to the head node. Drain every active GPU on every participating host.

The run uses the common multi-node launch environment: reachable and authorized host engines, Open MPI, noninteractive SSH for the caller-derived execution account, accessible CUDA-specific workload binaries, and a configured Multi-Node NVLink/IMEX environment. These operational prerequisites and the distinction between the dcgmi process user, head host-engine process user, and configured service account are described in Whose identity launches the workload. The exact host, MPI, and environment controls are in dcgmi mndiag.

DCGM reserves the multi-node diagnostic resource on every host before launch. Other activity on the participating GPUs can prevent reservation or launch; that is an orchestration failure, not a hardware verdict.

Parameters#

Supported DCGM parameter#

mnubergemm parameter#
Parameter	`dcgmi` form	Default	Validation and effect
`mnubergemm.time_to_run`	Valued assignment	`3600` seconds	A positive decimal integer from 1 through 2,147,483,647. It sets the workload’s `--time_to_run` value and DCGM’s run budget. Omit the parameter to use the default. An empty value, non-integral value, zero, negative value, or out-of-range value returns `DCGM_ST_BADPARAM`.

For example, use --parameters mnubergemm.time_to_run=600.

DCGM adds 60 seconds of startup allowance to the normal process-monitor deadline. Thus time_to_run=600 produces a 660-second process-execution deadline. Connection, authorization, reservation, and cleanup are separate phases.

Examples#

Run the default one-hour diagnostic:

$ dcgmi mndiag --hostList 'node01;node02' --run mnubergemm

Run it for ten minutes and request JSON:

$ dcgmi mndiag --hostList 'node01;node02' --run mnubergemm \
    --parameters mnubergemm.time_to_run=600 --json

Results#

mnubergemm diagnostic results#
Workload outcome	Diagnostic result	Meaning
The workload completes, provides usable results for the participating hosts and GPUs, and reports no error.	Overall and GPU results are `Pass`.	The combined compute, copy-engine, and NCCL workload completed without reporting an error. This is not an exhaustive statement of system health.
The workload reports an error for one or more GPUs.	Affected GPUs and the overall result are `Fail`.	DCGM associates the workload message with each affected GPU identified by the workload. The message is evidence for investigation; it does not by itself identify a defective component.
The workload does not provide a usable result for the complete participating host and GPU inventory.	Overall `Fail`.	The run did not establish a result for the requested configuration; inspect the captured workload logs.
The MPI workload exits unsuccessfully.	Overall `Fail`.	The command reports the MPI failure; inspect the captured stdout and stderr files.

mnubergemm results use only Pass and Fail. The diagnostic does not apply a separate bandwidth or error-rate threshold. A failure’s workload message is the evidence to interpret.

Failures before or around a result#

mnubergemm orchestration failures#
Condition	Status	Effect
A participating host has no active GPUs, the participating GPUs do not use one SKU, or the SKU is not supported.	`DCGM_ST_NOT_SUPPORTED`	The workload is not launched.
The head cannot connect to or authorize a participating host engine.	Connection, SSH, or authorization status	No diagnostic verdict is established.
A participating node cannot reserve the required diagnostic resource because it is already in use.	`DCGM_ST_IN_USE`	Wait for the conflicting activity to finish, or stop it, and retry.
`time_to_run` is invalid.	`DCGM_ST_BADPARAM`	Correct the request.
No readable, executable workload binary can be selected.	`DCGM_ST_INIT_ERROR`	Correct the package installation.
MPI cannot be launched, the expected workload process cannot be detected, or another GPU process is detected after launch.	Launch status such as `DCGM_ST_CHILD_SPAWN_FAILED`	Inspect MPI, SSH, service, and workload logs.
The deadline expires or the client interrupts the run.	Nonzero timeout or signal status	The run is incomplete and does not establish a pass.

A diagnostic Fail can accompany command status zero. The nonzero statuses above are command failures. See EXIT STATUS.

Executable and logs#

mnubergemm executable and log controls#
Item	Behavior
Default executable	`mnubergemm` under the installed `plugins/cuda12` or `plugins/cuda13` directory selected from the driver API major.
Captured stdout	`/var/log/nvidia-dcgm/dcgm_mndiag_mnubergemm_stdout.txt`.
Captured stderr	`/var/log/nvidia-dcgm/dcgm_mndiag_mnubergemm_stderr.txt`.

DCGM_HOME_DIR replaces /var/log/nvidia-dcgm as the log directory. An empty-but-present value makes the filenames relative to the host engine’s current working directory; leave the variable unset to use the default. Set it in the head host-engine process environment.