mnubergemm#

Purpose#

mnubergemm is the canonical name of the multi-node diagnostic supplied by DCGM 4.6. The head-node host engine launches a coordinated MPI workload on the requested hosts. The workload actively uses the participating GPUs and their multi-node data paths; DCGM collects the workload’s host, GPU, and error records into one diagnostic response.

This is a multi-node diagnostic, not a dcgmi diag plugin. Select it with dcgmi mndiag --run mnubergemm.

Availability#

mnubergemm availability#
Property	Value
First DCGM release	DCGM 4.3.0
Supported products in DCGM 4.6	GB200 NVL (PCI device ID `2941`) and GB300 NVL Bianca (PCI device ID `31c2`)
Selection cardinality	One multi-node diagnostic per `dcgmi mndiag` invocation
Result entities	An overall result and GPU results associated with the hosts reported by the workload

The active GPUs on a participating system must belong to the supported multi-node configuration. DCGM rejects a head node whose active GPUs have mixed SKUs or whose GPU SKU is not in the supported set.

Prerequisites and resource use#

mnubergemm is an active stress workload. Drain the requested GPUs from production work before running it. The host engines must be able to reserve the GPUs and must not find a conflicting MPI workload using them.

The run also requires the common multi-node environment: compatible DCGM and driver installations on every node, Open MPI, noninteractive SSH for the host-engine service account, reachable DCGM and MPI transports, and correctly configured IMEX channels on the supported multi-node NVLink system. The host-engine service account must be able to read and execute the workload and MPI binaries. See Prerequisites and Test Setup for the setup procedure.

Parameters#

DCGM 4.6 recognizes and validates the following customer-facing mnubergemm parameter:

mnubergemm parameters#
Parameter	Type	Default	Meaning and constraints
`mnubergemm.time_to_run`	Integer seconds	`3600`	Requested workload duration. The value must be greater than zero.

The process monitor allows an additional 60 seconds for workload startup before it terminates an overlong process. Thus a 600-second workload has a 660-second process-execution limit; connection, reservation, launch, and cleanup time are separate.

The common command-line parser can accept other syntactically valid mnubergemm.* assignments, and the runner may forward matching assignments to the bundled workload. DCGM does not validate those names or values as part of its public interface. The runner also supplies internal workload settings internally. Neither behavior makes those settings supported parameters; do not depend on them across DCGM releases.

Examples#

Run the default one-hour diagnostic on every GPU on two hosts:

$ dcgmi mndiag --hostList 'node01;node02' --run mnubergemm

Run for ten minutes on GPUs 0 through 3 on each host:

$ dcgmi mndiag \
    --hostList 'node01=0-3;node02=0-3' \
    --run mnubergemm \
    --parameters mnubergemm.time_to_run=600

See dcgmi mndiag for the complete host syntax and command options.

Results and failure conditions#

The DCGM 4.6 adapter recognizes information and error records produced by the workload. It uses those records, together with the MPI process status, to construct the result:

mnubergemm results#
Condition	Result or code	Interpretation
The output contains recognized host and GPU records and no workload error record.	Pass	The workload completed without reporting an error for a parsed GPU. This is evidence about this workload and run, not an exhaustive GPU health statement.
The workload emits one or more GPU error records.	Affected GPU and overall diagnostic fail; `DCGM_FR_UNKNOWN` with category `DCGM_FR_EC_HARDWARE_OTHER` and triage severity	DCGM preserves the workload’s error text for the affected host and GPU. Use that text to identify the reported operation or condition.
DCGM expected participating hosts but could not recognize any host in the workload output.	Overall diagnostic fail; `DCGM_FR_INTERNAL`	No usable per-host result was produced. Inspect the `mnubergemm` stdout and stderr logs named by DCGM.
The MPI process exits with a nonzero status.	Overall diagnostic fail	The result includes the MPI exit code and the paths of the captured stdout and stderr logs. This is an execution failure and does not by itself identify defective hardware.
The process exceeds `time_to_run` plus the 60-second startup allowance or is interrupted.	Failed or incomplete run	DCGM terminates or stops the workload and does not establish a passing diagnostic result.

The current adapter produces Pass or Fail results; it does not derive Warn or Skip from mnubergemm output. It also does not independently classify low bandwidth, memory corruption, NVLink faults, or similar defect types. Such a condition is a reported finding only when the workload emits an error record describing it. This distinction is why the workload’s error message is more authoritative than a generic list of possible stress-test failures.

Errors before the workload result#

Some failures prevent mnubergemm from producing a diagnostic result:

mnubergemm setup and orchestration errors#
Condition	Typical DCGM status	Meaning
The active GPUs use mixed SKUs or an unsupported SKU.	`DCGM_ST_NOT_SUPPORTED`	The diagnostic is not available for that system configuration.
GPUs cannot be reserved, another multi-node run owns the node, or a conflicting MPI process is detected.	`DCGM_ST_IN_USE`	Drain the resources and verify that no stale diagnostic or MPI process remains.
`mnubergemm.time_to_run` is missing its value, is not an integer, or is zero or negative.	`DCGM_ST_BADPARAM`	Correct the parameter before retrying.
The workload executable cannot be resolved or is inaccessible to the host-engine service account.	`DCGM_ST_INIT_ERROR`	Verify the installation and the service account’s executable and directory permissions.
MPI cannot launch, or DCGM cannot detect the expected workload process on every participating node.	`DCGM_ST_CHILD_SPAWN_FAILED` or the underlying launch status	Inspect the MPI, SSH, host-engine, and captured workload logs. No hardware verdict was established.
A participating host engine cannot be reached or cannot authorize the head node.	Connection- or authorization-specific nonzero status	Correct the listener, address, SSH tunnel, firewall, or concurrent head-node condition before interpreting the run as a diagnostic.

Command status and diagnostic result remain separate. In DCGM 4.6, a completed request can return command status zero while the MNUBERGEMM diagnostic result is Fail. See the exit-status discussion in dcgmi mndiag before using the result in automation. JSON output is release-specific and does not have a stable schema; see JSON output.