Multi-Node Diagnostics#

The multi-node diagnostics module coordinates a named diagnostic across multiple DCGM host engines. These diagnostics are selected with dcgmi mndiag --run. They are distinct from the single-node diagnostic plugins selected with dcgmi diag --run and should not be described as a plugin interface.

DCGM 4.6 accepts one multi-node diagnostic per request and provides one diagnostic, mnubergemm. Keeping the diagnostic catalogue separate from the command reference distinguishes the multi-node framework from the diagnostic that happens to be available in the current release.

Available diagnostics#

Multi-node diagnostic reference#

Canonical name

Purpose

Availability in DCGM 4.6

mnubergemm

Run a coordinated MPI workload across the selected hosts and GPUs and report GPU-scoped errors emitted while the workload exercises the supported multi-node system.

GB200 NVL and GB300 NVL systems

The multi-node diagnostic module was introduced in DCGM 4.3. GB300 NVL support was added in DCGM 4.4. The diagnostic page gives the complete version, product, parameter, result, and failure-condition reference.

Selection and parameters#

The --run value is a diagnostic name from the table above. The dcgmi 4.6 client compares the name without regard to case and defaults to mnubergemm when --run is omitted. It does not provide a command that enumerates multi-node diagnostics at runtime; this catalogue supplies the canonical names for the documented release.

Parameters use the form diagnostic.parameter=value. Syntactic acceptance by the common dcgmi parameter parser does not establish that a parameter is supported: the selected diagnostic can validate, ignore, or pass through an assignment. Use the diagnostic page for supported parameter names, values, and defaults.

Results and command status#

A multi-node request has two different kinds of outcome:

  • An orchestration error means DCGM could not connect to the participating host engines, reserve resources, start or monitor the workload, or collect a usable response. The command reports a DCGM error and returns a nonzero status.

  • A diagnostic result describes what the workload reported after it ran. A completed request can contain a failed diagnostic result while dcgmi itself returns status zero.

The current response contains an overall diagnostic result together with per-host and per-GPU information when the workload supplies it. See dcgmi mndiag for invocation, host-list, output, signal, and exit-status behavior. See Multi-Node Diagnostics for the environment setup and an end-to-end qualification workflow.