Multi-Node Diagnostics#

The multi-node diagnostics module coordinates one named workload across multiple DCGM host engines. Select the workload with dcgmi mndiag --run. These workloads are distinct from the single-node plugins selected with dcgmi diag --run.

The GPU/NVLink/NVSwitch data plane meant by Multi-Node NVLink fabric is defined in What crosses the node boundary.

Available diagnostics#

Multi-node diagnostic catalogue#

Canonical name

Use

Availability

mnubergemm

Reproduce failures that appear under combined single-precision matrix computation, GPU copy activity, and multi-GPU collective communication in one distributed job.

DCGM 4.3 and later

nvloom

Generate defined many-GPU traffic patterns, including GPU pairs, traffic concentrated at one GPU, rack-crossing transfers, multicast, sustained concurrent traffic, and latency probes.

DCGM 4.7 and later

mnnvbandwidth

Measure peer-memory transfers separately by direction, or locate failed pair endpoints with sequential 1 MiB GPU-issued read checks.

DCGM 4.7 and later

Selection#

One diagnostic is accepted per command. dcgmi compares the --run value without regard to case and defaults to mnubergemm.

The host list selects hosts and transports, not GPU subsets. On supported uniform systems, all active GPUs participate. See Execution inventory for the exact launch rule.

Every participating host must use the same supported GPU SKU.

Results#

A multi-node request has two independent outcomes:

  • The command return value reports whether DCGM connected, orchestrated, and rendered the run successfully.

  • The diagnostic result reports what the selected workload established. A completed command can report Fail while returning status zero.

The command grammar, Open MPI controls, text and JSON output, and exit behavior are defined by dcgmi mndiag. Each diagnostic page defines that workload’s command-line parameters, defaults, runtime model, result construction, packaged executable, logs, and failure conditions.

Use Multi-Node Diagnostics to follow a representative investigation and understand how the system paths, workload choice, and result interpretation fit together.