Multi-Node Diagnostics#

Overview#

The multi-node diagnostic module is module ID 10. It coordinates supported diagnostic workloads across host engines and is loaded when a multi-node diagnostic is first requested.

The module provides the orchestration around named multi-node diagnostics. A head-node host engine connects to the participating host engines, reserves the requested resources, launches the selected workload through MPI, monitors the run, and collects host and GPU results.

When multi-node diagnostics are useful#

Use this module after individual nodes are known to work and the remaining question involves coordinated operation across a supported multi-node NVLink system. It is intended for cluster commissioning, post-maintenance qualification, performance-consistency checks, and isolation of failures that appear only when GPUs on several hosts communicate and compute together.

Multi-node diagnostics are not a substitute for single-node triage. A deployment, memory, or PCIe problem on one host is usually faster to identify with Diagnostics before involving the cluster. A practical qualification sequence is:

Verify inventory, driver consistency, and host-engine connectivity on each node.
Run an appropriate single-node diagnostic on each node.
Confirm noninteractive SSH, MPI, and IMEX configuration required by the multi-node workload.
Run mndiag from the head node over the exact hosts and GPUs being qualified.
Inspect both the overall test result and the per-host/per-GPU details.

DCGM 4.6 provides the mnubergemm diagnostic. It actively uses GPU compute, memory, and interconnect resources across the selected nodes, so participating resources must be drained from ordinary workloads. The module is broader than that one diagnostic: mnubergemm is the current workload selected through the module, not another name for multi-node diagnostics as a whole.

Example: qualify a two-node allocation#

Suppose GPUs 0 through 3 on node01 and node02 form the allocation to test. From the head node, run a ten-minute test and request structured output:

$ dcgmi mndiag \
    --hostList 'node01=0-3;node02=0-3' \
    --parameters mnubergemm.time_to_run=600 \
    --json

The quoted host list is important because an unquoted semicolon is interpreted by the shell. A host without an explicit =gpu-list uses every GPU on that host. Include the head node in the list when it is intended to participate in the workload.

The head-node host engine orchestrates the run; participating host engines launch their local processes and return results. A completed request can still contain a failed diagnostic result. Automation must inspect the mnubergemm test status in the JSON document rather than treating process status zero by itself as a passing cluster. The JSON shape is release-specific, not a stable schema; pin and test the producing DCGM release as described in JSON output.

For a first run, a short explicit duration makes configuration errors easier to iterate on. Use a duration representative of the qualification objective once connectivity and launch are working.

Terminology#

Term	Description
DCGM	Data Center GPU Manager
`mnubergemm`	The multi-node diagnostic supplied by DCGM 4.6. It is an MPI workload that stresses GPUs across hosts simultaneously.
Head node	The node whose host engine coordinates the multi-node diagnostic.
MPI	Message Passing Interface, a standard interface used to launch and coordinate work spanning hosts. `mnubergemm` uses Open MPI.

Available diagnostics#

The multi-node diagnostic catalogue lists the canonical names accepted by dcgmi mndiag --run. It also provides the supported products, parameters, result rules, and failure conditions for each diagnostic. DCGM 4.6 has one entry, mnubergemm.

Prerequisites#

All of the nodes in the cluster, targeted in the test, will require the following to be installed and configured:

OpenMPI
Passwordless ssh access between all nodes in the cluster

Note

Each node is required to be using the same NVIDIA driver version.

Getting Started#

Command-line invocation#

Run dcgmi mndiag on the head node and include every node that should participate, including the head node itself. For example:

$ dcgmi mndiag --hostList 'node1;node2;node3'

The single quotes are significant because an unquoted semicolon separates commands in Bash instead of remaining part of the host list. See the shell-conventions note for the notation used by DCGM command examples.

The head-node host engine coordinates the run and collects the results. See dcgmi mndiag for the complete host-list grammar, options, output, and automation behavior. Diagnostic names, parameters, supported systems, and result interpretation are in the multi-node diagnostic reference.

Test Setup#

An installation of Open MPI (version 4.1.1 or a later ABI-compatible version) on each participating machine.

Note

This requirement should be satisfied automatically for systems in which the DCGM multi-node package has been installed using the system package manager.

IMEX channels: For the supported multi-node NVLink systems, configure IMEX (In-Memory EXtension) channels before running mnubergemm. Without working IMEX channels, NCCL initialization fails.

For the setup procedure, see the NVIDIA IMEX Channels Guide.

Warning

IMEX Channels Required for Multi-Node NVLink Systems

The mnubergemm diagnostic fails if IMEX channels are not set up correctly on the supported multi-node NVLink system. You may see errors similar to:

MNUB [E] G: 7 nvidia-node2 L: 3   FAILED: NCCL error @ ../../src/cluster_context.hpp:134 'unhandled system error (run with NCCL_DEBUG=INFO for details)' <- ncclCommInitRankConfig(&m_comm0, m_global_rank_total, m_id0, m_global_rank, &config)
MNUB [E] G: 6 nvidia-node2 L: 2   FAILED: NCCL error @ ../../src/cluster_context.hpp:134 'unhandled system error (run with NCCL_DEBUG=INFO for details)' <- ncclCommInitRankConfig(&m_comm0, m_global_rank_total, m_id0, m_global_rank, &config)

These messages indicate that NCCL could not initialize. Verify the IMEX configuration against the guide above before rerunning the diagnostic.

The value of the DCGM_MNDIAG_MPIRUN_PATH environment variable for the nv-hostengine process should be set to the path of the mpirun command.
The value of the LD_LIBRARY_PATH environment variable for the nv-hostengine process should be set to the directory containing the Open MPI libraries.

LD_LIBRARY_PATH changes the dynamic linker’s search for the host engine and inherited child processes; a value exported only by the dcgmi user does not configure the service. See shared-library loading, Customizing the systemd Service, and the versioned Open MPI launch prerequisites.

Note

This requirement should be satisfied automatically for systems in which

the DCGM multi-node package has been installed using the system package manager on a DCGM-supported Linux distribution and
the nvidia-dcgm.service systemd unit is used to manage the nv-hostengine daemon.

The multi-node package’s generated service environment and supported local overrides are described in Customizing the systemd Service.

The service account configured for nv-hostengine is available on every participating machine. The packaged nvidia-dcgm.service unit runs the host engine as root with --service-account nvidia-dcgm, so the account is nvidia-dcgm unless the unit has been customized. This is independent of the user that invokes dcgmi.

The account needs executable and directory access on every node even though it need not be an interactive login user. See service and child-process identities.
Either
1. login shells associated with the host-engine service account are configured to use Open MPI, or
2. the installation root directory of Open MPI is the same on each participating machine and Open MPI has been configured to propagate the installation root by default.
Non-interactive SSH needs to be configured between all nodes. Run the test as the host-engine service account, for example with the packaged unit:
```
# sudo -u nvidia-dcgm -H ssh <remote-node>
```

Warning

SSH Key Security and Non-Interactive Access

DCGM Multi-Node Diagnostics require non-interactive SSH access among the participating nodes because Open MPI can use a tree of remote launches, not only direct launches from the head node. The private key used for authentication must therefore be available to the configured host-engine service account wherever that account initiates a hop and, when no agent is available, stored without a passphrase.

Using an ssh-agent to load encrypted keys into memory will not work in this scenario:

$ eval $(ssh-agent)
$ ssh-add <path to SSH key>
$ dcgmi mndiag <args>...

This is because the SSH sessions required by the diagnostic are spawned by the nv-hostengine service account, not as a child of the user’s shell or dcgmi process. The host engine does not inherit that shell’s agent environment. An agent loaded by the interactive user therefore cannot provide the key.

OpenSSH rejects private-key files that are accessible by other users, so preserve the service account’s ownership and restrictive file permissions. See OpenSSH’s BatchMode setting and private-key file requirements, together with the Open MPI 5.0.10 non-interactive SSH requirements.

Note

Successful SSH is only the launch path: the DCGM listeners and the network transports selected by Open MPI must also be reachable among participating hosts, and Open MPI can allocate dynamic TCP or UDP ports. See the Open MPI multi-host troubleshooting guidance and the DCGM listener and firewall note.

The nv-hostengine daemon should be started on all the nodes.
Each node should have the same NVIDIA driver version.

Choosing the diagnostic and duration#

DCGM 4.6 selects mnubergemm by default. For a first run, set mnubergemm.time_to_run to a short positive duration while resolving configuration problems, then use a duration appropriate for the qualification goal. See mnubergemm for its supported parameter, default, timeout behavior, and exact result rules.

Sample output#

Executing: ./dcgmi mndiag --hostList "node1:5000;node2;node3;node4;node5;node6;node7:4000;node8"
Successfully ran multi-node diagnostic.
+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Metadata  ----------+------------------------------------------------|
| MNUBERGEMM Test           | Pass                                           |
| Driver Version            | 570.133                                        |
| DCGM Version              | 4.3.0                                          |
| Hosts Found               | 8                                              |
| Hosts With Errors         | 0                                              |
| Host List                 | 0: node1                                       |
|                           | 1: node2                                       |
|                           | 2: node3                                       |
|                           | 3: node4                                       |
|                           | 4: node5                                       |
|                           | 5: node6                                       |
|                           | 6: node7                                       |
|                           | 7: node8                                       |
| Total GPUs                | 32                                             |
|-----  Host Details  ------+------------------------------------------------|
| Host 0                    | Pass                                           |
|   GPUs: 0, 1, 2, 3        | Pass                                           |
| Host 1                    | Pass                                           |
|   GPUs: 0, 1, 2, 3        | Pass                                           |
| Host 2                    | Pass                                           |
|   GPUs: 0, 1, 2, 3        | Pass                                           |
| Host 3                    | Pass                                           |
|   GPUs: 0, 1, 2, 3        | Pass                                           |
| Host 4                    | Pass                                           |
|   GPUs: 0, 1, 2, 3        | Pass                                           |
| Host 5                    | Pass                                           |
|   GPUs: 0, 1, 2, 3        | Pass                                           |
| Host 6                    | Pass                                           |
|   GPUs: 0, 1, 2, 3        | Pass                                           |
| Host 7                    | Pass                                           |
|   GPUs: 0, 1, 2, 3        | Pass                                           |
+---------------------------+------------------------------------------------+

=========================================
Test completed at: Sat Jun 21 09:49:50 AM PDT 2025
Exit code: 0
Process monitoring will be stopped
=========================================

Interpreting the run#

First distinguish an orchestration error from a diagnostic finding. A connection, authorization, reservation, SSH, MPI launch, or process-monitoring error can prevent the workload from producing a result. Correct that setup or execution problem before drawing a conclusion about the GPUs.

After mnubergemm runs, inspect its overall result and every per-host and per-GPU entry. A failed GPU includes the error text emitted by the workload; that text identifies the reported operation or condition. Do not replace it with a generic assumption that every failure represents low bandwidth, memory corruption, or an interconnect defect. A passing result means this workload completed without a recognized error record during this run; it is not an exhaustive statement of system health.

The command’s process status and the diagnostic result are also separate. A completed request can return status zero while the diagnostic result is Fail. The mnubergemm result reference lists the source-verified pass, failure, and pre-execution conditions.

Multi-Node Diagnostics#

Overview#

When multi-node diagnostics are useful#

Example: qualify a two-node allocation#

Terminology#

Available diagnostics#

Prerequisites#

Getting Started#

Command-line invocation#

Test Setup#

Choosing the diagnostic and duration#

Sample output#

Interpreting the run#

Further reading#