dcgmi mndiag#

NAME#

dcgmi-mndiag - run a coordinated diagnostic across multiple hosts

SYNOPSIS#

dcgmi mndiag --hostList <host-spec>[;<host-spec>...]
              [--run mnubergemm]
              [--parameters <test.parameter=value>]...
              [--hostEngineAddress <address>] [--json]

DESCRIPTION#

dcgmi mndiag asks a head-node host engine to coordinate one named multi-node diagnostic across multiple test nodes. DCGM 4.6 provides the mnubergemm diagnostic, but the command and module are not synonymous with that workload. Every participating node must run a compatible host engine and satisfy the common and diagnostic-specific prerequisites.

The host list can select a TCP port or remote Unix socket for each node and can limit the GPU IDs used on that node. Hosts without a GPU list use all GPUs.

OPTIONS#

--hostList <host-spec>[;<host-spec>...]

Specify the test nodes. This option is required. Quote the value so the shell does not interpret semicolons. See the shell-conventions note when constructing the value interactively or in a script.

A host specification has one of these forms:

host
host:port
host:unix:///absolute/socket/path
host=gpu-list
host:port=gpu-list
host:unix:///absolute/socket/path=gpu-list

The default TCP port is 5555. gpu-list is * or a comma-separated list of numeric IDs and inclusive hyphen ranges, such as 0,2-4. IDs are sorted and duplicates are removed. The same host and transport cannot occur twice. One request accepts at most 72 host specifications and at most 32 distinct explicit GPU IDs per host.

--hostEngineAddress <address>

Connect to this head-node host engine. The default is localhost. This address is independent of the test-node entries in --hostList. See dcgmi for TCP and Unix-socket address syntax.

-r <test-name>, --run <test-name>

Select one multi-node diagnostic. DCGM 4.6 accepts mnubergemm without regard to case; it is also the default. The canonical names and the purpose of each diagnostic are listed in the multi-node diagnostic catalogue.

-p <test.parameter=value>, --parameters <test.parameter=value>

Pass a diagnostic parameter. The option is repeatable, and an argument can contain semicolon-separated assignments. Every assignment must contain = and a key may be defined only once.

The common parser checks the assignment syntax, count, and length, but does not establish whether a parameter is supported by the selected diagnostic. Parameters use the canonical, case-sensitive diagnostic.parameter=value form. A prefix that does not match the selected diagnostic is not applied. See the selected diagnostic’s page in the multi-node diagnostic reference for supported names, values, and defaults.

-j, --json

Use JSON output. See JSON output.

-h, --help

Print generated usage information and exit.

DIAGNOSTIC REFERENCE#

The multi-node diagnostic reference is the exhaustive catalogue for diagnostic names, supported products, diagnostic-specific prerequisites, parameters, result interpretation, and failure conditions. DCGM 4.6 provides only mnubergemm.

EXECUTION REQUIREMENTS#

The head-node host engine performs the orchestration and the participating host engines launch the selected workload. Every test node needs a compatible DCGM and driver stack, Open MPI, a configured mpirun path, and noninteractive SSH between nodes. These are host-engine execution requirements, not client-side dcgmi dependencies.

DCGM_MNDIAG_MPIRUN_PATH can select the mpirun executable explicitly; the host-engine environment must make the Open MPI libraries and noninteractive SSH credentials available to its child processes. The selected diagnostic can impose additional hardware, executable, and fabric requirements; see its diagnostic reference page.

Use Prerequisites and Test Setup as the configuration checklist. For mnubergemm requirements and exact result rules, see mnubergemm.

In DCGM 4.6, a completed multi-node API request and successful rendering return process status 0 even when the response’s diagnostic result is Fail. Automation must inspect MNUBERGEMM Test in text output or the test status in JSON. Connection, orchestration, timeout, signal, or rendering errors still return a nonzero DCGM status. JSON consumers must also account for the release-specific schema limitation in JSON output.

EXAMPLES#

Run on every GPU on three nodes:

$ dcgmi mndiag --hostList 'node1;node2;node3'

Use GPUs 0 through 3 on node 1 and GPUs 2 and 4 on node 2:

$ dcgmi mndiag --hostList 'node1=0-3;node2=2,4'

Use a custom TCP port and a remote Unix socket:

$ dcgmi mndiag \
    --hostList 'node1:5000;node2:unix:///run/dcgm/hostengine.sock'

Run for ten minutes and return JSON:

$ dcgmi mndiag --hostList 'node1;node2' \
    --parameters mnubergemm.time_to_run=600 --json