dcgmi mndiag#

NAME#

dcgmi-mndiag - run one coordinated diagnostic across multiple hosts

SYNOPSIS#

dcgmi mndiag (--hostList <host-spec>[;<host-spec>...] | --hostfile <path>)
              [-r <mnubergemm|nvloom|mnnvbandwidth>]
              [-p <prefix.parameter=value>]...
              [--hostEngineAddress <address>] [-j]

DESCRIPTION#

dcgmi mndiag connects to a head-node host engine and asks it to run one multi-node diagnostic. The head node connects to the participating host engines, reserves their resources, starts an MPI workload, monitors it, and prints the collected host and GPU results.

The accepted diagnostics are mnubergemm, nvloom, and mnnvbandwidth. They are multi-node workloads, not plugins selected by dcgmi diag.

Execution inventory#

The host list selects participating hosts and their host-engine transports. It does not select GPUs. The MPI process count is the number of hosts multiplied by the active-GPU count visible to the head-node host engine, and that same processes-per-node count is used on every host. On the supported uniform configurations, every active GPU on every participating host participates. Make those GPUs exclusively available to the diagnostic before starting a run.

Execution identity#

The effective UID of the dcgmi process is sent with the request. On the head node, nv-hostengine resolves that numeric UID and uses the resulting account for the SSH-tunnel and MPI-launch child processes. Running dcgmi through sudo therefore changes the requested execution identity.

This identity is distinct from both the operating-system user that runs nv-hostengine and the account configured with nv-hostengine --service-account. The packaged systemd unit runs nv-hostengine as root and configures nvidia-dcgm as its service account, but the mndiag path does not use that configured service account to select its SSH or MPI user.

Only the numeric UID is sent; the invoking shell environment and SSH agent are not. The UID must resolve on the head node, and noninteractive SSH must work from the head node for the resolved account. The head nv-hostengine must also either already run with that UID or have permission to switch its child processes to it. See Whose identity launches the workload for the complete identity and SSH setup model.

OPTIONS#

--hostList <host-spec>[;<host-spec>...]

Specify the participating hosts inline. Exactly one of --hostList and --hostfile is required. Quote a multi-host value so the shell does not interpret its semicolons. See the Linux shell conventions.

The effective host and transport forms are:

host
host:port
host:unix:///absolute/socket/path
host:unix://

A host without a port uses TCP port 5555. An explicit TCP port must be decimal and in the inclusive range 0 through 65535. The abbreviated host:unix:// form uses the default remote socket path /tmp/nv-hostengine. An explicit remote socket path must be absolute and cannot contain another colon.

IPv6 address literals are not accepted by this option. A host and normalized transport can occur only once; for example, node01 and node01:5555 are duplicates. Each hostname must also be unique independently of transport. Thus node01:5555 and node01:5556, or TCP and Unix-socket entries for node01, cannot appear in the same request.

One command accepts at most 576 entries. After dcgmi applies transport defaults, each normalized host entry can contain at most 255 characters.

--hostfile <path>

Read participating hosts from a file. Exactly one of --hostfile and --hostList is required. Put one host specification accepted by --hostList on each nonempty line. Leading and trailing spaces, tabs, and carriage returns are removed. A line whose first non-whitespace character is # and a blank line are ignored; inline comments are not supported. Each usable line must contain exactly one host specification. The file must be readable and must contain at least one usable entry. The 576-entry limit and all per-entry validation still apply.

--hostEngineAddress <address>

Connect to this head-node host engine. The default is localhost. This address identifies the coordinator and is independent of the test-node entries in --hostList or --hostfile. See dcgmi for the connection-address forms accepted by dcgmi.

-r <test-name>, --run <test-name>

Select one diagnostic. dcgmi accepts mnubergemm, nvloom, or mnnvbandwidth without regard to case and passes the canonical lowercase name to the host engine. The default is mnubergemm. See Multi-Node Diagnostics and the component references linked below.

-p <assignment>, --parameters <assignment>

Supply diagnostic or MPI-launch parameters. This option is repeatable, and one argument can contain semicolon-separated tokens.

Every token must contain =. The accepted shape is prefix.parameter=value; an empty value after = is syntactically valid.

The remaining common command-line rules are:

prefix must be the selected canonical diagnostic name or openmpi.
Parameter names are case-sensitive after --run has been canonicalized. DCGM does not normalize value case; whether a value is accepted belongs to the named consumer.
Do not repeat a complete parameter key across -p arguments.
At most 100 tokens are accepted.
Each resulting token must contain fewer than 1,050 characters.
Only the parameters documented for the selected diagnostic or below for Open MPI are part of the supported command interface. The diagnostic reference defines their accepted values and effects.

Quote an argument containing semicolons or whitespace. For example:

$ dcgmi mndiag --hostList 'node01;node02' --run nvloom \
    --parameters 'nvloom.time_to_run=1200;nvloom.suite=fabric-stress'

-j, --json

Render the command result as JSON instead of the text table. The exact schema for this release is documented in JSON OUTPUT. It is not a stable cross-release schema; see JSON output.

-h, --help

Print generated usage information and exit.

OPEN MPI LAUNCH CONTROLS#

openmpi.detect_interfaces=<value> is the only supported openmpi.* parameter. Automatic interface detection is enabled by default. The head-node host engine selects its non-loopback network interfaces that can reach the participating hosts and restricts Open MPI traffic to those interfaces.

Use openmpi.detect_interfaces=0 or openmpi.detect_interfaces=false to disable detection. These two values are exact and case-sensitive. Every other value, including an empty value, leaves detection enabled. The command cannot express a bare parameter entry.

If no usable interface is found, DCGM supplies neither restriction and Open MPI performs its default interface selection.

Open MPI also reads these inherited variables natively:

Inherited Open MPI interface variables#
Variable	Effect
`OMPI_MCA_btl_tcp_if_include`	Open MPI 4.x BTL TCP interface selection.
`OMPI_MCA_oob_tcp_if_include`	Open MPI 4.x out-of-band TCP interface selection.
`PRTE_MCA_oob_tcp_if_include`	Open MPI 5.x PRRTE out-of-band TCP interface selection.

When automatic detection succeeds, its interface selection takes precedence over the corresponding inherited settings. Disable automatic detection when the inherited MPI settings must control interface selection. Set these variables in the head nv-hostengine process environment, not only in the shell that invokes dcgmi.

Common host-engine environment#

Common multi-node host-engine environment#
Variable	Behavior
`DCGM_MNDIAG_MPIRUN_PATH`	Selects the `mpirun` executable. An unset or empty variable uses `/usr/bin/mpirun`. DCGM compares the version text from that executable with `ompi_info` in the same directory and logs a mismatch; the mismatch does not itself reject the run.
`DCGM_MPIRUN_ALLOW_RUN_AS_ROOT`	When this variable is present and the caller-derived mndiag execution account has UID 0, DCGM adds `--allow-run-as-root` to the MPI command. The value is not parsed; unset the variable to disable the behavior. The UID is the effective UID sent by the `dcgmi` process, not the UID that runs the head `nv-hostengine` or the account configured with `nv-hostengine --service-account`. Enable this setting when an MPI launch as root is intentional.
`DCGM_HOME_DIR`	Selects the directory used for captured workload logs. When unset, the directory is `/var/log/nvidia-dcgm`. If the variable is present but empty, the generated filenames are relative to the host engine’s current working directory; leave it unset to select the default.
`LD_LIBRARY_PATH`	Used normally by the dynamic loader for `mpirun` and its children. The packaged multi-node systemd generator sets a distribution-specific Open MPI library directory where one is required. Custom service configuration must preserve or replace that setting when MPI libraries are not on the system loader path.

These settings must be present in the head nv-hostengine process environment. Setting them only in the shell that invokes dcgmi does not configure an already-running host engine.

TEXT OUTPUT#

Without --json, dcgmi renders the diagnostic as a text report:

The opening sentence says the diagnostic ran successfully only for Pass, says it was skipped for Skip, and says it failed for every other result or for an orchestration failure.
Metadata contains the test name and result, Hosts Found, and Hosts With Errors. The latter is the number of distinct host IDs with a failed entity result. Identical driver and DCGM versions are printed once; differing versions are printed under the individual hosts.
When hosts are present, Metadata also contains Host List and Total GPUs. Total GPUs counts GPU result records, not inventory entries without results.
Host Details is omitted after an orchestration failure. Otherwise, any failed GPU makes its host Fail; absent a failure, a warning makes the host Warn; only skipped GPU results with no passing result make it Skip; all other combinations render Pass. All-pass GPUs are grouped on one row. Mixed results are printed per GPU, with GPU-scoped errors below failed GPUs. A host with no GPU result says No host result available when the overall result failed, or No GPU results available otherwise.
Error Summary renders GPU errors, global errors, and formatted pair-mode failures. When the first test is neither Pass nor Skip and its result contains no error message, the renderer synthesizes an MPI error from the captured process failure detail.
MNNVBandwidth pair mode adds Pairs Planned, Pairs Passed, and Pairs Failed to Metadata, and adds a Log Files section with the aggregate stdout and stderr paths.

JSON OUTPUT#

Do not rely on empty arrays being present. Array-valued members may be omitted when they have no entries.

Top-level JSON members#
Path	Presence	Value
`DCGM Diagnostic`	Always	String `DCGM Multi-Node Diagnostic`.
`metadata.num_hosts`	Always	Number of hosts found.
`metadata.num_entities`	Always	Number of entities found.
`runtime_error`	Failed command only	Human-readable connection, orchestration, timeout, or signal error.
`hosts`	Successful command with hosts	Host and entity inventory. It is deliberately omitted when the command itself failed.
`test_categories`	At least one test	One category object named `Multi-Node Tests` with a `tests` array.
`global_errors`	At least one global or synthesized error	Array of error objects.

Each hosts[] object contains host_id, hostname, dcgm_version, driver_version, num_entities, and entities. Each entity contains entity_group_id, entity_group, and entity_id; serial_num and device_id are included only when nonempty.

Each test_categories[0].tests[] object contains name, status, test_summary.status, and, when present, results. status uses the spellings in the result table above. If a test is neither Pass nor Skip and has no reported errors, mpi_error is added with warning and error_category set to MPI.

Each test results[] object contains host_id, entity_group_id, entity_group, entity_id, and status. Its optional warnings array contains objects with warning, error_id, error_category, and error_severity for errors matching that exact host and entity. Its optional info array contains matching informational message strings.

Ordinary global_errors[] objects contain warning, error_id, error_category, error_severity, and host_id. A synthesized MPI global error contains only warning and error_category.

The pair-mode planned, passed, and failed counts and aggregate log paths are available in text output but are not included in JSON output.

EXIT STATUS#

Command execution and diagnostic evaluation are separate:

A completed command that renders a diagnostic result returns DCGM_ST_OK/process status 0 even when the test result is Fail, Warn, Skip, or Not Run. Automation must inspect the rendered result.
Connection, authorization, reservation, launch, timeout, signal, cleanup, and rendering failures return their nonzero DCGM status when available.

While a multi-node diagnostic is running, SIGHUP, SIGINT (including Ctrl-C), SIGQUIT, or SIGTERM requests that DCGM stop the run. An interrupted run exits with a nonzero status and does not establish a passing diagnostic result.

dcgmi returns the signed value from main, but a POSIX shell exposes only its low eight bits. For a negative value n, $? is 256 + n. Do not compare $? to a negative DCGM constant.

Representative command return values#
Path	Signed return	POSIX `$?`
Completed command, regardless of diagnostic result	`DCGM_ST_OK` (0)	0
Invalid command parameter propagated by execution	`DCGM_ST_BADPARAM` (-1)	255
Unsupported participating GPU configuration	`DCGM_ST_NOT_SUPPORTED` (-6)	250
Timed-out operation	`DCGM_ST_TIMEOUT` (-11)	245

EXAMPLES#

Run the default diagnostic on all active GPUs of three hosts:

$ dcgmi mndiag --hostList 'node01;node02;node03'

Use a custom TCP port and a remote Unix socket:

$ dcgmi mndiag \
    --hostList 'node01:5000;node02:unix:///run/dcgm/hostengine.sock'

Read the hosts from a file and run MNNVBandwidth:

$ dcgmi mndiag --hostfile hosts.txt --run mnnvbandwidth

Disable automatic interface selection and rely on the head nv-hostengine process’s Open MPI environment:

$ dcgmi mndiag --hostList 'node01;node02' \
    --parameters openmpi.detect_interfaces=0

Request JSON for a ten-minute mnubergemm run:

$ dcgmi mndiag --hostList 'node01;node02' \
    --parameters mnubergemm.time_to_run=600 --json

DIAGNOSTIC REFERENCE#

The diagnostic-specific parameters, runtime, result construction, executable, logs, and failure conditions are defined in:

See Multi-Node Diagnostics for a representative investigation that introduces the system paths, execution identities, workload choice, and result interpretation.