mnnvbandwidth#

Purpose and availability#

mnnvbandwidth measures peer-GPU memory transfers across participating hosts. Its normal mode runs directional copy-engine writes and reads and reports a bandwidth matrix. Its all-pairs mode instead runs NVBandwidth’s GPU-issued peer-memory read test with a 1 MiB buffer for every unordered active-GPU pair in sequence so that a failed transfer can be associated with its two endpoints.

Select the normal run with:

$ dcgmi mndiag --hostList 'node01;node02' --run mnnvbandwidth

mnnvbandwidth is available in DCGM 4.7 and later. DCGM 4.7 packages NVBandwidth 0.10 for CUDA 12 and CUDA 13 on x86-64 and Arm64. The CUDA 12 binary is selected for a CUDA driver API major of 12; the CUDA 13 binary is selected for a major of 13 or later. A driver API major below 12 has no compatible packaged binary.

The built-in eligible devices are GB200 NVL (PCI device ID 2941) and GB300 NVL (PCI device ID 31c2). Every participating host must have active GPUs, and all participating GPUs must use the same eligible SKU.

Resource and launch model#

MNNVBandwidth actively transfers data. The host list selects hosts and host-engine transports, not GPU subsets. Drain every active GPU on every participating host. See Execution inventory.

The common launch requirements are reachable and authorized host engines, Open MPI, noninteractive SSH for the caller-derived execution account, accessible CUDA-specific binaries, and a functioning Multi-Node NVLink/IMEX configuration. See Whose identity launches the workload for the distinction between the dcgmi process user, head host-engine process user, and configured service account. Open MPI path and interface controls are defined in OPEN MPI LAUNCH CONTROLS.

Parameters#

Parameters use the case-sensitive mnnvbandwidth. prefix. Only the parameters listed below are supported; unknown names and invalid values are rejected before launch.

Normal-run parameters#

MNNVBandwidth valued parameters#

Parameter

dcgmi form

Default

Accepted value and effect

mnnvbandwidth.time_to_run

Valued assignment

3600 seconds

Decimal integer from 1 through 2,147,483,647. Sets DCGM’s base time budget and is not passed to NVBandwidth. Normal mode adds the common 60-second startup allowance; pair mode uses the value as its batch deadline.

mnnvbandwidth.testcase

Valued assignment

Two names listed below

One or more whitespace-separated tokens passed as individual values to --testcase. Mutually exclusive with testcasePrefixes.

mnnvbandwidth.testcasePrefixes

Valued assignment

Not set

One or more whitespace-separated tokens passed to --testcasePrefixes. Removes the default testcase selection and is mutually exclusive with testcase.

mnnvbandwidth.bufferSize

Valued assignment

Workload default

Decimal unsigned integer from 0 through 18,446,744,073,709,551,615 that sets NVBandwidth’s buffer size in MiB.

mnnvbandwidth.testSamples

Valued assignment

Workload default

Decimal integer from 1 through 2,147,483,647 that sets the number of samples.

mnnvbandwidth.targetNumPairs

Valued assignment

Computed

-1 requests all pairs, 0 requests no pairs, and a positive integer through 9,223,372,036,854,775,807 requests sampled pairs. NVBandwidth validates a positive value against the pairs available to its MPI world.

mnnvbandwidth.gpuPairs

Valued assignment

Not set

The sole accepted value is the exact string all. It selects the sequential all-pairs mode described below.

Numeric values must consume the entire token. A leading plus sign, surrounding whitespace, non-decimal text, or a value outside the stated integer type is rejected. bufferSize=0 is accepted.

Each testcase or testcasePrefixes token can contain letters, digits, underscore, hyphen, period, and colon, but a token cannot begin with a hyphen. Whitespace separates tokens. At least one token character is required. Quote the complete CLI assignment when it contains whitespace:

$ dcgmi mndiag --hostList 'node01;node02' --run mnnvbandwidth \
    --parameters 'mnnvbandwidth.testcase=TESTCASE_ONE TESTCASE_TWO'

Normal-run defaults#

Without a testcase or prefix override, DCGM passes both of these testcases:

  • multinode_device_to_device_memcpy_write_ce

  • multinode_device_to_device_memcpy_read_ce

DCGM also computes targetNumPairs when the caller omits it. Let N be the number of hosts multiplied by the active-GPU count visible to the head node:

  • If \(N \leq 16\), DCGM requests all directed pairs.

  • If \(N > 16\), DCGM requests N sampled pairs.

This threshold applies to the total MPI process count, not to a per-host GPU count. The normal run is one MPI launch using the common processes-per-node mapping.

Normal-mode runtime#

time_to_run is not passed to NVBandwidth. In normal mode, DCGM sets the process-monitor deadline to that value plus the common 60-second startup allowance. Thus time_to_run=300 permits 360 seconds in the process-execution phase.

Warning

Tests exceeding the total time budget will be terminated early and return a failed status. In these cases, partial results will not be provided. The multi-node diagnostic logs are the recommended source for additional details of such runs.

All-pairs mode#

mnnvbandwidth.gpuPairs=all makes DCGM build a sorted inventory from the active GPU IDs and UUIDs reported by every participating host. It rejects an empty host inventory, a host with no active GPUs, a missing GPU UUID, a duplicate host/GPU ID, a duplicate UUID, or fewer than two GPUs.

For N inventoried GPUs, DCGM generates every unordered pair:

\[\frac{N(N-1)}{2}\]

Pairs on one host and pairs on different hosts are both included. DCGM runs the pairs sequentially. Each launch uses two workload processes and selects the two endpoint GPUs by UUID.

Pair-mode parameter interactions#

Parameter behavior in all-pairs mode#

Parameter

Pair-mode behavior

time_to_run

One deadline for the entire sequence. Each pair receives only the time remaining; the value is not a per-pair budget. All-pairs mode does not add the normal-mode startup allowance to this batch deadline.

testcase

An explicit selection is retained. Otherwise DCGM selects multinode_device_to_device_memcpy_read_sm.

testcasePrefixes

An explicit prefix selection is retained and removes the default testcase.

bufferSize

An explicit value is retained. Otherwise DCGM passes 1 MiB.

testSamples, targetNumPairs

Removed before each pair launch; they do not apply.

A workload failure is recorded for that pair, and the batch continues when cleanup succeeds and time remains. An orchestration, timeout, or cleanup failure can abort the batch without a completed aggregate result.

Examples#

Run the normal default validation:

$ dcgmi mndiag --hostList 'node01;node02' --run mnnvbandwidth

Run one testcase with a five-minute overall budget:

$ dcgmi mndiag --hostList 'node01;node02' --run mnnvbandwidth \
    --parameters 'mnnvbandwidth.testcase=multinode_device_to_device_memcpy_read_ce' \
    --parameters mnnvbandwidth.time_to_run=300

Check every unordered active-GPU pair:

$ dcgmi mndiag --hostfile hosts.txt --run mnnvbandwidth \
    --parameters mnnvbandwidth.gpuPairs=all

Normal-mode results#

MNNVBandwidth normal-mode diagnostic results#

Workload outcome

Diagnostic result

Meaning

The selected testcases complete, the workload provides usable participant inventory and results, and it reports no error.

Pass

The requested directional transfers completed without a reported error. DCGM does not apply a minimum bandwidth threshold.

All selected work is waived or unavailable and no failure is reported.

Skip

The run did not exercise the requested transfer and does not establish a passing qualification.

The workload reports an error, a selected testcase does not complete, no usable result or participant inventory is produced, or the MPI workload exits unsuccessfully.

Fail

DCGM associates an error with an affected GPU when the workload identifies one; otherwise the error applies to the overall run. Inspect the captured logs for the workload report and MPI failure details.

Pair-mode result construction#

Each pair that completes without a reported error is counted as passed; a completed pair with an error is counted as failed. If any pair fails, the overall result is Fail; otherwise it is Pass.

Both endpoint GPUs of every failed pair receive failed entity results. A global error names the two host/GPU endpoints and the available reason. The command reports at most 32 pair-mode errors: if more than 32 pairs fail, it reports details for 31 and uses the final entry to report the number omitted. Passing endpoints that do not occur in any failed pair receive passing results. Marking both endpoints does not determine which endpoint caused the failure.

Text output reports Pairs Planned, Pairs Passed, and Pairs Failed and includes an aggregate Log Files section. These pair counts and paths are not included in JSON output; see JSON OUTPUT.

Failures before or around a result#

MNNVBandwidth orchestration failures#

Condition

Status

Effect

A participating host has no active GPUs, the participating GPUs do not use one SKU, or the SKU is not supported.

DCGM_ST_NOT_SUPPORTED

The workload is not launched.

The head cannot connect to or authorize a participating host engine, or a participant cannot supply its node inventory.

Connection, SSH, authorization, or inventory status

No completed diagnostic verdict is established.

A participating node cannot reserve the required diagnostic resource because it is already in use.

DCGM_ST_IN_USE

Wait for the conflicting activity to finish, or stop it, and retry.

A parameter is unknown, malformed, outside its accepted domain, or conflicts with another parameter.

DCGM_ST_BADPARAM

Correct the request.

No readable, executable mnnvbandwidth can be selected.

DCGM_ST_INIT_ERROR

Correct the package installation.

MPI cannot launch, the expected workload process cannot be detected, or another GPU process is detected after launch in normal mode.

Launch status such as DCGM_ST_CHILD_SPAWN_FAILED

Inspect MPI, SSH, service, and workload logs.

The normal process or all-pairs batch exceeds its budget, or the client interrupts it.

Nonzero timeout or signal status

The run is incomplete and does not establish a pass.

A normal-mode or aggregate pair Fail can accompany command status zero. Orchestration failures return nonzero. See EXIT STATUS.

Executable and logs#

MNNVBandwidth executable and log controls#

Item

Behavior

Default executable

mnnvbandwidth under the installed plugins/cuda12 or plugins/cuda13 directory selected from the driver API major.

Aggregate/normal stdout

/var/log/nvidia-dcgm/dcgm_mndiag_mnnvbandwidth_stdout.txt.

Aggregate/normal stderr

/var/log/nvidia-dcgm/dcgm_mndiag_mnnvbandwidth_stderr.txt.

In all-pairs mode, the aggregate logs contain the output from each pair. DCGM_HOME_DIR replaces /var/log/nvidia-dcgm as the log directory. An empty-but-present value makes the filenames relative to the host engine’s current working directory; leave the variable unset to use the default. Set it in the head host-engine process environment.

See also#

Multi-Node Diagnostics, mnubergemm, nvloom, dcgmi mndiag, Multi-Node Diagnostics, NVBandwidth Plugin