nvloom#

Purpose and availability#

nvloom is a suite-driven bandwidth and latency workload for reproducing failures that depend on many-GPU or rack-level traffic patterns. A suite can sample transfers from GPUs in each rack, transfer between GPU pairs, concentrate traffic at one GPU, cross rack boundaries, use multicast, probe latency, or sustain concurrent traffic. The packaged workload defines the exact testcases in each suite; DCGM selects the suite, launches it across the participating hosts, and reports inventory, completion, waivers, and errors. Captured workload logs retain the bandwidth summaries, which the current command renderers do not print.

See What crosses the node boundary for the distinction between this GPU memory fabric and the networks used for host-engine, SSH, and MPI control.

Select it with:

$ dcgmi mndiag --hostList 'node01;node02' --run nvloom

nvloom is available in DCGM 4.7 and later. The built-in eligible devices are GB200 NVL (PCI device ID 2941) and GB300 NVL (PCI device ID 31c2). Every participating host must have active GPUs, and all participating GPUs must use the same eligible SKU.

Packaged NVLoom variants#
Selected directory	Packaged workload	Selection rule
`cuda12`	NVLoom 1.4.1	CUDA driver API major 12.
`cuda13`	NVLoom 2.0.0	CUDA driver API major 13 or later.

Both directories are packaged for x86-64 and Arm64. A CUDA driver API major below 12 has no compatible packaged binary. No separate numeric driver-branch minimum applies.

Resource and launch model#

NVLoom is an active fabric workload. The host list selects hosts and host-engine transports, not GPU subsets. On supported uniform systems, every active GPU participates; drain every active GPU on every participating host. See Execution inventory.

The common launch requirements are reachable and authorized host engines, Open MPI, noninteractive SSH for the caller-derived execution account, an accessible CUDA-specific nvloom_cli, and a functioning Multi-Node NVLink/IMEX configuration. See Whose identity launches the workload for the distinction between the dcgmi process user, head host-engine process user, and configured service account. Open MPI path and interface controls are defined in OPEN MPI LAUNCH CONTROLS.

Parameters#

Only the parameter names listed below are supported. They are case-sensitive, and DCGM does not normalize their values. Except for time_to_run, the packaged NVLoom workload defines the accepted values. Suite and testcase names are case-sensitive.

NVLoom parameters#
Parameter	`dcgmi` form	Default	DCGM behavior
`nvloom.time_to_run`	Valued assignment	`3600` seconds	Requires a positive decimal integer from 1 through 2,147,483,647. Sets the overall DCGM budget and is transformed into the workload’s per-testcase `--duration`.
`nvloom.suite`	Valued assignment	`gpu-to-rack`	Selects a named suite. DCGM also uses the suite to divide the requested run time among its expected testcases.
`nvloom.testcase`	Valued assignment	Not set	Selects one named testcase. When present, DCGM uses a testcase count of one for duration division.
`nvloom.buffer_size`	Valued assignment	`536870912` (512 MiB)	Sets the NVLoom transfer buffer size in bytes.
`nvloom.iterations`	Valued assignment	`16`	Sets the number of workload iterations.
`nvloom.rich_output`	Empty assignment	Not set	Requests detailed per-measurement results, including the participating GPU pairs, from testcases that support rich output. Without it, those testcases report aggregate statistics such as a median. The details remain in the captured NVLoom output and do not change the `dcgmi --json` schema.
`nvloom.allocation_strategy`	Valued assignment	`reuse`	Selects the NVLoom buffer-allocation strategy.

nvloom.rich_output is a flag, not a Boolean-valued parameter. Because dcgmi requires an equals sign in each parameter token, specify the flag as:

$ dcgmi mndiag --hostList 'node01;node02' --run nvloom \
    --parameters nvloom.rich_output=

The packaged workload determines the valid suite, testcase, and workload parameter values.

Duration calculation#

Let T be time_to_run. DCGM computes:

available = max(T - 30, 0)
duration  = max(1, floor(available / testcase_count))

duration is passed to nvloom_cli. The testcase count is one when nvloom.testcase is present. Otherwise, DCGM uses the following lookup table for the suite or a fallback count of eight for an unrecognized suite.

NVLoom duration-accounting table#
Suite string	Testcase count	Packaged variants in the table
`gpu-to-rack`	8	NVLoom 1.4.1 and 2.0.0
`pairwise`	8	NVLoom 1.4.1 and 2.0.0
`pairwise-tma`	4	NVLoom 1.4.1 and 2.0.0
`fabric-stress`	4	NVLoom 1.4.1 and 2.0.0
`all-to-one`	4	NVLoom 1.4.1 and 2.0.0
`multicast`	8	NVLoom 1.4.1 and 2.0.0
`multicast-tma`	4	NVLoom 1.4.1 and 2.0.0
`egm`	8	NVLoom 1.4.1 and 2.0.0
`rack-to-rack`	8	NVLoom 1.4.1 and 2.0.0
`latency`	2	NVLoom 1.4.1 and 2.0.0
`rack-aware-all-to-one`	4	NVLoom 2.0.0
`egm-gpu-to-rack`	8	NVLoom 2.0.0

For a suite not listed in this table, DCGM uses eight testcases for duration accounting. The selected packaged workload determines whether the suite name is valid.

The normal process deadline is T plus a separate 60-second startup allowance. The 30-second subtraction affects the per-testcase workload duration; the 60-second allowance affects DCGM’s process monitor.

Examples#

Run the default suite, which samples GPU transfers by rack:

$ dcgmi mndiag --hostList 'node01;node02' --run nvloom

This example selects fabric-stress explicitly and gives it a 20-minute overall budget:

$ dcgmi mndiag --hostList 'node01;node02;node03;node04' --run nvloom \
    --parameters 'nvloom.time_to_run=1200;nvloom.suite=fabric-stress'

Supply a testcase name and request rich output:

$ dcgmi mndiag --hostList 'node01;node02' --run nvloom \
    --parameters 'nvloom.testcase=TESTCASE_NAME;nvloom.rich_output='

Results#

NVLoom diagnostic results#
Workload outcome	Diagnostic result	Meaning
Every participating host and GPU is accounted for, the selected work completes, no error is reported, and the run is not waiver-only.	Overall and GPU results are `Pass`.	The selected NVLoom traffic completed without a reported error. DCGM does not apply a separate bandwidth threshold.
All selected work is waived or non-applicable and no failure is reported.	Overall and entity results are `Skip`.	The run did not exercise the requested traffic and does not establish a passing qualification.
The workload reports an error for a GPU.	Affected GPU and overall result are `Fail`.	DCGM associates the workload message with the affected GPU. The message is evidence for investigation; it does not by itself identify a defective component.
Selected work does not complete, no usable result or participant inventory is produced, or an expected host or GPU is absent.	Overall `Fail`.	The run did not establish a result for the requested configuration; inspect the captured workload logs.
The MPI workload exits unsuccessfully.	Overall `Fail`.	The command reports the MPI failure; inspect the captured stdout and stderr files.

Bandwidth and latency summaries are not emitted by the current text or JSON renderers. Inspect the captured workload logs for those measurements.

Failures before or around a result#

NVLoom orchestration failures#
Condition	Status	Effect
A participating host has no active GPUs, the participating GPUs do not use one SKU, or the SKU is not supported.	`DCGM_ST_NOT_SUPPORTED`	The workload is not launched.
The head cannot connect to or authorize a participating host engine.	Connection, SSH, or authorization status	No diagnostic verdict is established.
A participating node cannot reserve the required diagnostic resource because it is already in use.	`DCGM_ST_IN_USE`	Wait for the conflicting activity to finish, or stop it, and retry.
A present `time_to_run` value is empty, non-integral, zero, negative, or outside the signed 32-bit range.	`DCGM_ST_BADPARAM`	Correct the request.
No readable, executable `nvloom_cli` can be selected.	`DCGM_ST_INIT_ERROR`	Correct the package installation.
MPI cannot launch, the expected workload process cannot be detected, or another GPU process is detected after launch.	Launch status such as `DCGM_ST_CHILD_SPAWN_FAILED`	Inspect MPI, SSH, service, and workload logs.
The deadline expires or the client interrupts the run.	Nonzero timeout or signal status	The run is incomplete and does not establish a pass.

Executable and logs#

NVLoom executable and log controls#
Item	Behavior
Default executable	`nvloom_cli` under the installed `plugins/cuda12` or `plugins/cuda13` directory selected from the driver API major.
Captured stdout	`/var/log/nvidia-dcgm/dcgm_mndiag_nvloom_stdout.txt`.
Captured stderr	`/var/log/nvidia-dcgm/dcgm_mndiag_nvloom_stderr.txt`.

DCGM_HOME_DIR replaces /var/log/nvidia-dcgm as the log directory. An empty-but-present value makes the filenames relative to the host engine’s current working directory; leave the variable unset to use the default. Set it in the head host-engine process environment.

A diagnostic Fail or Skip can accompany command status zero. Orchestration failures return nonzero. See EXIT STATUS.