NCCL Tests Plugin#

NCCL (pronounced “Nickel”) is a library providing inter-GPU communication primitives that are topology-aware and can be easily integrated into applications. NCCL Tests check both the performance and the correctness of NCCL operations. For more information about NCCL and its test suite, see the NCCL documentation and the NCCL Tests repository.

The NCCL Tests plugin validates system readiness for NCCL-based workloads by testing communication correctness on single-node systems.

Note

This plugin runs only single-node NCCL tests and does not require MPI. Multi-node NCCL tests are not supported.

Availability and prerequisites#

The canonical test name is nccl_tests. When available, the plugin participates in the level 3 suite. It is available only when its libraries and executable are installed, the executable path is configured in the host-engine environment, and the selected GPUs can form a supported single-node test.

Software requirements#

  • NCCL library (libnccl.so) - not bundled with DCGM, must be installed separately.

  • NCCL Tests project - not bundled with DCGM, must be built/installed separately from the NCCL Tests repository.

  • CUDA toolkit (required by NCCL).

Executable permissions#

The NCCL library requires read permission because it is loaded dynamically. The NCCL test binary has the following requirements:

  • When DCGM runs as a non-root user, the binary must be a regular file with read and execute permissions.

  • When DCGM runs as root, the binary:

    • Must be owned by the root user and root group (uid 0, gid 0).

    • Must not be writable by the group or other users.

    • Must be executable by its owner.

    • Must be a regular file. Symlinks are resolved before validation.

File ownership, mode bits, and the executing process’s credentials jointly determine access. See the Linux process-credentials note rather than broadening permissions merely to make the check pass.

Behavior#

The NCCL Tests plugin validates NCCL functionality on the system by executing test binaries from the NCCL Tests project. These tests verify that NCCL collective communication operations work correctly across the GPUs in the system.

The plugin performs three validation steps:

Binary execution

Verifies that the test binary can be launched successfully.

Exit status

Requires the binary to exit with status zero.

Output validation

Parses successful process output and verifies that the NCCL operations completed without reported errors.

Runtime and resource impact#

The plugin typically completes in a few minutes. Runtime depends on the number of GPUs and the interconnect topology and bandwidth; systems with more GPUs or slower interconnects can take longer.

Parameters#

The following table lists the parameters for the NCCL Tests plugin:

Parameter

Type

Default

Description

is_allowed

Bool

True

Specifies whether this test is allowed to run. When set to False, the test will be skipped.

Examples#

After the host engine has been started with the required environment, run only the NCCL diagnostic:

$ dcgmi diag -r nccl_tests

Run the level 3 diagnostic suite, which includes the NCCL test when the plugin is available:

$ dcgmi diag -r 3

Results and failure conditions#

NCCL Tests results#

Condition

Result or code

Interpretation

The executable exits with status zero and reports both a successful out-of-bounds check and a successful average-bus-bandwidth check.

Pass

The supplied single-node NCCL test completed the checks consumed by this plugin.

Either expected check reports failure, expected result lines are absent, or the executable exits nonzero.

DCGM_FR_NCCL_ERROR; fail

The accompanying message distinguishes an NCCL result from an output or process-status problem.

The executable cannot be launched or its output cannot be read.

DCGM_FR_INTERNAL; fail

This is an execution failure and does not establish an interconnect defect.

A root-run executable fails the ownership or mode security checks.

Permission-specific DCGM_FR_* code; fail

Correct ownership and permissions rather than weakening the security check.

DCGM_NCCL_TESTS_BIN_PATH is unset, names an invalid directory, or does not lead to the expected regular executable.

Plugin unavailable or skipped

No NCCL health result was produced.

No GPU was selected.

DCGM_FR_INTERNAL; fail

Correct the entity selection and rerun the test.

is_allowed=false.

DCGM_FR_TEST_DISABLED; skip

The external NCCL test was not run.

See also#

dcgmi diag, Diagnostics, Diagnostic Errors