NVIDIA NCCL#

NVIDIA Collective Communication Library (NCCL) provides GPU collective and point-to-point communication for distributed AI and HPC applications. On Vera-Rubin NVL72, NCCL must model three high-bandwidth paths: intra-tray Rubin GPU connectivity, rack-scale NVLink 6 through NVLink switch trays, and ConnectX-9 east-west networking.

Refer to the NCCL documentation and the NCCL source repository for public API and environment-variable details.

Release Stack#

Use the NCCL, CUDA toolkit, driver, network plugin, firmware, and platform image that are qualified together for the installed platform release. Confirm supported versions and features in public release documentation before using a release-dependent capability.

Avoid mixing components from different release trains. Before validating a new stack, preserve required release and site configuration while removing unnecessary application tuning and diagnostic overrides. Use that qualified configuration as the baseline, and record the effective settings and complete software and firmware set with every result so a regression can be reproduced.

Topology And Rail Validation#

Start from discovered topology instead of fixed interface names. NCCL should identify the 72-GPU NVLink 6 domain inside a rack and the intended CX9 devices for scale-out traffic. Network planes are independent fabrics, while rails group paths that provide locality and bandwidth within a plane. A valid mapping must preserve both GPU/NIC locality and the configured rail and plane layout.

Retain the release-qualified automatic rail assignment and NIC fusion defaults provided by the topology model and network plugin. If the deployment requires an explicit mapping, use the mapping delivered for that fabric and verify it against current PCIe, GPU, NIC, switch, and port discovery. Do not derive a persistent map from mlx5 numbering or from another rack configuration.

Validate in stages:

  • One compute tray to establish local GPU and NVLink behavior.

  • Multiple trays to expose switch, cable-backplane, rail, or plane asymmetry.

  • A full rack to validate the complete 72-GPU NVLink domain.

  • Multiple racks to validate CX9 rail balance and failure behavior.

At each stage, compare all-reduce, reduce-scatter, all-gather, all-to-all, and send/receive behavior. Exercise each network plane and confirm that traffic is balanced across the expected rails. Revalidate after image, driver, firmware, network-plugin, cabling, or fabric-policy changes.

Memory Localization And GPUDirect RDMA#

Rubin execution and memory can be localized. The resulting communication path depends on the allocation type and on support in the installed CUDA and NCCL stack. A localized allocation may require NCCL to use a non-localized internal buffer or host staging when a direct GPUDirect RDMA path is unavailable.

Use the allocation and registration path recommended by the current release. For each test, record whether buffers are user-allocated or library-allocated, localized or non-localized, registered or staged, and whether GPUDirect RDMA is actually active. Validate partitioned-GPU or MPS modes separately because their execution and memory placement can change both rank count and transport choice.

When performance differs between local and scale-out collectives, first check buffer placement, registration, GPU/NIC locality, and the selected rail before changing chunk sizes or transport policy. Change one control at a time and retain a comparison with the qualified baseline configuration.

Algorithms And Protocols#

Allow NCCL’s topology model to select algorithms and protocols by default. Validate Ring and Tree behavior, plus NVLS and NVLSTree when they are supported by the qualified release. Cover small, medium, and large messages because the best path and the dominant bottleneck can change with message size and collective type.

Treat Rubin-specific scheduling, localization, and data-movement modes as release-dependent capabilities. Enable or force one only when the installed release documents it, and compare the result against automatic selection. Check NVLink health in both directions before attributing a collective regression to algorithm choice.

Diagnostics And Regression Testing#

Use nccl-tests as a repeatable component test rather than as the only measure of application performance. Keep the launch policy, rank placement, message sweep, power profile, and rail mapping fixed when comparing releases.

Useful public diagnostics include:

  • NCCL_DEBUG=INFO for initialization and transport logs.

  • NCCL_DEBUG_SUBSYS=INIT,GRAPH,NET for focused startup, graph, and network detail.

  • NCCL_TOPO_DUMP_FILE to preserve NCCL’s detected topology.

  • NCCL_NET or NCCL_IB_HCA only for a controlled path-selection test.

  • NCCL_RUN_RAS_DIAGNOSTICS=1 for passive checks at communicator initialization: GPU inventory, driver versions, NVLink state, and NCCL settings compared across ranks.

  • NCCL_RUN_DIAGNOSTICS=1 for active checks at communicator initialization: GPU-to-GPU peer access, and from NCCL 2.32 network bandwidth over selected network paths between hosts using ib_write_bw.

Built-in diagnostics are available from NCCL 2.31. Export both variables as 1 to run both categories during communicator initialization. Passive checks require the reliability, availability, and serviceability (RAS) subsystem, enabled by default. ncclras -D requests only passive RAS checks for a running job; it does not rerun active tests. Capture the client’s standard output for on-demand reports.

The peer-access check covers GPU pairs expected to communicate directly, including across trays when multi-node NVLink is selected. Network checks run only when the communicator uses a network transport across at least two hosts; rack boundaries alone do not determine whether they run. They require ib_write_bw on every participating node and mutually resolvable hostnames. Only the first-channel device per rank is measured, not every rail. GPU-memory measurements can fall back to host memory when endpoint or tool support is unavailable.

At initialization, rank 0 prints a summary prefixed with NCCL DIAG. Collect all ranks’ output and review [INFO] results for failed, skipped, or incomplete checks. Diagnostic failures are informational and do not abort communicator initialization; a successful application exit is not proof that every check passed. See NCCL Diagnostics and RAS diagnostics for commands, prerequisites, and output format.

Archive the topology dump and logs from a known-good run. For a regression, classify the first failing scope as local NVLink, a switch or backplane segment, a network rail or plane, buffer localization or registration, or the complete software stack. Do not retain diagnostic overrides as permanent production settings after the underlying issue is resolved.