This is the NCCL 2.32.6 release notes. For previous NCCL release notes, refer to the NCCL Archives.

Compatibility

Key Features and Enhancements

This NCCL release includes the following key features and enhancements.

• Vera Rubin Performance: Added a pipeline-based cost model with Vera Rubin tuning data for Ring, Tree, NVLS, and NVLSTree. Added Vera Rubin models for symmetric AllGather, ReduceScatter, and AllReduce kernels. Sped up the topology graph search on large NVLink domains such as 32 to 40 GPU VR200 systems, reducing communicator initialization time. Tuner plugin interface v7 adds Rubin platform support.

• Device API Enhancements: Added an experimental Flow device API for building pipelined collective kernels over LSA and GIN connections with the Simple, LL, and LL128 protocols; the Flow API signatures are not stable and may change. Extended `ncclGetPeerPointer` and the resource-buffer peer pointer to address every load/store-reachable local peer by world rank, and added ncclDevCommCanGetPeerPointer to report whether a peer is reachable. Added the NCCL_GIN_CONNECTION_CUSTOM_ARRAY GIN connection type so applications can request connectivity to an explicit list of peers. Added ncclLsaCopyTma, a TMA-based variant of the ncclLsaCopy broadcast on Blackwell and later GPUs. Added a signal-efficient two-signal GIN barrier that GIN plugins can request through the v15 barrierOptions property. CFT teams now require and use CUDA logical-endpoint clique information to determine unicast and multicast team sizes. Added validation of GPU and NIC capabilities when establishing GDAKI connections, falling back to the proxy backend when GDAKI is unsupported and NCCL_GIN_TYPE is unset. Updated the GPI device backend for the latest libgpi device ABI, including 31-bit signal and counter IDs and extended transfer-size encoding, and updated the bundled GPUNetIO code to v5.0.0-rc3. Moved the IR-only C device ABI (ncclGin_C, ncclCoopAny) out of the public device headers into bindings/ir.

• Collectives and Runtime Enhancements: Added experimental collective kernels called General Kernels (GENK), built on top of the new Flow APIs. These kernels are compiled out by default to minimize binary size; build with SYM_GENK=1 and set NCCL_SYM_GENK_ENABLE=1 to try them. Added experimental AlltoAll_LL, a latency-oriented symmetric AlltoAll kernel for small and medium per-peer messages on LSA and mixed LSA/GIN communicators, selectable with NCCL_SYM_KERNEL=AlltoAll_LL. Added host RMA support on rail-only configurations such as NCCL_CROSS_NIC=0, where host RMA was previously disabled. Updated the GIN and RMA plugin interfaces to v15 and v17: connection setup receives an abort flag so ncclCommAbort can interrupt blocking GIN and RMA initialization, and the plugin rankStride is replaced by an explicit peerArray. Improved hierarchical CE collective bandwidth when inter-node traffic is distributed across multiple RMA contexts, including under CUDA graph capture. Reduced host munmap latency on ATS platforms such as GB200 by granting ncclMemAlloc memory access to the owning GPU only.

• Diagnostics and Profiling: Added profiler plugin interface v8 with opt-in NCCL-sourced GPU and CPU timestamps on every event. Added support for loading up to two profiler plugins at a time through a comma-separated NCCL_PROFILER_PLUGIN, each with its own event mask. Allowed NetPlugin profiler events to be collected independently of proxy events; profiler plugins must accept ProxyStep or NULL parents. Updated Inspector to record per-call profiler tags (requiring profiler v7 or later), emit the selected algorithm and protocol per collective in its JSON output, and ship a communication-pattern extractor; NCCL_INSPECTOR_NCCL_SOURCED_TIMINGS defaults to on. Added a RAS diagnostics check verifying that the network devices NCCL selected are consistent across the nodes of a communicator and within each node. Added the DIAG debug subsystem (NCCL_DEBUG_SUBSYS=DIAG) for active and RAS diagnostics details. Attached all GIN/GDAKI queue pairs to a dedicated hardware Q counter set so aggregate counters can be queried for GIN traffic; controlled by NCCL_GIN_GDAKI_Q_COUNTER (default on).

• Other Improvements: Enabled building NCCL for Windows on Arm64. Reduced library size by compressing device code with --compress-mode=size when building with CUDA 13 or newer. Added channel-major and step-major NVLS staging FIFO layouts that reduce NVLink uTLB pressure for large-message NVLS and NVLSTree on Rubin; NCCL_NVLS_FIFO_LAYOUT controls the selection. Built libnccl_device.bc and libnccl_device.ltoir with Clang/LLVM 23, and unified the IR build options as IR_ARCH and NCCL_EMIT_LLVM_IR. Relaxed ncclCommGrow so existing ranks may pass either the original ncclUniqueId or NULL. Added a thread-exclusive Put path to the EFA-GDA GIN backend and printed endpoint counts when a flush times out (Github PR #2478), (Github PR #2552). Invoked the build-time Python generators through python3, and resolved Git version metadata from the NCCL source tree for out-of-tree CMake builds (Github PR #2452), (Github PR #2451). Added a user-guide page on the determinism of NCCL reductions, and improved various other documentation.

• Contrib/Updates: Moved nccl_ep and nccl_m2n out of the NCCL repository; they are now developed at https://github.com/NVIDIA/nccl-extensions. Added contrib/tide, a community contribution for layout-aware FSDP2 collectives, built on top of NCCL device and host APIs. Updated nccl4rust for the NCCL 2.32 API and added a minimal AllReduce example.

Fixed Issues

The following issues have been resolved in NCCL 2.32.6:

• Fixed a PXN deadlock caused by an unnecessary blocking cudaHostRegister of legacy shared memory in the proxy process.

• Fixed single-node NVLS AllGather and ReduceScatter returning data in the wrong rank order when CUDA_VISIBLE_DEVICES permutes GPUs (Github PR #2377), (Issue #1906).

• Fixed a silent hang with network-registered buffers where the receiving proxy erased a rendezvous flag already published by the GPU (Github PR #2390), (Issue #2389).

• Fixed a NET/IB hang when the request counter wraps at UINT32_MAX and completions are matched to the wrong receive request.

• Fixed races in the NET/Socket transport, including a task-slot publication race that caused NULL data sends on aarch64, and a missing load ordering on the NET/IB CTS FIFO (Github PR #2365), (Github PR #2393), (Github Issue #1983).

• Fixed IB async-event threads handling events against the wrong device when NCCL_IB_DEVICE_PCI_ORDER reorders devices.

• Fixed communicator initialization failures on hosts with multi-port NICs when NCCL loads multiple network plugins.

• Fixed the example tuner's configuration optimizer selecting invalid measurements (Github PR #2453).

• Fixed CX9 NIC detection that misclassified BlueField-4 NICs as CX9 and missed CX9 virtual functions, preventing automatic rail and plane assignment.

• Fixed communicator initialization failures when combined per-rank topology data exceeds the topology XML fusion capacity (Github PR #2386), (Issue #2160).

• Fixed ncclAllReduce with ncclAvg on symmetric-memory buffers failing with an unhandled CUDA error for floating-point types (Github PR #2448), (Issue #2447).

• Fixed a non-blocking communicator becoming permanently unusable after one call returned ncclInvalidArgument (Github PR #2460), (Issue #2459).

• Fixed ncclCommShrink putting the parent communicator into an error state when the excluded rank list is invalid; it now returns ncclInvalidArgument (Github PR #2457).

• Fixed a crash in ncclCommInitRank when NCCL_ALGO, NCCL_PROTO, or NCCL_SYM_KERNEL contain empty entries (Github PR #2456), (Issue #2455).

• Fixed per-call maxCTAs bounds not being honored by kernel-based AlltoAll, Gather, and Scatter, and P2P channel counts exceeding maxCTAs after rounding to a power of two (Github PR #2435), (Issue #2421), (Issue #1598).

• Fixed ncclCommRegister returning success without a handle for CPU-backed buffers; it now returns a NULL handle and rejects a NULL handle pointer (Github PR #2433).

• Fixed ranks on distinct MIG compute instances of the same GPU being rejected as duplicate GPU ranks (Github Issue #1716).

• Fixed failures when cuMem host memory support is inconsistent across ranks. The enablement logic is no longer affected by CUDA forward compatibility.

• Fixed GDRCopy initialization failures on GPUs that support DMA-BUF export but not CPU mmap of DMA-BUF allocations.

• Fixed intermittent encrypted bootstrap failures caused by unread TLS session tickets.

• Fixed a hang or illegal memory access when a child communicator split from a `splitShare=1` parent reuses NVLS buffers in a different rank order.

• Fixed GIN and GIN-based symmetric kernels sending puts and signals to the wrong peer on communicators created with splitShare=1.

• Fixed CFT counted writes on the multicast path silently skipping memory binding, which could lead to hangs.

• Fixed a compiler error when calling put, putValue, or signal on a single-backend ncclGin_BackendOne or non-default ncclGin_BackendMask object (Github PR #2436).

• Fixed out-of-place AllGather performance on symmetric memory when the per-rank size is not a multiple of 256 bytes.

• Fixed a TRACE-only crash during tuning initialization for symmetric cost-model entries (Github Issue #2380).

• Fixed profiler kernel-channel completion timestamps being recorded before all warp groups finished (Github PR #2473).

• Fixed a use-after-free in RAS peer updates and a null dereference in RAS client collective handling (Github PR #2406).

• Fixed a command-injection vulnerability in the ib_write_bw diagnostics check by sanitizing host and device names.

• Fixed nccl4py cute GIN kernel compilation failures on CUDA 12, and CUDA compilation detection in headers when compiling with Clang (Github PR #2392), (Issue #1738).

• Fixed memory, file-descriptor, and resource leaks on various error paths (Github PR #2267), (Github PR #2364), (Github PR #2373), (Github PR #2376), (Github PR #2403), (Github PR #2404), (Github PR #2432), (Github Issue #2189).

Known Issues

• Compiling NCCL from source targeting PTX can fail due to CUDA device runtime library conflicts during linking. Directly targeting SASS does not have the same issue. This will be fixed in the next release.

Updating the GPG Repository Key

To best ensure the security and reliability of our RPM and Debian package repositories, NVIDIA is updating and rotating the signing keys used by apt, dnf/yum, and zypper package managers beginning on April 27, 2022. Failure to update your repository signing keys will result in package management errors when attempting to access or install NCCL packages. To ensure continued access to the latest NCCL release, please follow the updated NCCL installation guide.