This is the NCCL 2.32.3 release notes. For previous NCCL release notes, refer to the NCCL Archives.

Compatibility

Key Features and Enhancements

This NCCL release includes the following key features and enhancements.

• Vera Rubin Support: Added initial Rubin platform support including support for sm107, CX9 rail and plane detection, and MPS+MLoPart. NCCL 2.32.3 focuses on new functionalities and does not contain performance model tuning for Rubin. This will be part of the next release to improve out-of-box experience for Rubin users.

• Device API Enhancements: Added Compute Fabric Transport (CFT) counted-write and wait support. Added socket-based GIN support, enabling custom kernel development with TCP sockets. Added support in GDAKI for LAG-aware QP assignment based on the context ID (Github PR #2315). Added NCCL_WIN_GIN_ONLY so users can register a window only for GIN usage. Optimized GIN performance by skipping mcst operation when possible.

• Collectives and Runtime Enhancements: Added a ring-based hierarchical copy-engine AllGather implementation selectable with NCCL_HIER_CE_COLL_AG_RAIL_RING_ENABLE (Github PR #2299). Improved Blackwell symmetric AllGather performance, resource overhead modeling and kernel selection with a new cost model. Added optional TLS encryption for NCCL-owned socket traffic when NCCL is built with OpenSSL3, configured through the ncclSetEncryption API. Added ncclCollConfig_t::launchCompletionEvent, allowing callers to observe kernel-launch completion. Added ncclNvlsHostMode_t to ncclConfig_t so applications can disable host NVLS collectives per communicator. Optimized NVLS slot consumption to avoid resource exhaustion issues when using multiple communicators with NVLS.

• Diagnostics and Profiling: Added the ATTN log level for important non-fatal conditions, such as configuration fallbacks and plugin initialization failures. Expanded RAS capability with GPU-resident progress counters and a watchdog DMA mirror for diagnosing stalled collective kernels. Added additional RAS diagnostics functionalities for NVLink/NIC state and speed, PCI and GDR configuration, Xid/SXid events and others. Reported degraded NVLink fabric bandwidth with an ATTN message during communicator initialization. Reported mismatched NCCL Git revisions across communicator ranks during initialization.

• Other Improvements: Fixed PAT+NVLS performance drops on H100 platforms. Avoided address space exhaustion when repeatedly registering symmetric windows backed by a single physical memory allocation. Reduced communicator initialization overhead at large scale by avoiding scans of inactive proxy poll descriptors. Added an experimental built-in NetworkDirect transport on Windows, with automatic socket fallback. EFA team improved EFA GDA support with gin.get API and others (Github PR #2382)(Github PR #2398)(Github PR #2405). Reported Inspector ring-buffer drops and operation counts in JSON (Github PR #2304). NCCL now supports communicators using multiple MIG instances. Generated llms.txt for agents to better read NCCL documentation.

• Contrib/Updates: Added more functionality in NiiN. Fixed contrib/nccl_checkpoint rejecting otherwise compatible NCCL versions (Github Issue #2347).

Fixed Issues

The following issues have been resolved in NCCL 2.32.3:

• Fixed profiler overhead when no profiler plugin is loaded (Github Issue #2355).

• Fixed P2P IPC registration reuse producing out-of-bounds remote addresses when a registered allocation spans multiple cuMem segments (Github PR #2362).

• Fixed PAT connection-setup deadlocks when runtime connection is disabled and nodes have uneven local-rank counts (Github Issue #2385).

• Fixed non-thread-safe token parsing that could corrupt concurrent configuration parsing (Github Issue #2361).

• Fixed B40 TMA symmetric kernels crash due to insufficient shared memory.

• Fixed GIN GDAKI bug related to hop limit when using DOCA SDK.

• Fixed GIN Proxy and GPI descriptor shared-memory sizing and alignment.

• Fixed CFT window registration performed before ncclDevCommCreate.

• Fixed profiler API events reporting rank 0 instead of the originating communicator rank (Github Issue #2300).

• Fixed tuner plugins receiving uninitialized cost-model constants after the tuning rework.

Known Issues

• Rubin performance model has not been optimized with NCCL 2.32.3. Users can observe better performance on Rubin by increasing the number of CTAs NCCL uses. Automatic tuning will be improved in the next release.

• Socket GIN currently requires users to opt-in to enable GDRCopy.

• B100 PCIe MLoPart workloads can fail during memory allocation with CUDA error 101 (invalid device ordinal). Set NCCL_CUMEM_ENABLE=0 as a workaround.

Updating the GPG Repository Key

To best ensure the security and reliability of our RPM and Debian package repositories, NVIDIA is updating and rotating the signing keys used by apt, dnf/yum, and zypper package managers beginning on April 27, 2022. Failure to update your repository signing keys will result in package management errors when attempting to access or install NCCL packages. To ensure continued access to the latest NCCL release, please follow the updated NCCL installation guide.