NCCL Release 2.27.6

This is the NCCL 2.27.6 release notes. For previous NCCL release notes, refer to the NCCL Archives.

Compatibility

NCCL 2.27.6 has been tested with the following:

  • Deep learning framework containers. Refer to the Support Matrix for the supported container version.

  • This NCCL release supports CUDA 12.2, CUDA 12.4, and CUDA 12.9. The provided prebuilt binaries should work with other CUDA 12.x versions as well.

Key Features and Enhancements

This NCCL release includes the following key features and enhancements.

  • Improved support for DirectNIC (CX8): XDR speed detection and the reporting of the RDMA interfaces only when DirectNIC is enabled.

  • Completed P2C (PXN over C2C) support, which is now preferred over regular PXN and extends to send/receive operations as well. This feature is currently preliminary and is disabled by default; use NCCL_PXN_C2C=1 to enable.

  • Added support for compilation with GCC 14.

Fixed Issues

The following issues have been resolved in NCCL 2.27.6:

  • Fixed the unloading of network plugins that also provide tuner capability.

  • Fixed the change of the current device across the calls to ncclCommDestroy() and ncclCommAbort().

Known Issues

  • A note for users on MNNVL systems: please ensure an adequate stack size for NCCL threads. While the default Linux stack size limit of 8192 KB is known to be sufficient, we've seen crashes if the limit is changed to "unlimited," as it causes the glibc library to unexpectedly decrease the stack size of NCCL's background threads to just 2048 KB. Use ulimit -s in bash to print the current limit; if needed, reset it to 8192 KB using ulimit -s 8192. (You also need to ensure that the new setting is propagated to other nodes when launching a multi-node NCCL job).

Updating the GPG Repository Key

To best ensure the security and reliability of our RPM and Debian package repositories, NVIDIA is updating and rotating the signing keys used by apt, dnf/yum, and zypper package managers beginning on April 27, 2022. Failure to update your repository signing keys will result in package management errors when attempting to access or install NCCL packages. To ensure continued access to the latest NCCL release, please follow the updated NCCL installation guide.