NVIDIA Deep Learning NCCL Documentation
Release Notes (PDF) - v2.20.5 - Last updated March 6, 2024

NCCL Release 2.16.5

This is the NCCL 2.16.5 release notes. For previous NCCL release notes, refer to the NCCL Archives.

Compatibility

NCCL 2.16.5 has been tested with the following:

Known Issues

  • Performance is sub-optimal for NCCL communicators using 1 NIC per node on DGX H100/HGX H100 + NDR. It can be worked around by setting the following parameter:

    NCCL_MIN_NCHANNELS=4

Fixed Issues

The following issues have been resolved in NCCL 2.16.5:

  • Fix speed of IB NDR links

  • Fix handling of EINTR in socket polling

  • Improve proxy progress scheduling

  • Fix resource cleanup

  • Fix double free on case of init failure

  • Fix crash in case of communicator abort

  • Revert performance optimization causing significant regressions on some AMD platforms

Updating the GPG Repository Key

To best ensure the security and reliability of our RPM and Debian package repositories, NVIDIA is updating and rotating the signing keys used by apt, dnf/yum, and zypper package managers beginning on April 27, 2022. Failure to update your repository signing keys will result in package management errors when attempting to access or install NCCL packages. To ensure continued access to the latest NCCL release, please follow the updated NCCL installation guide.