Is this page helpful?

NVIDIA® NVSHMEM 3.1.7 Release Notes#

NVSHMEM is an implementation of the OpenSHMEM specification for NVIDIA GPUs. The NVSHMEM programming interface implements a Partitioned Global Address Space (PGAS) model across a cluster of NVIDIA GPUs. NVSHMEM provides an easy-to-use interface to allocate memory that is symmetrically distributed across GPUs. In addition to a CPU-side interface, NVSHMEM also provides an NVIDIA CUDA® kernel-side interface that allows NVIDIA CUDA® threads to access any location in the symmetrically distributed memory.

The release notes describe the key features, software enhancements and improvements, and known issues for NVIDIA® NVSHMEM 3.1.7 and earlier releases.

Key Features and Enhancements#

This NVSHMEM release includes the following key features and enhancements:

  • Added support for NVLINK SHARP (NVLS)-based collective algorithms on x86, Hopper, and Grace Hopper architecture-based single- and multi-node NVLINK platforms for popular deep-learning collective communications (ReduceScatter, Allgather, and Allreduce) device and on-stream APIs. This feature improves latency for small messages by a 2-3x speedup compared with one-shot algorithms over NVLINK.

  • Added support for GPU kernels that want to use a low-level query API to NVLS-enabled symmetric memory using the nvshmemx_mc_ptr host and device APIs for a given target team.

  • Added support for a new low-latency protocol (LL128) for Allgather collective communication device and on-stream APIs.

  • Enhanced support for the existing low-latency protocol (LL) warp-scoped collective to provide a 2x speedup over traditional algorithms when scaling the number of GPUs up to 32.

  • Added support for the half-precision (FP16/BF16) format on collective communication (ReduceScatter, Allgather, and Allreduce) on-device and on-stream APIs.

  • Added support for Python wheels using the PyPI repository and rpm/deb package distributions.

  • Added support for dynamic RDMA Global Identifier (GID) discovery for RoCE transports. This feature enables automatic fallback to the discovered GID without requiring users to specify the GID using a runtime variable.

  • Added support for a heterogeneous library build system. This feature allows the NVSHMEM static library to be built with a separate CUDA version from the NVSHMEM host library. This enables new features, such as NVLS in the host library, and allows applications compiled against lower versions of CUDA to link to the NVSHMEM device library. This makes the entire library portable to different CUDA minor versions and remain feature-complete. Users can specify a CUDA version for the device library by setting NVSHMEM_DEVICELIB_CUDA_HOME=<PATH TO CUDA>. Otherwise, the host CUDA version will be used.

  • Enhanced support for NVSHMEM on-stream signal APIs to use cuStreamWriteValue over P2P-connected GPUs when possible. This enables a zero-SM implementation of the on-stream signaling op.

  • Added support for DMABuf-based registration of NIC control structures in IBGDA to leverage DMABuf mainline support in newer Linux kernels over the proprietary solution nvidia-peermem.

  • Added sample code for the NVSHMEM UniqueID (UID) socket-based bootstrap modality under the examples directory.

  • Added support for NVSHMEM performance benchmarks to the release binary packages.

  • Enhanced collectives performance reporting by adding the Algorithmic Bandwidth (algoBW) and Bus Bandwidth (BusBW) metrics to the NVSHMEM performance benchmarks.

  • Enhanced the reduce-based collective symmetric memory scratch space to 512 KB to accommodate the additional space required for reducescatter-based collectives.

  • Fixed support for the Ninja build generator in the CMake build system.

  • Added support for the symmetric memory management unit test framework.

  • Fixed several minor bugs and memory leaks.

Compatibility#

NVSHMEM 3.1.7 has been tested with the following:

NVIDIA CUDA® Toolkit:

  • 11.8

  • 12.2

  • 12.6

CPUs

  • On x86 and NVIDIA GraceTM processors.

GPUs

  • Volta V100

  • Ampere A100

  • NVIDIA HopperTM

Limitations#

  • NVSHMEM is not compatible with the PMI client library on Cray systems and must use the NVSHMEM internal PMI-2 client library.

    • Jobs can be launched with the PMI bootstrap by specifying --mpi=pmi2 to Slurm and NVSHMEM_BOOTSTRAP_PMI=PMI-2 or directly by using the MPI or SHMEM bootstraps.

    • PMI-2 can also be set as the default PMI by setting NVSHMEM_DEFAULT_PMI2=1 when building NVSHMEM.

  • The libfabric transport does not yet support VMM, so VMM must be disabled by setting NVSHMEM_DISABLE_CUDA_VMM=1.

  • Systems with PCIe peer-to-peer communication require one of the following:

    • InfiniBand to support NVSHMEM atomics APIs.

    • Using NVSHMEM’s UCX transport that, if IB is absent, will use sockets for atomics.

  • nvshmem_barrier*, nvshmem_quiet, and nvshmem_wait_until only ensure ordering and visibility between the source and destination PEs and do not ensure global ordering and visibility.

  • When built with GDRcopy and when using InfiniBand on earlier versions of the 460 driver and previous branches, NVSHMEM cannot allocate the complete device memory because it cannot reuse the BAR1 space. This was fixed in the latest NVIDIA CUDA® 460 driver and in release 470 and later.

  • IBGDA transport support on NVIDIA Grace HopperTM systems is experimental in this release.

  • IBGDA does not work with DMABUF.

  • IBGDA does not work with CX-4 when the link layer is Ethernet (RoCE).

  • NVSHMEM is not supported on Grace + Ada L40 platforms.

Fixed Issues#

  • Fixed a bug related to the incorrect use of the NVSHMEM_DEVICE_TIMEOUT_POLLLING build-time variable.

  • Fixed a performance bug in the on-stream collectives perf test related to using cudaMemcpyAsync on the same CUDA stream, where cudaEvent is used to profile the start and end times of the on-stream communication kernel.

  • Fixed a bug related to virtual member functions of nvshmemi_symmetric_heap by forcing its access specifier to protected to limit access to inherited child classes only.

  • Fixed a bug related to recursive destructor memory corruption and nullptr access in the static member function of the nvshmemi_mem_transport class.

  • Fixed a bug related to the incorrect compile-time definitions for the NVML_GPU_FABRIC_STATE_COMPLETED and NVML_GPU_FABRIC_UUID_LEN constants.

  • Fixed a bug related to nvshmemx_collective_launch_query_gridsize, which could cause it to erroneously return a grid size of 0.

  • Fixed a bug related to nvshmem_init, which might cause the application to crash in MNNVL discovery when used with the CUDA compat libraries at runtime for CUDA Toolkit 12.4 or later.

  • Fixed a bug related to nvshmemx_collective_launch, which might cause duplicate initialization of the NVSHMEM device state.

  • Fixed a bug related to uninitialized variables in the IBGDA device code.

  • Fixed a bug related to out-of-bounds access (OOB) in the atomic BW performance test.

  • Fixed a bug related to missing C/C++ stdint headers on Ubuntu 24.04 and x86-based systems.

  • Fixed a bug related to the incorrect calculation of team-specific stride when creating a new team using nvshmem_team_split_strided.

Breaking Changes#

There are no breaking changes in this release.

Deprecated Features#

  • Removed host API-based nvshmem collectives performance benchmarks.

Known Issues#

  • Complex types, which are enabled by setting NVSHMEM_COMPLEX_SUPPORT at compile time, are not currently supported.

  • When enabling libfabric transport with NVSHMEM_LIBFABRIC_PROVIDER=EFA, certain operations are experimental and might result in the application kernel hanging with the following operations:

    • Device side nvshmem_put/nvshmem_get with nvshmem_barrier.

    • Host side nvshmem_put_on_stream/nvshmem_get_on_stream.

  • When enabling UCX remote transport with NVSHMEM_REMOTE_TRANSPORT=UCX, a data mismatch might be observed when scaling to 32 PEs or more on DGX-2 platforms.