Is this page helpful?

NVIDIA NVSHMEM 3.7.2 Release Notes#

NVIDIA® NVSHMEM is an implementation of the OpenSHMEM specification for NVIDIA GPUs. The NVSHMEM programming interface implements a Partitioned Global Address Space (PGAS) model across a cluster of NVIDIA GPUs. NVSHMEM provides an easy-to-use interface to allocate memory that is symmetrically distributed across the GPUs. In addition to a CPU-side interface, NVSHMEM provides a NVIDIA® CUDA® kernel-side interface that allows CUDA threads to access any location in the symmetrically-distributed memory.

The release notes describe the key features, software enhancements and improvements, and known issues for NVSHMEM 3.7.2 and earlier releases.

Key Features and Enhancements#

The NVSHMEM release includes the following key features and enhancements:

  • Bug Fixes

    • Fixed InfiniBand connection setup on ports that require global routing headers (GRH) in the GPUNetIO transport.

    • Fixed hangs in GPUNetIO transport on interfaces with an active MTU other than 4 KiB by configuring the queue-pair MTU from the active port.

Compatibility#

NVSHMEM 3.7.2 has been tested with the following:

  • CUDA Toolkit:

    • 12.8

    • 12.9

    • 13.2

    • 13.3

  • CPUs:

    • x86 processors

    • NVIDIA Grace™ processors

  • GPUs:

    • NVIDIA Ampere

    • NVIDIA Hopper™

    • NVIDIA Blackwell

  • NCCL 2.30.4

Limitations#

  • NVSHMEM is not compatible with the PMI client library on Cray systems, and must use the NVSHMEM internal PMI-2 client library.

    • You must launch jobs with the PMI bootstrap by specifying --mpi=pmi2 to Slurm and NVSHMEM_BOOTSTRAP_PMI=PMI-2, or directly by using the MPI or SHMEM bootstraps.

    • You must also set PMI-2 as the default PMI by setting NVSHMEM_DEFAULT_PMI2=1 when you build NVSHMEM.

  • The libfabric transport currently does not support VMM, so you must disable VMM by setting NVSHMEM_DISABLE_CUDA_VMM=1.

  • Systems with PCIe peer-to-peer communication must do one of the following:

    • Provide InfiniBand to support NVSHMEM atomics API calls.

    • Use NVSHMEM’s UCX transport, which uses sockets for atomics if InfiniBand is absent.

  • nvshmem_barrier*, nvshmem_quiet, and nvshmem_wait_until only ensure ordering and visibility between the source and destination PEs. They do not ensure global ordering and visibility.

  • When built with GDRCopy, and when using InfiniBand on versions of the 460 driver prior to 460.106.00, NVSHMEM cannot allocate the complete device memory because of the inability to reuse the BAR1 space. This has been fixed in the CUDA release fixed in the CUDA release 460 driver from 460.106.00 forward.

  • IBGDA does not work with CX-4 when the link layer is Ethernet (RoCE).

  • NVSHMEM is not supported on Grace with Ada L40 platforms.

  • NVSHMEM is not supported in virtualized environments (VM).

  • User buffers registered with nvshmemx_buffer_register_symmetric lack support for libfabric transport to perform GPU-GPU communication over remote networks (EFA, Slingshot, etc.).

  • When registering Extended GPU memory (EGM) user buffers with nvshmemx_buffer_register_symmetric, the buffers on different PEs must belong to distinct CPU sockets within a node. You can achieve this by selecting GPUs on a different NUMA domain using the CUDA_VISIBLE_DEVICES environment variable.

  • When using the Libfabric transport with NVSHMEM_LIBFABRIC_PROVIDER=EFA, you must ensure that the libfabric environment variable FI_EFA_ENABLE_SHM_TRANSFER is set to 0 before launching the application. While NVSHMEM sets this variable during initialization, it may be ignored by the EFA provider if it was already initialized by the launcher, for example when using mpirun.

Breaking Changes#

  • NVSHMEM now ships per-architecture LLVM bitcode device libraries named libnvshmem_device_sm_<arch>.bc. Tools that assumed the bitcode device library was a single, architecture-independent libnvshmem_device.bc file must be updated to select the bitcode file that matches the target GPU architecture. A compatibility libnvshmem_device.bc entry may be present for sm_90 builds, but it should not be treated as portable across GPU architectures.

Known Issues#

  • The internal layout of RC-connected QPs changed starting in 3.5.21, causing ABI compatibility breakage when enabling IBGDA.

  • Complex types, which are enabled by setting NVSHMEM_COMPLEX_SUPPORT at compile time, are not currently supported.

  • When you enable UCX remote transport with NVSHMEM_REMOTE_TRANSPORT=UCX, you may observe a data mismatch when scaling 32 PEs or more on DGX-2 platform.