Is this page helpful?

NVSHMEM Performance#

This section describes some key NVSHMEM performance considerations when developing applications using the NVSHMEM runtime.

Using 16-Byte Alignment Buffers in the Application#

Applications should prefer allocating GPU buffers that are 16-byte aligned because, internally, the NVSHMEM runtime will try to complete the data transfer using 16-byte writes, which are more efficient than 8-byte or unaligned writes.

Using nvshmem_*block for Heterogeneous Transports Configuration#

If the application workflow matches the control flow, use collective calls such as nvshmem_<OPS>_block. These calls provide better performance portability between remote transports and communication over NVIDIA (R) NVLink (R) because NVLink benefits from using multiple threads to drive higher transfer throughput.

Tuning Performance with Environment Variables#

NVSHMEM works well with its default settings, and most applications do not need to change them. When profiling shows that communication is a bottleneck, you can adjust the library’s behavior via environment variables. The following sections group the variables that most often affect performance by the area they control. For the complete list, accepted values, and defaults, refer to the Environment Variables reference.

Note

Change one setting at a time and measure its effect. The best values depend on your hardware, network, and communication pattern.

Remote Transport Selection#

The transport used for inter-node communication has a first-order effect on performance. NVSHMEM_REMOTE_TRANSPORT selects it, accepting ibrc (the default), ucx, libfabric, ibdevx, gpunetio, or none. On supported InfiniBand systems, the GPU-initiated GDAKI transports (IBGDA and GPUNetIO) can substantially improve performance of workloads sending out many very small messages in parallel. Refer to Using GPUDirect Async Kernel-Initiated Communication for how to enable them, and to Tuning the Queue-Pair Type and Configuration for IBGDA and GPUNetIO for queue-pair tuning.

NIC Selection and PE-to-NIC Mapping#

On multi-NIC nodes, mapping PEs to the appropriate NICs improves bandwidth and avoids contention. By default, each PE is assigned to the NIC closest to it. To customize this assignment, use the NVSHMEM_HCA_LIST, NVSHMEM_ENABLE_NIC_PE_MAPPING, and NVSHMEM_HCA_PE_MAPPING variables; refer to the Environment Variables reference for their accepted values and behavior.

Point-to-Point and Proxy Batching#

  • NVSHMEM_PROXY_REQUEST_BATCH_MAX, NVSHMEM_TRANSPORT_BATCH_MAX_OPS, NVSHMEM_IBGDA_NUM_REQUESTS_IN_BATCH, NVSHMEM_GPUNETIO_NUM_REQUESTS_IN_BATCH control how many requests are batched on the transport path.

TMA#

NVSHMEM_TMA_POLICY controls use of the Tensor Memory Accelerator for eligible device-side point-to-point operations over NVLink. Valid values are DISABLE, ENABLE, and FORCE. Refer to the TMA guide for details on the requirements and behavior of this path.

Isolating Performance with Feature Toggles#

NVSHMEM enables its hardware-acceleration paths by default. The NVSHMEM_DISABLE_* variables — for example, NVSHMEM_DISABLE_P2P, NVSHMEM_DISABLE_NVLS (NVLink SHARP collectives), and NVSHMEM_DISABLE_GDRCOPY — turn individual paths off even when they are available. These are primarily diagnostic: disabling a path can confirm whether it is responsible for a performance or correctness problem. Leave them enabled for production runs unless you have a specific reason to disable them.

Diagnostics and Profiling#

  • NVSHMEM_INFO prints the environment variable options that are in effect at startup.

  • NVSHMEM_DEBUG controls diagnostic logging. Refer to the Debugging FAQs for related guidance.

  • NVSHMEM_NVTX enables NVTX instrumentation so NVSHMEM operations are visible in profilers such as NVIDIA Nsight Systems. It accepts a comma-separated list of instrumentation groups.

Tuning the Queue-Pair Type and Configuration for IBGDA and GPUNetIO#

The NVSHMEM IBGDA transport and the GPUNetIO GPU data path implement the GPUDirect Async Kernel-initiated technology (GDAKI) for Mellanox Connect-X devices.

GDAKI offloads the entire process of sending memory over the remote network to the GPU. These GDAKI paths differ from other transports that rely on passing messages to a CPU proxy thread to initiate the transfer. Submitting the transfer directly from the GPU allows for a highly parallel submission of requests to the network interface. The increased message rate for small messages allows users to saturate the network bandwidth using a much smaller data buffer.

The increased parallelism comes with unique performance characteristics and challenges:

  • A thread on the GPU cannot fill out a work queue entry (WQE) as quickly as a CPU thread. This results in a higher latency for individual messages.

  • Providing a unique queue-pair (QP) for each GPU thread is not always feasible from a resource allocation perspective, and it increases quiet and fence latency. The following DCI and RC options balance the trade-offs between memory consumption and performance.

Deciding between DCI and RC QPs

RC QPs provide the lowest latency for message transfers. However, the memory overhead of creating a dedicated QP for each remote PE can be prohibitive for large-scale applications. For medium-scale and small-scale jobs, where total number of connections is not crucial, we recommend that you use RC QPs.

DCI QP configuration

  • NVSHMEM_IBGDA_NUM_DCI: The number of Dynamic Connection Initiators (DCI) to create per PE.

    Dynamically connected QPs allow for a DCI to connect to multiple remote Dynamic Connection Targets (DCT). In NVSHMEM, this means that each PE can have a DCI that connects to one or more DCTs on each remote PE. However, the flexibility and memory savings achieved by DCT are at the cost of reduced performance relative to a dedicated QP (for example, Reliable Connected or RC). Each time the DCI switches to a different DCT, the connection must be renegotiated with a half-handshake protocol that adds latency to the transfer.

    The default value is equal to the number of SMs on the device.

  • NVSHMEM_IBGDA_NUM_DCT: The number of DCTs created by each PE.

    This value should be set to at least 2 to guarantee full use of the underlying NIC bandwidth.

    The default value is 2.

  • NVSHMEM_IBGDA_DCI_MAP_BY includes the following options:

    • cta: Select which DCI(s) to use based on the CUDA thread block, also known as the Cooperative Thread Array (CTA), that calls the NVSHMEM API.

      A CTA is guaranteed to be scheduled to a Streaming Multiprocessor (SM), but multiple CTAs may exist on the same SM. This option is useful for applications that schedule workloads with multiple CTAs per SM. If NVSHMEM_IBGDA_NUM_DCI is not set, NVSHMEM creates one DCI per SM by default. For example, when launching a maximum number of CTAs smaller than the total number of SMs, set the value to the maximum number of CTAs that will be used simultaneously (Low Resource, Low Synchronization Overhead).

    • sm: This option is similar to the previous option but uses the underlying hardware SM to determine which DCI to use.

      This option is useful for users who are scheduling work in units that match the size of an SM. Like the option above, if NVSHMEM_IBGDA_NUM_DCI is not set, NVSHMEM creates one DCI per SM by default. This option has additional overhead relative to cta. Because multiple CTAs can map to an SM, heavier synchronization is required when submitting operations to the QP. For cta, you can use block-scoped atomics to avoid contention on the QP; for sm, a full device atomic is required, which has greater overhead.

    Note

    The cta mapping is the preferred option (Low Resource, High Synchronization Overhead).

    • warp: This option is useful for applications that drive I/O to multiple peers from a smaller subset of CTAs.

      For example, in an all-to-all communication pattern with 32 warps per CTA and 32 peers, users can set the number of DCIs to 32 and use the warp mapping to ensure that each warp has a dedicated DCI. By mapping this way, users can avoid the penalty of serializing submissions to the QP and renegotiating the connection for each warp submitting operations. This option is more memory intensive than the previous options. If NVSHMEM_IBGDA_NUM_DCI is not set, NVSHMEM creates num_warps_per_sm * num_sms DCIs by default (High Resource, Low Synchronization Overhead).

    • dct: If there are sufficient DCIs, this option locks each DCI to one specific DCT.

      This option forces a one-to-one mapping from DCIs to DCTs, which eliminates the need to reestablish a connection between iterations, even if the DCI is used by different warps, CTAs, or SMs over time. However, some of the memory-saving benefits of choosing DCT over RC are lost. Users can specify fewer DCIs in NVSHMEM_IBGDA_NUM_DCI to complete a one-to-one mapping with DCTs. In this case, each DCI will be locked to a small subset of DCTs. Users might specify a smaller number of DCIs than DCTs, so this option is still more memory efficient than RC, which requires the creation and connection on both ends for each QP. This option also requires a device-scoped atomic to guarantee correctness (High Resource, High Synchronization Overhead).

    The default value is cta.

RC QP configuration

  • NVSHMEM_IBGDA_NUM_RC_PER_PE: The number of Reliable Connected (RC) QPs on the local PE to create per remote PE.

    The default value is 2.

  • NVSHMEM_GPUNETIO_NUM_RC_PER_PE_GPU: The number of GPU-data-path RC QPs to create per remote PE for the GPUNetIO GDAKI path.

    The default value is 2.

NVSHMEM_IBGDA_RC_MAP_BY configures RC QP mapping for IBGDA.

NVSHMEM_GPUNETIO_RC_MAP_BY configures RC QP mapping for the GPUNetIO GDAKI path.

Both variables accept the following options:

  • none (default): Disable CTA-, SM-, or warp-based mapping and use atomic round-robin selection across the configured RC QPs and NICs. This mapping provides balanced RC QP and NIC use for high-message-rate workloads, especially when using multiple RC QPs or NICs.

  • cta: Select an RC QP based on the CTA ID.

  • sm: Select an RC QP based on the SM ID.

  • warp: Select an RC QP based on the warp ID.

The fixed cta, sm, and warp mappings remain available for workload-specific tuning. They use atomic round-robin NIC selection in multi-NIC configurations. Change the mapping only after measuring the application with the default none mapping.

When both DCI and RC are configured, or when neither is explicitly configured, IBGDA uses RC for connections. The none choice applies only to RC QP mapping and does not change IBGDA DCI mapping.