NVSHMEM Performance#

This section describes some key NVSHMEM performance considerations when developing applications using NVSHMEM runtime.

Using 16-Byte Alignment Buffers in the Application#

The application will prefer to use allocating GPU buffers that are 16-byte aligned because, internally, the NVSHMEM runtime will try to complete the data transfer using 16-byte writes that are the more efficient than 8-byte or unaligned writes.

Using nvshmem_*block for Heterogenous Transports Configuration#

If the application workflow matches the control flow, it will use collective calls such as nvshmem_<OPS>_block. These calls provide better performance portability between remote transports and communication over NVIDIA (R) NVLink (R) because NVLink benefits from using multiple threads to drive a higher transfer throughput.

Tuning Performance with Environment Variables#

NVSHMEM works well with its default settings, and most applications do not need to change them. When profiling shows that communication is a bottleneck, you can adjust the library’s behavior via environment variables. The following sections group the variables that most often affect performance by the area they control. For the complete list, accepted values, and defaults, refer to the Environment Variables reference.

Note

Change one setting at a time and measure its effect. The best values depend on your hardware, network, and communication pattern.

Remote Transport Selection#

The transport used for inter-node communication has a first-order effect on performance. NVSHMEM_REMOTE_TRANSPORT selects it, accepting ibrc (the default), ucx, libfabric, ibdevx, gpunetio, or none. On supported InfiniBand systems, the GPU-initiated GDAKI transports (IBGDA and GPUNetIO) can substantially improve performance of workloads sending out many very small messages in parallel. Refer to Using GPUDirect Async Kernel-Initiated Communication for how to enable them, and to Tuning the queue-pair Type and Configuration for IBGDA for queue-pair tuning.

NIC Selection and PE-to-NIC Mapping#

On multi-NIC nodes, mapping PEs to the appropriate NICs improves bandwidth and avoids contention. By default, each PE is assigned to the NIC closest to it. To customize this assignment, use the NVSHMEM_HCA_LIST, NVSHMEM_ENABLE_NIC_PE_MAPPING, and NVSHMEM_HCA_PE_MAPPING variables; refer to the Environment Variables reference for their accepted values and behavior.

Point-to-Point and Proxy Batching#

NVSHMEM_PROXY_REQUEST_BATCH_MAX, NVSHMEM_TRANSPORT_BATCH_MAX_OPS, NVSHMEM_IBGDA_NUM_REQUESTS_IN_BATCH, NVSHMEM_GPUNETIO_NUM_REQUESTS_IN_BATCH control how many requests are batched on the transport path.

TMA#

NVSHMEM_TMA_POLICY controls use of the Tensor Memory Accelerator for eligible device-side point-to-point operations over NVLink. Valid values are DISABLE, ENABLE, and FORCE. Refer to the TMA guide for details on the requirements and behavior of this path.

Isolating Performance with Feature Toggles#

NVSHMEM enables its hardware-acceleration paths by default. The NVSHMEM_DISABLE_* variables — for example, NVSHMEM_DISABLE_P2P, NVSHMEM_DISABLE_NVLS (NVLink SHARP collectives), and NVSHMEM_DISABLE_GDRCOPY — turn individual paths off even when they are available. These are primarily diagnostic: disabling a path can confirm whether it is responsible for a performance or correctness problem. Leave them enabled for production runs unless you have a specific reason to disable them.

Diagnostics and Profiling#

NVSHMEM_INFO prints the environment variable options that are in effect at startup.
NVSHMEM_DEBUG controls diagnostic logging. Refer to the Debugging FAQs for related guidance.
NVSHMEM_NVTX enables NVTX instrumentation so NVSHMEM operations are visible in profilers such as NVIDIA Nsight Systems. It accepts a comma-separated list of instrumentation groups.

Tuning the queue-pair Type and Configuration for IBGDA#

The NVSHMEM IBGDA transport is an implementation of the GPUDirect Async Kernel-initiated technology (GDAKI) for Mellanox Connect-X devices.

GDAKI offloads the entire process of sending memory over the remote network to the GPU. This transport differs from other transports that rely on passing messages to a CPU proxy thread to initiate the transfer. Submitting the transfer directly from the GPU allows for a highly parallel submission of requests to the network interface. The increased message rate for small messages allows users to saturate the network bandwidth using a much smaller data buffer.

The increased parallelism comes with unique performance characteristics and challenges:

A thread on the GPU cannot fill out a work queue entry (WQE) as quickly as a CPU thread. This results in a higher latency for individual messages.
Providing a unique queue-pair (QP) for each GPU thread is not always feasible from a resource allocation perspective, and it increase quiet and fence latency. The IBGDA transport provides several tuning parameters to balance the trade-offs between memory consumption and performance.
The first set of options controls the type of QP used in the transport, and here are the options:
NVSHMEM_IBGDA_NUM_DCI: The number of Dynamic Connection Initiators (DCI) to create per PE.

Dynamically connected QPs allow for a DCI to connect to multiple remote Dynamic Connection Targets (DCT). In NVSHMEM, this means that each PE can have a DCI that connects to one or more DCTs on each remote PE. However, the flexibility and memory savings achieved by DCT are at the cost of reduced performance relative to a dedicated QP (for example, Reliable Connected or RC). Each time the DCI switches to a different DCT, the Connection must be renegotiated with a half-handshake protocol that adds latency to the transfer.

The default value is equal to the number of SMs on the device.

NVSHMEM_IBGDA_NUM_DCT: The number of DCTs created by each PE.

This value should be set to at least 2 to guarantee full use of the underlying NICs bandwidth.

The default value is 2.

NVSHMEM_IBGDA_NUM_RC_PER_PE: The number of Reliable Connected (RC) QPs on the local PE to create per remote PE.

RC QPs provide the lowest latency for message transfers. However, the memory overhead of creating a dedicated QP for each remote PE can be prohibitive for large-scale applications. For medium-scale and small-scale jobs, where total number of connections is not crucial, we recommend that you use RC QPs.

The default value is 2.
If DCI and RC are specified by users, the transport will only use RC for connections. If neither is specified, the transport will default to using DCI. Both options have different default values based on the mapping type, but they can be overridden by users. This is advisable in certain cases as outlined below.
- The second set of options can be combined with the first set to provide better performance for certain workload types or to reduce the total number of QPs required by the host and target. These options come in two flavors that correspond to the two types of QPs.
  
  NVSHMEM_IBGDA_DCI_MAP_BY includes the following options:
  
  CTA: Select which DCI(s) to use based on the CUDA thread block, which is also known as the Cooperative Thread Array (CTA) that calls the NVSHMEM API.
  
  A CTA is guaranteed to be scheduled to a Streaming Multiprocessor (SM), but multiple CTAs might exist on the same SM. This option is useful for applications that schedule workloads where multiple CTAs are scheduled per SM. If IBGDA_NUM_DCI is not set by users, NVSHMEM will by default create one DCI per SM. For example, when you are launching a maximum number of CTAs that are smaller than the total number of SMs, set the default value to the maximum number of CTAs that will be used simultaneously by users (Low Resource, Low Synchronization Overhead).
  
  SM: This option is similar to the previous option but uses the underlying hardware SM to determine which DCI to use.
  
  This option is useful for users who are scheduling work in units that match the size of an SM. Like the option above, if IBGDA_NUM_DCI is not set by users, NVSHMEM will by default create one DCI per SM. This option has additional overhead relative to the CTA option. The fact that multiple CTAs can map to a SM requires heavier synchronization when submitting operations to the QP. For CTA, you can use block-scoped atomics to avoid contention on the QP, but in the SM case, a full device atomic needs to be performed that has greater overhead.

Note

The CTA is the preferred option (Low Resource, High Synchronization Overhead).

WARP: This option is useful for applications that drive IO to multiple peers from a smaller subset of CTAs.

For example, in an all-to-all communication pattern with 32 warps per CTA and 32 peers, users can set the number of DCIs to 32 and use the WARP mapping to ensure that each warp has a dedicated DCI. By mapping this way, users can avoid the penalty of serializing submissions to the QP and renegotiating the connection for each warp submitting operations. This options is more memory intensive than the previous optons. If IBGDA_NUM_DCI is not set by users, NVSHMEM will by default create num_warps_per_sm * num_sms DCIs (High Resource, Low Synchronization Overhead).

DCT: If there are sufficient DCIs, this option locks each DCI to one specific DCT.

This option forces one-to-one mapping for DCIs to DCTs, which eliminates the need to reestablish a connection between iterations, even if the DCI is used by different warps CTAs, or SMs over time. Hwoever, some of the memory saving benefits of choosing DC over RC are lost. Users can specify fewer DCIs in NVSHMEM_IBGDA_NUM_DCI to complete a one-to-one mapping with DCTs. In this case, each DCI will be locked to a small subset of DCTs. Users might specify a smaller number of DCIs than DCTs, so this option is still more memory efficient than RC, which requires the creation and connection on both ends for each QP. This option also requires a device-scoped atomic to guarantee correctness (High Resource, High Synchronization Overhead).

The default value is CTA.

NVSHMEM_IBGDA_RC_MAP_BY includes the following options:

For these options, there is no default number of QPs created, so users must specify the number of QPs to create based on mapping type (RCs have lower latency than DCIs).

CTA: This is semantically the same as the DCI option.

SM: This is semantically the same as the DCI option.

WARP: This is semantically the same as the DCI option.

The default value is CTA.