Is this page helpful?

GPUDirect RDMA and GPUDirect Storage#

NVIDIA GPUDirect Remote Direct Memory Access (RDMA) is a technology that enables direct data exchange between NVIDIA GPUs and third-party PCIe peer devices. GPUDirect RDMA enables network devices to access the vGPU frame buffer directly, bypassing CPU host memory. The third-party devices can be network interfaces such as NVIDIA ConnectX SmartNICs or BlueField DPUs, or video acquisition adapters.

GPUDirect Storage (GDS) enables a direct data path between local or remote storage, such as NFS servers or NVMe/NVMe over Fabric (NVMe-oF), and GPU memory. GDS performs direct memory access (DMA) transfers between GPU memory and storage. DMA avoids a bounce buffer through the CPU. This direct path increases system bandwidth and decreases the latency and utilization load on the CPU.

GPUDirect technology is supported only on a subset of vGPUs and guest OS releases.

GPUDirect RDMA and GPUDirect Storage Known Issues and Limitations#

Starting with GPUDirect Storage technology release 1.7.2, the following limitations apply:

Table 46 GPUDirect Storage Known Issues and Limitations#

Constraint

Applies to

Detail

GDS not supported

NVIDIA Ampere GPU architecture

GPUDirect Storage technology is not supported on Ampere-based GPUs.

Open GPU kernel modules required

Ada Lovelace, Hopper, and Blackwell

GDS is supported only with the guest driver for Linux based on NVIDIA Linux open GPU kernel modules.

Pre-1.7.2 GDS releases

Guest drivers on Linux kernels earlier than 6.6

GPUDirect Storage technology releases before 1.7.2 are supported only with those older guest kernel versions.

Guest OS allowlist

Linux guests only

Red Hat Enterprise Linux 8.8+, Ubuntu 24.04 LTS, and Ubuntu 22.04 LTS.

Hypervisor Platform Support for GPUDirect RDMA and GPUDirect Storage#

Version: 8.8+

Versions:

  • 24.04

  • 22.04

Versions:

  • 9

  • 8

vGPU Support for GPUDirect RDMA and GPUDirect Storage#

GPUDirect RDMA and GPUDirect Storage technology are supported on all time-sliced and MIG-backed NVIDIA vGPU for Compute on physical GPUs that support single root I/O virtualization (SR-IOV).

For a list of supported GPUs, refer to the Supported NVIDIA GPUs and Networking section in the NVIDIA AI Enterprise Infra Support Matrix.

Guest OS Releases Support for GPUDirect RDMA and GPUDirect Storage#

Linux only. GPUDirect technology is not supported on Windows.

Network Interface Cards Support for GPUDirect RDMA and GPUDirect Storage#

GPUDirect technology is supported on the following network interface cards:

  • NVIDIA ConnectX-8 SmartNIC

  • NVIDIA ConnectX-7 SmartNIC

  • NVIDIA ConnectX-6 SmartNIC

  • NVIDIA ConnectX-5 Ethernet adapter card