Is this page helpful?

GPUDirect RDMA and GPUDirect Storage#

NVIDIA GPUDirect Remote Direct Memory Access (RDMA) is a technology that enables direct data exchange between NVIDIA GPUs and third-party PCIe peer devices. GPUDirect RDMA enables network devices to access the vGPU frame buffer directly, bypassing CPU host memory. The third-party devices can be network interfaces such as NVIDIA ConnectX SmartNICs or BlueField DPUs, or video acquisition adapters.

GPUDirect Storage (GDS) enables a direct data path between local or remote storage, such as NFS servers or NVMe/NVMe over Fabric (NVMe-oF), and GPU memory. GDS performs direct memory access (DMA) transfers between GPU memory and storage. DMA avoids a bounce buffer through the CPU. This direct path increases system bandwidth and decreases the latency and utilization load on the CPU.

GPUDirect technology is supported only on a subset of vGPUs and guest OS releases.

GPUDirect RDMA and GPUDirect Storage Known Issues and Limitations#

Starting with GPUDirect Storage technology release 1.7.2, the following limitations apply:

  • GPUDirect Storage technology is not supported on GPUs based on the NVIDIA Ampere GPU architecture.

  • On GPUs based on the NVIDIA Ada Lovelace, Hopper, and Blackwell GPU architectures, GPUDirect Storage technology is supported only with the guest driver for Linux based on NVIDIA Linux open GPU kernel modules.

GPUDirect Storage technology releases before 1.7.2 are supported only with guest drivers with Linux kernel versions earlier than 6.6.

GPUDirect Storage technology is supported only on the following guest OS releases:

  • Red Hat Enterprise Linux 8.8+

  • Ubuntu 24.04 LTS

  • Ubuntu 22.04 LTS

Hypervisor Platform Support for GPUDirect RDMA and GPUDirect Storage#

Table 19 Hypervisor Platform Support for GPUDirect RDMA and GPUDirect Storage#

Hypervisor Platform

Version

Red Hat Enterprise Linux with KVM

8.8+

Ubuntu

  • 24.04

  • 22.04

VMware vSphere

  • 9

  • 8

vGPU Support for GPUDirect RDMA and GPUDirect Storage#

GPUDirect RDMA and GPUDirect Storage technology are supported on all time-sliced and MIG-backed NVIDIA vGPU for Compute on physical GPUs that support single root I/O virtualization (SR-IOV).

For a list of supported GPUs, refer to the Supported NVIDIA GPUs and Networking section in the NVIDIA AI Enterprise Infra Support Matrix.

Guest OS Releases Support for GPUDirect RDMA and GPUDirect Storage#

Linux only. GPUDirect technology is not supported on Windows.

Network Interface Cards Support for GPUDirect RDMA and GPUDirect Storage#

GPUDirect technology is supported on the following network interface cards:

  • NVIDIA ConnectX-8 SmartNIC

  • NVIDIA ConnectX-7 SmartNIC

  • NVIDIA ConnectX-6 SmartNIC

  • NVIDIA ConnectX-5 Ethernet adapter card