Peer-To-Peer (P2P) CUDA Transfers#
Peer-to-Peer (P2P) CUDA transfers enable device memory between vGPUs on different GPUs that are assigned to the same VM to be accessed from within CUDA kernels. NVLink is a high-bandwidth interconnect that enables fast communication between such vGPUs.
P2P CUDA transfers over NVLink are supported only on a subset of vGPUs, hypervisor releases, and guest OS releases.
For multi-vGPU VM assignment without the P2P focus, see Multi-vGPU. For the shared hypervisor support matrix that covers both Multi-vGPU and P2P, see Multi-vGPU and P2P.
Peer-to-Peer CUDA Transfers Known Issues and Limitations#
Only time-sliced vGPUs are supported. MIG-backed vGPUs are not supported.
P2P transfers over PCIe are not supported.
Warning
On NVIDIA A100 (Ampere) GPUs, P2P over NVLink and NVSwitch is disabled when Unified Virtual Memory (UVM) is enabled on a multi-vGPU VM. Inside the guest, nvidia-smi topo -m reports PIX (PCIe) instead of NV12 (NVLink). To restore NVLink P2P between vGPUs, disable UVM on the vGPU devices (for example, pciPassthru<n>.cfg.enable_uvm = 0 on VMware vSphere). This restriction is specific to the NVIDIA Ampere A100 architecture and does not apply to NVIDIA Hopper (H100/H200) or newer architectures.
vGPU Support for P2P#
Only NVIDIA vGPU for Compute time-sliced vGPUs allocated all of the physical GPU framebuffer on physical GPUs supporting NVLink are supported.
Board |
vGPU |
|---|---|
NVIDIA HGX B300 279 GB |
NVIDIA B300X-279C |
NVIDIA HGX B200 180 GB |
NVIDIA B200X-180C |
Board |
vGPU |
|---|---|
NVIDIA H800 PCIe 94 GB (H800 NVL) |
H800L-94C |
NVIDIA H800 PCIe 80 GB |
H800-80C |
NVIDIA H200 PCIe 141 GB (H200 NVL) |
H200-141C |
NVIDIA H200 SXM5 141 GB |
H200X-141C |
NVIDIA H100 PCIe 94 GB (H100 NVL) |
H100L-94C |
NVIDIA H100 SXM5 94 GB |
H100XL-94C |
NVIDIA H100 PCIe 80 GB |
H100-80C |
NVIDIA H100 SXM5 80 GB |
H100XM-80C |
NVIDIA H100 SXM5 64 GB |
H100XS-64C |
NVIDIA H20 SXM5 141 GB |
H20X-141C |
NVIDIA H20 SXM5 96 GB |
H20-96C |
Board |
vGPU |
|---|---|
|
A800D-80C |
NVIDIA A800 PCIe 40 GB active-cooled |
A800-40C |
NVIDIA A800 HGX 80 GB |
A800DX-80C [1] |
|
A100D-80C |
NVIDIA A100 HGX 80 GB |
A100DX-80C [1] |
NVIDIA A100 PCIe 40 GB |
A100-40C |
NVIDIA A100 HGX 40 GB |
A100X-40C [1] |
NVIDIA A40 |
A40-48C |
|
A30-24C |
NVIDIA A16 |
A16-16C |
NVIDIA A10 |
A10-24C |
NVIDIA RTX A6000 |
A6000-48C |
NVIDIA RTX A5500 |
A5500-24C |
NVIDIA RTX A5000 |
A5000-24C |