Peer-To-Peer (P2P) CUDA Transfers#
Peer-to-Peer (P2P) CUDA® transfers enable device memory between vGPUs on different GPUs that are assigned to the same VM to be accessed from within CUDA kernels. NVLink™ is a high-bandwidth interconnect. It enables fast communication between such vGPUs.
P2P CUDA transfers over NVLink are supported only on a subset of vGPUs, hypervisor releases, and guest OS releases.
For multi-vGPU™ VM assignment without the P2P focus, refer to Multi-vGPU. For the shared hypervisor matrix and page map, refer to Multi-vGPU and P2P.
Peer-to-Peer CUDA Transfers Known Issues and Limitations#
Only time-sliced vGPUs are supported. MIG-backed vGPUs are not supported.
P2P transfers over PCIe are not supported.
Warning
On NVIDIA® A100 (Ampere) GPUs, P2P over NVLink and NVSwitch™ is disabled when Unified Virtual Memory (UVM) is enabled on a multi-vGPU VM. Inside the guest, nvidia-smi topo -m reports PIX (PCIe) instead of NV12 (NVLink). To restore NVLink P2P between vGPUs, disable UVM on the vGPU devices (for example, pciPassthru<n>.cfg.enable_uvm = 0 on VMware vSphere). This restriction is specific to the NVIDIA Ampere A100 architecture and does not apply to NVIDIA Hopper™ (H100/H200) or newer architectures.
vGPU Support for P2P#
NVLink P2P supports only NVIDIA vGPU for Compute time-sliced vGPUs. Each vGPU must use all of the framebuffer of a physical GPU that supports NVLink.
Board |
vGPU |
|---|---|
NVIDIA HGX™ B300 288 GB |
NVIDIA B300X-269C |
NVIDIA HGX B200 180 GB |
NVIDIA B200X-180C |
Board |
vGPU |
|---|---|
NVIDIA H800 PCIe 94 GB (H800 NVL) |
H800L-94C |
NVIDIA H800 PCIe 80 GB |
H800-80C |
NVIDIA H200 PCIe 141 GB (H200 NVL) |
H200-141C |
NVIDIA H200 SXM5 141 GB |
H200X-141C |
NVIDIA H100 PCIe 94 GB (H100 NVL) |
H100L-94C |
NVIDIA H100 SXM5 94 GB |
H100XL-94C |
NVIDIA H100 PCIe 80 GB |
H100-80C |
NVIDIA H100 SXM5 80 GB |
H100XM-80C |
NVIDIA H100 SXM5 64 GB |
H100XS-64C |
NVIDIA H20 SXM5 141 GB |
H20X-141C |
NVIDIA H20 SXM5 96 GB |
H20-96C |
Board |
vGPU |
|---|---|
|
A800D-80C |
NVIDIA A800 PCIe 40 GB active-cooled |
A800-40C |
NVIDIA A800 HGX 80 GB |
A800DX-80C [1] |
|
A100D-80C |
NVIDIA A100 HGX 80 GB |
A100DX-80C [1] |
NVIDIA A100 PCIe 40 GB |
A100-40C |
NVIDIA A100 HGX 40 GB |
A100X-40C [1] |
NVIDIA A40 |
A40-48C |
|
A30-24C |
NVIDIA RTX™ A6000 |
A6000-48C |
NVIDIA RTX A5500 |
A5500-24C |
NVIDIA RTX A5000 |
A5000-24C |