PCIe Multi-GPU Systems
On multi-GPU systems, the NIM uses peer-to-peer memory copy to transfer data between GPUs whenever this feature is available; that is, when cudaDeviceCanAccessPeer() returns true.
However, on bare-metal Linux systems with PCIe topology, IOMMU-enabled peer-to-peer memory copy is not supported. For more information, refer to Host IOMMU Hardware, PCI Access Control Services, and VMs. When IOMMU is enabled, we recommend setting it to passthrough (by setting the Linux kernel parameter iommu=pt) to ensure optimal performance.
The following are the steps to set IOMMU to passthrough in GRUB:
-
Check whether IOMMU is enabled by running the following command:
- No matching output: IOMMU is not enabled. No further action is required.
- Matching output: IOMMU is enabled. Continue to the next step.
-
Check whether IOMMU is set to passthrough by running the following command:
- No matching output: Continue with the GRUB configuration steps.
- Matching output: IOMMU passthrough is already enabled. No further action is required.
-
Open
/etc/default/grubfile for edit and addiommu=pttoGRUB_CMDLINE_LINUXoption. For example: -
Based on the system’s OS, use
grub-mkconfigorgrub2-mkconfigto generate the configuration file:-
On systems with BIOS:
-
On systems with UEFI:
-
-
On Ubuntu and Debian, you might need to install
grub-mkconfig: -
Reboot the system:
-
Verify that IOMMU is set to passthrough by repeating step 2.