Is this page helpful?

PCIe Multi-GPU Systems#

On multi-GPU systems, the NIM uses peer-to-peer memory copy to transfer data between GPUs whenever this feature is available; that is, when cudaDeviceCanAccessPeer() returns true.

However, on bare-metal Linux systems with PCIe topology, IOMMU-enabled peer-to-peer memory copy is not supported. For more information, refer to Host IOMMU Hardware, PCI Access Control Services, and VMs. When IOMMU is enabled, we recommend setting it to passthrough (by setting the Linux kernel parameter iommu=pt) to ensure optimal performance.

The following are the steps to set IOMMU to passthrough in GRUB:

Check whether IOMMU is enabled by running the following command:
```
dmesg | grep -e DMAR -e IOMMU
```
- No matching output: IOMMU is not enabled. No further action is required.
- Matching output: IOMMU is enabled. Continue to the next step.
Check whether IOMMU is set to passthrough by running the following command:
```
dmesg | grep -i -e iommu=pt -e iommu.*passthrough
```
- No matching output: Continue with the GRUB configuration steps.
- Matching output: IOMMU passthrough is already enabled. No further action is required.
Open /etc/default/grub file for edit and add iommu=pt to GRUB_CMDLINE_LINUX option. For example:
```
.....

GRUB_CMDLINE_LINUX="crashkernel=auto quiet iommu=pt"

.....
```

Based on the system’s OS, use grub-mkconfig or grub2-mkconfig to generate the configuration file:

On systems with BIOS:

grub-mkconfig -o /boot/grub2/grub.cfg #on ubuntu, debian
grub2-mkconfig -o /boot/grub2/grub.cfg #on centos, rockylinux

On systems with UEFI:

grub-mkconfig -o /boot/efi/EFI/<os_name>/grub.cfg #on ubuntu, debian

grub2-mkconfig -o /boot/efi/EFI/<os_name>/grub.cfg #on centos, rockylinux

   #replace <os_name> with ubuntu, centos, debian, or rocky

On Ubuntu and Debian, you might need to install grub-mkconfig:
```
apt install grub-common
```
Reboot the system:
```
systemctl reboot
```
Verify that IOMMU is set to passthrough by repeating step 2.