Is this page helpful?

PCIe Multi-GPU Systems#

On hosts with more than one visible GPU, the NIM assigns each request to a GPU and keeps that request’s CUDA shared memory and Triton model instance on the same device. This pinned mode is the default. It is correct without changing the host IOMMU configuration, and it is the higher-throughput path on typical PCIe multi-GPU systems.

To let Triton’s scheduler place each batch on any visible GPU instead, set NV_AI4M_SVD_PIN_MODEL_TO_SHM_GPU=false. That unpinned mode requires IOMMU passthrough (iommu=pt). If passthrough is not active, the NIM logs a warning and stays in the pinned mode rather than starting a configuration that can corrupt cross-device shared-memory reads.

Accepted false values are false, 0, no, and off (case-insensitive). An unset, empty, or unrecognized value keeps the pinned default.

docker run -it --rm --name=synthetic-video-detector-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NV_AI4M_SVD_PIN_MODEL_TO_SHM_GPU=false \
  -p 8000:8000 \
  -p 8001:8001 \
  nvcr.io/nim/nvidia/synthetic-video-detector:latest

IOMMU Passthrough#

In unpinned mode, Triton schedules model instances across GPUs and the NIM uses peer-to-peer memory copy to transfer data between GPUs whenever this feature is available; that is, when cudaDeviceCanAccessPeer() returns true.

However, on bare-metal Linux systems with PCIe topology, IOMMU-enabled peer-to-peer memory copy is not supported. For more information, refer to Host IOMMU Hardware, PCI Access Control Services, and VMs. When IOMMU is enabled, set it to passthrough (by setting the Linux kernel parameter iommu=pt) before using unpinned Triton scheduling. An IOMMU that is disabled outright is not equivalent to passthrough; cross-device shared-memory reads can still return corrupted data.

The following are the steps to set IOMMU to passthrough in GRUB:

  1. Determine whether IOMMU is enabled by running the following command:

    dmesg | grep -e DMAR -e IOMMU

    • No matching output: IOMMU is not enabled. The default pinned mode still works. To use NV_AI4M_SVD_PIN_MODEL_TO_SHM_GPU=false, enable IOMMU in passthrough as described in the following steps.

    • Matching output: IOMMU is enabled. Continue to the next step.

  2. Determine whether IOMMU is set to passthrough by running the following command:

    dmesg | grep -i -e iommu=pt -e iommu.*passthrough

    • No matching output: Continue with the GRUB configuration steps.

    • Matching output: IOMMU passthrough is already enabled. No further action is required.

  3. Open /etc/default/grub file for edit and add iommu=pt to GRUB_CMDLINE_LINUX option. For example:

    .....

GRUB_CMDLINE_LINUX="crashkernel=auto quiet iommu=pt"

.....

  4. Based on the system’s OS, use grub-mkconfig or grub2-mkconfig to generate the configuration file:

    • On systems with BIOS:

      grub-mkconfig -o /boot/grub2/grub.cfg #on ubuntu, debian
grub2-mkconfig -o /boot/grub2/grub.cfg #on centos, rockylinux

    • On systems with UEFI:

      grub-mkconfig -o /boot/efi/EFI/<os_name>/grub.cfg #on ubuntu, debian

grub2-mkconfig -o /boot/efi/EFI/<os_name>/grub.cfg #on centos, rockylinux

   #replace <os_name> with ubuntu, centos, debian, or rocky

  5. On Ubuntu and Debian, you might need to install grub-mkconfig:

    apt install grub-common

  6. Reboot the system:

    systemctl reboot

  7. Verify that IOMMU is set to passthrough by repeating step 2.

On this page