Using GPU Pass-Through#

GPU pass-through is used to directly assign an entire physical GPU to one VM, bypassing the NVIDIA Virtual GPU Manager. In this mode of operation, the GPU is accessed exclusively by the NVIDIA driver running in the VM to which it is assigned; the GPU is not shared among VMs.

In pass-through mode, all GPUs that support NVIDIA vGPU software support error-correcting code (ECC).

GPU pass-through can be used in a server platform alongside NVIDIA vGPU, with some restrictions:

  • A physical GPU can host NVIDIA vGPUs, or can be used for pass-through, but cannot do both at the same time. Some hypervisors, for example VMware vSphere ESXi, require a host reboot to change a GPU from pass-through mode to vGPU mode.

  • A single VM cannot be configured for both vGPU and GPU pass-through at the same time.

  • The performance of a physical GPU passed through to a VM can be monitored only from within the VM itself. Such a GPU cannot be monitored by tools that operate through the hypervisor, such as XenCenter or nvidia-smi (see Monitoring GPU Performance).

  • The following BIOS settings must be enabled on your server platform:

    • VT-D/IOMMU

    • SR-IOV in Advanced Options

  • All GPUs directly connected to each other through NVLink must be assigned to the same VM.

You can assign multiple physical GPUs to one VM. The maximum number of physical GPUs that you can assign to a VM depends on the maximum number of PCIe pass-through devices per VM that your chosen hypervisor can support. For more information, refer to the documentation for your hypervisor, for example:

Note

If you intend to configure all GPUs in your server platform for pass-through, you do not need to install the NVIDIA Virtual GPU Manager.

Display Resolutions for Physical GPUs#

The display resolutions supported by a physical GPU depend on the NVIDIA vGPU software license that is applied to the GPU.

NVIDIA RTX Virtual Workstation Physical GPU Resolutions#

GPUs that are licensed with an NVIDIA RTX Virtual Workstation license support a maximum combined resolution based on the number of available pixels. You can choose between using a small number of high resolution displays or a larger number of lower resolution displays with these GPUs.

The following table lists the maximum number of displays per GPU at each supported display resolution for configurations in which all displays have the same resolution.

Available Pixels

Display Resolution

Displays per GPU

66355200

7680×4320

2

5120×2880 or lower

4

The following table provides examples of configurations with a mixture of display resolutions.

Available Pixels

Available Pixel Basis

Maximum Displays

Sample Mixed Display Configurations

66355200

2 7680×4320 displays

4

1 7680×4320 display plus 2 5120×2880 displays

1 7680×4320 display plus 3 4096×2160 displays

Note

You cannot use more than four displays even if the combined resolution of the displays is less than the number of available pixels from the GPU. For example, you cannot use five 4096×2160 displays even though the combined resolution of the displays (44236800) is less than the number of available pixels from the GPU (66355200).

NVIDIA Virtual Applications or NVIDIA vGPU for Compute Physical GPU Resolutions#

GPUs that are licensed with an NVIDIA Virtual Applications or an NVIDIA vGPU for Compute license support a single display with a fixed maximum resolution. The maximum resolution depends on the following factors:

  • The NVIDIA vGPU software license that is applied to the GPU

  • The operating system that is running on the system to which the GPU is assigned

License

Operating System

Maximum Display Resolution

Displays per GPU

NVIDIA Virtual Applications

Linux

2560×1600

1

Windows

1280×1024

1

Using GPU Pass-Through on XenServer#

You can configure a GPU for pass-through on XenServer by using XenCenter or by using the xe command.

The following additional restrictions apply when GPU pass-through is used in a server platform alongside NVIDIA vGPU:

  • The performance of a physical GPU passed through to a VM cannot be monitored through XenCenter.

  • nvidia-smi in dom0 no longer has access to the GPU.

  • Pass-through GPUs do not provide console output through XenCenter’s VM Console tab. Use a remote graphics connection directly into the VM to access the VM’s OS.

Configuring a VM for GPU Pass Through by Using XenCenter#

Select the Pass-through whole GPU option as the GPU type in the VM’s Properties:

Screen capture showing how to use XenCenter to configure a pass-through GPU

Figure 15 Using XenCenter to configure a pass-through GPU#

After configuring a XenServer VM for GPU pass through, install the NVIDIA graphics driver in the guest OS on the VM as explained in Installing the NVIDIA vGPU Software Graphics Driver.

Configuring a VM for GPU Pass Through by Using xe#

Create a vgpu object with the passthrough vGPU type:

[root@xenserver ~]# xe vgpu-type-list model-name="passthrough"
uuid ( RO)                : fa50b0f0-9705-6c59-689e-ea62a3d35237
         vendor-name ( RO):
          model-name ( RO): passthrough
    framebuffer-size ( RO): 0

[root@xenserver ~]# xe vgpu-create vm-uuid=753e77a9-e10d-7679-f674-65c078abb2eb vgpu-type-uuid=fa50b0f0-9705-6c59-689e-ea62a3d35237 gpu-group-uuid=585877ef-5a6c-66af-fc56-7bd525bdc2f6
6aa530ec-8f27-86bd-b8e4-fe4fde8f08f9
[root@xenserver ~]#

Caution

Do not assign pass-through GPUs using the legacy other-config:pci parameter setting. This mechanism is not supported alongside the XenCenter UI and xe vgpu mechanisms, and attempts to use it may lead to undefined results.

After configuring a XenServer VM for GPU pass through, install the NVIDIA graphics driver in the guest OS on the VM as explained in Installing the NVIDIA vGPU Software Graphics Driver.

Using GPU Pass-Through on a Linux with KVM Hypervisor#

NVIDIA vGPU software supports the following Linux with KVM hypervisors: Red Hat Enterprise Linux with KVM and Ubuntu.

You can configure a GPU for pass-through on a Linux with KVM hypervisor by using any of the following tools:

  • The Virtual Machine Manager (virt-manager) graphical tool

  • The virsh command

  • The QEMU command line

Before configuring a GPU for pass-through on Red Hat Enterprise Linux KVM or Ubuntu, ensure that the following prerequisites are met:

  • Red Hat Enterprise Linux KVM or Ubuntu is installed.

  • A virtual disk has been created.

    Note

    Do not create any virtual disks in /root.

  • A virtual machine has been created.

If you’re configuring a pass-through GPU that requires a large BAR address space on a UEFI VM, refer to NVIDIA vGPU software graphics driver fails to load on KVM-based hypervisors for a workaround to ensure that BAR resources are mapped into the VM.

Note

This workaround involves setting an experimental QEMU parameter.

Configuring a VM for GPU Pass-Through by Using Virtual Machine Manager (virt-manager)#

For more information about using Virtual Machine Manager, see the following topics in the documentation for Red Hat Enterprise Linux 7:

  1. Start virt-manager.

  2. In the virt-manager main window, select the VM that you want to configure for pass-through.

  3. From the Edit menu, choose Virtual Machine Details.

  4. In the virtual machine hardware information window that opens, click Add Hardware.

  5. In the Add New Virtual Hardware dialog box that opens, in the hardware list on the left, select PCI Host Device.

  6. From the Host Device list that appears, select the GPU that you want to assign to the VM and click Finish.

If you want to remove a GPU from the VM to which it is assigned, in the virtual machine hardware information window, select the GPU and click Remove.

After configuring the VM for GPU pass through, install the NVIDIA graphics driver in the guest OS on the VM as explained in Installing the NVIDIA vGPU Software Graphics Driver.

Configuring a VM for GPU Pass-Through by Using virsh#

For more information about using virsh, see the following topics in the documentation for Red Hat Enterprise Linux 7:

  1. Verify that the vfio-pci module is loaded.

    # lsmod | grep vfio-pci
    
  2. Obtain the PCI device bus/device/function (BDF) of the GPU that you want to assign in pass-through mode to a VM.

    # lspci | grep NVIDIA
    

    The NVIDIA GPUs listed in this example have the PCI device BDFs 85:00.0 and 86:00.0.

    # lspci | grep NVIDIA
     85:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
     86:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
    
  3. Obtain the full identifier of the GPU from its PCI device BDF.

    # virsh nodedev-list --cap pci| grep <transformed-bdf>
    
    <transformed-bdf>

    The PCI device BDF of the GPU with the colon and the period replaced with underscores, for example, 85_00_0.

    This example obtains the full identifier of the GPU with the PCI device BDF 85:00.0.

    # virsh nodedev-list --cap pci| grep 85_00_0
    pci_0000_85_00_0
    
  4. Obtain the domain, bus, slot, and function of the GPU.

    virsh nodedev-dumpxml <full-identifier>| egrep 'domain|bus|slot|function'
    
    <full-identifier>

    The full identifier of the GPU that you obtained in the previous step, for example, pci_0000_85_00_0.

    This example obtains the domain, bus, slot, and function of the GPU with the PCI device BDF 85:00.0.

    # virsh nodedev-dumpxml pci_0000_85_00_0| egrep 'domain|bus|slot|function'
        <domain>0x0000</domain>
        <bus>0x85</bus>
        <slot>0x00</slot>
        <function>0x0</function>
          <address domain='0x0000' bus='0x85' slot='0x00' function='0x0'/>
    
  5. In virsh, open for editing the XML file of the VM that you want to assign the GPU to.

    # virsh edit <vm-name>
    
    <vm-name>

    The name of the VM to that you want to assign the GPU to.

  6. Add a device entry in the form of an address element inside the source element to assign the GPU to the guest VM.

    You can optionally add a second address element after the source element to set a fixed PCI device BDF for the GPU in the guest operating system.

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='<domain>' bus='<bus>' slot='<slot>' function='<function>'/>
      </source>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    
    <domain>, <bus>, <slot>, <function>

    The domain, bus, slot, and function of the GPU, which you obtained in the previous step.

    This example adds a device entry for the GPU with the PCI device BDF 85:00.0 and fixes the BDF for the GPU in the guest operating system.

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x85' slot='0x00' function='0x0'/>
      </source>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    
  7. Start the VM that you assigned the GPU to.

    # virsh start <vm-name>
    
    <vm-name>

    The name of the VM that you assigned the GPU to.

After configuring the VM for GPU pass through, install the NVIDIA graphics driver in the guest OS on the VM as explained in Installing the NVIDIA vGPU Software Graphics Driver.

Configuring a VM for GPU Pass-Through by Using the QEMU Command Line#

  1. Obtain the PCI device bus/device/function (BDF) of the GPU that you want to assign in pass-through mode to a VM.

    # lspci | grep NVIDIA
    

    The NVIDIA GPUs listed in this example have the PCI device BDFs 85:00.0 and 86:00.0.

    # lspci | grep NVIDIA
     85:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
     86:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
    
  2. Add the following option to the QEMU command line:

    -device vfio-pci,host=<bdf>
    
    <bdf>

    The PCI device BDF of the GPU that you want to assign in pass-through mode to a VM, for example, 85:00.0.

    This example assigns the GPU with the PCI device BDF 85:00.0 in pass-through mode to a VM.

    -device vfio-pci,host=85:00.0
    

After configuring the VM for GPU pass through, install the NVIDIA graphics driver in the guest OS on the VM as explained in Installing the NVIDIA vGPU Software Graphics Driver.

Preparing a GPU Configured for vGPU for Use in Pass-Through Mode#

The mode in which a physical GPU is being used determines the Linux kernel module to which the GPU is bound. If you want to switch the mode in which a GPU is being used, you must unbind the GPU from its current kernel module and bind it to the kernel module for the new mode. After binding the GPU to the correct kernel module, you can then configure it for pass-through.

When the Virtual GPU Manager is installed on a Red Hat Enterprise Linux KVM or Ubuntu host, the physical GPUs on the host are bound to the nvidia kernel module. A physical GPU that is bound to the nvidia kernel module can be used only for vGPU. To enable the GPU to be passed through to a VM, the GPU must be unbound from nvidia kernel module and bound to the vfio-pci kernel module.

Before you begin, ensure that you have the domain, bus, slot, and function of the GPU that you are preparing for use in pass-through mode. For instructions, see Getting the BDF and Domain of a GPU on a Linux with KVM Hypervisor.

  1. If you are using a GPU that supports SR-IOV, such as a GPU based on the NVIDIA Ampere architecture, disable the virtual function for the GPU in the sysfs file system.

    If your GPU does not support SR-IOV, omit this step.

    Note

    Before performing this step, ensure that the GPU is not being used by any other processes, such as CUDA applications, monitoring applications, or the nvidia-smi command.

    Use the custom script sriov-manage provided by NVIDIA vGPU software for this purpose.

    # /usr/lib/nvidia/sriov-manage -d <domain>:<bus>:<slot>.<function>
    
    <domain>, <bus>, <slot>, <function>

    The domain, bus, slot, and function of the GPU, without the 0x prefix.

    This example disables the virtual function for the GPU with the domain 00, bus 06, slot 0000, and function 0.

    # /usr/lib/nvidia/sriov-manage -d 00:06:0000.0
    
  2. Determine the kernel module to which the GPU is bound by running the lspci command with the -k option on the NVIDIA GPUs on your host.

    # lspci -d 10de: -k
    

    The Kernel driver in use: field indicates the kernel module to which the GPU is bound.

    The following example shows that the NVIDIA Tesla M60 GPU with BDF 06:00.0 is bound to the nvidia kernel module and is being used for vGPU.

       06:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
             Subsystem: NVIDIA Corporation Device 115e
             Kernel driver in use: nvidia
    
  3. To ensure that no clients are using the GPU, acquire the unbind lock of the GPU.

    1. Ensure that no VM is running to which a vGPU on the physical GPU is assigned and that no process running on the host is using that GPU.

      Processes on the host that use the GPU include the nvidia-smi command and all processes based on the NVIDIA Management Library (NVML).

    2. Change to the directory in the proc file system that represents the GPU.

      # cd /proc/driver/nvidia/gpus/<domain>\:<bus>\:<slot>.<function>
      
    <domain>, <bus>, <slot>, <function>

    The domain, bus, slot, and function of the GPU, without a 0x prefix.

    This example changes to the directory in the proc file system that represents the GPU with the domain 0000 and PCI device BDF 06:00.0.

    # cd /proc/driver/nvidia/gpus/0000\:06\:00.0
    
    1. Write the value 1 to the unbindLock file in this directory.

      # echo 1 > unbindLock
      
    2. Confirm that the unbindLock file now contains the value 1.

      # cat unbindLock
      1
      

      If the unbindLock file contains the value 0, the unbind lock could not be acquired because a process or client is using the GPU.

  4. Unbind the GPU from nvidia kernel module.

    1. Change to the sysfs directory that represents the nvidia kernel module.

      # cd /sys/bus/pci/drivers/nvidia
      
    2. Write the domain, bus, slot, and function of the GPU to the unbind file in this directory.

      # echo <domain>:<bus>:<slot>.<function> > unbind
      
    <domain>, <bus>, <slot>, <function>

    The domain, bus, slot, and function of the GPU, without a 0x prefix.

    This example writes the domain, bus, slot, and function of the GPU with the domain 0000 and PCI device BDF 06:00.0.

    # echo 0000:06:00.0 > unbind
    
  5. Bind the GPU to the vfio-pci kernel module.

    1. Change to the sysfs directory that contains the PCI device information for the physical GPU.

      # cd /sys/bus/pci/devices/<domain>\:<bus>\:<slot>.<function>
      
    <domain>, <bus>, <slot>, <function>

    The domain, bus, slot, and function of the GPU, without a 0x prefix.

    This example changes to the sysfs directory that contains the PCI device information for the GPU with the domain 0000 and PCI device BDF 06:00.0.

    # cd /sys/bus/pci/devices/0000\:06\:00.0
    
    1. Write the kernel module name vfio-pci to the driver_override file in this directory.

      # echo vfio-pci > driver_override
      
    2. Change to the sysfs directory that represents the nvidia kernel module.

      # cd /sys/bus/pci/drivers/vfio-pci
      
    3. Write the domain, bus, slot, and function of the GPU to the bind file in this directory.

      # echo <domain>:<bus>:<slot>.<function> > bind
      
    <domain>, <bus>, <slot>, <function>

    The domain, bus, slot, and function of the GPU, without a 0x prefix.

    This example writes the domain, bus, slot, and function of the GPU with the domain 0000 and PCI device BDF 06:00.0.

    # echo 0000:06:00.0 > bind
    
    1. Change back to the sysfs directory that contains the PCI device information for the physical GPU.

      # cd /sys/bus/pci/devices/<domain>\:<bus>\:<slot>.<function>
      
    2. Clear the content of the driver_override file in this directory.

      # echo > driver_override
      

You can now configure the GPU for use in pass-through mode as explained in Using GPU Pass-Through on a Linux with KVM Hypervisor.

Using GPU Pass-Through on Microsoft Windows Server#

On supported versons of Microsoft Windows Server with Hyper-V role, you can use Discrete Device Assignment (DDA) to enable a VM to access a GPU directly.

Assigning a GPU to a VM on Microsoft Windows Server with Hyper-V#

Perform this task in Windows PowerShell. If you do not know the location path of the GPU that you want to assign to a VM, use Device Manager to obtain it.

If you are using an actively cooled NVIDIA Quadro graphics card such as the RTX 8000 or RTX 6000, you must also pass through the audio device on the graphics card.

Ensure that the following prerequisites are met:

  1. Obtain the location path of the GPU that you want to assign to a VM.

    1. In the device manager, context-click the GPU and from the menu that pops up, choose Properties.

    2. In the Properties window that opens, click the Details tab and in the Properties drop-down list, select Location paths.

    An example location path is as follows:

    PCIROOT(80)#PCI(0200)#PCI(0000)#PCI(1000)#PCI(0000)
    
  2. If you are using an actively cooled NVIDIA Quadro graphics card, obtain the location path of the audio device on the graphics card and disable the device.

    1. In the device manager, from the View menu, choose Devices by connection.

    2. Navigate to ACPI x64-based PC > Microsoft ACPI-Compliant System > PCI Express Root Complex > PCI-to-PCI Bridge.

    3. Context-click High Definition Audio Controller and from the menu that pops up, choose Properties.

    4. In the Properties window that opens, click the Details tab and in the Properties drop-down list, select Location paths.

    5. Context-click High Definition Audio Controller again and from the menu that pops up, choose Disable device.

  3. Dismount the GPU and, if present, the audio device from host to make them unavailable to the host so that they can be used solely by the VM.

    For each device that you are dismounting, type the following command:

    Dismount-VMHostAssignableDevice -LocationPath <gpu-device-location> -force
    
    <gpu-device-location>

    The location path of the GPU or the audio device that you obtained previously.

    This example dismounts the GPU at the location path PCIROOT(80)#PCI(0200)#PCI(0000)#PCI(1000)#PCI(0000).

    Dismount-VMHostAssignableDevice -LocationPath "PCIROOT(80)#PCI(0200)#PCI(0000)#PCI(1000)#PCI(0000)" -force
    
  4. Assign the GPU and, if present, the audio device that you dismounted in the previous step to the VM.

    For each device that you are assigning, type the following command:

    Add-VMAssignableDevice -LocationPath <gpu-device-location> -VMName <vm-name>
    
    <gpu-device-location>

    The location path of the GPU or the audio device that you dismounted in the previous step.

    <vm-name>

    The name of the VM to which you are attaching the GPU or the audio device.

    Note

    You can assign a pass-through GPU and, if present, its audio device to only one virtual machine at a time.

    This example assigns the GPU at the location path PCIROOT(80)#PCI(0200)#PCI(0000)#PCI(1000)#PCI(0000) to the VM VM1.

    Add-VMAssignableDevice -LocationPath "PCIROOT(80)#PCI(0200)#PCI(0000)#PCI(1000)#PCI(0000)" -VMName VM1
    
  5. Power on the VM.

    The guest OS should now be able to use the GPU and, if present, the audio device.

After assigning a GPU to a VM, install the NVIDIA graphics driver in the guest OS on the VM as explained in Installing the NVIDIA vGPU Software Graphics Driver.

Returning a GPU to the Host OS from a VM on Windows Server with Hyper-V#

Perform this task in the Windows PowerShell.

If you are using an actively cooled NVIDIA Quadro graphics card such as the RTX 8000 or RTX 6000, you must also return the audio device on the graphics card.

  1. List the GPUs and, if present, the audio devices that are currently assigned to the virtual machine (VM).

    Get-VMAssignableDevice -VMName <vm-name>
    
    <vm-name>

    The name of the VM whose assigned GPUs and audio devices you want to list.

  2. Shut down the VM to which the GPU and any audio devices are assigned.

  3. Remove the GPU and, if present, the audio device from the VM to which they are assigned.

    For each device that you are removing, type the following command:

    Remove-VMAssignableDevice –LocationPath <gpu-device-location> -VMName <vm-name>
    
    <gpu-device-location>

    The location path of the GPU or the audio device that you are removing, which you obtained previously.

    <vm-name>

    The name of the VM from which you are removing the GPU or the audio device.

    This example removes the GPU at the location path PCIROOT(80)#PCI(0200)#PCI(0000)#PCI(1000)#PCI(0000) from the VM VM1.

    Remove-VMAssignableDevice –LocationPath "PCIROOT(80)#PCI(0200)#PCI(0000)#PCI(1000)#PCI(0000)" -VMName VM1
    

    After the GPU and, if present, its audio device are removed from the VM, they are unavailable to the host operating system (OS) until you remount them on the host OS.

  4. Remount the GPU and, if present, its audio device on the host OS.

    For each device that you are remounting, type the following command:

    Mount-VMHostAssignableDevice –LocationPath <gpu-device-location>
    
    <gpu-device-location>

    The location path of the GPU or the audio device that you are remounting, which you specified in the previous step to remove the GPU or the audio device from the VM.

    This example remounts the GPU at the location path PCIROOT(80)#PCI(0200)#PCI(0000)#PCI(1000)#PCI(0000) on the host OS.

    Mount-VMHostAssignableDevice -LocationPath "PCIROOT(80)#PCI(0200)#PCI(0000)#PCI(1000)#PCI(0000)"
    

    The host OS should now be able to use the GPU and, if present, its audio device.

Using GPU Pass-Through on VMware vSphere#

On VMware vSphere, you can use Virtual Dedicated Graphics Acceleration (vDGA) to enable a VM to access a GPU directly. vDGA is a feature of VMware vSphere that dedicates a single physical GPU on an ESXi host to a single virtual machine.

Before configuring a vSphere VM with vDGA, ensure that these prerequisites are met

  1. Open the vCenter Web UI.

  2. In the vCenter Web UI, right-click the ESXi host and choose Configure.

  3. From the Hardware menu, choose PCI Devices.

  4. On the PCI Devices page that opens, click ALL PCI DEVICES and in the table of devices, select the GPU.

    Note

    When selecting the GPU to pass through, you must select only the physical device. To list only NVIDIA physical devices, set the filter on the Vendor Name field to NVIDIA and filter out any virtual function devices of the GPU by setting the filter on the ID field to 00.0.Screen capture showing the PCI Devices page.Screen capture showing the PCI Devices page.

  5. Click TOGGLE PASSTHROUGH.

  6. Reboot the ESXi host.

  7. After the ESXi host has booted, right-click the VM and choose Edit Settings.

  8. From the New Device menu, choose PCI Device and click Add.

  9. On the page that opens, from the New Device drop-down list, select the GPU.

  10. Click Reserve all memory and click OK.

  11. Start the VM.

For more information about vDGA, see the following topics in the VMware Horizon documentation:

After configuring a vSphere VM with vDGA, install the NVIDIA graphics driver in the guest OS on the VM as explained in Installing the NVIDIA vGPU Software Graphics Driver.