Is this page helpful?

NVIDIA vGPU for Compute Configuration#

Configuring MIG-Backed vGPU#

Enable Multi-Instance GPU (MIG) mode and configure GPU instances for MIG-backed vGPU deployments on supported GPUs.

Configuring a GPU for MIG-Backed vGPUs#

To support GPU Instances with NVIDIA vGPU, a GPU must be configured with MIG mode enabled, and GPU Instances and Compute Instances must be created on the physical GPU.

Prerequisites

  • The NVIDIA Virtual GPU Manager is installed on the hypervisor host.

  • You have root user privileges on your hypervisor host machine.

  • You have determined which GPU instances correspond to the vGPU types of the MIG-backed vGPUs you’ll create.

  • Other processes, such as CUDA applications, monitoring applications, or the nvidia-smi command, don’t use the GPU.

Steps

  1. Enable MIG mode for a GPU.

    Note

    For VMware vSphere, only enabling MIG mode is required because VMware vSphere creates the GPU Instances and Compute Instances.

  2. Create GPU instances on a MIG-enabled GPU.

  3. Create Compute instances in a GPU instance.

After configuring a GPU for MIG-backed vGPUs, create the vGPUs you need and add them to their VMs.

Enabling MIG Mode for a GPU#

Perform this task in your hypervisor command shell.

  1. Open a command shell as the root user on your hypervisor host machine. You can use a secure shell (SSH) on all supported hypervisors. Individual hypervisors may provide additional means for logging in. For details, refer to the documentation for your hypervisor.

  2. Determine whether MIG mode is enabled. Use the nvidia-smi command for this purpose. By default, MIG mode is disabled. This example shows that MIG mode is disabled on GPU 0.

    Note

    In the output from nvidia-smi, the NVIDIA A100 HGX 40GB GPU is referred to as A100-SXM4-40GB.

    $ nvidia-smi -i 0
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.16   Driver Version: 550.54.16    CUDA Version:  12.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:36:00.0 Off |                    0 |
| N/A   29C    P0    62W / 400W |      0MiB / 40537MiB |      6%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

  3. If MIG mode is disabled, enable it.

    $ nvidia-smi -i [gpu-ids] -mig 1

    gpu-ids - A comma-separated list of GPU indexes, PCI bus IDs, or UUIDs specifying the GPUs you want to enable MIG mode. If gpu-ids are omitted, MIG mode is enabled on all GPUs on the system.

    This example enables MIG mode on GPU 0.

    $ nvidia-smi -i 0 -mig 1
Enabled MIG Mode for GPU 00000000:36:00.0
All done.

    Note

    If another process is using the GPU, this command fails and displays a warning message that MIG mode for the GPU is in the pending enable state. Stop all GPU processes and retry the command.

  4. VMware vSphere ESXi with GPUs based only on the NVIDIA Ampere architecture - Reboot the hypervisor host. If you are using a different hypervisor or GPUs based on the NVIDIA Hopper GPU architecture or later, omit this step.

  5. Query the GPUs on which you enabled MIG mode to confirm that MIG mode is enabled. This example queries GPU 0 for the PCI bus ID and MIG mode in comma-separated values (CSV) format.

    $ nvidia-smi -i 0 --query-gpu=pci.bus_id,mig.mode.current --format=csv
pci.bus_id, mig.mode.current
00000000:36:00.0, Enabled

Creating GPU Instances on a MIG-Enabled GPU#

Note

If you are using VMware vSphere, omit this task. VMware vSphere creates the GPU instances automatically.

Perform this task in your hypervisor command shell.

  1. Open a command shell as the root user on your hypervisor host machine if necessary.

  2. List the GPU instance profiles that are available on your GPU. When you create a profile, you must specify the profiles by their IDs, not their names.

    $ nvidia-smi mig -lgip

    Example output:

    +--------------------------------------------------------------------------+
| GPU instance profiles:                                                   |
| GPU   Name          ID    Instances   Memory     P2P    SM    DEC   ENC  |
|                           Free/Total   GiB              CE    JPEG  OFA  |
|==========================================================================|
|   0  MIG 1g.5gb     19     7/7        4.95       No     14     0     0   |
|                                                          1     0     0   |
+--------------------------------------------------------------------------+
|   0  MIG 2g.10gb    14     3/3        9.90       No     28     1     0   |
|                                                          2     0     0   |
+--------------------------------------------------------------------------+
|   0  MIG 3g.20gb     9     2/2        19.79      No     42     2     0   |
|                                                          3     0     0   |
+--------------------------------------------------------------------------+
|   0  MIG 4g.20gb     5     1/1        19.79      No     56     2     0   |
|                                                          4     0     0   |
+--------------------------------------------------------------------------+
|   0  MIG 7g.40gb     0     1/1        39.59      No     98     5     0   |
|                                                          7     1     1   |
+--------------------------------------------------------------------------+

  3. Discover the GPU instance profiles that are mapped to the different vGPU types.

    $ nvidia-smi vgpu -s -v

    For example, for H100, some of the listing attributes of certain vGPU profiles look something like:

    # nvidia-smi vgpu -s -v -i 1
GPU 00000000:1A:00.0
vGPU Type ID : 0x335
Name : NVIDIA H100-1-10C
Class : Compute
GPU Instance Profile ID : 19
...
vGPU Type ID : 0x336
Name : NVIDIA H100-2-20C
Class : Compute
GPU Instance Profile ID : 14
…

  4. Create the GPU instances with a default compute instance corresponding to the vGPU types of the MIG-backed vGPUs you will create.

    $ nvidia-smi mig -cgi gpu-instance-profile-ids -C

    gpu-instance-profile-ids - A comma-separated list of GPU instance profile IDs specifying the GPU instances you want to create.

    This example creates two GPU instances of type 2g.10gb with profile ID 14.

    $ nvidia-smi mig -cgi 14,14 -C
Successfully created GPU instance ID  5 on GPU  2 using profile MIG 2g.10gb (ID 14)
Successfully created GPU instance ID  3 on GPU  2 using profile MIG 2g.10gb (ID 14)

Note

If you are creating a GPU Instance to support a 1:1 MIG-backed vGPU on a platform other than VMware vSphere, you can optionally create non-default Compute Instances for that vGPU by following the steps in Creating Non-Default Compute Instances in a GPU Instance for 1:1 vGPUs.

Creating Non-Default Compute Instances in a GPU Instance for 1:1 vGPUs#

This task is required only if you plan to use a 1:1 MIG-backed vGPU on a GPU Instance and wish to create non-default Compute Instances for that vGPU. This option is only available on platforms other than VMware vSphere.

Perform this task in your hypervisor command shell.

  1. Open a command shell as the root user on your hypervisor host machine if necessary.

  2. List the available GPU instances.

    $ nvidia-smi mig -lgi

    Example output:

    +----------------------------------------------------+
| GPU instances:                                     |
| GPU   Name          Profile  Instance   Placement  |
|                       ID       ID       Start:Size |
|====================================================|
|   2  MIG 2g.10gb      14        3          0:2     |
+----------------------------------------------------+
|   2  MIG 2g.10gb      14        5          4:2     |
+----------------------------------------------------+

  3. Create the compute instances that you need within each GPU instance.

    $ nvidia-smi mig -cci -gi gpu-instance-ids

    gpu-instance-ids - A comma-separated list of GPU instance IDs that specifies the GPU instances within which you want to create the compute instances.

    Caution

    To avoid an inconsistent state between a guest VM and the hypervisor host, do not create compute instances from the hypervisor on a GPU instance on which an active guest VM is running. Runtime changes to the vGPU’s Compute Instance configuration may be done by the guest VM itself, as explained in Modifying a MIG-Backed vGPU’s Configuration.

    This example creates a compute instance on each GPU instance 3 and 5.

    $ nvidia-smi mig -cci -gi 3,5
Successfully created compute instance on GPU  0 GPU instance ID  1 using profile ID  2
Successfully created compute instance on GPU  0 GPU instance ID  2 using profile ID  2

  4. Verify that the compute instances were created within each GPU instance.

    $ nvidia-smi

    Example output:

    +-----------------------------------------------------------------------------+
| MIG devices:                                                                |
+------------------+----------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |         Memory-Usage |        Vol|         Shared        |
|      ID  ID  Dev |           BAR1-Usage | SM     Unc| CE  ENC  DEC  OFA  JPG|
|                  |                      |        ECC|                       |
|==================+======================+===========+=======================|
|  2    3   0   0  |      0MiB /  9984MiB | 28      0 |  2   0    1    0    0 |
|                  |      0MiB / 16383MiB |           |                       |
+------------------+----------------------+-----------+-----------------------+
|  2    5   0   1  |      0MiB /  9984MiB | 28      0 |  2   0    1    0    0 |
|                  |      0MiB / 16383MiB |           |                       |
+------------------+----------------------+-----------+-----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|

    Note

    • Additional Compute Instances created in a VM at runtime are destroyed when the VM is shut down or rebooted. After the shutdown or reboot, only one Compute Instance remains in the VM.

Disabling MIG Mode for One or More GPUs#

If a GPU you want to use for time-sliced vGPUs or GPU passthrough has previously been configured for MIG-backed vGPUs, disable MIG mode on the GPU.

Prerequisites

  • The NVIDIA Virtual GPU Manager is installed on the hypervisor host.

  • You have root user privileges on your hypervisor host machine.

  • Other processes, such as CUDA applications, monitoring applications, or the nvidia-smi command, don’t use the GPU.

Steps

Perform this task in your hypervisor command shell.

  1. Open a command shell as the root user on your hypervisor host machine. You can use a secure shell (SSH) on all supported hypervisors. Individual hypervisors may provide additional means for logging in. For details, refer to the documentation for your hypervisor.

  2. Determine whether MIG mode is disabled. Use the nvidia-smi command for this purpose. By default, MIG mode is disabled but might have previously been enabled. This example shows that MIG mode is enabled on GPU 0.

    Note

    In the output from nvidia-smi, the NVIDIA A100 HGX 40GB GPU is referred to as A100-SXM4-40GB.

    $ nvidia-smi -i 0
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.16    Driver Version: 550.54.16   CUDA Version:  12.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      Off  | 00000000:36:00.0 Off |                    0 |
| N/A   29C    P0    62W / 400W |      0MiB / 40537MiB |      6%      Default |
|                               |                      |              Enabled |
+-------------------------------+----------------------+----------------------+

  3. If MIG mode is enabled, disable it.

    $ nvidia-smi -i [gpu-ids] -mig 0

    gpu-ids - A comma-separated list of GPU indexes, PCI bus IDs, or UUIDs specifying the GPUs you want to disable MIG mode. If gpu-ids are omitted, MIG mode is disabled for all GPUs in the system.

    This example disables MIG Mode on GPU 0.

    $ sudo nvidia-smi -i 0 -mig 0
Disabled MIG Mode for GPU 00000000:36:00.0
All done.

  4. Confirm that MIG mode was disabled. Use the nvidia-smi command for this purpose. This example shows that MIG mode is disabled on GPU 0.

    $ nvidia-smi -i 0
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.16    Driver Version: 550.54.16   CUDA Version:  12.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      Off  | 00000000:36:00.0 Off |                    0 |
| N/A   29C    P0    62W / 400W |      0MiB / 40537MiB |      6%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

Modifying a MIG-Backed vGPU’s Configuration From a Guest VM#

If you want to replace the compute instances created when the GPU was configured for MIG-backed vGPUs, you can delete them before adding the compute instances from within the guest VM.

Note

A MIG-backed vGPU that occupies an entire GPU instance is assigned all of the instance’s framebuffer. For such vGPUs, the maximum vGPUs per GPU instance in the tables in Virtual GPU Types for Supported GPUs is always 1.

Prerequisites

  • You have root user privileges in the guest VM.

  • Other processes, such as CUDA applications, monitoring applications, or the nvidia-smi command, don’t use the GPU instance.

Steps

Perform this task in a guest VM command shell.

  1. Open a command shell as the root user in the guest VM. You can use a secure shell (SSH) on all supported hypervisors. Individual hypervisors may provide additional means for logging in. For details, refer to the documentation for your hypervisor.

  2. List the available GPU instances

    $ nvidia-smi mig -lgi
+----------------------------------------------------+
| GPU instances:                                     |
   | GPU   Name          Profile  Instance   Placement  |
   |                       ID       ID       Start:Size |
   |====================================================|
   |   0  MIG 2g.10gb       0        0          0:8     |
   +----------------------------------------------------+

  3. Optional: If compute instances were created when the GPU was configured for MIG-backed vGPUs that you no longer require, delete them.

    $ nvidia-smi mig -dci -ci compute-instance-id -gi gpu-instance-id

    compute-instance-id - The ID of the compute instance that you want to delete.

    gpu-instance-id - The ID of the GPU instance from which you want to delete the compute instance.

    Note

    This command fails if another process is using the GPU instance. In this situation, stop all processes using the GPU instance and retry the command.

    This example deletes compute instance 0 from GPU instance 0 on GPU 0.

    $ nvidia-smi mig -dci -ci 0 -gi 0
Successfully destroyed compute instance ID  0 from GPU  0 GPU instance ID  0

  4. List the compute instance profiles that are available for your GPU instance.

    $ nvidia-smi mig -lcip

    This example shows that one MIG 2g.10gb compute instance or two MIG 1c.2g.10gb compute instances can be created within the GPU instance.

    $ nvidia-smi mig -lcip
    +-------------------------------------------------------------------------------+
    | Compute instance profiles:                                                    |
    | GPU    GPU      Name          Profile  Instances   Exclusive      Shared      |
    |      Instance                   ID     Free/Total     SM      DEC   ENC   OFA |
    |        ID                                                     CE    JPEG      |
    |===============================================================================|
    |   0     0       MIG 1c.2g.10gb   0      2/2           14       1     0     0  |
    |                                                                2     0        |
    +-------------------------------------------------------------------------------+
    |   0     0       MIG 2g.10gb      1*     1/1           28       1     0     0  |
    |                                                                2     0        |
    +-------------------------------------------------------------------------------+

  5. Create the compute instances that you need within the available GPU instance. Run the following command to create each compute instance individually.

    $ nvidia-smi mig -cci compute-instance-profile-id -gi gpu-instance-id

    compute-instance-profile-id - The compute instance profile ID that specifies the compute instance.

    gpu-instance-id - The GPU instance ID specifies the GPU instance within which you want to create the compute instance.

    Note

    This command fails if another process is using the GPU instance. In this situation, stop all GPU processes and retry the command.

    This example creates a MIG 2g.10gb compute instance on GPU instance 0.

    $ nvidia-smi mig -cci 1 -gi 0
Successfully created compute instance ID  0 on GPU  0 GPU instance ID  0 using profile MIG 2g.10gb (ID  1)

    This example creates two MIG 1c.2g.10gb compute instances on GPU instance 0 by running the same command twice.

      $ nvidia-smi mig -cci 0 -gi 0
  Successfully created compute instance ID  0 on GPU  0 GPU instance ID  0 using profile MIG 1c.2g.10gb (ID  0)
  $ nvidia-smi mig -cci 0 -gi 0
Successfully created compute instance ID  1 on GPU  0 GPU instance ID  0 using profile MIG 1c.2g.10gb (ID  0)

  6. Verify that the compute instances were created within the GPU instance. Use the nvidia-smi command for this purpose. This example confirms that a MIG 2g.10gb compute instance was created on GPU instance 0.

    $ nvidia-smi
Mon Mar 25 19:01:24 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.16    Driver Version: 550.54.16   CUDA Version:  12.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GRID A100X-2-10C     On  | 00000000:00:08.0 Off |                   On |
| N/A   N/A    P0    N/A /  N/A |   1058MiB / 10235MiB |     N/A      Default |
                                 |                      |              Enabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| MIG devices:                                                                |
+------------------+----------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |         Memory-Usage |        Vol|         Shared        |
|      ID  ID  Dev |           BAR1-Usage | SM     Unc| CE  ENC  DEC  OFA  JPG|
|                  |                      |        ECC|                       |
|==================+======================+===========+=======================|
|  0    0   0   0  |   1058MiB / 10235MiB | 28      0 |  2   0    1    0    0 |
|                  |      0MiB /  4096MiB |           |                       |
+------------------+----------------------+-----------+-----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

    This example confirms that two MIG 1c.2g.10gb compute instances were created on GPU instance 0.

    $ nvidia-smi
    Mon Mar 25 19:01:24 2024
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 550.54.16    Driver Version: 550.54.16   CUDA Version:  12.3     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  GRID A100X-2-10C     On  | 00000000:00:08.0 Off |                   On |
    | N/A   N/A    P0    N/A /  N/A |   1058MiB / 10235MiB |     N/A      Default |
    |                               |                      |              Enabled |
    +-------------------------------+----------------------+----------------------+

    +-----------------------------------------------------------------------------+
    | MIG devices:                                                                |
    +------------------+----------------------+-----------+-----------------------+
    | GPU  GI  CI  MIG |         Memory-Usage |        Vol|         Shared        |
    |      ID  ID  Dev |           BAR1-Usage | SM     Unc| CE  ENC  DEC  OFA  JPG|
    |                  |                      |        ECC|                       |
    |==================+======================+===========+=======================|
    |  0    0   0   0  |   1058MiB / 10235MiB | 14      0 |  2   0    1    0    0 |
    |                  |      0MiB /  4096MiB |           |                       |
    +------------------+                      +-----------+-----------------------+
    |  0    0   1   1  |                      | 14      0 |  2   0    1    0    0 |
    |                  |                      |           |                       |
    +------------------+----------------------+-----------+-----------------------+

    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+

Monitoring MIG-backed vGPU Activity#

Note

  1. MIG-backed vGPU activity cannot be monitored on GPUs based on the NVIDIA Ampere GPU architecture because the required hardware feature is absent.

  2. The --gpm-metrics option is supported only on MIG-backed vGPUs that are allocated all of the GPU instance’s frame buffer.

For more information, refer to the Monitoring MIG-backed vGPU Activity documentation.