GPU Operator with MIG

Multi-Instance GPU (MIG) allows GPUs based on the NVIDIA Ampere architecture (such as NVIDIA A100) to be securely partitioned into separate GPU Instances for CUDA applications. Refer to the MIG User Guide for more details on MIG.

This documents provides an overview of how to use the GPU Operator with nodes that support MIG.

Getting Started

Prerequisites

The GPU Operator starting with v1.7 supports the use of pre-installed drivers and the NVIDIA Container Toolkit (nvidia-docker2). Note that, the MIG Manager currently does not support the use of pre-installed drivers (with one of the reasons being that kubelet opens a handle to the NVIDIA driver when available).

As a result, the --set driver.enabled=false option in the Helm chart for use with pre-installed drivers is not supported when using the MIG Manager on NVIDIA Ampere GPUs. The MIG Manager only works when the Operator deploys the NVIDIA driver as a container.

Future releases of the MIG Manager will support the use of pre-installed drivers.

Initial Setup

In this example workflow, we start with a MIG strategy of single. The mixed strategy can also be specified and used in a similar manner.

Note

In a CSP IaaS environment such as Google Cloud, ensure that the mig-manager variable WITH_REBOOT is set to “true”. Refer to the note in the MIG User Guide for more information on the constraints with enabling MIG mode.

We can use the following option to install the GPU Operator:

$ helm install gpu-operator \
    deployments/gpu-operator \
    --set mig.strategy=single

At this point, all the pods, including the nvidia-mig-manager will be deployed on nodes that have MIG capable GPUs:

NAMESPACE                NAME                                                          READY   STATUS      RESTARTS   AGE
default                  gpu-operator-d6ccd4d8d-9cgzr                                  1/1     Running     2          6m58s
default                  gpu-operator-node-feature-discovery-master-867c4f7bfb-4nlq7   1/1     Running     0          6m58s
default                  gpu-operator-node-feature-discovery-worker-6rvr2              1/1     Running     1          6m58s
gpu-operator-resources   gpu-feature-discovery-sclxr                                   1/1     Running     0          6m39s
gpu-operator-resources   nvidia-container-toolkit-daemonset-tnh82                      1/1     Running     0          6m39s
gpu-operator-resources   nvidia-cuda-validator-qt6wq                                   0/1     Completed   0          3m11s
gpu-operator-resources   nvidia-dcgm-exporter-dh46q                                    1/1     Running     0          6m39s
gpu-operator-resources   nvidia-device-plugin-daemonset-t6qkz                          1/1     Running     0          6m39s
gpu-operator-resources   nvidia-device-plugin-validator-sd5f7                          0/1     Completed   0          105s
gpu-operator-resources   nvidia-driver-daemonset-f7ktr                                 1/1     Running     0          6m40s
gpu-operator-resources   nvidia-mig-manager-gzg8n                                      1/1     Running     0          79s
gpu-operator-resources   nvidia-operator-validator-vsccj                               1/1     Running     0          6m39s
kube-system              calico-kube-controllers-b656ddcfc-722nd                       1/1     Running     0          39m
kube-system              calico-node-vt5pz                                             1/1     Running     0          39m
kube-system              coredns-558bd4d5db-cmgzk                                      1/1     Running     0          39m
kube-system              coredns-558bd4d5db-hx98h                                      1/1     Running     0          39m
kube-system              etcd-a100-mig-k8s                                             1/1     Running     0          39m
kube-system              kube-apiserver-a100-mig-k8s                                   1/1     Running     0          39m
kube-system              kube-controller-manager-a100-mig-k8s                          1/1     Running     1          39m
kube-system              kube-proxy-7597j                                              1/1     Running     0          39m
kube-system              kube-scheduler-a100-mig-k8s                                   1/1     Running     1          39m

You can also check the labels applied to the node:

$ kubectl get node -o json | jq '.items[].metadata.labels'
"nvidia.com/cuda.driver.major": "460",
"nvidia.com/cuda.driver.minor": "73",
"nvidia.com/cuda.driver.rev": "01",
"nvidia.com/cuda.runtime.major": "11",
"nvidia.com/cuda.runtime.minor": "2",
"nvidia.com/gfd.timestamp": "1621375725",
"nvidia.com/gpu.compute.major": "8",
"nvidia.com/gpu.compute.minor": "0",
"nvidia.com/gpu.count": "1",
"nvidia.com/gpu.deploy.container-toolkit": "true",
"nvidia.com/gpu.deploy.dcgm-exporter": "true",
"nvidia.com/gpu.deploy.device-plugin": "true",
"nvidia.com/gpu.deploy.driver": "true",
"nvidia.com/gpu.deploy.gpu-feature-discovery": "true",
"nvidia.com/gpu.deploy.mig-manager": "true",
"nvidia.com/gpu.deploy.operator-validator": "true",
"nvidia.com/gpu.family": "ampere",
"nvidia.com/gpu.machine": "Google-Compute-Engine",
"nvidia.com/gpu.memory": "40536",
"nvidia.com/gpu.present": "true",
"nvidia.com/gpu.product": "A100-SXM4-40GB",
"nvidia.com/mig.strategy": "single"

Warning

The MIG Manager currently requires that all user workloads on the GPUs being configured be stopped. In some cases, the node may need to be rebooted (esp. in CSP IaaS), so the node may need to be cordoned before changing the MIG mode or the MIG geometry on the GPUs.

This requirement may be relaxed in future releases.

Configuring MIG Profiles

Now, let’s configure the GPU into a supported by setting the mig.config label on the GPU node.

Note

The mig-manager uses a ConfigMap called mig-parted-config in the gpu-operator-resources namespace in the daemonset to include supported MIG profiles. Refer to the ConfigMap to use when changing the label below or modify the ConfigMap appropriately for your use-case.

In this example, we use the 1g.5gb profile:

$ kubectl label nodes $NODE nvidia.com/mig.config=all-1g.5gb

The MIG manager will proceed to apply a mig.config.state label to the GPU and then terminate all the GPU pods in preparation to enable MIG mode and configure the GPU into the desired MIG geometry:

"nvidia.com/mig.config": "all-1g.5gb",
"nvidia.com/mig.config.state": "pending"
kube-system              kube-scheduler-a100-mig-k8s                                   1/1     Running     1          45m
gpu-operator-resources   nvidia-dcgm-exporter-dh46q                                    1/1     Terminating   0          13m
gpu-operator-resources   gpu-feature-discovery-sclxr                                   1/1     Terminating   0          13m
gpu-operator-resources   nvidia-device-plugin-daemonset-t6qkz                          1/1     Terminating   0          13m

Note

As described above, if the WITH_REBOOT option is set then the MIG manager will proceed to reboot the node:

"nvidia.com/mig.config": "all-1g.5gb",
"nvidia.com/mig.config.state": "rebooting"

Once the MIG manager has completed applying the configuration changes (including a node reboot if required), the node labels should appear as shown below:

"nvidia.com/cuda.driver.major": "460",
"nvidia.com/cuda.driver.minor": "73",
"nvidia.com/cuda.driver.rev": "01",
"nvidia.com/cuda.runtime.major": "11",
"nvidia.com/cuda.runtime.minor": "2",
"nvidia.com/gfd.timestamp": "1621442537",
"nvidia.com/gpu.compute.major": "8",
"nvidia.com/gpu.compute.minor": "0",
"nvidia.com/gpu.count": "7",
"nvidia.com/gpu.deploy.container-toolkit": "true",
"nvidia.com/gpu.deploy.dcgm-exporter": "true",
"nvidia.com/gpu.deploy.device-plugin": "true",
"nvidia.com/gpu.deploy.driver": "true",
"nvidia.com/gpu.deploy.gpu-feature-discovery": "true",
"nvidia.com/gpu.deploy.mig-manager": "true",
"nvidia.com/gpu.deploy.operator-validator": "true",
"nvidia.com/gpu.engines.copy": "1",
"nvidia.com/gpu.engines.decoder": "0",
"nvidia.com/gpu.engines.encoder": "0",
"nvidia.com/gpu.engines.jpeg": "0",
"nvidia.com/gpu.engines.ofa": "0",
"nvidia.com/gpu.family": "ampere",
"nvidia.com/gpu.machine": "Google-Compute-Engine",
"nvidia.com/gpu.memory": "4864",
"nvidia.com/gpu.multiprocessors": "14",
"nvidia.com/gpu.present": "true",
"nvidia.com/gpu.product": "A100-SXM4-40GB-MIG-1g.5gb",
"nvidia.com/gpu.slices.ci": "1",
"nvidia.com/gpu.slices.gi": "1",
"nvidia.com/mig.config": "all-1g.5gb",
"nvidia.com/mig.config.state": "success",
"nvidia.com/mig.strategy": "single"

The labels gpu.count and gpu.slices indicate that the devices are configured. We can also run nvidia-smi in the driver container to verify that the GPU has been configured:

$ sudo docker exec 629b93e200d9eea35be35a1b30991d007e48497d52a38e18a472945e44e52a8e nvidia-smi -L
GPU 0: A100-SXM4-40GB (UUID: GPU-5c89852c-d268-c3f3-1b07-005d5ae1dc3f)
  MIG 1g.5gb Device 0: (UUID: MIG-GPU-5c89852c-d268-c3f3-1b07-005d5ae1dc3f/7/0)
  MIG 1g.5gb Device 1: (UUID: MIG-GPU-5c89852c-d268-c3f3-1b07-005d5ae1dc3f/8/0)
  MIG 1g.5gb Device 2: (UUID: MIG-GPU-5c89852c-d268-c3f3-1b07-005d5ae1dc3f/9/0)
  MIG 1g.5gb Device 3: (UUID: MIG-GPU-5c89852c-d268-c3f3-1b07-005d5ae1dc3f/11/0)
  MIG 1g.5gb Device 4: (UUID: MIG-GPU-5c89852c-d268-c3f3-1b07-005d5ae1dc3f/12/0)
  MIG 1g.5gb Device 5: (UUID: MIG-GPU-5c89852c-d268-c3f3-1b07-005d5ae1dc3f/13/0)
  MIG 1g.5gb Device 6: (UUID: MIG-GPU-5c89852c-d268-c3f3-1b07-005d5ae1dc3f/14/0)

Finally, verify that the GPU Operator pods are in running state:

NAMESPACE                NAME                                                          READY   STATUS      RESTARTS   AGE
default                  gpu-operator-d6ccd4d8d-hhhq4                                  1/1     Running     4          38m
default                  gpu-operator-node-feature-discovery-master-867c4f7bfb-jt95x   1/1     Running     1          38m
default                  gpu-operator-node-feature-discovery-worker-rjpfb              1/1     Running     3          38m
gpu-operator-resources   gpu-feature-discovery-drzft                                   1/1     Running     0          97s
gpu-operator-resources   nvidia-container-toolkit-daemonset-885b5                      1/1     Running     1          38m
gpu-operator-resources   nvidia-cuda-validator-kh4tv                                   0/1     Completed   0          94s
gpu-operator-resources   nvidia-dcgm-exporter-6d5kd                                    1/1     Running     0          97s
gpu-operator-resources   nvidia-device-plugin-daemonset-kspv5                          1/1     Running     0          97s
gpu-operator-resources   nvidia-device-plugin-validator-mpgv9                          0/1     Completed   0          83s
gpu-operator-resources   nvidia-driver-daemonset-mgmdb                                 1/1     Running     3          38m
gpu-operator-resources   nvidia-mig-manager-svv7b                                      1/1     Running     1          35m
gpu-operator-resources   nvidia-operator-validator-w44q8                               1/1     Running     0          97s
kube-system              calico-kube-controllers-b656ddcfc-722nd                       1/1     Running     6          19h
kube-system              calico-node-vt5pz                                             1/1     Running     6          19h
kube-system              coredns-558bd4d5db-cmgzk                                      1/1     Running     5          19h
kube-system              coredns-558bd4d5db-hx98h                                      1/1     Running     5          19h
kube-system              etcd-a100-mig-k8s                                             1/1     Running     5          19h
kube-system              kube-apiserver-a100-mig-k8s                                   1/1     Running     5          19h
kube-system              kube-controller-manager-a100-mig-k8s                          1/1     Running     13         19h
kube-system              kube-proxy-7597j                                              1/1     Running     5          19h
kube-system              kube-scheduler-a100-mig-k8s                                   1/1     Running     11         19h

Reconfiguring MIG Profiles

The MIG manager supports dynamic reconfiguration of the MIG geometry. In this example, let’s reconfigure the GPU into a 3g.20gb profile:

$ kubectl label nodes $NODE nvidia.com/mig.config=all-3g.20gb --overwrite

We can see from the logs of the MIG manager that it has reconfigured the GPU into the new MIG geometry:

Applying the selected MIG config to the node
time="2021-05-19T16:42:14Z" level=debug msg="Parsing config file..."
time="2021-05-19T16:42:14Z" level=debug msg="Selecting specific MIG config..."
time="2021-05-19T16:42:14Z" level=debug msg="Running apply-start hook"
time="2021-05-19T16:42:14Z" level=debug msg="Checking current MIG mode..."
time="2021-05-19T16:42:14Z" level=debug msg="Walking MigConfig for (devices=all)"
time="2021-05-19T16:42:14Z" level=debug msg="  GPU 0: 0x20B010DE"
time="2021-05-19T16:42:14Z" level=debug msg="    Asserting MIG mode: Enabled"
time="2021-05-19T16:42:14Z" level=debug msg="    MIG capable: true\n"
time="2021-05-19T16:42:14Z" level=debug msg="    Current MIG mode: Enabled"
time="2021-05-19T16:42:14Z" level=debug msg="Checking current MIG device configuration..."
time="2021-05-19T16:42:14Z" level=debug msg="Walking MigConfig for (devices=all)"
time="2021-05-19T16:42:14Z" level=debug msg="  GPU 0: 0x20B010DE"
time="2021-05-19T16:42:14Z" level=debug msg="    Asserting MIG config: map[1g.5gb:7]"
time="2021-05-19T16:42:14Z" level=debug msg="Running pre-apply-config hook"
time="2021-05-19T16:42:14Z" level=debug msg="Applying MIG device configuration..."
time="2021-05-19T16:42:14Z" level=debug msg="Walking MigConfig for (devices=all)"
time="2021-05-19T16:42:14Z" level=debug msg="  GPU 0: 0x20B010DE"
time="2021-05-19T16:42:14Z" level=debug msg="    MIG capable: true\n"
time="2021-05-19T16:42:14Z" level=debug msg="    Updating MIG config: map[1g.5gb:7]"
time="2021-05-19T16:42:14Z" level=debug msg="Running apply-exit hook"
MIG configuration applied successfully
Restarting all GPU clients previouly shutdown by reenabling their component-specific nodeSelector labels
node/pramarao-a100-mig-k8s labeled
Changing the 'nvidia.com/mig.config.state' node label to 'success'

And the node labels have been updated appropriately:

"nvidia.com/gpu.product": "A100-SXM4-40GB-MIG-3g.20gb",
"nvidia.com/gpu.slices.ci": "3",
"nvidia.com/gpu.slices.gi": "3",
"nvidia.com/mig.config": "all-3g.20gb",

We can now proceed to run some sample workloads.

Running Sample CUDA Workloads

CUDA VectorAdd

Let’s run a simple CUDA sample, in this case vectorAdd by requesting a GPU resource as you would normally do in Kubernetes. In this case, Kubernetes will schedule the pod on a single MIG device and we use a nodeSelector to direct the pod to be scheduled on the node with the MIG devices.

$ cat << EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  containers:
  - name: vectoradd
    image: nvidia/samples:vectoradd-cuda11.2.1
    resources:
      limits:
        nvidia.com/gpu: 1
  nodeSelector:
    nvidia.com/gpu.product: A100-SXM4-40GB-MIG-1g.5gb
EOF

Concurrent Job Launch

Now, let’s try a more complex example. In this example, we will use Argo Workflows to launch concurrent jobs on MIG devices. In this example, the A100 has been configured into 2 MIG devices using the: 3g.20gb profile.

First, install the Argo Workflows components into your Kubernetes cluster.

$ kubectl create ns argo \
    && kubectl apply -n argo \
    -f https://raw.githubusercontent.com/argoproj/argo-workflows/stable/manifests/quick-start-postgres.yaml

Next, download the latest Argo CLI from the releases page and follow the instructions to install the binary.

Now, we will craft an Argo example that launches multiple CUDA containers onto the MIG devices on the GPU. We will reuse the same vectorAdd example from before. Here is the job description, saved as vector-add.yaml:

$ cat << EOF > vector-add.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: argo-mig-example-
spec:
entrypoint: argo-mig-result-example
templates:
- name: argo-mig-result-example
    steps:
    - - name: generate
        template: gen-mig-device-list
    # Iterate over the list of numbers generated by the generate step above
    - - name: argo-mig
        template: argo-mig
        arguments:
        parameters:
        - name: argo-mig
            value: "{{item}}"
        withParam: "{{steps.generate.outputs.result}}"

# Generate a list of numbers in JSON format
- name: gen-mig-device-list
    script:
    image: python:alpine3.6
    command: [python]
    source: |
        import json
        import sys
        json.dump([i for i in range(0, 2)], sys.stdout)

- name: argo-mig
    retryStrategy:
    limit: 10
    retryPolicy: "Always"
    inputs:
    parameters:
    - name: argo-mig
    container:
    image: nvidia/samples:vectoradd-cuda11.2.1
    resources:
        limits:
        nvidia.com/gpu: 1
    nodeSelector:
    nvidia.com/gpu.product: A100-SXM4-40GB-MIG-3g.20gb
EOF

Launch the workflow:

$ argo submit -n argo --watch vector-add.yaml

Argo will print out the pods that have been launched:

Name:                argo-mig-example-z6mqd
Namespace:           argo
ServiceAccount:      default
Status:              Succeeded
Conditions:
Completed           True
Created:             Wed Mar 24 14:44:51 -0700 (20 seconds ago)
Started:             Wed Mar 24 14:44:51 -0700 (20 seconds ago)
Finished:            Wed Mar 24 14:45:11 -0700 (now)
Duration:            20 seconds
Progress:            3/3
ResourcesDuration:   9s*(1 cpu),9s*(100Mi memory),1s*(1 nvidia.com/gpu)

STEP                       TEMPLATE                 PODNAME                           DURATION  MESSAGE
✔ argo-mig-example-z6mqd  argo-mig-result-example
├───✔ generate            gen-mig-device-list      argo-mig-example-z6mqd-562792713  8s
└─┬─✔ argo-mig(0:0)(0)    argo-mig                 argo-mig-example-z6mqd-845918106  2s
└─✔ argo-mig(1:1)(0)    argo-mig                 argo-mig-example-z6mqd-870679174  2s

If you observe the logs, you can see that the vector-add sample has completed on both devices:

$ argo logs -n argo @latest
argo-mig-example-z6mqd-562792713: [0, 1]
argo-mig-example-z6mqd-870679174: [Vector addition of 50000 elements]
argo-mig-example-z6mqd-870679174: Copy input data from the host memory to the CUDA device
argo-mig-example-z6mqd-870679174: CUDA kernel launch with 196 blocks of 256 threads
argo-mig-example-z6mqd-870679174: Copy output data from the CUDA device to the host memory
argo-mig-example-z6mqd-870679174: Test PASSED
argo-mig-example-z6mqd-870679174: Done
argo-mig-example-z6mqd-845918106: [Vector addition of 50000 elements]
argo-mig-example-z6mqd-845918106: Copy input data from the host memory to the CUDA device
argo-mig-example-z6mqd-845918106: CUDA kernel launch with 196 blocks of 256 threads
argo-mig-example-z6mqd-845918106: Copy output data from the CUDA device to the host memory
argo-mig-example-z6mqd-845918106: Test PASSED
argo-mig-example-z6mqd-845918106: Done

Architecture

The MIG manager is designed as a controller within Kubernetes. It watches for changes to the nvidia.com/mig.config label on the node and then applies the user requested MIG configuration When the label changes, the MIG Manager first stops all GPU pods (including the device plugin, gfd and dcgm-exporter), applies the MIG reconfiguration and restarts the GPU pods. The MIG reconfiguration may also involve a node reboot if required for enabling MIG mode.

The available MIG profiles are specified in a ConfigMap to the MIG manager daemonset. The user may choose one of these profiles to apply to the mig.config label to trigger a reconfiguration of the MIG geometry.

The MIG manager relies on the mig-parted tool to apply the configuration changes to the GPU, including enabling MIG mode (with a node reboot as required by some scenarios).