MIG Support in Kubernetes

This document provides steps on getting started and running some example CUDA workloads on MIG-enabled GPUs in a Kubernetes cluster.

Software Pre-requisites

The deployment workflow requires these prerequisites. Once these prerequisites have been met, you can proceed to deploy a MIG capable version of the NVIDIA k8s-device-plugin and the gpu-feature-discovery component in your cluster, so that Kubernetes can schedule pods on the available MIG devices.

  1. You already have a Kubernetes deployment up and running with access to at least one NVIDIA A100 GPU.

  2. The node with the NVIDIA A100 GPU is running the following versions of NVIDIA software:

    • NVIDIA datacenter driver >= 450.80.02

    • NVIDIA Container Toolkit (nvidia-docker2) >= 2.5.0 (and corresponding libnvidia-container >= 1.3.3)

  3. NVIDIA k8s-device-plugin: v0.7.0+

  4. NVIDIA gpu-feature-discovery: v0.2.0+

Getting Started

Install Kubernetes

As a first step, ensure that you have a Kubernetes deployment set up with a control plane and nodes joined to the cluster. Follow the Install Kubernetes guide for getting started with setting up a Kubernetes cluster.

Configuration Strategy


Setting up MIG Geometry

You can either use NVML (or its command-line interface nvidia-smi) to configure the desired MIG geometry. For automation, we recommend using tooling such as mig-parted that allows configuring MIG mode and creating the desired profiles on the GPUs.

In this step, let’s use mig-parted to configure the A100 into 7 GPUs (using the 1g.5gb profile):

$ sudo nvidia-mig-parted apply -f config.yaml -c all-1g.5gb

Now, the A100 should be configured into 7 MIG devices:

$ sudo nvidia-smi mig -lgi
| GPU instances:                                     |
| GPU   Name          Profile  Instance   Placement  |
|                       ID       ID       Start:Size |
|   0  MIG 1g.5gb       19        7          0:1     |
|   0  MIG 1g.5gb       19        8          1:1     |
|   0  MIG 1g.5gb       19        9          2:1     |
|   0  MIG 1g.5gb       19       10          3:1     |
|   0  MIG 1g.5gb       19       11          4:1     |
|   0  MIG 1g.5gb       19       12          5:1     |
|   0  MIG 1g.5gb       19       13          6:1     |

Deploying the NVIDIA Device Plugin and GFD

NVIDIA Device Plugin

Depending on the MIG configuration strategy used for the cluster, deploy the NVIDIA device plugin with the right options. In this example, we assume that the user has chosen a single MIG strategy for the cluster.

$ helm install \
   --generate-name \
   --set migStrategy=single \

At this point, the nvidia-device-plugin daemonset should be deployed and enumerated the MIG devices to Kubernetes:

2021/04/26 23:19:15 Loading NVML
2021/04/26 23:19:15 Starting FS watcher.
2021/04/26 23:19:15 Starting OS watcher.
2021/04/26 23:19:15 Retreiving plugins.
2021/04/26 23:19:16 Starting GRPC server for 'nvidia.com/gpu'
2021/04/26 23:19:16 Starting to serve 'nvidia.com/gpu' on /var/lib/kubelet/device-plugins/nvidia-gpu.sock
2021/04/26 23:19:16 Registered device plugin for 'nvidia.com/gpu' with Kubelet

GPU Feature Discovery

Next, we deploy the GPU Feature Discovery (GFD) plugin to label the GPU nodes so that users can specific MIG devices as resources in their podspec. Note that the GFD Helm chart also deploys the Node Feature Discovery (NFD) as a prerequisite:

$ helm repo add nvgfd https://nvidia.github.io/gpu-feature-discovery \
    && helm repo update
$ helm install \
   --generate-name \
   --set migStrategy=single \

At this point, we can verify that all pods are running:

NAMESPACE                NAME                                       READY   STATUS    RESTARTS   AGE
kube-system              calico-kube-controllers-6d8ccdbf46-wjst8   1/1     Running   1          4h58m
kube-system              calico-node-qp5wf                          1/1     Running   1          4h58m
kube-system              coredns-558bd4d5db-c6nhk                   1/1     Running   1          4h59m
kube-system              coredns-558bd4d5db-cgjr7                   1/1     Running   1          4h59m
kube-system              etcd-ipp1-0552                             1/1     Running   1          5h
kube-system              kube-apiserver-ipp1-0552                   1/1     Running   1          5h
kube-system              kube-controller-manager-ipp1-0552          1/1     Running   1          5h
kube-system              kube-proxy-d7tqd                           1/1     Running   1          4h59m
kube-system              kube-scheduler-ipp1-0552                   1/1     Running   1          5h
kube-system              nvidia-device-plugin-1619479152-646qm      1/1     Running   0          115m
node-feature-discovery   gpu-feature-discovery-1619479450-f7rvv     1/1     Running   0          110m
node-feature-discovery   nfd-master-74f76f6c68-zgt9d                1/1     Running   0          110m
node-feature-discovery   nfd-worker-pkdn2                           1/1     Running   0          110m

And the node has been labeled:

$ kubectl get node -o json | jq '.items[].metadata.labels'

with labels:

"node-role.kubernetes.io/master": "",
"node.kubernetes.io/exclude-from-external-load-balancers": "",
"nvidia.com/cuda.driver.major": "460",
"nvidia.com/cuda.driver.minor": "73",
"nvidia.com/cuda.driver.rev": "01",
"nvidia.com/cuda.runtime.major": "11",
"nvidia.com/cuda.runtime.minor": "2",
"nvidia.com/gfd.timestamp": "1619479472",
"nvidia.com/gpu.compute.major": "8",
"nvidia.com/gpu.compute.minor": "0",
"nvidia.com/gpu.count": "7",
"nvidia.com/gpu.engines.copy": "1",
"nvidia.com/gpu.engines.decoder": "0",
"nvidia.com/gpu.engines.encoder": "0",
"nvidia.com/gpu.engines.jpeg": "0",
"nvidia.com/gpu.engines.ofa": "0",
"nvidia.com/gpu.family": "ampere",
"nvidia.com/gpu.machine": "SYS-1019GP-TT-02-NC24B",
"nvidia.com/gpu.memory": "4864",
"nvidia.com/gpu.multiprocessors": "14",
"nvidia.com/gpu.product": "A100-PCIE-40GB-MIG-1g.5gb",
"nvidia.com/gpu.slices.ci": "1",
"nvidia.com/gpu.slices.gi": "1",
"nvidia.com/mig.strategy": "single"

We can now proceed to run some sample workloads.

Running Sample CUDA Workloads

CUDA VectorAdd

Let’s run a simple CUDA sample, in this case vectorAdd by requesting a GPU resource as you would normally do in Kubernetes. In this case, Kubernetes will schedule the pod on a single MIG device and we use a nodeSelector to direct the pod to be scheduled on the node with the MIG devices.

$ cat << EOF | kubectl create -f -
apiVersion: v1
kind: Pod
  name: cuda-vectoradd
  restartPolicy: OnFailure
  - name: vectoradd
    image: nvidia/samples:vectoradd-cuda11.2.1
        nvidia.com/gpu: 1
    nvidia.com/gpu.product: A100-SXM4-40GB-MIG-1g.5gb

Concurrent Job Launch

Now, let’s try a more complex example. In this example, we will use Argo Workflows to launch concurrent jobs on MIG devices. In this example, the A100 has been configured into 2 MIG devices using the: 3g.20gb profile.

First, install the Argo Workflows components into your Kubernetes cluster.

$ kubectl create ns argo \
    && kubectl apply -n argo \
    -f https://raw.githubusercontent.com/argoproj/argo-workflows/stable/manifests/quick-start-postgres.yaml

Next, download the latest Argo CLI from the releases page and follow the instructions to install the binary.

Now, we will craft an Argo example that launches multiple CUDA containers onto the MIG devices on the GPU. We will reuse the same vectorAdd example from before. Here is the job description, saved as vector-add.yaml:

$ cat << EOF > vector-add.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
generateName: argo-mig-example-
entrypoint: argo-mig-result-example
- name: argo-mig-result-example
    - - name: generate
        template: gen-mig-device-list
    # Iterate over the list of numbers generated by the generate step above
    - - name: argo-mig
        template: argo-mig
        - name: argo-mig
            value: "{{item}}"
        withParam: "{{steps.generate.outputs.result}}"

# Generate a list of numbers in JSON format
- name: gen-mig-device-list
    image: python:alpine3.6
    command: [python]
    source: |
        import json
        import sys
        json.dump([i for i in range(0, 2)], sys.stdout)

- name: argo-mig
    limit: 10
    retryPolicy: "Always"
    - name: argo-mig
    image: nvidia/samples:vectoradd-cuda11.2.1
        nvidia.com/gpu: 1
    nvidia.com/gpu.product: A100-SXM4-40GB-MIG-3g.20gb

Launch the workflow:

$ argo submit -n argo --watch vector-add.yaml

Argo will print out the pods that have been launched:

Name:                argo-mig-example-z6mqd
Namespace:           argo
ServiceAccount:      default
Status:              Succeeded
Completed           True
Created:             Wed Mar 24 14:44:51 -0700 (20 seconds ago)
Started:             Wed Mar 24 14:44:51 -0700 (20 seconds ago)
Finished:            Wed Mar 24 14:45:11 -0700 (now)
Duration:            20 seconds
Progress:            3/3
ResourcesDuration:   9s*(1 cpu),9s*(100Mi memory),1s*(1 nvidia.com/gpu)

STEP                       TEMPLATE                 PODNAME                           DURATION  MESSAGE
✔ argo-mig-example-z6mqd  argo-mig-result-example
├───✔ generate            gen-mig-device-list      argo-mig-example-z6mqd-562792713  8s
└─┬─✔ argo-mig(0:0)(0)    argo-mig                 argo-mig-example-z6mqd-845918106  2s
└─✔ argo-mig(1:1)(0)    argo-mig                 argo-mig-example-z6mqd-870679174  2s

If you observe the logs, you can see that the vector-add sample has completed on both devices:

$ argo logs -n argo @latest
argo-mig-example-z6mqd-562792713: [0, 1]
argo-mig-example-z6mqd-870679174: [Vector addition of 50000 elements]
argo-mig-example-z6mqd-870679174: Copy input data from the host memory to the CUDA device
argo-mig-example-z6mqd-870679174: CUDA kernel launch with 196 blocks of 256 threads
argo-mig-example-z6mqd-870679174: Copy output data from the CUDA device to the host memory
argo-mig-example-z6mqd-870679174: Test PASSED
argo-mig-example-z6mqd-870679174: Done
argo-mig-example-z6mqd-845918106: [Vector addition of 50000 elements]
argo-mig-example-z6mqd-845918106: Copy input data from the host memory to the CUDA device
argo-mig-example-z6mqd-845918106: CUDA kernel launch with 196 blocks of 256 threads
argo-mig-example-z6mqd-845918106: Copy output data from the CUDA device to the host memory
argo-mig-example-z6mqd-845918106: Test PASSED
argo-mig-example-z6mqd-845918106: Done