Container Device Interface Support in the GPU Operator

About the Container Device Interface

The Container Device Interface (CDI) is a specification for container runtimes such as cri-o, containerd, and podman that standardizes access to complex devices like NVIDIA GPUs by the container runtimes. CDI support is provided by the NVIDIA Container Toolkit and the Operator extends that support for Kubernetes clusters.

Use of CDI is transparent to cluster administrators and application developers. The benefits of CDI are largely to reduce development and support for runtime-specific plugins.

When CDI is enabled, two runtime classes, nvidia-cdi and nvidia-legacy, become available. These two runtime classes are in addition to the default runtime class, nvidia.

If you do not set CDI as the default runtime, the runtime resolves to the legacy runtime mode that the NVIDIA Container Toolkit provides on x86_64 machines or any architecture that has NVML libraries installed.

Optionally, you can specify the runtime class for a workload. See Optional: Specifying the Runtime Class for a Pod for an example.

Support for Multi-Instance GPU

Configuring CDI is supported with Multi-Instance GPU (MIG). Both the single and mixed strategies are supported.

Limitations and Restrictions

Enabling CDI is not supported with Red Hat OpenShift Container Platform. Refer to the Supported Operating Systems and Kubernetes Platforms.

Enabling CDI During Installation

Follow the instructions for installing the Operator with Helm on the Installing the NVIDIA GPU Operator page.

When you install the Operator with Helm, specify the --set cdi.enabled=true argument. Optionally, also specify the --set cdi.default=true argument to use the CDI runtime class by default for all pods.

Enabling CDI After Installation

Prerequisites

  • You installed version 22.3.0 or newer.

  • (Optional) Confirm that the only runtime class is nvidia by running the following command:

    $ kubectl get runtimeclasses

    Example Output

    NAME     HANDLER   AGE
nvidia   nvidia    47h

Procedure

To enable CDI support, perform the following steps:

  1. Enable CDI by modifying the cluster policy:

    $ kubectl patch clusterpolicies.nvidia.com/cluster-policy --type='json' \
    -p='[{"op": "replace", "path": "/spec/cdi/enabled", "value":true}]'

    Example Output

    clusterpolicy.nvidia.com/cluster-policy patched

  2. (Optional) Set the default container runtime mode to CDI by modifying the cluster policy:

    $ kubectl patch clusterpolicies.nvidia.com/cluster-policy --type='json' \
    -p='[{"op": "replace", "path": "/spec/cdi/default", "value":true}]'

    Example Output

    clusterpolicy.nvidia.com/cluster-policy patched

  3. (Optional) Confirm that the container toolkit and device plugin pods restart:

    $ kubectl get pods -n gpu-operator

    Example Output

    NAME                                                          READY   STATUS      RESTARTS   AGE
gpu-feature-discovery-qnw2q                                   1/1     Running     0          47h
gpu-operator-6d59774ff-hznmr                                  1/1     Running     0          2d
gpu-operator-node-feature-discovery-master-6d6649d597-7l8bj   1/1     Running     0          2d
gpu-operator-node-feature-discovery-worker-v86vj              1/1     Running     0          2d
nvidia-container-toolkit-daemonset-2768s                      1/1     Running     0          2m11s
nvidia-cuda-validator-ls4vc                                   0/1     Completed   0          47h
nvidia-dcgm-exporter-fxp9h                                    1/1     Running     0          47h
nvidia-device-plugin-daemonset-dvp4v                          1/1     Running     0          2m26s
nvidia-device-plugin-validator-kvxbs                          0/1     Completed   0          47h
nvidia-driver-daemonset-m86r7                                 1/1     Running     0          2d
nvidia-operator-validator-xg98r                               1/1     Running     0          47h

  4. Verify that the runtime classes include nvidia-cdi and nvidia-legacy:

    $ kubectl get runtimeclasses

    Example Output

    NAME            HANDLER         AGE
nvidia          nvidia          2d
nvidia-cdi      nvidia-cdi      5m7s
nvidia-legacy   nvidia-legacy   5m7s

Disabling CDI

To disable CDI support, perform the following steps:

  1. If your nodes use the CRI-O container runtime, then temporarily disable the GPU Operator validator:

    $ kubectl label nodes \
    nvidia.com/gpu.deploy.operator-validator=false \
    -l nvidia.com/gpu.present=true \
    --overwrite

    Tip

    You can run kubectl get nodes -o wide and view the CONTAINER-RUNTIME column to determine if your nodes use CRI-O.

  2. Disable CDI by modifying the cluster policy:

    $ kubectl patch clusterpolicies.nvidia.com/cluster-policy --type='json' \
    -p='[{"op": "replace", "path": "/spec/cdi/enabled", "value":false}]'

    Example Output

    clusterpolicy.nvidia.com/cluster-policy patched

  3. If you temporarily disabled the GPU Operator validator, re-enable the validator:

    $ kubectl label nodes \
    nvidia.com/gpu.deploy.operator-validator=true \
    nvidia.com/gpu.present=true \
    --overwrite

  4. (Optional) Verify that the nvidia-cdi and nvidia-legacy runtime classes are no longer available:

    $ kubectl get runtimeclass

    Example Output

    NAME     HANDLER   AGE
nvidia   nvidia    11d

Optional: Specifying the Runtime Class for a Pod

If you enabled CDI mode for the default container runtime, then no action is required to use CDI. However, you can use the following procedure to specify the legacy mode for a workload if you experience trouble.

If you did not enable CDI mode for the default container runtime, then you can use the following procedure to verify that CDI is enabled and as a routine practice to use the CDI mode of the container runtime.

  1. Create a file, such as cuda-vectoradd-cdi.yaml, with contents like the following example:

    apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  runtimeClassName: nvidia-cdi
  containers:
    - name: cuda-vectoradd
      image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubuntu20.04"
      resources:
        limits:
          nvidia.com/gpu: 1

    As an alternative, specify nvidia-legacy to use the legacy mode of the container runtime.

  2. (Optional) Create a temporary namespace:

    $ kubectl create ns demo

    Example Output

    namespace/demo created

  3. Start the pod:

    $ kubectl apply -n demo -f cuda-vectoradd-cdi.yaml

    Example Output

    pod/cuda-vectoradd created

  4. View the logs from the pod:

    $ kubectl logs -n demo cuda-vectoradd

    Example Output

    [Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

  5. Delete the temporary namespace:

$ kubectl delete ns demo

Example Output

namespace "demo" deleted