> This page is for Confidential Containers, version Dev. > For other versions, use one of these documentation indexes: > - v1.1.0 (default): https://docs.nvidia.com/confidential-computing/containers/v1.1.0/llms.txt > - v1.0.0: https://docs.nvidia.com/confidential-computing/containers/v1.0.0/llms.txt > - Dev: https://docs.nvidia.com/confidential-computing/containers/dev/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/confidential-computing/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/confidential-computing/_mcp/server. # Detailed Install Guide > As a Kubernetes Cluster Administrator, use these steps to deploy Kata Containers and the NVIDIA GPU Operator to your cluster and configure it for Confidential Containers. As a [Kubernetes Cluster Administrator](/confidential-computing/containers/dev/personas#coco-persona-kubernetes-cluster-administrator), use these steps to deploy Kata Containers and the NVIDIA GPU Operator to your cluster and configure it for Confidential Containers. If you want the fastest path and intend to run Confidential Containers on every node in your cluster, use the [Quickstart Install](/confidential-computing/containers/dev/install-quickstart) instead. Use this guide when you need per-node control, such as running Confidential Containers on some nodes and traditional GPU workloads on others, or when you want to use additional configuration options when installing the GPU Operator. ## Install Overview This guide assumes you completed [Prerequisites](/confidential-computing/containers/dev/prerequisites) on an existing Kubernetes cluster with GPU worker nodes. Install workflow: 1. [Prerequisites](/confidential-computing/containers/dev/prerequisites): prepare worker hosts and cluster software. 2. [Label nodes to deploy Confidential Containers components](#coco-label-nodes): select GPU workers for Confidential Containers workloads. 3. [Install Kata Containers](#coco-install-kata-chart): install runtime classes and node-level Kata components. 4. [Install the NVIDIA GPU Operator](#coco-install-gpu-operator): deploy Confidential Containers operands on target nodes. **Success criteria:** Helm releases report `STATUS: deployed`, the `kata-deploy` pod is `Running`, SNP and TDX runtime classes are available, and all GPU Operator operands are healthy on target nodes. After completing the installation, you can [Run a Sample Workload](/confidential-computing/containers/dev/run-sample-workload) to verify the deployment. ## Label Nodes for Confidential Containers Components The GPU Operator reads labels to determine what software components to deploy to a node. To configure a node for Confidential Container workloads, you label the node with the `nvidia.com/gpu.workload.config=vm-passthrough` label. Then, when the GPU Operator is installed in a subsequent step, it will deploy the software components needed to run Confidential Containers to the node. A node can only run one container runtime at a time, so a node configured for Confidential Container workloads cannot run traditional GPU container workloads. The labeling approach is useful if you want to run Confidential Containers workloads on some nodes and traditional GPU container workloads on other nodes in your cluster. For more details on how the GPU Operator deploys components to your cluster, refer to the [GPU Operator Cluster Topology Considerations](/confidential-computing/containers/dev/overview#coco-gpu-operator-cluster-topology) section in the architecture overview. > **Tip** > > Skip this section if you plan to use all nodes in your cluster to run Confidential Containers and instead set `sandboxWorkloads.defaultWorkload=vm-passthrough` when installing the GPU Operator. 1. Get a list of the nodes in your cluster: ```console kubectl get nodes ``` *Example Output:* ```output NAME STATUS ROLES AGE VERSION node-01 Ready 10d v1.34.0 node-02 Ready 10d v1.34.0 ``` Identify the GPU worker node or nodes you want to configure for Confidential Containers and use its name in the next step. 2. Set the `NODE_NAME` environment variable to the name of the node you want to configure: ```console export NODE_NAME="" ``` > **Note** > > Commands in this guide use the `$NODE_NAME` environment variable to reference this node. 3. Label the node for Confidential Containers: ```console kubectl label node $NODE_NAME nvidia.com/gpu.workload.config=vm-passthrough ``` *Example Output:* ```output node/ labeled ``` If you see ` not labeled`, the label may already be set. Continue to the next step to verify if the label was added. > **Note** > > To label multiple nodes at once, pass additional node names to the same command: > > ```console > kubectl label node nvidia.com/gpu.workload.config=vm-passthrough > ``` 4. Verify all target nodes have the label by listing all nodes with the label: ```console kubectl get nodes -l nvidia.com/gpu.workload.config=vm-passthrough ``` *Example Output:* ```output NAME STATUS ROLES AGE VERSION node-01 Ready 10d v1.34.0 ``` **Success criteria:** All nodes you intend to use for Confidential Container workloads appear in the output above. After all your desired nodes are labeled, you can continue to the next step to install Kata Containers. ## Install the Kata Containers Helm Chart Install Kata Containers using the `kata-deploy` Helm chart. The `kata-deploy` chart installs all required components from the Kata Containers project including the Kata Containers runtime binary, runtime configuration, UVM kernel, and images that NVIDIA uses for Confidential Containers and native Kata containers. The minimum required version is 4.0.0. 1. Set the chart version and registry path: ```console export VERSION="4.0.0" export CHART="oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy" ``` 2. Create a values file, such as `kata-nvidia-gpu-values.yaml`, to configure the `kata-deploy` chart for NVIDIA Confidential Containers: ```yaml title={"kata-nvidia-gpu-values.yaml"} # Example values file to enable NVIDIA GPU shims for the NVIDIA # Confidential Containers Reference Architecture. # Set to true for verbose kata-deploy and Kata runtime logging. debug: false deploymentMode: daemonset # Disable Node Feature Discovery (NFD) deployment by kata-deploy. # Both kata-deploy and the GPU Operator can deploy NFD. This reference # architecture relies on the NFD instance that the GPU Operator deploys # and manages, so the kata-deploy NFD subchart is turned off to avoid a # duplicate, conflicting deployment. node-feature-discovery: enabled: false # Install the nydus snapshotter on each node alongside containerd. # The confidential -snp and -tdx shims below use nydus to pull container # images directly into the confidential VM (guest pull), which keeps image # contents inside the trusted execution environment (TEE). snapshotter: setup: ["nydus"] # Disable the chart's default hypervisor/TEE shims and opt in only to # the NVIDIA GPU shims supported by this reference architecture. shims: disableAll: true # Non-confidential NVIDIA GPU passthrough shim used when Confidential # Computing mode is off on the node. The runtime class is restricted to # nodes where the GPU Operator's Confidential Computing Manager has # reported nvidia.com/cc.ready.state=false, so it will not schedule on # CC-ready nodes. The empty containerd snapshotter falls back to the # default (overlayfs); guest pull is not used for this non-confidential # path. qemu-nvidia-gpu: enabled: true supportedArches: - amd64 allowedHypervisorAnnotations: [] containerd: snapshotter: "" runtimeClass: # This label is automatically added by the GPU Operator. nodeSelector: nvidia.com/cc.ready.state: "false" # Confidential NVIDIA GPU passthrough for AMD SEV-SNP nodes. # Scheduled where the GPU Operator reports CC mode is on AND NFD # reports SEV-SNP support. Set agent.httpsProxy / agent.noProxy if # the guest needs a proxy to reach the registry. qemu-nvidia-gpu-snp: enabled: true supportedArches: - amd64 allowedHypervisorAnnotations: [] containerd: snapshotter: "nydus" forceGuestPull: false crio: guestPull: true agent: httpsProxy: "" noProxy: "" runtimeClass: # These labels are automatically added by the GPU Operator and NFD # respectively. nodeSelector: nvidia.com/cc.ready.state: "true" amd.feature.node.kubernetes.io/snp: "true" # Confidential NVIDIA GPU passthrough for Intel TDX nodes. # Same selectors and snapshotter behavior as the SNP shim above, # but pinned to TDX-capable hosts. qemu-nvidia-gpu-tdx: enabled: true supportedArches: - amd64 allowedHypervisorAnnotations: [] containerd: snapshotter: "nydus" forceGuestPull: false crio: guestPull: true agent: httpsProxy: "" noProxy: "" runtimeClass: # These labels are automatically added by the GPU Operator and NFD # respectively. nodeSelector: nvidia.com/cc.ready.state: "true" intel.feature.node.kubernetes.io/tdx: "true" # Default shim when a pod does not request a runtime class. Set to the # non-confidential shim so pods only run in a confidential VM when # they explicitly request the -snp or -tdx runtime class. defaultShim: amd64: qemu-nvidia-gpu # Can be changed to qemu-nvidia-gpu-snp or qemu-nvidia-gpu-tdx if preferred # Create one Kubernetes RuntimeClass per enabled shim above # (kata-qemu-nvidia-gpu, kata-qemu-nvidia-gpu-snp, kata-qemu-nvidia-gpu-tdx). # createDefault: false suppresses the generic "kata" RuntimeClass since # you should always reference a specific NVIDIA shim # by name in pod specs. runtimeClasses: enabled: true createDefault: false defaultName: "kata" ``` 3. Install the kata-deploy Helm chart with the values file: ```console helm install kata-deploy "${CHART}" \ --namespace kata-system --create-namespace \ -f kata-nvidia-gpu-values.yaml \ --wait --timeout 10m \ --version "${VERSION}" ``` *Example Output immediately after running the command:* ```output Pulled: ghcr.io/kata-containers/kata-deploy-charts/kata-deploy:4.0.0 Digest: sha256:aea41018779716ce2e0bf406d701637d10fb5a0792db51a08dfd3f76701eb933 ``` The `--wait` flag in the install command instructs Helm to wait until the release is deployed before returning. It can take a 2-3 minutes to return more output. *Example Output when the release is deployed:* ```output Pulled: ghcr.io/kata-containers/kata-deploy-charts/kata-deploy:4.0.0 Digest: sha256:aea41018779716ce2e0bf406d701637d10fb5a0792db51a08dfd3f76701eb933 LAST DEPLOYED: Wed Apr 1 17:03:00 2026 NAMESPACE: kata-system STATUS: deployed REVISION: 1 DESCRIPTION: Install complete TEST SUITE: None ``` > **Note** > > There is a [known Helm issue](https://github.com/helm/helm/issues/8660) on single node clusters, that may result in the Helm command finishing before all deployed pods are finished initializing. > If you are deploying to a single node cluster, you may need to wait for an additional few minutes after the Helm command completes for the `kata-deploy` pod to be in the Running state. `STATUS: deployed` confirms the Helm release succeeded and the chart resources were applied. The following steps confirm the Kata components are healthy. > **Note** > > Both `kata-deploy` and the GPU Operator can deploy Node Feature Discovery (NFD). > The values file disables the NFD deployment by `kata-deploy`. > The GPU Operator will deploy and manage NFD in the next step. 4. Verify that the `kata-deploy` pod is running: ```console kubectl get pods -n kata-system | grep kata-deploy ``` *Example Output:* ```output kata-deploy-b2lzs 1/1 Running 0 6m37s ``` A `READY` value of `1/1` and a `STATUS` of `Running` mean the `kata-deploy` pod installed the Kata components on the node successfully. If the pod is `Pending`, `ContainerCreating`, or `CrashLoopBackOff`, wait a minute and re-run the command. If the `kata-deploy` pod does not reach `Running`, refer to [View Kata Containers Logs](/confidential-computing/containers/dev/troubleshooting#coco-view-kata-logs) in [Troubleshooting](/confidential-computing/containers/dev/troubleshooting). 5. Verify that the `kata-qemu-nvidia-gpu-snp` and `kata-qemu-nvidia-gpu-tdx` runtime classes are available: ```console kubectl get runtimeclass | grep kata-qemu-nvidia-gpu ``` *Example Output:* ```output NAME HANDLER AGE kata-qemu-nvidia-gpu kata-qemu-nvidia-gpu 40s kata-qemu-nvidia-gpu-snp kata-qemu-nvidia-gpu-snp 40s kata-qemu-nvidia-gpu-tdx kata-qemu-nvidia-gpu-tdx 40s ``` Several runtimes are installed by the `kata-deploy` chart. The `kata-qemu-nvidia-gpu` runtime class is used with Kata Containers, in a non-Confidential Containers scenario. The `kata-qemu-nvidia-gpu-snp` for AMD-based systems or `kata-qemu-nvidia-gpu-tdx` for Intel-based systems runtime classes are used to deploy Confidential Containers workloads. The `kata-deploy` chart typically creates these runtime classes within 1-2 minutes after the `kata-deploy` pod reaches `Running`. If the SNP and TDX runtime classes are not listed immediately, the chart may still be initializing rather than failing. Wait 1-2 minutes and re-run the command. If they are still missing after the `kata-deploy` pod reports `Running`, the install did not complete correctly. Refer to [View Kata Containers Logs](/confidential-computing/containers/dev/troubleshooting#coco-view-kata-logs) in [Troubleshooting](/confidential-computing/containers/dev/troubleshooting) for help diagnosing the issue. On a single-node cluster, retry after a few minutes only if Helm returned before the `kata-deploy` pod reaches `Running` (refer to the note above). Otherwise, refer to the log steps below. **Success criteria:** Helm reports `STATUS: deployed`, the `kata-deploy` pod is `Running`, and both `kata-qemu-nvidia-gpu-snp` and `kata-qemu-nvidia-gpu-tdx` are available on the cluster. After all checks pass, continue to [Install the NVIDIA GPU Operator](#coco-install-gpu-operator). If the `kata-deploy` pod does not reach `Running` or the SNP and TDX runtime classes are missing, refer to [View Kata Containers Logs](/confidential-computing/containers/dev/troubleshooting#coco-view-kata-logs) in [Troubleshooting](/confidential-computing/containers/dev/troubleshooting). ## Install the NVIDIA GPU Operator Install the NVIDIA GPU Operator and configure it to deploy Confidential Container components. For more details on each of the GPU Operator components, refer to the [GPU Operator Cluster Topology Considerations](/confidential-computing/containers/dev/overview#coco-gpu-operator-components) section in the architecture overview. 1. Add and update the NVIDIA Helm repository: ```console helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \ && helm repo update ``` *Example Output:* ```output "nvidia" has been added to your repositories Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "nvidia" chart repository Update Complete. ⎈Happy Helming!⎈ ``` 2. Install the GPU Operator with the following configuration: > **Tip** > > Add `--set sandboxWorkloads.defaultWorkload=vm-passthrough` to configure every worker node for Confidential Containers workloads. > Refer to the [Label Nodes for Confidential Containers Components](#coco-label-nodes) section for more details on this use case. To customize the installation beyond the command below: * Refer to the [Common GPU Operator Configuration Settings](#coco-configuration-settings) section on this page for the Confidential Containers-specific configuration options you can specify when installing the GPU Operator. * Refer to the [Common chart customization options](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html#gpu-operator-helm-chart-options) in [Installing the NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html) for the additional general configuration options you can specify when installing the GPU Operator. ```console helm install --wait --timeout 10m --generate-name \ -n gpu-operator --create-namespace \ nvidia/gpu-operator \ --set sandboxWorkloads.enabled=true \ --set sandboxWorkloads.mode=kata \ --set nfd.enabled=true \ --set nfd.nodefeaturerules=true \ --version=v26.3.1 ``` *Example Output:* ```output NAME: gpu-operator LAST DEPLOYED: Tue Mar 10 17:58:12 2026 NAMESPACE: gpu-operator STATUS: deployed REVISION: 1 TEST SUITE: None ``` `STATUS: deployed` confirms the Helm release succeeded. The `--wait` flag instructs Helm to wait until the release is deployed before returning. It may take 3-5 minutes for the Helm command to complete. Use the following steps to confirm the GPU Operator components are deployed and configured correctly. 3. Verify that all GPU Operator pods, especially the Confidential Computing Manager, Kata Device Plugin and VFIO Manager operands, are running: ```console kubectl get pods -n gpu-operator ``` *Example Output:* ```output NAME READY STATUS RESTARTS AGE gpu-operator-1766001809-node-feature-discovery-gc-75776475sxzkp 1/1 Running 0 86s gpu-operator-1766001809-node-feature-discovery-master-6869lxq2g 1/1 Running 0 86s gpu-operator-1766001809-node-feature-discovery-worker-mh4cv 1/1 Running 0 86s gpu-operator-f48fd66b-vtfrl 1/1 Running 0 86s nvidia-cc-manager-7z74t 1/1 Running 0 61s nvidia-kata-sandbox-device-plugin-daemonset-d5rvg 1/1 Running 0 30s nvidia-sandbox-validator-6xnzc 1/1 Running 0 30s nvidia-vfio-manager-h229x 1/1 Running 0 62s ``` Each pod should report a `READY` value of `1/1` and a `STATUS` of `Running` or `Completed`. The `nvidia-cc-manager`, `nvidia-kata-sandbox-device-plugin-daemonset`, and `nvidia-vfio-manager` operands are specific to Confidential Containers and must be present on labeled nodes. Pods may briefly show `Pending` or `Init` while they start, which is expected. When all operands are `Running` or `Completed`, the GPU Operator components are deployed. 4. If you have host access to the worker node, you can perform the following validation step to confirm that the host uses the vfio-pci device driver for GPUs: ```console lspci -nnk -d 10de: ``` *Example Output:* ```output 65:00.0 3D controller [0302]: NVIDIA Corporation xxxxxxx [xxx] [10de:xxxx] (rev xx) Subsystem: NVIDIA Corporation xxxxxxx [xxx] [10de:xxxx] Kernel driver in use: vfio-pci Kernel modules: nvidiafb, nouveau ``` The `Kernel driver in use: vfio-pci` line in the output confirms the GPU is bound for VFIO passthrough into the confidential virtual machine. If the driver in use is `nvidia` or `nouveau` instead, the GPU is not ready for passthrough. Confirm your node meets the [Prerequisites](/confidential-computing/containers/dev/prerequisites#coco-prerequisites) section, including removing any NVIDIA GPU drivers on the host. **Success criteria:** All GPU Operator pods are `Running` or `Completed`. Your cluster is now configured to deploy workloads in Kata Containers. Continue to [Run a Sample Workload](/confidential-computing/containers/dev/run-sample-workload) to confirm everything is working as expected. If you are not seeing the expected output, view the logs for the GPU Operator pods or refer to [Troubleshooting](/confidential-computing/containers/dev/troubleshooting). ## Common GPU Operator Configuration Settings The following are the available GPU Operator configuration settings to enable Confidential Containers: | Parameter | Description | Default | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------- | | `sandboxWorkloads.enabled` | Enables sandbox workload management in the GPU Operator for virtual machine-style workloads and related operands. | `false` | | `sandboxWorkloads.defaultWorkload` | Specifies the default type of workload for the cluster, one of `container`, `vm-passthrough`, or `vm-vgpu`.Set to `vm-passthrough` if you plan to run all or mostly virtual machines in your cluster. | `container` | | `sandboxWorkloads.mode` | Specifies the sandbox mode to use when deploying sandbox workloads. Accepted values are `kubevirt` (default) and `kata`. Set to `kata` to run Confidential Containers workloads in Kata Containers. | `kubevirt` | | `kataSandboxDevicePlugin.env` | Optional list of environment variables passed to the NVIDIA Kata Device Plugin pod. Each list item is an `EnvVar` object with required `name` and optional `value` fields. Use the setting to configure `P_GPU_ALIAS` or `NVSWITCH_ALIAS` for the Kata sandbox device plugin. Refer to the [Configuring GPU or NVSwitch Resource Types Name](#coco-configuration-heterogeneous-clusters) section for more details. | `[]` (empty list) | ## Configuring GPU or NVSwitch Resource Types Name By default, the NVIDIA GPU Operator creates a resource type for GPUs and NVSwitches, `nvidia.com/pgpu` and `nvidia.com/nvswitch`. You can reference this name in your manifests to request GPU or NVSwitch resources for your workload. If you want to use a different name, you can set the `P_GPU_ALIAS` or `NVSWITCH_ALIAS` environment variables in the Kata device plugin to your preferred name. In clusters where all GPUs are the same model, a single resource type is typically sufficient. In heterogeneous clusters, where you have different GPU types on your nodes, you might want to use specific GPU types for your workload. To do this, specify an empty `P_GPU_ALIAS` environment variable in the Kata sandbox device plugin by adding the following to your GPU Operator installation: `--set kataSandboxDevicePlugin.env[0].name=P_GPU_ALIAS` and `--set kataSandboxDevicePlugin.env[0].value=""`. When this variable is set to `""`, the Kata device plugin creates GPU model-specific resource types, for example `nvidia.com/GH100_H200_141GB`, instead of the default `nvidia.com/pgpu` type. Use the exposed device resource types in pod specs by specifying respective resource limits. Similarly, you can set `NVSWITCH_ALIAS` to `""` to advertise model-specific NVSwitch resource types. The following example installs the GPU Operator with both `P_GPU_ALIAS` and `NVSWITCH_ALIAS` configured: ```console helm install --wait --timeout 10m --generate-name \ -n gpu-operator --create-namespace \ nvidia/gpu-operator \ --set sandboxWorkloads.enabled=true \ --set sandboxWorkloads.mode=kata \ --set nfd.enabled=true \ --set nfd.nodefeaturerules=true \ --set kataSandboxDevicePlugin.env[0].name=P_GPU_ALIAS \ --set kataSandboxDevicePlugin.env[0].value="" \ --set kataSandboxDevicePlugin.env[1].name=NVSWITCH_ALIAS \ --set kataSandboxDevicePlugin.env[1].value="" \ --version=v26.3.1 ``` After installing the GPU Operator, you can view the GPU or NVSwitch resource types available on a node by running the following command: ```console kubectl get node $NODE_NAME -o json | grep nvidia.com ``` > **Note** > > The `NODE_NAME` environment variable was set in the [Label Nodes](#coco-label-nodes) section. > If you want to view the resource types for a different node, you can update the `NODE_NAME` environment variable and run the command again. *Example Output:* ```output "nvidia.com/GH100_H200_141GB": "1" ``` You should see the resource type information for the GPUs and NVSwitches on the node. ## Next Steps > **Note** > > You now have a working Confidential Containers runtime. > > Attestation is what cryptographically verifies the TEE and releases secrets to a > production workload. For attestation concepts and a local > connectivity test, see the [Attestation](/confidential-computing/containers/dev/attestation) quickstart. For production attestation > deployment, refer to the upstream [Confidential Containers NVIDIA attestation guide](https://confidentialcontainers.org/docs/examples/nvidia-nim-confidential-gpu-attestation/). * [Run a Sample Workload](/confidential-computing/containers/dev/run-sample-workload) to verify your deployment. * To help manage the lifecycle of Kata Containers, install the [Kata Lifecycle Manager](https://github.com/kata-containers/lifecycle-manager). This Argo Workflows-based tool manages Kata Containers upgrades and day-two operations.