> This page is for Confidential Containers, version v1.1.0 (default).
> For other versions, use one of these documentation indexes:
> - v1.1.0 (default): https://docs.nvidia.com/confidential-computing/containers/v1.1.0/llms.txt
> - v1.0.0: https://docs.nvidia.com/confidential-computing/containers/v1.0.0/llms.txt
> - Dev: https://docs.nvidia.com/confidential-computing/containers/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/confidential-computing/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/confidential-computing/_mcp/server.

# Configuring Confidential Container Workloads

> As a Container User, use this page to configure confidential GPU workloads on a prepared cluster.

As a [Container User](/confidential-computing/containers/personas#coco-persona-container-user), use this page to configure confidential GPU workloads on a prepared cluster.

A Confidential Container workload is a standard Kubernetes pod that runs inside a TEE-protected
virtual machine and requests one or more GPUs through the NVIDIA Kata sandbox device plugin.
Compared with a traditional GPU pod, a Confidential Container workload pod manifest differs in
various ways. In particular, the pod manifest:

* Selects a TEE-aware Kata runtime class instead of the default `runc`-based runtime.

* Requests GPU and NVSwitch resources using the resource types advertised by the NVIDIA
  Kata sandbox device plugin, which can be either default names or model-specific names.

* For NVSwitch-based HGX systems, requests every GPU and NVSwitch on the node together so
  that all devices reside inside the same Confidential Container virtual machine.

* For an attested production deployment, includes a Kata agent security policy that limits
  which Agent API calls the untrusted host can make into the guest.

**Before this page:** Complete the [Detailed Install Guide](/confidential-computing/containers/confidential-containers-deploy) and verify the cluster with [Run a Sample Workload](/confidential-computing/containers/run-sample-workload) (`Test PASSED` in pod logs).
For install steps, refer to [Prerequisites](/confidential-computing/containers/prerequisites) and [Detailed Install Guide](/confidential-computing/containers/confidential-containers-deploy).

This page describes each of these decisions and provides single-GPU and multi-GPU passthrough
manifest examples that you can copy and adapt to your environment.
The install sample uses a minimal manifest and does not attach an agent security policy.

## Select a Container Runtime Class

A Confidential Container workload must set `spec.runtimeClassName` to a TEE-aware Kata
runtime that NVIDIA provides through the `kata-deploy` Helm chart.
Select the runtime class based on the CPU TEE on the target worker node:

| Node TEE    | Runtime class              |
| ----------- | -------------------------- |
| AMD SEV-SNP | `kata-qemu-nvidia-gpu-snp` |
| Intel TDX   | `kata-qemu-nvidia-gpu-tdx` |

The `kata-deploy` chart also installs a `kata-qemu-nvidia-gpu` runtime class.
That class is intended for non-confidential Kata workloads.

## Reference GPU and NVSwitch Resource Types

The NVIDIA Kata sandbox device plugin advertises GPUs and NVSwitches to Kubernetes as extended resources.
Your pod manifest requests those resources under `resources.limits`.
You can use either the default resource types or model-specific resource types.

By default, every passthrough GPU is advertised as `nvidia.com/pgpu` and every NVSwitch is advertised as `nvidia.com/nvswitch`.
These names are stable across GPU models, which keeps manifests portable when every node in your cluster has the same GPU type.

A sample resource request using the default resource type is shown below:

```yaml
resources:
  limits:
    nvidia.com/pgpu: "1"
```

In heterogeneous clusters, where worker nodes use different GPU models, you can configure the Kata sandbox device plugin to advertise resources under model-specific names by setting
`P_GPU_ALIAS=""` (and optionally `NVSWITCH_ALIAS=""`) on the plugin.
With this configuration, GPUs are exposed as resources such as `nvidia.com/GH100_H200_141GB`,
which lets a workload pin itself to a specific accelerator model.

Refer to [Configuring GPU or NVSwitch Resource Types Name](/confidential-computing/containers/confidential-containers-deploy#coco-configuration-heterogeneous-clusters)
for the GPU Operator install flags that enable this behavior.

Use the model-specific resource name in workloads that must target a specific accelerator:

```yaml
resources:
  limits:
    nvidia.com/GH100_H200_141GB: "1"
```

Set the `NODE_NAME` environment variable to the node you want to check:

```console
export NODE_NAME="<node-name>"
```

To list the GPU and NVSwitch resource types advertised on a node, run:

```console
kubectl get node $NODE_NAME -o json | grep nvidia.com
```

*Example Output:*

```output
"nvidia.com/GH100_H200_141GB": "1"
```

## Single-GPU Passthrough

A single-GPU workload requests one GPU and runs inside its own Confidential Container virtual
machine.
This pattern is the recommended starting point for verifying a deployment and for most
independent workloads that do not require NVLink between GPUs.

1. Create a file, such as `cuda-vectoradd-kata.yaml`:

   ```yaml {7,14}
   apiVersion: v1
   kind: Pod
   metadata:
     name: cuda-vectoradd-kata
     namespace: default
   spec:
     runtimeClassName: kata-qemu-nvidia-gpu-snp # or kata-qemu-nvidia-gpu-tdx
     restartPolicy: Never
     containers:
       - name: cuda-vectoradd
         image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04"
         resources:
           limits:
             nvidia.com/pgpu: "1"
             memory: 16Gi
   ```

   > **Note**
   >
   > If you configured the Kata sandbox device plugin to use model-specific resource types,
   > replace `nvidia.com/pgpu` with the appropriate model-specific name, such as
   > `nvidia.com/GH100_H200_141GB`.

2. Create the pod:

   ```console
   kubectl apply -f cuda-vectoradd-kata.yaml
   ```

3. Verify the workload completes successfully:

   ```console
   kubectl logs cuda-vectoradd-kata
   ```

   *Example Output:*

   ```output
   [Vector addition of 50000 elements]
   Copy input data from the host memory to the CUDA device
   CUDA kernel launch with 196 blocks of 256 threads
   Copy output data from the CUDA device to the host memory
   Test PASSED
   Done
   ```

Refer to [Run a Sample Workload](/confidential-computing/containers/run-sample-workload) for the end-to-end verification flow including
deletion and pending-pod guidance.

## Multi-GPU Passthrough

Multi-GPU passthrough assigns every GPU and NVSwitch on a node to a single Confidential
Container virtual machine.
This configuration is required for NVSwitch (NVLink) based HGX systems running confidential
workloads.

> **Info**
>
> You must assign all the GPUs and NVSwitches on the node to the same Confidential Container
> virtual machine.
> Configuring only a subset of GPUs for Confidential Computing on a single node is not
> supported.

### NVIDIA Hopper PPCIE Mode

For NVIDIA Hopper GPUs, multi-GPU passthrough requires protected PCIe (PPCIE) mode, which
claims exclusive use of the NVSwitches for a single Confidential Container.
The NVIDIA Confidential Computing Manager for Kubernetes transitions GPUs into the correct
mode based on the `cc.mode` label that you set.

1. Set the `NODE_NAME` environment variable to the node you want to configure:

   ```console
   export NODE_NAME="<node-name>"
   ```

2. Apply the `ppcie` CC mode label to the node:

   ```console
   kubectl label node $NODE_NAME nvidia.com/cc.mode=ppcie --overwrite
   ```

Refer to [Managing the Confidential Computing Mode](/confidential-computing/containers/configure-cc-mode) for full details
on setting the CC mode and verifying the change.

NVIDIA Blackwell GPUs use NVLink encryption, which places the switches outside of the
Trusted Computing Base (TCB), so the default CC mode of `on` is sufficient and no additional
configuration is required.

### Run a Multi-GPU Workload

1. Create a file, such as `multi-gpu-kata.yaml`, with a pod manifest that requests every GPU
   and NVSwitch on the node:

   ```yaml {7,14,15,16}
   apiVersion: v1
   kind: Pod
   metadata:
     name: multi-gpu-kata
     namespace: default
   spec:
     runtimeClassName: kata-qemu-nvidia-gpu-snp # or kata-qemu-nvidia-gpu-tdx
     restartPolicy: Never
     containers:
       - name: cuda-sample
         image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04"
         resources:
           limits:
             nvidia.com/pgpu: "8"
             nvidia.com/nvswitch: "4" # Only for NVIDIA Hopper GPUs with PPCIE mode
             memory: 128Gi
   ```

   > **Note**
   >
   > If you configured `P_GPU_ALIAS` or `NVSWITCH_ALIAS` for heterogeneous clusters,
   > replace `nvidia.com/pgpu` and `nvidia.com/nvswitch` with the corresponding
   > model-specific resource types.
   > Refer to [Reference GPU and NVSwitch Resource Types](#coco-resource-types)
   > for details.

2. Create the pod:

   ```console
   kubectl apply -f multi-gpu-kata.yaml
   ```

   *Example Output:*

   ```output
   pod/multi-gpu-kata created
   ```

3. Verify the pod is running:

   ```console
   kubectl get pod multi-gpu-kata
   ```

   *Example Output:*

   ```output
   NAME             READY   STATUS    RESTARTS   AGE
   multi-gpu-kata   1/1     Running   0          30s
   ```

4. Verify that all GPUs are visible inside the container:

   ```console
   kubectl exec multi-gpu-kata -- nvidia-smi -L
   ```

   *Example Output:*

   ```output
   GPU 0: NVIDIA H100 (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
   GPU 1: NVIDIA H100 (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
   GPU 2: NVIDIA H100 (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
   GPU 3: NVIDIA H100 (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
   GPU 4: NVIDIA H100 (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
   GPU 5: NVIDIA H100 (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
   GPU 6: NVIDIA H100 (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
   GPU 7: NVIDIA H100 (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
   ```

5. Delete the pod:

   ```console
   kubectl delete -f multi-gpu-kata.yaml
   ```

## Attach a Kata Agent Security Policy

The Kata agent runs inside the guest virtual machine and manages the container lifecycle.
Because the Kata shim on the host is outside the TEE, the untrusted host can still issue Agent
API calls into the guest unless you restrict those Agent API calls.
An *agent security policy* is a Rego policy that the agent enforces so that only the operations
your workload declared are allowed.

The [sample workload](/confidential-computing/containers/run-sample-workload) omits an agent policy so that you can confirm
GPU passthrough with the smallest possible manifest.
Do not treat that YAML as a production template.

The Kata Containers `genpolicy` tool reads your Kubernetes YAML, infers the intended Agent API
calls, encodes the policy in base64, and appends it as an annotation on the same file.
The Kata agent in the guest receives that annotation.
Use `genpolicy` 4.1.0, which is the version this architecture validates with that agent.
Refer to [Supported Software Components](/confidential-computing/containers/supported-platforms#coco-supported-software-components) for the component matrix.

1. Download `genpolicy` 4.1.0 from the
   [Kata Containers 4.1.0 release](https://github.com/kata-containers/kata-containers/releases/tag/4.1.0).
   For usage details, refer to the
   [Agent Policy generation tool](https://github.com/kata-containers/kata-containers/blob/4.1.0/src/tools/genpolicy/README.md)
   documentation.

2. Run `genpolicy` against the manifest to deploy:

   ```console
   genpolicy -y cuda-vectoradd-kata.yaml
   ```

   Review the generated policy before you apply the manifest.
   The automatically-generated policy is a starting point.
   The policy can allow operations you do not want, such as `kubectl exec`, or omit operations your
   workload needs.

   > **Info**
   >
   > Review and, if needed, edit the policy so that it matches your threat model before you
   > use it.

3. Apply the annotated manifest:

   ```console
   kubectl apply -f cuda-vectoradd-kata.yaml
   ```

## Next Steps

* Refer to [Attestation](/confidential-computing/containers/attestation) for Trustee concepts and a local connectivity test.
  A complete attestation scenario must not only cover the TEE but also the Kata Agent API surface.
  For this purpose, the agent security policy contained in the workload annotation is included in the measured
  payload that is attested with the guest.
  The agent security policy is distinct from the Trustee attestation policy that decides whether
  to release secrets.
  Refer to the upstream [Understanding CoCo Policies](https://confidentialcontainers.org/docs/getting-started/securing-workloads/#understanding-coco-policies).

* Refer to [Managing the Confidential Computing Mode](/confidential-computing/containers/configure-cc-mode) to change the CC mode on GPUs at the cluster or node level.

* Refer to [Troubleshooting](/confidential-computing/containers/troubleshooting) if a workload does not schedule or the pod stays `Pending`.