OpenSandbox with Kata

Use this path when sandboxed GRPO / NeMo Gym sandbox pods must be isolated from the host kernel. We document Kata QEMU (RuntimeClass/kata-qemu) because it runs each sandbox in a QEMU VM with its own guest kernel. Other kernel-isolated runtimes (for example Kata Firecracker, or a different hardware-isolated RuntimeClass) may work the same way, but they have not been tested with NeMo Platform. Kata is optional; most installs only need the shared-kernel OpenSandbox path. This does not sandbox the rest of the platform (API, DPO, SFT, inference).

Complete the OpenSandbox namespace, Secret, and platform-values steps first; this page only adds virtualization prerequisites and the Kata server overlay.

OpenSandbox [secure_runtime] is server-global: one server process cannot mix shared-kernel and Kata. Kata is added as an extra RuntimeClass, so the cluster default OCI runtime stays in place for every other workload. For production, install one OpenSandbox server and set opensandbox.domain to that Service.

Upstream: Secure container runtime.

Node virtualization prerequisites

Each Kata worker needs:

• Hardware virtualization enabled in BIOS: Intel VMX, AMD SVM, or Arm virtualization extensions
• /dev/kvm on the node
• Nested virtualization if the node itself is a VM
• KVM / vhost kernel modules loaded
• CRI-O or containerd with a Kata handler

Confirm virtualization on the node. Do not use Node Feature Discovery CPUID labels for this; they report processor capability, not BIOS enablement. On the worker:

1
ls /dev/kvm
2
kvm-ok   # cpu-checker package on Debian/Ubuntu
3
grep -cE '(vmx|svm)' /proc/cpuinfo
4
lscpu | grep -i virt

Operators often already know BIOS virtualization is on; when it is the firmware default these checks still pass. After kata-deploy, kubectl get nodes -l katacontainers.io/kata-runtime=true shows nodes that received the runtime.

Install kata-deploy (QEMU only)

Pin kata-deploy 4.0.0. Restrict shims to QEMU so the cluster gets RuntimeClass/kata-qemu. The top-level nodeSelector is the kata-deploy DaemonSet. Select a custom label you apply after the host checks above (Intel VMX or AMD SVM, plus /dev/kvm) — not a runtime label. Do not select katacontainers.io/kata-runtime on the DaemonSet; kata-deploy applies that label after it lands. Keep it on the RuntimeClass only. If every node has working KVM, omit the DaemonSet selector so kata-deploy runs cluster-wide.

1
# example kata-deploy values — replace the custom label with one you apply
2
nodeSelector:
3
  virt.nvidia.com/kvm: "true"  # apply after confirming BIOS VMX or SVM and /dev/kvm
4
shims:
5
  disableAll: true
6
  qemu:
7
    enabled: true
8
    runtimeClass:
9
      nodeSelector:
10
        katacontainers.io/kata-runtime: "true"

1
helm upgrade --install kata-deploy oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy \
2
  --version 4.0.0 \
3
  --namespace kube-system \
4
  --create-namespace \
5
  -f kata-values.yaml
6
7
kubectl -n kube-system rollout status ds/kata-deploy --timeout=600s
8
kubectl get runtimeclass kata-qemu

A smoke pod with runtimeClassName: kata-qemu should reach Running and print a guest kernel (uname -r differs from the node kernelVersion).

CRI-O overlay.skip_mount_home retrofit

On containerd + runc (the usual default), kata-deploy registers Kata as an additional RuntimeClass. It does not replace runc. Skip this section.

This retrofit applies only to CRI-O (OKE, OpenShift, and similar). kata-deploy writes a CRI-O drop-in that includes overlay.skip_mount_home=true and restarts cri-o. That option only applies to overlay mounts created after it is active.

On nodes that already ran containers, a bare cri-o restart re-adopts old merged mounts. The Kata guest can then see an empty virtiofs shared/ directory and CreateContainer hangs.

For those nodes: cordon → drain → restart cri-o → uncordon. Fresh nodes that never ran containers before kata-deploy do not need this.

OpenSandbox Kata server

Use the same controller as the shared-kernel server. Install a second server only if you are comparing runtimes; production should run one:

1
export NMP_NAMESPACE=nemo-platform
2
export OPENSANDBOX_DIR=/path/to/OpenSandbox
3
export EXAMPLES=k8s/helm/examples/opensandbox
4
5
sed -i.bak "s/REPLACE_WITH_RELEASE_NAMESPACE/${NMP_NAMESPACE}/g" \
6
  "${EXAMPLES}/opensandbox-server-kata-qemu.yaml"
7
8
# Adjust batchsandbox-template-kata-qemu.yaml nodeSelector to match kata-deploy.
9
# imagePullSecrets.name is nvcrimagepullsecret; edit it if your Secret differs.
10
kubectl apply -f "${EXAMPLES}/batchsandbox-template-kata-qemu.yaml"
11
12
kubectl create secret generic opensandbox-server-kata-api-key \
13
  -n opensandbox-system --from-literal=api-key="$(openssl rand -hex 32)"
14
15
kubectl get secret opensandbox-server-kata-api-key -n opensandbox-system -o json \
16
  | jq 'del(.metadata.uid,.metadata.resourceVersion,.metadata.creationTimestamp,.metadata.namespace)
17
        | .metadata.name="opensandbox-server-api-key"' \
18
  | kubectl apply -n "$NMP_NAMESPACE" -f -
19
20
helm upgrade --install opensandbox-server-kata \
21
  "${OPENSANDBOX_DIR}/kubernetes/charts/opensandbox-server" \
22
  --namespace opensandbox-system \
23
  -f "${EXAMPLES}/opensandbox-server-kata-qemu.yaml"

The overlay sets:

1
[secure_runtime]
2
type = "kata"
3
k8s_runtime_class = "kata-qemu"

[kubernetes] namespace must still be the platform job namespace.

Point the platform at the Kata Service

1
sandboxClusterCapable: true
2
opensandbox:
3
  domain: opensandbox-server-kata.opensandbox-system.svc.cluster.local
4
  protocol: http
5
  apiKeySecret: opensandbox-server-api-key
6
  apiKeySecretKey: api-key

Verify

1
export OPEN_SANDBOX_WORKLOAD_NS="$NMP_NAMESPACE"
2
./k8s/helm/examples/opensandbox/verify/kata-qemu.sh

Expect runtimeClassName=kata-qemu, scheduling on a kata-labeled node, and a guest kernel that differs from the node kernel.

Kata still needs admission controls that reject privileged pods, host namespaces/devices, and unsafe hostPath mounts. Those policies are cluster-admin work; the platform chart does not install them.