Skip to navigation

The NeMo Helix Helm chart does not install OpenSandbox. Install the upstream charts, then point the platform at the running server.

Install OpenSandbox when you run sandboxed GRPO or Evaluator Gym jobs, or Harbor taskset runs: untrusted custom environment FileSets run in isolated pods, not in the training or evaluation container. It does not sandbox the rest of the platform (API, DPO, SFT, inference). For Evaluator-specific settings, see Configure Sandboxed Gym for Evaluator. For Harbor taskset runs on Kubernetes, also see Configure Harbor Taskset Runs.

This page is the shared-kernel path: sandbox pods use the cluster default OCI runtime. That is often runc (containerd) or crun (CRI-O, including OKE and OpenShift). A kernel-isolated runtime is not required. Use OpenSandbox with Kata when those Gym/GRPO sandbox pods must be isolated from the host kernel. The documented path is Kata QEMU because it gives each sandbox its own guest kernel; other isolated runtimes may work but have not been tested. Example Helm values live in k8s/helm/examples/opensandbox/.

OpenSandbox [secure_runtime] is server-global. One OpenSandbox server cannot mix shared-kernel and Kata, so point the platform at one Service DNS. The cluster keeps its default OCI runtime for every other workload.

Upstream Helm, TOML, and SDK documentation: Kubernetes deployment, configuration, server configuration reference.

Prerequisites

  • A local OpenSandbox checkout with Helm charts under kubernetes/charts/, or a published chart tarball
  • kubectl access to the cluster
  • The NeMo Helix Helm release namespace (jobs run here; it is also [kubernetes] namespace in the server values)

Namespace rule

OpenSandbox config.toml [kubernetes] namespace must be the same namespace the platform jobs run in (the Helm release namespace). Control-plane Deployments, Services, and template ConfigMaps can stay in opensandbox-system. Sandbox pods remount the job-storage PVC by claim name and resolve image-pull secrets in their own namespace, so a dedicated opensandbox workload namespace will fail PVC remounts and image pulls.

Values jobs need

Training and other job pods are OpenSandbox clients. They do not need a kubeconfig to the sandbox API. The Python SDK builds http(s)://{domain}/v1 from:

Client needHow it is suppliedNotes
API hostOPEN_SANDBOX_DOMAINIn-cluster Service DNS, no scheme. Default example: opensandbox-server.opensandbox-system.svc.cluster.local
API keyOPEN_SANDBOX_API_KEYKubernetes Secret in the job namespace (not opensandbox-system)
Schemeplatform.sandbox_server_protocol / Gym host_provider_options.connection.protocolSet http for the in-cluster Service. If unset, NeMo-RL health URLs default to https and stall
Proxyuse_server_proxyLeave true. Jobs cannot reach sandbox pod IPs
Request timeouthost_provider_options.connection.request_timeout_sFor the create call, defaults to the Gym host’s ready wait plus 30s; later calls use the SDK default of 30s. Must be longer than the server’s sandbox_create_timeout_seconds, because the create request stays open while the sandbox image is pulled
CapabilitysandboxClusterCapable / platform.sandbox_cluster_capableDefaults to false (sandboxed GRPO fail-closes)
Job PVCrl.job_storage_pvc_claim or evaluator.sandbox_job_storage_pvc_claimSandboxes remount the same claim used by the selected Jobs execution profile

Gym sample YAML uses OPENSANDBOX_DOMAIN / OPENSANDBOX_API_KEY. NeMo Helix and the SDK use OPEN_SANDBOX_*. Do not mix the names.

Install the shared-kernel server

Replace NHX_NAMESPACE with the platform release namespace.

export NHX_NAMESPACE=nemo-helix
export OPENSANDBOX_DIR=/path/to/OpenSandbox
export EXAMPLES=k8s/helm/examples/opensandbox
sed -i.bak "s/REPLACE_WITH_RELEASE_NAMESPACE/${NHX_NAMESPACE}/g" \
"${EXAMPLES}/opensandbox-server.yaml"
kubectl create namespace opensandbox-system --dry-run=client -o yaml | kubectl apply -f -
kubectl create namespace "$NHX_NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f "${EXAMPLES}/batchsandbox-template.yaml"
kubectl create secret generic opensandbox-server-api-key \
-n opensandbox-system --from-literal=api-key="$(openssl rand -hex 32)"
# Jobs cannot secretKeyRef across namespaces. Copy the key into the job namespace.
kubectl get secret opensandbox-server-api-key -n opensandbox-system -o json \
| jq 'del(.metadata.uid,.metadata.resourceVersion,.metadata.creationTimestamp,.metadata.namespace)' \
| kubectl apply -n "$NHX_NAMESPACE" -f -
# The BatchSandbox template hard-codes imagePullSecrets.name: nvcrimagepullsecret.
# That Secret must exist in $NHX_NAMESPACE. If yours has a different name, edit
# imagePullSecrets in batchsandbox-template.yaml (and batchsandbox-template-kata-qemu.yaml
# if you use Kata) before kubectl apply.
kubectl get secret nvcrimagepullsecret -n "$NHX_NAMESPACE"
helm upgrade --install opensandbox-controller \
"${OPENSANDBOX_DIR}/kubernetes/charts/opensandbox-controller" \
--namespace opensandbox-system \
-f "${EXAMPLES}/opensandbox-controller.yaml"
helm upgrade --install opensandbox-server \
"${OPENSANDBOX_DIR}/kubernetes/charts/opensandbox-server" \
--namespace opensandbox-system \
-f "${EXAMPLES}/opensandbox-server.yaml"

Point the platform at the server

sandboxClusterCapable: true
opensandbox:
domain: opensandbox-server.opensandbox-system.svc.cluster.local
protocol: http
apiKeySecret: opensandbox-server-api-key
apiKeySecretKey: api-key

When sandboxClusterCapable is true, Kubernetes and Volcano job pods receive OPEN_SANDBOX_DOMAIN and a Secret-backed OPEN_SANDBOX_API_KEY. The chart still does not create OpenSandbox Deployments.

These values let sandboxed GRPO run. Kata adds kernel isolation. The NetworkPolicy values lock down the sandbox’s cluster access when the cluster CNI enforces NetworkPolicy, and they are not required for the job to start. A CNI that only stores the policies leaves that access open. See Secure Sandboxed GRPO.

Isolate sandbox pods

Every sandbox pod runs execd, the OpenSandbox command server. It listens on the pod IP and runs any command it is sent, as the image’s user. OpenSandbox gives it no credential, creates no NetworkPolicy, and its egress sidecar filters only traffic leaving a sandbox.

The platform protects sandboxes in two layers:

  • A per-sandbox execd token. Harbor evaluations give every sandbox they create its own random token. execd then refuses any request without it, whether the request comes straight to the pod IP or through the OpenSandbox server’s proxy with the server API key. This works on every cluster and needs no configuration. The task in the sandbox can read its own token, which only reaches its own sandbox. The token needs execd v1.0.15 or later, the first release that reads EXECD_ACCESS_TOKEN. The example server values pin v1.0.21. An older execd ignores the token and accepts every request.
  • An ingress NetworkPolicy. When sandboxClusterCapable or platformConfig.platform.sandbox_cluster_capable is true, the chart creates a NetworkPolicy in the sandbox namespace that admits sandbox pod ingress only from opensandbox.namespace. It doesn’t depend on networkPolicies.enabled and can’t be turned off; add anything else that needs to reach sandboxes to opensandbox.sandboxNetworkPolicy.extraIngress. It also covers ports other than execd, such as Compose services. Platform clients reach sandboxes through the OpenSandbox server’s proxy, so nothing else needs to connect to them.

Sandboxes created by other clients get no token, and only the NetworkPolicy protects them.

Known issue: the token is visible to anyone who can read pods. OpenSandbox only accepts env vars as plain values, so the token is stored as plain text in the sandbox’s pod spec and BatchSandbox resource. Anyone who can read pods in the sandbox namespace, such as a user with the built-in view role, can read it. To use it, they still need a way to send requests to the sandbox, which the NetworkPolicy blocks on clusters that enforce it.

The token travels in the X-EXECD-ACCESS-TOKEN header. If anything between the platform and the OpenSandbox server, such as an ingress controller or gateway, strips unknown headers, every command fails with 401. Pass the header through.

opensandbox:
sandboxNetworkPolicy:
# Optional: admit only the server pods instead of the whole server namespace.
serverPodSelector:
matchLabels:
app.kubernetes.io/name: opensandbox-server
  • The policy lands in opensandbox.composeServices.sandboxNamespace, or the release namespace when that is empty. Both must match [kubernetes] namespace in the server values.
  • If opensandbox.domain is an in-cluster Service name, its namespace must match opensandbox.namespace. Otherwise the chart refuses to render, since the policy would shut the server out.
  • If the server runs in the sandbox namespace, set serverPodSelector. Otherwise the chart refuses to render, since admitting that namespace would admit every sandbox.
  • A client that sets use_server_proxy: false connects to the pod IP directly and is refused unless extraIngress admits it.
  • Every request to a sandbox goes through the OpenSandbox server, which forwards it over a pool of at most 100 connections per server process. A running command holds one connection until it finishes, so about 100 trials running at once fill the pool, and the next request waits up to 180 seconds for a free connection and then fails with 500. The default evaluation caps (max_cluster_concurrent_jobs: 50) stay under this. Raise them past it only after giving the server more capacity.
  • In ingress gateway mode, add the gateway to extraIngress before upgrading, or it can’t reach sandboxes.
  • The CNI must enforce NetworkPolicy. One that doesn’t, such as flannel, stores the policy and ignores it.

Verify

export OPEN_SANDBOX_WORKLOAD_NS="$NHX_NAMESPACE"
./k8s/helm/examples/opensandbox/verify/shared-kernel.sh

The verifier checks the server Deployment, /health, creates a short-lived sandbox in the job namespace, and asserts an empty runtimeClassName (cluster default OCI runtime).