OpenSandbox
The NeMo Helix Helm chart does not install OpenSandbox. Install the upstream charts, then point the platform at the running server.
Install OpenSandbox when you run sandboxed GRPO or Evaluator Gym jobs, or Harbor taskset runs: untrusted custom environment FileSets run in isolated pods, not in the training or evaluation container. It does not sandbox the rest of the platform (API, DPO, SFT, inference). For Evaluator-specific settings, see Configure Sandboxed Gym for Evaluator. For Harbor taskset runs on Kubernetes, also see Configure Harbor Taskset Runs.
This page is the shared-kernel path: sandbox pods use the cluster default OCI runtime. That is often runc (containerd) or crun (CRI-O, including OKE and OpenShift). A kernel-isolated runtime is not required. Use OpenSandbox with Kata when those Gym/GRPO sandbox pods must be isolated from the host kernel. The documented path is Kata QEMU because it gives each sandbox its own guest kernel; other isolated runtimes may work but have not been tested. Example Helm values live in k8s/helm/examples/opensandbox/.
OpenSandbox [secure_runtime] is server-global. One OpenSandbox server cannot mix shared-kernel and Kata, so point the platform at one Service DNS. The cluster keeps its default OCI runtime for every other workload.
Upstream Helm, TOML, and SDK documentation: Kubernetes deployment, configuration, server configuration reference.
Prerequisites
- A local OpenSandbox checkout with Helm charts under
kubernetes/charts/, or a published chart tarball kubectlaccess to the cluster- The NeMo Helix Helm release namespace (jobs run here; it is also
[kubernetes] namespacein the server values)
Namespace rule
OpenSandbox config.toml [kubernetes] namespace must be the same namespace the platform jobs run in (the Helm release namespace). Control-plane Deployments, Services, and template ConfigMaps can stay in opensandbox-system. Sandbox pods remount the job-storage PVC by claim name and resolve image-pull secrets in their own namespace, so a dedicated opensandbox workload namespace will fail PVC remounts and image pulls.
Values jobs need
Training and other job pods are OpenSandbox clients. They do not need a kubeconfig to the sandbox API. The Python SDK builds http(s)://{domain}/v1 from:
Gym sample YAML uses OPENSANDBOX_DOMAIN / OPENSANDBOX_API_KEY. NeMo Helix and the SDK use OPEN_SANDBOX_*. Do not mix the names.
Install the shared-kernel server
Replace NHX_NAMESPACE with the platform release namespace.
Point the platform at the server
When sandboxClusterCapable is true, Kubernetes and Volcano job pods receive OPEN_SANDBOX_DOMAIN and a Secret-backed OPEN_SANDBOX_API_KEY. The chart still does not create OpenSandbox Deployments.
These values let sandboxed GRPO run. Kata adds kernel isolation. The NetworkPolicy values lock down the sandbox’s cluster access when the cluster CNI enforces NetworkPolicy, and they are not required for the job to start. A CNI that only stores the policies leaves that access open. See Secure Sandboxed GRPO.
Isolate sandbox pods
Every sandbox pod runs execd, the OpenSandbox command server. It listens on the pod IP and runs any command it is sent, as the image’s user. OpenSandbox gives it no credential, creates no NetworkPolicy, and its egress sidecar filters only traffic leaving a sandbox.
The platform protects sandboxes in two layers:
- A per-sandbox
execdtoken. Harbor evaluations give every sandbox they create its own random token.execdthen refuses any request without it, whether the request comes straight to the pod IP or through the OpenSandbox server’s proxy with the server API key. This works on every cluster and needs no configuration. The task in the sandbox can read its own token, which only reaches its own sandbox. The token needsexecdv1.0.15 or later, the first release that readsEXECD_ACCESS_TOKEN. The example server values pin v1.0.21. An olderexecdignores the token and accepts every request. - An ingress NetworkPolicy. When
sandboxClusterCapableorplatformConfig.platform.sandbox_cluster_capableis true, the chart creates a NetworkPolicy in the sandbox namespace that admits sandbox pod ingress only fromopensandbox.namespace. It doesn’t depend onnetworkPolicies.enabledand can’t be turned off; add anything else that needs to reach sandboxes toopensandbox.sandboxNetworkPolicy.extraIngress. It also covers ports other thanexecd, such as Compose services. Platform clients reach sandboxes through the OpenSandbox server’s proxy, so nothing else needs to connect to them.
Sandboxes created by other clients get no token, and only the NetworkPolicy protects them.
Known issue: the token is visible to anyone who can read pods. OpenSandbox only accepts env vars as plain values, so the token is stored as plain text in the sandbox’s pod spec and BatchSandbox resource. Anyone who can read pods in the sandbox namespace, such as a user with the built-in view role, can read it. To use it, they still need a way to send requests to the sandbox, which the NetworkPolicy blocks on clusters that enforce it.
The token travels in the X-EXECD-ACCESS-TOKEN header. If anything between the platform and the OpenSandbox server, such as an ingress controller or gateway, strips unknown headers, every command fails with 401. Pass the header through.
- The policy lands in
opensandbox.composeServices.sandboxNamespace, or the release namespace when that is empty. Both must match[kubernetes] namespacein the server values. - If
opensandbox.domainis an in-cluster Service name, its namespace must matchopensandbox.namespace. Otherwise the chart refuses to render, since the policy would shut the server out. - If the server runs in the sandbox namespace, set
serverPodSelector. Otherwise the chart refuses to render, since admitting that namespace would admit every sandbox. - A client that sets
use_server_proxy: falseconnects to the pod IP directly and is refused unlessextraIngressadmits it. - Every request to a sandbox goes through the OpenSandbox server, which forwards it over a pool of at most 100 connections per server process. A running command holds one connection until it finishes, so about 100 trials running at once fill the pool, and the next request waits up to 180 seconds for a free connection and then fails with
500. The default evaluation caps (max_cluster_concurrent_jobs: 50) stay under this. Raise them past it only after giving the server more capacity. - In ingress gateway mode, add the gateway to
extraIngressbefore upgrading, or it can’t reach sandboxes. - The CNI must enforce NetworkPolicy. One that doesn’t, such as flannel, stores the policy and ignores it.
Verify
The verifier checks the server Deployment, /health, creates a short-lived sandbox in the job namespace, and asserts an empty runtimeClassName (cluster default OCI runtime).