> This page is for version 0.11.
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt
> - 0.11: https://docs.nvidia.com/nemo-helix/v0.11/llms.txt
> - 0.10: https://docs.nvidia.com/nemo-helix/v0.10/llms.txt
> - 0.9: https://docs.nvidia.com/nemo-helix/v0.9/llms.txt
> - 0.8: https://docs.nvidia.com/nemo-helix/v0.8/llms.txt
> - 0.7: https://docs.nvidia.com/nemo-helix/v0.7/llms.txt
> - 0.6: https://docs.nvidia.com/nemo-helix/v0.6/llms.txt
> - 0.5: https://docs.nvidia.com/nemo-helix/v0.5/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.

# OpenSandbox

> Install OpenSandbox as a cluster engine and point NeMo Helix jobs at the in-cluster server.

The NeMo Helix Helm chart does **not** install [OpenSandbox](https://github.com/opensandbox-group/OpenSandbox). Install the upstream charts, then point the platform at the running server.

Install OpenSandbox when you run **sandboxed GRPO or Evaluator Gym jobs**, or **Harbor taskset runs**: untrusted custom
environment FileSets run in isolated pods, not in the training or evaluation container. It does
**not** sandbox the rest of the platform (API, DPO, SFT, inference). For Evaluator-specific settings,
see [Configure Sandboxed Gym for Evaluator](/nemo-helix/v0.11/documentation/evaluate-models/agent-eval/gym-sandbox-configuration).
For Harbor taskset runs on Kubernetes, also see [Configure Harbor Taskset Runs](/nemo-helix/v0.11/documentation/kubernetes-deployment/setup/helm/harbor-sandbox-profiles).

This page is the **shared-kernel** path: sandbox pods use the cluster default OCI runtime. That is often **runc** (containerd) or **crun** (CRI-O, including OKE and OpenShift). **A kernel-isolated runtime is not required.** Use [OpenSandbox with Kata](/nemo-helix/v0.11/documentation/kubernetes-deployment/setup/helm/opensandbox-kata) when those Gym/GRPO sandbox pods must be isolated from the host kernel. The documented path is Kata QEMU because it gives each sandbox its own guest kernel; other isolated runtimes may work but have not been tested. Example Helm values live in [`k8s/helm/examples/opensandbox/`](https://github.com/NVIDIA-NeMo/nemo-helix/tree/main/k8s/helm/examples/opensandbox).

OpenSandbox `[secure_runtime]` is **server-global**. One OpenSandbox server cannot mix shared-kernel and Kata, so point the platform at **one** Service DNS. The cluster keeps its default OCI runtime for every other workload.

Upstream Helm, TOML, and SDK documentation: [Kubernetes deployment](https://github.com/opensandbox-group/OpenSandbox/blob/server/v0.2.1/docs/kubernetes/deployment.md), [configuration](https://github.com/opensandbox-group/OpenSandbox/blob/server/v0.2.1/docs/getting-started/configuration.md), [server configuration reference](https://github.com/opensandbox-group/OpenSandbox/blob/server/v0.2.1/server/configuration.md).

## Prerequisites

* A local [OpenSandbox](https://github.com/opensandbox-group/OpenSandbox) checkout with Helm charts under `kubernetes/charts/`, or a published chart tarball
* `kubectl` access to the cluster
* The NeMo Helix Helm release namespace (jobs run here; it is also `[kubernetes] namespace` in the server values)

## Namespace rule

OpenSandbox `config.toml` `[kubernetes] namespace` **must be the same namespace the platform jobs run in** (the Helm release namespace). Control-plane Deployments, Services, and template ConfigMaps can stay in `opensandbox-system`. Sandbox pods remount the job-storage PVC by claim name and resolve image-pull secrets in their own namespace, so a dedicated `opensandbox` workload namespace will fail PVC remounts and image pulls.

## Values jobs need

Training and other job pods are OpenSandbox **clients**. They do not need a kubeconfig to the sandbox API. The Python SDK builds `http(s)://{domain}/v1` from:

| Client need     | How it is supplied                                                                   | Notes                                                                                                                                                                                                                                                   |
| --------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| API host        | `OPEN_SANDBOX_DOMAIN`                                                                | In-cluster Service DNS, **no scheme**. Default example: `opensandbox-server.opensandbox-system.svc.cluster.local`                                                                                                                                       |
| API key         | `OPEN_SANDBOX_API_KEY`                                                               | Kubernetes Secret in the **job** namespace (not `opensandbox-system`)                                                                                                                                                                                   |
| Scheme          | `platform.sandbox_server_protocol` / Gym `host_provider_options.connection.protocol` | Set **`http`** for the in-cluster Service. If unset, NeMo-RL health URLs default to **https** and stall                                                                                                                                                 |
| Proxy           | `use_server_proxy`                                                                   | Leave true. Jobs cannot reach sandbox pod IPs                                                                                                                                                                                                           |
| Request timeout | `host_provider_options.connection.request_timeout_s`                                 | For the create call, defaults to the Gym host's ready wait plus 30s; later calls use the SDK default of 30s. Must be longer than the server's `sandbox_create_timeout_seconds`, because the create request stays open while the sandbox image is pulled |
| Capability      | `sandboxClusterCapable` / `platform.sandbox_cluster_capable`                         | Defaults to `false` (sandboxed GRPO fail-closes)                                                                                                                                                                                                        |
| Job PVC         | `rl.job_storage_pvc_claim` or `evaluator.sandbox_job_storage_pvc_claim`              | Sandboxes remount the same claim used by the selected Jobs execution profile                                                                                                                                                                            |

Gym sample YAML uses `OPENSANDBOX_DOMAIN` / `OPENSANDBOX_API_KEY`. NeMo Helix and the SDK use `OPEN_SANDBOX_*`. Do not mix the names.

## Install the shared-kernel server

Replace `NHX_NAMESPACE` with the platform release namespace.

```sh
export NHX_NAMESPACE=nemo-helix
export OPENSANDBOX_DIR=/path/to/OpenSandbox
export EXAMPLES=k8s/helm/examples/opensandbox

sed -i.bak "s/REPLACE_WITH_RELEASE_NAMESPACE/${NHX_NAMESPACE}/g" \
  "${EXAMPLES}/opensandbox-server.yaml"

kubectl create namespace opensandbox-system --dry-run=client -o yaml | kubectl apply -f -
kubectl create namespace "$NHX_NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -

kubectl apply -f "${EXAMPLES}/batchsandbox-template.yaml"

kubectl create secret generic opensandbox-server-api-key \
  -n opensandbox-system --from-literal=api-key="$(openssl rand -hex 32)"

# Jobs cannot secretKeyRef across namespaces. Copy the key into the job namespace.
kubectl get secret opensandbox-server-api-key -n opensandbox-system -o json \
  | jq 'del(.metadata.uid,.metadata.resourceVersion,.metadata.creationTimestamp,.metadata.namespace)' \
  | kubectl apply -n "$NHX_NAMESPACE" -f -

# The BatchSandbox template hard-codes imagePullSecrets.name: nvcrimagepullsecret.
# That Secret must exist in $NHX_NAMESPACE. If yours has a different name, edit
# imagePullSecrets in batchsandbox-template.yaml (and batchsandbox-template-kata-qemu.yaml
# if you use Kata) before kubectl apply.
kubectl get secret nvcrimagepullsecret -n "$NHX_NAMESPACE"

helm upgrade --install opensandbox-controller \
  "${OPENSANDBOX_DIR}/kubernetes/charts/opensandbox-controller" \
  --namespace opensandbox-system \
  -f "${EXAMPLES}/opensandbox-controller.yaml"

helm upgrade --install opensandbox-server \
  "${OPENSANDBOX_DIR}/kubernetes/charts/opensandbox-server" \
  --namespace opensandbox-system \
  -f "${EXAMPLES}/opensandbox-server.yaml"
```

## Point the platform at the server

```yaml
sandboxClusterCapable: true
opensandbox:
  domain: opensandbox-server.opensandbox-system.svc.cluster.local
  protocol: http
  apiKeySecret: opensandbox-server-api-key
  apiKeySecretKey: api-key
```

When `sandboxClusterCapable` is true, Kubernetes and Volcano job pods receive `OPEN_SANDBOX_DOMAIN` and a Secret-backed `OPEN_SANDBOX_API_KEY`. The chart still does not create OpenSandbox Deployments.

These values let sandboxed GRPO run. Kata adds kernel isolation. The NetworkPolicy values lock down the sandbox's cluster access when the cluster CNI enforces NetworkPolicy, and they are not required for the job to start. A CNI that only stores the policies leaves that access open. See [Secure Sandboxed GRPO](/nemo-helix/v0.11/documentation/kubernetes-deployment/setup/helm/sandboxed-grpo).

## Isolate sandbox pods

Every sandbox pod runs `execd`, the OpenSandbox command server. It listens on the pod IP and runs any command it is sent, as the image's user. OpenSandbox gives it no credential, creates no NetworkPolicy, and its egress sidecar filters only traffic leaving a sandbox.

The platform protects sandboxes in two layers:

* **A per-sandbox `execd` token.** Harbor evaluations give every sandbox they create its own random token. `execd` then refuses any request without it, whether the request comes straight to the pod IP or through the OpenSandbox server's proxy with the server API key. This works on every cluster and needs no configuration. The task in the sandbox can read its own token, which only reaches its own sandbox. The token needs `execd` v1.0.15 or later, the first release that reads `EXECD_ACCESS_TOKEN`. The example server values pin v1.0.21. An older `execd` ignores the token and accepts every request.
* **An ingress NetworkPolicy.** When `sandboxClusterCapable` or `platformConfig.platform.sandbox_cluster_capable` is true, the chart creates a NetworkPolicy in the sandbox namespace that admits sandbox pod ingress only from `opensandbox.namespace`. It doesn't depend on `networkPolicies.enabled` and can't be turned off; add anything else that needs to reach sandboxes to `opensandbox.sandboxNetworkPolicy.extraIngress`. It also covers ports other than `execd`, such as Compose services. Platform clients reach sandboxes through the OpenSandbox server's proxy, so nothing else needs to connect to them.

Sandboxes created by other clients get no token, and only the NetworkPolicy protects them.

**Known issue: the token is visible to anyone who can read pods.** OpenSandbox only accepts env vars as plain values, so the token is stored as plain text in the sandbox's pod spec and BatchSandbox resource. Anyone who can read pods in the sandbox namespace, such as a user with the built-in `view` role, can read it. To use it, they still need a way to send requests to the sandbox, which the NetworkPolicy blocks on clusters that enforce it.

The token travels in the `X-EXECD-ACCESS-TOKEN` header. If anything between the platform and the OpenSandbox server, such as an ingress controller or gateway, strips unknown headers, every command fails with `401`. Pass the header through.

```yaml
opensandbox:
  sandboxNetworkPolicy:
    # Optional: admit only the server pods instead of the whole server namespace.
    serverPodSelector:
      matchLabels:
        app.kubernetes.io/name: opensandbox-server
```

* The policy lands in `opensandbox.composeServices.sandboxNamespace`, or the release namespace when that is empty. Both must match `[kubernetes] namespace` in the server values.
* If `opensandbox.domain` is an in-cluster Service name, its namespace must match `opensandbox.namespace`. Otherwise the chart refuses to render, since the policy would shut the server out.
* If the server runs in the sandbox namespace, set `serverPodSelector`. Otherwise the chart refuses to render, since admitting that namespace would admit every sandbox.
* A client that sets `use_server_proxy: false` connects to the pod IP directly and is refused unless `extraIngress` admits it.
* Every request to a sandbox goes through the OpenSandbox server, which forwards it over a pool of at most 100 connections per server process. A running command holds one connection until it finishes, so about 100 trials running at once fill the pool, and the next request waits up to 180 seconds for a free connection and then fails with `500`. The default evaluation caps (`max_cluster_concurrent_jobs: 50`) stay under this. Raise them past it only after giving the server more capacity.
* In ingress gateway mode, add the gateway to `extraIngress` before upgrading, or it can't reach sandboxes.
* The CNI must enforce NetworkPolicy. One that doesn't, such as flannel, stores the policy and ignores it.

## Verify

```sh
export OPEN_SANDBOX_WORKLOAD_NS="$NHX_NAMESPACE"
./k8s/helm/examples/opensandbox/verify/shared-kernel.sh
```

The verifier checks the server Deployment, `/health`, creates a short-lived sandbox in the job namespace, and asserts an empty `runtimeClassName` (cluster default OCI runtime).