OpenSandbox Best Practices

One sandbox is one Kubernetes pod on a shared cluster. Your requests decide how much of the cluster you occupy, your limits decide what your commands may use, and both are shared with everyone else’s runs. Most “OpenSandbox is flaky” reports trace back to one of the traps below. See the OpenSandbox Provider page for the config schema.

The short list

1. Start from the shipped config — pass --config nemo_gym/sandbox/providers/opensandbox/configs/opensandbox.yaml and override only the keys you need. Its connection settings are the ones that are hard to get right (connection fields).
2. Start with low requests and set limits to the max CPU/memory you need — small resource_requests so you pack densely, resources sized to your real peaks; never the same map for both (requests are not limits).
3. Check what CPU count your sandbox actually sees — on many clusters it is the host’s cores, not your limit, and that varies cluster to cluster. Be mindful of it: leave derive_cpu_env on and give agents a bash timeout (host-core fan-out).
4. Use background execution — the shipped default; keep the max poll interval in the tens of seconds (background execution and timeouts).
5. Retry creates, never commands — a retried command can run twice (retries).
6. Turn on egress controls for any benchmark where the agent must not reach the internet — they are off by default (egress controls).
7. Clean up zombie sandboxes — reap an interrupted run by its run id, set ttl_s as the backstop, and keep attribution on so the run is findable (be a good tenant).
8. Check the failure table before filing a bug — most alarming errors at scale are known and benign (failure signatures). To see what a sandbox is actually getting and using, and to tell an OOM from a timeout, see Debugging OpenSandbox Sandboxes.

Connection fields that must be right

The shipped nemo_gym/sandbox/providers/opensandbox/configs/opensandbox.yaml has these set correctly; if you copy or override it, keep them.

Requests are not limits

resources becomes the pod’s limits. Scheduling requests are a separate map under provider_options.resource_requests. If you set only resources, the server uses it as both. These are Kubernetes requests and limits applied to the sandbox container.

1
sandbox_config:
2
  resources:                # limits: what a command may burst to
3
    cpu: 4
4
    memory_mib: 16384
5
    disk_gib: 30
6
  provider_options:
7
    resource_requests:      # requests: what the scheduler reserves for you
8
      cpu: 0.5
9
      memory_mib: 512
10
      disk_gib: 30

Low requests are safe because agent workloads are bursty: median memory use is ~1–2 GB against a 16 GiB limit, with occasional 7–16 GB peaks during builds and tests. The limits cover the peaks; the small requests keep the cluster usable. The values above are the validated defaults for SWE-bench-style environments; raise a limit only after measuring memory.peak or CPU throttling inside the sandbox — not from a failure message.

The host-core fan-out trap

What a sandbox reports as its CPU count varies cluster to cluster. On a plain Kubernetes node, nproc, os.cpu_count() and os.cpus() report the host’s cores (often 128–192), not your limit. Clusters that run LXCFS overlay /proc/cpuinfo, /proc/meminfo and /sys/devices/system/cpu/online with cgroup-aware values, so the same tools report your limit instead. Check before assuming either — run nproc and python -c 'import os; print(os.cpu_count())' in one sandbox on the cluster you are about to use.

Where the host count leaks through, tools that size worker pools from it — old jest, Python multiprocessing, pytest -n auto, make -j$(nproc), OpenBLAS — fan out to 100+ processes in a 4-CPU, 16 GiB box. The result is an OOM kill, or a near-miss that thrashes at the memory ceiling and runs 10× slower. It depends on each repo’s tooling, so it looks random.

derive_cpu_env (default true in the SWE-bench configs) is the portable mitigation: it injects cap variables derived from resources.cpu — OMP_NUM_THREADS, PYTHON_CPU_COUNT, PYTEST_XDIST_AUTO_NUM_WORKERS, GOMAXPROCS, CARGO_BUILD_JOBS, UV_THREADPOOL_SIZE, and others; see CPU_CAP_ENV_VARS in nemo_gym/sandbox/utils.py. Explicit sandbox_config.env keys win. Leave it on even where LXCFS is present; it is harmless there and protects you when the same config moves to a cluster without it.

Two things it cannot fix:

• Runners that ignore the environment: old jest, plain make, ninja, and Python multiprocessing before 3.13. On clusters without LXCFS, cap these in the task’s setup command or the image.
• Agent-written runaway code. This is the majority of OOMs on coding benchmarks, and Kubernetes group-OOM kills the whole sandbox, ending the rollout. Give agents a bash timeout (2 minutes is the validated default; 8 minutes measured quality-negative) so runaway commands fail fast and the agent recovers.

Background execution and timeouts

With operations.background_exec: false, exec() holds one streaming response open for the whole command; any load balancer with a fixed idle timeout drops that stream mid-command and the client hangs. With background_exec: true (the shipped setting) the provider instead submits the command with background=true, gets an execution id back, polls its status with short GETs, and fetches the output once when the status reports finished. It costs a few extra round trips per command and survives any idle timeout.

connection.request_timeout_s still governs the submit and the final output fetch.

Which timeout you are waiting on

Create is synchronous on the server: schedule the pod, lazy-load the image, start the exec daemon. On a healthy cluster that is p50 ≈ 7 s, p90 ≈ 20 s, p99 ≈ 2 min. On a full cluster, pods sit Pending for as long as your timeout allows.

• create.timeout_s must exceed ready_timeout_s; it bounds the whole create including the readiness wait (the shipped config: 1500 s over 1200 s).
• ttl_s is the server-side backstop if your client dies. Set it above your longest rollout.

Retries: creates yes, commands no

create.retries is safe: a failed create leaves nothing behind. operations.retries (shipped: 5, exponential backoff from retry_delay_s to retry_max_delay_s) covers the SDK calls after create — status and endpoint lookups, file transfer, close — plus a command submit that failed at TCP connect before the body reached the exec daemon. All of those are idempotent or provably never started, so retrying them is safe. operations.command_retries defaults to 0 and should stay there for agent commands, because a proxy 502 on POST /command does not mean the command never started — the server maps every transport error to 502, including a dropped read after the body reached the exec daemon, which runs commands on a context a dropped connection never cancels. A retried git apply runs twice. Raise it only for provably idempotent workloads.

Keep create.skip_health_check: false. Skipping it lets your first command race a pod whose exec daemon is not listening yet, and the resulting 502 kills the rollout.

Egress controls are off by default

Without a network policy the server creates no egress sidecar, and the sandbox has unrestricted internet access. For benchmarks that means an agent can pip install its way around a task, fetch the upstream fix from GitHub, or exfiltrate the test set. Harness-level deny lists (such as opencode’s git fetch / curl github.com permission rules) are advisory; the network policy is the enforced layer.

Enable it per sandbox with provider_options.network_policy. Rules are evaluated in order, match FQDNs or wildcard domains, and fall through to default_action:

1
sandbox_config:
2
  provider_options:
3
    network_policy:
4
      default_action: deny
5
      egress:
6
        - action: allow
7
          target: pypi.org
8
        - action: allow
9
          target: "*.pythonhosted.org"
10
        - action: allow
11
          target: "*.internal.example.com"   # the hosts the harness itself must reach

What happens on create: the server injects the egress sidecar, drops NET_ADMIN from the sandbox container so only the sidecar can touch the network stack, and routes DNS through the sidecar. Things to know:

• dns vs dns+nft is a server setting, not yours. In dns mode only name resolution is filtered — a hard-coded IP bypasses it. dns+nft also enforces at the IP layer with nftables (resolved IPs of allowed domains are admitted with a TTL), which is what a real default-deny needs. Ask which mode your cluster runs before relying on the policy for anti-cheating.
• Allow what the harness needs, then test with one rollout. Under default_action: deny, anything the agent process calls out to — the Gym model server, a package index, an internal artifact host — must be listed, or the first model call fails inside the sandbox with a connection error that looks like an infrastructure outage.
• A create with network_policy is rejected if the server has no egress image configured; that is a cluster setup gap, not a config typo.
• OPENSANDBOX_EGRESS_* variables in the sandbox env configure the sidecar (proxy behavior, nameserver exemptions). Without a network_policy they are silently dropped, because there is no sidecar to read them.

Be a good tenant

• Keep attribution on. Inside Kubernetes (no Slurm variables, OS user root) set NEMO_GYM_TEAM / NEMO_GYM_USER / NEMO_GYM_WORKLOAD on the pod, or nobody can tell whose run is filling the cluster.

• Clean up interrupted runs by the run id logged at the first create (--reap to delete, omit to audit):

  $
  python -m nemo_gym.sandbox.providers.opensandbox.cleanup_sandboxes \
  >
    --domain "$OPENSANDBOX_DOMAIN" --api-key "$OPENSANDBOX_API_KEY" \
  >
    --run-id <run-id> --user <user> --reap

• No apt-get per sandbox. A tool install per rollout is 10–40 s of network time times thousands of rollouts, plus a failure point. Bake tools into the image; mount the agent binary instead of downloading it.

• Stop sandboxes in the background. Teardown is a provider round-trip you never need to wait for; ttl_s reaps anything a background stop misses.

Failure signatures that are not what they look like

Sandboxes run with the default capability set — no SYS_PTRACE, no SYS_ADMIN. strace/gdb attach and mount namespaces fail for that reason, not because of your config.