OpenSandbox Provider
The opensandbox provider creates sandboxes through the OpenSandbox SDK and server API. It is the Kubernetes-backed provider used by agentic software engineering environments such as mini_swe_agent_2.
Running at scale? Read OpenSandbox Best Practices for resource requests vs limits, timeout and retry guidance, and the failure signatures that are not bugs.
SciCode-Verified grading
Local grading remains the default. Select --benchmark scicode_verified/configs/opensandbox
to run scientific grading remotely. Configure OPENSANDBOX_DOMAIN,
OPENSANDBOX_API_KEY, and SCICODE_SANDBOX_IMAGE with an immutable image digest.
The overlay
expects /opt/scicode/2024/bin/python, /opt/scicode/2025/bin/python, and the corrected
HDF5 at /data/test_data_cleaned.h5; see the
image instructions.
scicode_verified_resources_server.resources_servers.scicode.num_processes
limits concurrent grading attempts, including creation and cleanup (64 by default for this remote overlay).
Each attempt requests one CPU and 8 GiB; adjust concurrency and
sandbox.memory_mib to your service quota and measured workload. Model generation
concurrency is independent. Infrastructure failures propagate without being scored
as incorrect answers; ordinary assertions and code timeouts retain the existing
grading semantics. Failed rows need investigation before retrying.
Setup
Obtain service access
To use Gym with OpenSandbox, connect to a running deployment and obtain its
service key. Set that key as OPENSANDBOX_API_KEY.
Choose an access route before running an agent:
- An existing organization-managed deployment: request access from the team operating that OpenSandbox service, using its approved internal access runbook.
- Your own deployment: follow the upstream
OpenSandbox server deployment guide
and server configuration reference.
The operator configures a non-empty
api_keyunder the server’s[server]section and securely supplies that value to clients asOPENSANDBOX_API_KEY. Deploy and secure the server before connecting Gym.
Configure the client
Install the sandbox extra in the runtime image or virtual environment that creates sandboxes:
For package installs, use:
Set connection values through the provider config. The shipped config reads these environment variables:
Sandbox, model, and image-registry credentials are separate. Keep secrets out of source control, issues, and resolved-config logs.
Multi-cell front ends
A deployment can put several OpenSandbox servers (“cells”) behind one front end,
such as a cell router. Point OPENSANDBOX_DOMAIN at the front end; no other Gym
setting changes:
- Sandbox IDs carry the owning cell as a prefix (for example
c5-<uuid>). Treat them as opaque strings; later calls for that sandbox are routed by the prefix. - Keep
connection.use_server_proxy: true(the default). Endpoint lookups then return the owning cell’s server-proxy URL, so commands, file transfers, and PTY traffic go from Gym straight to the cell. The host that runs Gym must be able to reach every cell’s endpoint, not only the front end. - The cell’s server proxy authenticates these direct calls with the same API key.
The provider sends
OPENSANDBOX_API_KEYas a header on them. If you call the OpenSandbox SDK directly, pass the key inConnectionConfig.headersas well asapi_key:api_keyalone covers only lifecycle calls, and execd requests get401. - Compose collections stay on one cell; see Compose networking.
Check access before allocating sandboxes
From the Gym server’s host or container:
- Confirm the endpoint resolves and is reachable. The shipped
.svc.cluster.localaddress normally requires cluster network/DNS access. Use the operator’s TLS/CA configuration. - Use the public
/healthroute to check reachability, then the deployment’s read-only authenticated check to verify credentials. - Confirm image-registry and required download access, plus model connectivity
from the component making requests. Inside a sandbox,
localhostrefers to the sandbox itself. - Agree initial concurrency, lifetime, and cleanup within the deployment’s limits.
Resolve access, quota, or image-pull failures with the operator before retrying. After these checks, run an agent evaluation to verify the complete workflow.
Provider Config
Load the shipped OpenSandbox config
and select it with sandbox_provider: sandbox. This excerpt shows selected
defaults; load the full file using the launch command below.
Keep create.skip_health_check: false to avoid commands racing sandbox readiness.
Three create settings work against the task’s ready_timeout_s:
create.request_timeout_s(elseconnection.request_timeout_s) is capped atready_timeout_sfor the create request, which blocks while the pod is allocated, so a pod the cluster cannot place fails within one ready timeout. When neither is configured the SDK’s own default (30 s) applies and stays on the handle.create.timeout_sbounds one create attempt as configured: the request plus the SDK’s readiness wait, which starts only once the request returns. Unset, it defaults to the capped request timeout plusready_timeout_splus 30 s. A configured value below that sum is respected and logged once.create.fail_fast_on_pending(defaultfalse): a create whose sandbox is stillPendingat the timeout is deleted and retried like any other timeout, with the pending reason in the log, becausePendingalso covers a full cluster and an image still pulling. Set it totrueto report it once instead, when every spec is known to be placeable and a pod stillPendingat the ready timeout means a request above a limit.
A resource_requests entry above the sandbox’s limit is clamped to the limit with a warning; a spec that
declares no resources is held to the SDK’s default limits (1 CPU, 2048 MiB), since those are what the
server applies.
When a create attempt times out or the caller cancels it, the provider finds the sandbox the attempt
left behind by the attempt’s metadata key, deletes it and cancels the create request. If it finds none
(the server had not recorded the sandbox yet, or the lookup failed), it lets the request finish and
deletes the sandbox the server’s answer names. aclose() waits up to operations.close_timeout_s for
those requests.
Set create.renew_ttl_s and create.renew_interval_s to keep every sandbox on a lease
(renew_ttl_s: 1200, renew_interval_s: 300, renew_timeout_s: 30). The lease is off in the shipped config.
A config that checkpoints sandboxes should set it: a server that crashes leaves its sandboxes running, a
restore replaces them with new ones, and without a lease the old ones run until the spec’s ttl_s. Each sandbox is
then created with renew_ttl_s, or the spec’s ttl_s when that is shorter, and the process that created it
renews it every renew_interval_s (jittered by up to 10% earlier), with renew_timeout_s per call, but never
past the spec’s ttl_s from creation. The lease runs from the create request: the service starts it when it
receives the request and answers once the sandbox is ready. It must therefore be longer than the time the
request stays open (placement and the image pull, which a sandbox created from a snapshot repeats) plus the
provider-side setup that follows (the reconnect, the readiness probe and setup.commands); the first renewal
comes that much sooner, and at once if more than one interval has already passed. A create that takes longer
than the whole lease loses its sandbox. Stopping a sandbox stops its renewal, and so does a crash of the process that
created it: the service then deletes the sandbox within one renew_ttl_s instead of at the end of the spec’s
lifetime. A failed renewal is logged and tried again within what is left of the lease, at most one interval
later and sooner as the expiry nears, with jitter so that sandboxes created together do not retry in step. A
sandbox survives an outage if the API answers again at least one
renew_timeout_s plus a few seconds before its lease runs out, so live sandboxes expire once the OpenSandbox API
is unreachable for longer than about renew_ttl_s - renew_interval_s - renew_timeout_s. Without renew_ttl_s,
renew_interval_s alone refreshes the spec’s whole ttl_s at every interval, and a renewal that fails fails
the sandbox’s later operations.
The snapshots of partial-rollout checkpoints are deleted only when the controller releases a checkpoint it no longer restores. The OpenSandbox service does not expire snapshots, and deleting the source sandbox leaves them. So what a job leaves behind depends on its controller:
- A controller that releases as it goes leaves the snapshots of the checkpoints it had not released when the job ended.
- A controller that never releases leaves every snapshot of every checkpoint: one per live sandbox at each commit. The server logs a warning once its index holds more than a few checkpoints.
Delete them by the name prefix of the job’s checkpoint.state_dir, whose id is in
<checkpoint.state_dir>/state_dir_id (this is not the run_id of a rollouts manifest):
A checkpoint.state_dir belongs to one job and its restarts. Two jobs that run at the same time must not share
one, for example by launching two experiments from a config that sets the same absolute path: they would share
the name prefix above and the record of which snapshots belong to which checkpoint, and each job’s release would
delete the other’s snapshots. Put the job’s id in the path. The directory must also be on a filesystem whose
flock holds across every node that runs the server: on Lustre, mount with -o flock, since localflock only
holds within one node.
A release lists every snapshot on the service to find the ones a crashed process left unrecorded, because the
service’s name filter matches whole names only. operations.snapshot_list_timeout_s bounds that listing and
operations.snapshot_delete_timeout_s each delete, retries included; one that runs out fails the release, and
the next release tries again.
Run an agent with the provider config by passing it alongside the agent and model configs:
The provider constructor accepts these optional config sections:
Failure reporting and recovery
Creation, readiness, lifecycle and runtime failures use NgFailureEvent(boundary="sandbox"); see Error Handling.
Status and log reads retry through operations.outcome_policy, whose HTTP rules take precedence over kind rules.
Partial overrides merge with defaults. With the default aiohttp transport, provider retries honor an observed HTTP 429 Retry-After header;
bounded recovery waits stop at their existing deadline. Supported actions are retry and fail; budgets and replay safety remain
execution constraints. Command submissions are single-attempt. command_retries remains accepted for config
compatibility, but neither it nor a retry rule enables command replay. Transport-level 429 retries are disabled for OpenSandbox so they
cannot bypass this policy. Lifecycle SDK operations use the same event/reporting path.
Creation and reconnect recovery use create.outcome_policy, with retry defaults for transport failures,
SDK internal failures, create deadlines, Pending observations and failed readiness probes. Setup-command failures
and unclassified requests fail by default. HTTP rules take precedence over kind rules. Error text alone never
selects a retry. create.fail_fast_on_pending still prevents another Pending attempt when enabled.
Policy evaluates the current event’s HTTP status, then its kind, then the default action. It does not search
causes. A lifecycle observation wrapping a failed HTTP 410 request has no HTTP status of its own; the cause
retains 410. Likewise, an expired readiness probe reports provider_not_ready above those events. The default
create policy retries that readiness failure within create.retries, even with by_status_code: {410: fail}.
To stop that create path, configure create.outcome_policy.by_failure_kind: {provider_not_ready: fail}.
Failed create attempts and recovered PTY requests are retained in the request failure report.
The status-recovery window covers reads of the same execution ID and is capped by the existing command budget; diagnostic calls cannot restart it. Failed attempts enter the request’s failure report before retry, including when a later read succeeds. Collection persists that report through the shared failure-reporting path. An ambiguous outcome is not proof of OOM or model fault, and this provider does not assign benchmark scores.
Closing a sandbox
Sandbox deletion is idempotent. If a DELETE reply is lost, the provider can retry
within operations.retries; a subsequent not-found response confirms that the
sandbox is gone. All DELETE attempts and retry delays share one
operations.close_timeout_s deadline. Permission failures and caller cancellation
remain terminal. If termination fails or its deadline expires, the provider keeps
the SDK handle available for the owner’s retry and does not close its transport.
This recovery does not change CREATE or command retry policies.
After confirmed deletion, local SDK transport cleanup has its own existing
close_timeout_s bound. Allow both phases, renewal shutdown and other workload
cleanup inside the caller’s outer deadline: a 12-second close setting allows up
to 12 seconds for DELETE and another 12 seconds for transport cleanup, before
other cleanup work. Increasing retries never extends the DELETE deadline.
Image policy and setup commands
Two blocks belong to the deployment, not to any task or server. images.rewrites is an ordered list of
{from, to} prefix rules applied to every image a sandbox asks for, for a mirror or a derived image that adds
what the platform needs. images.auth maps an image prefix to {username, password}; the longest matching
prefix is sent with the pull, unless the spec carries its own image_auth. setup.commands run as root in
every new sandbox after it is ready and before it is handed out, within setup.timeout_s each; a failing
command fails the create and is not retried. A sandbox whose image runs as a non-root user on a cell that drops
CAP_SETUID refuses the switch to root before the command runs; the command then runs as the image’s own user
(logged for each such command), so one written to no-op where it does not apply still does, and one that needs root
there fails the create on its own result. setup.env is set in every sandbox under the spec’s own
variables, for platform knobs such as the exec daemon’s grace period. Keep task-specific settings out of these
blocks: they apply to everything the provider creates. That includes Compose sidecars and the provider’s own
helper containers (an Alpine image with neither Python nor apt-get), so a command must no-op where it does
not apply, as the example below does.
Task folders
When the harbor Environment Server runs task folders on OpenSandbox:
- Prepare reads the configuration of private images with the
sandbox:block’simages.auth(never thegpu_sandbox_providerblock’s), at the reference the task names. The provider matches each prefix against the reference it pulls, afterimages.rewrites, so an image a rewrite moves between private registries needs an entry for each. A declareddocker_imageis pulled as written; a Dockerfile’sFROMimage and every Compose service image are pulled at their recorded digest, which spells a Docker Hub image withoutdocker.io/. So key a Docker Hub image asmyorg/(prepare also acceptsdocker.io/myorg/for its own reads), and an image on another registry by its host prefix. Outside a Compose task, a declareddocker_imagewrittendocker.io/myorg/imgalso needs adocker.io/myorg/entry. - Images built from a task’s Dockerfile must be pushed to a registry, so set
builds.repository(see Harbor image builds). The build tool pushes with its own login, but the provider pulls withimages.auth, so a private repository needs an entry there too. - Compose tasks need
networking.enabled: true, andnetworking.loopback_forwarding: truewhen a service usesnetwork_mode: service:<name>. Named volumes needshared_storage. See Run on OpenSandbox for every setting a Compose task needs.
SandboxSpec Provider Options
Set OpenSandbox-specific create options under SandboxSpec.provider_options, or under sandbox_spec.provider_options in an agent config:
Supported options are:
Unknown provider option keys raise ValueError; values with the wrong type raise TypeError. This catches config drift before a sandbox allocation is attempted.
Network Policy
provider_options.network_policy is the sandbox’s egress policy: a defaultAction (allow or deny) and ordered egress rules of {action, target}, where a target is a host name, a *.domain wildcard or an IP literal. restricted_network_policy in nemo_gym.utils.network_policy (also importable from nemo_gym.agents) builds one from endpoint URLs. A harness that creates its own sandbox applies its network_access setting with restrict_sandbox_network in nemo_gym.agents.utils.sandbox_network, which builds that policy for the model and tool endpoints (see Restrict the egress of an owned sandbox). A resources server that lends a sandbox with LentSandboxes passes network_access="model_only" and model_server, and each lent sandbox then reaches only that model server’s host. With network_access="model_and_tools" and tool_servers (the servers whose tools the harness calls over MCP, such as the resources server itself), it also reaches their hosts; the Harbor environment server’s phase_network_policy (environment_servers/harbor/network_policy.py) builds one from a Harbor phase’s network_mode and allowed_hosts.
What the control plane does with it, verified live with the OpenSandbox 0.1.15 SDK against an OpenSandbox deployment. The probes read neither the control plane’s own version nor its egress enforcement mode, dns or dns+nft, which is a server setting (see OpenSandbox Best Practices). Confirm that setting separately; a connection timeout alone does not identify the enforcement layer:
- A sandbox created without a
network_policyhas no egress sidecar: it is unrestricted, and its policy can neither be read nor changed afterwards (the egress endpoints answer 502). - A sandbox created with a policy keeps its
defaultActionfor life. The SDK’spatch_egress_rulesanddelete_egress_ruleschange the rules of a running sandbox and the change is enforced within seconds, but no call changesdefaultAction, and*is not a catch-all target: adeny *rule underdefaultAction: allowblocks nothing, and anallow *rule underdenyopens nothing. - Under
defaultAction: deny, name resolution failed and a connection to a raw IP timed out on the deployment probed. The probes did not establish which layer caused the timeout:dnsfilters name resolution only, so a hard-coded IP bypasses the policy and ano-networkphase is weaker there, whiledns+nftadditionally enforces at the IP layer; confirm which mode your cluster runs before relying on it. Anallowrule for a host name, a*.domainwildcard or an IP literal opened exactly that target.
A public sandbox therefore cannot be switched to a restricted policy once it runs. A workload with a restricted phase creates that phase’s sandbox restricted: the harbor Environment Server applies a task’s [verifier].network_mode to the separate verifier’s own sandbox at creation (see Shared and separate verification), which restricts that sandbox for its whole life, image entrypoint and artifact staging included, where Harbor restricts only the verify call itself; and a harness that needs a restricted agent sandbox passes the policy at create as the sandboxed agents do.
Relevant SandboxSpec Fields
Resource Mapping
SandboxResources is translated into OpenSandbox resource quantities:
The provider also normalizes metadata values for backend labels by replacing unsupported characters and truncating values to the provider limit.
Job Attribution
The provider merges nemo-gym.nvidia.com/team, nemo-gym.nvidia.com/user,
nemo-gym.nvidia.com/workload, and nemo-gym.nvidia.com/run keys into each sandbox’s
metadata when they can be resolved. OpenSandbox propagates sandbox metadata as Kubernetes
labels on the sandbox resources, so running sandboxes are attributable both through the
OpenSandbox list API metadata filter and directly at the cluster level:
Each field resolves in order: the provider’s attribution config, then NEMO_GYM_TEAM /
NEMO_GYM_USER / NEMO_GYM_WORKLOAD environment variables, then Slurm job environment
variables (SLURM_JOB_ACCOUNT / SLURM_JOB_USER / SLURM_JOB_NAME). user additionally
falls back to the OS login name (root is ignored — containers run as root by default, so
it would attribute the image, not a person), and workload to NEMO_GYM_CONFIG_PATH, the
server instance name the gym CLI sets on every server process it spawns. Fields that cannot
be resolved are omitted. Explicit sandbox_spec.metadata and default_metadata keys always
take precedence over attribution keys.
run identifies one launch of the creating process: NEMO_GYM_RUN_ID if set, else an id
generated per process. The resolved attribution is logged once at the first sandbox create —
note the run value from the logs to list or clean up exactly that run’s sandboxes (for
example after killing a run partway through), which team / user / workload cannot do
when the same user launches the same workload twice:
Attribution in Kubernetes deployments
When the gym servers themselves run in Kubernetes (for example a driver pod launching an
eval), none of the automatic sources resolve: there are no Slurm variables, and the
container’s OS login is typically root, which is ignored. Set the NEMO_GYM_* variables
on the pod spec — either directly or via the downward API from labels the pod already
carries:
Lifecycle
The provider creates one OpenSandbox sandbox per Gym sandbox:
If create.skip_health_check is enabled, the provider reconnects to the sandbox before running the NeMo Gym readiness probe so follow-up operations use a fresh SDK handle.
File Transfer
The provider uses the OpenSandbox SDK file API for uploads and downloads:
upload()reads the local file and writes bytes to the target sandbox path.download()reads bytes from the sandbox path and writes them to the local target path.
Startup files from SandboxSpec.files use the same provider file path semantics before the first command runs.
User and Runtime Notes
The neutral user argument to exec() maps onto OpenSandbox command options:
Commands with a reported exit status return SandboxExecResult. A missing or contradictory final outcome raises NgFailureEvent with provider_command_outcome_unknown. A signal death near the configured deadline does not establish that the command timer fired, so its reported exit code is preserved. Gym observes its own deadline directly. For session commands it records provider_command_timeout, performs cleanup and returns the existing timeout result. For background commands (operations.background_exec: true) it records provider_command_timeout, interrupts a command still running at the deadline and returns code 124 with error_type="timeout"; execd’s own timer is set command_timeout_headroom_s / 2 later and only stops a command whose interrupt was refused. A client request timeout instead reports provider_io_timeout; it does not establish that the remote command stopped.
If background status reads keep reporting running=True until Gym’s overall client wait expires, Gym reports
provider_command_outcome_unknown and retains that last Running observation in the reason. Failed status reads
that exhaust the recovery budget instead report provider_status_unavailable, with the failed read as cause.
Neither event proves that the remote command stopped.
A reported signal exit retains error_type=None; it is not automatically excluded as a timeout. In AnySWE,
a patch from such an agent execution can still be verified. A signal exit from a grading command proceeds to
the benchmark’s log parser and can produce a counted zero if the logs do not establish success. Consumers must
make grading and inclusion decisions explicitly from available evidence; the provider does not infer them
from elapsed time or a signal code.
The provider’s create.image_pull_policy defaults to IfNotPresent. Valid values are Always, IfNotPresent, and Never. The resolved policy is written into OpenSandbox create extensions as both imagePullPolicy and opensandbox.extensions.image-pull-policy unless those keys are already present in provider_options.extensions.
Create retries handle eligible allocation, connection and server-side failures through create.outcome_policy.
Command submissions remain single-attempt, including when operations.command_retries is nonzero.
execd, the exec daemon inside each sandbox, holds every command response open for EXECD_API_GRACE_SHUTDOWN (default 1s) after the command’s last event, which is fixed latency on each command and on the readiness probe. Set it in the sandbox spec’s env to shorten it; the shipped SWE-bench and OpenCode configs use 50ms, and the provider’s operations.background_poll_initial_s (0.5) is sized to that.
connection.tls_verify (default false) controls certificate verification on every connection the provider opens, both the SDK transport and PTY WebSockets. The default skips verification so that https:// endpoints with a certificate the client cannot verify, such as a validation cluster behind a self-signed load balancer, work without extra setup; set tls_verify: true for endpoints with a verifiable certificate. For https endpoints, either set connection.protocol: https or include the scheme in the domain (connection.domain: https://sandbox.example, or OPENSANDBOX_DOMAIN=https://sandbox.example): a scheme in the domain sets the protocol and takes precedence over connection.protocol. The cleanup_sandboxes.py helper mirrors the setting with --tls-verify and reads tls_verify from --connection-config.
Setting connection.keepalive_expiry_s: null preserves the configured tls_verify behavior. With transport_backend: httpx, tls_verify: true and disable_connection_pooling: false, it uses the SDK’s default transport. SDK 0.1.15 drops command-status HTTP 410 response bodies on that path. Every transport the provider supplies, including the default aiohttp one, retains those bodies up to 64 KiB for diagnostics. OOM confirmation comes from control-plane lifecycle evidence independently of the command response body.