Skip to navigation

Sandbox API

View as MarkdownOpen in Claude

The nemo_gym.sandbox module is the provider-neutral interface for creating isolated execution environments, running commands, and moving files in or out of those environments. Agents and resources servers call the same API while provider pages document backend-specific setup, configuration, and isolation properties.

Import caller-facing APIs from the public package boundary:

from nemo_gym.sandbox import AsyncSandbox, Sandbox, SandboxResources, SandboxSpec

Treat nemo_gym.sandbox as the stable caller-facing API. Provider modules under nemo_gym.sandbox.providers are implementation details unless you are adding or configuring a provider.

Install Provider Dependencies

The public API is part of nemo-gym. Runtime backends can have optional dependencies, so install the extras or system packages required by the provider you configure in your agent or resources server.

Core Types

APIPurpose
AsyncSandboxAsync facade for FastAPI servers, async agents, and rollout code. Use this inside async code.
SandboxSync facade for synchronous harnesses. It owns a private event loop and rejects calls from an already-running async loop.
SandboxSpecProvider-neutral sandbox creation request. Includes image, TTL, working directory, files, metadata, resources, entrypoint, and provider options.
SandboxResourcesTyped resource request with CPU, memory, disk, and GPU fields.
SandboxExecResultCommand result with stdout, stderr, return_code, and optional error_type.
SandboxStatusProvider-neutral lifecycle status: starting, running, stopped, error, or unknown.
SandboxCreateErrorCreation exception retained for providers awaiting migration.
SandboxCreateVerificationErrorReadiness-check exception retained for providers awaiting migration.
SandboxPtyErrorPTY-session exception retained for providers awaiting migration.

First-Run Example

Create a small local script after your provider is available. Replace the provider name and settings with the backend configured for your environment.

import asyncio
from nemo_gym.sandbox import AsyncSandbox, SandboxResources, SandboxSpec
provider_config = {"apptainer": {}}
spec = SandboxSpec(
image="docker://python:3.13.14-slim",
ttl_s=1800,
ready_timeout_s=300,
workdir="/sandbox",
files={
"/sandbox/hello.py": "print('hello from sandbox')\n",
},
resources=SandboxResources(cpu=1, memory_mib=1024, disk_gib=5),
metadata={"example": "first-run"},
)
async def main() -> None:
async with AsyncSandbox(provider_config, spec) as sandbox:
await sandbox.start()
result = await sandbox.exec("python /sandbox/hello.py", timeout_s=60)
print(result.stdout or result.stderr)
raise SystemExit(result.return_code)
asyncio.run(main())

Run it from your local checkout or application environment:

python first_sandbox.py

exec() returns a SandboxExecResult. Nonzero process exits are reported in return_code; providers should reserve exceptions for sandbox runtime failures such as allocation, transport, or lifecycle errors.

Provider Config Blocks

Agents can refer to a named provider block instead of embedding provider credentials or backend settings in the agent config. The sandbox_provider field names a top-level config block, and that block contains exactly one provider key plus optional reserved keys:

sandbox:
default_metadata:
sandbox-api: opensandbox-sdk
opensandbox:
connection:
domain: ${oc.env:OPENSANDBOX_DOMAIN}
api_key: ${oc.env:OPENSANDBOX_API_KEY}
protocol: http

The agent then references that block by name:

mini_swe_agent_2:
responses_api_agents:
mini_swe_agent_2:
sandbox_provider: sandbox
sandbox_spec:
resources:
cpu: 2
memory_mib: 8192
provider_options:
platform:
os: linux
arch: amd64

Every provider config can bind the same name, such as sandbox, so swapping backends is swapping the provider config path passed to gym env start. If one run needs multiple sandboxes, give each block a distinct name and reference each one separately.

default_metadata is merged into SandboxSpec.metadata before create. The agent’s own sandbox_spec.metadata wins on key conflicts.

The helpers used by agents are public:

from nemo_gym.sandbox import resolve_provider_config, resolve_provider_metadata
provider_config = resolve_provider_config("sandbox", global_config_dict)
provider_metadata = resolve_provider_metadata("sandbox", global_config_dict)

Inline single-key mappings are also accepted when a caller does not use Hydra config:

provider_config = {"opensandbox": {"connection": {"domain": "sandbox.example"}}}

Lifecycle

AsyncSandbox and Sandbox are lifecycle objects. Construct one with a provider config and optional SandboxSpec, call start(), run commands or transfer files, then call stop(). Context managers close the provider on exit, but they do not start the sandbox automatically.

from nemo_gym.sandbox import AsyncSandbox, SandboxResources, SandboxSpec
spec = SandboxSpec(
image="ghcr.io/example/eval-image:py312",
ttl_s=18000,
ready_timeout_s=1200,
workdir="/workspace",
resources=SandboxResources(cpu=2, memory_mib=8192, disk_gib=20),
metadata={"benchmark": "my-benchmark", "task_id": "task-001"},
)
async with AsyncSandbox(provider_config, spec) as sandbox:
await sandbox.start()
result = await sandbox.exec(
"python -m pytest -q",
timeout_s=600,
user="root",
)
passed = result.return_code == 0

Sync vs. Async

Use AsyncSandbox inside FastAPI handlers, async resources servers, async agents, and rollout collection code.

Use Sandbox only in synchronous code, such as a third-party harness adapter that does not expose async hooks.

from nemo_gym.sandbox import Sandbox, SandboxSpec
with Sandbox(provider_config, SandboxSpec(image="ghcr.io/example/eval-image:py312")) as sandbox:
sandbox.start()
result = sandbox.exec("python --version", timeout_s=30)
output = "\n".join(part for part in (result.stdout, result.stderr) if part)

Do not call Sandbox from FastAPI handlers, async resources servers, or async agents. It blocks the caller by design. Use AsyncSandbox in async code.

Interactive Terminals

AsyncSandbox.pty opens interactive terminal sessions on providers that implement SupportsSandboxPty: OpenSandbox (execd PTY sessions, which also re-attach by id from another process) and Docker (docker exec -i -t under a host pseudo-terminal; sessions live in the calling process). await sandbox.pty.create(rows=..., cols=...) returns a session with read, write, resize, send_signal, wait_exit and close; await sandbox.pty.exec(command) runs one command in a terminal and collects its output; pty=False gives a pipe session with stdout and stderr split. Providers without the capability raise NotImplementedError, so use exec() there. The sync Sandbox facade does not mirror the PTY namespace.

SandboxSpec Fields

SandboxSpec is intentionally provider-neutral. Providers map these fields onto their own runtime primitives.

FieldDescription
imageContainer image to create.
ttl_sSandbox lifetime in seconds, when supported by the provider.
ready_timeout_sTime to wait for sandbox readiness.
workdirDefault working directory for exec() calls.
envEnvironment variables injected into the sandbox. Forward only values required by the task.
filesFiles to upload at startup, keyed by remote target path. A str value is written as UTF-8 text and a bytes value is written unchanged.
metadataString metadata for tracing, debugging, and backend labels. Providers may normalize values for their runtime.
resourcesSandboxResources or a mapping with cpu, memory_mib, disk_gib, gpu, and gpu_type.
entrypointOptional container entrypoint override.
provider_optionsProvider-specific options that do not fit the common schema.

You can pass resources as either a SandboxResources instance or a mapping:

spec = SandboxSpec(
image="ghcr.io/example/eval-image:py312",
resources={
"cpu": 2,
"memory_mib": 8192,
"disk_gib": 20,
},
)

Unknown resource keys raise a ValueError, which catches config drift early.

Startup Files and File Transfer

Use files for small files that should exist before the first command runs. A str value is written as UTF-8 text, and a bytes value, such as an image, is written unchanged:

from pathlib import Path
spec = SandboxSpec(
image="ghcr.io/example/eval-image:py312",
workdir="/workspace",
files={
"/workspace/input.txt": "hello\n",
"/workspace/figure.png": Path("figure.png").read_bytes(),
},
)

Use upload() and download() for local files:

await sandbox.upload(local_path, "/workspace/archive.tar.gz")
await sandbox.download("/workspace/log.txt", output_path)

upload() and download() operate on files. If you need structured values, serialize them locally before uploading and parse the downloaded file locally after the sandbox command completes.

To move in-memory content into or out of a sandbox that is already running, use upload_bytes() and download_bytes(), or upload_text() and read_text() for UTF-8 text, from nemo_gym.sandbox. They go through upload() and download(), so the content never passes through a shell:

from nemo_gym.sandbox import download_bytes, upload_bytes, upload_text
await upload_bytes(sandbox, path="/workspace/figure.png", data=image_bytes)
await upload_text(sandbox, path="/workspace/notes.md", text="# Notes\n")
plot = await download_bytes(sandbox, path="/workspace/plot.png")

For media in a prompt, nemo_gym.core.split_text_and_media decodes a message’s content into its text and its media files. stage_media() writes each file to /tmp/nemo-gym-media/<sha256>.<extension>, named by its content, and returns the sandbox paths to pass to a harness, in order. Every rollout of a task then gets the same paths, and so the same prompt. The sandbox’s default user creates the directory, and each file is uploaded under a unique name, readable by every user, then renamed onto its path. Sessions that share a sandbox can stage the same image at once and never read a partly written file. Sessions do not delete staged files, because another session may be reading them. A sandbox with its own /tmp drops them when it is destroyed. The local provider, and Apptainer without --contain, use the host’s /tmp, where the files stay until the host clears it. stage_media() raises SandboxUnavailableError when the provider cannot run a staging command, such as on a timeout, and SandboxTransferError when a command fails inside the sandbox.

from nemo_gym.core import split_text_and_media
from nemo_gym.sandbox import stage_media
text, media = split_text_and_media(message.content)
paths = await stage_media(sandbox, media)

Status and Cleanup

Call status() when a runner needs to distinguish a stopped sandbox from a provider error:

status = await sandbox.status()
if status.value == "error":
...

Always stop sandboxes in cleanup paths. stop() is idempotent on the public facade and closes provider-scoped resources after ending the sandbox lifecycle.

sandbox = AsyncSandbox(provider_config, spec)
try:
await sandbox.start()
result = await sandbox.exec("pytest -q", timeout_s=600)
finally:
await sandbox.stop()

Image Rewrites

Use rewrite_image() when a benchmark’s upstream image needs to run through an internal registry mirror.

from nemo_gym.sandbox import rewrite_image
image = rewrite_image(
"docker.io/library/python:3.13.14-slim",
[{"from": "docker.io/", "to": "mirror.example.com/dockerhub/"}],
)

Rewrites are ordered. The first matching from prefix wins.

Error Handling

Command process exits, including nonzero exits, return SandboxExecResult. For allocation, readiness, lifecycle and runtime failures, new and migrated providers must raise NgFailureEvent with boundary="sandbox". Its registered failure_kind, operation, observed HTTP status, typed details and explicit cause carry the evidence; callers apply their own outcome policy. A failure kind alone does not decide attribution, retries, health or a zero score.

The registered kinds and their meanings live in failure_kinds.py. The provider_* namespace covers sandboxes, browsers and containers; boundary="sandbox" identifies the API boundary for sandbox events.

A command exit code of 137, a signal kill near a deadline, an error message, or a sidecar’s OOM does not establish the command’s cause. Forward NgFailureEvent unchanged when only forwarding; when reporting a distinct failure at a caller’s boundary, create its event and use raise ... from error.

Local argument/configuration validation and task cancellation are separate from operational failure events. Do not translate cancellation into a sandbox failure. PTY read(timeout_s=...) and wait_exit(timeout_s=...) raise TimeoutError when the caller’s bounded wait expires: that timeout alone does not mean the PTY or command failed.

Resources servers and agents forward failure events to the owning task boundary for its outcome policy, preserving their causes. An infrastructure failure alone does not justify a zero reward. See Failure observations for recording, transport and saved reports, and policy by execution stage for shared OOM defaults.

Required provider migrations

OpenSandbox implements the failure-event contract above. The following built-in providers must migrate in follow-up changes:

ProviderConfig key
Apptainerapptainer
Daytonadaytona
Dockerdocker
E2Be2b
ECS Fargateecs_fargate
Enrootenroot
Locallocal
OpenShellopenshell

For each migration:

  • Cover allocation, readiness, lifecycle, command and file failures, plus PTY failures where supported. Include operational failures currently returned as error-valued results; keep ordinary command exits as SandboxExecResult.
  • Classify provider evidence into registered kinds and preserve explicit causes.
  • Record recovered attempts and use configurable outcome policies within existing retry budgets and replay constraints.
  • Update callers and tests together, retaining their retry and grading behavior.

A provider’s migration is complete when all its supported operational failure paths and their callers use the shared contract.

Until a provider migrates, callers must retain its existing exception handling. SandboxCreateError, SandboxCreateVerificationError and SandboxPtyError remain compatibility classes for those paths; they are not an alternative contract for new providers. Remove them only after their remaining providers and callers have migrated. The provider guide describes the required implementation.