nemo_gym.sandbox.providers.opensandbox.provider
nemo_gym.sandbox.providers.opensandbox.provider
OpenSandbox provider implementation.
Module Contents
Classes
Functions
Data
DEFAULT_ATTRIBUTION_KEY_PREFIX
IMAGE_PULL_POLICY_ANNOTATION_EXTENSION_KEY
IMAGE_PULL_POLICY_EXTENSION_KEY
API
Job attribution merged into every sandbox’s metadata (Kubernetes labels on the sandbox).
OpenSandbox propagates sandbox metadata as Kubernetes labels on the sandbox resources, so
attribution is queryable both through the OpenSandbox list API and at the cluster level
(e.g. kubectl get pods -l nemo-gym.nvidia.com/team=my-team). key_prefix namespaces
the label keys (Kubernetes prefixed-key convention); set it to "" for bare
team / user / workload / run keys.
Unset fields are auto-detected: NEMO_GYM_TEAM / NEMO_GYM_USER / NEMO_GYM_WORKLOAD
environment variables first, then Slurm job env vars (SLURM_JOB_ACCOUNT /
SLURM_JOB_USER / SLURM_JOB_NAME), then the OS login name for user (root is
ignored) and the gym CLI’s NEMO_GYM_CONFIG_PATH server instance name for workload.
Fields that cannot be resolved are omitted. run scopes sandboxes to one launch of the
creating process (NEMO_GYM_RUN_ID, else generated per process and logged) so a run’s
sandboxes can be listed and cleaned up exactly. Explicit SandboxSpec.metadata keys
always take precedence over attribution keys.
OpenSandbox server connection settings.
With the legacy httpx backend, keepalive_expiry_s must stay below the
server’s own keep-alive idle timeout (uvicorn defaults to 5s), or sockets are reused after the
server has closed them; null falls back to the SDK’s default transport only
when certificate verification is enabled and pooling is not disabled.
transport_backend=aiohttp uses Gym’s global client and connector limits;
provider-local pooling settings apply only to transport_backend=httpx.
tls_verify applies to every connection the provider opens (SDK transport
and PTY sockets) and is off by
default; set it for endpoints whose certificate the client can verify.
domain may carry its scheme (https://sandbox.example). The scheme is
moved into protocol and takes precedence over a configured protocol,
so every URL the provider builds itself (the PTY WebSocket target) agrees
with the SDK’s base URL; the SDK receives the host plus any path prefix
(sandbox.example:8080/prefix). The SDK would accept a scheme in
domain on its own, but the provider reads protocol directly, hence
the normalization here. Only scheme://host[:port][/path-prefix] is
accepted: a query string or fragment is a configuration error.
OpenSandbox create/reconnect retry settings.
Bases: SandboxCreateError
Raised when OpenSandbox cannot create a sandbox.
Bases: OpenSandboxCreateError
Raised when OpenSandbox sandbox creation exceeds the client timeout.
Bases: SandboxCreateVerificationError
Raised when a newly-created sandbox cannot execute a probe command.
Operator assertion that direct sandbox IPs are mutually reachable.
Retry and timeout settings for SDK operations after create.
Post-create probe settings.
Provider backed by the OpenSandbox SDK/server API.
Use Gym’s global HTTP pool, or the explicitly selected legacy backend.
Reconnect after SDK create so follow-up calls use a fresh SDK handle.
Create a sandbox through opensandbox.Sandbox.create.
Run a command inside an OpenSandbox sandbox.
Run a command as a background execution polled via short requests.
The logs endpoint returns one combined stream, so unlike the foreground
path stdout carries both streams and stderr is set only when the
sandbox itself reports an error.
Return the provider-owned shared transport, building it on first use.
Briefly poll the sandbox status; describe an OOM kill, else None.
With any_death every terminal state is reported, not just an OOM
kill — for callers that need to know whether the sandbox is gone at
all, not specifically why. When the backend stops answering it usually
takes the control plane a moment to record why, so poll for up to 5s
before giving up.
Return the aiohttp client for one PTY session (same tls_verify as the SDK transport).
True only when execd itself reports the PTY session does not exist.
A proxy 404 (route not registered yet) lacks execd’s error code, and a failed check is treated as unknown so the attach proceeds as before.
Resolve the sandbox’s execd base URL, headers and request timeout.
Read one file from an OpenSandbox sandbox.
Add the configured default image pull policy to SDK create extensions.
Release sessions that ended on their own; their aiohttp client is
only freed by close(). Called from create/attach so the tracking
set cannot grow without bound.
Retry backend-connect 502s that command_retries deliberately skips.
A proxy 502 is a TCP-connect failure: the command never reached execd, so
retrying under operations.retries cannot double-run it (unlike a real
command failure). When that budget is exhausted the backend is dead, so
raise a typed error and fail fast instead of retrying for hours.
Write one file into an OpenSandbox sandbox.
Close provider-owned resources.
Re-attach to an existing execd PTY session by id.
Terminate the sandbox and close local SDK resources.
Probe capabilities and verify the shared memory allocated at creation.
Rebuild a live handle from an OpenSandbox sandbox id via the SDK.
Running sandboxes are health-checked unless the caller opts out. A paused sandbox has no exec daemon to check; resume rebuilds its endpoints and performs the health check instead.
Create one sandbox through the configured OpenSandbox path.
Job attribution keys (team / user / workload / run) are merged into the
spec’s metadata (explicit spec keys win) so every sandbox is attributable via its labels.
Open an interactive execd PTY session inside a sandbox.
Download one file from an OpenSandbox sandbox.
Resolve one client-reachable direct or server-proxied service URL.
Run a command inside an OpenSandbox sandbox.
Preserve background services that redirect their stdout and stderr.
Run loopback listeners in a foreground command owned by the collection.
Resolve a direct container IP, independently of client proxy mode.
Pause a sandbox and wait until it reports paused.
Local PTY clients are detached first, while execd can still answer the close handshake; server sessions are never deleted, so they remain attachable if the pause request fails. After resume, the Kubernetes backend has replaced the runtime (open a new PTY); the Docker backend thawed it (re-attach by id).
Resume a paused sandbox and rebuild its SDK clients and endpoints.
One pause_resume_timeout_s deadline covers the request, endpoint
rebuild and readiness check. On timeout the server-side state is
unknown: reconnect and check status() before retrying. See
pause() for what happens to PTY sessions.
Return a descriptor for reattaching to this sandbox by id.
OpenSandbox sandboxes are reachable by id from any process that has the
connection config, so the id alone is enough to reconnect and no sandbox
server is needed to share one. scope is ignored: OpenSandbox has no
lease concept of its own.
Append validated peer aliases to the sandbox hosts file.
Return placement metadata required by the configured shared storage.
Mount a project directory, or the shared root for bootstrap when source is None.
Return the current OpenSandbox lifecycle status.
Upload one local file into an OpenSandbox sandbox.
Require explicit deployment support for inter-sandbox networking.
Require explicit deployment support for loopback TCP listeners.
Reject requirements without an operator-configured implementation.
Recognized per-sandbox create options read from SandboxSpec.provider_options.
image_auth, network_policy, platform, and volumes entries are passed through to the
OpenSandbox SDK, so their inner fields are validated by the SDK rather than here.
Operator-supplied capability probes and create-time runtime metadata.
Shared host mount and optional deployment-specific placement metadata.
Bases: RuntimeError
Raised when the server proxy cannot open a TCP connection to a sandbox’s exec daemon.
A submission 502 means the command never started. A status or log polling 502 can mean the backend died while the command was running.
Bases: Filter
Hide the SDK warning emitted when terminate reaches its desired state.
Match kill errors meaning the sandbox is already gone (terminate’s goal state).
Only the terminate path may treat this as success; other operations must keep failing loudly on not-found.
Return whether a sandbox create failure is likely transient.
Return whether an SDK operation can be retried.
Split a configured OpenSandbox domain into (domain, scheme).
"https://sandbox.example:8080/prefix/" -> ("sandbox.example:8080/prefix", "https");
a bare host is returned unchanged with scheme None. Surrounding whitespace
is stripped either way. Only scheme://host[:port][/path-prefix] is accepted:
other schemes, a missing host, or a query string / fragment raise ValueError.
Validate a Kubernetes-compatible container image pull policy.