Adding a Sandbox Provider
Add a provider when NeMo Gym needs to create sandboxes through a new runtime backend, such as a container service, HPC isolation layer, or in-house execution platform. The public AsyncSandbox and Sandbox facades stay the same; the provider owns runtime-specific create, command, file transfer, status, and cleanup behavior.
Provider Contract
Providers implement the SandboxProvider protocol from nemo_gym.sandbox. Keep common caller fields on SandboxSpec; put backend-specific options in SandboxSpec.provider_options.
New provider implementations must follow this lifecycle and failure contract:
- Return a
SandboxHandlefromcreate()only after the sandbox is ready enough to run commands and transfer files. - Return command status through
SandboxExecResultfor process exits, including nonzero exits. - For allocation, readiness and runtime failures, raise
NgFailureEventwithboundary="sandbox", the operation and a kind registered in failure_kinds.py. Each provider classifies its own SDK errors. The sharednemo_gym.sandbox.failures.sandbox_failurefactory builds the event from an explicitfailure_kind, observedhttp_statusand sandbox details; it does not interpret SDK exceptions or choose policy actions. Useraise ... from errorto retain the original cause. - A provider-request timeout leaves remote completion unknown. Report
provider_io_timeout; do not invent a command timeout. - Report
provider_oom_killedonly from authoritative current-sandbox evidence, retaining its source. - Choose runtime actions through configurable
OutcomePolicyConfigrules. Bound retries by the operation’s existing budget and never replay a command whose delivery is uncertain. Record failed attempts withobserve_failurebefore retrying. It records the attempt in the current request’s failure report without changing the result; see Failure observations. - Make
close()safe to call from cleanup paths. - Use
aclose()for provider-scoped resources such as SDK clients.
OpenSandbox implements this contract. See Required provider migrations for the built-in providers that must adopt it in follow-up changes.
Provider Config
Provider constructors receive the single provider-specific mapping from a named sandbox block:
resolve_provider_config("sandbox", global_config_dict) returns only the single provider mapping:
resolve_provider_metadata("sandbox", global_config_dict) returns the optional default_metadata block. Agents merge those values into SandboxSpec.metadata before provider create, with agent-level metadata taking precedence.
Each named block must contain exactly one non-reserved provider key. The reserved keys today are default_metadata and volume_sources (SANDBOX_BLOCK_RESERVED_KEYS in nemo_gym/sandbox/config.py); a provider never sees either.
Provider Options
Use SandboxSpec.provider_options for per-sandbox options that do not belong in the neutral schema. Validate that mapping before allocating runtime resources so bad configs fail early.
Then resolve it inside create():
This keeps common fields such as image, env, files, metadata, and resources portable while still giving a provider room for runtime-specific behavior.
Registry
The registry in nemo_gym.sandbox.providers.registry maps provider names from config to provider classes. Lookup order is:
- Explicit in-process registration with
register_provider(). - Built-in providers shipped with NeMo Gym.
- Installed Python entry points in the
nemo_gym.sandbox_providersgroup.
External packages and tests can register a provider directly:
In-tree built-in providers should use a lazy loader in registry.py so importing nemo_gym.sandbox does not eagerly import optional provider dependencies.
Out-of-tree packages can publish a provider entry point in their pyproject.toml:
Two entry points with the same provider name raise an error because selection would be nondeterministic. An entry point shadowed by a registered provider or a built-in provider is warned and ignored.
After registration, callers select the provider with a single-key provider config:
Provider Pages
Each provider page should use the same shape so users can compare backends quickly:
- Setup and optional dependencies
- Named provider config block
provider_optionsaccepted bySandboxSpec- Resource mapping and isolation properties
- Minimal
gym env startor local first-run example - Provider-specific operational notes