> This page is for version 0.7.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # OpenSandbox Provider > Configure NeMo Gym sandboxes backed by an OpenSandbox server. The `opensandbox` provider creates sandboxes through the OpenSandbox SDK and server API. It is the Kubernetes-backed provider used by agentic software engineering environments such as `mini_swe_agent_2`. > **Tip** > > Running at scale? Read [OpenSandbox Best Practices](/infrastructure/sandbox/opensandbox-best-practices) > for resource requests vs limits, timeout and retry guidance, and the failure signatures that are not bugs. ## Setup ### Obtain service access To use Gym with OpenSandbox, connect to a running deployment and obtain its service key. Set that key as `OPENSANDBOX_API_KEY`. Choose an access route before running an agent: * **An existing organization-managed deployment:** request access from the team operating that OpenSandbox service, using its approved internal access runbook. * **Your own deployment:** follow the upstream [OpenSandbox server deployment guide](https://github.com/opensandbox-group/OpenSandbox/blob/main/docs/components/server.md) and [server configuration reference](https://github.com/opensandbox-group/OpenSandbox/blob/main/server/configuration.md). The operator configures a non-empty `api_key` under the server's `[server]` section and securely supplies that value to clients as `OPENSANDBOX_API_KEY`. Deploy and secure the server before connecting Gym. ### Configure the client Install the sandbox extra in the runtime image or virtual environment that creates sandboxes: ```bash uv sync --extra sandbox ``` For package installs, use: ```bash pip install "nemo-gym[sandbox]" ``` Set connection values through the provider config. The shipped config reads these environment variables: | Variable | Purpose | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OPENSANDBOX_DOMAIN` | Reachable server host and optional port, optionally with its scheme (`https://host` sets `connection.protocol`). Override the shipped in-cluster default unless your process can reach that service. | | `OPENSANDBOX_API_KEY` | Required by the shipped config; obtain it from the service operator or configure it on your own server as described above. Inject it through your approved secret manager or protected runtime environment. | Sandbox, model, and image-registry credentials are separate. Keep secrets out of source control, issues, and resolved-config logs. ### Check access before allocating sandboxes From the Gym server's host or container: * Confirm the endpoint resolves and is reachable. The shipped `.svc.cluster.local` address normally requires cluster network/DNS access. Use the operator's TLS/CA configuration. * Use the public `/health` route to check reachability, then the deployment's read-only authenticated check to verify credentials. * Confirm image-registry and required download access, plus model connectivity from the component making requests. Inside a sandbox, `localhost` refers to the sandbox itself. * Agree initial concurrency, lifetime, and cleanup within the deployment's limits. Resolve access, quota, or image-pull failures with the operator before retrying. After these checks, run an agent evaluation to verify the complete workflow. ## Provider Config Load the [shipped OpenSandbox config](https://github.com/NVIDIA-NeMo/Gym/blob/main/nemo_gym/sandbox/providers/opensandbox/configs/opensandbox.yaml) and select it with `sandbox_provider: sandbox`. This excerpt shows selected defaults; load the full file using the launch command below. ```yaml sandbox: default_metadata: sandbox-api: opensandbox-sdk opensandbox: connection: domain: ${oc.env:OPENSANDBOX_DOMAIN,opensandbox-server.opensandbox-system.svc.cluster.local} api_key: ${oc.env:OPENSANDBOX_API_KEY} protocol: http request_timeout_s: 300 use_server_proxy: true create: request_timeout_s: 1200 timeout_s: 1500 connect_attempt_timeout_s: 1500 skip_health_check: false retries: 10 retry_delay_s: 5.0 retry_max_delay_s: 90.0 probe: timeout_s: 60 deadline_s: 180 stable_count: 2 stable_delay_s: 1.0 operations: retries: 5 retry_delay_s: 1.0 retry_max_delay_s: 45.0 command_retries: 0 close_timeout_s: 30 ``` Keep `create.skip_health_check: false` to avoid commands racing sandbox readiness. Keep `create.timeout_s` above the task's `ready_timeout_s`: the shipped mini-SWE values are 1500 and 1200 seconds respectively. Adjust them together. Run an agent with the provider config by passing it alongside the agent and model configs: ```bash gym env start \ --config responses_api_agents/mini_swe_agent_2/configs/mini_swe_agent_2.yaml \ --config nemo_gym/sandbox/providers/opensandbox/configs/opensandbox.yaml \ --config responses_api_models/vllm_model/configs/vllm_model.yaml ``` The provider constructor accepts four optional config sections: | Section | Purpose | | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `connection` | OpenSandbox SDK connection settings: domain, API key, protocol, request timeout, server proxy mode, and `tls_verify`. | | `create` | Sandbox create timeout, retry, image pull policy, and health-check behavior. | | `probe` | Post-create command probe used to verify command execution before the sandbox is returned. | | `operations` | Retry and timeout behavior for SDK operations after create, including command retries, close timeout, background command polling, and the dedicated `status_poll_timeout_s` budget for background status polls. | ## SandboxSpec Provider Options Set OpenSandbox-specific create options under `SandboxSpec.provider_options`, or under `sandbox_spec.provider_options` in an agent config: ```yaml sandbox_spec: provider_options: image_auth: username: ${oc.env:REGISTRY_USERNAME} password: ${oc.env:REGISTRY_PASSWORD} platform: os: linux arch: amd64 skip_health_check: false extensions: imagePullPolicy: IfNotPresent ``` Supported options are: | Option | Purpose | | ------------------- | -------------------------------------------------------------------------------------------- | | `image_auth` | Registry username and password passed to the OpenSandbox SDK for private image pulls. | | `platform` | Mapping passed to the OpenSandbox SDK `PlatformSpec`, such as `os` and `arch`. | | `snapshot_id` | OpenSandbox snapshot ID to create from instead of only an image. | | `volumes` | List of mappings passed to the OpenSandbox SDK `Volume` model. | | `skip_health_check` | Per-sandbox override for SDK create health checking unless the provider config forces it on. | | `extensions` | String map passed to OpenSandbox create extensions. | Unknown provider option keys raise `ValueError`; values with the wrong type raise `TypeError`. This catches config drift before a sandbox allocation is attempted. ## Relevant SandboxSpec Fields | Field | OpenSandbox behavior | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | | `image` | Container image passed to OpenSandbox create. | | `ttl_s` | Converted to an OpenSandbox sandbox timeout. | | `ready_timeout_s` | Converted to an OpenSandbox ready timeout. | | `env` | Passed to OpenSandbox create and command execution. | | `files` | Seed files uploaded before the first command. | | `metadata` | Passed to OpenSandbox create after provider-specific normalization. | | `resources` | Mapped to OpenSandbox resource quantities. | | `entrypoint` | Passed to OpenSandbox create when set. | | `provider_options` | Per-sandbox OpenSandbox options such as `image_auth`, `platform`, `snapshot_id`, `volumes`, `skip_health_check`, and `extensions`. | ## Resource Mapping `SandboxResources` is translated into OpenSandbox resource quantities: | SandboxResources field | OpenSandbox resource key | | ---------------------- | --------------------------------------------- | | `cpu` | `cpu` | | `memory_mib` | `memory`, formatted as `Mi` | | `disk_gib` | `ephemeral-storage`, formatted as `Gi` | | `gpu` | `gpu` | | `gpu_type` | `gpu_type` | The provider also normalizes metadata values for backend labels by replacing unsupported characters and truncating values to the provider limit. ## Job Attribution The provider merges `nemo-gym.nvidia.com/team`, `nemo-gym.nvidia.com/user`, `nemo-gym.nvidia.com/workload`, and `nemo-gym.nvidia.com/run` keys into each sandbox's metadata when they can be resolved. OpenSandbox propagates sandbox metadata as Kubernetes labels on the sandbox resources, so running sandboxes are attributable both through the OpenSandbox list API metadata filter and directly at the cluster level: ```bash kubectl get pods -l nemo-gym.nvidia.com/team=my-team ``` Each field resolves in order: the provider's `attribution` config, then `NEMO_GYM_TEAM` / `NEMO_GYM_USER` / `NEMO_GYM_WORKLOAD` environment variables, then Slurm job environment variables (`SLURM_JOB_ACCOUNT` / `SLURM_JOB_USER` / `SLURM_JOB_NAME`). `user` additionally falls back to the OS login name (`root` is ignored — containers run as root by default, so it would attribute the image, not a person), and `workload` to `NEMO_GYM_CONFIG_PATH`, the server instance name the gym CLI sets on every server process it spawns. Fields that cannot be resolved are omitted. Explicit `sandbox_spec.metadata` and `default_metadata` keys always take precedence over attribution keys. `run` identifies one launch of the creating process: `NEMO_GYM_RUN_ID` if set, else an id generated per process. The resolved attribution is logged once at the first sandbox create — note the `run` value from the logs to list or clean up exactly that run's sandboxes (for example after killing a run partway through), which `team` / `user` / `workload` cannot do when the same user launches the same workload twice: ```bash kubectl delete batchsandboxes -l nemo-gym.nvidia.com/run= ``` ```yaml sandbox: opensandbox: attribution: enabled: true # set false to disable automatic attribution key_prefix: nemo-gym.nvidia.com/ # label-key namespace; "" for bare team/user/workload/run keys # team: my-team # override auto-detection # user: my-user # workload: my-benchmark # run: my-run-id ``` ### Attribution in Kubernetes deployments When the gym servers themselves run in Kubernetes (for example a driver pod launching an eval), none of the automatic sources resolve: there are no Slurm variables, and the container's OS login is typically `root`, which is ignored. Set the `NEMO_GYM_*` variables on the pod spec — either directly or via the downward API from labels the pod already carries: ```yaml env: - name: NEMO_GYM_TEAM value: my-team - name: NEMO_GYM_USER valueFrom: fieldRef: fieldPath: metadata.labels['owner'] ``` ## Lifecycle The provider creates one OpenSandbox sandbox per Gym sandbox: | Operation | OpenSandbox behavior | | --------- | ------------------------------------------------------------------------------------ | | Create | `opensandbox.Sandbox.create(...)` with normalized `SandboxSpec` fields. | | Exec | `handle.raw.commands.run(...)` with command options derived from `exec()` arguments. | | Status | `handle.raw.get_info()` mapped onto `SandboxStatus`. | | Close | `handle.raw.kill()` followed by local SDK handle cleanup. | If `create.skip_health_check` is enabled, the provider reconnects to the sandbox before running the NeMo Gym readiness probe so follow-up operations use a fresh SDK handle. ## File Transfer The provider uses the OpenSandbox SDK file API for uploads and downloads: * `upload()` reads the local file and writes bytes to the target sandbox path. * `download()` reads bytes from the sandbox path and writes them to the local target path. Startup files from `SandboxSpec.files` use the same provider file path semantics before the first command runs. ## User and Runtime Notes The neutral `user` argument to `exec()` maps onto OpenSandbox command options: | `user` value | Behavior | | ------------ | ----------------------------------------------------- | | `None` | Use the sandbox default user. | | `"root"` | Run the command as provided. | | Integer uid | Pass the uid through the OpenSandbox command options. | | Other string | Wrap the command with `su -s /bin/sh -c ... `. | Command failures return `SandboxExecResult` with the command's exit code. If OpenSandbox reports an execution error without an exit code, the provider returns code `125` with `error_type="sandbox"`. The provider's `create.image_pull_policy` defaults to `IfNotPresent`. Valid values are `Always`, `IfNotPresent`, and `Never`. The resolved policy is written into OpenSandbox create extensions as both `imagePullPolicy` and `opensandbox.extensions.image-pull-policy` unless those keys are already present in `provider_options.extensions`. Create retries handle transient allocation, connection, image pull, and server-side errors. Command retries are controlled separately by `operations.command_retries`. execd, the exec daemon inside each sandbox, holds every command response open for `EXECD_API_GRACE_SHUTDOWN` (default 1s) after the command's last event, which is fixed latency on each command and on the readiness probe. Set it in the sandbox spec's `env` to shorten it; the shipped SWE-bench and OpenCode configs use `50ms`, and the provider's `operations.background_poll_initial_s` (0.5) is sized to that. `connection.tls_verify` (default `false`) controls certificate verification on every connection the provider opens, both the SDK transport and PTY WebSockets. The default skips verification so that `https://` endpoints with a certificate the client cannot verify, such as a validation cluster behind a self-signed load balancer, work without extra setup; set `tls_verify: true` for endpoints with a verifiable certificate. For `https` endpoints, either set `connection.protocol: https` or include the scheme in the domain (`connection.domain: https://sandbox.example`, or `OPENSANDBOX_DOMAIN=https://sandbox.example`): a scheme in the domain sets the protocol and takes precedence over `connection.protocol`. The `cleanup_sandboxes.py` helper mirrors the setting with `--tls-verify` and reads `tls_verify` from `--connection-config`. Setting `connection.keepalive_expiry_s: null` preserves the configured `tls_verify` behavior. It uses the SDK's default transport only when `tls_verify: true` and `disable_connection_pooling: false`; otherwise, the provider supplies a transport to apply those settings.