nemo_gym.sandbox.providers.opensandbox.provider

View as Markdown

OpenSandbox provider implementation.

Module Contents

Classes

NameDescription
OpenSandboxAttributionConfigJob attribution merged into every sandbox’s metadata (Kubernetes labels on the sandbox).
OpenSandboxConnectionConfigOpenSandbox server connection settings.
OpenSandboxCreateConfigOpenSandbox create/reconnect retry settings.
OpenSandboxCreateErrorRaised when OpenSandbox cannot create a sandbox.
OpenSandboxCreateTimeoutErrorRaised when OpenSandbox sandbox creation exceeds the client timeout.
OpenSandboxCreateVerificationErrorRaised when a newly-created sandbox cannot execute a probe command.
OpenSandboxOperationConfigRetry and timeout settings for SDK operations after create.
OpenSandboxProbeConfigPost-create probe settings.
OpenSandboxProviderProvider backed by the OpenSandbox SDK/server API.
OpenSandboxProviderOptionsRecognized per-sandbox create options read from SandboxSpec.provider_options.
SandboxBackendUnreachableErrorRaised when the server proxy cannot open a TCP connection to a sandbox’s exec daemon.

Functions

Data

ATTRIBUTION_KEY_PREFIX_RE

DEFAULT_ATTRIBUTION_KEY_PREFIX

DEFAULT_IMAGE_PULL_POLICY

IMAGE_PULL_POLICY_ANNOTATION_EXTENSION_KEY

IMAGE_PULL_POLICY_EXTENSION_KEY

LOGGER

METADATA_VALUE_RE

RETRYABLE_ERROR_MARKERS

RETRYABLE_HTTP_STATUS_CODES

STATUS_CODE_RE

VALID_IMAGE_PULL_POLICIES

API

class nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxAttributionConfig(
enabled: bool = True,
team: str | None = None,
user: str | None = None,
workload: str | None = None,
run: str | None = None,
key_prefix: str = DEFAULT_ATTRIBUTION_KEY_PREFIX
)
Dataclass

Job attribution merged into every sandbox’s metadata (Kubernetes labels on the sandbox).

OpenSandbox propagates sandbox metadata as Kubernetes labels on the sandbox resources, so attribution is queryable both through the OpenSandbox list API and at the cluster level (e.g. kubectl get pods -l nemo-gym.nvidia.com/team=my-team). key_prefix namespaces the label keys (Kubernetes prefixed-key convention); set it to "" for bare team / user / workload / run keys.

Unset fields are auto-detected: NEMO_GYM_TEAM / NEMO_GYM_USER / NEMO_GYM_WORKLOAD environment variables first, then Slurm job env vars (SLURM_JOB_ACCOUNT / SLURM_JOB_USER / SLURM_JOB_NAME), then the OS login name for user (root is ignored) and the gym CLI’s NEMO_GYM_CONFIG_PATH server instance name for workload. Fields that cannot be resolved are omitted. run scopes sandboxes to one launch of the creating process (NEMO_GYM_RUN_ID, else generated per process and logged) so a run’s sandboxes can be listed and cleaned up exactly. Explicit SandboxSpec.metadata keys always take precedence over attribution keys.

enabled
bool = True
key_prefix
str = DEFAULT_ATTRIBUTION_KEY_PREFIX
run
str | None = None
team
str | None = None
user
str | None = None
workload
str | None = None
nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxAttributionConfig.__post_init__() -> None
class nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxConnectionConfig(
domain: str | None = None,
api_key: str | None = None,
protocol: str | None = None,
request_timeout_s: int | None = None,
use_server_proxy: bool = False,
disable_connection_pooling: bool = False,
keepalive_expiry_s: float | None = 3.0,
max_keepalive_connections: int = 20,
max_connections: int | None = 100,
connect_retries: int = 2,
transport_backend: str = 'httpx'
)
Dataclass

OpenSandbox server connection settings.

keepalive_expiry_s must stay below the server’s own keep-alive idle timeout (uvicorn defaults to 5s), or pooled sockets are reused after the server has closed them; null falls back to the SDK’s default transport. transport_backend is “httpx” or “aiohttp” (via the optional httpx-aiohttp bridge, falling back to httpx when it is absent). The pool is shared, so max_connections also caps in-flight sandbox operations per process; null means no cap.

api_key
str | None = None
connect_retries
int = 2
disable_connection_pooling
bool = False
domain
str | None = None
keepalive_expiry_s
float | None = 3.0
max_connections
int | None = 100
max_keepalive_connections
int = 20
protocol
str | None = None
request_timeout_s
int | None = None
transport_backend
str = 'httpx'
use_server_proxy
bool = False
class nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxCreateConfig(
request_timeout_s: int | None = None,
timeout_s: float | None = None,
retries: int = 2,
retry_delay_s: float = 5.0,
retry_max_delay_s: float = 60.0,
image_pull_policy: str | None = DEFAULT_IMAGE_PULL_POLICY,
skip_health_check: bool = False,
connect_attempt_timeout_s: float = 30.0,
connect_poll_s: float = 2.0
)
Dataclass

OpenSandbox create/reconnect retry settings.

connect_attempt_timeout_s
float = 30.0
connect_poll_s
float = 2.0
image_pull_policy
str | None = DEFAULT_IMAGE_PULL_POLICY
request_timeout_s
int | None = None
retries
int = 2
retry_delay_s
float = 5.0
retry_max_delay_s
float = 60.0
skip_health_check
bool = False
timeout_s
float | None = None
nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxCreateConfig.__post_init__() -> None
class nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxCreateError()

Bases: SandboxCreateError

Raised when OpenSandbox cannot create a sandbox.

class nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxCreateTimeoutError()

Bases: OpenSandboxCreateError

Raised when OpenSandbox sandbox creation exceeds the client timeout.

class nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxCreateVerificationError()

Bases: SandboxCreateVerificationError

Raised when a newly-created sandbox cannot execute a probe command.

class nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxOperationConfig(
retries: int = 3,
retry_delay_s: float = 1.0,
retry_max_delay_s: float = 15.0,
command_retries: int = 0,
close_timeout_s: float | None = 30.0,
background_exec: bool = False,
background_poll_initial_s: float = 0.25,
background_poll_interval_s: float = 2.0
)
Dataclass

Retry and timeout settings for SDK operations after create.

background_exec
bool = False
background_poll_initial_s
float = 0.25
background_poll_interval_s
float = 2.0
close_timeout_s
float | None = 30.0
command_retries
int = 0
retries
int = 3
retry_delay_s
float = 1.0
retry_max_delay_s
float = 15.0
nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxOperationConfig.__post_init__() -> None
class nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProbeConfig(
command: str | None = 'printf nemo-gym-sandbox-re...,
expected_stdout: str | None = 'nemo-gym-sandbox-ready',
timeout_s: int = 30,
deadline_s: float | None = None,
stable_count: int = 1,
stable_delay_s: float = 0.0
)
Dataclass

Post-create probe settings.

command
str | None = 'printf nemo-gym-sandbox-ready'
deadline_s
float | None = None
expected_stdout
str | None = 'nemo-gym-sandbox-ready'
stable_count
int = 1
stable_delay_s
float = 0.0
timeout_s
int = 30
nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProbeConfig.__post_init__() -> None
class nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider(
connection: nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxConnectionConfig | collections.abc.Mapping[str, typing.Any] | None = None,
create: nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxCreateConfig | collections.abc.Mapping[str, typing.Any] | None = None,
probe: nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProbeConfig | collections.abc.Mapping[str, typing.Any] | None = None,
operations: nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxOperationConfig | collections.abc.Mapping[str, typing.Any] | None = None,
attribution: nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxAttributionConfig | collections.abc.Mapping[str, typing.Any] | None = None
)

Provider backed by the OpenSandbox SDK/server API.

_attribution
_connection
_create
= _coerce_config(create, OpenSandboxCreateConfig)
_operations
_probe
= _coerce_config(probe, OpenSandboxProbeConfig)
_transport
Any | None = None
name
= 'opensandbox'
nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._attribution_metadata() -> dict[str, str]
nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._await_sdk_call(
awaitable: typing.Any,
operation: str,
sandbox_id: str,
timeout_s: float | None
) -> typing.Any
async
nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._await_sdk_operation(
operation_factory: typing.Callable[[], typing.Awaitable[typing.Any]],
operation: str,
sandbox_id: str,
timeout_s: float | None,
retries: int | None = None
) -> typing.Any
async
nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._build_transport() -> typing.Any

Build the SDK transport with the configured pool limits.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._cleanup_failed_create_handle(
handle: nemo_gym.sandbox.providers.base.SandboxHandle
) -> None
async
nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._command_retry_count() -> int
nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._connect_after_create(
handle: nemo_gym.sandbox.providers.base.SandboxHandle,
spec: nemo_gym.sandbox.providers.base.SandboxSpec
) -> nemo_gym.sandbox.providers.base.SandboxHandle
async

Reconnect after SDK create so follow-up calls use a fresh SDK handle.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._connection_config(
request_timeout_s: int | float | None = None
) -> typing.Any
nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._create_once(
spec: nemo_gym.sandbox.providers.base.SandboxSpec
) -> nemo_gym.sandbox.providers.base.SandboxHandle
async

Create a sandbox through opensandbox.Sandbox.create.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._create_with_retries(
spec: nemo_gym.sandbox.providers.base.SandboxSpec
) -> nemo_gym.sandbox.providers.base.SandboxHandle
async
nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._exec(
handle: nemo_gym.sandbox.providers.base.SandboxHandle,
command: str,
cwd: str | None = None,
env: dict[str, str] | None = None,
timeout_s: int | float | None = None,
user: str | int | None = None,
retries: int | None = None
) -> nemo_gym.sandbox.providers.base.SandboxExecResult
async

Run a command inside an OpenSandbox sandbox.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._exec_background(
handle: nemo_gym.sandbox.providers.base.SandboxHandle,
command: str,
opts_kwargs: dict[str, typing.Any],
sdk_timeout_s: float | None,
total_timeout_s: int | float | None,
retries: int
) -> nemo_gym.sandbox.providers.base.SandboxExecResult
async

Run a command as a background execution polled via short requests.

The logs endpoint returns one combined stream, so unlike the foreground path stdout carries both streams and stderr is set only when the sandbox itself reports an error.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._get_transport() -> typing.Any

Return the provider-owned shared transport, building it on first use.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._read_file(
handle: nemo_gym.sandbox.providers.base.SandboxHandle,
source_path: str
) -> bytes
async

Read one file from an OpenSandbox sandbox.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._resolve_extensions(
extensions: collections.abc.Mapping[str, str]
) -> dict[str, str]

Add the configured default image pull policy to SDK create extensions.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._submit_command(
operation_factory: typing.Callable[[], typing.Awaitable[typing.Any]],
operation: str,
sandbox_id: str,
timeout_s: float | None,
retries: int
) -> typing.Any
async

Retry backend-connect 502s that command_retries deliberately skips.

A proxy 502 is a TCP-connect failure: the command never reached execd, so retrying under operations.retries cannot double-run it (unlike a real command failure). When that budget is exhausted the backend is dead, so raise a typed error and fail fast instead of retrying for hours.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._verify_created_handle(
handle: nemo_gym.sandbox.providers.base.SandboxHandle
) -> None
async
nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider._write_file(
handle: nemo_gym.sandbox.providers.base.SandboxHandle,
target_path: str,
data: str | bytes
) -> None
async

Write one file into an OpenSandbox sandbox.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider.aclose() -> None
async

Close provider-owned resources.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider.close(
handle: nemo_gym.sandbox.providers.base.SandboxHandle
) -> None
async

Terminate the sandbox and close local SDK resources.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider.connect(
descriptor: collections.abc.Mapping[str, typing.Any]
) -> nemo_gym.sandbox.providers.base.SandboxHandle
async

Rebuild a live handle from an OpenSandbox sandbox id via the SDK.

Health-checks unless the caller opts out: a sandbox id only proves the workload exists, not that its exec daemon is listening yet, so an unchecked handle turns that gap into a 502 on the first call.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider.create(
spec: nemo_gym.sandbox.providers.base.SandboxSpec
) -> nemo_gym.sandbox.providers.base.SandboxHandle
async

Create one sandbox through the configured OpenSandbox path.

Job attribution keys (team / user / workload / run) are merged into the spec’s metadata (explicit spec keys win) so every sandbox is attributable via its labels.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider.download_file(
handle: nemo_gym.sandbox.providers.base.SandboxHandle,
source_path: str,
target_path: pathlib.Path
) -> None
async

Download one file from an OpenSandbox sandbox.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider.exec(
handle: nemo_gym.sandbox.providers.base.SandboxHandle,
command: str,
cwd: str | None = None,
env: dict[str, str] | None = None,
timeout_s: int | float | None = None,
user: str | int | None = None
) -> nemo_gym.sandbox.providers.base.SandboxExecResult
async

Run a command inside an OpenSandbox sandbox.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider.serialize_handle(
handle: nemo_gym.sandbox.providers.base.SandboxHandle,
scope: str | None = None
) -> dict[str, typing.Any]
async

Return a descriptor for reattaching to this sandbox by id.

OpenSandbox sandboxes are reachable by id from any process that has the connection config, so the id alone is enough to reconnect and no sandbox server is needed to share one. scope is ignored: OpenSandbox has no lease concept of its own.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider.status(
handle: nemo_gym.sandbox.providers.base.SandboxHandle
) -> nemo_gym.sandbox.providers.base.SandboxStatus
async

Return the current OpenSandbox lifecycle status.

nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProvider.upload_file(
handle: nemo_gym.sandbox.providers.base.SandboxHandle,
source_path: pathlib.Path,
target_path: str
) -> None
async

Upload one local file into an OpenSandbox sandbox.

class nemo_gym.sandbox.providers.opensandbox.provider.OpenSandboxProviderOptions(
image_auth: collections.abc.Mapping[str, typing.Any] | None = None,
platform: collections.abc.Mapping[str, typing.Any] | None = None,
snapshot_id: str | None = None,
volumes: tuple[collections.abc.Mapping[str, typing.Any], ...] = (),
skip_health_check: bool | None = None,
extensions: collections.abc.Mapping[str, str] = dict(),
resource_requests: collections.abc.Mapping[str, typing.Any] | None = None
)
Dataclass

Recognized per-sandbox create options read from SandboxSpec.provider_options.

image_auth, platform, and volumes entries are passed through to the OpenSandbox SDK, so their inner fields are validated by the SDK rather than here.

extensions
Mapping[str, str] = field(default_factory=dict)
image_auth
Mapping[str, Any] | None = None
platform
Mapping[str, Any] | None = None
resource_requests
Mapping[str, Any] | None = None
skip_health_check
bool | None = None
snapshot_id
str | None = None
volumes
tuple[Mapping[str, Any], ...] = ()
classmethod
class nemo_gym.sandbox.providers.opensandbox.provider.SandboxBackendUnreachableError()

Bases: RuntimeError

Raised when the server proxy cannot open a TCP connection to a sandbox’s exec daemon.

The proxy’s 502 is a connect failure, so the submitted command never started. Persistent 502s mean the backend is gone (e.g. the container was OOM-killed and sandbox pods never restart); retrying cannot revive it.

nemo_gym.sandbox.providers.opensandbox.provider._exception_status_code(
exception: BaseException
) -> int | None
nemo_gym.sandbox.providers.opensandbox.provider._has_retryable_error_marker(
exception: BaseException
) -> bool
nemo_gym.sandbox.providers.opensandbox.provider._is_missing_sandbox_delete_error(
exception: BaseException
) -> bool
nemo_gym.sandbox.providers.opensandbox.provider._is_retryable_create_error(
exception: BaseException
) -> bool

Return whether a sandbox create failure is likely transient.

nemo_gym.sandbox.providers.opensandbox.provider._is_retryable_sdk_operation_error(
exception: BaseException,
seen: set[int] | None = None
) -> bool

Return whether an SDK operation can be retried.

nemo_gym.sandbox.providers.opensandbox.provider._log_create_retry(
retry_state: typing.Any
) -> None
nemo_gym.sandbox.providers.opensandbox.provider._log_operation_retry(
retry_state: typing.Any,
operation: str = '?',
sandbox_id: str = '?'
) -> None
nemo_gym.sandbox.providers.opensandbox.provider._metadata_map(
values: dict[str, typing.Any]
) -> dict[str, str]
nemo_gym.sandbox.providers.opensandbox.provider._metadata_value(
value: typing.Any
) -> str
nemo_gym.sandbox.providers.opensandbox.provider._normalize_spec(
spec: nemo_gym.sandbox.providers.base.SandboxSpec
) -> nemo_gym.sandbox.providers.base.SandboxSpec
nemo_gym.sandbox.providers.opensandbox.provider._require_opensandbox_sdk() -> tuple[typing.Any, typing.Any, typing.Any, typing.Any, typing.Any]
nemo_gym.sandbox.providers.opensandbox.provider._require_tenacity() -> tuple[typing.Any, typing.Any, typing.Any, typing.Any]
nemo_gym.sandbox.providers.opensandbox.provider._resource_map(
resources: nemo_gym.sandbox.providers.base.SandboxResources
) -> dict[str, str]
nemo_gym.sandbox.providers.opensandbox.provider._resource_quantity(
value: float | int
) -> str
nemo_gym.sandbox.providers.opensandbox.provider._sdk_error_attributes(
exception: BaseException,
operation: str,
sandbox_id: str,
attempt_number: int | None = None,
max_attempts: int | None = None,
sleep_s: float | None = None
) -> dict[str, typing.Any]
nemo_gym.sandbox.providers.opensandbox.provider._string_map(
values: collections.abc.Mapping[str, typing.Any]
) -> dict[str, str]
nemo_gym.sandbox.providers.opensandbox.provider._to_image_spec(
image: str,
image_auth: collections.abc.Mapping[str, typing.Any] | None
) -> typing.Any
nemo_gym.sandbox.providers.opensandbox.provider._to_platform_spec(
platform: dict[str, typing.Any]
) -> typing.Any
nemo_gym.sandbox.providers.opensandbox.provider._to_sandbox_status(
state: typing.Any
) -> nemo_gym.sandbox.providers.base.SandboxStatus
nemo_gym.sandbox.providers.opensandbox.provider._to_volumes(
volumes: list[collections.abc.Mapping[str, typing.Any]]
) -> list[typing.Any]
nemo_gym.sandbox.providers.opensandbox.provider.validate_image_pull_policy(
image_pull_policy: str
) -> str

Validate a Kubernetes-compatible container image pull policy.

nemo_gym.sandbox.providers.opensandbox.provider.ATTRIBUTION_KEY_PREFIX_RE = re.compile('(?=.{1,253}$)[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-...
nemo_gym.sandbox.providers.opensandbox.provider.DEFAULT_ATTRIBUTION_KEY_PREFIX = 'nemo-gym.nvidia.com/'
nemo_gym.sandbox.providers.opensandbox.provider.DEFAULT_IMAGE_PULL_POLICY = 'IfNotPresent'
nemo_gym.sandbox.providers.opensandbox.provider.IMAGE_PULL_POLICY_ANNOTATION_EXTENSION_KEY = 'opensandbox.extensions.image-pull-policy'
nemo_gym.sandbox.providers.opensandbox.provider.IMAGE_PULL_POLICY_EXTENSION_KEY = 'imagePullPolicy'
nemo_gym.sandbox.providers.opensandbox.provider.LOGGER = logging.getLogger(__name__)
nemo_gym.sandbox.providers.opensandbox.provider.METADATA_VALUE_RE = re.compile('[^A-Za-z0-9_.-]+')
nemo_gym.sandbox.providers.opensandbox.provider.RETRYABLE_ERROR_MARKERS = ('all connection attempts failed', 'connection refused', 'connection reset', 'ga...
nemo_gym.sandbox.providers.opensandbox.provider.RETRYABLE_HTTP_STATUS_CODES = {408, 409, 425, 429, 500, 502, 503, 504}
nemo_gym.sandbox.providers.opensandbox.provider.STATUS_CODE_RE = re.compile('(?:status code|http)\\D+(\\d{3})', re.IGNORECASE)
nemo_gym.sandbox.providers.opensandbox.provider.VALID_IMAGE_PULL_POLICIES = {'Always', 'IfNotPresent', 'Never'}