Skip to navigation

nemo_gym.sandbox.providers.enroot.provider

View as MarkdownOpen in Claude

Enroot provider implementation.

Enroot (https://github.com/NVIDIA/enroot) is an unprivileged container runtime. Unlike Apptainer it has no persistent daemon/instance concept, so this provider emulates a long-lived sandbox by launching a detached enroot start running a sleeping init and then re-entering it with enroot exec <pid>. See README.md for the enroot-specific design notes (non-daemonizing start, PID-based liveness, #-registry URIs, pinned ENROOT_* paths).

Module Contents

Classes

NameDescription
EnrootCreateConfigSettings for creating an Enroot sandbox container.
EnrootCreateErrorRaised when Enroot cannot create a sandbox.
EnrootCreateVerificationErrorRaised when a newly-created sandbox cannot execute a probe command.
EnrootExecConfigSettings for running commands inside an Enroot sandbox.
EnrootProbeConfigPost-create probe settings: a test command confirming the sandbox is usable.
EnrootProviderSandbox provider backed by the local Enroot CLI.
_EnrootInstanceProvider-private state stashed on SandboxHandle.raw.

Functions

NameDescription
_coerce_mountsNormalize spec.provider_options['mounts'] into a list of enroot fstab entries.
_find_container_init_pidFind the container init PID by scanning /proc for its unique cmdline.
_is_missing_container-
_is_runtime_failureBest-effort: did enroot itself fail to run the command (vs the command failing)?
_read_proc_cmdlineReturn /proc/<pid>/cmdline as a space-joined string, or ” if unreadable.
_require_enrootReturn the enroot binary path or hard-error if it is not installed.
_resource_gpu_envMap a neutral GPU request onto NVIDIA_VISIBLE_DEVICES for the enroot hook.
_translate_docker_uriTranslate a docker image reference into an enroot import URI.

Data

CONTAINER_NAME_PREFIX

DEFAULT_INIT_COMMAND

DEFAULT_MOUNT_POINT

DOCKER_HUB_HOSTS

ENROOT_MISSING_CONTAINER_MARKERS

ENROOT_RUNTIME_ERROR_MARKERS

LOGGER

READY_PROBE_COMMAND

READY_PROBE_EXPECTED

SANDBOX_RUNTIME_RETURN_CODE

API

class nemo_gym.sandbox.providers.enroot.EnrootCreateConfig(
mount_point: str = DEFAULT_MOUNT_POINT,
base_dir: str | None = None,
data_path: str | None = None,
cache_path: str | None = None,
runtime_path: str | None = None,
sqsh_cache_dir: str | None = None,
rw: bool = True,
remap_root: bool = False,
bypass_entrypoint: bool = True,
init_command: str = DEFAULT_INIT_COMMAND,
import_timeout_s: float | None = 1800,
create_timeout_s: float | None = 600,
start_timeout_s: float | None = 600,
start_poll_s: float = 0.5,
extra_import_args: list[str] = list(),
extra_create_args: list[str] = list(),
extra_start_args: list[str] = list()
)
Dataclass

Settings for creating an Enroot sandbox container.

base_dir
str | None = None
bypass_entrypoint
bool = True
cache_path
str | None = None
create_timeout_s
float | None = 600
data_path
str | None = None
extra_create_args
list[str] = field(default_factory=list)
extra_import_args
list[str] = field(default_factory=list)
extra_start_args
list[str] = field(default_factory=list)
import_timeout_s
float | None = 1800
init_command
str = DEFAULT_INIT_COMMAND
mount_point
str = DEFAULT_MOUNT_POINT
remap_root
bool = False
runtime_path
str | None = None
rw
bool = True
sqsh_cache_dir
str | None = None
start_poll_s
float = 0.5
start_timeout_s
float | None = 600
nemo_gym.sandbox.providers.enroot.EnrootCreateConfig.__post_init__() -> None
class nemo_gym.sandbox.providers.enroot.EnrootCreateError()

Bases: SandboxCreateError

Raised when Enroot cannot create a sandbox.

class nemo_gym.sandbox.providers.enroot.EnrootCreateVerificationError()

Bases: SandboxCreateVerificationError

Raised when a newly-created sandbox cannot execute a probe command.

class nemo_gym.sandbox.providers.enroot.EnrootExecConfig(
default_timeout_s: float | None = 180,
default_mounts: list[str] = list(),
extra_exec_args: list[str] = list(),
concurrency: int = 32,
exec_shell: str = 'sh'
)
Dataclass

Settings for running commands inside an Enroot sandbox.

concurrency
int = 32
default_mounts
list[str] = field(default_factory=list)
default_timeout_s
float | None = 180
exec_shell
str = 'sh'
extra_exec_args
list[str] = field(default_factory=list)
nemo_gym.sandbox.providers.enroot.EnrootExecConfig.__post_init__() -> None
class nemo_gym.sandbox.providers.enroot.EnrootProbeConfig(
command: str | None = READY_PROBE_COMMAND,
expected_stdout: str | None = READY_PROBE_EXPECTED,
timeout_s: int = 30,
deadline_s: float | None = None,
stable_count: int = 1,
stable_delay_s: float = 0.0
)
Dataclass

Post-create probe settings: a test command confirming the sandbox is usable.

command
str | None = READY_PROBE_COMMAND
deadline_s
float | None = None
expected_stdout
str | None = READY_PROBE_EXPECTED
stable_count
int = 1
stable_delay_s
float = 0.0
timeout_s
int = 30
nemo_gym.sandbox.providers.enroot.EnrootProbeConfig.__post_init__() -> None
class nemo_gym.sandbox.providers.enroot.EnrootProvider(
exec: nemo_gym.sandbox.providers.enroot.provider.EnrootExecConfig | collections.abc.Mapping[str, typing.Any] | None = None,
create: nemo_gym.sandbox.providers.enroot.provider.EnrootCreateConfig | collections.abc.Mapping[str, typing.Any] | None = None,
probe: nemo_gym.sandbox.providers.enroot.provider.EnrootProbeConfig | collections.abc.Mapping[str, typing.Any] | None = None
)

Sandbox provider backed by the local Enroot CLI.

_binary
= _require_enroot()
_create_config
= _coerce_config(create, EnrootCreateConfig)
_enroot_env
_exec_config
= _coerce_config(exec, EnrootExecConfig)
_import_locks
dict[str, Lock] = {}
_probe
= _coerce_config(probe, EnrootProbeConfig)
_semaphore
= asyncio.Semaphore(self._exec_config.concurrency)
_sqsh_cache_dir
= Path(cfg.sqsh_cache_dir or base / 'sqsh')
name
= 'enroot'
nemo_gym.sandbox.providers.enroot.EnrootProvider._await_container_pid(
err_f: typing.IO[bytes]
) -> int
async

Poll enroot list until the container’s init PID appears, or raise.

nemo_gym.sandbox.providers.enroot.EnrootProvider._cleanup_failed_create_handle(
) -> None
async

Best-effort teardown of a sandbox that failed to start or verify.

nemo_gym.sandbox.providers.enroot.EnrootProvider._ensure_image(
image: str
) -> pathlib.Path
async

Return a local squashfs path for image, importing (and caching) if needed.

nemo_gym.sandbox.providers.enroot.EnrootProvider._kill_start_group(
) -> None

Best-effort SIGTERM then SIGKILL of the detached start’s process group.

A handle rebuilt by connect() has no proc and never reaped the group, so it signals the group only while its leader still carries the container marker; a process-group ID reused after the creator stopped is left alone.

nemo_gym.sandbox.providers.enroot.EnrootProvider._lookup_container(
name: str
) -> tuple[int | None, bool]
async

Return (pid, present). pid is the running init PID or None; present is whether the rootfs exists at all (running or not).

nemo_gym.sandbox.providers.enroot.EnrootProvider._pid_has_container_marker(
pid: int,
name: str
) -> bool
staticmethod

Return True if /proc/<pid>/cmdline still carries the container’s unique marker.

This guards against PID reuse: if the init exits and Linux reuses the PID for an unrelated process, the container marker won’t appear in the new process’s cmdline, so exec() returns a sandbox error instead of joining the wrong namespace.

nemo_gym.sandbox.providers.enroot.EnrootProvider._read_temp(
handle: typing.IO[bytes]
) -> str
staticmethod
nemo_gym.sandbox.providers.enroot.EnrootProvider._resolve_image(
image: str
) -> tuple[str | None, pathlib.Path | None]

Return (import_uri, sqsh_path). Exactly one is non-None.

A local .sqsh file is used directly (no import). Anything else is translated to an enroot import URI.

nemo_gym.sandbox.providers.enroot.EnrootProvider._run(
argv: list[str],
timeout_s: float | None,
stdin: bytes | None = None
) -> tuple[int, str, str]
async

Run an enroot CLI command. Returns (return_code, stdout, stderr).

Enforces timeout via asyncio.wait_for and kills the whole process group on timeout so child processes do not linger. Bounds concurrency with a shared semaphore. Decodes output with errors=“replace”. Every call runs with the pinned ENROOT_* environment.

nemo_gym.sandbox.providers.enroot.EnrootProvider._start_detached(
argv: list[str]
) -> tuple[typing.Any, typing.IO[bytes], typing.IO[bytes]]
async

Launch the long-lived enroot start init without awaiting its exit.

enroot start does not daemonize — it stays in the foreground for the whole container lifetime. We launch it detached in its own session (start_new_session=True), capture output to temp files (so an early exit leaves diagnosable stderr), and return the process handle plus the temp files. The caller confirms readiness by polling enroot list.

nemo_gym.sandbox.providers.enroot.EnrootProvider._verify_created_handle(
) -> None
async

Run the readiness probe until the sandbox responds, or raise.

  • probe.command is None -> skip (no verification).
  • probe.deadline_s is None -> single attempt; a failure raises immediately.
  • probe.deadline_s is set -> poll until the sandbox passes the probe stable_count consecutive times, or the deadline elapses.
nemo_gym.sandbox.providers.enroot.EnrootProvider.aclose() -> None
async

No provider-wide resources to close.

nemo_gym.sandbox.providers.enroot.EnrootProvider.close(
) -> None
async

Kill the container init, remove the rootfs, and clean up the staging dir.

nemo_gym.sandbox.providers.enroot.EnrootProvider.connect(
descriptor: collections.abc.Mapping[str, object]
async

Rebuild a handle to a running sandbox from a serialize_handle descriptor.

Nothing is created or restarted. The descriptor must come from this host and UID with the same Enroot data and runtime paths, and its staging directory must still exist. Release the handle with AsyncSandbox.disconnect(); stop() would tear down the creator’s sandbox.

Raises:

  • ValueError: the descriptor is malformed or was made on another host, UID or set of Enroot paths.
  • RuntimeError: the container init has exited or its PID was reused.
nemo_gym.sandbox.providers.enroot.EnrootProvider.create(
async

Import/create the rootfs, launch a detached init, and return a ready handle.

nemo_gym.sandbox.providers.enroot.EnrootProvider.download_file(
source_path: str,
target_path: pathlib.Path
) -> None
async

Download one sandbox file to the host.

Fast path (source under the bind mount): read directly from the host side of the shared folder. Fallback (arbitrary path): cp inside the container into the shared folder, then read the host side.

nemo_gym.sandbox.providers.enroot.EnrootProvider.exec(
command: str,
cwd: str | None = None,
env: dict[str, str] | None = None,
timeout_s: int | float | None = None,
user: str | int | None = None,
stdin: bytes | None = None
async

Run a command inside the container via enroot exec <pid>.

Maps the neutral user parameter onto enroot:

  • None -> run as the default (launching) user, or root if the container was started with remap_root.
  • “root” / 0 -> run directly (requires create.remap_root=true to be root inside the container).
  • other user/uid -> wrap in su to switch to that user (requires root inside the container).

stdin, when given, is piped to the command’s standard input.

nemo_gym.sandbox.providers.enroot.EnrootProvider.serialize_handle(
scope: str | None = None
) -> dict[str, object]
async

Share a running sandbox with another worker on the same host and UID.

The descriptor belongs to the trusted control plane. Both workers must use the same Enroot data/runtime directories. The creator retains ownership; a borrower releases its client with disconnect(). scope does not grant a separate lease for this local provider.

The sandbox environment stays out of the descriptor because it may hold task secrets. enroot exec builds each command’s environment from the init process’s /proc/<pid>/environ, so a borrower still sees it.

nemo_gym.sandbox.providers.enroot.EnrootProvider.status(
async

Return the container’s lifecycle status via enroot list -f.

Liveness is keyed off the PID column, not name presence: enroot lists created-but-not-running rootfs with an empty PID. Running -> RUNNING; present without a PID, or absent -> STOPPED; timeout/parse error -> UNKNOWN.

nemo_gym.sandbox.providers.enroot.EnrootProvider.upload_file(
source_path: pathlib.Path,
target_path: str
) -> None
async

Upload one host file into the sandbox.

Fast path (target under the bind mount): write directly to the host side of the shared folder. Fallback (arbitrary path): stage into the shared folder, then cp inside the container.

class nemo_gym.sandbox.providers.enroot.provider._EnrootInstance(
name: str,
sqsh_path: pathlib.Path,
staging_dir: pathlib.Path,
mount_point: str,
image: str,
env: dict[str, str] = dict(),
container_pid: int | None = None,
start_pgid: int | None = None,
proc: typing.Any = None
)
Dataclass

Provider-private state stashed on SandboxHandle.raw.

container_pid
int | None = None
env
dict[str, str] = field(default_factory=dict)
image
str
mount_point
str
name
str
sqsh_path
Path
staging_dir
Path
start_pgid
int | None = None
nemo_gym.sandbox.providers.enroot.provider._coerce_mounts(
value: typing.Any
) -> list[str]

Normalize spec.provider_options['mounts'] into a list of enroot fstab entries.

Accepts a single "src:dst[:type:opts]" string or a list of them. These are extra per-sandbox mounts, added on top of the staging mount and the provider-level exec.default_mounts.

nemo_gym.sandbox.providers.enroot.provider._find_container_init_pid(
base_init: str,
marker: str
) -> int | None

Find the container init PID by scanning /proc for its unique cmdline.

Nested inside pyxis, enroot runs as real root without a per-container user namespace, so enroot list cannot map a PID to the container name. The init is also reparented out of the enroot start process tree once the container’s PID namespace is set up, so a tree walk misses it. Instead we tag each container’s init with its unique name (... # <name>) and find the process whose cmdline is the init: it starts with sh -c (the provider passes argv0=“sh”, so this holds even where /bin/sh is dash) and carries both the base init loop and the unique marker. Scanning all of /proc is reparent-proof; the marker keeps it unambiguous under concurrency. The enroot start wrapper also carries the marker but its cmdline starts with the enroot binary, not sh -c , so it is excluded.

nemo_gym.sandbox.providers.enroot.provider._is_missing_container(
stderr: str
) -> bool
nemo_gym.sandbox.providers.enroot.provider._is_runtime_failure(
stderr: str
) -> bool

Best-effort: did enroot itself fail to run the command (vs the command failing)?

nemo_gym.sandbox.providers.enroot.provider._read_proc_cmdline(
pid: int
) -> str

Return /proc/<pid>/cmdline as a space-joined string, or ” if unreadable.

nemo_gym.sandbox.providers.enroot.provider._require_enroot() -> str

Return the enroot binary path or hard-error if it is not installed.

nemo_gym.sandbox.providers.enroot.provider._resource_gpu_env(
) -> dict[str, str]

Map a neutral GPU request onto NVIDIA_VISIBLE_DEVICES for the enroot hook.

nemo_gym.sandbox.providers.enroot.provider._translate_docker_uri(
image: str
) -> str

Translate a docker image reference into an enroot import URI.

Enroot’s docker scheme separates the registry host with # rather than / (docker://[USER@][REGISTRY#]IMAGE[:TAG]). A leading component that looks like a registry host (contains a . or :, or is localhost) is treated as the registry; otherwise the reference is a Docker Hub name.

The Docker Hub canonical hostnames are dropped rather than passed as the REGISTRY fragment: the public registry API is served from registry-1.docker.io, not the literal docker.io alias, so forwarding docker://docker.io#repo points enroot at an endpoint that returns non-JSON and the import fails. Dropping them lets enroot use its configured Hub default (e.g. docker://swebench/foo:tag).

nemo_gym.sandbox.providers.enroot.provider.CONTAINER_NAME_PREFIX = 'nemo-gym-'
nemo_gym.sandbox.providers.enroot.provider.DEFAULT_INIT_COMMAND = 'while true; do sleep 86400; done'
nemo_gym.sandbox.providers.enroot.provider.DEFAULT_MOUNT_POINT = '/sandbox'
nemo_gym.sandbox.providers.enroot.provider.DOCKER_HUB_HOSTS = frozenset({'docker.io', 'index.docker.io', 'registry-1.docker.io'})
nemo_gym.sandbox.providers.enroot.provider.ENROOT_MISSING_CONTAINER_MARKERS = ('does not exist', 'no such', 'not found')
nemo_gym.sandbox.providers.enroot.provider.ENROOT_RUNTIME_ERROR_MARKERS = ('no such process', 'no such file or directory: /proc', 'nsenter')
nemo_gym.sandbox.providers.enroot.provider.LOGGER = logging.getLogger(__name__)
nemo_gym.sandbox.providers.enroot.provider.READY_PROBE_COMMAND = f'printf enroot-sandbox-ready > {DEFAULT_MOUNT_POINT}/.nemo-gym-ready && printf ...
nemo_gym.sandbox.providers.enroot.provider.READY_PROBE_EXPECTED = 'enroot-sandbox-ready'
nemo_gym.sandbox.providers.enroot.provider.SANDBOX_RUNTIME_RETURN_CODE = 125