nemo_gym.sandbox.transfer

Sandbox transfer helpers.

Files move between this host and a sandbox as tar archives: upload_dir and download_dir pack on one side and unpack on the other, and run the sandbox side through exec_as_root, so the task’s file modes travel unchanged and a target the image’s default user cannot write (/tests, /logs) is still reachable. upload_path and download_path move a single file or a directory the same way, and remote_kind tells which a path is. upload_archive replays the archive a download kept (archive_copy) into another sandbox, so what the verifier receives is byte for byte what the agent’s sandbox packed. tar_extract_command is the shell line that unpacks an uploaded archive with GNU or BusyBox tar. SandboxTransferError reports a transfer command that failed inside the sandbox, SandboxProbeError a probe the sandbox could not answer, and SandboxUnavailableError a command the provider could not run at all, which is Gym’s outage rather than the task’s. Environment servers and the agents that seed a sandbox share this module.

Module Contents

Classes

Name	Description
`SandboxProbeError`	The sandbox could not answer a probe or run a command (lost container, provider timeout).
`SandboxTransferError`	A transfer command failed inside the sandbox.
`SandboxUnavailableError`	The provider could not run a transfer command at all (its result carries an `error_type`).

Functions

Name	Description
`_lands_clear`	Whether extracting into `directory` stays clear of `keep_out` as `upload_archive` describes.
`_pack`	-
`_prepare_restore_target`	Clear and open `target` (a directory) or open its parent (a file) as `upload_path` describes.
`_unpack`	Extract `archive` into `target`.
`_upload_and_extract`	Upload `archive` and extract it as root into `target` inside the sandbox, then remove the upload.
`download_dir`	Copy the contents of `source` inside the sandbox into the local `target` folder.
`download_path`	Copy a file or directory out of the sandbox; returns what it was, or `None` when absent.
`exec_as_root`	Run `command` as root, or as the default user when the sandbox cannot switch users.
`remote_kind`	`"dir"`, `"file"` or `None` for a path inside the sandbox.
`tar_extract_command`	The shell line that unpacks the uploaded archive `remote` into `target` with GNU or BusyBox tar.
`upload_archive`	Restore at `target` the archive another sandbox’s root `tar` produced (a download’s `archive_copy`).
`upload_dir`	Copy the contents of `source` into `target` inside the sandbox.
`upload_path`	Restore a collected file or directory into the sandbox at `target`.

Data

API

class nemo_gym.sandbox.transfer.SandboxProbeError()

Bases: RuntimeError

The sandbox could not answer a probe or run a command (lost container, provider timeout).

Not a SandboxTransferError: a caller that treats a failed transfer as “this artifact is not there” must not treat an outage the same way.

class nemo_gym.sandbox.transfer.SandboxTransferError()

Bases: RuntimeError

A transfer command failed inside the sandbox.

class nemo_gym.sandbox.transfer.SandboxUnavailableError()

Bases: RuntimeError

The provider could not run a transfer command at all (its result carries an error_type).

A timeout, a lost container or an exec the provider refused is Gym’s outage, not a property of the task or the sandbox’s files, so it is not a SandboxTransferError: a caller that scores a deterministic transfer failure (permission denied, cannot mkdir) lets this one propagate and the episode is retried.

nemo_gym.sandbox.transfer._lands_clear(
    sandbox: nemo_gym.sandbox.AsyncSandbox,
    directory: str,
    keep_out: tuple[str, ...]
) -> bool

async

Whether extracting into directory stays clear of keep_out as upload_archive describes.

The probe only reads: directory need not exist yet, and nothing is created, emptied or opened for it. A sandbox that answers the probe with nothing (a fake) leaves only the name to judge; one that cannot run it is an outage and raises.

nemo_gym.sandbox.transfer._pack(
    source: pathlib.Path,
    archive: pathlib.Path
) -> None

nemo_gym.sandbox.transfer._prepare_restore_target(
    sandbox: nemo_gym.sandbox.AsyncSandbox,
    is_dir: bool,
    target: str
) -> str

async

Clear and open target (a directory) or open its parent (a file) as upload_path describes.

Returns the directory an archive is then extracted into. The named path is what is cleared and opened: a link already there is left as a link, which the extraction then follows (see keep_out on upload_archive).

nemo_gym.sandbox.transfer._unpack(
    archive: pathlib.Path,
    target: pathlib.Path,
    names: tuple[str, ...] | None = None,
    skip_unsafe: bool = False,
    links: str = 'data'
) -> None

Extract archive into target.

links="data" applies tarfile.data_filter, which refuses links that point outside the extraction root: right for verifier output that is read on this host. links="keep" applies tarfile.tar_filter instead, which keeps symbolic links as links wherever they point (an agent’s virtualenv links to /usr/bin/python3): right for artifacts that only pass through this host on their way into another sandbox, where they are re-archived and never followed. Hard links are created on this host whatever the mode, so their target must stay inside target in both modes, as must every member’s own path, and a device node or FIFO is refused in both modes (tar_filter alone would create it on this host). In "keep" mode files and folders are made readable by their owner (as data_filter does), so a tree the agent left unreadable can still be re-archived; the host copy’s modes are a staging convenience, the modes the agent left are in the archive itself (see upload_archive).

nemo_gym.sandbox.transfer._upload_and_extract(
    sandbox: nemo_gym.sandbox.AsyncSandbox,
    archive: pathlib.Path,
    target: str,
    timeout_s: float,
    fixup: str = '',
    what: object = None
) -> None

async

Upload archive and extract it as root into target inside the sandbox, then remove the upload.

nemo_gym.sandbox.transfer.download_dir(
    sandbox: nemo_gym.sandbox.AsyncSandbox,
    source: str,
    target: pathlib.Path,
    timeout_s: float = 600,
    names: tuple[str, ...] | None = None,
    skip_unsafe: bool = False,
    links: str = 'data',
    exclude: tuple[str, ...] = (),
    archive_copy: pathlib.Path | None = None
) -> None

async

Copy the contents of source inside the sandbox into the local target folder.

links is passed to _unpack: "keep" for artifacts that are restored into another sandbox, "data" (the default) for files read on this host. exclude holds an artifact’s exclude rules, passed to tar --exclude exactly as Harbor passes them, unanchored: so node_modules leaves out every directory of that name at any depth, while ./cache matches only the top-level one because member names start with ./. archive_copy keeps the archive the sandbox’s root tar produced at that path, with the modes the sandbox held, for upload_archive; it stays only when the host copy unpacked, so a refused member (one that escapes, a device node) leaves no archive behind either.

nemo_gym.sandbox.transfer.download_path(
    sandbox: nemo_gym.sandbox.AsyncSandbox,
    source: str,
    target: pathlib.Path,
    timeout_s: float = 600,
    links: str = 'data',
    exclude: tuple[str, ...] = (),
    archive_copy: pathlib.Path | None = None
) -> str | None

async

Copy a file or directory out of the sandbox; returns what it was, or None when absent.

A single file travels through the same archive as a directory, so its mode bits (an executable the agent built) arrive with it; exclude applies to directories only. archive_copy is passed to download_dir: a file’s archive holds that one member, named as the file is.

nemo_gym.sandbox.transfer.exec_as_root(
    sandbox: nemo_gym.sandbox.AsyncSandbox,
    command: str,
    timeout_s: float
) -> nemo_gym.sandbox.providers.base.SandboxExecResult

async

Run command as root, or as the default user when the sandbox cannot switch users.

nemo_gym.sandbox.transfer.remote_kind(
    sandbox: nemo_gym.sandbox.AsyncSandbox,
    path: str
) -> str | None

async

"dir", "file" or None for a path inside the sandbox.

Only an answer counts as absence: a probe the sandbox could not run (a provider error, a lost container, a non-zero exit of the test itself) raises SandboxProbeError, so an outage is never read as “the agent wrote nothing”.

nemo_gym.sandbox.transfer.tar_extract_command(
    remote: str,
    target: str
) -> str

The shell line that unpacks the uploaded archive remote into target with GNU or BusyBox tar.

nemo_gym.sandbox.transfer.upload_archive(
    sandbox: nemo_gym.sandbox.AsyncSandbox,
    archive: pathlib.Path,
    kind: str,
    target: str,
    timeout_s: float = 600,
    keep_out: tuple[str, ...] = ()
) -> bool

async

Restore at target the archive another sandbox’s root tar produced (a download’s archive_copy).

kind is what download_path reported, "dir" or "file". The target is prepared as upload_path prepares it; then root tar extracts the archive there, so the files arrive with the modes the agent’s sandbox held (read-only, world-writable and set-id bits included), as Harbor’s docker cp round trip keeps them. None of those modes is applied on this host: the staging copy download_dir unpacked beside the archive went through _unpack’s filter, and the archive exists only because that copy was accepted. A file’s archive holds the one member named as the file is, so it is extracted into target’s parent, which target must name the same way.

keep_out names directories the restore must land clear of when a link diverts it: before anything touches the target, the directory the archive would be extracted into is resolved in the sandbox to its physical place (_lands_clear), and when that place is not the one named (a link an earlier restore left in the way, say) and is one of keep_out, inside one or above one, nothing is prepared or extracted, a warning says so and False is returned. The probe comes first because preparing the target is destructive: it empties the directory and creates what is missing, through whatever link sits in the path, with or without a trailing slash. A restore that lands where its name says is the caller’s to judge by that name. Returns True once the archive is extracted.

nemo_gym.sandbox.transfer.upload_dir(
    sandbox: nemo_gym.sandbox.AsyncSandbox,
    source: pathlib.Path,
    target: str,
    timeout_s: float = 600,
    make_readable: bool = False
) -> None

async

Copy the contents of source into target inside the sandbox.

File modes travel with the archive. make_readable additionally opens the tree to every user, for a solution the task user must read; a verifier tree keeps its modes because tests may check them.

nemo_gym.sandbox.transfer.upload_path(
    sandbox: nemo_gym.sandbox.AsyncSandbox,
    source: pathlib.Path,
    target: str,
    timeout_s: float = 600
) -> None

async

Restore a collected file or directory into the sandbox at target.

A directory replaces whatever the image ships at target (a file the agent deleted stays deleted) and is made world-writable, as Harbor does when it restores artifacts: verifiers drop privileges before running agent output and need to write next to it. A file lands in its parent, which is created world-writable when missing; an existing parent is made world-writable when the sandbox can (a+rwx keeps a sticky bit) and otherwise only has to be writable, so a root-owned /tmp keeps its mode and an image that cannot switch to root can still receive the file. Modes travel in both cases.

nemo_gym.sandbox.transfer.LOGGER = logging.getLogger(__name__)

nemo_gym.sandbox.transfer.TRANSFER_USER = 'root'

nemo_gym.sandbox.transfer._NO_ROOT_MARKERS = ('CAP_SETUID', 'identity switch', 'switching to uid=0', 'operation not permitted...

nemo_gym.sandbox.transfer._PHYSICAL_PROBE = 'p={quoted}; t=\'\'; while ! cd -P "$p" 2>/dev/null; do [ "$p" = / ] && exit 3; ...