nemo_gym.sandbox.transfer

Bases: RuntimeError

The sandbox could not answer a probe or run a command (lost container, provider timeout).

Not a SandboxTransferError: a caller that treats a failed transfer as “this artifact is not there” must not treat an outage the same way.

Bases: RuntimeError

A transfer command failed inside the sandbox.

Bases: RuntimeError

The provider could not run a transfer command at all (its result carries an error_type).

A timeout, a lost container or an exec the provider refused is Gym’s outage, not a property of the task or the sandbox’s files, so it is not a SandboxTransferError: a caller that scores a deterministic transfer failure (permission denied, cannot mkdir) lets this one propagate and the episode is retried.

Whether extracting into directory stays clear of keep_out as upload_archive describes.

The probe only reads: directory need not exist yet, and nothing is created, emptied or opened for it. A sandbox that answers the probe with nothing (a fake) leaves only the name to judge; one that cannot run it is an outage and raises.

Clear and open target (a directory) or open its parent (a file) as upload_path describes.

Returns the directory an archive is then extracted into. The named path is what is cleared and opened: a link already there is left as a link, which the extraction then follows (see keep_out on upload_archive).

Extract archive into target.

protect names top-level entries of target that belong to this host (a transcript copy, a staging folder with its manifest): a member at or under one of them is left out of the extraction, with a warning, so what the sandbox holds under the same name (an agent-planted file in a folder the agent could write) never replaces the host’s record. links="data" applies tarfile.data_filter, which refuses links that point outside the extraction root: right for verifier output that is read on this host. links="keep" applies tarfile.tar_filter instead, which keeps symbolic links as links wherever they point (an agent’s virtualenv links to /usr/bin/python3): right for artifacts that only pass through this host on their way into another sandbox, where they are re-archived and never followed. Hard links are created on this host whatever the mode, so their target must stay inside target in both modes, as must every member’s own path, and a device node or FIFO is refused in both modes (tar_filter alone would create it on this host). In "keep" mode files and folders are made readable by their owner (as data_filter does), so a tree the agent left unreadable can still be re-archived; the host copy’s modes are a staging convenience, the modes the agent left are in the archive itself (see upload_archive).

Upload archive and extract it as root into target inside the sandbox, then remove the upload.

Copy the contents of source inside the sandbox into the local target folder.

links is passed to _unpack: "keep" for artifacts that are restored into another sandbox, "data" (the default) for files read on this host, as is protect: top-level names under target this host owns, which no member of the archive may write (see there). exclude holds an artifact’s exclude rules, passed to tar --exclude exactly as Harbor passes them, unanchored: so node_modules leaves out every directory of that name at any depth, while ./cache matches only the top-level one because member names start with ./. archive_copy keeps the archive the sandbox’s root tar produced at that path, with the modes the sandbox held, for upload_archive; it stays only when the host copy unpacked, so a refused member (one that escapes, a device node) leaves no archive behind either.

Copy a file or directory out of the sandbox; returns what it was, or None when absent.

A single file travels through the same archive as a directory, so its mode bits (an executable the agent built) arrive with it; exclude applies to directories only. archive_copy is passed to download_dir: a file’s archive holds that one member, named as the file is.

Run command as root, or as the default user when the sandbox cannot switch users.

"dir", "file" or None for a path inside the sandbox.

Only an answer counts as absence: a probe the sandbox could not run (a provider error, a lost container, a non-zero exit of the test itself) raises SandboxProbeError, so an outage is never read as “the agent wrote nothing”.

The shell line that unpacks the uploaded archive remote into target with GNU or BusyBox tar.

With replace_links, a symbolic link found at the path of any member of the archive is removed first, in the same shell line, before the extraction: the tars differ on what a link in the way means. GNU tar replaces a link at a directory member’s path with the directory; BusyBox tar (Alpine and other minimal images) keeps the link and writes the member’s children through it, wherever it points, so an archive merged as root into a tree another user has written to (a step’s workdir/ into the agent’s working directory) would land where that user’s link says. Harbor’s docker cp removes the entry in the way (moby’s archive.Unpack), as GNU tar does; the pre-clearing gives both tars that outcome. The members are read as tar -t prints them, so a member whose name holds a newline (GNU tar escapes it in the listing) is not matched; a link at the target itself is followed by either tar, as docker cp resolves its destination.

The clearing and the extraction are two commands, not one operation: a process still running in the target’s tree (one a previous turn’s agent left behind in the working directory it owns) can plant a link again at a member’s path after the loop removed it and before tar -x reaches it, and BusyBox tar then writes that member through it as root. The pre-clearing gives BusyBox tar GNU tar’s outcome against a link already in place, not against one re-planted during the extraction; GNU tar and docker cp remove a link in the way right before they create each member and carry a window of the same shape, so Harbor’s merge shares it. Unpacking into a root-owned staging folder and moving the entries over would not close it: a directory member merges into a directory the previous agents may already have at that name, so the move is a recursive copy through the same user-owned tree, with the same window at every level, and a link at the destination that points at a directory sends mv into that directory. What such a process gains is where the archive’s own members land, not their content. The window is noted here rather than closed.

Restore at target the archive another sandbox’s root tar produced (a download’s archive_copy).

kind is what download_path reported, "dir" or "file". The target is prepared as upload_path prepares it; then root tar extracts the archive there, so the files arrive with the modes the agent’s sandbox held (read-only, world-writable and set-id bits included), as Harbor’s docker cp round trip keeps them. None of those modes is applied on this host: the staging copy download_dir unpacked beside the archive went through _unpack’s filter, and the archive exists only because that copy was accepted. A file’s archive holds the one member named as the file is, so it is extracted into target’s parent, which target must name the same way.

keep_out names directories the restore must land clear of when a link diverts it: before anything touches the target, the directory the archive would be extracted into is resolved in the sandbox to its physical place (_lands_clear), and when that place is not the one named (a link an earlier restore left in the way, say) and is one of keep_out, inside one or above one, nothing is prepared or extracted, a warning says so and False is returned. The probe comes first because preparing the target is destructive: it empties the directory and creates what is missing, through whatever link sits in the path, with or without a trailing slash. A restore that lands where its name says is the caller’s to judge by that name. Returns True once the archive is extracted.

Copy the contents of source into target inside the sandbox.

File modes travel with the archive. make_readable additionally opens the tree to every user, for a solution the task user must read; a verifier tree keeps its modes because tests may check them. ignore is called with each entry’s path relative to source (posix form); an entry it names is left out, its subtree with it. The environment server passes the task digest’s rule (nemo_gym.data.tasks.harbor.digest.is_ignored) for a task’s own folders, so a tests/helper.pyc or an environment/__pycache__ that the digest skips never reaches the sandbox; artifacts restored by upload_path travel whole. replace_links is for a merge into a tree another user has written to: a symbolic link at any member’s path is removed, as root, before the extraction, so the member lands where its name says on BusyBox tar as it does on GNU tar (tar_extract_command).

Restore a collected file or directory into the sandbox at target.

A directory replaces whatever the image ships at target (a file the agent deleted stays deleted) and is made world-writable, as Harbor does when it restores artifacts: verifiers drop privileges before running agent output and need to write next to it. A file lands in its parent, which is created world-writable when missing; an existing parent is made world-writable when the sandbox can (a+rwx keeps a sticky bit) and otherwise only has to be writable, so a root-owned /tmp keeps its mode and an image that cannot switch to root can still receive the file. Modes travel in both cases.