nemo_gym.data.tasks.harbor.hub

Bases: ValueError

The reference cannot be resolved or fetched.

Hash the task folders a fetch keeps and refuse one whose content differs from the manifest.

A rewritten manifest records what is on disk, so without this check a fetch that adds a task would bless a hand-edited folder next to it. A folder recorded without a digest (a manifest written before digests existed) is accepted and recorded now.

The manifest a previous fetch wrote, or None (a malformed one is rewritten).

One a newer Gym wrote (a schema this Gym does not read) raises instead, so the fetch stops with the folder untouched rather than replacing a manifest this Gym cannot read in full.

The (git_url, commit, path) pins a previous fetch recorded, or {}.

Refuse the task folders a previous fetch pinned for tasks that dataset no longer lists.

The manifest a fetch writes pins the dataset’s tasks alone, so such a folder would stay on disk unrecorded and the next prepare would refuse the whole folder as edited by hand, after a pull that reported success. The folder holds another version’s task (a committed entry’s source moved to a version with fewer tasks, the registry revised this one, or harbor:<dataset> now resolves to such a version), so it is refused here, named, with what to move aside, before anything is cloned or rewritten: the registry’s counterpart of the store’s refusal of a manifest written for another version (tasks removed, nemo_gym.data.tasks.harbor.package_store).

Refuse the task folders a previous fetch pinned at a commit other than the one dataset pins them at.

A recorded pin is kept when the registry still names the same repository and path, so a version that pins a task at another commit (a committed entry’s source moved to it, harbor:<dataset> now resolves to it, or the registry revised this version in place) used to keep the folder of the commit fetched before and record it under the new reference: the other version’s content, run by name as this one’s. Harbor caches a git task by (git_url, git_commit_id, path) and downloads a task pinned at another commit afresh (harbor.models.task.id.GitTaskId), so the folder is refused here, named with both commits, before anything is cloned or rewritten; moved aside, the fetch clones the commit the registry pins. A registry pin that is not a commit (HEAD, a branch) keeps the recorded one while the reference stays the same: the fetch pinned what it pointed at then. A fetch of another reference into the folder (the version changed) resolves such a pin afresh, as the first fetch of that version into a folder of its own did before entries shared one, and Harbor never serves a task without a resolved commit from its cache (harbor.tasks.client); a commit that moved since is refused here the same way, named with both commits.

Refuse links that could import host files, before any task bytes are copied.

Total order over versions: numeric versions first (by component), then the rest by text.

Publish the pins with the digests of their folders, and the reference fetched; only pinned tasks are recorded.

digests caches the hashing across the per-task checkpoints, and naming the pinned tasks keeps each checkpoint from scanning the dataset folder again: a folder the fetch did not pin stays unrecorded, so prepare refuses it instead of running it as part of the dataset.

The harbor: reference target as the fetchers record it, so two spellings of one reference compare equal.

A package-store reference is read as the store reads it (reference: harbor:org/name is @latest, a digest is its sha256: form). A registry reference is itself: a version it names is the version, and harbor:<dataset> without one is the registry’s pick, another reference than any version it may resolve to. A malformed reference raises HubError, as HubRef.parse does.

Return target after checking that it resolves to a path strictly inside root; else raise error.

Called before anything under a dataset folder is removed or replaced, so a spoofed name or a symlink pointing elsewhere can never make a fetch touch files outside the dataset.

Where pulled datasets live: $NEMO_GYM_DATASETS_DIR, else environments/harbor/ under the working root.

That root is nemo_gym.WORKING_DIR: the Gym checkout of an editable install, the working directory of a package install, whose PARENT_DIR is site-packages (shared, often read-only) and could hold neither a dataset nor the registry cache.

Each fetched dataset is one folder under it (environments/harbor/<dataset>/), gitignored there next to the committed per-dataset entries, which keep only their README, YAML files and dataset.toml. An entry that declares a [gym] source is pulled into its own folder when that folder is under the working root and not installed, else (an entry a package install ships under site-packages, a project-local .venv included, or one an extra search root commits, or any entry when NEMO_GYM_DATASETS_DIR is set) into <datasets dir>/<entry>/ (nemo_gym.data.tasks.harbor.entries).

Whether the harbor: reference declared (a committed entry’s [gym] source) names the fetch a manifest records as recorded (ref), without asking the hub.

A package-store reference matches by package and by tag, latest or digest, read as the store reads them (reference: harbor:org/name is @latest, a digest is its sha256: form). A registry reference matches by dataset and, when it pins one, by version: the version an unpinned harbor:<dataset> gets is the registry’s pick, which the record holds and only a pull can revise. A record that does not parse as a reference matches nothing.

Where a fetch of target lands when that follows from the reference alone, without resolving it.

A package-store dataset’s folder is named from its reference (<name> for latest, <name>-<tag>, <name>-<12 digest chars> for a digest pin, PackageRef.folder_name), and a registry dataset’s from its name and version; None for a registry reference whose version the registry picks. gym dataset pull reads the folder’s manifest here before the fetch to report an unchanged dataset.

Fetch tasks, retaining the recorded source pins of existing folders.

Publish a pin after each task, so a later failure leaves completed tasks reusable. A folder without a trustworthy pin (including interruption before pin publication) is refused; the user can move it aside and retry without losing local edits. The manifest records each folder’s digest beside its pin; when a fetch adds tasks to an existing folder, the kept folders must still match their recorded digests.

The tasks land in root/<name>-<version>/ (RegistryDataset.folder_name) unless folder names the dataset folder itself: a committed entry under environments/harbor/ whose dataset.toml declares this dataset as its [gym] source (nemo_gym.data.tasks.harbor.entries). The folder’s other files (the entry’s README, YAML overlays and dataset.toml) are not task folders and stay as they are. A folder holding a task of another version (the entry’s source moved to a version that no longer lists it, or that pins it at another commit, a HEAD pin that moved since included) is refused with the folder named, as the store’s fetch refuses a manifest written for another version (_refuse_dropped_folders, _refuse_repinned_folders), never left unrecorded beside the fetched tasks or recorded as this version’s.

Resolve and fetch a harbor: reference; return the dataset folder.

The dataset lands in its own folder under root (datasets_dir unless given), named after the dataset and its version, or in folder when a caller names the destination: the folder of a committed entry that declares the reference as its [gym] source (nemo_gym.data.tasks.harbor.entries.plan_pull decides that; this function does not read entries). The registry cache stays under root/.harbor/.

A package-store fetch refuses a task folder whose content no longer matches the store (see nemo_gym.data.tasks.harbor.package_store.fetch_package_dataset). Whatever the store fetch refuses or cannot read (a PackageStoreError) is raised as a HubError, so a caller handles one error type for a harbor: reference of either source.

Read the registry, validating downloads before atomically replacing the cache.

A damaged cache raises HubError with its path and recovery instructions. Failed downloads or updates leave any previous cache intact.

Refuse task folders under folder that neither the source names nor a previous fetch recorded.

The manifest a fetch writes is the record prepare trusts, so a folder nobody fetched must not sit beside the fetched ones when it is written: it would run and score as part of the dataset.

Pin HEAD (or a branch name) to the commit it points at right now.

cache memoizes the answer per (git_url, ref) so one fetch asks each remote once.

Where a fetch of target lands, resolving a registry reference whose version the registry picks.

expected_folder names the folder when the reference alone does; for harbor:<dataset> without a version the registry’s cache under root/.harbor/ (downloaded, or refreshed when refresh_registry) is read and the version it picks names the folder, as fetch_ref will. gym dataset pull resolves such a reference here, before the fetch, so it can read the folder’s manifest and report an unchanged dataset.

Return git_url if it is an https, ssh or git URL (or user@host:path); else raise.

Require one folder component, reserving the dataset manifest’s name; raise error otherwise.

Task names arrive from registry.json and from package-store replies, and the fetchers use them as path components, so empty names, ., anything containing .., a path separator or NUL is refused. Other characters, such as $, which real Harbor task names contain, are accepted.