> This page is for version 0.7.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.data.tasks.harbor.hub Resolve `harbor:[@]` through the Harbor registry. The registry is one JSON file listing datasets; each task entry names a git repository, a commit and a path. Fetching copies the task folders into Gym's pulled datasets folder, `environments/harbor/` under the working root, so that every server reads the same bytes and nothing downloads at run time. A `HEAD` pin is resolved to a commit before anything is copied. ## Module Contents ### Classes | Name | Description | | -------------------------------------------------------------------- | -------------------------------------------- | | [`HubError`](#nemo_gym-data-tasks-harbor-hub-HubError) | The reference cannot be resolved or fetched. | | [`HubRef`](#nemo_gym-data-tasks-harbor-hub-HubRef) | - | | [`RegistryDataset`](#nemo_gym-data-tasks-harbor-hub-RegistryDataset) | - | | [`RegistryTask`](#nemo_gym-data-tasks-harbor-hub-RegistryTask) | - | ### Functions | Name | Description | | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | [`_check_kept_folders`](#nemo_gym-data-tasks-harbor-hub-_check_kept_folders) | Hash the task folders a fetch keeps and refuse one whose content differs from the manifest. | | [`_git`](#nemo_gym-data-tasks-harbor-hub-_git) | - | | [`_numeric_version`](#nemo_gym-data-tasks-harbor-hub-_numeric_version) | - | | [`_parse_registry`](#nemo_gym-data-tasks-harbor-hub-_parse_registry) | - | | [`_read_manifest`](#nemo_gym-data-tasks-harbor-hub-_read_manifest) | The manifest a previous fetch wrote, or `None` (a malformed one is rewritten). | | [`_read_manifest_pins`](#nemo_gym-data-tasks-harbor-hub-_read_manifest_pins) | The `(git_url, commit, path)` pins a previous fetch recorded, or `{}`. | | [`_refuse_dropped_folders`](#nemo_gym-data-tasks-harbor-hub-_refuse_dropped_folders) | Refuse the task folders a previous fetch pinned for tasks that `dataset` no longer lists. | | [`_refuse_repinned_folders`](#nemo_gym-data-tasks-harbor-hub-_refuse_repinned_folders) | Refuse the task folders a previous fetch pinned at a commit other than the one `dataset` pins them at. | | [`_validate_task_path`](#nemo_gym-data-tasks-harbor-hub-_validate_task_path) | - | | [`_validate_task_tree`](#nemo_gym-data-tasks-harbor-hub-_validate_task_tree) | Refuse links that could import host files, before any task bytes are copied. | | [`_version_key`](#nemo_gym-data-tasks-harbor-hub-_version_key) | Total order over versions: numeric versions first (by component), then the rest by text. | | [`_write_manifest`](#nemo_gym-data-tasks-harbor-hub-_write_manifest) | Publish the pins with the digests of their folders, and the `reference` fetched; only pinned tasks are recorded. | | [`canonical_reference`](#nemo_gym-data-tasks-harbor-hub-canonical_reference) | The `harbor:` reference `target` as the fetchers record it, so two spellings of one reference compare equal. | | [`contained_path`](#nemo_gym-data-tasks-harbor-hub-contained_path) | Return `target` after checking that it resolves to a path strictly inside `root`; else raise `error`. | | [`datasets_dir`](#nemo_gym-data-tasks-harbor-hub-datasets_dir) | Where pulled datasets live: `$NEMO_GYM_DATASETS_DIR`, else `environments/harbor/` under the working root. | | [`declares_recorded_fetch`](#nemo_gym-data-tasks-harbor-hub-declares_recorded_fetch) | Whether the `harbor:` reference `declared` (a committed entry's `[gym] source`) names the fetch a manifest | | [`expected_folder`](#nemo_gym-data-tasks-harbor-hub-expected_folder) | Where a fetch of `target` lands when that follows from the reference alone, without resolving it. | | [`fetch_dataset`](#nemo_gym-data-tasks-harbor-hub-fetch_dataset) | Fetch tasks, retaining the recorded source pins of existing folders. | | [`fetch_ref`](#nemo_gym-data-tasks-harbor-hub-fetch_ref) | Resolve and fetch a `harbor:` reference; return the dataset folder. | | [`is_hub_ref`](#nemo_gym-data-tasks-harbor-hub-is_hub_ref) | - | | [`load_registry`](#nemo_gym-data-tasks-harbor-hub-load_registry) | Read the registry, validating downloads before atomically replacing the cache. | | [`refuse_unrecorded_folders`](#nemo_gym-data-tasks-harbor-hub-refuse_unrecorded_folders) | Refuse task folders under `folder` that neither the source names nor a previous fetch recorded. | | [`resolve_commit`](#nemo_gym-data-tasks-harbor-hub-resolve_commit) | Pin `HEAD` (or a branch name) to the commit it points at right now. | | [`resolve_dataset`](#nemo_gym-data-tasks-harbor-hub-resolve_dataset) | - | | [`resolved_folder`](#nemo_gym-data-tasks-harbor-hub-resolved_folder) | Where a fetch of `target` lands, resolving a registry reference whose version the registry picks. | | [`validate_git_url`](#nemo_gym-data-tasks-harbor-hub-validate_git_url) | Return `git_url` if it is an https, ssh or git URL (or `user@host:path`); else raise. | | [`validate_task_name`](#nemo_gym-data-tasks-harbor-hub-validate_task_name) | Require one folder component, reserving the dataset manifest's name; raise `error` otherwise. | ### Data [`DATASETS_DIR_ENV`](#nemo_gym-data-tasks-harbor-hub-DATASETS_DIR_ENV) [`REF_PREFIX`](#nemo_gym-data-tasks-harbor-hub-REF_PREFIX) [`REGISTRY_SOURCE`](#nemo_gym-data-tasks-harbor-hub-REGISTRY_SOURCE) [`REGISTRY_URL`](#nemo_gym-data-tasks-harbor-hub-REGISTRY_URL) [`_GIT_URL_SCHEME`](#nemo_gym-data-tasks-harbor-hub-_GIT_URL_SCHEME) [`_GIT_URL_SCP`](#nemo_gym-data-tasks-harbor-hub-_GIT_URL_SCP) [`_NAME`](#nemo_gym-data-tasks-harbor-hub-_NAME) [`_NAME_SEPARATORS`](#nemo_gym-data-tasks-harbor-hub-_NAME_SEPARATORS) [`_PACKAGE_NAME`](#nemo_gym-data-tasks-harbor-hub-_PACKAGE_NAME) [`_SHA`](#nemo_gym-data-tasks-harbor-hub-_SHA) [`_STAGING_PREFIX`](#nemo_gym-data-tasks-harbor-hub-_STAGING_PREFIX) [`_UNSAFE`](#nemo_gym-data-tasks-harbor-hub-_UNSAFE) [`logger`](#nemo_gym-data-tasks-harbor-hub-logger) ### API ```python class nemo_gym.data.tasks.harbor.hub.HubError() ``` **Bases:** `ValueError` The reference cannot be resolved or fetched. ```python class nemo_gym.data.tasks.harbor.hub.HubRef( name: str, version: str | None ) ``` Dataclass **`is_package`** `bool` `org/name` references live in Harbor's package store, not in `registry.json`. --- **`name`** `str` --- **`version`** `str | None` --- ```python nemo_gym.data.tasks.harbor.hub.HubRef.parse( target: str ) -> nemo_gym.data.tasks.harbor.hub.HubRef ``` classmethod ```python class nemo_gym.data.tasks.harbor.hub.RegistryDataset( name: str, version: str, description: str, tasks: tuple[nemo_gym.data.tasks.harbor.hub.RegistryTask, ...] ) ``` Dataclass **`description`** `str` --- **`folder_name`** `str` `-`, so two versions of one dataset never share a folder. --- **`name`** `str` --- **`reference`** `str` The reference a fetch of this dataset records in its manifest: `harbor:@`, the version resolved, so a fetch of `harbor:` records which version the registry picked. --- **`tasks`** `tuple[RegistryTask, ...]` --- **`version`** `str` --- ```python class nemo_gym.data.tasks.harbor.hub.RegistryTask( name: str, git_url: str, git_commit_id: str, path: str ) ``` Dataclass **`git_commit_id`** `str` --- **`git_url`** `str` --- **`name`** `str` --- **`path`** `str` --- ```python nemo_gym.data.tasks.harbor.hub._check_kept_folders( folder: pathlib.Path, kept: list[str], previous: nemo_gym.data.tasks.harbor.manifest.Manifest | None, digests: dict[str, str] ) -> None ``` Hash the task folders a fetch keeps and refuse one whose content differs from the manifest. A rewritten manifest records what is on disk, so without this check a fetch that adds a task would bless a hand-edited folder next to it. A folder recorded without a digest (a manifest written before digests existed) is accepted and recorded now. ```python nemo_gym.data.tasks.harbor.hub._git( args: str = (), cwd: pathlib.Path | None = None ) -> str ``` ```python nemo_gym.data.tasks.harbor.hub._numeric_version( version: str ) -> tuple[int, ...] | None ``` ```python nemo_gym.data.tasks.harbor.hub._parse_registry( payload: bytes ) -> list[nemo_gym.data.tasks.harbor.hub.RegistryDataset] ``` ```python nemo_gym.data.tasks.harbor.hub._read_manifest( folder: pathlib.Path ) -> nemo_gym.data.tasks.harbor.manifest.Manifest | None ``` The manifest a previous fetch wrote, or `None` (a malformed one is rewritten). One a newer Gym wrote (a schema this Gym does not read) raises instead, so the fetch stops with the folder untouched rather than replacing a manifest this Gym cannot read in full. ```python nemo_gym.data.tasks.harbor.hub._read_manifest_pins( manifest: nemo_gym.data.tasks.harbor.manifest.Manifest | None ) -> dict[str, tuple[str, str, str]] ``` The `(git_url, commit, path)` pins a previous fetch recorded, or `{}`. ```python nemo_gym.data.tasks.harbor.hub._refuse_dropped_folders( folder: pathlib.Path, previous: dict[str, tuple[str, str, str]], names: set[str], dataset: nemo_gym.data.tasks.harbor.hub.RegistryDataset ) -> None ``` Refuse the task folders a previous fetch pinned for tasks that `dataset` no longer lists. The manifest a fetch writes pins the dataset's tasks alone, so such a folder would stay on disk unrecorded and the next prepare would refuse the whole folder as edited by hand, after a pull that reported success. The folder holds another version's task (a committed entry's `source` moved to a version with fewer tasks, the registry revised this one, or `harbor:` now resolves to such a version), so it is refused here, named, with what to move aside, before anything is cloned or rewritten: the registry's counterpart of the store's refusal of a manifest written for another version (`tasks removed`, `nemo_gym.data.tasks.harbor.package_store`). ```python nemo_gym.data.tasks.harbor.hub._refuse_repinned_folders( folder: pathlib.Path, repinned: dict[str, tuple[str, str]], dataset: nemo_gym.data.tasks.harbor.hub.RegistryDataset ) -> None ``` Refuse the task folders a previous fetch pinned at a commit other than the one `dataset` pins them at. A recorded pin is kept when the registry still names the same repository and path, so a version that pins a task at another commit (a committed entry's `source` moved to it, `harbor:` now resolves to it, or the registry revised this version in place) used to keep the folder of the commit fetched before and record it under the new reference: the other version's content, run by name as this one's. Harbor caches a git task by `(git_url, git_commit_id, path)` and downloads a task pinned at another commit afresh (`harbor.models.task.id.GitTaskId`), so the folder is refused here, named with both commits, before anything is cloned or rewritten; moved aside, the fetch clones the commit the registry pins. A registry pin that is not a commit (`HEAD`, a branch) keeps the recorded one while the reference stays the same: the fetch pinned what it pointed at then. A fetch of another reference into the folder (the version changed) resolves such a pin afresh, as the first fetch of that version into a folder of its own did before entries shared one, and Harbor never serves a task without a resolved commit from its cache (`harbor.tasks.client`); a commit that moved since is refused here the same way, named with both commits. ```python nemo_gym.data.tasks.harbor.hub._validate_task_path( path: str ) -> None ``` ```python nemo_gym.data.tasks.harbor.hub._validate_task_tree( source: pathlib.Path ) -> None ``` Refuse links that could import host files, before any task bytes are copied. ```python nemo_gym.data.tasks.harbor.hub._version_key( version: str ) -> tuple ``` Total order over versions: numeric versions first (by component), then the rest by text. ```python nemo_gym.data.tasks.harbor.hub._write_manifest( folder: pathlib.Path, pins: dict[str, tuple[str, str, str]], digests: dict[str, str], reference: str ) -> None ``` Publish the pins with the digests of their folders, and the `reference` fetched; only pinned tasks are recorded. `digests` caches the hashing across the per-task checkpoints, and naming the pinned tasks keeps each checkpoint from scanning the dataset folder again: a folder the fetch did not pin stays unrecorded, so prepare refuses it instead of running it as part of the dataset. ```python nemo_gym.data.tasks.harbor.hub.canonical_reference( target: str ) -> str ``` The `harbor:` reference `target` as the fetchers record it, so two spellings of one reference compare equal. A package-store reference is read as the store reads it (`reference`: `harbor:org/name` is `@latest`, a digest is its `sha256:` form). A registry reference is itself: a version it names is the version, and `harbor:` without one is the registry's pick, another reference than any version it may resolve to. A malformed reference raises `HubError`, as `HubRef.parse` does. ```python nemo_gym.data.tasks.harbor.hub.contained_path( target: pathlib.Path, root: pathlib.Path, error: type[ValueError] = HubError ) -> pathlib.Path ``` Return `target` after checking that it resolves to a path strictly inside `root`; else raise `error`. Called before anything under a dataset folder is removed or replaced, so a spoofed name or a symlink pointing elsewhere can never make a fetch touch files outside the dataset. ```python nemo_gym.data.tasks.harbor.hub.datasets_dir() -> pathlib.Path ``` Where pulled datasets live: `$NEMO_GYM_DATASETS_DIR`, else `environments/harbor/` under the working root. That root is `nemo_gym.WORKING_DIR`: the Gym checkout of an editable install, the working directory of a package install, whose `PARENT_DIR` is `site-packages` (shared, often read-only) and could hold neither a dataset nor the registry cache. Each fetched dataset is one folder under it (`environments/harbor//`), gitignored there next to the committed per-dataset entries, which keep only their README, YAML files and `dataset.toml`. An entry that declares a `[gym] source` is pulled into its own folder when that folder is under the working root and not installed, else (an entry a package install ships under `site-packages`, a project-local `.venv` included, or one an extra search root commits, or any entry when `NEMO_GYM_DATASETS_DIR` is set) into `//` (`nemo_gym.data.tasks.harbor.entries`). ```python nemo_gym.data.tasks.harbor.hub.declares_recorded_fetch( declared: str, recorded: str ) -> bool ``` Whether the `harbor:` reference `declared` (a committed entry's `[gym] source`) names the fetch a manifest records as `recorded` (`ref`), without asking the hub. A package-store reference matches by package and by tag, `latest` or digest, read as the store reads them (`reference`: `harbor:org/name` is `@latest`, a digest is its `sha256:` form). A registry reference matches by dataset and, when it pins one, by version: the version an unpinned `harbor:` gets is the registry's pick, which the record holds and only a pull can revise. A record that does not parse as a reference matches nothing. ```python nemo_gym.data.tasks.harbor.hub.expected_folder( target: str, root: pathlib.Path | None = None ) -> pathlib.Path | None ``` Where a fetch of `target` lands when that follows from the reference alone, without resolving it. A package-store dataset's folder is named from its reference (`` for `latest`, `-`, `-<12 digest chars>` for a digest pin, `PackageRef.folder_name`), and a registry dataset's from its name and version; `None` for a registry reference whose version the registry picks. `gym dataset pull` reads the folder's manifest here before the fetch to report an unchanged dataset. ```python nemo_gym.data.tasks.harbor.hub.fetch_dataset( dataset: nemo_gym.data.tasks.harbor.hub.RegistryDataset, root: pathlib.Path, folder: pathlib.Path | None = None ) -> pathlib.Path ``` Fetch tasks, retaining the recorded source pins of existing folders. Publish a pin after each task, so a later failure leaves completed tasks reusable. A folder without a trustworthy pin (including interruption before pin publication) is refused; the user can move it aside and retry without losing local edits. The manifest records each folder's digest beside its pin; when a fetch adds tasks to an existing folder, the kept folders must still match their recorded digests. The tasks land in `root/-/` (`RegistryDataset.folder_name`) unless `folder` names the dataset folder itself: a committed entry under `environments/harbor/` whose `dataset.toml` declares this dataset as its `[gym] source` (`nemo_gym.data.tasks.harbor.entries`). The folder's other files (the entry's README, YAML overlays and `dataset.toml`) are not task folders and stay as they are. A folder holding a task of another version (the entry's source moved to a version that no longer lists it, or that pins it at another commit, a `HEAD` pin that moved since included) is refused with the folder named, as the store's fetch refuses a manifest written for another version (`_refuse_dropped_folders`, `_refuse_repinned_folders`), never left unrecorded beside the fetched tasks or recorded as this version's. ```python nemo_gym.data.tasks.harbor.hub.fetch_ref( target: str, root: pathlib.Path | None = None, refresh_registry: bool = False, folder: pathlib.Path | None = None ) -> pathlib.Path ``` Resolve and fetch a `harbor:` reference; return the dataset folder. The dataset lands in its own folder under `root` (`datasets_dir` unless given), named after the dataset and its version, or in `folder` when a caller names the destination: the folder of a committed entry that declares the reference as its `[gym] source` (`nemo_gym.data.tasks.harbor.entries.plan_pull` decides that; this function does not read entries). The registry cache stays under `root/.harbor/`. A package-store fetch refuses a task folder whose content no longer matches the store (see `nemo_gym.data.tasks.harbor.package_store.fetch_package_dataset`). Whatever the store fetch refuses or cannot read (a `PackageStoreError`) is raised as a `HubError`, so a caller handles one error type for a `harbor:` reference of either source. ```python nemo_gym.data.tasks.harbor.hub.is_hub_ref( target: str ) -> bool ``` ```python nemo_gym.data.tasks.harbor.hub.load_registry( cache_dir: pathlib.Path, refresh: bool = False ) -> list[nemo_gym.data.tasks.harbor.hub.RegistryDataset] ``` Read the registry, validating downloads before atomically replacing the cache. A damaged cache raises `HubError` with its path and recovery instructions. Failed downloads or updates leave any previous cache intact. ```python nemo_gym.data.tasks.harbor.hub.refuse_unrecorded_folders( folder: pathlib.Path, known: set[str], what: str, error: type[ValueError] = HubError ) -> None ``` Refuse task folders under `folder` that neither the source names nor a previous fetch recorded. The manifest a fetch writes is the record prepare trusts, so a folder nobody fetched must not sit beside the fetched ones when it is written: it would run and score as part of the dataset. ```python nemo_gym.data.tasks.harbor.hub.resolve_commit( git_url: str, git_commit_id: str, cache: dict[tuple[str, str], str] | None = None ) -> str ``` Pin `HEAD` (or a branch name) to the commit it points at right now. `cache` memoizes the answer per `(git_url, ref)` so one fetch asks each remote once. ```python nemo_gym.data.tasks.harbor.hub.resolve_dataset( ref: nemo_gym.data.tasks.harbor.hub.HubRef, registry: list[nemo_gym.data.tasks.harbor.hub.RegistryDataset] ) -> nemo_gym.data.tasks.harbor.hub.RegistryDataset ``` ```python nemo_gym.data.tasks.harbor.hub.resolved_folder( target: str, root: pathlib.Path | None = None, refresh_registry: bool = False ) -> pathlib.Path ``` Where a fetch of `target` lands, resolving a registry reference whose version the registry picks. `expected_folder` names the folder when the reference alone does; for `harbor:` without a version the registry's cache under `root/.harbor/` (downloaded, or refreshed when `refresh_registry`) is read and the version it picks names the folder, as `fetch_ref` will. `gym dataset pull` resolves such a reference here, before the fetch, so it can read the folder's manifest and report an unchanged dataset. ```python nemo_gym.data.tasks.harbor.hub.validate_git_url( git_url: str ) -> str ``` Return `git_url` if it is an https, ssh or git URL (or `user@host:path`); else raise. ```python nemo_gym.data.tasks.harbor.hub.validate_task_name( name: str, error: type[ValueError] = HubError ) -> str ``` Require one folder component, reserving the dataset manifest's name; raise `error` otherwise. Task names arrive from `registry.json` and from package-store replies, and the fetchers use them as path components, so empty names, `.`, anything containing `..`, a path separator or NUL is refused. Other characters, such as `$`, which real Harbor task names contain, are accepted. ```python nemo_gym.data.tasks.harbor.hub.DATASETS_DIR_ENV = 'NEMO_GYM_DATASETS_DIR' ``` ```python nemo_gym.data.tasks.harbor.hub.REF_PREFIX = 'harbor:' ``` ```python nemo_gym.data.tasks.harbor.hub.REGISTRY_SOURCE = 'harbor-registry' ``` ```python nemo_gym.data.tasks.harbor.hub.REGISTRY_URL = 'https://raw.githubusercontent.com/harbor-framework/harbor/main/registry.json' ``` ```python nemo_gym.data.tasks.harbor.hub._GIT_URL_SCHEME = re.compile('(https|ssh|git)://[A-Za-z0-9]', re.IGNORECASE) ``` ```python nemo_gym.data.tasks.harbor.hub._GIT_URL_SCP = re.compile('[A-Za-z0-9][A-Za-z0-9._-]*@[A-Za-z0-9][A-Za-z0-9.-]*:(?!-)[^:]*') ``` ```python nemo_gym.data.tasks.harbor.hub._NAME = re.compile('[A-Za-z0-9][A-Za-z0-9._-]*') ``` ```python nemo_gym.data.tasks.harbor.hub._NAME_SEPARATORS = ('/', '\\', '\x00') ``` ```python nemo_gym.data.tasks.harbor.hub._PACKAGE_NAME = re.compile('[A-Za-z0-9][A-Za-z0-9._-]*/[A-Za-z0-9][A-Za-z0-9._-]*') ``` ```python nemo_gym.data.tasks.harbor.hub._SHA = re.compile('[0-9a-f]{40}') ``` ```python nemo_gym.data.tasks.harbor.hub._STAGING_PREFIX = '.nemo-gym-harbor-' ``` ```python nemo_gym.data.tasks.harbor.hub._UNSAFE = re.compile('[^A-Za-z0-9._-]') ``` ```python nemo_gym.data.tasks.harbor.hub.logger = logging.getLogger(__name__) ```