> This page is for version 0.5.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.data.tasks.harbor.entries Committed dataset entries under `environments/harbor//` and where a pull lands. An entry is the folder a Gym checkout commits for one dataset: its README, deployment YAML overlays and a `dataset.toml` whose `[gym]` table may declare `source`, the hub reference the task folders come from (`harbor:/@` for the package store, `harbor:@` for the registry). `gym dataset pull ` fetches that reference into the entry's own folder, where the pulled task folders and `manifest.toml` are gitignored next to the committed files, and a pull of the reference itself lands there too, so one dataset never lives in two sibling folders. A reference no entry declares lands in `/-/` as before. `gym dataset validate ` and `gym eval run ` resolve the name to that folder without fetching (`entry_target`, where an existing path of that name keeps meaning the path, since those commands take paths; the pull takes none, so a bare name is always the entry there). An entry is pulled into its own folder when that folder is under the working root (`nemo_gym.WORKING_DIR`: the Gym checkout of an editable install, the working directory of a package install) and is not installed. An installed entry, one a package install ships under `site-packages` or another Python installation prefix (`_installed_entry`; shared, often read-only, and under the working root too when the project keeps its venv there), and an entry committed outside the working root (one a `NEMO_GYM_EXTRA_ROOTS` root commits) are pulled into `//` (`datasets_dir`, the writable default every other pull uses), and `NEMO_GYM_DATASETS_DIR` moves every pulled dataset there (an entry without a source is never pulled, so it stays at its folder). An entry's tasks then live in `//` with a copy of the entry's Gym layer beside them, its `dataset.toml` and whatever dataset-level `task_data.py` or `verifier.py` it commits (`gym_layer`), kept current with the entry's files (`prepare_destination`), so the files that shape the tasks, and the dataset digest that covers them, are the same wherever the pull landed. That copy folder, told by the `LAYER_RECORD_FILE` the placing writes beside the copies, is never itself an entry: lookup by name and by reference (`find_entry`, `committed_entries`) skips it and goes on to the root that commits the entry, so the pull keeps following the entry's files, while a folder of the entry's name a user committed in an earlier root (no record file) keeps precedence, as any shadowing component does. ## Module Contents ### Classes | Name | Description | | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------- | | [`DatasetEntry`](#nemo_gym-data-tasks-harbor-entries-DatasetEntry) | A committed entry: its name (the folder's), its folder and the `[gym] source` it declares, if any. | | [`PullPlan`](#nemo_gym-data-tasks-harbor-entries-PullPlan) | What a pull of a target fetches and where: the hub reference, and the entry it lands in, if one. | ### Functions | Name | Description | | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- | | [`_committed_folder`](#nemo_gym-data-tasks-harbor-entries-_committed_folder) | The folder `environments/harbor/<name>` of the first search root that has one, or `None`. | | [`_entry`](#nemo_gym-data-tasks-harbor-entries-_entry) | The entry at `folder`; a `dataset.toml` that does not read raises `ValueError` with the file named. | | [`_install_prefixes`](#nemo_gym-data-tasks-harbor-entries-_install_prefixes) | The running interpreter's installation prefixes: `purelib` and `platlib`, its site-packages folders and | | [`_installed_entry`](#nemo_gym-data-tasks-harbor-entries-_installed_entry) | Whether `folder` is installed: it lies under a Python installation prefix, so it is a package's, shared and | | [`_layer_digests`](#nemo_gym-data-tasks-harbor-entries-_layer_digests) | The digest of each Gym-layer file by its relative path, as the record of the copies holds them. | | [`_pull_copy`](#nemo_gym-data-tasks-harbor-entries-_pull_copy) | Whether `folder` is the copy a pull placed at `<datasets dir>/<entry>/` for an entry elsewhere, told by | | [`_record_copies`](#nemo_gym-data-tasks-harbor-entries-_record_copies) | Record the copies of `layer` now at `destination` in `LAYER_RECORD_FILE`, when the record differs. | | [`_recorded_copies`](#nemo_gym-data-tasks-harbor-entries-_recorded_copies) | The digests `LAYER_RECORD_FILE` at `destination` holds, or `None` without a readable record. | | [`_refuse_copies_edited_in_place`](#nemo_gym-data-tasks-harbor-entries-_refuse_copies_edited_in_place) | Refuse copies of the entry's Gym layer that are neither the entry's files nor the ones this module placed. | | [`_remove_copy`](#nemo_gym-data-tasks-harbor-entries-_remove_copy) | Remove the copy of a file the entry no longer commits, and the folders left empty by it. | | [`_roots`](#nemo_gym-data-tasks-harbor-entries-_roots) | - | | [`_under`](#nemo_gym-data-tasks-harbor-entries-_under) | Whether `folder` resolves to `root` or a path under it (`root` resolved too). | | [`_under_working_root`](#nemo_gym-data-tasks-harbor-entries-_under_working_root) | Whether `folder` lies under `nemo_gym.WORKING_DIR`, the root whose `environments/harbor/` is the | | [`committed_entries`](#nemo_gym-data-tasks-harbor-entries-committed_entries) | Every entry that declares a source, across the search roots; an earlier root wins a name. | | [`entry_at`](#nemo_gym-data-tasks-harbor-entries-entry_at) | The committed entry whose pull `folder` holds when that is not the entry's own folder: | | [`entry_for_ref`](#nemo_gym-data-tasks-harbor-entries-entry_for_ref) | The entry whose source names the same dataset and version as the `harbor:` reference `target`, or `None`. | | [`entry_target`](#nemo_gym-data-tasks-harbor-entries-entry_target) | The entry `target` names on a command line, or `None` when it is a path or a hub reference. | | [`find_entry`](#nemo_gym-data-tasks-harbor-entries-find_entry) | The entry called `name` in the first search root that commits one, or `None`. | | [`gym_layer`](#nemo_gym-data-tasks-harbor-entries-gym_layer) | The Gym-layer files under `folder` by relative path, with their content: `dataset.toml`, a dataset-level | | [`plan_pull`](#nemo_gym-data-tasks-harbor-entries-plan_pull) | How `gym dataset pull TARGET` lands, or `None` when `target` is neither a hub reference nor an entry. | | [`prepare_destination`](#nemo_gym-data-tasks-harbor-entries-prepare_destination) | `entry`'s destination with the entry's Gym layer current there, ready for a fetch or a prepare. | ### Data [`ENTRIES_SUBDIR`](#nemo_gym-data-tasks-harbor-entries-ENTRIES_SUBDIR) [`LAYER_FILES`](#nemo_gym-data-tasks-harbor-entries-LAYER_FILES) [`LAYER_RECORD_FILE`](#nemo_gym-data-tasks-harbor-entries-LAYER_RECORD_FILE) [`_ENTRY_NAME`](#nemo_gym-data-tasks-harbor-entries-_ENTRY_NAME) [`_FETCH_SOURCES`](#nemo_gym-data-tasks-harbor-entries-_FETCH_SOURCES) ### API ```python class nemo_gym.data.tasks.harbor.entries.DatasetEntry( name: str, folder: pathlib.Path, source: str | None ) ``` Dataclass A committed entry: its name (the folder's), its folder and the `[gym] source` it declares, if any. **`destination`** `Path` Where the entry's tasks are pulled to and read from: the entry's own folder when it is under the working root (`nemo_gym.WORKING_DIR`) and not installed, else `//` (`nemo_gym.data.tasks.harbor.hub.datasets_dir`). An installed entry, one a package install ships under `site-packages` or another installation prefix (`_installed_entry`), falls back to the datasets dir wherever the working root is, a project-local `.venv` under it included, since the installation may be shared or read-only and is in any case not where the user's pulls go; so does an entry committed outside the working root (one a `NEMO_GYM_EXTRA_ROOTS` root commits), and `NEMO_GYM_DATASETS_DIR` moves every pull there, the working root's entries included. The name is taken by the first root that commits it (`find_entry` order), so the fallback folder is another committed entry's own folder (the shadowed one's) only when the same name is committed twice, and that case is the user's to resolve. An entry without a source has no pull to move: its tasks are wherever its folder holds them (a checkout the user placed there), so it stays at its folder and resolves by name as it does by path, wherever it is and whether or not the variable is set. --- **`folder`** `Path` --- **`name`** `str` --- **`source`** `str | None` --- ```python class nemo_gym.data.tasks.harbor.entries.PullPlan( ref: str, entry: nemo_gym.data.tasks.harbor.entries.DatasetEntry | None ) ``` Dataclass What a pull of a target fetches and where: the hub reference, and the entry it lands in, if one. **`entry`** `DatasetEntry | None` --- **`folder`** `Path | None` The dataset folder to fetch into, or `None` for the hub's own folder name under the datasets dir. --- **`ref`** `str` --- ```python nemo_gym.data.tasks.harbor.entries._committed_folder( name: str ) -> pathlib.Path | None ``` The folder `environments/harbor/` of the first search root that has one, or `None`. The copy a pull placed there for an entry committed in a later root (`_pull_copy`: the working root's `//` of an installed entry, after its first pull) is not that folder: it is skipped and the search goes on, so the name keeps resolving to the entry whose files the copy follows. ```python nemo_gym.data.tasks.harbor.entries._entry( folder: pathlib.Path ) -> nemo_gym.data.tasks.harbor.entries.DatasetEntry ``` The entry at `folder`; a `dataset.toml` that does not read raises `ValueError` with the file named. ```python nemo_gym.data.tasks.harbor.entries._install_prefixes() -> list[pathlib.Path] ``` The running interpreter's installation prefixes: `purelib` and `platlib`, its site-packages folders and the user site-packages (a venv's `site` module may lack `getsitepackages`: then the first two stand). ```python nemo_gym.data.tasks.harbor.entries._installed_entry( folder: pathlib.Path ) -> bool ``` Whether `folder` is installed: it lies under a Python installation prefix, so it is a package's, shared and often read-only, and never where a user's pulls go, whatever the working directory. Installed is a path with a `site-packages` or `dist-packages` component (as given or resolved), or one under the running interpreter's `purelib`/`platlib` (`sysconfig.get_paths`), its site-packages folders (`site.getsitepackages`) or its user site-packages (`site.getusersitepackages`). ```python nemo_gym.data.tasks.harbor.entries._layer_digests( files: dict[pathlib.Path, bytes] ) -> dict[str, str] ``` The digest of each Gym-layer file by its relative path, as the record of the copies holds them. ```python nemo_gym.data.tasks.harbor.entries._pull_copy( folder: pathlib.Path ) -> bool ``` Whether `folder` is the copy a pull placed at `//` for an entry elsewhere, told by the `LAYER_RECORD_FILE` only `prepare_destination` writes, beside the copies of the entry's Gym layer. Such a folder is never itself an entry: the entry is in a later root, and lookup goes on to it. ```python nemo_gym.data.tasks.harbor.entries._record_copies( destination: pathlib.Path, layer: dict[pathlib.Path, bytes] ) -> None ``` Record the copies of `layer` now at `destination` in `LAYER_RECORD_FILE`, when the record differs. Written whole and renamed into place, so an interrupted write leaves the previous record, not a torn one. ```python nemo_gym.data.tasks.harbor.entries._recorded_copies( destination: pathlib.Path ) -> dict[str, str] | None ``` The digests `LAYER_RECORD_FILE` at `destination` holds, or `None` without a readable record. ```python nemo_gym.data.tasks.harbor.entries._refuse_copies_edited_in_place( entry: nemo_gym.data.tasks.harbor.entries.DatasetEntry, destination: pathlib.Path, layer: dict[pathlib.Path, bytes], copies: dict[pathlib.Path, bytes] ) -> None ``` Refuse copies of the entry's Gym layer that are neither the entry's files nor the ones this module placed. The copies are at a destination elsewhere than the entry's folder (`NEMO_GYM_DATASETS_DIR`, or an entry committed outside the working root, `DatasetEntry.destination`). The record of the placed copies is `LAYER_RECORD_FILE` beside them (`_record_copies`), which only `prepare_destination` writes: copies whose digests it holds, every one of them, were placed there and may be replaced. The folder's `manifest.toml` is no such record, not even one a hub fetch wrote: a prepare of the folder by its path in a shell without `NEMO_GYM_DATASETS_DIR` rewrites its Gym-layer digests over an edited copy, as it does for any dataset whose own `dataset.toml` changed, so a copy checked against those digests read as the one on record and the next pull overwrote the edit. A `local` manifest beside the differing copies is named as such: it is prepare's record of a dataset the user put at `//` by hand and ran by its path, before `NEMO_GYM_DATASETS_DIR` or an entry committed outside the working root moved the entry's pull there, or in a shell without the variable, so the folder is not the entry's pull at all; the refusal says which case it is and what to do (move the folder aside; with the variable, running it by its path without it is the other way), and either way nothing is overwritten. ```python nemo_gym.data.tasks.harbor.entries._remove_copy( destination: pathlib.Path, path: pathlib.Path ) -> None ``` Remove the copy of a file the entry no longer commits, and the folders left empty by it. ```python nemo_gym.data.tasks.harbor.entries._roots() -> list[pathlib.Path] ``` ```python nemo_gym.data.tasks.harbor.entries._under( folder: pathlib.Path, root: pathlib.Path ) -> bool ``` Whether `folder` resolves to `root` or a path under it (`root` resolved too). ```python nemo_gym.data.tasks.harbor.entries._under_working_root( folder: pathlib.Path ) -> bool ``` Whether `folder` lies under `nemo_gym.WORKING_DIR`, the root whose `environments/harbor/` is the default datasets dir (the checkout of an editable install, the working directory of a package install). A project-local `.venv` lies under it too, which is why `_installed_entry` is asked first. ```python nemo_gym.data.tasks.harbor.entries.committed_entries() -> list[nemo_gym.data.tasks.harbor.entries.DatasetEntry] ``` Every entry that declares a source, across the search roots; an earlier root wins a name. A name is taken by the first root that has a folder of it, as `find_entry` resolves it, whether or not that folder declares a source: a source-less `tb4` in an earlier root shadows a sourced one in a later root for the name, so it shadows it for the reference too, and the pull of that reference lands in the hub's own folder rather than in an entry the name no longer reaches. The copy a pull placed for an entry of a later root (`_pull_copy`) takes no name: it is skipped as `find_entry` skips it, so the installed entry it follows keeps the name and the reference. Only folders with a `dataset.toml` are read (a pulled hub folder has none unless its user wrote one) and one whose table does not read is skipped here: the mapping must not fail every pull over a stray file, and the entry's own pull reports it. ```python nemo_gym.data.tasks.harbor.entries.entry_at( folder: pathlib.Path ) -> nemo_gym.data.tasks.harbor.entries.DatasetEntry | None ``` The committed entry whose pull `folder` holds when that is not the entry's own folder: `//` for the entry called `name` under a moved `NEMO_GYM_DATASETS_DIR`, for an installed entry (one a package install ships) or for an entry committed outside the working root, or `None` for every other folder (an entry's own folder included: nothing is copied there). The folder is looked up as the name is (`_committed_folder`), so the copy itself, once placed, is passed over for the entry it follows and the path form agrees with the name form. A path target naming such a folder is prepared with the entry's Gym layer kept current beside the tasks, or refused, exactly as the pull and the entry's name are (`prepare_destination`), so the three commands agree on the folder and an edited copy is refused in the same words by each. An unreadable `dataset.toml` of the entry raises `ValueError` naming the file, as `find_entry` does: it is the table the folder's copy follows. The folder's own table is never read here: the folder is compared with the first root's `environments/harbor/` before any entry is read, so a plain folder under the datasets dir (every pulled hub folder of an editable checkout, whose datasets dir is that root's `environments/harbor/`) is nobody's pull elsewhere, and a `dataset.toml` written into it that does not read is reported by the prepare that follows, as it was before. ```python nemo_gym.data.tasks.harbor.entries.entry_for_ref( target: str ) -> nemo_gym.data.tasks.harbor.entries.DatasetEntry | None ``` The entry whose source names the same dataset and version as the `harbor:` reference `target`, or `None`. The two are compared as the fetchers record them (`canonical_reference`), so a digest with or without its `sha256:` prefix, or `harbor:org/name` with or without `@latest`, finds the entry that declares the other spelling instead of landing beside it in the hub's own folder. `harbor:org/name` (the store's `latest`) and `harbor:org/name@4.0.0` are different references, so a plain reference never lands in an entry pinned to a version. Two entries declaring one source is a mistake in the checkout and is refused with both named. ```python nemo_gym.data.tasks.harbor.entries.entry_target( target: str ) -> nemo_gym.data.tasks.harbor.entries.DatasetEntry | None ``` The entry `target` names on a command line, or `None` when it is a path or a hub reference. A bare name (one path component, no `./` or `/`) that is not an existing path and that a search root commits under `environments/harbor/` is the entry. An existing folder of that name, or an explicit path, keeps meaning the path, as it does for Hydra overrides. ```python nemo_gym.data.tasks.harbor.entries.find_entry( name: str ) -> nemo_gym.data.tasks.harbor.entries.DatasetEntry | None ``` The entry called `name` in the first search root that commits one, or `None`. An entry need not declare a source: one whose tasks come from elsewhere (a checkout the user points `gym eval run` at) still resolves by name to its folder. A `dataset.toml` that does not read raises `ValueError` naming the file, as prepare would for the folder. ```python nemo_gym.data.tasks.harbor.entries.gym_layer( folder: pathlib.Path ) -> dict[pathlib.Path, bytes] ``` The Gym-layer files under `folder` by relative path, with their content: `dataset.toml`, a dataset-level `task_data.py` or `verifier.py` (`LAYER_FILES`), each when present. These are the files the dataset digest covers besides the task folders, so they are what an entry's destination elsewhere must hold a copy of for a run there to be the entry's. ```python nemo_gym.data.tasks.harbor.entries.plan_pull( target: str ) -> nemo_gym.data.tasks.harbor.entries.PullPlan | None ``` How `gym dataset pull TARGET` lands, or `None` when `target` is neither a hub reference nor an entry. A hub reference is fetched into the entry that declares it as its source, else into the hub's own folder. An entry name is fetched from the source it declares, into its folder; an entry without a source has nothing to pull and says so. The pull takes no path, so a bare name is looked up as an entry outright: a folder of the same name in the working directory does not hide it, as it would from validate and run (`entry_target`). ```python nemo_gym.data.tasks.harbor.entries.prepare_destination( entry: nemo_gym.data.tasks.harbor.entries.DatasetEntry, create: bool = True ) -> pathlib.Path ``` `entry`'s destination with the entry's Gym layer current there, ready for a fetch or a prepare. The entry's own folder needs nothing. A destination elsewhere (`NEMO_GYM_DATASETS_DIR`, an installed entry or one committed outside the working root, `DatasetEntry.destination`) holds a copy of the entry's Gym layer (`gym_layer`: `dataset.toml`, a dataset-level `task_data.py` or `verifier.py`), so the files that shape the tasks and the digest that covers them are the entry's wherever the pull landed: the copies are placed before the first task lands (`create` makes the folder; a prepare, which fetches nothing, passes `False` and leaves a missing folder missing) and are replaced, a file the entry no longer commits removed, when the entry's files changed since, as long as the copies are still the ones on record: the ones this function placed, whose digests it writes to `LAYER_RECORD_FILE` beside them as it places them (`_record_copies`; the record is brought up to date whenever the copies are found to be the entry's files). Only this function writes that record, so no other command can vouch for an edited copy: the manifest's own Gym-layer digests are rewritten by a prepare of the folder by its path in a shell without `NEMO_GYM_DATASETS_DIR`, which cannot know the folder is an entry's pull, and were once what the copies were checked against, so such a prepare absorbed an edit of the copy and the next pull overwrote it. Whenever the copies are replaced the manifest is brought up to date with them (`refresh_gym_layer`), so its digests follow the copies even when nothing rewrites the manifest afterwards (a registry fetch with every folder in place, a fetch that fails, a prepare refused before its manifest check). A copy that matches neither the entry's file nor the record was edited in place and is refused, never overwritten, whatever the folder's manifest says: the entry's files are where the layer is edited. `gym dataset pull `, `gym dataset validate ` and `gym eval run ` all come through here, so they agree on the folder. ```python nemo_gym.data.tasks.harbor.entries.ENTRIES_SUBDIR = Path('environments') / 'harbor' ``` ```python nemo_gym.data.tasks.harbor.entries.LAYER_FILES = (DATASET_FILE, *DATASET_FILES) ``` ```python nemo_gym.data.tasks.harbor.entries.LAYER_RECORD_FILE = '.gym-layer.json' ``` ```python nemo_gym.data.tasks.harbor.entries._ENTRY_NAME = re.compile('[A-Za-z0-9][A-Za-z0-9._-]*') ``` ```python nemo_gym.data.tasks.harbor.entries._FETCH_SOURCES = (PACKAGE_STORE_SOURCE, REGISTRY_SOURCE) ```