nemo_gym.data.tasks.harbor.entries

A committed entry: its name (the folder’s), its folder and the [gym] source it declares, if any.

What a pull of a target fetches and where: the hub reference, and the entry it lands in, if one.

The folder environments/harbor/<name> of the first search root that has one, or None.

The copy a pull placed there for an entry committed in a later root (_pull_copy: the working root’s <datasets dir>/<name>/ of an installed entry, after its first pull) is not that folder: it is skipped and the search goes on, so the name keeps resolving to the entry whose files the copy follows.

The entry at folder; a dataset.toml that does not read raises ValueError with the file named.

The running interpreter’s installation prefixes: purelib and platlib, its site-packages folders and the user site-packages (a venv’s site module may lack getsitepackages: then the first two stand).

Whether folder is installed: it lies under a Python installation prefix, so it is a package’s, shared and often read-only, and never where a user’s pulls go, whatever the working directory. Installed is a path with a site-packages or dist-packages component (as given or resolved), or one under the running interpreter’s purelib/platlib (sysconfig.get_paths), its site-packages folders (site.getsitepackages) or its user site-packages (site.getusersitepackages).

The digest of each Gym-layer file by its relative path, as the record of the copies holds them.

Whether folder is the copy a pull placed at <datasets dir>/<entry>/ for an entry elsewhere, told by the LAYER_RECORD_FILE only prepare_destination writes, beside the copies of the entry’s Gym layer. Such a folder is never itself an entry: the entry is in a later root, and lookup goes on to it.

Record the copies of layer now at destination in LAYER_RECORD_FILE, when the record differs.

Written whole and renamed into place, so an interrupted write leaves the previous record, not a torn one.

The digests LAYER_RECORD_FILE at destination holds, or None without a readable record.

Refuse copies of the entry’s Gym layer that are neither the entry’s files nor the ones this module placed.

The copies are at a destination elsewhere than the entry’s folder (NEMO_GYM_DATASETS_DIR, or an entry committed outside the working root, DatasetEntry.destination). The record of the placed copies is LAYER_RECORD_FILE beside them (_record_copies), which only prepare_destination writes: copies whose digests it holds, every one of them, were placed there and may be replaced. The folder’s manifest.toml is no such record, not even one a hub fetch wrote: a prepare of the folder by its path in a shell without NEMO_GYM_DATASETS_DIR rewrites its Gym-layer digests over an edited copy, as it does for any dataset whose own dataset.toml changed, so a copy checked against those digests read as the one on record and the next pull overwrote the edit. A local manifest beside the differing copies is named as such: it is prepare’s record of a dataset the user put at <datasets dir>/<entry>/ by hand and ran by its path, before NEMO_GYM_DATASETS_DIR or an entry committed outside the working root moved the entry’s pull there, or in a shell without the variable, so the folder is not the entry’s pull at all; the refusal says which case it is and what to do (move the folder aside; with the variable, running it by its path without it is the other way), and either way nothing is overwritten.

Remove the copy of a file the entry no longer commits, and the folders left empty by it.

Whether folder resolves to root or a path under it (root resolved too).

Whether folder lies under nemo_gym.WORKING_DIR, the root whose environments/harbor/ is the default datasets dir (the checkout of an editable install, the working directory of a package install). A project-local .venv lies under it too, which is why _installed_entry is asked first.

Every entry that declares a source, across the search roots; an earlier root wins a name.

A name is taken by the first root that has a folder of it, as find_entry resolves it, whether or not that folder declares a source: a source-less tb4 in an earlier root shadows a sourced one in a later root for the name, so it shadows it for the reference too, and the pull of that reference lands in the hub’s own folder rather than in an entry the name no longer reaches. The copy a pull placed for an entry of a later root (_pull_copy) takes no name: it is skipped as find_entry skips it, so the installed entry it follows keeps the name and the reference. Only folders with a dataset.toml are read (a pulled hub folder has none unless its user wrote one) and one whose table does not read is skipped here: the mapping must not fail every pull over a stray file, and the entry’s own pull reports it.

The committed entry whose pull folder holds when that is not the entry’s own folder: <datasets dir>/<name>/ for the entry called name under a moved NEMO_GYM_DATASETS_DIR, for an installed entry (one a package install ships) or for an entry committed outside the working root, or None for every other folder (an entry’s own folder included: nothing is copied there). The folder is looked up as the name is (_committed_folder), so the copy itself, once placed, is passed over for the entry it follows and the path form agrees with the name form. A path target naming such a folder is prepared with the entry’s Gym layer kept current beside the tasks, or refused, exactly as the pull and the entry’s name are (prepare_destination), so the three commands agree on the folder and an edited copy is refused in the same words by each. An unreadable dataset.toml of the entry raises ValueError naming the file, as find_entry does: it is the table the folder’s copy follows. The folder’s own table is never read here: the folder is compared with the first root’s environments/harbor/<name> before any entry is read, so a plain folder under the datasets dir (every pulled hub folder of an editable checkout, whose datasets dir is that root’s environments/harbor/) is nobody’s pull elsewhere, and a dataset.toml written into it that does not read is reported by the prepare that follows, as it was before.

The entry whose source names the same dataset and version as the harbor: reference target, or None.

The two are compared as the fetchers record them (canonical_reference), so a digest with or without its sha256: prefix, or harbor:org/name with or without @latest, finds the entry that declares the other spelling instead of landing beside it in the hub’s own folder. harbor:org/name (the store’s latest) and harbor:org/name@4.0.0 are different references, so a plain reference never lands in an entry pinned to a version. Two entries declaring one source is a mistake in the checkout and is refused with both named.

The entry target names on a command line, or None when it is a path or a hub reference.

A bare name (one path component, no ./ or /) that is not an existing path and that a search root commits under environments/harbor/ is the entry. An existing folder of that name, or an explicit path, keeps meaning the path, as it does for Hydra overrides.

The entry called name in the first search root that commits one, or None.

An entry need not declare a source: one whose tasks come from elsewhere (a checkout the user points gym eval run at) still resolves by name to its folder. A dataset.toml that does not read raises ValueError naming the file, as prepare would for the folder.

The Gym-layer files under folder by relative path, with their content: dataset.toml, a dataset-level task_data.py or verifier.py (LAYER_FILES), each when present. These are the files the dataset digest covers besides the task folders, so they are what an entry’s destination elsewhere must hold a copy of for a run there to be the entry’s.

How gym dataset pull TARGET lands, or None when target is neither a hub reference nor an entry.

A hub reference is fetched into the entry that declares it as its source, else into the hub’s own folder. An entry name is fetched from the source it declares, into its folder; an entry without a source has nothing to pull and says so. The pull takes no path, so a bare name is looked up as an entry outright: a folder of the same name in the working directory does not hide it, as it would from validate and run (entry_target).

entry’s destination with the entry’s Gym layer current there, ready for a fetch or a prepare.

The entry’s own folder needs nothing. A destination elsewhere (NEMO_GYM_DATASETS_DIR, an installed entry or one committed outside the working root, DatasetEntry.destination) holds a copy of the entry’s Gym layer (gym_layer: dataset.toml, a dataset-level task_data.py or verifier.py), so the files that shape the tasks and the digest that covers them are the entry’s wherever the pull landed: the copies are placed before the first task lands (create makes the folder; a prepare, which fetches nothing, passes False and leaves a missing folder missing) and are replaced, a file the entry no longer commits removed, when the entry’s files changed since, as long as the copies are still the ones on record: the ones this function placed, whose digests it writes to LAYER_RECORD_FILE beside them as it places them (_record_copies; the record is brought up to date whenever the copies are found to be the entry’s files). Only this function writes that record, so no other command can vouch for an edited copy: the manifest’s own Gym-layer digests are rewritten by a prepare of the folder by its path in a shell without NEMO_GYM_DATASETS_DIR, which cannot know the folder is an entry’s pull, and were once what the copies were checked against, so such a prepare absorbed an edit of the copy and the next pull overwrote it. Whenever the copies are replaced the manifest is brought up to date with them (refresh_gym_layer), so its digests follow the copies even when nothing rewrites the manifest afterwards (a registry fetch with every folder in place, a fetch that fails, a prepare refused before its manifest check). A copy that matches neither the entry’s file nor the record was edited in place and is refused, never overwritten, whatever the folder’s manifest says: the entry’s files are where the layer is edited. gym dataset pull <entry>, gym dataset validate <entry> and gym eval run <entry> all come through here, so they agree on the folder.