nemo_gym.data.tasks.harbor.package_store

org/name@ref; ref is a tag such as 4.0.0, latest or sha256:<digest>.

nemo_gym.data.tasks.harbor.package_store.PackageRef.folder_name(
    resolved_digest: str
) -> str

nemo_gym.data.tasks.harbor.package_store.PackageRef.of(
    ref: nemo_gym.data.tasks.harbor.hub.HubRef
) -> nemo_gym.data.tasks.harbor.package_store.PackageRef

Read-only client for Harbor’s package store.

nemo_gym.data.tasks.harbor.package_store.PackageStore._dataset_tasks(
    version_id: str
) -> tuple[nemo_gym.data.tasks.harbor.package_store.PackageTask, ...]

nemo_gym.data.tasks.harbor.package_store.PackageStore._json(
    method: str,
    path: str,
    params: dict[str, str] | None = None,
    body: typing.Any = None
) -> typing.Any

nemo_gym.data.tasks.harbor.package_store.PackageStore._request(
    method: str,
    path: str,
    params: dict[str, str] | None = None,
    body: typing.Any = None
) -> bytes

nemo_gym.data.tasks.harbor.package_store.PackageStore._rows(
    table: str,
    params: dict[str, str]
) -> list[dict[str, typing.Any]]

nemo_gym.data.tasks.harbor.package_store.PackageStore.archive_path(
    task: nemo_gym.data.tasks.harbor.package_store.PackageTask
) -> str

nemo_gym.data.tasks.harbor.package_store.PackageStore.download_task(
    task: nemo_gym.data.tasks.harbor.package_store.PackageTask,
    target: pathlib.Path
) -> None

nemo_gym.data.tasks.harbor.package_store.PackageStore.resolve_dataset(
    ref: nemo_gym.data.tasks.harbor.package_store.PackageRef
) -> nemo_gym.data.tasks.harbor.package_store.ResolvedDataset

Bases: ValueError

The reference cannot be resolved or a package cannot be fetched.

Derived facts only: per-task digests and package pins, and the reference this fetch was made from (PackageRef.reference); the dataset’s name and version stay in dataset.toml.

Every task of the store’s version is pinned; a digest is recorded only for the tasks in digests, the folders this fetch has verified (or trusts from the previous manifest, see _checkpoint_digests). Nothing is hashed here: a folder the loop has not reached yet may hold a local edit, and a manifest that vouched for it would make the next fetch, and prepare, accept the edit as the store’s copy. A task whose folder is not verified keeps its pin and gets no digest, which is how an interrupted fetch names the tasks still to come, and a folder the store does not name stays unrecorded.

Refuse folder when its manifest was written for another package, or another version, than dataset.

Refuse folder when its manifest was written for another package than dataset.

The manifest holds no copy of the dataset’s name (that lives in dataset.toml), and one written before the fetch’s reference was recorded has no ref either, but every task it pins carries the task’s own package, <org>/<task>, as the store names it. A dataset may collect tasks published under any org (Harbor’s dataset.toml pins each task as org/name, and the store resolves each to its own package), so a pin is compared with the package dataset resolved that task to, never with <dataset org>/<task>. Folder names stay readable (<name>, <name>-<tag>), so those pins are what tell alice/evals@1.0 from bob/evals@1.0 in the same evals-1.0 folder when the two pin different packages for a task; two datasets pinning the very same task packages hold the same bytes, and either completes the folder, while a task one pins and the other lacks is a version difference (_stale_reason). A manifest another fetcher wrote (the registry path) is refused by its source. The remedy is the same as for an edited task folder.

The digests a progress manifest may record: what this fetch verified, plus the previous manifest’s digest for a task it has not reached yet whose folder is still there.

The previous digest was verified when it was written and pins the same task version (_recorded_digests), so keeping it loses nothing and keeps the folder checked against the copy the previous fetch wrote rather than against the store’s hash. It is never recomputed from the folder: the loop checks that folder when it gets there and refuses an edit. A task whose folder is gone drops its digest and is fetched again.

Record which dataset folder holds before its first task lands.

Called when the folder has no manifest, right before the first download. The claim is the manifest with every task pinned and only the digests verified so far (none in a fresh folder; a folder already there for a later task is not hashed, the loop checks it when it gets there), created exclusively; each task downloaded after it is recorded as it lands. Of two fetches racing for an unclaimed folder exactly one writes the claim, and the other reads it back and is refused unless it fetches the same dataset (then both complete the folder, and PackageStore.download_task keeps whichever copy of a task lands first). An interrupted fetch leaves the claim behind, so a later fetch of another package, or of another version of this one, is refused before it looks at any task, and a later fetch of the same dataset completes the folder.

Whether the task folder at target is the store’s task, by either digest.

A freshly unpacked archive matches Harbor’s published hash by archive_hash (its files are exactly the published set, names Gym’s folder digest skips included), and a folder a run has since left byte-code in matches it by content_hash unless the package itself ships such a name. Either match means the folder is the store’s copy.

The manifest a previous fetch left in folder, or None when there is none.

One this Gym cannot read stops the fetch with the folder untouched: a manifest a newer Gym wrote (a schema this Gym does not read) raises as itself, and a malformed one (not TOML, or not the expected tables) as a PackageStoreError. The manifest is the folder’s claim, which package and version its task folders belong to, so a folder whose claim cannot be read is never completed, reused or rewritten; the remedy is the same as for an edited task folder.

Gym’s digest of each task folder as the previous fetch wrote it, by task name.

Only an entry that pins the task version dataset names counts, so a hand-edited or foreign manifest never vouches for a folder.

How the store’s version differs from the one the manifest was written for, or None.

Fetch every task of a package-store dataset into root/<folder>/<task>/.

The dataset folder is root/<name>[-<tag | 12 digest chars>]/ (PackageRef.folder_name) unless folder names it: the folder of a committed entry under environments/harbor/ that declares this dataset as its [gym] source (nemo_gym.data.tasks.harbor.entries). The entry’s own files there (README, YAML overlays, dataset.toml) are not task folders and are left alone; the manifest’s claim and the per-task checks below are the same whichever folder the tasks land in.

A task folder whose digest already matches is kept, so a rerun after a partial fetch completes it: a folder the manifest records is compared with the digest recorded for it (Gym’s digest of the folder as fetched, which a package shipping a name that digest skips makes differ from the store’s hash), and a folder no manifest records yet (the fetch was interrupted) with the store’s hash by either digest. A task folder whose digest differs (it was edited locally, or the store republished it) stops the fetch with both digests named, so edits are never discarded silently; deleting or moving the folder aside and running again fetches the store’s copy. A folder whose manifest was written for another version of the dataset (latest moved), or for another package (its task pins name other packages than the store resolves this dataset’s tasks to), is refused the same way, so one folder never mixes two versions or two packages. That manifest exists from the first download on: an unclaimed folder is claimed (_claim_folder, the pins without digests, created exclusively) before its first task lands, so an interrupted fetch leaves a folder that says whose it is, and two fetches racing for one folder never fill it with two datasets. A manifest this Gym cannot read, one a newer Gym wrote or a malformed one, stops the fetch with the folder untouched, since this Gym cannot tell what it claimed. Anything else in a task’s place (a folder without task.toml, a file, a symlink) also stops the fetch: only what this function fetched is ever replaced. Returns the dataset folder.