nemo_gym.tasks.harbor.package_store

org/name@ref; ref is a tag such as 4.0.0, latest or sha256:<digest>.

nemo_gym.tasks.harbor.package_store.PackageRef.folder_name(
    resolved_digest: str
) -> str

Read-only client for Harbor’s package store.

nemo_gym.tasks.harbor.package_store.PackageStore._dataset_tasks(
    version_id: str
) -> tuple[nemo_gym.tasks.harbor.package_store.PackageTask, ...]

nemo_gym.tasks.harbor.package_store.PackageStore._json(
    method: str,
    path: str,
    params: dict[str, str] | None = None,
    body: typing.Any = None
) -> typing.Any

nemo_gym.tasks.harbor.package_store.PackageStore._request(
    method: str,
    path: str,
    params: dict[str, str] | None = None,
    body: typing.Any = None
) -> bytes

nemo_gym.tasks.harbor.package_store.PackageStore._rows(
    table: str,
    params: dict[str, str]
) -> list[dict[str, typing.Any]]

nemo_gym.tasks.harbor.package_store.PackageStore.archive_path(
    task: nemo_gym.tasks.harbor.package_store.PackageTask
) -> str

nemo_gym.tasks.harbor.package_store.PackageStore.download_task(
    task: nemo_gym.tasks.harbor.package_store.PackageTask,
    target: pathlib.Path
) -> None

nemo_gym.tasks.harbor.package_store.PackageStore.resolve_dataset(
    ref: nemo_gym.tasks.harbor.package_store.PackageRef
) -> nemo_gym.tasks.harbor.package_store.ResolvedDataset

Bases: ValueError

The reference cannot be resolved or a package cannot be fetched.

Refuse folder when its manifest’s [dataset] table pins another dataset than dataset.

Folder names stay readable (<name>, <name>-<tag>), so the manifest’s [dataset] table is what tells alice/evals@1.0 from bob/evals@1.0 in the same evals-1.0 folder, and a latest fetch from the version the tag has since moved to. Either mismatch names the recorded identity and the requested one, and a manifest whose table names no package-store dataset at all is refused the same way: a folder with a manifest is never treated as unclaimed. The remedy is the same as for an edited task folder.

Record which dataset folder holds before its first task lands.

Called when the folder has no manifest, right before the first download. The claim is the manifest with the [dataset] identity, every task pinned and only the digests verified so far (none in a fresh folder), created exclusively; each task downloaded after it is recorded as it lands. Of two fetches racing for an unclaimed folder exactly one writes the claim, and the other reads it back and is refused unless it fetches the same dataset (then both complete the folder, and PackageStore.download_task keeps whichever copy of a task lands first). An interrupted fetch leaves the claim behind, so a later fetch of another package, or of another version of this one, is refused before it looks at any task, and a later fetch of the same dataset completes the folder.

Whether the task folder at target is the store’s task, by either digest.

A freshly unpacked archive matches Harbor’s published hash by archive_hash (its files are exactly the published set, names Gym’s folder digest skips included), and a folder a run has since left byte-code in matches it by content_hash unless the package itself ships such a name. Either match means the folder is the store’s copy.

The manifest: the dataset pin, then per task the store’s hash and, once the task has landed, Gym’s digest.

Every task is pinned from the first write on; a task whose folder is not there yet has no digest, so the manifest an interrupted fetch leaves names exactly the tasks still to come.

The manifest a previous fetch left in folder, parsed, or None when there is none.

One this Gym cannot read stops the fetch with the folder untouched: not TOML, or without the [dataset] table, or with [dataset] or [tasks] not a table, is a PackageStoreError. The manifest is the folder’s claim, which package and version its task folders belong to, so a folder whose claim cannot be read is never completed, reused or rewritten; the remedy is the same as for an edited task folder.

The [dataset] table of a parsed manifest (string fields only), or {}.

Gym’s digest of each task folder as the previous fetch wrote it, by task name.

Only an entry that pins the task version dataset names counts, so a hand-edited or foreign manifest never vouches for a folder, and a task the claim pinned but that has not landed yet (its entry has no digest) is not recorded.

Write manifest.toml atomically: a failed write leaves the previous, truthful file intact.

The complete file is written to a temporary sibling (unique per process and thread, so concurrent fetches of one folder never share it) and moved into place, so a reader never sees a partial manifest. A rewrite keeps the permissions the file had; a new manifest gets the usual umask-aware mode.

With exclusive, the manifest is created only where none is and FileExistsError is raised otherwise, the target untouched (the temporary is written and removed either way): the complete file is hard-linked into place, the atomic create-unless-present that os.replace lacks, so of two writers racing for an empty folder exactly one succeeds. A filesystem without hard links falls back to an exclusive create of the target itself, still one winner, though a reader may then glimpse the file before its last byte. _claim_folder claims a folder this way before its first task lands.

Write text to fd and flush it to disk; the descriptor is closed.

Fetch every task of a package-store dataset into root/<folder>/<task>/.

A task folder whose digest already matches is kept, so a rerun after a partial fetch completes it: a folder the manifest records is compared with the digest recorded for it (Gym’s digest of the folder as fetched, which a package shipping a name that digest skips makes differ from the store’s hash), and a folder no manifest records yet (the fetch was interrupted) with the store’s hash by either digest. A task folder whose digest differs (it was edited locally, or the store republished it) stops the fetch with both digests named, so edits are never discarded silently; deleting or moving the folder aside and running again fetches the store’s copy. A dataset folder whose manifest.toml pins another package, or an earlier resolution of this one (latest has moved), stops the fetch before any task is looked at, with the recorded and the requested identity named, so one folder never mixes two versions or two packages. That manifest exists from the first download on: an unclaimed folder is claimed (_claim_folder, the identity and the pins without digests, created exclusively) before its first task lands, so an interrupted fetch leaves a folder that says whose it is, and two fetches racing for one folder never fill it with two datasets. A manifest this Gym cannot read stops the fetch with the folder untouched, since this Gym cannot tell what it claimed. Anything else in a task’s place (a folder without task.toml, a file, a symlink) also stops the fetch: only what this function fetched is ever replaced. Returns the dataset folder.