> This page is for version 0.5.1.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.data.tasks.harbor.package_store

Harbor package-store datasets: `harbor:<org>/<name>[@<tag | sha256:digest>]`.

Harbor publishes some datasets (Terminal Bench 4 among them) as packages rather than
git checkouts. A dataset version is a list of task packages, each pinned by the
content hash Gym also uses as its task digest. Fetching a dataset resolves the version,
downloads every task archive, checks each against Harbor's published hash and writes
`manifest.toml` with the per-task digests and package pins (see `nemo_gym.data.tasks.harbor.manifest`):
per task the store's hash and Gym's digest of the folder as fetched. The manifest is also the folder's
claim: it is written with the task pins and no digests before the first task lands, so a folder an
interrupted fetch leaves behind still says which package and version it belongs to. A later fetch into
the same folder reads that back first, refuses a folder claimed for another package or another version
of the dataset, and compares each task folder with the digest recorded for it.

The store is Harbor's public PostgREST and storage API. Only the public
(publishable) key is needed to read published packages.

## Module Contents

### Classes

| Name                                                                               | Description                                                                          |
| ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| [`PackageRef`](#nemo_gym-data-tasks-harbor-package_store-PackageRef)               | `org/name@ref`; `ref` is a tag such as `4.0.0`, `latest` or `sha256:&lt;digest&gt;`. |
| [`PackageStore`](#nemo_gym-data-tasks-harbor-package_store-PackageStore)           | Read-only client for Harbor's package store.                                         |
| [`PackageStoreError`](#nemo_gym-data-tasks-harbor-package_store-PackageStoreError) | The reference cannot be resolved or a package cannot be fetched.                     |
| [`PackageTask`](#nemo_gym-data-tasks-harbor-package_store-PackageTask)             | -                                                                                    |
| [`ResolvedDataset`](#nemo_gym-data-tasks-harbor-package_store-ResolvedDataset)     | -                                                                                    |

### Functions

| Name                                                                                             | Description                                                                                            |
| ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ |
| [`_build_manifest`](#nemo_gym-data-tasks-harbor-package_store-_build_manifest)                   | Derived facts only: per-task digests and package pins, and the reference this fetch was made from      |
| [`_check_recorded_dataset`](#nemo_gym-data-tasks-harbor-package_store-_check_recorded_dataset)   | Refuse `folder` when its manifest was written for another package, or another version, than `dataset`. |
| [`_check_recorded_identity`](#nemo_gym-data-tasks-harbor-package_store-_check_recorded_identity) | Refuse `folder` when its manifest was written for another package than `dataset`.                      |
| [`_checkpoint_digests`](#nemo_gym-data-tasks-harbor-package_store-_checkpoint_digests)           | The digests a progress manifest may record: what this fetch verified, plus the previous manifest's     |
| [`_claim_folder`](#nemo_gym-data-tasks-harbor-package_store-_claim_folder)                       | Record which dataset `folder` holds before its first task lands.                                       |
| [`_holds_store_copy`](#nemo_gym-data-tasks-harbor-package_store-_holds_store_copy)               | Whether the task folder at `target` is the store's `task`, by either digest.                           |
| [`_read_manifest`](#nemo_gym-data-tasks-harbor-package_store-_read_manifest)                     | The manifest a previous fetch left in `folder`, or `None` when there is none.                          |
| [`_recorded_digests`](#nemo_gym-data-tasks-harbor-package_store-_recorded_digests)               | Gym's digest of each task folder as the previous fetch wrote it, by task name.                         |
| [`_stale_reason`](#nemo_gym-data-tasks-harbor-package_store-_stale_reason)                       | How the store's version differs from the one the manifest was written for, or `None`.                  |
| [`_write_manifest`](#nemo_gym-data-tasks-harbor-package_store-_write_manifest)                   | -                                                                                                      |
| [`fetch_package_dataset`](#nemo_gym-data-tasks-harbor-package_store-fetch_package_dataset)       | Fetch every task of a package-store dataset into `root/&lt;folder&gt;/&lt;task&gt;/`.                  |

### Data

[`PACKAGE_STORE_PUBLIC_KEY`](#nemo_gym-data-tasks-harbor-package_store-PACKAGE_STORE_PUBLIC_KEY)

[`PACKAGE_STORE_SOURCE`](#nemo_gym-data-tasks-harbor-package_store-PACKAGE_STORE_SOURCE)

[`PACKAGE_STORE_URL`](#nemo_gym-data-tasks-harbor-package_store-PACKAGE_STORE_URL)

[`_DIGEST`](#nemo_gym-data-tasks-harbor-package_store-_DIGEST)

[`_PAGE_SIZE`](#nemo_gym-data-tasks-harbor-package_store-_PAGE_SIZE)

[`_SAFE`](#nemo_gym-data-tasks-harbor-package_store-_SAFE)

[`logger`](#nemo_gym-data-tasks-harbor-package_store-logger)

### API

```python
class nemo_gym.data.tasks.harbor.package_store.PackageRef(
    org: str,
    name: str,
    ref: str = 'latest'
)
```

Dataclass

`org/name@ref`; `ref` is a tag such as `4.0.0`, `latest` or `sha256:<digest>`.

**`digest`** `str | None`

---

**`name`** `str`

---

**`org`** `str`

---

**`package`** `str`

---

**`ref`** `str = 'latest'`

---

**`reference`** `str`

The reference as a manifest records it: `harbor:<org>/<name>@<tag | latest | sha256:digest>`, so that
`harbor:org/name` and `@latest`, or a digest with and without its `sha256:` prefix, read as one.

---

```python
nemo_gym.data.tasks.harbor.package_store.PackageRef.folder_name(
    resolved_digest: str
) -> str
```

`<name>` for `latest`, `<name>-<tag>` for a tag, `<name>-<12 digest chars>` for a digest pin.

`latest` is what a plain `harbor:org/name` reference means, so its folder carries the plain
name and `manifest.toml` records the task packages it resolved to. A pinned tag or digest keeps
its suffix, so two pins of one dataset never share a folder.

```python
nemo_gym.data.tasks.harbor.package_store.PackageRef.of(
    ref: nemo_gym.data.tasks.harbor.hub.HubRef
) -> nemo_gym.data.tasks.harbor.package_store.PackageRef
```

classmethod

The store's reading of a `harbor:<org>/<name>[@<ref>]` reference: without `@<ref>` it is `latest`.

```python
class nemo_gym.data.tasks.harbor.package_store.PackageStore(
    base_url: str = PACKAGE_STORE_URL,
    api_key: str = PACKAGE_STORE_PUBLIC_KEY,
    timeout_s: float = 120
)
```

Read-only client for Harbor's package store.

**`base_url`** `= base_url.rstrip('/')`

---

```python
nemo_gym.data.tasks.harbor.package_store.PackageStore._dataset_tasks(
    version_id: str
) -> tuple[nemo_gym.data.tasks.harbor.package_store.PackageTask, ...]
```

```python
nemo_gym.data.tasks.harbor.package_store.PackageStore._json(
    method: str,
    path: str,
    params: dict[str, str] | None = None,
    body: typing.Any = None
) -> typing.Any
```

The decoded JSON reply; a body that is not JSON (an HTML error page, a reply cut short) is a store error.

```python
nemo_gym.data.tasks.harbor.package_store.PackageStore._request(
    method: str,
    path: str,
    params: dict[str, str] | None = None,
    body: typing.Any = None
) -> bytes
```

```python
nemo_gym.data.tasks.harbor.package_store.PackageStore._rows(
    table: str,
    params: dict[str, str]
) -> list[dict[str, typing.Any]]
```

```python
nemo_gym.data.tasks.harbor.package_store.PackageStore.archive_path(
    task: nemo_gym.data.tasks.harbor.package_store.PackageTask
) -> str
```

```python
nemo_gym.data.tasks.harbor.package_store.PackageStore.download_task(
    task: nemo_gym.data.tasks.harbor.package_store.PackageTask,
    target: pathlib.Path
) -> None
```

Download and unpack one task package into `target`, checking its digest.

The unpacked archive is checked with `archive_hash`, the
exclusion-free digest Harbor publishes: the archive holds exactly the files Harbor's packager
kept, so a file Gym's folder digest would skip (a `.pyc` the task ships on purpose) still
counts, and a file the packager never put there is a mismatch. The folder digest of what was
unpacked is what `fetch_package_dataset` records in the manifest and compares reruns with,
so such a package is fetched once and kept thereafter. `target` is
`<dataset folder>/<task name>`, checked to really resolve inside the dataset folder; an empty
folder there is replaced, a concurrent fetch's copy of the same package is kept, and anything
else stops the fetch.

The unpacked package is promoted with one `os.rename` of the staging folder, a sibling of
`target` on the same filesystem, so the destination is only ever absent, a complete previous
copy or the complete new one: no rename-away, no copy fallback and nothing to roll back.

```python
nemo_gym.data.tasks.harbor.package_store.PackageStore.resolve_dataset(
    ref: nemo_gym.data.tasks.harbor.package_store.PackageRef
) -> nemo_gym.data.tasks.harbor.package_store.ResolvedDataset
```

Resolve a dataset reference to its version row and task list.

```python
class nemo_gym.data.tasks.harbor.package_store.PackageStoreError()
```

**Bases:** `ValueError`

The reference cannot be resolved or a package cannot be fetched.

```python
class nemo_gym.data.tasks.harbor.package_store.PackageTask(
    org: str,
    name: str,
    content_hash: str
)
```

Dataclass

**`content_hash`** `str`

---

**`name`** `str`

---

**`org`** `str`

---

**`package`** `str`

---

```python
class nemo_gym.data.tasks.harbor.package_store.ResolvedDataset(
    ref: nemo_gym.data.tasks.harbor.package_store.PackageRef,
    version_id: str,
    content_hash: str,
    tasks: tuple[nemo_gym.data.tasks.harbor.package_store.PackageTask, ...]
)
```

Dataclass

**`content_hash`** `str`

---

**`ref`** `PackageRef`

---

**`tasks`** `tuple[PackageTask, ...]`

---

**`version_id`** `str`

---

```python
nemo_gym.data.tasks.harbor.package_store._build_manifest(
    folder: pathlib.Path,
    dataset: nemo_gym.data.tasks.harbor.package_store.ResolvedDataset,
    digests: dict[str, str]
) -> nemo_gym.data.tasks.harbor.manifest.Manifest
```

Derived facts only: per-task digests and package pins, and the reference this fetch was made from
(`PackageRef.reference`); the dataset's name and version stay in `dataset.toml`.

Every task of the store's version is pinned; a digest is recorded only for the tasks in `digests`, the
folders this fetch has verified (or trusts from the previous manifest, see `_checkpoint_digests`).
Nothing is hashed here: a folder the loop has not reached yet may hold a local edit, and a manifest that
vouched for it would make the next fetch, and `prepare`, accept the edit as the store's copy. A task
whose folder is not verified keeps its pin and gets no digest, which is how an interrupted fetch names
the tasks still to come, and a folder the store does not name stays unrecorded.

```python
nemo_gym.data.tasks.harbor.package_store._check_recorded_dataset(
    folder: pathlib.Path,
    previous: nemo_gym.data.tasks.harbor.manifest.Manifest | None,
    dataset: nemo_gym.data.tasks.harbor.package_store.ResolvedDataset
) -> None
```

Refuse `folder` when its manifest was written for another package, or another version, than `dataset`.

```python
nemo_gym.data.tasks.harbor.package_store._check_recorded_identity(
    folder: pathlib.Path,
    previous: nemo_gym.data.tasks.harbor.manifest.Manifest | None,
    dataset: nemo_gym.data.tasks.harbor.package_store.ResolvedDataset
) -> None
```

Refuse `folder` when its manifest was written for another package than `dataset`.

The manifest holds no copy of the dataset's name (that lives in `dataset.toml`), and one written before the
fetch's reference was recorded has no `ref` either, but every task it
pins carries the task's own package, `<org>/<task>`, as the store names it. A dataset may collect
tasks published under any org (Harbor's `dataset.toml` pins each task as `org/name`, and the store
resolves each to its own package), so a pin is compared with the package `dataset` resolved that task
to, never with `<dataset org>/<task>`. Folder names stay readable (`<name>`, `<name>-<tag>`), so
those pins are what tell `alice/evals@1.0` from `bob/evals@1.0` in the same `evals-1.0` folder when
the two pin different packages for a task; two datasets pinning the very same task packages hold the
same bytes, and either completes the folder, while a task one pins and the other lacks is a version
difference (`_stale_reason`). A manifest another fetcher wrote (the registry path) is refused by
its source. The remedy is the same as for an edited task folder.

```python
nemo_gym.data.tasks.harbor.package_store._checkpoint_digests(
    folder: pathlib.Path,
    digests: dict[str, str],
    recorded: dict[str, str]
) -> dict[str, str]
```

The digests a progress manifest may record: what this fetch verified, plus the previous manifest's
digest for a task it has not reached yet whose folder is still there.

The previous digest was verified when it was written and pins the same task version
(`_recorded_digests`), so keeping it loses nothing and keeps the folder checked against the copy the
previous fetch wrote rather than against the store's hash. It is never recomputed from the folder: the
loop checks that folder when it gets there and refuses an edit. A task whose folder is gone drops its
digest and is fetched again.

```python
nemo_gym.data.tasks.harbor.package_store._claim_folder(
    folder: pathlib.Path,
    dataset: nemo_gym.data.tasks.harbor.package_store.ResolvedDataset,
    digests: dict[str, str]
) -> None
```

Record which dataset `folder` holds before its first task lands.

Called when the folder has no manifest, right before the first download. The claim is the manifest
with every task pinned and only the digests verified so far (none in a fresh folder; a folder already
there for a later task is not hashed, the loop checks it when it gets there), created
exclusively; each task downloaded after it is recorded as it lands. Of two fetches
racing for an unclaimed folder exactly one writes the claim, and the other reads it back and is
refused unless it fetches the same dataset (then both complete the folder, and
`PackageStore.download_task` keeps whichever copy of a task lands first). An interrupted fetch
leaves the claim behind, so a later fetch of another package, or of another version of this one, is
refused before it looks at any task, and a later fetch of the same dataset completes the folder.

```python
nemo_gym.data.tasks.harbor.package_store._holds_store_copy(
    target: pathlib.Path,
    task: nemo_gym.data.tasks.harbor.package_store.PackageTask
) -> bool
```

Whether the task folder at `target` is the store's `task`, by either digest.

A freshly unpacked archive matches Harbor's published hash by `archive_hash` (its files are
exactly the published set, names Gym's folder digest skips included), and a folder a run has
since left byte-code in matches it by `content_hash` unless the package itself ships such a
name. Either match means the folder is the store's copy.

```python
nemo_gym.data.tasks.harbor.package_store._read_manifest(
    folder: pathlib.Path
) -> nemo_gym.data.tasks.harbor.manifest.Manifest | None
```

The manifest a previous fetch left in `folder`, or `None` when there is none.

One this Gym cannot read stops the fetch with the folder untouched: a manifest a newer Gym wrote
(a schema this Gym does not read) raises as itself, and a malformed one (not TOML, or not the
expected tables) as a `PackageStoreError`. The manifest is the folder's claim, which package
and version its task folders belong to, so a folder whose claim cannot be read is never completed,
reused or rewritten; the remedy is the same as for an edited task folder.

```python
nemo_gym.data.tasks.harbor.package_store._recorded_digests(
    previous: nemo_gym.data.tasks.harbor.manifest.Manifest | None,
    dataset: nemo_gym.data.tasks.harbor.package_store.ResolvedDataset
) -> dict[str, str]
```

Gym's digest of each task folder as the previous fetch wrote it, by task name.

Only an entry that pins the task version `dataset` names counts, so a hand-edited or foreign
manifest never vouches for a folder.

```python
nemo_gym.data.tasks.harbor.package_store._stale_reason(
    previous: nemo_gym.data.tasks.harbor.manifest.Manifest | None,
    dataset: nemo_gym.data.tasks.harbor.package_store.ResolvedDataset
) -> str | None
```

How the store's version differs from the one the manifest was written for, or `None`.

```python
nemo_gym.data.tasks.harbor.package_store._write_manifest(
    folder: pathlib.Path,
    dataset: nemo_gym.data.tasks.harbor.package_store.ResolvedDataset,
    digests: dict[str, str]
) -> None
```

```python
nemo_gym.data.tasks.harbor.package_store.fetch_package_dataset(
    ref: nemo_gym.data.tasks.harbor.package_store.PackageRef,
    root: pathlib.Path,
    store: nemo_gym.data.tasks.harbor.package_store.PackageStore | None = None,
    folder: pathlib.Path | None = None
) -> pathlib.Path
```

Fetch every task of a package-store dataset into `root/<folder>/<task>/`.

The dataset folder is `root/<name>[-<tag | 12 digest chars>]/` (`PackageRef.folder_name`) unless
`folder` names it: the folder of a committed entry under `environments/harbor/` that declares this
dataset as its `[gym] source` (`nemo_gym.data.tasks.harbor.entries`). The entry's own files there (README,
YAML overlays, `dataset.toml`) are not task folders and are left alone; the manifest's claim and the
per-task checks below are the same whichever folder the tasks land in.

A task folder whose digest already matches is kept, so a rerun after a partial
fetch completes it: a folder the manifest records is compared with the digest recorded
for it (Gym's digest of the folder as fetched, which a package shipping a name that
digest skips makes differ from the store's hash), and a folder no manifest records yet
(the fetch was interrupted) with the store's hash by either digest. A task folder whose
digest differs (it was edited locally, or the store republished it) stops the fetch with
both digests named, so edits are never discarded silently; deleting or moving the folder
aside and running again fetches the store's copy. A folder whose manifest was written for
another version of the dataset (`latest` moved), or for another package (its task pins
name other packages than the store resolves this dataset's tasks to), is refused the same
way, so one folder never mixes two versions or two packages. That manifest exists from the
first download on: an unclaimed folder is claimed (`_claim_folder`, the pins without
digests, created exclusively) before its first task lands, so an interrupted fetch leaves a
folder that says whose it is, and two fetches racing for one folder never fill it with two
datasets. A manifest this Gym cannot read, one a newer Gym wrote or a malformed one, stops
the fetch with the folder untouched, since this Gym cannot tell what it claimed. Anything
else in a task's place (a folder without `task.toml`, a file, a symlink) also stops the
fetch: only what this function fetched is ever replaced. Returns the dataset folder.

```python
nemo_gym.data.tasks.harbor.package_store.PACKAGE_STORE_PUBLIC_KEY = 'sb_publishable_Z-vuQbpvpG-PStjbh4yE0Q_e-d3MTIH'
```

```python
nemo_gym.data.tasks.harbor.package_store.PACKAGE_STORE_SOURCE = 'harbor-package-store'
```

```python
nemo_gym.data.tasks.harbor.package_store.PACKAGE_STORE_URL = 'https://ofhuhcpkvzjlejydnvyd.supabase.co'
```

```python
nemo_gym.data.tasks.harbor.package_store._DIGEST = re.compile('(sha256:)?([a-f0-9]{64})')
```

```python
nemo_gym.data.tasks.harbor.package_store._PAGE_SIZE = 1000
```

```python
nemo_gym.data.tasks.harbor.package_store._SAFE = re.compile('[^A-Za-z0-9._-]+')
```

```python
nemo_gym.data.tasks.harbor.package_store.logger = logging.getLogger(__name__)
```