> This page is for version 0.4.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.tasks.harbor.package_store

Harbor package-store datasets: `harbor:<org>/<name>[@<tag | sha256:digest>]`.

Harbor publishes some datasets (Terminal Bench 4 among them) as packages rather than
git checkouts. A dataset version is a list of task packages, each pinned by the
content hash Gym also uses as its task digest. Fetching a dataset resolves the version,
downloads every task archive, checks each against Harbor's published hash and writes
`manifest.toml`, whose `[dataset]` table pins the package (`org/name`), the reference
and the content hash the fetch resolved, and whose `[tasks]` entries pin every task package and
record Gym's digest of each task folder as fetched. The manifest is also the folder's claim: it is
written with the identity and the task pins, and no digests, before the first task lands, so a
folder an interrupted fetch leaves behind still says which package and version it belongs to. A
later fetch into the same folder reads that back first, refuses a folder claimed for another
package or an earlier resolution of this one, and compares each task folder with the digest
recorded for it.

The store is Harbor's public PostgREST and storage API. Only the public
(publishable) key is needed to read published packages.

## Module Contents

### Classes

| Name                                                                          | Description                                                                          |
| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| [`PackageRef`](#nemo_gym-tasks-harbor-package_store-PackageRef)               | `org/name@ref`; `ref` is a tag such as `4.0.0`, `latest` or `sha256:<digest>`. |
| [`PackageStore`](#nemo_gym-tasks-harbor-package_store-PackageStore)           | Read-only client for Harbor's package store.                                         |
| [`PackageStoreError`](#nemo_gym-tasks-harbor-package_store-PackageStoreError) | The reference cannot be resolved or a package cannot be fetched.                     |
| [`PackageTask`](#nemo_gym-tasks-harbor-package_store-PackageTask)             | -                                                                                    |
| [`ResolvedDataset`](#nemo_gym-tasks-harbor-package_store-ResolvedDataset)     | -                                                                                    |

### Functions

| Name                                                                                        | Description                                                                                                |
| ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| [`_check_recorded_identity`](#nemo_gym-tasks-harbor-package_store-_check_recorded_identity) | Refuse `folder` when its manifest's `[dataset]` table pins another dataset than `dataset`.                 |
| [`_claim_folder`](#nemo_gym-tasks-harbor-package_store-_claim_folder)                       | Record which dataset `folder` holds before its first task lands.                                           |
| [`_holds_store_copy`](#nemo_gym-tasks-harbor-package_store-_holds_store_copy)               | Whether the task folder at `target` is the store's `task`, by either digest.                               |
| [`_manifest_text`](#nemo_gym-tasks-harbor-package_store-_manifest_text)                     | The manifest: the dataset pin, then per task the store's hash and, once the task has landed, Gym's digest. |
| [`_read_manifest`](#nemo_gym-tasks-harbor-package_store-_read_manifest)                     | The manifest a previous fetch left in `folder`, parsed, or `None` when there is none.                      |
| [`_recorded_dataset`](#nemo_gym-tasks-harbor-package_store-_recorded_dataset)               | The `[dataset]` table of a parsed manifest (string fields only), or `{}`.                        |
| [`_recorded_digests`](#nemo_gym-tasks-harbor-package_store-_recorded_digests)               | Gym's digest of each task folder as the previous fetch wrote it, by task name.                             |
| [`_write_manifest`](#nemo_gym-tasks-harbor-package_store-_write_manifest)                   | Write `manifest.toml` atomically: a failed write leaves the previous, truthful file intact.                |
| [`_write_text`](#nemo_gym-tasks-harbor-package_store-_write_text)                           | Write `text` to `fd` and flush it to disk; the descriptor is closed.                                       |
| [`fetch_package_dataset`](#nemo_gym-tasks-harbor-package_store-fetch_package_dataset)       | Fetch every task of a package-store dataset into `root/<folder>/<task>/`.                      |

### Data

[`MANIFEST_FILE`](#nemo_gym-tasks-harbor-package_store-MANIFEST_FILE)

[`PACKAGE_STORE_PUBLIC_KEY`](#nemo_gym-tasks-harbor-package_store-PACKAGE_STORE_PUBLIC_KEY)

[`PACKAGE_STORE_URL`](#nemo_gym-tasks-harbor-package_store-PACKAGE_STORE_URL)

[`_DIGEST`](#nemo_gym-tasks-harbor-package_store-_DIGEST)

[`_NO_HARD_LINKS`](#nemo_gym-tasks-harbor-package_store-_NO_HARD_LINKS)

[`_PAGE_SIZE`](#nemo_gym-tasks-harbor-package_store-_PAGE_SIZE)

[`_SAFE`](#nemo_gym-tasks-harbor-package_store-_SAFE)

[`_UNREADABLE_REMEDY`](#nemo_gym-tasks-harbor-package_store-_UNREADABLE_REMEDY)

[`logger`](#nemo_gym-tasks-harbor-package_store-logger)

### API

```python
class nemo_gym.tasks.harbor.package_store.PackageRef(
    org: str,
    name: str,
    ref: str = 'latest'
)
```

Dataclass

`org/name@ref`; `ref` is a tag such as `4.0.0`, `latest` or `sha256:<digest>`.

**`digest`** `str | None`

---

**`name`** `str`

---

**`org`** `str`

---

**`package`** `str`

---

**`ref`** `str = 'latest'`

---

```python
nemo_gym.tasks.harbor.package_store.PackageRef.folder_name(
    resolved_digest: str
) -> str
```

`<name>` for `latest`, `<name>-<tag>` for a tag, `<name>-<12 digest chars>` for a digest pin.

`latest` is what a plain `harbor:org/name` reference means, so its folder carries the plain
name and `manifest.toml` records the content hash it resolved to. A pinned tag or digest keeps
its suffix, so two pins of one dataset never share a folder.

```python
class nemo_gym.tasks.harbor.package_store.PackageStore(
    base_url: str = PACKAGE_STORE_URL,
    api_key: str = PACKAGE_STORE_PUBLIC_KEY,
    timeout_s: float = 120
)
```

Read-only client for Harbor's package store.

**`base_url`** `= base_url.rstrip('/')`

---

```python
nemo_gym.tasks.harbor.package_store.PackageStore._dataset_tasks(
    version_id: str
) -> tuple[nemo_gym.tasks.harbor.package_store.PackageTask, ...]
```

```python
nemo_gym.tasks.harbor.package_store.PackageStore._json(
    method: str,
    path: str,
    params: dict[str, str] | None = None,
    body: typing.Any = None
) -> typing.Any
```

The decoded JSON reply; a body that is not JSON (an HTML error page, a reply cut short) is a store error.

```python
nemo_gym.tasks.harbor.package_store.PackageStore._request(
    method: str,
    path: str,
    params: dict[str, str] | None = None,
    body: typing.Any = None
) -> bytes
```

```python
nemo_gym.tasks.harbor.package_store.PackageStore._rows(
    table: str,
    params: dict[str, str]
) -> list[dict[str, typing.Any]]
```

```python
nemo_gym.tasks.harbor.package_store.PackageStore.archive_path(
    task: nemo_gym.tasks.harbor.package_store.PackageTask
) -> str
```

```python
nemo_gym.tasks.harbor.package_store.PackageStore.download_task(
    task: nemo_gym.tasks.harbor.package_store.PackageTask,
    target: pathlib.Path
) -> None
```

Download and unpack one task package into `target`, checking its digest.

The unpacked archive is checked with `archive_hash`, the
exclusion-free digest Harbor publishes: the archive holds exactly the files Harbor's packager
kept, so a file Gym's folder digest would skip (a `.pyc` the task ships on purpose) still
counts, and a file the packager never put there is a mismatch. The folder digest of what was
unpacked is what `fetch_package_dataset` records in the manifest and compares reruns with,
so such a package is fetched once and kept thereafter. `target` is
`<dataset folder>/<task name>`, checked to really resolve inside the dataset folder; an empty
folder there is replaced, a concurrent fetch's copy of the same package is kept, and anything
else stops the fetch.

The unpacked package is promoted with one `os.rename` of the staging folder, a sibling of
`target` on the same filesystem, so the destination is only ever absent, a complete previous
copy or the complete new one: no rename-away, no copy fallback and nothing to roll back.

```python
nemo_gym.tasks.harbor.package_store.PackageStore.resolve_dataset(
    ref: nemo_gym.tasks.harbor.package_store.PackageRef
) -> nemo_gym.tasks.harbor.package_store.ResolvedDataset
```

Resolve a dataset reference to its version row and task list.

```python
class nemo_gym.tasks.harbor.package_store.PackageStoreError()
```

**Bases:** `ValueError`

The reference cannot be resolved or a package cannot be fetched.

```python
class nemo_gym.tasks.harbor.package_store.PackageTask(
    org: str,
    name: str,
    content_hash: str
)
```

Dataclass

**`content_hash`** `str`

---

**`name`** `str`

---

**`org`** `str`

---

**`package`** `str`

---

```python
class nemo_gym.tasks.harbor.package_store.ResolvedDataset(
    ref: nemo_gym.tasks.harbor.package_store.PackageRef,
    version_id: str,
    content_hash: str,
    tasks: tuple[nemo_gym.tasks.harbor.package_store.PackageTask, ...]
)
```

Dataclass

**`content_hash`** `str`

---

**`ref`** `PackageRef`

---

**`tasks`** `tuple[PackageTask, ...]`

---

**`version_id`** `str`

---

```python
nemo_gym.tasks.harbor.package_store._check_recorded_identity(
    folder: pathlib.Path,
    dataset: nemo_gym.tasks.harbor.package_store.ResolvedDataset,
    previous: dict[str, typing.Any] | None
) -> None
```

Refuse `folder` when its manifest's `[dataset]` table pins another dataset than `dataset`.

Folder names stay readable (`<name>`, `<name>-<tag>`), so the manifest's `[dataset]` table is
what tells `alice/evals@1.0` from `bob/evals@1.0` in the same `evals-1.0` folder, and a
`latest` fetch from the version the tag has since moved to. Either mismatch names the recorded
identity and the requested one, and a manifest whose table names no package-store dataset at all is
refused the same way: a folder with a manifest is never treated as unclaimed. The remedy is the same
as for an edited task folder.

```python
nemo_gym.tasks.harbor.package_store._claim_folder(
    folder: pathlib.Path,
    dataset: nemo_gym.tasks.harbor.package_store.ResolvedDataset,
    digests: dict[str, str]
) -> None
```

Record which dataset `folder` holds before its first task lands.

Called when the folder has no manifest, right before the first download. The claim is the manifest
with the `[dataset]` identity, every task pinned and only the digests verified so far (none in a
fresh folder), created exclusively; each task downloaded after it is recorded as it lands. Of two
fetches racing for an unclaimed folder exactly one writes the claim, and the other reads it back and
is refused unless it fetches the same dataset (then both complete the folder, and
`PackageStore.download_task` keeps whichever copy of a task lands first). An interrupted fetch
leaves the claim behind, so a later fetch of another package, or of another version of this one, is
refused before it looks at any task, and a later fetch of the same dataset completes the folder.

```python
nemo_gym.tasks.harbor.package_store._holds_store_copy(
    target: pathlib.Path,
    task: nemo_gym.tasks.harbor.package_store.PackageTask
) -> bool
```

Whether the task folder at `target` is the store's `task`, by either digest.

A freshly unpacked archive matches Harbor's published hash by `archive_hash` (its files are
exactly the published set, names Gym's folder digest skips included), and a folder a run has
since left byte-code in matches it by `content_hash` unless the package itself ships such a
name. Either match means the folder is the store's copy.

```python
nemo_gym.tasks.harbor.package_store._manifest_text(
    dataset: nemo_gym.tasks.harbor.package_store.ResolvedDataset,
    digests: dict[str, str]
) -> str
```

The manifest: the dataset pin, then per task the store's hash and, once the task has landed, Gym's digest.

Every task is pinned from the first write on; a task whose folder is not there yet has no `digest`, so
the manifest an interrupted fetch leaves names exactly the tasks still to come.

```python
nemo_gym.tasks.harbor.package_store._read_manifest(
    folder: pathlib.Path
) -> dict[str, typing.Any] | None
```

The manifest a previous fetch left in `folder`, parsed, or `None` when there is none.

One this Gym cannot read stops the fetch with the folder untouched: not TOML, or without the
`[dataset]` table, or with `[dataset]` or `[tasks]` not a table, is a `PackageStoreError`.
The manifest is the folder's claim, which package and version its task folders belong to, so a folder
whose claim cannot be read is never completed, reused or rewritten; the remedy is the same as for an
edited task folder.

```python
nemo_gym.tasks.harbor.package_store._recorded_dataset(
    manifest: dict[str, typing.Any]
) -> dict[str, str]
```

The `[dataset]` table of a parsed manifest (string fields only), or `{}`.

```python
nemo_gym.tasks.harbor.package_store._recorded_digests(
    previous: dict[str, typing.Any] | None,
    dataset: nemo_gym.tasks.harbor.package_store.ResolvedDataset
) -> dict[str, str]
```

Gym's digest of each task folder as the previous fetch wrote it, by task name.

Only an entry that pins the task version `dataset` names counts, so a hand-edited or foreign
manifest never vouches for a folder, and a task the claim pinned but that has not landed yet (its
entry has no `digest`) is not recorded.

```python
nemo_gym.tasks.harbor.package_store._write_manifest(
    folder: pathlib.Path,
    dataset: nemo_gym.tasks.harbor.package_store.ResolvedDataset,
    digests: dict[str, str],
    exclusive: bool = False
) -> None
```

Write `manifest.toml` atomically: a failed write leaves the previous, truthful file intact.

The complete file is written to a temporary sibling (unique per process and thread, so concurrent
fetches of one folder never share it) and moved into place, so a reader never sees a partial manifest.
A rewrite keeps the permissions the file had; a new manifest gets the usual umask-aware mode.

With `exclusive`, the manifest is created only where none is and `FileExistsError` is raised
otherwise, the target untouched (the temporary is written and removed either way): the complete file is
hard-linked into place, the atomic create-unless-present that `os.replace` lacks, so of two writers
racing for an empty folder exactly one succeeds. A filesystem without hard links falls back to an
exclusive create of the target itself, still one winner, though a reader may then glimpse the file
before its last byte. `_claim_folder` claims a folder this way before its first task lands.

```python
nemo_gym.tasks.harbor.package_store._write_text(
    fd: int,
    text: str
) -> None
```

Write `text` to `fd` and flush it to disk; the descriptor is closed.

```python
nemo_gym.tasks.harbor.package_store.fetch_package_dataset(
    ref: nemo_gym.tasks.harbor.package_store.PackageRef,
    root: pathlib.Path,
    store: nemo_gym.tasks.harbor.package_store.PackageStore | None = None
) -> pathlib.Path
```

Fetch every task of a package-store dataset into `root/<folder>/<task>/`.

A task folder whose digest already matches is kept, so a rerun after a partial
fetch completes it: a folder the manifest records is compared with the digest recorded
for it (Gym's digest of the folder as fetched, which a package shipping a name that
digest skips makes differ from the store's hash), and a folder no manifest records yet
(the fetch was interrupted) with the store's hash by either digest. A task folder whose
digest differs (it was edited locally, or the store republished it) stops the fetch with
both digests named, so edits are never discarded silently; deleting or moving the folder
aside and running again fetches the store's copy. A dataset folder whose `manifest.toml`
pins another package, or an earlier resolution of this one (`latest` has moved), stops
the fetch before any task is looked at, with the recorded and the requested identity named,
so one folder never mixes two versions or two packages. That manifest exists from the first
download on: an unclaimed folder is claimed (`_claim_folder`, the identity and the
pins without digests, created exclusively) before its first task lands, so an interrupted
fetch leaves a folder that says whose it is, and two fetches racing for one folder never
fill it with two datasets. A manifest this Gym cannot read stops the fetch with the folder
untouched, since this Gym cannot tell what it claimed. Anything else in a task's place (a
folder without `task.toml`, a file, a symlink) also stops the fetch: only what this
function fetched is ever replaced. Returns the dataset folder.

```python
nemo_gym.tasks.harbor.package_store.MANIFEST_FILE = 'manifest.toml'
```

```python
nemo_gym.tasks.harbor.package_store.PACKAGE_STORE_PUBLIC_KEY = 'sb_publishable_Z-vuQbpvpG-PStjbh4yE0Q_e-d3MTIH'
```

```python
nemo_gym.tasks.harbor.package_store.PACKAGE_STORE_URL = 'https://ofhuhcpkvzjlejydnvyd.supabase.co'
```

```python
nemo_gym.tasks.harbor.package_store._DIGEST = re.compile('(sha256:)?([a-f0-9]{64})')
```

```python
nemo_gym.tasks.harbor.package_store._NO_HARD_LINKS = {errno.EPERM, errno.ENOTSUP, errno.EOPNOTSUPP, errno.ENOSYS, errno.EACCES}
```

```python
nemo_gym.tasks.harbor.package_store._PAGE_SIZE = 1000
```

```python
nemo_gym.tasks.harbor.package_store._SAFE = re.compile('[^A-Za-z0-9._-]+')
```

```python
nemo_gym.tasks.harbor.package_store._UNREADABLE_REMEDY = 'The manifest records which package and version the task folders belong to, so t...
```

```python
nemo_gym.tasks.harbor.package_store.logger = logging.getLogger(__name__)
```