nemo_gym.data.tasks.harbor.manifest

manifest.toml in memory. tasks maps a task name to its digest and pin keys.

nemo_gym.data.tasks.harbor.manifest.Manifest._write_text(
    fd: int
) -> None

nemo_gym.data.tasks.harbor.manifest.Manifest.build(
    folder: pathlib.Path,
    source: str,
    pins: dict[str, dict[str, str]] | None = None,
    digests: dict[str, str] | None = None,
    names: collections.abc.Iterable[str] | None = None,
    ref: str | None = None
) -> nemo_gym.data.tasks.harbor.manifest.Manifest

nemo_gym.data.tasks.harbor.manifest.Manifest.read(
    folder: pathlib.Path
) -> nemo_gym.data.tasks.harbor.manifest.Manifest | None

nemo_gym.data.tasks.harbor.manifest.Manifest.text() -> str

nemo_gym.data.tasks.harbor.manifest.Manifest.write(
    folder: pathlib.Path,
    exclusive: bool = False
) -> pathlib.Path

Bases: ValueError

The manifest cannot be read, or the task folders no longer match it.

Bases: ManifestError

The manifest was written by a newer Gym (a schema_version this Gym does not read).

Distinct from a malformed manifest: the file is intact and says more than this Gym understands, so a fetcher must leave it alone rather than rewrite it with what it knows.

value as a TOML basic string, ASCII only.

JSON string escapes are valid TOML basic-string escapes (the quote, the backslash, the controls and U+007F as \u007f, other non-ASCII as \uXXXX) except that JSON writes a character above U+FFFF as a UTF-16 surrogate pair, which TOML forbids; that pair becomes TOML’s \UXXXXXXXX.

The first five of names and how many there are in all.

Write the manifest; a folder that cannot be written (read-only shared storage) keeps its in-memory identity.

Recompute the manifest of folder from disk and reconcile it with the one on disk.

• No manifest: the folder is a local dataset; one is written with source local, unless git tracks the folder, whose identity is then computed in memory and nothing written.
• A manifest naming a source this Gym does not know (written by a newer Gym, or edited): refused, never read as local, because this Gym cannot tell whether the folders still match it.
• A manifest without digests (written before they existed): upgraded in place, pins kept.
• A manifest with per-task digests but no dataset digest (the package-store format before the schema was numbered): the folders are held to the recorded digests, and an unchanged folder is upgraded in place, the recorded digests kept, so a cache written by an earlier Gym keeps its fetch record.
• A fetched manifest pinning a task with no digest: the fetch that wrote it was interrupted before that task landed (a fetcher claims the folder with every pin before its first download), so the folder is refused, named with the tasks still to come, until the fetch is run again.
• Task folders differ from a local manifest: the folder is its own source of truth, so the manifest is rewritten and the new digest reported.
• No manifest, or a local one, in a folder that a fetched dataset’s check reaches (a subfolder of a fetched dataset, within its discovery range, see enclosing_manifest): refused, and no local manifest written there. The task folders under it are that dataset’s, recorded in its manifest and held to it by its check and by check_task_manifest; a local manifest beside them would prepare an edited one as local and shadow the fetched manifest for the single-task check.
• Task folders differ from a fetched manifest: the folder was edited by hand, or its source moved since the fetch. ManifestError says which tasks and what to do.
• Only dataset.toml or a dataset-level task_data.py / verifier.py moved: the Gym layer changed on purpose, the task folders are the same, so the derived facts are rewritten without a fetch, and the rewrite says which.

digests are folder digests a loader already computed, extended in place with the rest. Returns the manifest now on disk.

Hold one task folder to the manifest of the dataset folder whose check reaches it, when that is a fetch’s.

A single task folder target (<dataset>/<task>, or a task deeper under the dataset folder) is prepared with the dataset.toml and image records beside it but is no folder of task folders, so check_manifest never runs for it; without this check an edited task of a fetched dataset prepared alone, as local, while the dataset folder was refused. The manifest that holds the task is the one the dataset folder’s check holds it to, not only its parent’s: that check discovers tasks under subfolders too (iter_task_folders, at most MAX_DISCOVERY_DEPTH levels down, never inside a hidden folder or through a link) and names each by its folder name, so a task moved into a subfolder of a fetched dataset is still the task its manifest records and is held to it here, while a task that check never sees (hidden, deeper, under a hidden or linked folder, or inside another task folder, as a fixture a task ships) is held to nothing, and a manifest above that range is not read (enclosing_manifest; a folder in between that carries a fetched manifest of its own, such as a dataset fetched inside another folder, holds its tasks itself, and a local one in between is passed over, as is a manifest.toml this Gym cannot read as a fetch’s record, another tool’s for one). The task’s own parent is read as it was: it is the dataset root the task is prepared with, so its manifest is the task’s dataset’s whatever it says, and one naming a source this Gym does not know, or written by a newer Gym, refuses the task as it refuses the dataset folder. The task’s folder digest is compared with the one that manifest records for it, with task_differences’ words: a folder that differs was edited by hand or its source moved, one the manifest does not record was never fetched, one that cannot be hashed is refused as the dataset folder’s check refuses it, and one recorded without a digest is a fetch that never finished. Nothing is written: the manifest stays the dataset folder’s record, and it is not upgraded or rewritten here. Returns that manifest, for the task’s identity to carry its source (the task is still prepared with the dataset.toml and image records beside it, as any single task folder target is: the manifest holds its content, not the settings the preparation applies), or None when no fetched manifest in that range holds the task (a standalone task folder, whose parent is whatever directory holds it, or one whose nearest manifest is local: the folder is its own source of truth, and the dataset folder’s next prepare records the edit) or that manifest is from before digests were recorded (pins only; the dataset folder’s prepare upgrades it).

digests are folder digests already computed, keyed by task name as check_manifest keys them: the task’s is used when present (a loader’s, to check the folder as it loaded) and hashed and stored otherwise.

The dataset’s identity: sorted per-task digests, then dataset.toml, then the dataset-level files when there are any (dataset_files other than the digest of nothing).

Digest of the dataset-level Gym files that shape tasks (DATASET_FILES): name and content of each one present, in order; the digest of nothing when the dataset ships none (every hub dataset).

Digest of dataset.toml alone, or of nothing when the file is absent.

The nearest fetched manifest above path whose dataset folder’s check reaches path, with its folder.

A dataset folder’s check (check_manifest) sees the task folders iter_task_folders discovers: at most MAX_DISCOVERY_DEPTH levels down, never inside a hidden folder, never through a symbolic link, never inside a task folder it found (a task’s own files are its content, not more tasks). So the folders whose manifest can hold path are its nearest levels parents, reached from path through visible folders that are not links or task folders; a manifest further up, or above a hidden, linked or task folder, belongs to a dataset whose check never sees path and is not read. Only a manifest this Gym reads as a fetch’s record (a source it knows other than local) claims the folders below it: a local one is passed over (a local folder inside a fetched dataset’s range is no dataset of its own, check_manifest refuses to prepare it as one, so the fetched manifest above holds its task folders), and so is a manifest.toml that is no record this Gym can read, such as another tool’s in a project root, or a newer Gym’s, which claims nothing here and is refused where it is the folder’s own, by that folder’s check. None when no such manifest is in reach.

Which part of the Gym layer differs between two manifests of one task set, for a rewrite notice.

A manifest from before the dataset digest existed (per-task digests only, see Manifest.records_content) is rewritten for that alone, whatever dataset.toml held when it was written.

Bring a fetched manifest’s Gym-layer digests up to date with folder without touching its task records.

A fetch records dataset.toml and the dataset-level Gym files as it found them, and rewrites the manifest only when it fetches something: a registry fetch with every folder in place, or a pull that only replaced the copy of an entry’s dataset.toml beside the tasks (nemo_gym.data.tasks.harbor.entries.prepare_destination), leaves dataset_toml_digest and the dataset digest describing a file that is no longer there. This recomputes those digests from disk and keeps every task’s recorded digest and pin, and the fetch’s reference, as they are (nothing is hashed, so a hand-edited folder is still caught by check_manifest, which rehashes), then writes the manifest when its text changed, with the notice check_manifest prints for the same rewrite (Rewrote ...; dataset digest <old> -> <new>), so a change of the dataset’s identity is announced whichever function applies it. Returns the manifest now on disk: None without one, and the old one untouched when it records pins only (check_manifest upgrades those) or names a source this Gym does not know (check_manifest refuses those).

The one-line notice that folder’s manifest was rewritten, and how the dataset digest moved.

One line naming the tasks whose digest differs between a manifest and the folders, or None.

A task is missing when the manifest recorded its digest and the folder is gone (on_disk lacks it, or maps it to None because its pin outlived the folder), changed when both digests exist and differ, and not in the manifest when only the folder exists. One of unhashable (its folder is there but has no digest now, see Manifest.build) whose digest the manifest recorded is cannot be hashed, not missing.

The task folders under folder, as the loader discovers them (iter_task_folders: recursively, hidden folders skipped, links not followed), sorted by path. A task is named by its folder’s name there too.

Whether git tracks files under folder: the repository’s own example datasets, or a checkout.

Such a folder is under version control already, so prepare computes its identity in memory instead of leaving a derived manifest.toml in the working tree. No git, no repository or any other failure reads as not tracked.