nemo_gym.data.tasks.harbor.digest

Whole-folder digest of a Harbor task.

Same record format as Harbor’s package publisher and Gym’s Terminal Bench 4 loader: SHA256 over sorted relative_path NUL file_sha256 LF records covering task.toml, instruction.md, README.md, trajectory.json and everything under environment/, tests/, solution/ and steps/. The digest travels in task_data under DIGEST_KEY so the resources server can refuse a row whose folder changed after materialization. content_hash refuses directory symlinks because this file-content format does not represent their targets.

Beside it, DATASET_DIGEST_KEY carries the digest of the whole dataset (manifest.toml’s [dataset].digest, see nemo_gym.data.tasks.harbor.manifest). Rollout collection copies both _ng_* keys from the row onto the results row it writes, so every rollouts.jsonl row says which task folder and which dataset it ran. context_digest is the second digest here: Harbor’s environment id of a Dockerfile build context, which keys and names the images prepare builds. inputs_digest is the third: everything a build of that context reads, recorded with the built image so a context whose Harbor id is unchanged but whose inputs are not (a mode, a symbolic link, an empty folder, a build setting) is rebuilt, and inputs_differences spells out where two trees of one Harbor id part, for the task left out of a shared build.

Module Contents

Functions

Name	Description
`_covered_files`	The regular files under the resolved task folder `path` that a digest covers, in record order.
`_covered_paths`	Every entry under the resolved task folder `path` that a digest reads, links included: the top-level task
`_describe_entry`	-
`_record`	One `relative_path NUL file_sha256 LF` record of `file` under the resolved task folder `path`.
`_tree_entries`	Every entry under the context folder `context` as `inputs_digest` reads it, sorted by relative posix
`archive_hash`	Harbor’s published content hash of the unpacked task package at `path`.
`content_hash`	Digest the task folder at `path`.
`context_digest`	Harbor’s environment id of a Dockerfile build context (a task’s `environment/` folder).
`inputs_differences`	Where the context folders `ours` and `theirs` differ in what `inputs_digest` reads, in path order.
`inputs_digest`	Digest of everything an image build of the context folder at `context` reads.
`is_covered`	Whether the task path at `relative` (a posix path under the task folder) is one `content_hash`
`is_ignored`	Whether `content_hash` skips the task file at `relative` (a posix path under the task folder):

Data

API

nemo_gym.data.tasks.harbor.digest._covered_files(
    path: pathlib.Path
) -> list[pathlib.Path]

The regular files under the resolved task folder path that a digest covers, in record order.

nemo_gym.data.tasks.harbor.digest._covered_paths(
    path: pathlib.Path
) -> list[pathlib.Path]

Every entry under the resolved task folder path that a digest reads, links included: the top-level task files and the covered folders with everything below them. Entries outside them (an author’s .venv/ or .git/) are not the task’s and never reach a digest.

nemo_gym.data.tasks.harbor.digest._describe_entry(
    entry: tuple[str, int, str] | None,
    against: tuple[str, int, str] | None
) -> str | None

nemo_gym.data.tasks.harbor.digest._record(
    path: pathlib.Path,
    file: pathlib.Path
) -> bytes

One relative_path NUL file_sha256 LF record of file under the resolved task folder path.

nemo_gym.data.tasks.harbor.digest._tree_entries(
    context: pathlib.Path
) -> list[tuple[str, str, int, str]]

Every entry under the context folder context as inputs_digest reads it, sorted by relative posix path: (relative_path, kind, st_mode, payload).

nemo_gym.data.tasks.harbor.digest.archive_hash(
    path: pathlib.Path
) -> str

Harbor’s published content hash of the unpacked task package at path.

The same records as content_hash over every covered file, with no exclusions: Harbor’s packager applies the task’s .gitignore (or its defaults, the same names as _IGNORED) when it chooses the archive’s files, and the published hash covers exactly those files. A downloaded archive therefore matches its advertised hash by this digest, whatever the publisher excluded or kept, while content_hash stays the digest of a task folder on disk, where byte-code caches and editor leftovers appear after a run and must not change the task’s identity.

nemo_gym.data.tasks.harbor.digest.content_hash(
    path: pathlib.Path
) -> str

Digest the task folder at path.

nemo_gym.data.tasks.harbor.digest.context_digest(
    path: pathlib.Path
) -> str

Harbor’s environment id of a Dockerfile build context (a task’s environment/ folder).

environment_content_hash over the folder: 32 hex characters of SHA-256 over the sorted (relative path, bytes) of its regular files. Symbolic links and anything under a _CONTENT_HASH_IGNORE_NAMES member (__pycache__, .git, .DS_Store) are skipped, and file modes are not part of it: parity with Harbor’s image name is the contract, not a complete cache identity. An empty folder hashes to its name, as Harbor hashes it when the task declares no docker_image (one that does never reaches a build). Two contexts with the same files get the same id, so it keys the build record (see nemo_gym.data.tasks.harbor.image_build); whether a recorded image is reused is inputs_digest’s call.

nemo_gym.data.tasks.harbor.digest.inputs_differences(
    ours: pathlib.Path,
    theirs: pathlib.Path
) -> list[tuple[str, str | None, str | None]]

Where the context folders ours and theirs differ in what inputs_digest reads, in path order.

One item per entry that is in one tree only or reads differently in the two: (relative_path, in ours, in theirs), each side a short description (a file 0755, a link 0777 -> target, a folder 0755, with (content <sha256 prefix>) on a file whose bytes differ from the other side’s) or None where the tree has no such entry. Empty exactly when the two trees’ inputs_digest agree.

nemo_gym.data.tasks.harbor.digest.inputs_digest(
    context: pathlib.Path,
    settings: collections.abc.Mapping[str, str] | None = None
) -> str

Digest of everything an image build of the context folder at context reads.

Where context_digest is Harbor’s name for the context, this is the build’s identity: SHA-256 (64 hex characters) over every entry under context, sorted by relative posix path, as relative_path NUL kind NUL st_mode NUL payload LF. kind is f for a regular file (payload: the SHA-256 of its bytes), l for a symbolic link (payload: its target, read, not followed), d for a folder (no payload, so an empty folder counts) and o for anything else; st_mode is the full lstat mode in octal, permission bits included. Nothing is skipped: the names Harbor’s hash ignores (__pycache__, .git, .DS_Store) and symbolic links are part of it, since a build tool reads them. settings are the build’s inputs outside the tree (the Dockerfile path, the build arguments), appended as NUL key NUL value LF records sorted by key. Folder ownership is not part of it: a build does not read it, and it differs between hosts that hold the same folder.

nemo_gym.data.tasks.harbor.digest.is_covered(
    relative: str
) -> bool

Whether the task path at relative (a posix path under the task folder) is one content_hash covers: one of the top-level task files, or anything under environment/, tests/, solution/ or steps/ (those folders themselves included), and not a name is_ignored skips. A folder that is covered may still hold skipped names after a run; they do not change the task’s identity.

nemo_gym.data.tasks.harbor.digest.is_ignored(
    relative: str
) -> bool

Whether content_hash skips the task file at relative (a posix path under the task folder): editor leftovers and byte-code caches, matched on any path component.

nemo_gym.data.tasks.harbor.digest.DATASET_DIGEST_KEY = '_ng_dataset_digest'

nemo_gym.data.tasks.harbor.digest.DIGEST_KEY = '_ng_digest'

nemo_gym.data.tasks.harbor.digest.GITIGNORE = '.gitignore'

nemo_gym.data.tasks.harbor.digest._DIRECTORIES = ('environment', 'tests', 'solution', 'steps')

nemo_gym.data.tasks.harbor.digest._IGNORED = ('__pycache__', '*.pyc', '.DS_Store', '*.swp', '*.swo', '*~')

nemo_gym.data.tasks.harbor.digest._KIND_NAMES = {'f': 'a file', 'l': 'a link', 'd': 'a folder', 'o': 'another kind of entry'}

nemo_gym.data.tasks.harbor.digest._SINGLE_FILES = ('task.toml', 'instruction.md', 'README.md', 'trajectory.json', 'task_data.json'...