nemo_gym.data.tasks.harbor.records

The two record files prepare writes next to the task folders, and how they are read.

compose-images.json records image configurations, built-images.json records the images built per context digest and service. This module names the files and the service keys, takes the sidecar lock, replaces a file atomically, reads it with one retry against a torn write, and reads the per-service entries with the legacy main-only fallback. The loader, the recorders, the build modules and the environment server all read through it.

Module Contents

Functions

Name	Description
`read_built_images`	The built images recorded in `folder` (the tasks’ parent), or `{}` when none were.
`read_image_configs`	The image configurations recorded in `folder` (the tasks’ parent), or `{}` when none were.
`read_json_record`	The JSON object in the record file at `path`, or `{}` when the file is missing or not an object.
`record_lock`	Hold an exclusive lock on `path`’s sidecar lock file (`<file>.lock`) for a read-modify-write of it.
`recorded_forced_tasks`	The tasks one `built-images.json` entry records as built under `builds.force` over their declared
`recorded_service_images`	The images one `built-images.json` entry records, by Compose service name (`recorded_service_records`
`recorded_service_records`	The per-service records of one `built-images.json` entry, by Compose service name: `image` plus `tool`
`write_json_record`	Replace the record file at `path` with `data` atomically: written sorted and indented to a temporary

Data

RECORD_READ_RETRY_S

VERIFIER_SERVICE

API

nemo_gym.data.tasks.harbor.records.read_built_images(
    folder: pathlib.Path
) -> dict[str, typing.Any]

The built images recorded in folder (the tasks’ parent), or {} when none were.

nemo_gym.data.tasks.harbor.records.read_image_configs(
    folder: pathlib.Path
) -> dict[str, typing.Any]

The image configurations recorded in folder (the tasks’ parent), or {} when none were.

nemo_gym.data.tasks.harbor.records.read_json_record(
    path: pathlib.Path
) -> dict[str, typing.Any]

The JSON object in the record file at path, or {} when the file is missing or not an object.

Writers replace the file atomically (write_json_record), so a torn read should not happen; as belt and braces a file that does not parse is read once more after a short pause (a record an older writer is still truncating and rewriting) and HarborTaskError names it if it stays invalid.

nemo_gym.data.tasks.harbor.records.record_lock(
    path: pathlib.Path
) -> typing.Iterator[None]

Hold an exclusive lock on path’s sidecar lock file (<file>.lock) for a read-modify-write of it.

Two Gym processes can share one datasets folder (gym eval run and gym dataset validate of one harbor: ref both record into it), so every read, merge and replace of a record file happens under this lock or a concurrent writer’s entries are lost. flock is per open file description, so threads of one process are serialised as well. The lock file stays next to the record; it carries no content.

nemo_gym.data.tasks.harbor.records.recorded_forced_tasks(
    entry: typing.Any
) -> set[str]

The tasks one built-images.json entry records as built under builds.force over their declared docker_image (forced_tasks); the loader runs the recorded main for them and the declared image for every other task that declares one. set() for an entry without the key.

nemo_gym.data.tasks.harbor.records.recorded_service_images(
    entry: typing.Any
) -> dict[str, str]

The images one built-images.json entry records, by Compose service name (recorded_service_records without the flags).

nemo_gym.data.tasks.harbor.records.recorded_service_records(
    entry: typing.Any
) -> dict[str, dict[str, typing.Any]]

The per-service records of one built-images.json entry, by Compose service name: image plus tool and pushed as recorded for that service.

A services map is read as written; a service record without the two flags (written before they were kept per service) takes the entry’s own, as does the single main record of an entry with image and no services, written before sidecars were built. Anything else records nothing.

nemo_gym.data.tasks.harbor.records.write_json_record(
    path: pathlib.Path,
    data: dict[str, typing.Any]
) -> None

Replace the record file at path with data atomically: written sorted and indented to a temporary file in the same directory, fsynced, then moved over path with os.replace, so a reader opens either the old file or the new one and never a truncated one. Callers merging into an existing record hold record_lock around the read, the merge and this call.

nemo_gym.data.tasks.harbor.records.BUILT_IMAGES_FILE = 'built-images.json'

nemo_gym.data.tasks.harbor.records.IMAGE_CONFIGS_FILE = 'compose-images.json'

nemo_gym.data.tasks.harbor.records.MAIN_SERVICE = 'main'

nemo_gym.data.tasks.harbor.records.RECORD_LOCK_SUFFIX = '.lock'

nemo_gym.data.tasks.harbor.records.RECORD_READ_RETRY_S = 0.05

nemo_gym.data.tasks.harbor.records.VERIFIER_SERVICE = 'verifier'