nemo_gym.tasks.harbor.records

The two record files prepare writes next to the task folders, and how they are read.

compose-images.json records image configurations, built-images.json records the images built per context digest and service. This module names the files and the service keys, takes the sidecar lock, replaces a file atomically, reads it with one retry against a torn write, and reads the per-service entries with the legacy main-only fallback. The loader, the recorders, the build modules and the environment server all read through it.

Module Contents

Functions

Name	Description
`read_built_images`	The built images recorded in `folder` (the tasks’ parent), or `{}` when none were.
`read_image_configs`	The image configurations recorded in `folder` (the tasks’ parent), or `{}` when none were.
`read_json_record`	The JSON object in the record file at `path`, or `{}` when the file is missing or not an object.
`record_lock`	Hold an exclusive lock on `path`’s sidecar lock file (`<file>.lock`) for a read-modify-write of it.
`write_json_record`	Replace the record file at `path` with `data` atomically: written sorted and indented to a temporary

Data

BUILT_IMAGES_FILE

IMAGE_CONFIGS_FILE

RECORD_LOCK_SUFFIX

RECORD_READ_RETRY_S

API

nemo_gym.tasks.harbor.records.read_built_images(
    folder: pathlib.Path
) -> dict[str, typing.Any]

The built images recorded in folder (the tasks’ parent), or {} when none were.

nemo_gym.tasks.harbor.records.read_image_configs(
    folder: pathlib.Path
) -> dict[str, typing.Any]

The image configurations recorded in folder (the tasks’ parent), or {} when none were.

nemo_gym.tasks.harbor.records.read_json_record(
    path: pathlib.Path
) -> dict[str, typing.Any]

The JSON object in the record file at path, or {} when the file is missing or not an object.

Writers replace the file atomically (write_json_record), so a torn read should not happen; as belt and braces a file that does not parse is read once more after a short pause (a record an older writer is still truncating and rewriting) and HarborTaskError names it if it stays invalid.

nemo_gym.tasks.harbor.records.record_lock(
    path: pathlib.Path
) -> typing.Iterator[None]

Hold an exclusive lock on path’s sidecar lock file (<file>.lock) for a read-modify-write of it.

Two Gym processes can share one datasets folder (gym eval run and gym dataset validate of one harbor: ref both record into it), so every read, merge and replace of a record file happens under this lock or a concurrent writer’s entries are lost. flock is per open file description, so threads of one process are serialised as well. The lock file stays next to the record; it carries no content.

nemo_gym.tasks.harbor.records.write_json_record(
    path: pathlib.Path,
    data: dict[str, typing.Any]
) -> None

Replace the record file at path with data atomically: written sorted and indented to a temporary file in the same directory, fsynced, then moved over path with os.replace, so a reader opens either the old file or the new one and never a truncated one. Callers merging into an existing record hold record_lock around the read, the merge and this call.

nemo_gym.tasks.harbor.records.BUILT_IMAGES_FILE = 'built-images.json'

nemo_gym.tasks.harbor.records.IMAGE_CONFIGS_FILE = 'compose-images.json'

nemo_gym.tasks.harbor.records.RECORD_LOCK_SUFFIX = '.lock'

nemo_gym.tasks.harbor.records.RECORD_READ_RETRY_S = 0.05