nemo_gym.tasks.harbor.build_records

Decide when a recorded image can be reused and record a finished build.

Two record files describe built images: compose-images.json holds each image’s configuration with the inputs digest of what was built, built-images.json maps a context digest to the images of its services. This module computes the inputs digest over the settled plan entry, compares a recorded image against the platform and repository of the current build, reports which services need rebuilding, and merges a build result into both files under the record lock. It knows nothing about the build tools.

Module Contents

Functions

Name	Description
`_canonical_json`	`value` (a parsed YAML section) as one line of JSON that is the same for the same section: keys sorted and
`build_inputs_digest`	The inputs digest of a settled entry (`settle_entry`): `inputs_digest`
`build_inputs_mismatch`	Why the image the `built-images.json` entry `record` names was built from other inputs than `entry`’s
`platform_key`	`os/arch` of a platform string, with a variant such as `linux/arm64/v8` dropped.
`pop_stale_build_inputs`	Remove from `recorded` (`built-images.json` as read) the entries of `plan`’s contexts whose build inputs
`record_built_image`	Record one built image next to the task folders: its context digest in `built-images.json`
`recorded_image_mismatch`	Why a recorded built image cannot serve this builder, or `None` when it can.
`settle_entry`	`entry` with `main`’s `build` section of the task’s Compose file applied, as `ImageBuilder.build`

API

nemo_gym.tasks.harbor.build_records._canonical_json(
    value: typing.Any
) -> str

value (a parsed YAML section) as one line of JSON that is the same for the same section: keys sorted and read as text (YAML allows other scalars as keys), and a scalar JSON has no form for, a date say, as its text.

nemo_gym.tasks.harbor.build_records.build_inputs_digest(
    entry: nemo_gym.tasks.harbor.build_plan.BuildPlanEntry
) -> str

The inputs digest of a settled entry (settle_entry): inputs_digest of its context (every entry of the tree with its mode, content or link target, nothing skipped) plus what the build reads outside the tree: main’s build section of the task’s Compose file as a whole, as it stands after interpolation (Compose interpolates every string of the file against the build’s environment, so an inline Dockerfile, a label or a cache setting changes the image while no file does), its two paths by their resolved forms relative to the context (the text may spell them through CONTEXT_DIR, an absolute path that differs between hosts holding the same folder) and the build arguments as resolved (a null or bare argument is read from the environment, which the section does not show). Recorded in the built image’s configuration record and compared before a recorded image is reused (build_inputs_mismatch).

nemo_gym.tasks.harbor.build_records.build_inputs_mismatch(
    entry: nemo_gym.tasks.harbor.build_plan.BuildPlanEntry,
    record: dict[str, typing.Any],
    image_configs: dict[str, typing.Any],
    repository: str | None,
    platform: str | None
) -> str | None

Why the image the built-images.json entry record names was built from other inputs than entry’s context holds now, or None when the inputs match and the recorded image may be reused.

The context’s Harbor hash names the image and keys the record, but it covers file names and bytes only; the inputs_digest in the image’s compose-images.json record (image_configs, written by the build with build_inputs_digest) covers the rest, so a context whose name is unchanged is still rebuilt under that name when its inputs are not. An image whose record has no digest (written before it was recorded, or missing) counts as changed. Settling the entry can fail as the build would (BuildError: a Compose file that does not interpolate or is refused); that is reported as the reason too, so the caller rebuilds, and the rebuild reports the error in full.

nemo_gym.tasks.harbor.build_records.platform_key(
    platform: str | None
) -> str

os/arch of a platform string, with a variant such as linux/arm64/v8 dropped.

nemo_gym.tasks.harbor.build_records.pop_stale_build_inputs(
    recorded: dict[str, typing.Any],
    plan: list[nemo_gym.tasks.harbor.build_plan.BuildPlanEntry],
    image_configs: dict[str, typing.Any],
    repository: str | None,
    platform: str | None
) -> list[tuple[nemo_gym.tasks.harbor.build_plan.BuildPlanEntry, dict[str, typing.Any], str]]

Remove from recorded (built-images.json as read) the entries of plan’s contexts whose build inputs no longer match (build_inputs_mismatch against image_configs), and return them as (entry, record, why) for the caller to report: a stale record is treated as none, so the context is rebuilt under the same name and its records replaced.

nemo_gym.tasks.harbor.build_records.record_built_image(
    folder: pathlib.Path,
    result: nemo_gym.tasks.harbor.build_plan.BuildResult,
    overwrite: bool = False
) -> pathlib.Path

Record one built image next to the task folders: its context digest in built-images.json and its configuration in compose-images.json. Existing entries are kept unless overwrite.

Both files are merged and replaced under their sidecar locks (record_lock), the built-images lock first and the compose-images lock inside it, always in that order; record_image_configs takes only the compose-images lock, so the two writers cannot deadlock and neither loses the other’s entries. The configuration lands before the build entry, as before, so a reader that finds the entry finds its configuration.

nemo_gym.tasks.harbor.build_records.recorded_image_mismatch(
    entry: dict[str, typing.Any],
    image_configs: dict[str, typing.Any],
    repository: str | None,
    platform: str
) -> str | None

Why a recorded built image cannot serve this builder, or None when it can.

A local (unpushed) image serves only local builds; a pushed one serves both. The recorded platform has to be the one asked for.

nemo_gym.tasks.harbor.build_records.settle_entry(
    entry: nemo_gym.tasks.harbor.build_plan.BuildPlanEntry,
    repository: str | None,
    platform: str | None
) -> nemo_gym.tasks.harbor.build_plan.BuildPlanEntry

entry with main’s build section of the task’s Compose file applied, as ImageBuilder.build applies it before the build command: the file is checked and interpolated against the build’s own environment (compose_build_env, for the image name and platform a builder with repository and platform uses), and its settings fill the entry (main_build_settings). Raises BuildError where the build would.