> This page is for version Main (default). > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.data.tasks.harbor.build_records Decide when a recorded image can be reused and record a finished build. Two record files describe built images: compose-images.json holds each image's configuration with the inputs digest of what was built, built-images.json maps a context digest to the images of its services. This module computes the inputs digest over the settled plan entry, compares a recorded image against the platform and repository of the current build, reports which services need rebuilding, and merges a build result into both files under the record lock. It knows nothing about the build tools. ## Module Contents ### Functions | Name | Description | | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | [`_canonical_json`](#nemo_gym-data-tasks-harbor-build_records-_canonical_json) | `value` (a parsed YAML section) as one line of JSON that is the same for the same section: keys sorted and | | [`_dependency_digests`](#nemo_gym-data-tasks-harbor-build_records-_dependency_digests) | `(service, inputs digest)`, sorted by service, for each service whose image `entry`'s build reads: the | | [`build_inputs_digest`](#nemo_gym-data-tasks-harbor-build_records-build_inputs_digest) | The inputs digest of a settled entry (`settle_entry`): `inputs_digest` | | [`build_inputs_mismatch`](#nemo_gym-data-tasks-harbor-build_records-build_inputs_mismatch) | Why the image the `built-images.json` entry `record` holds for `entry`'s service was built from other | | [`platform_key`](#nemo_gym-data-tasks-harbor-build_records-platform_key) | `os/arch` of a platform string, with a variant such as `linux/arm64/v8` dropped. | | [`record_built_image`](#nemo_gym-data-tasks-harbor-build_records-record_built_image) | Record one built image next to the task folders: under its context digest and service in | | [`record_forced_tasks`](#nemo_gym-data-tasks-harbor-build_records-record_forced_tasks) | Add `tasks` to the `forced_tasks` of the context `digest`'s entry in `built-images.json`: the tasks | | [`recorded_image_mismatch`](#nemo_gym-data-tasks-harbor-build_records-recorded_image_mismatch) | Why the image a `built-images.json` entry records for `service` cannot serve this builder, or `None` | | [`settle_entry`](#nemo_gym-data-tasks-harbor-build_records-settle_entry) | `entry` with its service's `build` section of the task's Compose file applied, as | | [`settle_with_document`](#nemo_gym-data-tasks-harbor-build_records-settle_with_document) | `entry` with its service's `build` section of the checked and interpolated Compose `document` applied | ### API ```python nemo_gym.data.tasks.harbor.build_records._canonical_json( value: typing.Any ) -> str ``` `value` (a parsed YAML section) as one line of JSON that is the same for the same section: keys sorted and read as text (YAML allows other scalars as keys), and a scalar JSON has no form for, a date say, as its text. ```python nemo_gym.data.tasks.harbor.build_records._dependency_digests( entry: nemo_gym.data.tasks.harbor.build_plan.BuildPlanEntry, document: dict[str, typing.Any] | None, env: dict[str, str], seen: tuple[str, ...] ) -> tuple[tuple[str, str], ...] ``` `(service, inputs digest)`, sorted by service, for each service whose image `entry`'s build reads: the `service:` references of its `build.additional_contexts` (a mapping, or Compose's `name=service:x` list) that name a service with a `build` section of `document`. Each is settled as the entry is and digested with its own dependencies (`build_inputs_digest`), so the digest is transitive: a change to `db`'s inputs changes `main`'s when `main` builds `FROM base` with `base: service:db`, and `docker compose build` would build a different `main`. A service already on the path (`seen`: Compose refuses the cycle at build) adds nothing. Only `docker compose build` honours such references (the direct tools refuse them, `ImageBuilder._refuse_on_direct_path`), and only it follows them to a rebuild of the consumer. ```python nemo_gym.data.tasks.harbor.build_records.build_inputs_digest( entry: nemo_gym.data.tasks.harbor.build_plan.BuildPlanEntry ) -> str ``` The inputs digest of a settled entry (`settle_entry`): `inputs_digest` of its context (every entry of the tree with its mode, content or link target, nothing skipped) plus what the build reads outside the tree: the service's `build` section of the task's Compose file as a whole, as it stands after interpolation (Compose interpolates every string of the file against the build's environment, so an inline Dockerfile, a label or a cache setting changes the image while no file does), its two paths by their resolved forms relative to the context (the text may spell them through `CONTEXT_DIR`, an absolute path that differs between hosts holding the same folder), the build arguments as resolved (a `null` or bare argument is read from the environment, which the section does not show) and the inputs digest of each service whose image the build reads through a `service:` reference of its `additional_contexts` (`BuildPlanEntry.dependency_digests`: the section names the service, not what that service is built from). The services of one context share the tree and differ in the section, so each image carries its own. Recorded in the built image's configuration record and compared before a recorded image is reused (`build_inputs_mismatch`). ```python nemo_gym.data.tasks.harbor.build_records.build_inputs_mismatch( entry: nemo_gym.data.tasks.harbor.build_plan.BuildPlanEntry, record: dict[str, typing.Any], image_configs: dict[str, typing.Any], repository: str | None, platform: str | None ) -> str | None ``` Why the image the `built-images.json` entry `record` holds for `entry`'s service was built from other inputs than the context holds now, or `None` when the inputs match and the recorded image may be reused. The context's Harbor hash names the image and keys the record, but it covers file names and bytes only; the `inputs_digest` in the image's `compose-images.json` record (`image_configs`, written by the build with `build_inputs_digest`) covers the rest, so a context whose name is unchanged is still rebuilt under that name when its inputs are not. An image whose record has no digest (written before it was recorded, or missing) counts as changed. Settling the entry can fail as the build would (`BuildError`: a Compose file that does not interpolate or is refused); that is reported as the reason too, so the caller rebuilds, and the rebuild reports the error in full. ```python nemo_gym.data.tasks.harbor.build_records.platform_key( platform: str | None ) -> str ``` `os/arch` of a platform string, with a variant such as `linux/arm64/v8` dropped. ```python nemo_gym.data.tasks.harbor.build_records.record_built_image( folder: pathlib.Path, result: nemo_gym.data.tasks.harbor.build_plan.BuildResult, overwrite: bool = False ) -> pathlib.Path ``` Record one built image next to the task folders: under its context digest and service in `built-images.json` and by image in `compose-images.json`. Existing entries are kept unless `overwrite`. The digest's entry is merged, not replaced: `services[]` is set to this build's `image`, `tool` and `pushed` (the other services keep theirs: one context's services may be built by differently configured runs, and a pushed `main` must not vouch for a local sidecar or the reverse), `tasks` unioned, and the top-level keys older readers use kept equal to `main`'s: `image` (absent while `main` is not recorded) with `tool` and `pushed` (the first recorded service's while `main` is not recorded). A service recorded once keeps its image unless `overwrite`, like `compose-images.json` keeps its entries. Both files are merged and replaced under their sidecar locks (`record_lock`), the built-images lock first and the compose-images lock inside it, always in that order; `record_image_configs` takes only the compose-images lock, so the two writers cannot deadlock and neither loses the other's entries. The configuration lands before the build entry, as before, so a reader that finds the entry finds its configuration. ```python nemo_gym.data.tasks.harbor.build_records.record_forced_tasks( folder: pathlib.Path, digest: str, tasks: typing.Iterable[str] ) -> None ``` Add `tasks` to the `forced_tasks` of the context `digest`'s entry in `built-images.json`: the tasks whose declared `docker_image` a prepare under `builds.force` built over, when the context's `main` was already recorded and is reused (a fresh build records them through `record_built_image`). No entry, no write. ```python nemo_gym.data.tasks.harbor.build_records.recorded_image_mismatch( entry: dict[str, typing.Any], image_configs: dict[str, typing.Any], repository: str | None, platform: str, service: str = MAIN_SERVICE ) -> str | None ``` Why the image a `built-images.json` entry records for `service` cannot serve this builder, or `None` when it can. A service the entry does not record is missing. A local (unpushed) image serves only local builds; a pushed one serves both, judged by that service's own `pushed` flag (`recorded_service_records`: the entry's flag stands in for a record written before the flags were kept per service), since the services of one context may have been built by differently configured runs. The recorded platform has to be the one asked for. ```python nemo_gym.data.tasks.harbor.build_records.settle_entry( entry: nemo_gym.data.tasks.harbor.build_plan.BuildPlanEntry, repository: str | None, platform: str | None ) -> nemo_gym.data.tasks.harbor.build_plan.BuildPlanEntry ``` `entry` with its service's `build` section of the task's Compose file applied, as `ImageBuilder.build_many` applies it before the build command: the file is checked and interpolated against the build's own environment (`compose_build_env`, for the image name and platform a builder with `repository` and `platform` uses), and its settings fill the entry (`settle_with_document`). Raises `BuildError` where the build would. ```python nemo_gym.data.tasks.harbor.build_records.settle_with_document( entry: nemo_gym.data.tasks.harbor.build_plan.BuildPlanEntry, document: dict[str, typing.Any] | None, env: dict[str, str] ) -> nemo_gym.data.tasks.harbor.build_plan.BuildPlanEntry ``` `entry` with its service's `build` section of the checked and interpolated Compose `document` applied (`service_build_settings`) and the inputs digests of the services it builds from filled in (`BuildPlanEntry.dependency_digests`, see `_dependency_digests`). What `ImageBuilder.build_many` settles the entries it builds with, and `settle_entry` the one it checks, so the digest recorded and the digest compared are computed alike.