> This page is for version 0.5.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.data.tasks.harbor.build_plan Plan the image builds of a task folder. Reads task folders and dataset.toml, decides which Dockerfiles need a build rather than a pull or an overlay, finds the sidecars with a build section and the separate verifier's tests context, and groups the work by context digest into BuildPlanEntry values with BuildResult as the record of one finished build. Tool detection and the storage root live here because the plan names the tool. The module runs no subprocess and reads no Compose variables. It is the root of the build modules: build_compose, build_records, build_tools and image_build import it and it imports none of them. ## Module Contents ### Classes | Name | Description | | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | [`BuildConflict`](#nemo_gym-data-tasks-harbor-build_plan-BuildConflict) | Why `plan_builds` planned a task for nothing: one of its folders has the Harbor hash of `first`'s (the | | [`BuildError`](#nemo_gym-data-tasks-harbor-build_plan-BuildError) | A build, push or inspect command failed; nothing was recorded for the image. | | [`BuildPlanEntry`](#nemo_gym-data-tasks-harbor-build_plan-BuildPlanEntry) | One service to build of one distinct build context, and the tasks that share the context. | | [`BuildResult`](#nemo_gym-data-tasks-harbor-build_plan-BuildResult) | A built image, ready to record: the image of `service` for the context `digest`. | ### Functions | Name | Description | | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | [`_conflict_reason`](#nemo_gym-data-tasks-harbor-build_plan-_conflict_reason) | Why `task_id`'s `context` is left out of the entry `first`'s `first_context` holds under `digest`: | | [`_context_conflict`](#nemo_gym-data-tasks-harbor-build_plan-_context_conflict) | The `BuildConflict` of `task_id`'s folder `context` with the folder `plan_builds` already holds | | [`_plan_context`](#nemo_gym-data-tasks-harbor-build_plan-_plan_context) | Add `task_id`'s services `planned` of `context` to `plan_builds`'s tables, under the context's Harbor | | [`default_storage_root`](#nemo_gym-data-tasks-harbor-build_plan-default_storage_root) | Where buildah keeps its layers: node-local when the cluster scratch disk exists, else the temp dir. | | [`detect_tool`](#nemo_gym-data-tasks-harbor-build_plan-detect_tool) | The builder to use, or `None` when none is installed. | | [`needs_build`](#nemo_gym-data-tasks-harbor-build_plan-needs_build) | Whether a Dockerfile is outside what pull and overlay mode can run. | | [`plan_builds`](#nemo_gym-data-tasks-harbor-build_plan-plan_builds) | The services to build among the tasks under `folder`, one entry per distinct build context and service. | | [`sidecar_build_services`](#nemo_gym-data-tasks-harbor-build_plan-sidecar_build_services) | The services other than `main` of `context`'s `docker-compose.yaml` that carry a `build` section, in | ### Data [`BUILD_TOOLS`](#nemo_gym-data-tasks-harbor-build_plan-BUILD_TOOLS) [`DEFAULT_PLATFORM`](#nemo_gym-data-tasks-harbor-build_plan-DEFAULT_PLATFORM) [`DOCKERFILE_NAME`](#nemo_gym-data-tasks-harbor-build_plan-DOCKERFILE_NAME) [`HARBOR_COMPOSE_FILE`](#nemo_gym-data-tasks-harbor-build_plan-HARBOR_COMPOSE_FILE) [`LOGGER`](#nemo_gym-data-tasks-harbor-build_plan-LOGGER) [`REPOSITORY_ENV`](#nemo_gym-data-tasks-harbor-build_plan-REPOSITORY_ENV) ### API ```python class nemo_gym.data.tasks.harbor.build_plan.BuildConflict( first: str, reason: str ) ``` Dataclass Why `plan_builds` planned a task for nothing: one of its folders has the Harbor hash of `first`'s (the task planned first under it) but not its build inputs. `reason` spells the difference out for the task's skip. **`first`** `str` --- **`reason`** `str` --- ```python class nemo_gym.data.tasks.harbor.build_plan.BuildError() ``` **Bases:** `RuntimeError` A build, push or inspect command failed; nothing was recorded for the image. ```python class nemo_gym.data.tasks.harbor.build_plan.BuildPlanEntry( digest: str, context: pathlib.Path, tasks: tuple[str, ...], env: dict[str, str] = dict(), cpus: int | None = None, memory_mb: int | None = None, service: str = MAIN_SERVICE, build_context: pathlib.Path | None = None, dockerfile: str = DOCKERFILE_NAME, build_args: dict[str, str] = dict(), compose_build: dict[str, typing.Any] | None = None, target: str | None = None, dockerfile_inline: bool = False, dependency_digests: tuple[tuple[str, str], ...] = (), verifier: bool = False, forced_tasks: tuple[str, ...] = () ) ``` Dataclass One service to build of one distinct build context, and the tasks that share the context. `digest` is Harbor's environment id of the context (32 hex characters), the key of the build record and the body of the image name; `context` is the task's `environment/` folder, the hash root, Compose's project directory and `CONTEXT_DIR`, whatever the service's own build context is. `service` is the Compose service the entry builds: `main` (the task's own image, `hb__`) or a sidecar of the task's `docker-compose.yaml` with a `build:` section (`hb__-`). The entries of one context share `digest`, `context`, `tasks` and the settings below, and differ in `service` and the build section. `env`, `cpus` and `memory_mb` are the first task's effective `[environment]` settings, the variables Harbor's `docker compose build` sees (`compose_build_env`); `env` is empty when the context ships no `docker-compose.yaml`, as Harbor only resolves the task's env for a Compose task (`_uses_compose`). Tasks sharing a context but declaring different settings share this image anyway, as they do in Harbor (the name is the content hash); `plan_builds` logs them. Tasks whose folders share the hash but not what a build reads (a mode, a link, an empty folder, a name the hash skips) do not share the entry: the first task's folder is the context and the others are left out, see `plan_builds`. `build_context`, `dockerfile`, `build_args`, `target` and `dockerfile_inline` are the service's `build` section of the task's `docker-compose.yaml` (`service_build_settings`), the settings `docker compose build` merges over Harbor's template: the direct build of buildah and podman applies them through `ImageBuilder.build_argv`. `compose_build` is that section whole, as interpolated, for the inputs digest of the image (`build_inputs_digest`), and `dependency_digests` the inputs digests of the services this one builds from (the `service:` references of its `additional_contexts`), by service, part of that digest too. `plan_builds` leaves them at their defaults (the context and its `Dockerfile`); `ImageBuilder.build_many` fills them once the file has been checked (`build_records.settle_with_document`), so a file that does not interpolate or is refused fails that context's build, not the planning of the whole dataset. `verifier` marks the separate verifier's image (service `verifier`, see the module docstring): `context` and `digest` are then the task's `tests/` folder and Harbor's hash of it, `env`, `cpus` and `memory_mb` the effective verifier environment's, and the entry builds Harbor's template service `main` of that context (`compose_service`), so a `tests/docker-compose.yaml` is read for `main`'s build section, as Harbor's verifier environment reads it. A Compose sidecar that happens to be named `verifier` (kumo's) is an ordinary service entry of the `environment/` context with this flag unset. **`build_args`** `dict[str, str] = field(default_factory=dict)` --- **`build_context`** `Path | None = None` --- **`build_dir`** `Path` The folder the tool builds: the service's `build.context` resolved, else the context itself. --- **`compose_build`** `dict[str, Any] | None = None` --- **`compose_service`** `str` The service of the context's Compose project this entry builds: `main` for the separate verifier's image (Harbor's template service of the `tests/` context), else `service` itself. --- **`context`** `Path` --- **`cpus`** `int | None = None` --- **`dependency_digests`** `tuple[tuple[str, str], ...] = ()` --- **`digest`** `str` --- **`dockerfile`** `str = DOCKERFILE_NAME` --- **`dockerfile_inline`** `bool = False` --- **`dockerfile_path`** `Path` The Dockerfile the tool builds, the service's `build.dockerfile` relative to `build_dir`. --- **`env`** `dict[str, str] = field(default_factory=dict)` --- **`forced_tasks`** `tuple[str, ...] = ()` --- **`is_main`** `bool` --- **`memory_mb`** `int | None = None` --- **`service`** `str = MAIN_SERVICE` --- **`target`** `str | None = None` --- **`tasks`** `tuple[str, ...]` --- **`verifier`** `bool = False` --- ```python class nemo_gym.data.tasks.harbor.build_plan.BuildResult( digest: str, tasks: tuple[str, ...], tag: str, image: str, tool: str, pushed: bool, record: dict[str, typing.Any] = dict(), service: str = MAIN_SERVICE, forced_tasks: tuple[str, ...] = () ) ``` Dataclass A built image, ready to record: the image of `service` for the context `digest`. **`digest`** `str` --- **`forced_tasks`** `tuple[str, ...] = ()` --- **`image`** `str` --- **`pushed`** `bool` --- **`record`** `dict[str, Any] = field(default_factory=dict)` --- **`service`** `str = MAIN_SERVICE` --- **`tag`** `str` --- **`tasks`** `tuple[str, ...]` --- **`tool`** `str` --- ```python nemo_gym.data.tasks.harbor.build_plan._conflict_reason( task_id: str, context: pathlib.Path, first: str, first_context: pathlib.Path, digest: str ) -> str ``` Why `task_id`'s `context` is left out of the entry `first`'s `first_context` holds under `digest`: the differences `inputs_differences` finds, the first three spelled out. ```python nemo_gym.data.tasks.harbor.build_plan._context_conflict( task_id: str, context: pathlib.Path, digest: str, entries: dict[str, list[str]], contexts: dict[str, pathlib.Path], trees: dict[str, str] ) -> nemo_gym.data.tasks.harbor.build_plan.BuildConflict | None ``` The `BuildConflict` of `task_id`'s folder `context` with the folder `plan_builds` already holds under its Harbor hash `digest`, or `None` when no folder is held yet or the two read the same to a build. Harbor's hash covers names and bytes; the build reads more. The held folder's inputs digest is computed once. ```python nemo_gym.data.tasks.harbor.build_plan._plan_context( task_id: str, context: pathlib.Path, digest: str, planned: list[str], environment: nemo_gym.data.tasks.harbor.models.HarborEnvironment, entries: dict[str, list[str]], contexts: dict[str, pathlib.Path], services: dict[str, list[str]], settings: dict[str, tuple[dict[str, str], int | None, int | None]] ) -> None ``` Add `task_id`'s services `planned` of `context` to `plan_builds`'s tables, under the context's Harbor hash `digest`, with `environment`'s settings (`[environment]`'s for the task's own context, the effective verifier environment's for `tests/`) when the context is new, logging a task that declares others for a known one. The folder has passed `_context_conflict`. ```python nemo_gym.data.tasks.harbor.build_plan.default_storage_root() -> pathlib.Path ``` Where buildah keeps its layers: node-local when the cluster scratch disk exists, else the temp dir. ```python nemo_gym.data.tasks.harbor.build_plan.detect_tool( repository: str | None = None ) -> str | None ``` The builder to use, or `None` when none is installed. With a repository the image is pushed, so any tool serves: buildah, then podman, then docker. Without one the image has to stay where a sandbox provider can start it, and only the Docker daemon is such a place (buildah and podman keep images in a store no provider reads), so only docker qualifies. ```python nemo_gym.data.tasks.harbor.build_plan.needs_build( dockerfile_text: str, image_configs: dict[str, typing.Any] | None = None ) -> bool ``` Whether a Dockerfile is outside what pull and overlay mode can run. A file whose instructions can only be resolved against its base image's configuration is resolved against the record in `image_configs` (`compose-images.json`, as prepare records it before planning builds). Without a record the file is classed as a build, so a context is never left unbuilt because the base image was unknown. ```python nemo_gym.data.tasks.harbor.build_plan.plan_builds( folder: pathlib.Path, select: typing.Callable[[str], bool] | None = None, conflicts: dict[str, nemo_gym.data.tasks.harbor.build_plan.BuildConflict] | None = None, force: bool = False ) -> list[nemo_gym.data.tasks.harbor.build_plan.BuildPlanEntry] ``` The services to build among the tasks under `folder`, one entry per distinct build context and service. `folder` is a task folder or a folder of task folders; `select` limits the scan to the tasks whose folder name it accepts. `conflicts`, when given, receives by task id the `BuildConflict` of each task planned for nothing: one of its folders (`environment/` or `tests/`) has the Harbor hash of a task planned before it (the same file names and contents, so the same `hb__` image names and build record) but not the same build inputs (`inputs_digest`: a mode, a symbolic link, an empty folder or a name the hash skips differs). One image cannot be built from both folders under the one name, so the entry keeps the first task's folder and tasks, and the task is in no entry at all, its other folder's included, since all of its folders are checked before any is added: prepare leaves it out with the reason, as it leaves out a task whose build fails, or plans again without the first task when that one is left out itself. A task's `main` counts when it has `environment/Dockerfile`, declares no `[environment].docker_image` after the dataset's overrides, and the Dockerfile is outside pull and overlay mode; a prebuilt `main` is never built, unless `force` is set: Harbor's `--force-build` builds the Dockerfile instead of pulling the declared image, whatever the file holds (one pull or overlay mode could run too), and the loader then runs the recorded build. Every other service of the task's `environment/docker-compose.yaml` (the one Compose file Harbor reads) with a `build` section counts whatever `main`'s mode, as Harbor's `docker compose build` builds it (`sidecar_build_services`). Tasks whose `task.toml` does not load are left for the loader to report. Entries are grouped by context in the order of the first task that uses it, `main` first and the sidecars in file order. The Compose file is not interpolated here, only scanned for `build` keys: resolving each section is the build's job (`ImageBuilder.build_many`), so a file one task gets wrong fails that task's build and nothing else. A task in separate verifier mode whose effective verifier environment declares no `docker_image` and that ships `tests/Dockerfile` (`verifier_build_context`) plans one more entry, service `verifier` with `tests/` as its context and digest (its own group, after the task's `environment/` entries), whatever the file holds, as Harbor builds it whatever it holds. A shared verifier, a declared or inherited `docker_image` or a task without the file plans none (the loader refuses the last shape when no image is declared either, as Harbor does). A `tests/docker-compose.yaml` is read for `main`'s build section only; one that declares other services fails the verifier's build before any command (`ImageBuilder.build_many`). ```python nemo_gym.data.tasks.harbor.build_plan.sidecar_build_services( context: pathlib.Path ) -> list[str] ``` The services other than `main` of `context`'s `docker-compose.yaml` that carry a `build` section, in file order; `[]` without the file, or for one that does not parse or has no services mapping (the loader reports such a file). Only the presence of the key is read here, nothing is interpolated. ```python nemo_gym.data.tasks.harbor.build_plan.BUILD_TOOLS = ('buildah', 'podman', 'docker') ``` ```python nemo_gym.data.tasks.harbor.build_plan.DEFAULT_PLATFORM = '/'.join(PLATFORM) ``` ```python nemo_gym.data.tasks.harbor.build_plan.DOCKERFILE_NAME = 'Dockerfile' ``` ```python nemo_gym.data.tasks.harbor.build_plan.HARBOR_COMPOSE_FILE = 'docker-compose.yaml' ``` ```python nemo_gym.data.tasks.harbor.build_plan.LOGGER = logging.getLogger(__name__) ``` ```python nemo_gym.data.tasks.harbor.build_plan.REPOSITORY_ENV = 'NEMO_GYM_BUILD_REPOSITORY' ```