nemo_gym.data.tasks.harbor.build_plan

Why plan_builds planned a task for nothing: one of its folders has the Harbor hash of first’s (the task planned first under it) but not its build inputs. reason spells the difference out for the task’s skip.

Bases: RuntimeError

A build, push or inspect command failed; nothing was recorded for the image.

One service to build of one distinct build context, and the tasks that share the context.

digest is Harbor’s environment id of the context (32 hex characters), the key of the build record and the body of the image name; context is the task’s environment/ folder, the hash root, Compose’s project directory and CONTEXT_DIR, whatever the service’s own build context is. service is the Compose service the entry builds: main (the task’s own image, hb__<digest>) or a sidecar of the task’s docker-compose.yaml with a build: section (hb__<digest>-<service>). The entries of one context share digest, context, tasks and the settings below, and differ in service and the build section. env, cpus and memory_mb are the first task’s effective [environment] settings, the variables Harbor’s docker compose build sees (compose_build_env); env is empty when the context ships no docker-compose.yaml, as Harbor only resolves the task’s env for a Compose task (_uses_compose). Tasks sharing a context but declaring different settings share this image anyway, as they do in Harbor (the name is the content hash); plan_builds logs them. Tasks whose folders share the hash but not what a build reads (a mode, a link, an empty folder, a name the hash skips) do not share the entry: the first task’s folder is the context and the others are left out, see plan_builds.

build_context, dockerfile, build_args, target and dockerfile_inline are the service’s build section of the task’s docker-compose.yaml (service_build_settings), the settings docker compose build merges over Harbor’s template: the direct build of buildah and podman applies them through ImageBuilder.build_argv. compose_build is that section whole, as interpolated, for the inputs digest of the image (build_inputs_digest), and dependency_digests the inputs digests of the services this one builds from (the service: references of its additional_contexts), by service, part of that digest too. plan_builds leaves them at their defaults (the context and its Dockerfile); ImageBuilder.build_many fills them once the file has been checked (build_records.settle_with_document), so a file that does not interpolate or is refused fails that context’s build, not the planning of the whole dataset.

verifier marks the separate verifier’s image (service verifier, see the module docstring): context and digest are then the task’s tests/ folder and Harbor’s hash of it, env, cpus and memory_mb the effective verifier environment’s, and the entry builds Harbor’s template service main of that context (compose_service), so a tests/docker-compose.yaml is read for main’s build section, as Harbor’s verifier environment reads it. A Compose sidecar that happens to be named verifier (kumo’s) is an ordinary service entry of the environment/ context with this flag unset.

A built image, ready to record: the image of service for the context digest.

Why task_id’s context is left out of the entry first’s first_context holds under digest: the differences inputs_differences finds, the first three spelled out.

The BuildConflict of task_id’s folder context with the folder plan_builds already holds under its Harbor hash digest, or None when no folder is held yet or the two read the same to a build. Harbor’s hash covers names and bytes; the build reads more. The held folder’s inputs digest is computed once.

Add task_id’s services planned of context to plan_builds’s tables, under the context’s Harbor hash digest, with environment’s settings ([environment]’s for the task’s own context, the effective verifier environment’s for tests/) when the context is new, logging a task that declares others for a known one. The folder has passed _context_conflict.

Where buildah keeps its layers: node-local when the cluster scratch disk exists, else the temp dir.

The builder to use, or None when none is installed.

With a repository the image is pushed, so any tool serves: buildah, then podman, then docker. Without one the image has to stay where a sandbox provider can start it, and only the Docker daemon is such a place (buildah and podman keep images in a store no provider reads), so only docker qualifies.

Whether a Dockerfile is outside what pull and overlay mode can run.

A file whose instructions can only be resolved against its base image’s configuration is resolved against the record in image_configs (compose-images.json, as prepare records it before planning builds). Without a record the file is classed as a build, so a context is never left unbuilt because the base image was unknown.

The services to build among the tasks under folder, one entry per distinct build context and service.

folder is a task folder or a folder of task folders; select limits the scan to the tasks whose folder name it accepts. conflicts, when given, receives by task id the BuildConflict of each task planned for nothing: one of its folders (environment/ or tests/) has the Harbor hash of a task planned before it (the same file names and contents, so the same hb__<hash> image names and build record) but not the same build inputs (inputs_digest: a mode, a symbolic link, an empty folder or a name the hash skips differs). One image cannot be built from both folders under the one name, so the entry keeps the first task’s folder and tasks, and the task is in no entry at all, its other folder’s included, since all of its folders are checked before any is added: prepare leaves it out with the reason, as it leaves out a task whose build fails, or plans again without the first task when that one is left out itself. A task’s main counts when it has environment/Dockerfile, declares no [environment].docker_image after the dataset’s overrides, and the Dockerfile is outside pull and overlay mode; a prebuilt main is never built, unless force is set: Harbor’s --force-build builds the Dockerfile instead of pulling the declared image, whatever the file holds (one pull or overlay mode could run too), and the loader then runs the recorded build. Every other service of the task’s environment/docker-compose.yaml (the one Compose file Harbor reads) with a build section counts whatever main’s mode, as Harbor’s docker compose build builds it (sidecar_build_services). Tasks whose task.toml does not load are left for the loader to report. Entries are grouped by context in the order of the first task that uses it, main first and the sidecars in file order. The Compose file is not interpolated here, only scanned for build keys: resolving each section is the build’s job (ImageBuilder.build_many), so a file one task gets wrong fails that task’s build and nothing else.

A task in separate verifier mode whose effective verifier environment declares no docker_image and that ships tests/Dockerfile (verifier_build_context) plans one more entry, service verifier with tests/ as its context and digest (its own group, after the task’s environment/ entries), whatever the file holds, as Harbor builds it whatever it holds. A shared verifier, a declared or inherited docker_image or a task without the file plans none (the loader refuses the last shape when no image is declared either, as Harbor does). A tests/docker-compose.yaml is read for main’s build section only; one that declares other services fails the verifier’s build before any command (ImageBuilder.build_many).

The services other than main of context’s docker-compose.yaml that carry a build section, in file order; [] without the file, or for one that does not parse or has no services mapping (the loader reports such a file). Only the presence of the key is read here, nothing is interpolated.