nemo_gym.data.tasks.harbor.build_tools
nemo_gym.data.tasks.harbor.build_tools
Drive docker, buildah or podman for one planned build.
ImageBuilder owns the per-tool argv for build, compose build, push and inspect, the generated containers.conf and registries.conf environment, the compose plugin and heredoc preflights, the pushed digest and the inspect normalisation that turns tool output into the recorded configuration. It is the only build module that runs subprocesses, through the Runner seam. ImageBuilder resolves detect_tool from this module’s globals, which is where tests patch it.
Module Contents
Classes
Functions
Data
API
Build the services of one context into named images with docker, buildah or podman, and push them when asked.
main’s image is named as Harbor names it, hb__<hash>, a sidecar’s hb__<hash>-<service> (see
tag_for). repository is the registry path the names go under (registry/path/hb__<hash>) and
the images are pushed there. Without a repository an image stays local under its name in the Docker daemon,
the only local store a sandbox provider starts images from, so buildah and podman need a repository. push
defaults to whether a repository was named. platform (linux/amd64 by default, the platform sandboxes
run) is passed to the tool, and an image that comes out for another platform is refused before it is pushed
or recorded; a platform the records could not tell from the baseline (an amd64 micro-architecture level above
it, see platform_pair) is refused before anything is built. run is the command seam.
docker builds the way Harbor’s Docker environment does: one docker compose build over Harbor’s build
Compose file, the task’s own and Gym’s images override (compose_build_argv), never docker build.
buildah and podman build each service’s Dockerfile directly (build_argv).
Fail before anything is built when docker compose (the Compose v2 plugin) is missing.
Fail before a Dockerfile with a heredoc is built when this host’s buildah or podman cannot parse one.
Ubuntu 24.04’s packaged buildah 1.33.7 and podman 4.9.3 are linked against openshift/imagebuilder 1.2.5,
which reads the body of RUN <<EOF as instructions and fails with Unknown instruction: "ECHO". The
preflight builds HEREDOC_PROBE_DOCKERFILE (FROM scratch plus a COPY heredoc; no base image
is pulled) once per builder with the build’s own flags and environment, and a failure is one
BuildError naming the tool, its version, the Dockerfile and the remedy (HEREDOC_REMEDY)
instead of one Unknown instruction per task. The outcome is cached either way.
Inspect the built tag, refuse another platform, push when configured, and return the result.
The build section keys only docker compose build can honour, refused or ignored by buildah and podman.
<tool> --version’s first line, or the tool’s name when it will not say.
build_many for one service, returning its result.
The direct Dockerfile build of buildah and podman (docker builds through compose_build_argv).
--format docker builds the image format BuildKit builds, so SHELL applies to the RUN steps and
HEALTHCHECK and ONBUILD reach the image (the tools’ default OCI format has none of the three).
The service’s build section of the task’s Compose file, which docker compose build would apply (over
Harbor’s template for main), is applied the same way: its context and Dockerfile, --build-arg per arg
and --target (service_build_settings).
Build the services entries of one context, push when configured, inspect, and yield each result to
record as soon as it is done, in order. Raises BuildError; what was yielded before stays built.
The task’s Compose file is read here, not at planning: it is checked once for the context
(validate_task_compose) and each entry’s service’s build section fills it
(service_build_settings), so a file that does not interpolate, escapes environment/ or carries a
refused key fails this context’s build and the caller leaves its tasks out, the way it leaves out a task
whose build command fails. The build’s environment is Harbor’s (compose_build_env) with
MAIN_IMAGE_NAME the main tag, whether or not main is among the entries, as Harbor always exports
it.
docker runs one docker compose build for the context, Harbor’s build step, and then inspects and pushes
each image; partial says entries are not every service the context builds (the others are
recorded), so the build names them rather than building the whole project. buildah and podman run one build
per service (build_argv); on that path dockerfile_inline and additional_contexts (even
service: references, which need --build-context and a build order) are refused naming the service,
before the context’s first build, since the checked file already says so, and no_cache and pull are
ignored, as the direct tools have no Compose to honour them.
Harbor’s docker compose build line for the context (_run_docker_compose_command,
https://github.com/harbor-framework/harbor/blob/v0.22.0/src/harbor/environments/docker/docker.py#L606).
The project name stands in for Harbor’s session id (Compose’s default image name is
<project>-<service>; main is named explicitly, see harbor_compose_document); the
project directory is the context, so relative paths in the task’s Compose file resolve as Harbor
resolves them; the files are Harbor’s build template, then the task’s docker-compose.yaml when
it has one, in Harbor’s order, then Gym’s images override (images_compose_document), last so its
names win. services names the services to build; Harbor passes none, and so does a full build, so every
service with a build section is built. A partial build (some services of the context recorded, or
main not planned) names the ones it builds.
The image’s OCI configuration in the compose-images.json record shape (image unset).
The image’s name under the repository when there is one: hb__<hash> for main,
hb__<hash>-<service> for a sidecar.
The hash is the entry’s digest, Harbor’s environment id of the context, and hb__<hash> is the
value Harbor exports as MAIN_IMAGE_NAME for it (the name a task Compose file that sets
image: ${MAIN_IMAGE_NAME} builds under; Harbor’s own template leaves the tag to Compose). A sidecar
takes Compose’s default form for a service without image:, <project>-<service>, under the project
name Gym passes to docker compose build (hb__<hash>; Harbor’s is the session id, so its sidecar tags
change per run and are never published). The separate verifier’s image takes the sidecar form too,
hb__<hash>-verifier with Harbor’s hash of tests/: the hash is the one Harbor exports as
MAIN_IMAGE_NAME for its verifier environment, and the suffix names the role where Harbor’s own tag is the
per-session <session>-main. One flat name per image, sanitised like Harbor’s, stays within the
one repository path builds.repository promises; a slash would open a path per hash.
tag_for by digest and service.
The environment buildah and podman commands run with, or None (inherit) for docker.
The host’s containers.conf may give every container a default bind mount (this deployment mounts
/etc/localtime read-only into each one through containers.conf.d), and buildah and podman apply
those mounts to build steps too, so a Dockerfile that installs tzdata (Ubuntu with python3 pulls
it in) fails with mv: cannot move '/etc/localtime.dpkg-new' to '/etc/localtime': Device or resource busy. The builder writes a containers.conf with no default volumes under its storage root, once, and
hands it to the tool as CONTAINERS_CONF_OVERRIDE, the layer containers-common applies over the host
files. A CONTAINERS_CONF_OVERRIDE the caller already set is left as it is.
Both tools resolve an unqualified FROM (FROM oven/bun:1) through the host’s registries.conf, and
Debian’s and Ubuntu’s stock file defines no unqualified-search registry, so the build fails with
short-name "oven/bun:1" did not resolve to an alias where docker compose build pulls from docker.io.
The builder writes a registries.conf naming docker.io as the one unqualified-search registry under its
storage root and hands it to the tool as CONTAINERS_REGISTRIES_CONF. That file replaces the host’s main
registries.conf (the drop-ins under registries.conf.d, short-name aliases included, still load), so a
site whose main file carries mirrors or blocked registries exports its own CONTAINERS_REGISTRIES_CONF,
which is left as it is. docker needs neither file and inherits the process environment.
The set fields of an inspect’s Docker.config section (the docker-format image configuration), or {}.
Why a registry step may have answered 403, when the tool’s output says it did: a proxy the build tool
inherits (from its environment, or from its own configuration when the environment names none) can answer
403 to the registry ping unless no_proxy carries the registry host. Names only, never values.
The manifest digest the registry assigned to the pushed image.
Turn a tool’s inspect JSON into the compose-images.json record shape (without image).
buildah reports {"OCIv1": {"architecture", "os", "config"}, "Docker": {"config"}, "FromImageID"}; podman and
docker report a list with Architecture, Os, Config and Id. The image is built in docker format
(IMAGE_FORMAT), whose Shell, Healthcheck and OnBuild the OCI image configuration cannot
hold: buildah reports them under Docker.config, read here over the OCI config; docker reports them in
Config; podman’s Config is the OCI image configuration (its inspect does not expose Shell and
OnBuild, although the image carries them) and Healthcheck sits beside Config as a top-level key,
read here when Config has none.
The proxy-related variables in env (http_proxy, HTTPS_PROXY, no_proxy, all_proxy and the
like), names only: their values can carry credentials.