nemo_gym.tasks.harbor.build_tools

Drive docker, buildah or podman for one planned build.

ImageBuilder owns the per-tool argv for build, compose build, push and inspect, the generated containers.conf and registries.conf environment, the compose plugin and heredoc preflights, the pushed digest and the inspect normalisation that turns tool output into the recorded configuration. It is the only build module that runs subprocesses, through the Runner seam. ImageBuilder resolves detect_tool from this module’s globals, which is where tests patch it.

Module Contents

Classes

Name	Description
`ImageBuilder`	Build one context into a named image with docker, buildah or podman, and push it when asked.

Functions

Name	Description
`_docker_config`	The set fields of an inspect’s `Docker.config` section (the docker-format image configuration), or `{}`.
`_proxy_note`	Why a registry step may have answered 403, when the tool’s output says it did: a proxy the build tool
`_pushed_digest`	The manifest digest the registry assigned to the pushed image.
`_run_default`	-
`normalize_inspect`	Turn a tool’s inspect JSON into the `compose-images.json` record shape (without `image`).
`proxy_variable_names`	The proxy-related variables in `env` (`http_proxy`, `HTTPS_PROXY`, `no_proxy`, `all_proxy` and the

Data

BUILT_CONFIG_KEYS

COMPOSE_PLUGIN

CONTAINERS_CONF_FILE

CONTAINERS_CONF_OVERRIDE

CONTAINERS_CONF_TEXT

CONTAINERS_REGISTRIES_CONF

HEREDOC_PROBE_DOCKERFILE

REGISTRIES_CONF_FILE

REGISTRIES_CONF_TEXT

API

class nemo_gym.tasks.harbor.build_tools.ImageBuilder(
    tool: str | None = None,
    repository: str | None = None,
    push: bool | None = None,
    platform: str | None = None,
    storage_root: pathlib.Path | None = None,
    run: nemo_gym.tasks.harbor.build_plan.Runner | None = None
)

Build one context into a named image with docker, buildah or podman, and push it when asked.

The image is named as Harbor names it, hb__<hash> (see tag_for). repository is the registry path the name goes under (registry/path/hb__<hash>) and the image is pushed there. Without a repository the image stays local under that name in the Docker daemon, the only local store a sandbox provider starts images from, so buildah and podman need a repository. push defaults to whether a repository was named. platform (linux/amd64 by default, the platform sandboxes run) is passed to the tool, and an image that comes out for another platform is refused before it is pushed or recorded. run is the command seam.

docker builds the way Harbor’s Docker environment does: docker compose build over Harbor’s build Compose file and the task’s own (compose_build_argv), never docker build. buildah and podman build the Dockerfile directly (build_argv).

_heredoc_failure

str | None = None

_run

= run or _run_default

_tool_env

dict[str, str] | None = None

platform

= platform or DEFAULT_PLATFORM

push

= bool(self.repository) if push is None else push

repository

= repository.rstrip('/') if repository else None

storage_root

tool

= tool or detect_tool(self.repository)

nemo_gym.tasks.harbor.build_tools.ImageBuilder._check(
    argv: list[str],
    what: str,
    env: dict[str, str] | None = None
) -> subprocess.CompletedProcess

nemo_gym.tasks.harbor.build_tools.ImageBuilder._ensure_compose_plugin() -> None

Fail before anything is built when docker compose (the Compose v2 plugin) is missing.

nemo_gym.tasks.harbor.build_tools.ImageBuilder._ensure_heredoc_support(
    dockerfile: pathlib.Path
) -> None

Fail before a Dockerfile with a heredoc is built when this host’s buildah or podman cannot parse one.

Ubuntu 24.04’s packaged buildah 1.33.7 and podman 4.9.3 are linked against openshift/imagebuilder 1.2.5, which reads the body of RUN <<EOF as instructions and fails with Unknown instruction: "ECHO". The preflight builds HEREDOC_PROBE_DOCKERFILE (FROM scratch plus a COPY heredoc; no base image is pulled) once per builder with the build’s own flags and environment, and a failure is one BuildError naming the tool, its version, the Dockerfile and the remedy (HEREDOC_REMEDY) instead of one Unknown instruction per task. The outcome is cached either way.

nemo_gym.tasks.harbor.build_tools.ImageBuilder._storage_flags() -> list[str]

nemo_gym.tasks.harbor.build_tools.ImageBuilder._tool_version() -> str

<tool> --version’s first line, or the tool’s name when it will not say.

nemo_gym.tasks.harbor.build_tools.ImageBuilder.build(
    entry: nemo_gym.tasks.harbor.build_plan.BuildPlanEntry
) -> nemo_gym.tasks.harbor.build_plan.BuildResult

Build, push when configured, inspect, and return the result to record. Raises BuildError.

The task’s Compose file is read here, not at planning: its main.build settings fill the entry (main_build_settings) once the file is checked, so a file that does not interpolate, escapes environment/ or carries a refused key fails this context’s build and the caller leaves its tasks out, the way it leaves out a task whose build command fails.

nemo_gym.tasks.harbor.build_tools.ImageBuilder.build_argv(
    entry: nemo_gym.tasks.harbor.build_plan.BuildPlanEntry,
    tag: str
) -> list[str]

The direct Dockerfile build of buildah and podman (docker builds through compose_build_argv).

--format docker builds the image format BuildKit builds, so SHELL applies to the RUN steps and HEALTHCHECK and ONBUILD reach the image (the tools’ default OCI format has none of the three). main’s build section of the task’s Compose file, which docker compose build would merge over Harbor’s template, is applied the same way: its context and Dockerfile, --build-arg per arg and --target (main_build_settings).

nemo_gym.tasks.harbor.build_tools.ImageBuilder.compose_build_argv(
    entry: nemo_gym.tasks.harbor.build_plan.BuildPlanEntry,
    compose_file: pathlib.Path
) -> list[str]

Harbor’s docker compose build line for the context (_run_docker_compose_command, https://github.com/harbor-framework/harbor/blob/v0.22.0/src/harbor/environments/docker/docker.py#L606).

The project name stands in for Harbor’s session id (Compose’s default image name is <project>-<service>; main is named explicitly, see harbor_compose_document); the project directory is the context, so relative paths in the task’s Compose file resolve as Harbor resolves them; the files are Harbor’s build template, then the task’s docker-compose.yaml when it has one, in Harbor’s order.

nemo_gym.tasks.harbor.build_tools.ImageBuilder.inspect(
    tag: str
) -> dict[str, typing.Any]

The image’s OCI configuration in the compose-images.json record shape (image unset).

nemo_gym.tasks.harbor.build_tools.ImageBuilder.inspect_argv(
    tag: str
) -> list[str]

nemo_gym.tasks.harbor.build_tools.ImageBuilder.push_argv(
    tag: str,
    digest_file: pathlib.Path
) -> list[str]

nemo_gym.tasks.harbor.build_tools.ImageBuilder.tag_for(
    entry: nemo_gym.tasks.harbor.build_plan.BuildPlanEntry
) -> str

The image’s name, hb__<hash>, under the repository when there is one.

The hash is the entry’s digest, Harbor’s environment id of the context, and hb__<hash> is the value Harbor exports as MAIN_IMAGE_NAME for it (the name a task Compose file that sets image: ${MAIN_IMAGE_NAME} builds under; Harbor’s own template leaves the tag to Compose).

nemo_gym.tasks.harbor.build_tools.ImageBuilder.tool_env() -> dict[str, str] | None

The environment buildah and podman commands run with, or None (inherit) for docker.

The host’s containers.conf may give every container a default bind mount (this deployment mounts /etc/localtime read-only into each one through containers.conf.d), and buildah and podman apply those mounts to build steps too, so a Dockerfile that installs tzdata (Ubuntu with python3 pulls it in) fails with mv: cannot move '/etc/localtime.dpkg-new' to '/etc/localtime': Device or resource busy. The builder writes a containers.conf with no default volumes under its storage root, once, and hands it to the tool as CONTAINERS_CONF_OVERRIDE, the layer containers-common applies over the host files. A CONTAINERS_CONF_OVERRIDE the caller already set is left as it is.

Both tools resolve an unqualified FROM (FROM oven/bun:1) through the host’s registries.conf, and Debian’s and Ubuntu’s stock file defines no unqualified-search registry, so the build fails with short-name "oven/bun:1" did not resolve to an alias where docker compose build pulls from docker.io. The builder writes a registries.conf naming docker.io as the one unqualified-search registry under its storage root and hands it to the tool as CONTAINERS_REGISTRIES_CONF. That file replaces the host’s main registries.conf (the drop-ins under registries.conf.d, short-name aliases included, still load), so a site whose main file carries mirrors or blocked registries exports its own CONTAINERS_REGISTRIES_CONF, which is left as it is. docker needs neither file and inherits the process environment.

nemo_gym.tasks.harbor.build_tools._docker_config(
    data: dict[str, typing.Any]
) -> dict[str, typing.Any]

The set fields of an inspect’s Docker.config section (the docker-format image configuration), or {}.

nemo_gym.tasks.harbor.build_tools._proxy_note(
    output: str,
    env: dict[str, str]
) -> str

Why a registry step may have answered 403, when the tool’s output says it did: a proxy the build tool inherits (from its environment, or from its own configuration when the environment names none) can answer 403 to the registry ping unless no_proxy carries the registry host. Names only, never values.

nemo_gym.tasks.harbor.build_tools._pushed_digest(
    tool: str,
    digest_file: pathlib.Path,
    completed: subprocess.CompletedProcess,
    run: nemo_gym.tasks.harbor.build_plan.Runner,
    tag: str
) -> str

The manifest digest the registry assigned to the pushed image.

nemo_gym.tasks.harbor.build_tools._run_default(
    argv: list[str],
    env: dict[str, str] | None = None
) -> subprocess.CompletedProcess

nemo_gym.tasks.harbor.build_tools.normalize_inspect(
    tool: str,
    data: typing.Any
) -> dict[str, typing.Any]

Turn a tool’s inspect JSON into the compose-images.json record shape (without image).

buildah reports {"OCIv1": {"architecture", "os", "config"}, "Docker": {"config"}, "FromImageID"}; podman and docker report a list with Architecture, Os, Config and Id. The image is built in docker format (IMAGE_FORMAT), whose Shell, Healthcheck and OnBuild the OCI image configuration cannot hold: buildah reports them under Docker.config, read here over the OCI config; docker reports them in Config; podman’s Config is the OCI image configuration (its inspect does not expose Shell and OnBuild, although the image carries them) and Healthcheck sits beside Config as a top-level key, read here when Config has none.

nemo_gym.tasks.harbor.build_tools.proxy_variable_names(
    env: dict[str, str]
) -> list[str]

The proxy-related variables in env (http_proxy, HTTPS_PROXY, no_proxy, all_proxy and the like), names only: their values can carry credentials.

nemo_gym.tasks.harbor.build_tools.BUILT_CONFIG_KEYS = (*RECORDED_CONFIG_KEYS, 'OnBuild', 'Shell')

nemo_gym.tasks.harbor.build_tools.COMPOSE_PLUGIN = 'docker-compose-plugin'

nemo_gym.tasks.harbor.build_tools.CONTAINERS_CONF_FILE = 'containers.conf'

nemo_gym.tasks.harbor.build_tools.CONTAINERS_CONF_OVERRIDE = 'CONTAINERS_CONF_OVERRIDE'

nemo_gym.tasks.harbor.build_tools.CONTAINERS_CONF_TEXT = '[containers]\nvolumes = []\n'

nemo_gym.tasks.harbor.build_tools.CONTAINERS_REGISTRIES_CONF = 'CONTAINERS_REGISTRIES_CONF'

nemo_gym.tasks.harbor.build_tools.HEREDOC_PROBE_DOCKERFILE = 'FROM scratch\nCOPY <<EOF /probe\nprobe\nEOF\n'

nemo_gym.tasks.harbor.build_tools.HEREDOC_PROBE_TAG = 'localhost/nemo-gym/heredoc-probe'

nemo_gym.tasks.harbor.build_tools.HEREDOC_REMEDY = "Dockerfile heredocs need a buildah or podman built with openshift/imagebuilder ...

nemo_gym.tasks.harbor.build_tools.HTTP_403 = re.compile('(?<![0-9a-fA-F])403(?![0-9a-fA-F])')

nemo_gym.tasks.harbor.build_tools.IMAGE_FORMAT = 'docker'

nemo_gym.tasks.harbor.build_tools.LOGGER = logging.getLogger(__name__)

nemo_gym.tasks.harbor.build_tools.REGISTRIES_CONF_FILE = 'registries.conf'

nemo_gym.tasks.harbor.build_tools.REGISTRIES_CONF_TEXT = 'unqualified-search-registries = ["docker.io"]\n'