nemo_gym.tasks.harbor.build_plan

Plan the image builds of a task folder.

Reads task folders and dataset.toml, decides which Dockerfiles need a build rather than a pull or an overlay, finds the sidecars with a build section and the separate verifier’s tests context, and groups the work by context digest into BuildPlanEntry values with BuildResult as the record of one finished build. Tool detection and the storage root live here because the plan names the tool. The module runs no subprocess and reads no Compose variables. It is the root of the build modules: build_compose, build_records, build_tools and image_build import it and it imports none of them.

Module Contents

Classes

Name	Description
`BuildError`	A build, push or inspect command failed; nothing was recorded for the image.
`BuildPlanEntry`	One distinct build context and the tasks that share it.
`BuildResult`	A built image, ready to record.

Functions

Name	Description
`default_storage_root`	Where buildah keeps its layers: node-local when the cluster scratch disk exists, else the temp dir.
`detect_tool`	The builder to use, or `None` when none is installed.
`needs_build`	Whether a Dockerfile is outside what pull and overlay mode can run.
`plan_builds`	The distinct build contexts among the tasks under `folder` whose Dockerfile needs a build.

Data

BUILD_TOOLS

DEFAULT_PLATFORM

DOCKERFILE_NAME

HARBOR_COMPOSE_FILE

API

class nemo_gym.tasks.harbor.build_plan.BuildError()

Bases: RuntimeError

A build, push or inspect command failed; nothing was recorded for the image.

class nemo_gym.tasks.harbor.build_plan.BuildPlanEntry(
    digest: str,
    context: pathlib.Path,
    tasks: tuple[str, ...],
    env: dict[str, str] = dict(),
    cpus: int | None = None,
    memory_mb: int | None = None,
    build_context: pathlib.Path | None = None,
    dockerfile: str = DOCKERFILE_NAME,
    build_args: dict[str, str] = dict(),
    compose_build: dict[str, typing.Any] | None = None,
    target: str | None = None,
    dockerfile_inline: bool = False
)

Dataclass

One distinct build context and the tasks that share it.

digest is Harbor’s environment id of the context (32 hex characters), the key of the build record and the body of the image name. env, cpus and memory_mb are the first task’s effective [environment] settings, the variables Harbor’s docker compose build sees (compose_build_env); env is empty when the context ships no docker-compose.yaml, as Harbor only resolves the task’s env for a Compose task (_uses_compose). Tasks sharing a context but declaring different settings share this image anyway, as they do in Harbor (the name is the content hash); plan_builds logs them.

build_context, dockerfile, build_args, target and dockerfile_inline are main’s build section of the task’s docker-compose.yaml (main_build_settings), the settings docker compose build merges over Harbor’s template: the direct build of buildah and podman applies them through ImageBuilder.build_argv. plan_builds leaves them at their defaults (the context and its Dockerfile); ImageBuilder.build fills them once the file has been checked, so a file that does not interpolate or is refused fails that context’s build, not the planning of the whole dataset.

build_args

dict[str, str] = field(default_factory=dict)

build_context

Path | None = None

build_dir

Path

The folder the tool builds: main.build.context resolved, else the context itself.

compose_build

dict[str, Any] | None = None

context

Path

cpus

int | None = None

digest

str

dockerfile

str = DOCKERFILE_NAME

dockerfile_inline

bool = False

dockerfile_path

Path

The Dockerfile the tool builds, main.build.dockerfile relative to build_dir.

env

dict[str, str] = field(default_factory=dict)

memory_mb

int | None = None

target

str | None = None

tasks

tuple[str, ...]

class nemo_gym.tasks.harbor.build_plan.BuildResult(
    digest: str,
    tasks: tuple[str, ...],
    tag: str,
    image: str,
    tool: str,
    pushed: bool,
    record: dict[str, typing.Any] = dict()
)

Dataclass

A built image, ready to record.

digest

str

image

str

pushed

bool

record

dict[str, Any] = field(default_factory=dict)

tag

str

tasks

tuple[str, ...]

tool

str

nemo_gym.tasks.harbor.build_plan.default_storage_root() -> pathlib.Path

Where buildah keeps its layers: node-local when the cluster scratch disk exists, else the temp dir.

nemo_gym.tasks.harbor.build_plan.detect_tool(
    repository: str | None = None
) -> str | None

The builder to use, or None when none is installed.

With a repository the image is pushed, so any tool serves: buildah, then podman, then docker. Without one the image has to stay where a sandbox provider can start it, and only the Docker daemon is such a place (buildah and podman keep images in a store no provider reads), so only docker qualifies.

nemo_gym.tasks.harbor.build_plan.needs_build(
    dockerfile_text: str,
    image_configs: dict[str, typing.Any] | None = None
) -> bool

Whether a Dockerfile is outside what pull and overlay mode can run.

A file whose instructions can only be resolved against its base image’s configuration is resolved against the record in image_configs (compose-images.json, as prepare records it before planning builds). Without a record the file is classed as a build, so a context is never left unbuilt because the base image was unknown.

nemo_gym.tasks.harbor.build_plan.plan_builds(
    folder: pathlib.Path,
    select: typing.Callable[[str], bool] | None = None
) -> list[nemo_gym.tasks.harbor.build_plan.BuildPlanEntry]

The distinct build contexts among the tasks under folder whose Dockerfile needs a build.

folder is a task folder or a folder of task folders; select limits the scan to the tasks whose folder name it accepts. A task counts when it has environment/Dockerfile, declares no [environment].docker_image after the dataset’s overrides, and the Dockerfile is outside pull and overlay mode. Tasks whose task.toml does not load are left for the loader to report. Entries are ordered by the first task that uses them. The task’s Compose file is not read here: nothing in it changes whether a context is built, and reading it is the build’s job (ImageBuilder.build), so a file one task gets wrong fails that task’s build and nothing else.

nemo_gym.tasks.harbor.build_plan.BUILD_TOOLS = ('buildah', 'podman', 'docker')

nemo_gym.tasks.harbor.build_plan.DEFAULT_PLATFORM = '/'.join(PLATFORM)

nemo_gym.tasks.harbor.build_plan.DOCKERFILE_NAME = 'Dockerfile'

nemo_gym.tasks.harbor.build_plan.HARBOR_COMPOSE_FILE = 'docker-compose.yaml'

nemo_gym.tasks.harbor.build_plan.LOGGER = logging.getLogger(__name__)

nemo_gym.tasks.harbor.build_plan.MAIN_SERVICE = 'main'

nemo_gym.tasks.harbor.build_plan.REPOSITORY_ENV = 'NEMO_GYM_BUILD_REPOSITORY'

nemo_gym.tasks.harbor.build_plan.Runner = Callable[..., subprocess.CompletedProcess]