nemo_gym.tasks.harbor.build_compose

The task’s docker-compose.yaml as docker compose build reads it.

Harbor builds with docker compose build over its own template followed by the task’s Compose file, with Harbor’s infrastructure variables in the environment. This module writes that template and Gym’s images override, resolves the build environment in Harbor’s order, interpolates the task file the way Compose does, reads each service’s build section into the plan, and refuses what must not run on the build host: secrets, ssh, extra contexts and hosts, host networking, cache credentials and paths outside the environment folder. The build-side variable helpers deliberately stay separate from the loader’s copies in compose_variables.

Module Contents

Functions

Name	Description
`_cache_credentials`	The credential key a `cache_from` / `cache_to` entry carries, if any.
`_compose_scalar`	A YAML scalar as Compose passes it to the build: `true`/`false` lower-case, else its text.
`_contained_build_path`	`value` (a Compose build path, already interpolated) resolved against `base` and checked to stay under
`_refuse`	-
`_warn_unset`	One warning per context and variable for a bare `${VAR}` the build resolved to an empty string.
`compose_build_env`	The environment of `docker compose build`, merged as Harbor’s `_compose_env_vars` merges it
`compose_interpolation_variables`	The variables `docker compose` interpolates the task’s Compose file with for a command run with `env`:
`harbor_compose_document`	The Compose file Harbor’s Docker environment builds a task with, resolved for one build context.
`image_name`	`hb__<digest>` sanitised as Harbor sanitises it, under `repository` when there is one: the name
`main_build_settings`	`services.main.build` of the task’s Compose file as `BuildPlanEntry` fields, or `{}` without one.
`resolve_task_env`	A task’s `[environment].env` the way Harbor hands it to `docker compose` (`resolve_env_vars`,
`task_compose_file`	The task’s own Compose file in `context`: `docker-compose.yaml`, the one name Harbor reads, or `None`.
`validate_task_compose`	Check the task’s Compose file before `docker compose build` runs it on the build host, and return it
`write_compose_file`	Write `harbor_compose_document` under `folder` as `docker-compose-build.yaml` and return it.

Data

CACHE_CREDENTIAL_KEYS

COMPOSE_BUILD_FILE

COMPOSE_DOTENV

COMPOSE_ENV_FILES

HARBOR_ENV_TEMPLATE

LOGGER

MAIN_IMAGE_VARIABLE

REFUSED_BUILD_KEYS

_WARNED_UNSET

API

nemo_gym.tasks.harbor.build_compose._cache_credentials(
    entry: typing.Any
) -> str | None

The credential key a cache_from / cache_to entry carries, if any.

nemo_gym.tasks.harbor.build_compose._compose_scalar(
    value: typing.Any
) -> str

A YAML scalar as Compose passes it to the build: true/false lower-case, else its text.

nemo_gym.tasks.harbor.build_compose._contained_build_path(
    value: typing.Any,
    base: pathlib.Path,
    context: pathlib.Path,
    path: pathlib.Path,
    service: str,
    key: str
) -> pathlib.Path

value (a Compose build path, already interpolated) resolved against base and checked to stay under context with no symlink escape; otherwise BuildError.

nemo_gym.tasks.harbor.build_compose._refuse(
    path: pathlib.Path,
    service: str,
    key: str,
    why: str
) -> nemo_gym.tasks.harbor.build_plan.BuildError

nemo_gym.tasks.harbor.build_compose._warn_unset(
    context: pathlib.Path,
    tasks: tuple[str, ...],
    unset: set[str]
) -> None

One warning per context and variable for a bare ${VAR} the build resolved to an empty string.

nemo_gym.tasks.harbor.build_compose.compose_build_env(
    entry: nemo_gym.tasks.harbor.build_plan.BuildPlanEntry,
    image: str,
    platform: str | None
) -> dict[str, str]

The environment of docker compose build, merged as Harbor’s _compose_env_vars merges it (https://github.com/harbor-framework/harbor/blob/v0.22.0/src/harbor/environments/docker/docker.py#L579): the process environment, then the task’s [environment].env (resolve_task_env), then Harbor’s infrastructure variables, which win: MAIN_IMAGE_NAME, CONTEXT_DIR, CPUS and MEMORY (as <n>M) from the task’s resources, and the legacy HOST_*_PATH / ENV_*_PATH log mounts, plus the two the loader defines for the same file (nemo_gym.tasks.harbor.image_configs.compose_variables): TEST_DIR and COMPOSE_PROJECT_NAME (hb__<hash>, the image name; a stable stand-in, not Harbor’s per-trial project name), so the build and the seed interpolate a task’s file from the same set. A task variable named like an infrastructure one is ignored with a warning, as merge_compose_env does. DOCKER_DEFAULT_PLATFORM carries builds.platform, the one setting Harbor has no equivalent of.

nemo_gym.tasks.harbor.build_compose.compose_interpolation_variables(
    context: pathlib.Path,
    env: dict[str, str],
    tasks: tuple[str, ...] = ()
) -> dict[str, str]

The variables docker compose interpolates the task’s Compose file with for a command run with env: that environment over the project directory’s .env file (Compose reads it from --project-directory, which Harbor sets to context; a variable set in the environment wins over the file). .env values are read as Compose’s dotenv reader reads them (compose-go/dotenv): an unquoted or double-quoted value is itself interpolated, in file order, against the environment and then the lines above it, with the document’s ${VAR}, ${VAR:-default} and $VAR forms and $$ (or \$ in double quotes) for a literal dollar; a single-quoted value is literal. A reference is resolved as it is in the document (substitute_with_empty, the loader’s rule): NET=${X:-host} in .env is host to the check as it is to Compose, a bare ${X} nobody sets is an empty string with one warning naming tasks, and ${X:?msg} is refused. COMPOSE_ENV_FILES set in env is refused: it points Compose at other files than the .env read here.

nemo_gym.tasks.harbor.build_compose.harbor_compose_document(
    context: pathlib.Path,
    image: str
) -> dict[str, typing.Any]

The Compose file Harbor’s Docker environment builds a task with, resolved for one build context.

Harbor’s docker-compose-build.yaml (v0.22.0, quoted in full; it is the first -f of every docker compose command Harbor runs for a task, see _docker_compose_paths at https://github.com/harbor-framework/harbor/blob/v0.22.0/src/harbor/environments/docker/docker.py#L316)::

services: main: build: context: ${CONTEXT_DIR} pull_policy: build command: [ “sh”, “-c”, “sleep infinity” ]

This is that file with CONTEXT_DIR resolved to the task’s environment/ folder and one addition: image set to hb__<hash>, the value Harbor exports as MAIN_IMAGE_NAME (ComposeInfraEnvVars.main_image_name at https://github.com/harbor-framework/harbor/blob/v0.22.0/src/harbor/environments/docker/docker.py#L238). Harbor’s template sets no image: its compose build tags <session>-main unless the task’s own Compose file says image: ${MAIN_IMAGE_NAME}. Gym needs a stable name to inspect, push and record the result under, so it sets that value itself. No dockerfile is named, as in Harbor’s file: Compose defaults to Dockerfile, and naming it would clash with a task file’s dockerfile_inline (Compose refuses a service declaring both). Every other field is Harbor’s, verbatim.

nemo_gym.tasks.harbor.build_compose.image_name(
    digest: str,
    repository: str | None
) -> str

hb__<digest> sanitised as Harbor sanitises it, under repository when there is one: the name ImageBuilder.tag_for builds under and MAIN_IMAGE_NAME in the build’s environment.

nemo_gym.tasks.harbor.build_compose.main_build_settings(
    entry: nemo_gym.tasks.harbor.build_plan.BuildPlanEntry,
    document: dict[str, typing.Any] | None,
    env: dict[str, str]
) -> dict[str, typing.Any]

services.main.build of the task’s Compose file as BuildPlanEntry fields, or {} without one.

document is the file as validate_task_compose returned it for env, the build’s environment (compose_build_env): interpolated as Compose interpolates it, its paths checked to stay under environment/. This is what docker compose build applies from the task file over Harbor’s template, read here so the direct build of buildah and podman applies the same: the build context (a path, or the string form of the section) and dockerfile, resolved under environment/ (_contained_build_path again, so a caller handing in an unchecked document still cannot escape it); args as a mapping or Compose’s KEY=value list, where a mapping value of null and a bare list KEY alike take the variable’s value from the build’s environment and are dropped when it is unset, as Compose drops them, so the Dockerfile’s own ARG default stands; target; and whether a dockerfile_inline is set. args of another shape is a BuildError.

nemo_gym.tasks.harbor.build_compose.resolve_task_env(
    env: dict[str, str],
    tasks: tuple[str, ...] = ()
) -> dict[str, str]

A task’s [environment].env the way Harbor hands it to docker compose (resolve_env_vars, https://github.com/harbor-framework/harbor/blob/v0.22.0/src/harbor/utils/env.py#L94): a whole value of the form ${VAR} or ${VAR:-default} is read from the process environment, the default standing in when the variable is unset; any other value is literal. A ${VAR} with no default and no value fails the build naming it, as Harbor fails the trial.

nemo_gym.tasks.harbor.build_compose.task_compose_file(
    context: pathlib.Path
) -> pathlib.Path | None

The task’s own Compose file in context: docker-compose.yaml, the one name Harbor reads, or None.

The loader accepts the other spellings (docker-compose.yml, compose.yaml, compose.yml) at run time; Harbor’s Docker environment never appends them to a build, so a context shipping one is refused here rather than built from a file Harbor would ignore, under the same hb__<hash> as Harbor’s build.

nemo_gym.tasks.harbor.build_compose.validate_task_compose(
    context: pathlib.Path,
    env: dict[str, str],
    tasks: tuple[str, ...] = ()
) -> dict[str, typing.Any] | None

Check the task’s Compose file before docker compose build runs it on the build host, and return it interpolated (None when the context ships none). Anything refused raises BuildError naming the service and key.

The file is third-party content and the build host often holds the push credentials, so what it may ask of the host is bounded to what a legitimate task build needs: main may carry build args, a target, a context and a Dockerfile under environment/ (symlinks resolved, like the loader’s env_file rule), or a dockerfile_inline. Refused on any service with a build section: secrets, ssh, extra_hosts, network: host, additional_contexts other than service: references, and cache_from / cache_to entries carrying credentials; refused anywhere: a top-level include and an extends with a file, which pull further files in unchecked. main’s image has to resolve to MAIN_IMAGE_NAME when set (${MAIN_IMAGE_NAME}, ${MAIN_IMAGE_NAME:-...} and ${MAIN_IMAGE_NAME-...} all do, since Harbor always exports the variable), or the build would be tagged under another name than the one recorded.

Compose interpolates the file against the command’s environment before it reads a single key, so the checks run on the document interpolated the same way: env is the environment the build runs with (compose_build_env), the project directory’s .env under it with its values expanded as Compose’s dotenv reader expands them (compose_interpolation_variables), with Compose’s ${VAR}, ${VAR:-default}, ${VAR-default}, ${VAR:?error}, ${VAR:+alt} and $VAR forms on every string. Otherwise network: ${X:-host} or cache_to: [type=gha,${X:-token}=abc] would pass a check on the raw text and resolve to the refused value at build time. A bare ${VAR} nobody sets is an empty string with one warning naming tasks and the variable, as docker compose build makes it and as the loader resolves the same file at seed (substitute_with_empty), so the document checked is the document built; ${VAR:?message} is refused with the message. main’s image is interpolated the same way before it is compared, so any spelling Compose resolves to the main image passes and a literal other image, or a reference to another variable, is refused with what it resolves to.

A sidecar with a build section is refused: the loader records a sidecar from its image: (one without is refused as “builds are not supported”) and no provider starts a locally built sidecar, so building it here would spend the time on an image nothing uses, or on one that then differs from the registry image the task runs with. Sidecars name a prebuilt image: and carry no build:.

nemo_gym.tasks.harbor.build_compose.write_compose_file(
    folder: pathlib.Path,
    context: pathlib.Path,
    image: str
) -> pathlib.Path

Write harbor_compose_document under folder as docker-compose-build.yaml and return it.

nemo_gym.tasks.harbor.build_compose.CACHE_CREDENTIAL_KEYS = frozenset({'token', 'secret', 'password', 'credentials', 'access_key_id', 'secre...

nemo_gym.tasks.harbor.build_compose.COMPOSE_BUILD_FILE = 'docker-compose-build.yaml'

nemo_gym.tasks.harbor.build_compose.COMPOSE_DOTENV = '.env'

nemo_gym.tasks.harbor.build_compose.COMPOSE_ENV_FILES = 'COMPOSE_ENV_FILES'

nemo_gym.tasks.harbor.build_compose.HARBOR_ENV_TEMPLATE = re.compile('\\$\\{([^}:]+)(?::-(.*))?\\}')

nemo_gym.tasks.harbor.build_compose.LOGGER = logging.getLogger(__name__)

nemo_gym.tasks.harbor.build_compose.MAIN_IMAGE_VARIABLE = 'MAIN_IMAGE_NAME'

nemo_gym.tasks.harbor.build_compose.REFUSED_BUILD_KEYS = ('secrets', 'ssh', 'extra_hosts')

nemo_gym.tasks.harbor.build_compose._WARNED_UNSET: set[tuple[str, str]] = set()