> This page is for version 0.6.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.data.tasks.harbor.build_compose The task's docker-compose.yaml as docker compose build reads it. Harbor builds with docker compose build over its own template followed by the task's Compose file, with Harbor's infrastructure variables in the environment. This module writes that template and Gym's images override, resolves the build environment in Harbor's order, interpolates the task file the way Compose does, reads each service's build section into the plan, and refuses what must not run on the build host: secrets, ssh, extra contexts and hosts, host networking, BuildKit entitlements, cache credentials and paths outside the environment folder. The build-side variable helpers deliberately stay separate from the loader's copies in compose_variables. ## Module Contents ### Functions | Name | Description | | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | [`_cache_credentials`](#nemo_gym-data-tasks-harbor-build_compose-_cache_credentials) | The credential key a `cache_from` / `cache_to` entry carries, if any. | | [`_compose_scalar`](#nemo_gym-data-tasks-harbor-build_compose-_compose_scalar) | A YAML scalar as Compose passes it to the build: `true`/`false` lower-case, else its text. | | [`_contained_build_path`](#nemo_gym-data-tasks-harbor-build_compose-_contained_build_path) | `value` (a Compose build path, already interpolated) resolved against `base` and checked to stay under | | [`_refuse`](#nemo_gym-data-tasks-harbor-build_compose-_refuse) | - | | [`_warn_unset`](#nemo_gym-data-tasks-harbor-build_compose-_warn_unset) | One warning per context and variable for a bare `${VAR}` the build resolved to an empty string. | | [`build_section`](#nemo_gym-data-tasks-harbor-build_compose-build_section) | `services.<service>.build` of a Compose document as a mapping (the string form read as its `context`), | | [`compose_build_env`](#nemo_gym-data-tasks-harbor-build_compose-compose_build_env) | The environment of `docker compose build`, merged as Harbor's `_compose_env_vars` merges it | | [`compose_interpolation_variables`](#nemo_gym-data-tasks-harbor-build_compose-compose_interpolation_variables) | The variables `docker compose` interpolates the task's Compose file with for a command run with `env`: | | [`harbor_compose_document`](#nemo_gym-data-tasks-harbor-build_compose-harbor_compose_document) | The Compose file Harbor's Docker environment builds a task with, resolved for one build context. | | [`image_name`](#nemo_gym-data-tasks-harbor-build_compose-image_name) | `hb__<digest>` sanitised as Harbor sanitises it, under `repository` when there is one: the name | | [`images_compose_document`](#nemo_gym-data-tasks-harbor-build_compose-images_compose_document) | Gym's images override for `docker compose build`: `image` per built service, `main` included. | | [`resolve_task_env`](#nemo_gym-data-tasks-harbor-build_compose-resolve_task_env) | A task's `[environment].env` the way Harbor hands it to `docker compose` (`resolve_env_vars`, | | [`service_build_settings`](#nemo_gym-data-tasks-harbor-build_compose-service_build_settings) | The entry's service's `build` section of the task's Compose file as `BuildPlanEntry` fields, or | | [`task_compose_file`](#nemo_gym-data-tasks-harbor-build_compose-task_compose_file) | The task's own Compose file in `context`: `docker-compose.yaml`, the one name Harbor reads, or `None`. | | [`validate_task_compose`](#nemo_gym-data-tasks-harbor-build_compose-validate_task_compose) | Check the task's Compose file before `docker compose build` runs it on the build host, and return it | | [`write_compose_files`](#nemo_gym-data-tasks-harbor-build_compose-write_compose_files) | Write Harbor's build template (`harbor_compose_document`) and Gym's images override | ### Data [`CACHE_CREDENTIAL_KEYS`](#nemo_gym-data-tasks-harbor-build_compose-CACHE_CREDENTIAL_KEYS) [`COMPOSE_BUILD_FILE`](#nemo_gym-data-tasks-harbor-build_compose-COMPOSE_BUILD_FILE) [`COMPOSE_DOTENV`](#nemo_gym-data-tasks-harbor-build_compose-COMPOSE_DOTENV) [`COMPOSE_ENV_FILES`](#nemo_gym-data-tasks-harbor-build_compose-COMPOSE_ENV_FILES) [`COMPOSE_IMAGES_FILE`](#nemo_gym-data-tasks-harbor-build_compose-COMPOSE_IMAGES_FILE) [`HARBOR_ENV_TEMPLATE`](#nemo_gym-data-tasks-harbor-build_compose-HARBOR_ENV_TEMPLATE) [`LOGGER`](#nemo_gym-data-tasks-harbor-build_compose-LOGGER) [`MAIN_IMAGE_VARIABLE`](#nemo_gym-data-tasks-harbor-build_compose-MAIN_IMAGE_VARIABLE) [`REFUSED_BUILD_KEYS`](#nemo_gym-data-tasks-harbor-build_compose-REFUSED_BUILD_KEYS) [`_WARNED_UNSET`](#nemo_gym-data-tasks-harbor-build_compose-_WARNED_UNSET) ### API ```python nemo_gym.data.tasks.harbor.build_compose._cache_credentials( entry: typing.Any ) -> str | None ``` The credential key a `cache_from` / `cache_to` entry carries, if any. ```python nemo_gym.data.tasks.harbor.build_compose._compose_scalar( value: typing.Any ) -> str ``` A YAML scalar as Compose passes it to the build: `true`/`false` lower-case, else its text. ```python nemo_gym.data.tasks.harbor.build_compose._contained_build_path( value: typing.Any, base: pathlib.Path, context: pathlib.Path, path: pathlib.Path, service: str, key: str ) -> pathlib.Path ``` `value` (a Compose build path, already interpolated) resolved against `base` and checked to stay under `context` with no symlink escape; otherwise `BuildError`. ```python nemo_gym.data.tasks.harbor.build_compose._refuse( path: pathlib.Path, service: str, key: str, why: str ) -> nemo_gym.data.tasks.harbor.build_plan.BuildError ``` ```python nemo_gym.data.tasks.harbor.build_compose._warn_unset( context: pathlib.Path, tasks: tuple[str, ...], unset: set[str] ) -> None ``` One warning per context and variable for a bare `${VAR}` the build resolved to an empty string. ```python nemo_gym.data.tasks.harbor.build_compose.build_section( document: dict[str, typing.Any] | None, service: str ) -> dict[str, typing.Any] | None ``` `services..build` of a Compose document as a mapping (the string form read as its `context`), or `None` when the service has none. ```python nemo_gym.data.tasks.harbor.build_compose.compose_build_env( entry: nemo_gym.data.tasks.harbor.build_plan.BuildPlanEntry, image: str, platform: str | None ) -> dict[str, str] ``` The environment of `docker compose build`, merged as Harbor's `_compose_env_vars` merges it ([https://github.com/harbor-framework/harbor/blob/v0.22.0/src/harbor/environments/docker/docker.py#L579](https://github.com/harbor-framework/harbor/blob/v0.22.0/src/harbor/environments/docker/docker.py#L579)): the process environment, then the task's `[environment].env` (`resolve_task_env`), then Harbor's infrastructure variables, which win: `MAIN_IMAGE_NAME`, `CONTEXT_DIR`, `CPUS` and `MEMORY` (as `M`) from the task's resources, and the legacy `HOST_*_PATH` / `ENV_*_PATH` log mounts, plus the two the loader defines for the same file (`nemo_gym.data.tasks.harbor.image_configs.compose_variables`): `TEST_DIR` and `COMPOSE_PROJECT_NAME` (`hb__`, the image name; a stable stand-in, not Harbor's per-trial project name), so the build and the seed interpolate a task's file from the same set. A task variable named like an infrastructure one is ignored with a warning, as `merge_compose_env` does. `DOCKER_DEFAULT_PLATFORM` carries `builds.platform`, the one setting Harbor has no equivalent of. ```python nemo_gym.data.tasks.harbor.build_compose.compose_interpolation_variables( context: pathlib.Path, env: dict[str, str], tasks: tuple[str, ...] = () ) -> dict[str, str] ``` The variables `docker compose` interpolates the task's Compose file with for a command run with `env`: that environment over the project directory's `.env` file (Compose reads it from `--project-directory`, which Harbor sets to `context`; a variable set in the environment wins over the file). `.env` values are read as Compose's dotenv reader reads them (`compose-go/dotenv`): an unquoted or double-quoted value is itself interpolated, in file order, against the environment and then the lines above it, with the document's `${VAR}`, `${VAR:-default}` and `$VAR` forms and `$$` (or `\$` in double quotes) for a literal dollar; a single-quoted value is literal. A reference is resolved as it is in the document (`substitute_with_empty`, the loader's rule): `NET=${X:-host}` in `.env` is `host` to the check as it is to Compose, a bare `${X}` nobody sets is an empty string with one warning naming `tasks`, and `${X:?msg}` is refused. `COMPOSE_ENV_FILES` set in `env` is refused: it points Compose at other files than the `.env` read here. ```python nemo_gym.data.tasks.harbor.build_compose.harbor_compose_document( context: pathlib.Path, image: str ) -> dict[str, typing.Any] ``` The Compose file Harbor's Docker environment builds a task with, resolved for one build context. Harbor's `docker-compose-build.yaml` (v0.22.0, quoted in full; it is the first `-f` of every `docker compose` command Harbor runs for a task, see `_docker_compose_paths` at [https://github.com/harbor-framework/harbor/blob/v0.22.0/src/harbor/environments/docker/docker.py#L316](https://github.com/harbor-framework/harbor/blob/v0.22.0/src/harbor/environments/docker/docker.py#L316)): ```python services: main: build: context: ${CONTEXT_DIR} pull_policy: build command: [ "sh", "-c", "sleep infinity" ] ``` This is that file with `CONTEXT_DIR` resolved to the task's `environment/` folder and one addition: `image` set to `hb__`, the value Harbor exports as `MAIN_IMAGE_NAME` (`ComposeInfraEnvVars.main_image_name` at [https://github.com/harbor-framework/harbor/blob/v0.22.0/src/harbor/environments/docker/docker.py#L238](https://github.com/harbor-framework/harbor/blob/v0.22.0/src/harbor/environments/docker/docker.py#L238)). Harbor's template sets no `image`: its `compose build` tags `-main` unless the task's own Compose file says `image: ${MAIN_IMAGE_NAME}`. Gym needs a stable name to inspect, push and record the result under, so it sets that value itself. No `dockerfile` is named, as in Harbor's file: Compose defaults to `Dockerfile`, and naming it would clash with a task file's `dockerfile_inline` (Compose refuses a service declaring both). Every other field is Harbor's, verbatim. ```python nemo_gym.data.tasks.harbor.build_compose.image_name( digest: str, repository: str | None ) -> str ``` `hb__` sanitised as Harbor sanitises it, under `repository` when there is one: the name `ImageBuilder.tag_for` builds under and `MAIN_IMAGE_NAME` in the build's environment. ```python nemo_gym.data.tasks.harbor.build_compose.images_compose_document( images: collections.abc.Mapping[str, str] ) -> dict[str, typing.Any] ``` Gym's images override for `docker compose build`: `image` per built service, `main` included. Harbor passes its template, then the task's file, then its own generated overrides (`docker.py#L363-L371` at v0.22.0). Gym's one override, passed last, names every service it builds: `main` as `hb__` (the template already says so) and each sidecar as `hb__-`. Last wins in Compose, so a sidecar's own `image:` in the task file (kumo's `${MAIN_IMAGE_NAME}_verifier`) is overridden without a `docker tag` step, and a sidecar without one gets Gym's name instead of Compose's `-` default, which under the project name Gym passes is the same string. ```python nemo_gym.data.tasks.harbor.build_compose.resolve_task_env( env: dict[str, str], tasks: tuple[str, ...] = () ) -> dict[str, str] ``` A task's `[environment].env` the way Harbor hands it to `docker compose` (`resolve_env_vars`, [https://github.com/harbor-framework/harbor/blob/v0.22.0/src/harbor/utils/env.py#L94](https://github.com/harbor-framework/harbor/blob/v0.22.0/src/harbor/utils/env.py#L94)): a whole value of the form `${VAR}` or `${VAR:-default}` is read from the process environment, the default standing in when the variable is unset; any other value is literal. A `${VAR}` with no default and no value fails the build naming it, as Harbor fails the trial. ```python nemo_gym.data.tasks.harbor.build_compose.service_build_settings( entry: nemo_gym.data.tasks.harbor.build_plan.BuildPlanEntry, document: dict[str, typing.Any] | None, env: dict[str, str] ) -> dict[str, typing.Any] ``` The entry's service's `build` section of the task's Compose file as `BuildPlanEntry` fields, or `{}` without one. `document` is the file as `validate_task_compose` returned it for `env`, the build's environment (`compose_build_env`): interpolated as Compose interpolates it, its paths checked to stay under `environment/`. This is what `docker compose build` applies from the task file (over Harbor's template for `main`), read here so the direct build of buildah and podman applies the same: the build `context` (a path, or the string form of the section) and `dockerfile`, resolved under `environment/` (`_contained_build_path` again, so a caller handing in an unchecked document still cannot escape it); `args` as a mapping or Compose's `KEY=value` list, where a mapping value of `null` and a bare list `KEY` alike take the variable's value from the build's environment and are dropped when it is unset, as Compose drops them, so the Dockerfile's own `ARG` default stands; `target`; and whether a `dockerfile_inline` is set. `args` of another shape is a `BuildError`. ```python nemo_gym.data.tasks.harbor.build_compose.task_compose_file( context: pathlib.Path ) -> pathlib.Path | None ``` The task's own Compose file in `context`: `docker-compose.yaml`, the one name Harbor reads, or `None`. The loader accepts the other spellings (`docker-compose.yml`, `compose.yaml`, `compose.yml`) at run time; Harbor's Docker environment never appends them to a build, so a context shipping one is refused here rather than built from a file Harbor would ignore, under the same `hb__` as Harbor's build. ```python nemo_gym.data.tasks.harbor.build_compose.validate_task_compose( context: pathlib.Path, env: dict[str, str], tasks: tuple[str, ...] = (), builds_main: bool = True ) -> dict[str, typing.Any] | None ``` Check the task's Compose file before `docker compose build` runs it on the build host, and return it interpolated (`None` when the context ships none). Anything refused raises `BuildError` naming the service and key. The file is third-party content and the build host often holds the push credentials, so what it may ask of the host is bounded to what a legitimate task build needs: a service with a `build` section may carry args, a target, a context and a Dockerfile under `environment/` (symlinks resolved, like the loader's `env_file` rule), or a `dockerfile_inline`. Refused on any service with a `build` section: `secrets`, `ssh`, `extra_hosts`, `entitlements` and `privileged` (BuildKit's `--allow network.host` / `security.insecure`, which let a `RUN --network=host` line reach the build host's network), `network: host`, `additional_contexts` other than `service:` references, and `cache_from` / `cache_to` entries carrying credentials; refused anywhere: a top-level `include` and an `extends` with a `file`, which pull further files in unchecked. `main`'s `image` has to resolve to `MAIN_IMAGE_NAME` when set and `main` is among the services being built (`builds_main`): `${MAIN_IMAGE_NAME}`, `${MAIN_IMAGE_NAME:-...}` and `${MAIN_IMAGE_NAME-...}` all do, since Harbor always exports the variable; otherwise the build would be tagged under another name than the one recorded. A sidecar-only build (a prebuilt, pull-mode or overlay `main`, or one already recorded) neither reads nor tags `main`'s image, so its spelling is not checked, as Harbor's prebuilt mode never reads it. Compose interpolates the file against the command's environment before it reads a single key, so the checks run on the document interpolated the same way: `env` is the environment the build runs with (`compose_build_env`), the project directory's `.env` under it with its values expanded as Compose's dotenv reader expands them (`compose_interpolation_variables`), with Compose's `${VAR}`, `${VAR:-default}`, `${VAR-default}`, `${VAR:?error}`, `${VAR:+alt}` and `$VAR` forms on every string. Otherwise `network: ${X:-host}` or `cache_to: [type=gha,${X:-token}=abc]` would pass a check on the raw text and resolve to the refused value at build time. A bare `${VAR}` nobody sets is an empty string with one warning naming `tasks` and the variable, as `docker compose build` makes it and as the loader resolves the same file at seed (`substitute_with_empty`), so the document checked is the document built; `${VAR:?message}` is refused with the message. `main`'s `image` is interpolated the same way before it is compared, so any spelling Compose resolves to the main image passes and a literal other image, or a reference to another variable, is refused with what it resolves to. A sidecar with a `build` section passes the same checks as `main`'s and is built (`plan_builds`), as Harbor builds it. Its `image:`, when it has one, is accepted whatever it says: the build is tagged with Gym's name for it (`ImageBuilder.tag_for`) and the seed document is filled from the record by service name, so no rule on its spelling is needed. ```python nemo_gym.data.tasks.harbor.build_compose.write_compose_files( folder: pathlib.Path, context: pathlib.Path, main_image: str, images: collections.abc.Mapping[str, str] ) -> tuple[pathlib.Path, pathlib.Path] ``` Write Harbor's build template (`harbor_compose_document`) and Gym's images override (`images_compose_document`) under `folder` and return their paths, in the order they are passed. ```python nemo_gym.data.tasks.harbor.build_compose.CACHE_CREDENTIAL_KEYS = frozenset({'token', 'secret', 'password', 'credentials', 'access_key_id', 'secre... ``` ```python nemo_gym.data.tasks.harbor.build_compose.COMPOSE_BUILD_FILE = 'docker-compose-build.yaml' ``` ```python nemo_gym.data.tasks.harbor.build_compose.COMPOSE_DOTENV = '.env' ``` ```python nemo_gym.data.tasks.harbor.build_compose.COMPOSE_ENV_FILES = 'COMPOSE_ENV_FILES' ``` ```python nemo_gym.data.tasks.harbor.build_compose.COMPOSE_IMAGES_FILE = 'docker-compose-images.yaml' ``` ```python nemo_gym.data.tasks.harbor.build_compose.HARBOR_ENV_TEMPLATE = re.compile('\\$\\{([^}:]+)(?::-(.*))?\\}') ``` ```python nemo_gym.data.tasks.harbor.build_compose.LOGGER = logging.getLogger(__name__) ``` ```python nemo_gym.data.tasks.harbor.build_compose.MAIN_IMAGE_VARIABLE = 'MAIN_IMAGE_NAME' ``` ```python nemo_gym.data.tasks.harbor.build_compose.REFUSED_BUILD_KEYS = ('secrets', 'ssh', 'extra_hosts', 'entitlements', 'privileged') ``` ```python nemo_gym.data.tasks.harbor.build_compose._WARNED_UNSET: set[tuple[str, str]] = set() ```