nemo_gym.data.tasks.harbor.image_configs

One warning per task and service (shared_services), at prepare and on the server alike: the merge is Compose’s and not refused (an overlay written for a dataset may customize a service it knows by name, as Harbor’s extra -f files do), but an overlay service that carries the deployment’s values should not merge onto a definition a third-party task wrote under the same name.

Give the service fields of a task’s Compose document their meaning for a sandbox group; mutates document.

See nemo_gym.sandbox.compose_config.check_compose_fields: deploy.resources become sandbox resources (GPUs included), restart and pull_policy take their recorded forms, platform must be platform, the one the deployment runs (sandbox_provider_options.platform when set, else linux/amd64, the one prepare records images for; see compose_config.deployment_platform), networks and extra_hosts their mapping forms, post_start hooks are checked, and tmpfs, ulimits, sysctls and security_opt must be among what the service’s provider can apply. provider and gpu_provider are sandbox provider names (such as opensandbox) of the task’s block and of the gpu_sandbox_provider block; each service is checked against the one it runs on, as the server routes it: the task’s services run on gpu_provider when task.toml declares GPUs, and a sidecar that reserves GPUs through deploy.resources runs there as well. Anything else raises ImageConfigError naming the task, the file and the field, so prepare leaves the task out before a sandbox starts; with the job-level overlay the document came from, the file named is the task’s merged with the overlay files (compose_source), since the service or the field may be theirs. Without a provider name the capability check runs at seed instead, against the live provider.

The document checked is the one the server hands the adapter (environment_servers/harbor/app.py, _compose_document): the services prepare built carrying their recorded images without build: or pull_policy: build (with_built_images, applied by the loader), and main’s build: and pull_policy: build dropped when they only restate Harbor’s own build of the task’s Dockerfile (without_restated_main_build; a build key Gym cannot apply without a build is this task’s reason). Every other pull_policy stays: it is the service’s runtime pull policy, normalized here and applied by the adapter where the provider can, and build on a service nothing built fails the check. A sidecar build: prepare recorded no build for and any other key the adapter does not start are then refused by the check’s allow-list.

Every service the task’s artifacts entries and [[verifier.collect]] hooks name must be in the group (main always is: the group starts it from the task image). A task’s own file names them in practice; a job-level overlay file can reset one away (kafka: !reset against a hook with service = "kafka"), which only the merged document shows, so prepare skips such a task with the reason and the seed refuses it terminally, where the group’s start would fail and be retried for every episode. Raises ImageConfigError naming the task, its Compose source (compose_source) and the services.

The check that needs the recorded images: a sidecar whose image runs as a non-root user, which the group starts without host injection, must declare no extra_hosts (check_sidecar_extra_hosts). Prepare runs it once the images are recorded; the seed runs it on the merged document before anything is provisioned, where adapt_non_root_sidecars would otherwise raise it inside the group’s start. Raises ImageConfigError naming the task and its Compose source (with the overlay files merged onto it, compose_source).

The task’s Compose overlay, if it has one.

Every image a Compose task starts: the task image for main and each sidecar’s image. A service without an image and without a recorded build is refused by name; with the job-level overlay the document came from, naming the overlay files merged onto the task’s (compose_source), since the service may be theirs.

What a message about the task’s Compose document names as its source: environment/<file> for the task’s own file; with a job-level overlay that names Compose files, that file merged with Compose overlay files [...] (the files in merge order), or the overlay files alone for a task that declares no Compose file, since a field, a bind source or a service the group cannot run may then come from any of them.

The base image of every task Dockerfile the loader resolves under folder, with the tasks using it.

folder is a task folder or a folder of task folders, as discover_tasks reads it; with select, only the tasks whose folder name it accepts count. The image a task declares is the effective one, exactly as load_task sees it: task.toml after the dataset’s [gym] overrides. A per-task docker_image override thus turns a Dockerfile with RUN lines into a prebuilt image whose FROM is never resolved, so there is nothing to record for it. Tasks whose task.toml does not load, and a dataset whose dataset.toml does not load (which fails every task), are left for the loader to report.

Whether a service’s pull_policy is Compose’s build, the one that names a build step, not a pull.

The task’s Compose overlay with env_file entries inlined, ${VAR} references resolved and the images prepare built standing in for the services it built (with_built_images), main only when the loader resolved the task to its built image.

Returns None for a task without a Compose file, unless the job-level overlay names Compose files (HarborComposeOverlay): then a task without a file gets Harbor’s one-service group (main alone, synthesized_group) and every task’s document takes the overlay files merged onto it, in order, as Compose merges the extra -f files Harbor passes after the task’s (merge_compose). Each overlay file is interpolated with the task’s variables (the deployment’s, when given, above the task’s own: see operator_variables below); its relative paths resolve against the project directory, the task’s environment folder, as Compose resolves every -f file’s and Harbor’s --project-directory names it, never against the overlay file’s own folder: its env_file entries are made absolute against it before the merge (an overlay’s env_file may be any readable file; a task’s stays under the task) and its bind sources are left for the planner, which resolves them against the same folder as the task’s. The env files of the merged document are read once after the merge, the merged environment winning over them, as Compose reads them after merging the -f files. An overlay file that does not parse or merge raises ImageConfigError naming it; an error in the merged document (an env file it names, an interpolation left over, a services key the overlay removed) names the task’s file and the overlay files merged onto it (compose_source).

Prepare and the resources server both read the file through this function, so the document the sandbox starts is the one prepare validated, provided both run with the same process environment (see compose_variables). A required variable (${VAR:?msg}) without a value, a task variable the host does not set or a missing env_file raises ImageConfigError naming it; a variable nobody sets is an empty string, with a warning. The record is the one load_task read (task.built_images): nothing is hashed here, which runs once per episode on the server.

operator_variables are the deployment’s values (the environment server’s compose_variables), which the server resolves at seed: they fill the names neither the task, the process environment nor Harbor’s own variables set (SWE-Interact’s simulated user reads its model endpoint and key that way). In the overlay files they stand above the task’s own sources instead (its [environment].env, environment/.env and the process environment), below Harbor’s infrastructure set: the overlay files are the operator’s like the values, so a task cannot redirect what they do with the deployment’s values by setting one of their names itself. The pass-through spellings of an overlay service’s environment (a bare NAME in the list form, NAME: with no value in the mapping form) follow the same order as its ${NAME}: they are resolved before the merge (resolve_passthrough_environment), where the task’s own are read after it, in the task’s order. With them a name still unset fails here, before anything is provisioned; without them, as at prepare, it is an empty string with a warning, the images and structure being all prepare reads.

Resolve the volumes of a task’s Compose document for a sandbox group; mutates document.

Bind mounts of the task’s own files become x-sandbox.uploads entries (performed into the service once it starts), named volumes stay for the provider’s shared storage, and external volumes or bind sources outside the task resolve through volume_sources (the sandbox block’s mapping). Each service resolves through the mapping of the block it runs on, the way check_task_compose_fields routes the capability check: when gpu_volume_sources is given (the gpu_sandbox_provider block’s mapping), every service of a task that declares GPUs and a GPU-reserving sidecar of any task resolve through it. shared_storage and gpu_shared_storage say whether each block’s provider has shared storage, routed the same way: False leaves a task whose named volumes or shared logs binds need it out at prepare, naming the setting, rather than failing every seed; None (unknown) leaves that to the adapter. Anything the group cannot run raises ImageConfigError naming the volume, path or field, so prepare fails before a sandbox starts; with the job-level overlay the document came from, the message names the overlay files merged onto the task’s (compose_source). Returns the uploads per service.

Record the base image of every Dockerfile under folder next to the task folders, before they load.

select limits the scan to the tasks whose folder name it accepts (see dockerfile_base_images). Returns the file’s path, or None when no task has a Dockerfile the loader resolves. platform is as record_image_configs takes it.

Write compose-images.json in folder for the Compose tasks among tasks.

Returns the file’s path, or None when no task uses Compose. With a job-level overlay that names Compose files every task is a Compose task (load_compose_document): the overlay’s images are recorded with the tasks’, and a plain image task’s own image too, since the group starts it as main. Each Compose document’s service fields are checked on the way (see check_task_compose_fields), with sandbox_provider and gpu_sandbox_provider naming the providers of the sandbox: and gpu_sandbox_provider blocks for the capability check and compose_platform the deployment’s, which a service’s platform must match; platform is the one the images are recorded for, as record_image_configs takes it (the built images’ builds.platform). Each document’s volumes are planned next (see plan_task_compose_volumes), with volume_sources standing for the deployment’s external volumes; a task that declares GPUs, and a GPU-reserving sidecar of any task, are checked against gpu_volume_sources instead when that is given (the gpu_sandbox_provider block’s mapping, which is what the server resolves them against; an empty mapping when that block has none), shared_storage/gpu_shared_storage say whether each block’s provider has the shared storage named volumes and shared logs binds need (False leaves such a task out, naming the setting), and the services the task’s artifacts and [[verifier.collect]] hooks name must be in the merged document (check_task_compose_services; an overlay file can reset one away). A task whose Compose file does not resolve (an ImageConfigError from load_compose_document), carries a field the group cannot run, declares volumes it cannot run (the planner’s ImageConfigError), or lacks a service its hooks name does not stop the others (with an overlay, the reason names the overlay files merged onto the task’s, compose_source): with skipped given it is recorded there by task id with the reason and left out, otherwise one ImageConfigError at the end names every such task. Its images are still recorded, so a fixed task file needs no new registry round-trip. Once the image configurations are recorded, a sidecar whose image runs as a non-root user (which the group starts without host injection) must declare no extra_hosts (check_task_sidecar_extra_hosts), the way the adapter decides it at seed. An image the registry cannot resolve still raises for the whole set, as record_image_configs says.

Record the configuration of references (image reference to the tasks using it) into folder.

Entries already recorded for the platform of this run are kept as they are, so a pinned recording never changes under a completed run; only references missing from the file are resolved. An entry an older Gym wrote (no current record_version: it lacks fields the Dockerfile resolver reads), or one recorded for another platform (builds.platform was another when the file was written; the sandbox adapter refuses such a record at seed), is read again, at the digest it was pinned to rather than the tag, so the pin does not move either: behind a tag that digest is the multi-arch index, which names a manifest for the platform now asked for, so the configuration follows builds.platform under the same pin. A pin that is a single manifest of another platform cannot, and is reported with the remedy. Returns the file’s path, or None when there is nothing to record. A reference the registry cannot resolve raises ImageConfigError naming the image, the tasks and the registry’s answer. platform (os/arch, builds.platform; linux/amd64 when None) is the platform the default client resolves references for, and the one a record must be of to be kept; a client given carries its own.

The registry is asked outside the file’s sidecar lock (record_lock), so a slow registry never holds up another process; the file is then read again, merged and replaced under the lock, and an entry another process recorded in the meantime is kept over the one resolved here, as a pinned entry of this platform always is (a legacy or other-platform entry is replaced by its re-read).

own_services without the services the overlay document extra replaced or dropped, for the next overlay file’s shared_services: a service extra tags !override is the overlay’s definition from then on, one it tags !reset is gone, and a services mapping tagged either leaves none of the task’s. The next file then merges into the overlay’s definition, not the task’s (the standalone file plus the glue file of the README’s example), and is not warned about; a service an earlier overlay file only merged into keeps the task’s fields under it and stays shared.

Why main’s build section (as the Compose file spells it) is more than a restatement of Harbor’s own build of context_dir’s Dockerfile, as a reason naming the keys, or None when it only restates it and can be dropped. The rule of without_restated_main_build, on one section, for the loader’s interpolated document (context_dir is the task’s environment/ folder) and for the raw file prepare screens before a build (literal_contexts: the spellings of that folder an uninterpolated file carries, ${CONTEXT_DIR}).

A compose-images.json entry this run reads as it is: of the current record version and of platform (os, architecture), the one the run records images for. Anything else is resolved again at its pin by prepare, and refused at seed by the server, which cannot resolve it (_check_compose_images).

The services, main aside, that a task’s own Compose file declares (own_services) and the overlay document extra merges into rather than replaces.

Compose merges -f files by service name, the task’s file included, so a task that declares a name an overlay file declares keeps its definition (image, entrypoint, command, …) under the overlay’s fields, the overlay’s environment among them. A service extra tags !override starts the overlay’s definition alone, one it tags !reset is dropped and one it writes with no value adds nothing, so none of those is shared; main is the task’s own service, which the glue files extend by design (depends_on, networks).

Whether task.toml asks for GPUs (agent or verifier environment): the task then runs on the GPU block.

document with the recorded image of every service in images (service name to image, as built-images.json records them) set as its image and its build removed, along with a pull_policy: build, the policy that names that build.

Harbor builds every service with a build: section from the task’s Compose file at every start (main with args, a target or a dockerfile_inline, and the sidecars from their own contexts), and Harbor’s own template sets pull_policy: build. Gym builds those images once at prepare, with the same file, and records them by service name; here each recorded image stands in for its service’s build, so a sidecar’s own image: spelling (kumo’s ${MAIN_IMAGE_NAME}_verifier, a local tag under Harbor) is replaced by the name Gym built and recorded. The section and the build policy describe a build that already happened; the sandbox Compose adapter starts images only and refuses the section as an unsupported field, so they are dropped here rather than failing every episode of a task prepare accepted. Any other pull_policy (always, missing, never) is the service’s runtime pull policy, not part of its build, and stays for the field check as on any pulled service: it now governs the pull of the recorded image. Services not in images are untouched: a task in pull or overlay mode keeps whatever its file gives main, so an unsupported field there is still reported, and a sidecar prepare recorded no build for (with or without an image: of its own) is reported by compose_image_references.

document with pull_policy removed from every service.

For the raw-file screen prepare runs before a build (prepare_builds.refused_before_build), which reads keys only and accepts this one whatever its value. The documents the loader and the server hand the field check keep it: a service’s always, missing or never is the sandbox’s pull policy, which the adapter applies per sandbox where the provider can (check_compose_fields normalizes the spelling), and only the build policy, which names a build step rather than a pull, is dropped with the build it belongs to, by with_built_images for a service prepare built and by without_restated_main_build for a pull or overlay main restating Harbor’s template. build: is not touched here either: a sidecar’s is still refused (no provider starts a locally built sidecar).

document with main’s build: dropped when it only restates Harbor’s own build of the task’s Dockerfile, for a task in pull or overlay mode, and a reason instead when that section carries a key Gym cannot apply without a build. Any other task is returned as it is.

Harbor builds main from environment/Dockerfile through its own template, which already sets build.context: ${CONTEXT_DIR} (docker-compose-build.yaml at v0.22.0, appended before the task file), so a task file that repeats context, dockerfile: Dockerfile or pull changes nothing there; orca-bench, terminal_bench_aai and data-eng-bench do, some 600 hub tasks. Gym runs such a Dockerfile without a build (the loader folded it into the task in pull mode, or runs its RUN lines at seed in overlay mode), so the restatement is dropped as a built task’s section is (with_built_images). args, target, dockerfile_inline, another dockerfile or another context would change the image Harbor builds and have no equivalent without one, so the task is reported, not started on the wrong image.