nemo_gym.data.tasks.harbor.task

A task folder read into memory.

task_id is the folder name: [task].name is an org/name label that is not unique across a dataset. folder_digest is the folder’s content hash alone (content_hash, what manifest.toml records per task and a fetch verifies); digest is the one rows carry and the server checks, that hash plus the dataset.toml settings applied to the task (task_digest), equal to folder_digest when the dataset has no [gym] table. image is the prebuilt image when the task declares one, else the base image of a pull-mode or overlay-mode Dockerfile pinned to the digest its recorded configuration was read at (the multi-arch index behind the written tag, see nemo_gym.data.tasks.harbor.image_configs; the written reference when nothing is recorded), else None (a build is required). workdir, env and user merge task.toml with what the Dockerfile recorded; task.toml wins.

Dockerfile ENV, WORKDIR and USER are resolved to literals as docker build resolves them, starting from the base image’s configuration recorded in IMAGE_CONFIGS_FILE next to the task folders (see nemo_gym.data.tasks.harbor.dockerfile).

overlay is Gym-derived data, not a Harbor field: the Dockerfile’s RUN lines (with the WORKDIR, ENV and USER in effect for each) that the server runs in the pulled base image at seed. Empty for a prebuilt image or a pull-mode Dockerfile.

overlay_workdirs are the Dockerfile’s WORKDIR instructions for an overlay-mode task, in order with the user in effect at each, so the server creates every one of them as a build would, not only the ones a RUN line happens to use.

sandbox_user is the user the sandbox runs commands as when no user is asked for: the default user of the image the sandbox starts from. For a Dockerfile Gym runs itself that is the base image’s recorded User (None is root), not the Dockerfile’s final USER or [agent].user; for an image prepare built (forced over a declared docker_image too) it is the built image’s recorded User, whatever [agent].user says; for a declared docker_image built from a pull-mode Dockerfile it is that Dockerfile’s final USER. When neither is known the task’s user stands in for it.

built_image is the image prepare built from the task’s environment/ folder when the Dockerfile needed a build (built-images.json); image is then that image and the task runs in pull mode from it. None in the other two modes and for a prebuilt image.

verifier_image is the image a separate verifier runs in when it is not the task’s own: the [verifier.environment].docker_image the task declares, else the image prepare built from tests/Dockerfile (service verifier under Harbor’s hash of tests/ in built-images.json, see verifier_build_context). None in shared mode, and in separate mode when the verifier runs in the task’s own image because the effective verifier environment is a copy of [environment] that carries its docker_image (Harbor’s prebuilt verifier). Every other separate-mode shape loads with a verifier image or not at all: a task whose effective verifier environment names no image and ships no tests/Dockerfile is refused as Harbor refuses it, and one whose tests/Dockerfile has no recorded build is refused like an unrecorded environment/ build (load_task).

context_id is Harbor’s environment id of environment/ (context_digest, the key of built-images.json and the body of the hb__<hash> image names), computed once here for every task with an environment/ folder (the build-mode branches read it, and the Compose loader needs it at every seed, for a Compose file or for the one-service group a job-level overlay extends) and None for a task without one; built_images is every image that record holds for the context, by Compose service name (main and the sidecars with a build: section, recorded_service_images), {} when none is recorded. The Compose loader substitutes them at prepare and at seed without hashing the folder again; main’s entry is applied only when built_image is set, since a record of main may be another task’s build of the same context or one that predates a docker_image override.

entrypoint is the image’s Entrypoint as Docker would start it: the built image’s recorded one, or the base image’s for a Dockerfile the loader resolves unless that Dockerfile wrote ENTRYPOINT []. None when the image records none or no configuration is recorded (a prebuilt docker_image). image_shell is the image’s Shell (Docker’s /bin/sh -c when it sets none), what the daemon runs a CMD-SHELL healthcheck through, None likewise when nothing is recorded. entrypoint_user and entrypoint_workdir are the User and WorkingDir the image starts with (None is root and /): what Docker gives its ENTRYPOINT and its HEALTHCHECK, independent of [agent].user and [environment].workdir, which only shape the commands run beside them. They are known whenever the image’s configuration is, with or without an entrypoint. image_healthcheck is the recorded Dockerfile HEALTHCHECK block (Test, Interval, … as Docker records them), or None.

verifier_image_config is the recorded configuration of the separate verifier’s image, verifier_image (what Docker starts it with: its Entrypoint, User, WorkingDir and Healthcheck, the way the fields above describe the task image’s): the image prepare built from tests/Dockerfile, or a declared [verifier.environment].docker_image whose reference compose-images.json records, as it does the base image of this task’s Dockerfile (pull or overlay mode) and of the other tasks’. The record is the image’s own, never the Dockerfile’s final USER and WORKDIR or its RUN lines: Harbor up --waits the declared image as it is. None for a declared image nothing is recorded for (as for a prebuilt docker_image), when the verifier grades in the task’s own image (the task’s own fields describe that start), and in shared mode.

turns holds the task’s [[steps]] in order; empty for a single-turn task. A multi-turn task never sends instruction to the agent: each turn sends its own, and the top-level file may hold material the agent must not see (SWE-Interact puts the full hidden specification there).

task_data is the task’s own field values, read from task_data.json in its folder (a parameterized task’s rendered row), {} for a task without one. Materialize writes them into the row’s task_data beside the reserved _ng_* keys, and the dataset’s task_data.py validates them when it ships one (nemo_gym.data.tasks.harbor.task_data).

seed_trajectory is Harbor’s seed trajectory when the task ships one, relative to the task folder: steps/<first step>/trajectory.json for a multi-step task, else trajectory.json (one or the other, as Harbor’s _task_trajectory_path picks it); None otherwise. The environment server runs such a task for the oracle alone and refuses it for any other agent, naming the file.

nemo_gym.data.tasks.harbor.task.HarborTask.turn_solution_dir(
    turn: nemo_gym.data.tasks.harbor.task.HarborTurn
) -> pathlib.Path

One task step loaded with its steps/<name>/ folder, which the harbor server runs as one agent turn.

config is the step as task.toml writes it (HarborTaskStep) and instruction is steps/<name>/instruction.md, so HarborTaskStep is to HarborTurn what HarborTaskConfig is to HarborTask. One turn per task step is this server’s choice, as in Harbor’s multi-step trial, not something a task step implies: Harbor’s simulated-user trials run many turns of the agent under test inside one step.

verifier_mode is the step’s own, as Harbor’s resolve_step_verifier_mode resolves it (the step’s explicit mode, else separate when it declares a [steps.verifier.environment], else the task’s). verifier_image and verifier_image_config are, for a separate turn, what HarborTask.verifier_image and HarborTask.verifier_image_config are for a single-step task: the image the turn’s verifier runs in when it is not the task’s own (None runs the task image), by the same rules, and that image’s recorded configuration when there is one.

root is the dataset folder whose built-images.json the loader read: the task’s parent, or the dataset root of a task discovered deeper (the record is never next to a group folder).

Whether an agent user runs with uid 0 (None is the image’s own user, root in most images).

Read every [[steps]] entry as a turn, refusing the per-step features the server does not run yet.

Each turn needs steps/<name>/instruction.md and a test.sh, its own under steps/<name>/tests/ or the task’s under tests/, which is the script Harbor runs for the step. A step’s own [steps.agent].user is read as written: the server switches the sandbox’s default user, the working directory’s owner and the harness’s user to it for that turn (and refuses a name:group pair at seed on a provider that switches user by name only, as it refuses the task’s own). A step’s workdir/, workdir/setup.sh, [steps.healthcheck] and artifacts are read by the server per turn (HarborTurn.workdir_dir, HarborTurn.setup_script, HarborTaskStep.healthcheck and HarborTaskStep.artifacts, collected after the turn’s agent with the task’s); a setup.sh beside instruction.md instead of inside workdir/ is not a Harbor file and is warned about, since Harbor 0.22.0 runs steps/<name>/workdir/setup.sh only. A step without its own [steps.verifier].timeout_sec is graded within Harbor’s step default of 600 s (times the dataset’s timeout_multiplier, like every budget), not the task’s [verifier].timeout_sec (HarborTaskConfig.step_verifier_timeout_sec); when the task’s figure differs, the steps it does not reach are warned about once, since an author who set it for the steps would expect it to apply.

Whether any built-images.json entry of root records task_name as built under builds.force over its declared image: one file read, so a declared-image task whose Dockerfile the loader can run pays Harbor’s hash of environment/ (the key its own record sits under) only when such a record may exist.

The separate turns’ verifier images, by the rules load_task applies to a single-step task’s.

Harbor starts a step’s separate verifier from steps/<name>/tests/ when that folder exists, else from tests/ (_verifier_env_build_context), with the step’s effective environment (step_verifier_environment): a declared docker_image is run as it is, otherwise the folder’s Dockerfile is built. Prepare builds tests/Dockerfile only, so a step whose verifier would be built from its own steps/<name>/tests/ is refused naming the image to declare; one built from tests/ reads the recorded build, as the task’s does. The docker-compose.yaml of the folder the step starts from (the step’s own tests/ or the task’s, which Harbor stacks on the step verifier’s template, _docker_compose_paths) is refused as the task’s verifier’s is: one that is not read as written (not valid YAML, not a mapping, or carrying a top-level key other than services, name, version and x- extensions, an include or a networks block among them: verifier_compose_refusal), services beside main, and a runtime setting on main (verifier_compose_main_overrides; main’s image, its build section and a pull_policy of build are ones beside a step verifier with a docker_image), which would reach Harbor’s step verifier and never this server’s.

The seed trajectory the task ships, relative to the task folder, or None; warns when there is one.

Harbor 0.22.0 loads trajectory.json beside the instruction (steps/<first step>/trajectory.json for a multi-step task, where a top-level one is ignored: Trial._task_trajectory_path picks one path or the other, never both) into the agent’s session as prior context, as its --load-trajectory option does, for every agent but the oracle and nop (the oracle runs the task’s own solution, so the prior context does not apply to it, and task authors validate with it); an agent that cannot load an ATIF trajectory fails the trial before any environment spend. This server hands no prior context to a harness yet, so running the task for a model would grade it without the context the task was written for: the task loads, the oracle runs it as Harbor’s does, and the environment server refuses it at seed for any other agent, naming the file. The file stays part of the task’s identity and its package (content_hash covers it), so a fetched task verifies against its advertised hash.

The image prepare built for the task’s Dockerfile, when built-images.json records one.

root is the dataset folder the record sits in (the task’s parent by default).

The images prepare built for the task’s environment/ context, by Compose service name ({} when none are recorded): main for the task’s own Dockerfile, each built sidecar under its service name.

root is the dataset folder the record sits in (the task’s parent by default).

A folder with task.toml is one task; otherwise its subfolders are searched for tasks.

The search is recursive (see iter_task_folders), so a dataset may group its tasks in folders; task_id stays the task folder’s own name and must be unique across the dataset. Dataset-level files (dataset.toml, the recorded image configurations) are read from root.

With skipped, a child that fails to load is recorded there under its folder name and the others still load; without it the first failure raises. A root that is itself a task always raises.

The image whose recorded configuration the task’s Dockerfile resolves against, if it has one.

A prebuilt docker_image already holds whatever its Dockerfile’s RUN lines did, so only a pull-mode Dockerfile contributes settings next to a declared image. Dockerfiles that need a build contribute nothing (load_task reports them).

Whether the task ships a Compose overlay under environment/ (COMPOSE_FILE_NAMES).

Every task folder under root, sorted by path: a folder with task.toml is a task, any other folder is searched in turn, up to max_depth levels down. Hidden folders are skipped and symbolic links are never followed (a linked task folder is listed, and load_task refuses it).

A task’s ID is its folder name, so two task folders of the same name anywhere under root raise HarborTaskError here, before any caller (base-image recording, Dockerfile builds, discovery) does work keyed by that name.

Read path as one task. Raises HarborTaskError when it is not one.

root is the dataset folder whose files shape the task (dataset.toml and the recorded image configurations); it is the task’s parent folder unless the task was discovered deeper. image_configs is that folder’s recorded image configurations when the caller has read them already (discover_tasks reads the shared file once); otherwise they are read here.

instruction.md as the agent sees it: leading canary marker lines and blank lines dropped.

Harbor tasks open with an HTML comment or heading carrying a canary GUID for contamination tracking. It is not part of the task, so it is removed the way Gym’s Terminal Bench servers always did; everything after it is kept byte for byte.

The digest a task is materialized and served under: its folder’s content hash plus the dataset slice applied to it.

dataset.toml’s [gym] table can change the effective task without touching the folder, so the defaults and the task’s own override are folded in whenever the table shapes tasks (GymDatasetConfig.shapes_tasks). A dataset whose table shapes none, because it has no table or declares only identity, run and catalog keys (license, type, allowed_agents, domain, verified, [gym.defaults].num_repeats), keeps the plain folder hash, which is also what the package store checks: none of those keys says what the task is, so none enters its digest. shared_verifier_digest is the digest of the dataset root’s verifier.py when it grades the task: grading code outside the folder is part of the task’s identity too.