nemo_gym.data.tasks.harbor.prepare

What gym dataset pull fetched: the reference, where it landed and the manifest left there, current.

nemo_gym.data.tasks.harbor.prepare.PulledDataset.fetched_line() -> str

nemo_gym.data.tasks.harbor.prepare.PulledDataset.superseded_line() -> str

nemo_gym.data.tasks.harbor.prepare.PulledDataset.text() -> str

Refuse a [gym.tasks."<id>"] table that names no task folder: a misspelt id would otherwise apply to nothing.

A target that is itself a task folder is prepared without its siblings, so the check is skipped for it.

Refuse folder’s task folders as entry’s pull unless its manifest records a fetch of the entry’s source.

The manifest’s source says which fetcher wrote it and its ref which reference the fetch was made from (declares_recorded_fetch reads the declaration against it); both are checked offline, so an entry whose source moved to another tag since the pull names the pull instead of running the tasks of the tag it had. A manifest that records no reference (written before it was recorded) names the pull too, and the pull records it without fetching a folder that still matches.

The dataset’s manifest, reconciled with the loaded tasks’ folder digests.

A folder of task folders carries manifest.toml, reconciled by check_manifest before the builds (checked, with the folder digests it hashed). The loader hashed the folders that loaded again, with the same function (HarborTask.folder_digest); a digest that differs means a folder changed under the preparation, so the manifest is reconciled once more with the loader’s digests (and a fetched dataset refused). The manifest records folder digests only: HarborTask.digest, the one rows carry, also folds in the dataset.toml settings, which the manifest covers once through dataset_toml_digest, so a settings edit changes the dataset digest but is not a task file edit. A single task folder names no dataset folder of its own (its parent is whatever holds it), so its identity is computed in memory over that one task, with the dataset.toml beside it, and nothing is written (checked is None exactly then). When a fetched dataset’s check reaches the task (its manifest is the nearest fetched one within discovery range, wherever under that folder the task sits), parent is that manifest, which check_task_manifest held the folder to before the builds: the identity is that same in-memory one with the fetch’s source instead of local (still over the dataset.toml beside the task: the manifest holds the task’s content, not the settings a nested task is prepared with), and a folder that changed under the preparation is held to the manifest once more with the loader’s digest, as a dataset folder is.

The dataset label the rows of folder carry as task_source.

A dataset folder target is labelled by its own name, the taskset. A single task folder target is labelled by its parent’s name when the parent is a dataset: it holds a dataset.toml or a manifest.toml, so the table, image records and digests the preparation uses are the parent’s, and two single-task runs of one dataset carry one label. A task folder whose parent holds neither is a standalone task (the parent is only the directory that happens to hold it: a scratch folder, a user’s home) and is labelled by its own name, as a hand-built taskset is, so two unrelated task folders prepared from one parent do not share a label.

reasons (task id to why it was left out) as indented lines; a reason that starts with its task id says it once.

The manifest in folder as it stands, or None; the fetch itself reports an unreadable one.

The error of a preparation that keeps no task, listing every task left out and why.

The dataset’s [gym] table, or None when dataset.toml does not parse (the loader reports that).

Write run_config.yaml and return it with the Hydra overrides for collection.

The dataset’s [gym] table sets the run’s defaults: the split the collector runs is the entry’s type (+split= on the command line still wins), and the run config’s top-level num_repeats is [gym.defaults].num_repeats, below every config file and command-line override, so +num_repeats= replaces it. The count lives at the run level and not in the entry: collate would duplicate rows by an entry count and the collector would multiply on top, and the metrics sidecar beside the rows records the entry, so a moving entry count would conflict run to run. The oracle’s validation run is not an evaluation: validate_target adds +num_repeats=1 unless the command line sets a count, so each reference solution plays once. An example dataset is refused here again (refuse_example_dataset; prepare_target refused it before anything was written), so a hand-built taskset of smoke-test samples does not run either.

The entry carries the dataset’s allowed_agents, and the collector’s pairing guard checks the agent block’s harness against it at dispatch. The oracle plays the reference solution, not a model, and is always admitted (prepare_target checks no agent for it), so its run config carries the entry without the list: the list speaks of model harnesses, and the guard would otherwise refuse the validation run.

The folder a committed entry’s target resolves to: where its pull landed (DatasetEntry.destination).

Nothing is fetched here: gym dataset validate <entry> and gym eval run <entry> read what gym dataset pull <entry> left, and the manifest check that follows refuses a folder that no longer matches that pull. For an entry that declares a source, the folder must hold that pull’s record: task folders with no manifest.toml, or with one that records another source (local, or the other hub fetcher’s) or a fetch of another reference than the entry declares now (its source changed since the pull), are refused with the pull to run, never run as a local dataset under the entry’s name, since nothing then ties them to the declared reference (the pull of a package-store entry keeps and records the folders that still match the store, and re-records the reference when only that changed; the registry fetch says in its own words what to move aside). When the pull lands elsewhere than the entry’s folder (NEMO_GYM_DATASETS_DIR, or an entry committed outside the working root) the copy of the entry’s Gym layer beside the tasks is brought up to date, or refused, exactly as the pull would (prepare_destination), so the two commands never disagree about one folder. A folder without a pulled task names the pull to run (or, for an entry that declares no source, says the task folder path is the target). Tasks are looked for as the loader finds them (iter_task_folders: nested under group folders too), so a layout the folder’s path prepares, the entry’s name prepares as well.

Fetch (for hub references) and load the tasks, then write the rows file.

A task folder that does not load is skipped with a warning (and listed in PreparedTaskset.skipped) so the rest of the dataset still prepares; so is a task whose image fails to build or whose Compose file does not resolve, with the build’s or the resolver’s reason. A dataset that ships task_data.py is refused as a whole, with the task named, when a task’s task_data.json does not satisfy its TaskData or when the module does not import (nemo_gym.data.tasks.harbor.task_data); both are checked before any image is recorded or built, and the loaded tasks are checked again before a row is written.

Tasks matching an exclude glob, or not matching an only glob, are left out of the rows and listed on stdout. Dockerfiles outside pull and overlay mode are built once and recorded before the tasks load, as build (the builds: configuration block) says; the default builds locally with docker, and pushes with the first tool on PATH when a repository is named. Compose service fields are checked against sandbox_provider and gpu_sandbox_provider, the provider names of the sandbox: and GPU blocks, which decide the tmpfs, ulimits, sysctls and security_opt each service’s provider can apply, and against compose_platform, the deployment’s (see nemo_gym.data.tasks.harbor.image_configs.check_task_compose_fields). Compose volumes are checked against volume_sources, the deployment’s mapping of external volumes and foreign bind sources to shared storage (see nemo_gym.sandbox.config); a task that declares GPUs is checked against gpu_volume_sources when given, the mapping of the block the server runs such tasks on; shared_storage and gpu_shared_storage say whether each block’s provider has the shared storage named volumes and shared logs binds need, so a task that needs it is left out at prepare, naming the setting, when the block has none (None: unknown, decided at seed). compose_overlay is the job-level Compose overlay (the server’s compose_overlay block): its files are merged onto every task’s Compose document, a plain image task included, and its images recorded with the tasks’. rows_name names the rows file inside the taskset’s output folder: a run scopes it to its agent, so two agents prepared with different selections on one taskset keep their own rows. registry_credentials (registry host or image prefix to username and password, nemo_gym.cli.eval.registry_credentials_from_overrides) are what the base images and the Compose images are read from their registries with; without them a private image stops the preparation with the registry’s refusal. A template target (is_template_folder) is rendered into <render_root>/<taskset>/rendered/ first, output_root when render_root is not given. The rendered folder is the dataset root (its manifest.toml, compose-images.json and built-images.json live in it), so every command that prepares one template must render it into one place: gym dataset validate keeps its rows and results under results/harbor/validation/ but renders where gym eval run does, and an image built for the template under one command is reused, not rebuilt and pushed, under the other.

Before any image is recorded, built or pushed, manifest.toml beside the task folders is reconciled with their digests (check_manifest): a fetched dataset whose folders changed is refused, a local folder gets a manifest. The digests hashed for that check are reused for the tasks that do not load; the loader’s own folder digests are checked against them. Selection does not enter the dataset digest; it covers every task folder, so two runs of different subsets still name one dataset. A task folder that cannot be hashed (a file under it escapes it through a link, or cannot be read) is recorded in the manifest by name without a digest and does not enter the dataset digest; the loader refuses it, so it is skipped like any task that does not load, and selection applies to it as to any other task.

agent is the harness the run selected (its implementation name, what --agent resolves to). It is checked against the dataset’s [gym] allowed_agents as soon as dataset.toml is at hand, after a hub fetch and before any manifest check, image record, build or server start; a harness the dataset does not admit is refused with both named. None skips the check (the oracle’s validation run, which plays the reference solution and not a model), and so does allow_unsupported_pairing, the run’s pairing override (--allow-unsupported-pairing or NEMO_GYM_ALLOW_UNSUPPORTED_PAIRING, nemo_gym.cli.eval.pairing_override_from_overrides): the collector’s guard waives the same list at dispatch under it, so an explicit experimental run proceeds without editing the dataset’s declaration. An example dataset ([gym] type) is refused at the same point whatever the harness, before anything is written: that type marks committed smoke-test samples, which neither gym eval run nor gym dataset validate runs.

Fetch target (a harbor: reference or a committed entry’s name) and report where it landed.

The destination comes from plan_pull: the entry that declares the reference as its source (its own folder, or <datasets dir>/<entry>/ with its dataset.toml placed first when NEMO_GYM_DATASETS_DIR or an entry committed outside the working root moved the destination, destination), else the hub’s own folder under the datasets dir. The fetch is the one every gym eval run harbor:... makes, so a stale, edited or foreign folder is refused in the same words (a refusal naming a committed entry’s own folder says what to delete there, with_entry_hint); nothing is prepared, built or run. The manifest left after the fetch gives the task count and the dataset digest; its Gym-layer digests are first brought up to date with the folder (refresh_gym_layer), since a fetch that downloads nothing rewrites nothing and would otherwise report the digest of a dataset.toml that changed since the last pull, which the next prepare would then rewrite. unchanged compares the manifest on disk before anything was touched with the one reported; a registry reference whose version the registry picks is resolved first (resolved_folder) so that folder is known. When the Gym layer’s digests moved while the task record stayed (the entry’s dataset.toml changed, or one written by hand into a hub folder), the rewrite is announced in the manifest check’s words whichever step applied it: the refresh above, or the fetch itself, which rewrites the manifest without a word when every folder is in place (the package store’s does). A hub folder the reference landed in before an entry declared it (terminal-bench-4.0.0/ beside terminal_bench_4/) is left alone and named in the report, since nothing reads it any more. refresh_registry downloads the registry again instead of reading .harbor/registry.json; it is prepare_target’s pass-through, and deleting that cache file does the same for the command.

Entry point for gym dataset pull <target>.

Refuse a harness the dataset’s [gym] allowed_agents does not admit, naming the harness and the list.

Raised before any fetch result is built on, any image recorded or built, or any server or sandbox started.

Refuse a dataset whose [gym] type is example: that type marks committed smoke-test samples, which no split runs.

prepare_target raises it as soon as dataset.toml is read, before any manifest check, image record, build or rows file; build_run raises it again as a backstop for a hand-built PreparedTaskset.

Map --agent NAME[/FLAVOR] to responses_api_agents/<NAME>/configs/<FLAVOR>.yaml.

NAME may omit a trailing _agent or _sandboxed_agent (simple selects simple_agent). A short name that matches two harnesses, such as hermes (hermes_agent and hermes_sandboxed_agent) or terminus_2, is refused with both full names, so a run never lands on the wrong harness by default.

Every distinct image a run of tasks starts, sorted: each task’s image, the image of a separate verifier ([verifier.environment].docker_image), the image of each step’s separate verifier (HarborTurn.verifier_image) and every Compose service image (the job-level overlay’s services included), the last as the recorded configuration pins it (compose-images.json, what resolve_compose hands the sandbox), else as written when it has no record.

Whether the harness behind impl_name calls the model server from inside the task sandbox.

The harness says so itself: calls_model_from_sandbox = True on the agent class its entrypoint runs, read from the entrypoint source (no agent module is imported here), or sandbox_model_base_url in its config.

Split tasks into the ones to run and (task_id, reason) pairs left out.

Why --only-tasks/--exclude-tasks leave task_id out, or None when they keep it.

The fetchers’ and the manifest check’s refusals name the dataset folder and say to delete it, or move it aside, and run again. When that folder is a committed entry’s own (not the hub’s folder, nor the entry’s pull elsewhere than its folder, under a moved NEMO_GYM_DATASETS_DIR or for an entry committed outside the working root, which hold nothing git tracks), the same refusal says what to delete there: the pulled task folders and manifest.toml, not the README, YAML files and dataset.toml the entry commits, and that another version is pulled by changing source in that dataset.toml. The manifest check’s refusal of an interrupted fetch (run the fetch again) names the pull to run again instead, wherever the folder is: the fetch finishes it, and deleting what it landed would throw the partial fetch away. Returns exc itself for every other folder, and when the message does not name the folder.