Task Folder Files and Storage

A task-folder run writes to the dataset folder (pulled task folders, manifest.toml and the image records) and to the run’s results folder. Builds also fill the build tool’s image store, and the server copies each task it runs into the system temp folder until it shuts down. This page lists what each file is for and what you can delete.

<taskset> is the name of the target’s folder: the dataset folder (for an entry or harbor: reference, the folder it was pulled into), or the task folder for a single task. results/harbor/ is relative to the directory you run gym from.

Pulled datasets

Pulled datasets live in the datasets folder: environments/harbor/ in the Gym checkout, or under the working directory for a package install. Set NEMO_GYM_DATASETS_DIR to move it. Earlier versions pulled into datasets/ under the directory you ran gym from; a checkout that still holds datasets there refetches them at the new default, so move each one with mv datasets/<dataset> environments/harbor/<dataset> or keep the old folder by setting NEMO_GYM_DATASETS_DIR to its absolute path. A dataset run by a committed entry’s name is fetched again either way, into a folder named after the entry. Each fetched dataset is one folder there:

A committed entry under environments/harbor/<entry>/ keeps a README, YAML overlays and a dataset.toml; git ignores the task folders a pull adds beside them. While NEMO_GYM_DATASETS_DIR is set, or when an entry is installed elsewhere (such as under site-packages) or committed outside the checkout, the entry is pulled into <entry>/ under the datasets folder with copies of its Gym files, recorded in a hidden .gym-layer.json. Edit the entry’s own files, not the copies.

manifest.toml beside the task folders holds derived facts only. Equal manifests mean identical task folders and dataset files, but not identical images, which the image records pin. It holds:

• [dataset]: source (harbor-registry, harbor-package-store, git or local), ref (the fetched reference), digest, dataset_toml_digest, and dataset_files_digest when task_data.py or verifier.py exist.
• [tasks."<id>"]: the task’s folder digest and its upstream pin.

Prepare writes a manifest.toml with source = "local" for a dataset folder you author, unless git tracks the folder. A single task folder gets none; a dataset folder inside a fetched one is refused. A later fetch downloads only the task folders missing from disk, even ones the manifest records. It keeps the folders whose digests still match. Registry references (harbor:<dataset>[@<version>]) resolve through Harbor’s registry, cached at .harbor/registry.json in the datasets folder. Nothing refreshes it, and it picks the version of a reference without one, so move it aside to see datasets and versions added since.

Image records

Prepare writes two record files at the dataset root, beside the task folders:

A record found on disk is reused without asking the registry. Each built image is recorded once it is built (and pushed, with a repository), so an interrupted prepare resumes at the next build. Both files are replaced atomically under a lock file beside them (compose-images.json.lock, built-images.json.lock), so concurrent commands keep each other’s entries.

Run outputs

<agent> is the instance name in the harness’s config. The rollouts stem follows --output: --output results/harbor/x/full.jsonl writes full_run_summary.json, full_failures.jsonl and so on. The health check’s reports keep their names there, and the rows, run config, skipped.json, images.json and verifier/ stay in results/harbor/<taskset>/.

The run summary

rollouts_run_summary.json explains a results folder on its own. It holds:

• command, target, taskset, agent and dataset (folder, source and digests).
• task_source: the dataset label the rows carry.
• files: every file of the run by relative path. A file this run did not write is null; one an earlier run left under the same name is listed under stale_files.
• counts: selected, results, masked, failed and missing.
• tasks: per task and repeat, its status (result, failed or missing), reward, mask_sample, failure_kind and the input its row carried. A result adds verifier (mode, run time in seconds, network, return code, parsed rewards), response_metadata (with the oracle’s status) and, for a multi-step task, per-step turns and stopped_after_turn. A failure adds failure (class, stage, message, attempt). Both list the relative paths of any verifier logs. A path missing on disk is listed under missing_paths.
• skipped: the tasks skipped.json lists. unvalidated: the tasks gym dataset validate left out because they have no solution/.

Paths other than dataset.folder and summary_dir are relative to the summary’s folder, so a results folder can move as a whole. A command writes the summary once it starts its servers, whatever happens next. So does a gym dataset validate in which no task has a solution/. A command refused earlier, while it prepares the tasks or on a --resume that no longer matches them, writes none.

Verifier log folders

Each verification gets a fresh verifier/session-<id>/ folder; the row’s verifier_logs_dir is its absolute path. It holds what the sandbox’s /logs/verifier held, except that the server writes trajectory.json, the entrypoint logs and artifacts/ itself:

• reward.txt or reward.json.
• test-stdout.txt: everything test.sh printed.
• trajectory.json: the transcript the verifier read, in the Agent Trajectory Interchange Format (ATIF).
• entrypoint-stdout.txt: the image entrypoint’s output, when it ran.
• artifacts/: the collected artifacts, except for a single-step task verified in the agent’s sandbox.

Digests

The harbor Environment Server refuses a row whose task folder no longer matches its digest, so run the command again after editing a task. Every rollouts.jsonl row carries both digests, and the run config records the dataset digest under tasksets.<name>.dataset_digest.

What a rerun reuses

What is safe to delete

• A results folder, once you no longer need its results. For a parameterized dataset, its rendered/ folder is the dataset folder and holds manifest.toml and the image records. Deleting it rebuilds every built image on the next run.
• A pulled dataset folder, for a fresh fetch. The image records live there too, so every task that needs a build is built again. In a committed entry’s folder, delete only the task folders and manifest.toml.
• .harbor/registry.json, to refresh the registry.
• One entry of built-images.json, to force that image to rebuild. A changed Dockerfile or context gets a new hash and rebuilds on its own.
• One entry of compose-images.json, keyed by the image reference as written, to pick up a tag that moved, such as a republished FROM myorg/env:latest. The next prepare reads that image again. Moving the whole file aside reads every image again and also rebuilds every built image.
• The buildah storage folder (/raid/scratch/<user>/nemo-gym-buildah, or nemo-gym-buildah in the system temp folder), when no build runs.

Otherwise keep compose-images.json and built-images.json with the task folders while you still run the dataset.