> This page is for version Main (default). > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. # Task Folder Files and Storage > Where task-folder runs write their files, which digests identify a task and a dataset, what a rerun reuses and what you can delete. A task-folder run writes to the dataset folder (pulled task folders, `manifest.toml` and the image records) and to the run's results folder. Builds also fill the build tool's image store, and the server copies each task it runs into the system temp folder until it shuts down. This page lists what each file is for and what you can delete. | Location | What lives there | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | `environments/harbor//` | Pulled task folders and `manifest.toml` | | `/compose-images.json`, `built-images.json` | Image records | | `results/harbor//` | Rows, run config, rollouts, run summary and verifier logs of `gym eval run` | | `results/harbor/validation//` | The same, for `gym dataset validate`, but a parameterized dataset still renders into `results/harbor//rendered/` | `` is the name of the target's folder: the dataset folder (for an entry or `harbor:` reference, the folder it was pulled into), or the task folder for a single task. `results/harbor/` is relative to the directory you run `gym` from. ## Pulled datasets Pulled datasets live in the **datasets folder**: `environments/harbor/` in the Gym checkout, or under the working directory for a package install. Set `NEMO_GYM_DATASETS_DIR` to move it. Earlier versions pulled into `datasets/` under the directory you ran `gym` from; a checkout that still holds datasets there refetches them at the new default, so move each one with `mv datasets/ environments/harbor/` or keep the old folder by setting `NEMO_GYM_DATASETS_DIR` to its absolute path. A dataset run by a committed entry's name is fetched again either way, into a folder named after the entry. Each fetched dataset is one folder there: | Reference | Folder | | ----------------------------------------------------------------- | ----------------------------------------------- | | `harbor:/` (the store's `latest`) | `/` | | `harbor:/@` | `-/` | | `harbor:/@sha256:` | `-/` | | `harbor:[@]` (registry dataset) | `-/` | | A reference that a committed entry declares as its `[gym] source` | The entry's folder, such as `terminal_bench_4/` | A committed entry under `environments/harbor//` keeps a README, YAML overlays and a `dataset.toml`; git ignores the task folders a pull adds beside them. While `NEMO_GYM_DATASETS_DIR` is set, or when an entry is installed elsewhere (such as under `site-packages`) or committed outside the checkout, the entry is pulled into `/` under the datasets folder with copies of its Gym files, recorded in a hidden `.gym-layer.json`. Edit the entry's own files, not the copies. `manifest.toml` beside the task folders holds derived facts only. Equal manifests mean identical task folders and dataset files, but not identical images, which the image records pin. It holds: * `[dataset]`: `source` (`harbor-registry`, `harbor-package-store`, `git` or `local`), `ref` (the fetched reference), `digest`, `dataset_toml_digest`, and `dataset_files_digest` when `task_data.py` or `verifier.py` exist. * `[tasks.""]`: the task's folder digest and its upstream pin. Prepare writes a `manifest.toml` with `source = "local"` for a dataset folder you author, unless git tracks the folder. A single task folder gets none; a dataset folder inside a fetched one is refused. A later fetch downloads only the task folders missing from disk, even ones the manifest records. It keeps the folders whose digests still match. Registry references (`harbor:[@]`) resolve through Harbor's registry, cached at `.harbor/registry.json` in the datasets folder. Nothing refreshes it, and it picks the version of a reference without one, so move it aside to see datasets and versions added since. ## Image records Prepare writes two record files at the dataset root, beside the task folders: | File | What it holds | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `compose-images.json` | The configuration (`Entrypoint`, `Env`, `User`, `WorkingDir` and more) of each image prepare reads or builds. The base image of a pull-mode or overlay-mode Dockerfile and each Compose service image are pinned to a digest; a built image is pinned once pushed. A prebuilt `docker_image` gets no record unless a pull-mode Dockerfile (`FROM` that image) sits beside it or the task runs under Compose. | | `built-images.json` | Every built image, keyed by the hash of its build context: the image per Compose service, the tool, whether it was pushed, and the tasks that use it. | A record found on disk is reused without asking the registry. Each built image is recorded once it is built (and pushed, with a repository), so an interrupted prepare resumes at the next build. Both files are replaced atomically under a lock file beside them (`compose-images.json.lock`, `built-images.json.lock`), so concurrent commands keep each other's entries. ## Run outputs | File under `results/harbor//` | What it is | | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `tasks_.jsonl` | The rows of a run with one harness. Each row carries `_ng_digest`, `_ng_dataset_digest` and `task_source`. | | `run_config_.yaml` | The generated run config: the task mapping, the dataset digests and the generated server instances. | | `skipped.json` | Task ID to reason, for every selected task that did not load or build, or whose Compose file was refused. | | `images.json` | Every distinct image a run of the prepared tasks starts, for pre-pulling. | | `rendered/` | Rendered task folders of a parameterized dataset. | | `rollouts.jsonl` | One result row per episode. | | `rollouts_failures.jsonl`, `rollouts_materialized_inputs.jsonl`, `rollouts_aggregate_metrics.json` | Failed episodes, the inputs `--resume` reads, and aggregate metrics. | | `rollouts_run_summary.json` | The run summary. | | `quality_summary.json`, `rollout_verdicts.jsonl` | The health check's reports. | | `verifier/session-/` | One folder per verification. | `` is the instance name in the harness's config. The `rollouts` stem follows `--output`: `--output results/harbor/x/full.jsonl` writes `full_run_summary.json`, `full_failures.jsonl` and so on. The health check's reports keep their names there, and the rows, run config, `skipped.json`, `images.json` and `verifier/` stay in `results/harbor//`. ## The run summary `rollouts_run_summary.json` explains a results folder on its own. It holds: * `command`, `target`, `taskset`, `agent` and `dataset` (folder, source and digests). * `task_source`: the dataset label the rows carry. * `files`: every file of the run by relative path. A file this run did not write is `null`; one an earlier run left under the same name is listed under `stale_files`. * `counts`: selected, results, masked, failed and missing. * `tasks`: per task and repeat, its `status` (`result`, `failed` or `missing`), `reward`, `mask_sample`, `failure_kind` and the input its row carried. A result adds `verifier` (mode, run time in seconds, network, return code, parsed rewards), `response_metadata` (with the oracle's status) and, for a multi-step task, per-step `turns` and `stopped_after_turn`. A failure adds `failure` (class, stage, message, attempt). Both list the relative paths of any verifier logs. A path missing on disk is listed under `missing_paths`. * `skipped`: the tasks `skipped.json` lists. `unvalidated`: the tasks `gym dataset validate` left out because they have no `solution/`. Paths other than `dataset.folder` and `summary_dir` are relative to the summary's folder, so a results folder can move as a whole. A command writes the summary once it starts its servers, whatever happens next. So does a `gym dataset validate` in which no task has a `solution/`. A command refused earlier, while it prepares the tasks or on a `--resume` that no longer matches them, writes none. ## Verifier log folders Each verification gets a fresh `verifier/session-/` folder; the row's `verifier_logs_dir` is its absolute path. It holds what the sandbox's `/logs/verifier` held, except that the server writes `trajectory.json`, the entrypoint logs and `artifacts/` itself: * `reward.txt` or `reward.json`. * `test-stdout.txt`: everything `test.sh` printed. * `trajectory.json`: the transcript the verifier read, in the Agent Trajectory Interchange Format (ATIF). * `entrypoint-stdout.txt`: the image entrypoint's output, when it ran. * `artifacts/`: the collected artifacts, except for a single-step task verified in the agent's sandbox. ## Digests | Digest | Covers | Changes when | | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Task digest (`_ng_digest`) | `task.toml`, `instruction.md`, `README.md`, `trajectory.json`, `task_data.json` and everything under `environment/`, `tests/`, `solution/` and `steps/`; plus the dataset's `[gym.defaults]` and the task's override when `dataset.toml` sets them, and a dataset-level `verifier.py` that grades the task | A covered file or setting changes. Byte-code caches and editor leftovers do not count, and neither does `num_repeats`. | | Dataset digest (`_ng_dataset_digest`) | The sorted task folder digests, `dataset.toml`, and `task_data.py` and `verifier.py` when present | Any of those changes. | | Context hash (`hb__`) | Harbor's hash of a Dockerfile build context | A regular file's name or bytes change, outside `.git`, `__pycache__` and `.DS_Store`. Other changes, such as a file mode, a link or a build argument, rebuild under the same name. | The `harbor` Environment Server refuses a row whose task folder no longer matches its digest, so run the command again after editing a task. Every `rollouts.jsonl` row carries both digests, and the run config records the dataset digest under `tasksets..dataset_digest`. ## What a rerun reuses | Item | Reused when | | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Pulled task folders | The manifest records them and their digests match. A folder that changed on disk or upstream, such as when the store's `latest` moved, is refused, not fetched over; move it aside to fetch again. | | Image configuration records | Always, once recorded for the run's platform. A pull-mode or overlay-mode task or a Compose service starts from the digest recorded there, so a tag that moves later is not picked up. | | Built images | The context hash and build inputs match, and the record fits the run: a pushed image serves every run, a local one only runs that push nothing. | | Rows and run config | Never. Each run writes them again. | | Rollouts | Only with `--resume`, and only when the task selection and digests match. | ## What is safe to delete * **A results folder**, once you no longer need its results. For a parameterized dataset, its `rendered/` folder is the dataset folder and holds `manifest.toml` and the image records. Deleting it rebuilds every built image on the next run. * **A pulled dataset folder**, for a fresh fetch. The image records live there too, so every task that needs a build is built again. In a committed entry's folder, delete only the task folders and `manifest.toml`. * **`.harbor/registry.json`**, to refresh the registry. * **One entry of `built-images.json`**, to force that image to rebuild. A changed Dockerfile or context gets a new hash and rebuilds on its own. * **One entry of `compose-images.json`**, keyed by the image reference as written, to pick up a tag that moved, such as a republished `FROM myorg/env:latest`. The next prepare reads that image again. Moving the whole file aside reads every image again and also rebuilds every built image. * **The buildah storage folder** (`/raid/scratch//nemo-gym-buildah`, or `nemo-gym-buildah` in the system temp folder), when no build runs. Otherwise keep `compose-images.json` and `built-images.json` with the task folders while you still run the dataset. > **Tip** > > On a cluster, put the datasets folder and results that must outlive the job on shared storage. Keep buildah layers > on node-local disk: they go under `/raid/scratch` when it exists; elsewhere, point `TMPDIR` at local disk. buildah > reads each build context (`environment/`, or a Compose service's `build.context`) in place inside the dataset > folder. Its heredoc support fails on Lustre, so keep a dataset whose Dockerfiles use heredocs on local disk while it > builds.