> This page is for version Main (default). > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. # Task Folder Format > The files of a task folder and a dataset of task folders, the task.toml and dataset.toml fields Gym reads, Dockerfile modes, Harbor compatibility and what the loader refuses. Field names and defaults are Harbor's (`task.toml` schema 1.4). To write your first task, start with [Author Task Folders](/nemo/gym/main/data/task-folders/harbor-tasks). The contract for `tests/` is on [Task Folder Verifiers](/nemo/gym/main/data/task-folders/task-folder-verifiers). ## Task folder layout | Path | Required | What it holds | | ---------------- | ------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `task.toml` | Yes | The task's settings. | | `instruction.md` | Single-step tasks | The agent's prompt. Leading canary lines (an HTML comment or `#` line containing `canary`) are dropped. | | `environment/` | No | A `Dockerfile` or `docker-compose.yaml`, plus their files. With a prebuilt `docker_image` and neither file, its files go to the working directory. | | `tests/` | Single-step tasks, unless the dataset has `verifier.py` | `test.sh` or `verifier.py`, plus fixtures. With both, `test.sh` grades. A multi-step task may keep its tests in `steps//tests/` instead. | | `solution/` | No | `solve.sh` or `answer.md`. Only the oracle of `gym dataset validate` sees it. | | `task_data.json` | No | The task's own fields. See [Typed task fields](/nemo/gym/main/data/task-folders/task-folder-patterns#typed-task-fields). | | `steps/` | Multi-step tasks | One folder per step. | The dataset root may hold `dataset.toml`, `task_data.py` (the model that types task fields) and `verifier.py` (a verifier for every single-step task without `tests/`). Do not edit the records Gym writes there at prepare. The folder name is the task ID, unique across the dataset. Task folders may sit up to six levels down. Hidden folders are skipped, and a task folder must not be a symbolic link. `gym eval run` and `gym dataset validate` prepare the dataset, stamping each row with a digest of its task folder, dataset settings and dataset `verifier.py`. The `harbor` Environment Server refuses a row whose digest no longer matches, so prepare again after an edit. See [Digests](/nemo/gym/main/reference/task-folder-storage#digests). ## Task settings These tables list every `task.toml` key Gym acts on. Gym reads other keys, such as `schema_version`, `[metadata]` and `source`, without acting on them, except to refuse [unsupported settings](#what-the-loader-refuses). ### Top level | Key | Default | What it does | | ---------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `[task]` | none | Harbor keys a dataset on its `name` (`org/name`); Gym uses the folder name. | | `artifacts` | `[]` | Paths copied out of the agent's sandbox for a separate verifier, and after each step of a multi-step task. See [Shared and separate verification](/nemo/gym/main/data/task-folders/task-folder-verifiers#shared-and-separate-verification). | | `[[steps]]` | none | Makes the task multi-step. See [Multi-Step Task Folders](/nemo/gym/main/data/task-folders/multi-step-tasks). | | `multi_step_reward_strategy` | `"mean"` | `mean` or `final`. | Like Harbor, Gym ignores unknown keys (logging each one) and reads the legacy spellings `version`, `memory`, `storage` and `allow_internet`, which it never writes back. ### \[environment] | Key | Default | What it does | | --------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `docker_image` | none | A prebuilt image. Without it, Gym uses `environment/Dockerfile`; with neither, every episode fails at seed, even with a Compose file. Beside it, a pull-mode Dockerfile's settings apply and other Dockerfiles are ignored, unless `builds.force` builds the Dockerfile. In a single-sandbox task, the image's `ENTRYPOINT` and `HEALTHCHECK` do not run, unless a pull-mode `environment/Dockerfile` sits beside it; then those of its `FROM` image run. In a Compose task they run. | | `build_timeout_sec` | `600` | The budget all overlay-mode `RUN` lines share. Builds at prepare have no time limit. | | `cpus`, `memory_mb`, `storage_mb` | none | Resource limits, sent with `storage_mb` rounded up to whole GiB. Providers differ in what they enforce (Docker ignores `storage_mb`), and the server's `sandbox_resources_override` replaces any it names. | | `gpus`, `gpu_types` | none | GPU count and allowed types. Only the first type is sent, and only when the server sets `request_gpu_type: true`. GPU tasks run on the server's `gpu_sandbox_provider` block when set. | | `env` | `{}` | Variables for the agent's sandbox. A whole value `${NAME}` or `${NAME:-default}` is read from the server's host, and an unset `${NAME}` fails the episode at seed. | | `workdir` | none | The working directory. Unset, the Dockerfile's final `WORKDIR`, else the image's. | | `healthcheck` | none | A `command` polled at seed. After `start_period_sec` (`0`), `retries` (`3`) consecutive failures `interval_sec` (`5`) apart fail the seed (retryable). `timeout_sec` (`30`) bounds each attempt; `start_interval_sec` (`5`) paces the start period. | | `mcp_servers` | `[]` | Tool servers for the task. See [A task with its own MCP tool](/nemo/gym/main/data/task-folders/task-folder-patterns#a-task-with-its-own-mcp-tool). | | `skills_dir` | none | A skills folder for harnesses that read one; a relative path resolves in the working directory. When the sandbox's folder is missing or empty and the task ships `environment/skills/`, the server uploads it there at seed. | A task whose `environment/` holds `docker-compose.yaml` runs as a group of sandboxes, one per service, on OpenSandbox only; see [Multi-Container Tasks](/nemo/gym/main/data/task-folders/compose-tasks). ### \[agent] `timeout_sec` (default `28800`) is the agent's budget. `user` is the user the agent runs as; unset, the Dockerfile's final `USER`, else the image's. A numeric user is looked up by name in the image first. A bare numeric `USER` with no passwd entry runs as that integer user ID on Docker and on OpenSandbox's plain exec; other providers, and OpenSandbox's PTY and bash sessions, need a named user. A `uid:gid` pair runs only on Docker. ### \[verifier] | Key | Default | What it does | | ------------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `timeout_sec` | `600` | Bound on `test.sh`. A step uses `[steps.verifier].timeout_sec` (default `600`) instead. | | `user` | none | The user `test.sh` runs as. Unset, the user the verifier's image starts with, never `[agent].user`. In the task's own image, root when `[agent].user` is set and Gym has no record of the image's user. | | `env` | `{}` | The variables `test.sh` receives, templated like `[environment].env` but with the server's `secrets` checked first. | | `environment_mode` | `shared` | `shared` runs `test.sh` in the agent's sandbox, `separate` in a fresh one. | | `environment` | none | A `[verifier.environment]` table with the keys of `[environment]`. Selects separate mode. | | `collect` | `[]` | `[[verifier.collect]]` hooks, run before `test.sh` and any artifact collection: `command`, `service` (`main`), `timeout_sec` (`60`) and `user` (unset: the service's default, in `main` the task's agent user). A hook that fails, times out or names a missing service only logs a warning; one the sandbox cannot run masks the sample. | | `network_mode`, `allowed_hosts` | `public`, none | `no-network`, or `allowlist` with `allowed_hosts`. Needs OpenSandbox and separate mode, on every step of a multi-step task. An allowlist fails at seed if the deployment's `network_policy` denies by default. | ### \[solution] `env` holds variables for the oracle's `solve.sh`, resolved on the server's host. ## Dataset settings `dataset.toml` holds Harbor's `[dataset]` table, which Gym leaves to Harbor, and Gym's `[gym]` table. Deployment settings, such as the sandbox provider, never go here. The `[gym]` tables below are complete: Gym refuses any other key in `[gym]`, `[gym.defaults]` or `[gym.tasks.""]`. ```toml [dataset] name = "local/my-dataset" # Harbor requires org/name version = "0.1.0" [gym] license = "Apache 2.0" allowed_agents = ["terminus_2_sandboxed_agent", "hermes_agent"] [gym.defaults] num_repeats = 3 [gym.tasks."hello".environment] memory_mb = 4096 ``` ### \[gym] | Key | Default | What it does | | ---------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | `validation` with a license, else `benchmark` | The split the rows enter: `train`, `validation` or `benchmark`. Gym refuses to run `example`, which marks smoke-test samples. | | `license` | none | One of Gym's dataset license values. Required for `train` and `validation`. | | `allowed_agents` | any | Harnesses that may run the dataset, by full folder name under `responses_api_agents/`, such as `hermes_agent`. The oracle is always admitted. `--allow-unsupported-pairing` skips the check. | | `domain` | none | One of Gym's domain values, such as `coding`. | | `verified` | `false` | Whether the rewards were baselined and reviewed. | | `source` | none | The hub reference a committed entry under `environments/harbor/` is pulled from. | None of these keys enters a task digest. ### \[gym.defaults] | Key | Default | What it does | | --------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `num_repeats` | `1` | Rollouts per task for `gym eval run`. `+num_repeats=` replaces it. | | `timeout_multiplier` | `1.0` | Scales every `[agent]` and `[verifier]` `timeout_sec`, including a step's own. Collect hooks, health checks and build timeouts do not scale. | | `resource_multiplier` | `1.0` | Scales `cpus`, `memory_mb` and `storage_mb` of `[environment]` and `[verifier.environment]`, rounded up. GPUs, `[steps.verifier.environment]` and Compose sidecars do not scale. | ### \[gym.tasks."task-id"] Per-task overrides, keyed by folder name and applied before the multipliers. `environment` and `verifier` take any key of the task's table: `env` merges over the task's own, `verifier.environment` merges field by field like `[environment]` (or, on a task without one, creates it and so selects separate mode), and every other key replaces it. `verifier` also takes the Gym-only `inputs` list (see [Sidecar inputs](/nemo/gym/main/data/task-folders/task-folder-verifiers#sidecar-inputs)). `artifacts` adds entries after the task's own. Once `dataset.toml` sets a multiplier other than `1.0` or any override, every task's digest includes the defaults and its own (possibly empty) override, so adding the first such key, or removing the last, changes every digest. > **Note** > > `harbor add`, `harbor remove` and `harbor sync` rewrite `dataset.toml` without the `[gym]` table. Keep the > dataset in git and restore the table afterwards. ## Dockerfile modes Prepare puts each distinct `environment/Dockerfile` in one of three modes. | Mode | Dockerfile | How it runs | | ------- | ------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | | Pull | `FROM` plus `WORKDIR`, `ENV`, `USER` and `LABEL` | The base image starts, pinned to its digest at prepare, with the Dockerfile's settings applied. | | Overlay | The same plus `RUN` lines | The base image starts and each `RUN` line runs at seed. Nothing is built or pushed. A failing line fails the seed with its output. | | Build | Anything else, such as `COPY`, `ARG`, multi-stage builds, `RUN --mount`, a heredoc `RUN`, or a non-empty `ENTRYPOINT` or `CMD` | Gym builds the image once from `environment/` and runs it like a pulled image. | Pull and overlay mode read the base image's configuration from its registry at prepare. A built image is named `hb__` after Harbor's content hash of `environment/`, so identical folders share one image. A task whose build fails or is turned off is listed in `skipped.json`. A base image the registry cannot resolve, or a needed build with no usable build tool, stops the whole preparation. To configure builds, see [Harbor image builds](/nemo/gym/main/reference/configuration#harbor-image-builds). ## Compatibility with Harbor Task folders use the Harbor task format. Gym runs them with its own harnesses and sandbox providers, so Harbor's agents and trial runtime are not used. See [Harbor Environment Server](/nemo/gym/main/agent-interactions/harbor). ### Supported features * Single-step and multi-step tasks. * Prebuilt images and every [Dockerfile mode](#dockerfile-modes). * Multi-container tasks with Compose. See [Multi-Container Tasks](/nemo/gym/main/data/task-folders/compose-tasks). * Shared and separate verification, with `reward.txt` or `reward.json`. * Collect hooks and artifacts. * Task MCP servers and skills, with a harness that supports them. See [A task with its own MCP tool](/nemo/gym/main/data/task-folders/task-folder-patterns#a-task-with-its-own-mcp-tool). * `[environment.healthcheck]`, and the image's `ENTRYPOINT` and `HEALTHCHECK` when its configuration is on record. ### Additional features * The `[gym]` table in `dataset.toml`, for dataset settings such as the split, license and per-task overrides. * Python verifiers (`tests/verifier.py`) and reference answers in `solution/answer.md`. * Typed task fields and parameterized tasks. * [Sidecar inputs](/nemo/gym/main/data/task-folders/task-folder-verifiers#sidecar-inputs) for the verifier, with `[verifier].inputs`. * Protection against a planted reward: before a shared verifier runs, Gym clears `/tests` and `/logs/verifier`, apart from the sidecar inputs. Ship verifier helpers in the task's `tests/` folder. See [How the verifier runs](/nemo/gym/main/data/task-folders/task-folder-verifiers#how-the-verifier-runs). `gym dataset publish --harbor` writes a copy that Harbor can run. The copy leaves out the `[gym]` table, and publishing refuses Python verifiers that read `response` or grade in separate mode, and tasks that share a `[task].name`. ### Current limitations * Windows and TPU tasks are not supported. * Network restrictions apply only to a verifier in separate mode, on OpenSandbox. * Compose tasks run only on OpenSandbox. * A task that seeds the agent with a transcript (`trajectory.json`, or `steps//trajectory.json` in a multi-step task) runs only with the oracle. ## What the loader refuses The loader refuses a task with the reason in its message. `gym eval run` lists refused tasks in `skipped.json` and stops unless you pass `--skip-unloadable`. `gym dataset validate` lists them as `skipped: ` and exits 1. When no task loads at all, it prints the error instead of a table and exits 2. | Category | Causes | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Missing files | No `instruction.md` (single-step), no `tests/` folder (single-step, without a dataset `verifier.py`), or a step with no `steps//` folder, `instruction.md` there, or `test.sh` in its own or the task's `tests/`. A folder without `task.toml` is never refused: Gym searches it for task folders. | | Missing images | A build-mode Dockerfile, base image or `tests/Dockerfile` with no record from prepare, or a separate verifier with no image. | | Unsupported settings | Windows, TPU, network restrictions outside separate-mode `[verifier]`, or a shared verifier with its own environment. | | Invalid values | A malformed `task.toml` or override, a bad `inputs` entry, an MCP server without an endpoint, or an artifact whose source is `/` or under `/tests` or `/logs/verifier`, whose path has a `..` part, or whose `destination` is absolute. | | Dataset settings | `type = "example"`, a harness outside `allowed_agents` (without `--allow-unsupported-pairing`), or a `[gym.tasks]` table naming no task. These stop the whole command. | A `tests/` folder with neither `test.sh` nor `verifier.py` loads. Unless a separate verifier's image ships `/tests/test.sh`, it scores 0 with `harbor:missing_reward`. For fixes, see [Task Folders](/nemo/gym/main/troubleshooting/task-folders).