Task Folder Format

Field names and defaults are Harbor’s (task.toml schema 1.4). To write your first task, start with Author Task Folders. The contract for tests/ is on Task Folder Verifiers.

Task folder layout

Path	Required	What it holds
`task.toml`	Yes	The task’s settings.
`instruction.md`	Single-step tasks	The agent’s prompt. Leading canary lines (an HTML comment or `#` line containing `canary`) are dropped.
`environment/`	No	A `Dockerfile` or `docker-compose.yaml`, plus their files. With a prebuilt `docker_image` and neither file, its files go to the working directory.
`tests/`	Single-step tasks, unless the dataset has `verifier.py`	`test.sh` or `verifier.py`, plus fixtures. With both, `test.sh` grades. A multi-step task may keep its tests in `steps/<name>/tests/` instead.
`solution/`	No	`solve.sh` or `answer.md`. Only the oracle of `gym dataset validate` sees it.
`task_data.json`	No	The task’s own fields. See Typed task fields.
`steps/`	Multi-step tasks	One folder per step.

The dataset root may hold dataset.toml, task_data.py (the model that types task fields) and verifier.py (a verifier for every single-step task without tests/). Do not edit the records Gym writes there at prepare.

The folder name is the task ID, unique across the dataset. Task folders may sit up to six levels down. Hidden folders are skipped, and a task folder must not be a symbolic link.

gym eval run and gym dataset validate prepare the dataset, stamping each row with a digest of its task folder, dataset settings and dataset verifier.py. The harbor Environment Server refuses a row whose digest no longer matches, so prepare again after an edit. See Digests.

Task settings

These tables list every task.toml key Gym acts on. Gym reads other keys, such as schema_version, [metadata] and source, without acting on them, except to refuse unsupported settings.

Top level

Key	Default	What it does
`[task]`	none	Harbor keys a dataset on its `name` (`org/name`); Gym uses the folder name.
`artifacts`	`[]`	Paths copied out of the agent’s sandbox for a separate verifier, and after each step of a multi-step task. See Shared and separate verification.
`[[steps]]`	none	Makes the task multi-step. See Multi-Step Task Folders.
`multi_step_reward_strategy`	`"mean"`	`mean` or `final`.

Like Harbor, Gym ignores unknown keys (logging each one) and reads the legacy spellings version, memory, storage and allow_internet, which it never writes back.

[environment]

Key	Default	What it does
`docker_image`	none	A prebuilt image. Without it, Gym uses `environment/Dockerfile`; with neither, every episode fails at seed, even with a Compose file. Beside it, a pull-mode Dockerfile’s settings apply and other Dockerfiles are ignored, unless `builds.force` builds the Dockerfile. In a single-sandbox task, the image’s `ENTRYPOINT` and `HEALTHCHECK` do not run, unless a pull-mode `environment/Dockerfile` sits beside it; then those of its `FROM` image run. In a Compose task they run.
`build_timeout_sec`	`600`	The budget all overlay-mode `RUN` lines share. Builds at prepare have no time limit.
`cpus`, `memory_mb`, `storage_mb`	none	Resource limits, sent with `storage_mb` rounded up to whole GiB. Providers differ in what they enforce (Docker ignores `storage_mb`), and the server’s `sandbox_resources_override` replaces any it names.
`gpus`, `gpu_types`	none	GPU count and allowed types. Only the first type is sent, and only when the server sets `request_gpu_type: true`. GPU tasks run on the server’s `gpu_sandbox_provider` block when set.
`env`	`{}`	Variables for the agent’s sandbox. A whole value `${NAME}` or `${NAME:-default}` is read from the server’s host, and an unset `${NAME}` fails the episode at seed.
`workdir`	none	The working directory. Unset, the Dockerfile’s final `WORKDIR`, else the image’s.
`healthcheck`	none	A `command` polled at seed. After `start_period_sec` (`0`), `retries` (`3`) consecutive failures `interval_sec` (`5`) apart fail the seed (retryable). `timeout_sec` (`30`) bounds each attempt; `start_interval_sec` (`5`) paces the start period.
`mcp_servers`	`[]`	Tool servers for the task. See A task with its own MCP tool.
`skills_dir`	none	A skills folder for harnesses that read one; a relative path resolves in the working directory. When the sandbox’s folder is missing or empty and the task ships `environment/skills/`, the server uploads it there at seed.

A task whose environment/ holds docker-compose.yaml runs as a group of sandboxes, one per service, on OpenSandbox only; see Multi-Container Tasks.

[agent]

timeout_sec (default 28800) is the agent’s budget. user is the user the agent runs as; unset, the Dockerfile’s final USER, else the image’s.

A numeric user is looked up by name in the image first. A bare numeric USER with no passwd entry runs as that integer user ID on Docker and on OpenSandbox’s plain exec; other providers, and OpenSandbox’s PTY and bash sessions, need a named user. A uid:gid pair runs only on Docker.

[verifier]

Key	Default	What it does
`timeout_sec`	`600`	Bound on `test.sh`. A step uses `[steps.verifier].timeout_sec` (default `600`) instead.
`user`	none	The user `test.sh` runs as. Unset, the user the verifier’s image starts with, never `[agent].user`. In the task’s own image, root when `[agent].user` is set and Gym has no record of the image’s user.
`env`	`{}`	The variables `test.sh` receives, templated like `[environment].env` but with the server’s `secrets` checked first.
`environment_mode`	`shared`	`shared` runs `test.sh` in the agent’s sandbox, `separate` in a fresh one.
`environment`	none	A `[verifier.environment]` table with the keys of `[environment]`. Selects separate mode.
`collect`	`[]`	`[[verifier.collect]]` hooks, run before `test.sh` and any artifact collection: `command`, `service` (`main`), `timeout_sec` (`60`) and `user` (unset: the service’s default, in `main` the task’s agent user). A hook that fails, times out or names a missing service only logs a warning; one the sandbox cannot run masks the sample.
`network_mode`, `allowed_hosts`	`public`, none	`no-network`, or `allowlist` with `allowed_hosts`. Needs OpenSandbox and separate mode, on every step of a multi-step task. An allowlist fails at seed if the deployment’s `network_policy` denies by default.

[solution]

env holds variables for the oracle’s solve.sh, resolved on the server’s host.

Dataset settings

dataset.toml holds Harbor’s [dataset] table, which Gym leaves to Harbor, and Gym’s [gym] table. Deployment settings, such as the sandbox provider, never go here. The [gym] tables below are complete: Gym refuses any other key in [gym], [gym.defaults] or [gym.tasks."<id>"].

[dataset]
name = "local/my-dataset"   # Harbor requires org/name
version = "0.1.0"
[gym]
license = "Apache 2.0"
allowed_agents = ["terminus_2_sandboxed_agent", "hermes_agent"]
[gym.defaults]
num_repeats = 3
[gym.tasks."hello".environment]
memory_mb = 4096

[gym]

Key	Default	What it does
`type`	`validation` with a license, else `benchmark`	The split the rows enter: `train`, `validation` or `benchmark`. Gym refuses to run `example`, which marks smoke-test samples.
`license`	none	One of Gym’s dataset license values. Required for `train` and `validation`.
`allowed_agents`	any	Harnesses that may run the dataset, by full folder name under `responses_api_agents/`, such as `hermes_agent`. The oracle is always admitted. `--allow-unsupported-pairing` skips the check.
`domain`	none	One of Gym’s domain values, such as `coding`.
`verified`	`false`	Whether the rewards were baselined and reviewed.
`source`	none	The hub reference a committed entry under `environments/harbor/` is pulled from.

None of these keys enters a task digest.

[gym.defaults]

Key	Default	What it does
`num_repeats`	`1`	Rollouts per task for `gym eval run`. `+num_repeats=<n>` replaces it.
`timeout_multiplier`	`1.0`	Scales every `[agent]` and `[verifier]` `timeout_sec`, including a step’s own. Collect hooks, health checks and build timeouts do not scale.
`resource_multiplier`	`1.0`	Scales `cpus`, `memory_mb` and `storage_mb` of `[environment]` and `[verifier.environment]`, rounded up. GPUs, `[steps.verifier.environment]` and Compose sidecars do not scale.

[gym.tasks.”task-id”]

Per-task overrides, keyed by folder name and applied before the multipliers. environment and verifier take any key of the task’s table: env merges over the task’s own, verifier.environment merges field by field like [environment] (or, on a task without one, creates it and so selects separate mode), and every other key replaces it. verifier also takes the Gym-only inputs list (see Sidecar inputs). artifacts adds entries after the task’s own. Once dataset.toml sets a multiplier other than 1.0 or any override, every task’s digest includes the defaults and its own (possibly empty) override, so adding the first such key, or removing the last, changes every digest.

harbor add, harbor remove and harbor sync rewrite dataset.toml without the [gym] table. Keep the dataset in git and restore the table afterwards.

Dockerfile modes

Prepare puts each distinct environment/Dockerfile in one of three modes.

Mode	Dockerfile	How it runs
Pull	`FROM` plus `WORKDIR`, `ENV`, `USER` and `LABEL`	The base image starts, pinned to its digest at prepare, with the Dockerfile’s settings applied.
Overlay	The same plus `RUN` lines	The base image starts and each `RUN` line runs at seed. Nothing is built or pushed. A failing line fails the seed with its output.
Build	Anything else, such as `COPY`, `ARG`, multi-stage builds, `RUN --mount`, a heredoc `RUN`, or a non-empty `ENTRYPOINT` or `CMD`	Gym builds the image once from `environment/` and runs it like a pulled image.

Pull and overlay mode read the base image’s configuration from its registry at prepare. A built image is named hb__<hash> after Harbor’s content hash of environment/, so identical folders share one image. A task whose build fails or is turned off is listed in skipped.json. A base image the registry cannot resolve, or a needed build with no usable build tool, stops the whole preparation. To configure builds, see Harbor image builds.

Compatibility with Harbor

Task folders use the Harbor task format. Gym runs them with its own harnesses and sandbox providers, so Harbor’s agents and trial runtime are not used. See Harbor Environment Server.

Supported features

Single-step and multi-step tasks.
Prebuilt images and every Dockerfile mode.
Multi-container tasks with Compose. See Multi-Container Tasks.
Shared and separate verification, with reward.txt or reward.json.
Collect hooks and artifacts.
Task MCP servers and skills, with a harness that supports them. See A task with its own MCP tool.
[environment.healthcheck], and the image’s ENTRYPOINT and HEALTHCHECK when its configuration is on record.

Additional features

The [gym] table in dataset.toml, for dataset settings such as the split, license and per-task overrides.
Python verifiers (tests/verifier.py) and reference answers in solution/answer.md.
Typed task fields and parameterized tasks.
Sidecar inputs for the verifier, with [verifier].inputs.
Protection against a planted reward: before a shared verifier runs, Gym clears /tests and /logs/verifier, apart from the sidecar inputs. Ship verifier helpers in the task’s tests/ folder. See How the verifier runs.

gym dataset publish --harbor writes a copy that Harbor can run. The copy leaves out the [gym] table, and publishing refuses Python verifiers that read response or grade in separate mode, and tasks that share a [task].name.

Current limitations

Windows and TPU tasks are not supported.
Network restrictions apply only to a verifier in separate mode, on OpenSandbox.
Compose tasks run only on OpenSandbox.
A task that seeds the agent with a transcript (trajectory.json, or steps/<first step>/trajectory.json in a multi-step task) runs only with the oracle.

What the loader refuses

The loader refuses a task with the reason in its message. gym eval run lists refused tasks in skipped.json and stops unless you pass --skip-unloadable. gym dataset validate lists them as skipped: <reason> and exits 1. When no task loads at all, it prints the error instead of a table and exits 2.

Category	Causes
Missing files	No `instruction.md` (single-step), no `tests/` folder (single-step, without a dataset `verifier.py`), or a step with no `steps/<name>/` folder, `instruction.md` there, or `test.sh` in its own or the task’s `tests/`. A folder without `task.toml` is never refused: Gym searches it for task folders.
Missing images	A build-mode Dockerfile, base image or `tests/Dockerfile` with no record from prepare, or a separate verifier with no image.
Unsupported settings	Windows, TPU, network restrictions outside separate-mode `[verifier]`, or a shared verifier with its own environment.
Invalid values	A malformed `task.toml` or override, a bad `inputs` entry, an MCP server without an endpoint, or an artifact whose source is `/` or under `/tests` or `/logs/verifier`, whose path has a `..` part, or whose `destination` is absolute.
Dataset settings	`type = "example"`, a harness outside `allowed_agents` (without `--allow-unsupported-pairing`), or a `[gym.tasks]` table naming no task. These stop the whole command.

A tests/ folder with neither test.sh nor verifier.py loads. Unless a separate verifier’s image ships /tests/test.sh, it scores 0 with harbor:missing_reward. For fixes, see Task Folders.