Task Folder Verifiers

A task folder carries its own verifier in tests/, unless the dataset’s verifier.py grades it. After the agent’s session closes, the harbor Environment Server runs it inside a sandbox and reads the reward it writes. No Resources Server takes part. This page is the contract a verifier must meet. The [verifier] fields are listed in Task settings.

How the verifier runs

In the default shared mode, the server does this in the agent’s sandbox:

1. Runs the task’s [[verifier.collect]] hooks.
2. Empties /tests and /logs/verifier, apart from declared sidecar inputs, so the agent cannot plant a reward, a helper or a test report. Ship helpers in the task’s tests/ folder, not in the image’s /tests.
3. Uploads tests/ to /tests and makes test.sh executable.
4. Writes the agent’s transcript to /logs/agent/trajectory.json in ATIF v1.7, the format Harbor’s verifiers read.
5. Runs bash /tests/test.sh from the working directory under timeout, with its output in /logs/verifier/test-stdout.txt.

Verification needs no Dockerfile of its own: test.sh runs in the task’s own image, the one [environment].docker_image or environment/Dockerfile gives the agent, or in the verifier’s image in separate mode. That image must have bash, timeout, tar, gzip and find. Debian- and Ubuntu-based images include them; Alpine lacks bash. Without them test.sh never starts and the sample is masked, as verifier_error when bash or timeout is missing and as provider_unavailable for the other three.

Reward files

test.sh writes its result to /logs/verifier/reward.txt or /logs/verifier/reward.json. The exit code of test.sh does not affect the score.

• reward.txt holds one number, which is the reward.
• reward.json holds a non-empty JSON object of numbers; booleans count as numbers. A reward key is the score, and the other keys are metrics, kept per row in verifier_rewards but not in aggregate metrics. Without a reward key, every key is a component: the score is their sum, and the result names them in reward_components.
• reward.json wins whenever it exists, even when it is malformed.

Two tasks of one dataset may write different keys. Gym pads no missing key and renames none.

These rows count in the score, and a reward written before a timeout still stands. Other failures, such as a lost sandbox or a reward that could not be copied out, set mask_sample: the rollout stays in rollouts.jsonl but is left out of the score.

Python verifiers

A single-step task may grade with tests/verifier.py instead of tests/test.sh. When tests/test.sh exists it wins, and a verifier.py beside it is an ordinary helper. A verifier.py at the dataset root grades every single-step task that has no tests/ folder; see Shared verifier code.

The module defines verify, which declares the arguments it reads and returns the reward:

from pathlib import Path
from nemo_gym.verifiers.files import read_text
def verify(workspace: Path, task_data: dict) -> float:
    answer = read_text(workspace, "answer.txt")
    return float(answer is not None and answer.strip() == task_data["answer"])

verify can declare any of these four arguments, by name. The runner passes only the ones it declares (every available one to **kwargs), so the example above gets workspace and task_data alone.

verify returns a finite number, or a dict of finite numbers. A reward key is the score and the other keys are metrics. Without reward, the dict must hold exactly one key, which is the score. Unlike in reward.json, several keys are not summed. An async def verify is awaited. A verify that raises, requires an argument the runner does not supply, or returns anything else writes no reward, which scores 0 with harbor:missing_reward and the error in test-stdout.txt.

The helpers use the standard library only:

• nemo_gym.verifiers: final_text, exact_match and numeric_match.
• nemo_gym.verifiers.files: read_text, file_exists and text_file_equals. They read only inside the workspace. read_text drops one trailing newline unless you pass strip_trailing_newline=False.

No verifier code runs on the host. At verification the server uploads a shim over /tests: a test.sh, a runner, a copy of nemo_gym.verifiers, the task’s fields and the agent’s response. Its test.sh runs python3, else python, and needs 3.8 or newer; without one, the sample scores 0 with harbor:missing_reward and is not masked. The shim runs like any test.sh, under the same timeout, user and mode.

In separate mode, a verifier that reads workspace needs the working directory in artifacts. gym dataset publish --harbor refuses a Python verifier that task.toml puts in separate mode, or that declares response, even with a default.

Reference answers

A task graded by its answer can ship solution/answer.md instead of solution/solve.sh. The oracle of gym dataset validate copies the file byte for byte to answer.md in the working directory, as the task’s user, and submits its text as the response. It reads the file with base64, so the image needs that tool. A verifier that reads workspace / "answer.md" grades the same under Gym and under Harbor, where gym dataset publish --harbor turns the file into a solve.sh that writes it.

Sidecar inputs

A Compose sidecar may write a file into /logs/verifier during the rollout for test.sh to read, in shared mode only. The shared verifier empties that folder first, so declare each such file in dataset.toml:

[gym.tasks."<id>".verifier]
inputs = ["actions.jsonl"]

Each entry is a plain top-level file name: no path separators, no *, ? or [, and no grader output such as reward.json. A declared file survives only as a regular file. In a multi-step task it survives the reset before every turn. Harbor has no such field. See Multi-Container Tasks.

Shared and separate verification

Verification can be shared or separate, meaning the verifier runs in the agent’s sandbox or in a separate sandbox.

In shared mode (environment_mode = "shared", the default), test.sh runs in the agent’s sandbox. Processes the agent started keep running beside it. Use shared verification when the verifier needs to inspect the actual environment the agent changed. For example, Terminal-Bench’s nginx-request-logging tests whether Nginx is responding at localhost:8080. Copying its configuration files into a fresh container sandbox would not prove that the agent successfully started the Nginx server and restarting it during verificatio nwould change what is actaully being tested.

In separate mode, set by environment_mode = "separate" or a [verifier.environment] table, a fresh sandbox is created for the verifier. Use separate verification when success can be determined from exported artifacts and there are concerns about the agent’s environment affecting verification. Imagine an agent is asked in a SWE task to fix a bug in a Python project. While working, the agent changes a Python file and upgrades a Python package. The package upgrade is not something we want to impact verification, but this change may affect verification.

When using separate verification, the server runs the collect hooks, copies /logs/artifacts and the task’s artifacts out of the agent’s sandbox, stops it, and runs test.sh in a fresh sandbox that takes its env and workdir from [verifier.environment], or from [environment] without one. That sandbox receives the artifacts at their own paths, the task’s tests/ (unless its image ships /tests/test.sh) and the transcript. A verifier that reads the agent’s files needs them in artifacts, for example artifacts = ["/workspace"].

An artifacts entry is a path, or a table with source, destination (where the host copy lands in the verifier logs), service (a Compose service, main by default) and exclude patterns. A source of /, or under /tests or /logs/verifier, is refused.

The separate verifier’s image:

A separate-mode task with none of these is refused. When Gym has the verifier image’s configuration on record, its ENTRYPOINT and HEALTHCHECK run before test.sh. A tests/docker-compose.yaml may hold only services, name, version and x- keys at the top, and only service main. main may set image, build and pull_policy; for a prebuilt verifier image, only a pull_policy other than build.

[verifier].network_mode (no-network, or allowlist with allowed_hosts) restricts the separate verifier’s sandbox; a restriction on any other table is refused at load. The episode fails at seed without OpenSandbox, or for an allowlist under a deny-by-default sandbox_provider_options.network_policy. Under shared verification the task is refused at load. The server option separate_verifier_for_artifacts: true verifies every task that declares artifacts separately; see Configure the server.

Environment variables and secrets

A whole value ${VAR} or ${VAR:-default} in [environment].env, [verifier].env or [verifier.environment].env is read from the server’s host environment when used. A missing required value fails the episode before any sandbox starts.

test.sh sees its sandbox’s environment ([environment].env in shared mode) plus [verifier].env. Judge API keys and other secrets go in the server’s secrets block and fill ${NAME} templates in [verifier].env and [steps.verifier].env only, and win over the host environment. A sandbox with a CPU limit also gets CPU-count hints such as OMP_NUM_THREADS; a variable the task sets wins, as does one the image sets when Gym has its configuration on record.

Users

test.sh runs as [verifier].user. Unset, it runs as the user the verifier’s image starts with, never [agent].user; when Gym has no record of that user and [agent].user is set, it runs as root. A numeric image USER runs test.sh as that uid. A [verifier].user is used as written, so write a numeric one as a TOML integer, not a string. A user with a group (name:group) works on the Docker provider only and is refused at seed elsewhere.

Image entrypoint

When Gym has the image’s configuration on record (a built image, or a pull-mode or overlay-mode Dockerfile), the seed starts the image’s ENTRYPOINT with sh -c "sleep infinity" in place of its CMD, beside the sandbox’s keep-alive process, then waits for its HEALTHCHECK and, as at every seed, [environment.healthcheck]. An entrypoint that exits non-zero fails the seed, or masks the sample with harbor:entrypoint_failed if it dies during the rollout. Its output is kept as entrypoint-stdout.txt (a separate verifier image’s as verifier-entrypoint-stdout.txt) with the verifier logs. ENTRYPOINT [] turns it off. Outside a Compose task, a prebuilt docker_image has no record, so its ENTRYPOINT and HEALTHCHECK do not run, unless a pull-mode environment/Dockerfile (FROM that image) sits beside it. A Compose task’s main runs both from the record prepare makes of every service image.

Scoring patterns

What a verifier checks, from file state to a model judge, is the same question for task folders and Resources Servers. See Build Verifiers for scoring patterns.