nemo_gym.data.tasks.harbor.python_verifier
nemo_gym.data.tasks.harbor.python_verifier
Python verifiers: which module grades a task, and the test.sh shim that runs it inside the sandbox.
A task may grade with tests/verifier.py instead of tests/test.sh; a dataset may ship one verifier.py
beside dataset.toml that grades every task without a tests/ folder. tests/test.sh always wins when it is
present: many hub datasets ship a tests/verifier.py that their test.sh calls as a helper, and such a task is a
plain test.sh task to Gym (the module is uploaded with tests/ like any other file). The module’s verify
function (see nemo_gym.verifiers) never runs on the host. The harbor environment server uploads a shim folder
over /tests after the task’s own tests/: a test.sh that finds a Python (3.8 or newer, PYTHON_FLOOR;
an older one is refused with the version named, exit 127, no reward), the runner, a vendored copy
of nemo_gym.verifiers, the task’s fields and, at run time, the agent’s final response, and runs the
verifier with the workspace path. gym dataset publish --harbor writes the same shim into each expanded
task’s tests/ (without the response, which Harbor cannot provide): the task’s own tests/verifier.py stays
where it is and the shim runs it there, a dataset-level module travels under tests/.ng/verifier.py, so the
published folder grades by tests/test.sh the same way under Harbor in shared verifier mode, and loads back into
Gym as a plain test.sh task. Publish refuses a Python
verifier in separate mode: Harbor uploads nothing into a separate verifier container, so the shim could not reach it.
The shim clears any reward.json or reward.txt already in /logs/verifier before the runner starts, and the
runner does the same before verify runs. Harbor does not: in shared mode its trial bind-mounts the verifier log
folder into the agent’s container for the whole run and reads reward.json before reward.txt, so a file the agent
planted there would otherwise stand as the verdict (beside, or instead of, the runner’s own). Gym’s server empties the
folder before every upload; the published artifact is graded by Harbor, which is why the shim clears it itself.
Module Contents
Classes
Functions
Data
API
Bases: HarborTaskError
A task’s Python verifier is not usable (a module the shim cannot wrap).
What the shim needs to know about the task it wraps.
The parameter names of def verify in the module, read from its source without importing it.
None when the module defines verify some other way (verify = partial(...)), which the host cannot
resolve without running task code; the in-sandbox runner then decides at run time.
The module that grades task_folder, or None when tests/test.sh does.
tests/verifier.py when the task has one and no tests/test.sh; else the dataset root’s verifier.py
when the task has no tests/ folder at all. tests/test.sh wins whenever it exists: on the hub a
tests/verifier.py next to it is the helper that test.sh runs (the shape of whole datasets, such as
multimedia-terminalbench, osworld-verified or o11y-bench), not a second grader, so the task is a test.sh task
and the module an ordinary test file. A tests/ folder holding neither is left to test.sh grading
(None): Harbor admits that shape when a separate verifier image ships its own /tests/test.sh, and the
loader kept admitting it; such a task without one fails at verification with no test.sh to run, as before.
Whether verify declares response: such a verifier cannot be published to Harbor.
The test.sh the sandbox runs: clear stale reward files, find a Python of at least PYTHON_FLOOR,
hand the runner the paths, let it write the reward. No Python, or an older one, is exit 127 with the reason on
stderr and no reward (the missing-reward outcome).
Write the shim folder at target (created; existing shim files replaced) and return it.
target/test.sh, target/.ng/run_verifier.py, target/.ng/nemo_gym/verifiers/* (the installed
package’s modules, with an empty nemo_gym/__init__.py so the sandbox never imports the real package),
target/.ng/task_data.json, target/.ng/response.json when a response is given, and
target/.ng/verifier.py when the module is not in the task’s own tests/.
The generated tree is world-readable (0755 folders, 0644 files, test.sh 0755) whatever umask the
process runs under. Gym’s upload and Harbor’s docker cp both keep the archive’s modes and chown nothing, and
test.sh runs as the task’s [verifier].user: under umask 077 the runner and its helpers would land
0600 in a 0700 folder that a non-root verifier user could not open, and grading would fail before
verify ran. Only the shim’s own files are set, and target itself when this call created it (publish’s
tests/ for a task graded by the dataset’s module, which Harbor copies into /tests modes and all); a
tests/ the task shipped keeps the modes its author gave it.