nemo_gym.verifiers.runner

The in-sandbox runner of a Python verifier: python3 run_verifier.py --verifier ... --workspace ....

Copied into the verifier sandbox by the test.sh shim (nemo_gym.data.tasks.harbor.python_verifier) and run there as a script, so it is standard library only and imports nothing from nemo_gym. It puts its own .ng folder first on sys.path, where the shim vendored nemo_gym/verifiers, then the verifier module’s own folder, then the tests folder around the shim (a helper module beside the task’s tests/verifier.py stays importable when the module was published under .ng/), so the verifier’s from nemo_gym.verifiers import ... resolves without an install and always to the vendored copy: a nemo_gym package in the task’s own tests/, in the image’s site-packages or on its PYTHONPATH never replaces the helpers. That order holds however the interpreter was started (as test.sh runs it, python3 /tests/.ng/run_verifier.py, sys.path[0] is already this folder; under -I it is absent): each of the three entries is moved to its place, not merely added when missing. It loads the verifier module, calls verify with the arguments the function declares (workspace, task_data, response, state; positional-only parameters are passed by position), and writes Harbor’s reward files: reward.txt for a number, reward.json for a dict. An async def verify is awaited (asyncio.run). response exists only when the shim was given one (--response): the environment server always passes the agent’s final response, a published Harbor task never has one, so a verify that requires response without a default is a contract error there rather than a verifier quietly called with None.

Exit codes: 0 when a reward was written, 2 for a shim or verifier-contract problem (no verify, an argument it asks for that is not available, a return value that is not a number or a dict of numbers), 1 when verify raised. No reward file is written on 1 or 2, which the environment server and Harbor score as a missing reward, with this script’s stderr in the verifier log; and none is left either: before verify runs, any reward.json or reward.txt already in the reward folder (a file, a link or a folder of that name) is removed, as the test.sh shim removes them before this script starts, so a file the agent planted there never stands as the verdict under Harbor, which shares the folder with the agent’s container and reads reward.json first.

Module Contents

Functions

Name	Description
`_number`	-
`clear_rewards`	Remove every reward file already in `reward_dir` (a file, a link or a folder of that name), if the folder
`load_verify`	-
`main`	-
`place_on_path`	Put `folders` at the front of `sys.path` in the order given, highest priority first, whatever the
`select_arguments`	The arguments to call `verify` with: the known names it declares, each from `available`.
`write_reward`	Harbor’s reward files from `verify`’s return value; raises `RuntimeError` for anything else.

Data

KNOWN_ARGUMENTS

REWARD_FILES

API

nemo_gym.verifiers.runner._number(
    value: typing.Any
) -> float | None

nemo_gym.verifiers.runner.clear_rewards(
    reward_dir: pathlib.Path
) -> None

Remove every reward file already in reward_dir (a file, a link or a folder of that name), if the folder exists: the verdict is what verify writes in this run, never something found there.

nemo_gym.verifiers.runner.load_verify(
    path: pathlib.Path
) -> typing.Any

nemo_gym.verifiers.runner.main(
    argv: list[str] | None = None
) -> int

nemo_gym.verifiers.runner.place_on_path(
    folders: pathlib.Path = ()
) -> None

Put folders at the front of sys.path in the order given, highest priority first, whatever the interpreter put there already: an entry already on the path (sys.path[0] is this script’s own folder when it runs as test.sh runs it, without -I; PYTHONPATH may add any other) is moved, so no folder lower in the order is ever searched before one higher in it.

nemo_gym.verifiers.runner.select_arguments(
    verify: typing.Any,
    available: dict[str, typing.Any]
) -> tuple[list[typing.Any], dict[str, typing.Any]]

The arguments to call verify with: the known names it declares, each from available.

Returns (positional, keywords): a positional-only parameter (def verify(workspace, /)) is passed by position, every other one by name. A parameter the runner does not know, or that is not in available in this run (response without --response), is left to its default; one without a default is a contract error. *args/**kwargs receive nothing (a **kwargs verifier gets every available argument).

nemo_gym.verifiers.runner.write_reward(
    result: typing.Any,
    reward_dir: pathlib.Path
) -> None

Harbor’s reward files from verify’s return value; raises RuntimeError for anything else.

nemo_gym.verifiers.runner.KNOWN_ARGUMENTS = ('workspace', 'task_data', 'response', 'state')

nemo_gym.verifiers.runner.REWARD_FILES = ('reward.json', 'reward.txt')