nemo_gym.verifiers

Helpers for a Harbor task’s Python verifier (tests/verifier.py, or verifier.py at the dataset root).

A Python verifier is a module with a verify function. It runs inside the task’s verifier sandbox, never on the host: the harbor environment server (and gym dataset publish --harbor) wraps it in a tests/test.sh shim that carries this package along, so from nemo_gym.verifiers import ... works in the sandbox with nothing installed. Everything here is therefore standard library only, and this package imports nothing else from nemo_gym.

verify declares the arguments it wants and the runner passes those alone (by name, or by position for a positional-only parameter such as def verify(workspace, /)):

workspace (pathlib.Path): the task’s working directory, where the agent worked;
task_data (dict): the task’s own fields (task_data.json, a parameterized row);
response (dict or None): the agent’s final Responses API response (not available to a published Harbor task, whose test.sh sees only the container);
state (dict): {"workspace": Path}, the RFC’s spelling of the workspace.

It returns a number (the reward) or a dict with a reward key and other numeric keys as metrics; an async def verify is awaited. A verify that requires response without a default is a contract error (no reward) wherever no response exists, which is every published Harbor task.

Submodules

nemo_gym.verifiers.files
nemo_gym.verifiers.runner

Package Contents

Functions

Name	Description
`_parse_number`	-
`exact_match`	`1.0` when the two strings agree, after stripping surrounding whitespace by default.
`final_text`	The text of the agent’s final assistant message in a Responses API response (`""` when there is none).
`numeric_match`	`1.0` when the last number in `actual` equals `expected` within the tolerances.

Data

_NUMBER

API

nemo_gym.verifiers._parse_number(
    text: str
) -> float | None

nemo_gym.verifiers.exact_match(
    actual: str,
    expected: str,
    strip: bool = True,
    case_sensitive: bool = True
) -> float

1.0 when the two strings agree, after stripping surrounding whitespace by default.

nemo_gym.verifiers.final_text(
    response: typing.Mapping[str, typing.Any] | None
) -> str

The text of the agent’s final assistant message in a Responses API response ("" when there is none).

Walks response["output"] from the end and returns the concatenated output_text parts of the last message item; a response that is None or carries no message yields "".

nemo_gym.verifiers.numeric_match(
    actual: str | float,
    expected: str | float,
    rel_tol: float = 1e-06,
    abs_tol: float = 0.0
) -> float

1.0 when the last number in actual equals expected within the tolerances.

Strings are searched for their last number ("The answer is 72." reads as 72; thousands separators are dropped); a string without one scores 0.0.

nemo_gym.verifiers._NUMBER = re.compile('[-+]?(?:\\d+(?:[.,]\\d+)*(?:\\.\\d+)?|\\.\\d+)(?:[eE][-+]?\\d+)?')