Build Verifiers

A verifier scores an agent’s behavior on a task. Where you implement it depends on how you define your tasks:

• JSONL rows plus a Resources Server. You implement the verifier in a Resources Server, the component that owns the verifier, per-task state, and any environment-specific tools. The rest of this page covers this path.
• Task folders. The verifier is the task folder’s tests/, or one verifier.py beside dataset.toml that grades every single-step task without tests/. The harbor Environment Server runs it in a sandbox after the agent finishes, with no Resources Server. See Task Folder Verifiers.

The pattern pages below apply to both paths: they describe how to score a rollout, whichever component runs the check. Their examples use a Resources Server’s verify().

Resources Server Interfaces

You build a Resources Server by subclassing one of two base classes. At runtime it runs as a web server that is driven over HTTP:

• SimpleResourcesServer — /seed_session initializes per-task state at the start (optional), and /verify scores the rollout at the end. An Environment Server that drives agent sessions, such as single_agent_turn, calls both in order. Only on the legacy_agent path, where the agent runs the whole episode itself, does the agent call them. Tools are exposed as additional endpoints or over MCP.
• GymnasiumServer — scores incrementally via /step (with /reset at the start) instead of a single /verify, for environments that need per-turn observations and rewards. GymnasiumServer is provided by the bundled resources_servers/gymnasium/ package, not the core nemo_gym library — import it with from resources_servers.gymnasium import GymnasiumServer.