Evaluate
For the underlying concepts, read Evaluation and Environments. For server-level architecture, read Architecture.
NeMo Gym evaluation is environment-native: the same dataset, resources server, verifier, and rollout machinery supports model comparison, harness comparison, benchmark scoring, ablation studies, and training-data analysis.
Gym Evaluation Loop
- Discover — find an environment or benchmark that matches your target capability:
gym list benchmarks,gym list environments,gym search <query>. - Configure — choose a model and agent harness; pre-flight check your config with
gym env validate. - Run — collect rollouts with fixed sampling settings and an explicit repeat count:
gym eval run. Resume an interrupted run with--resume. - Scale — shard across jobs and merge outputs with
gym eval aggregate. - Verify health — confirm that rollout records contain complete, internally consistent trajectory and model-call evidence.
- Profile — compute pass@1, pass@k, and per-task variance:
gym eval profile. - Diagnose — inspect failures, sometimes-pass tasks, and missing rows with BLADE.
- Decide — use results to determine whether the next action is model training, harness work, verifier repair, prompt change, or skill improvement.
- Reverify — if you change a verifier parameter (grading mode, threshold, judge prompt), recompute rewards on the same rollouts without re-running inference:
gym eval reverify.
The most important rule: vary one thing at a time. Changing the model, harness, prompt, verifier, and dataset together can still produce a useful release-gate score — but it cannot explain what caused the difference.
Models
NeMo Gym accesses models through the model server, which keeps provider-specific details behind the Responses API boundary — the same evaluation can compare hosted models, self-hosted vLLM instances, local checkpoints, or pre- and post-training snapshots.
Agent Harnesses
An agent harness is the orchestration layer that turns a model into an agent — it manages conversation state, routes tool calls, and decides when a task is complete. Harness changes often affect metrics as much as model changes do.
Benchmarks
A benchmark is a versioned evaluation protocol built on top of an environment’s resources server: the same verifier plus a canonical dataset split, prompt configuration, and documented repeat count. All environments can be used for training; a benchmark is the evaluation-configured overlay.