Evaluate
For the underlying concepts, read Evaluation and Environments. For server-level architecture, read Architecture.
NeMo Gym evaluation is environment-native: the same dataset, resources server, verifier, and rollout machinery supports model comparison, harness comparison, benchmark scoring, ablation studies, and training-data analysis.
Gym Evaluation Loop
- Discover — find an environment or benchmark that matches your target capability:
gym list benchmarks,gym list environments,gym search <query>. - Configure — choose a model and agent harness; pre-flight check your config with
gym env validate. - Run — collect rollouts with fixed sampling settings and an explicit repeat count:
gym eval run. Resume an interrupted run with--resume. - Scale — shard across jobs and merge outputs with
gym eval aggregate. - Profile — compute pass@1, pass@k, and per-task variance:
gym eval profile. - Diagnose — inspect failures, sometimes-pass tasks, and missing rows with BLADE.
- Decide — use results to determine whether the next action is model training, harness work, verifier repair, prompt change, or skill improvement.
The most important rule: vary one thing at a time. Changing the model, harness, prompt, verifier, and dataset together can still produce a useful release-gate score — but it cannot explain what caused the difference.
Models
NeMo Gym accesses models through the model server, which keeps provider-specific details behind the Responses API boundary — the same evaluation can compare hosted models, self-hosted vLLM instances, local checkpoints, or pre- and post-training snapshots.
Set up a model server for evaluation.
Review how model, agent, and resources servers fit together.
Agent Harnesses
An agent harness is the orchestration layer that turns a model into an agent — it manages conversation state, routes tool calls, and decides when a task is complete. Harness changes often affect metrics as much as model changes do.
Set up a built-in or custom agent harness.
Evaluate skills as a run-level variable — swap skill sets without touching the dataset, and compare variants using the content-hashed skills_ref.
Benchmarks
A benchmark is a fixed evaluation configuration built on top of an environment’s resources server — same verifier, frozen dataset split, documented repeat count. All environments can be used for training; a benchmark is the evaluation-configured overlay.
How to choose a benchmark, what the categories cover, and how to interpret results.
Full table of built-in benchmarks and training environments by category.
Contribution checklist for adding a new benchmark to Gym.
Next Steps
Run a small evaluation and inspect the generated outputs.
Step-by-step walkthroughs for specific benchmarks and evaluation workflows.
Customize metrics and key metrics for an environment.
Use BLADE to find why scores changed, which tasks failed, and what intervention to prioritize.
Compute per-task pass rates and variance with gym eval profile.
Use evaluation results to drive post-training.