> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/v0.5.1/evaluation/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# Evaluate

> Run repeatable model and agent evaluations with NeMo Gym environments.

> **Info**
>
> For the underlying concepts, read [Evaluation](/about/concepts/evaluation) and [Environments](/about/concepts/environments). For server-level architecture, read [Architecture](/about/architecture).

NeMo Gym evaluation is environment-native: the same dataset, resources server, verifier, and rollout machinery supports model comparison, harness comparison, benchmark scoring, ablation studies, and training-data analysis.

## Gym Evaluation Loop

1. **Discover** — find an environment or benchmark that matches your target capability: `gym list benchmarks`, `gym list environments`, `gym search <query>`.
2. **Configure** — choose a model and agent harness; pre-flight check your config with `gym env validate`.
3. **Run** — collect rollouts with fixed sampling settings and an explicit repeat count: `gym eval run`. Resume an interrupted run with `--resume`.
4. **Scale** — shard across jobs and merge outputs with `gym eval aggregate`.
5. **Profile** — compute pass\@1, pass\@k, and per-task variance: `gym eval profile`.
6. **Diagnose** — inspect failures, sometimes-pass tasks, and missing rows with BLADE.
7. **Decide** — use results to determine whether the next action is model training, harness work, verifier repair, prompt change, or skill improvement.
8. **Reverify** — if you change a verifier parameter (grading mode, threshold, judge prompt), recompute rewards on the same rollouts without re-running inference: `gym eval reverify`.

The most important rule: vary one thing at a time. Changing the model, harness, prompt, verifier, and dataset together can still produce a useful release-gate score — but it cannot explain what caused the difference.

## Models

NeMo Gym accesses models through the model server, which keeps provider-specific details behind the Responses API boundary — the same evaluation can compare hosted models, self-hosted vLLM instances, local checkpoints, or pre- and post-training snapshots.

#### [Configure Models](/model-server)

Set up a model server for evaluation.

#### [Architecture](/about/architecture)

Review how model, agent, and resources servers fit together.

## Agent Harnesses

An agent harness is the orchestration layer that turns a model into an agent — it manages conversation state, routes tool calls, and decides when a task is complete. Harness changes often affect metrics as much as model changes do.

#### [Configure Agents](/agent-server)

Set up a built-in or custom agent harness.

#### [Agent Skills](/agent-server/agent-skills)

Evaluate skills as a run-level variable — swap skill sets without touching the dataset, and compare variants using the content-hashed `skills_ref`.

## Benchmarks

A benchmark is a fixed evaluation configuration built on top of an environment's resources server — same verifier, frozen dataset split, documented repeat count. All environments can be used for training; a benchmark is the evaluation-configured overlay.

#### [Benchmarks](/evaluation/benchmarks)

How to choose a benchmark, what the categories cover, and how to interpret results.

#### [Browse Environments](/evaluation/environment-list)

Full table of built-in benchmarks and training environments by category.

#### [Add a Benchmark](/contribute/environments/adding-a-benchmark)

Contribution checklist for adding a new benchmark to Gym.

## Next Steps

#### [Quickstart](/get-started/quickstart)

Run a small evaluation and inspect the generated outputs.

#### [Evaluation Tutorials](/tutorials/evaluation-tutorials)

Step-by-step walkthroughs for specific benchmarks and evaluation workflows.

#### [Aggregate Metrics](/evaluation/aggregate-metrics)

Customize metrics and key metrics for an environment.

#### [Diagnose Results](/evaluation/diagnose-results)

Use BLADE to find why scores changed, which tasks failed, and what intervention to prioritize.

#### [Reward Profiling](/reference/cli-commands#gym-eval-profile)

Compute per-task pass rates and variance with `gym eval profile`.

#### [Reverify Rollouts](/tutorials/evaluation-tutorials/reverify-rollouts)

Recompute rewards from existing rollouts without re-running inference using `gym eval reverify`.

#### [Training](/training-tutorials)

Use evaluation results to drive post-training.

> Run repeatable model and agent evaluations with NeMo Gym environments.

## Docs

- [Benchmarks](https://docs.nvidia.com/nemo/gym/v0.5.1/evaluation/benchmarks.md): What benchmarks are, how they relate to environments, how to choose one, and how to interpret results.
- [Browse Environments](https://docs.nvidia.com/nemo/gym/v0.5.1/evaluation/environment-list.md): Browse built-in benchmark and training environments.
- [Aggregate Metrics](https://docs.nvidia.com/nemo/gym/v0.5.1/evaluation/aggregate-metrics.md): Compute summary statistics, customize per-environment metrics, and merge shard outputs from gym eval aggregate.
- [Diagnose Results](https://docs.nvidia.com/nemo/gym/v0.5.1/evaluation/diagnose-results.md): Use BLADE and other analysis skills to diagnose evaluation results — why scores changed, which tasks failed, and what intervention to prioritize.