> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/evaluator/nightly/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/evaluator/_mcp/server.

# Home

LLM evaluation framework: benchmark environments, pluggable solvers, multi-format reporting.

[Installation](/get-started/install)

[Quickstart](/get-started/quickstart)

---

## Get Started

#### [Installation](/get-started/install)

Install from source and configure extras (scoring, skills, harbor, proxy).

#### [Quickstart](/get-started/quickstart)

Run your first evaluation in under 5 minutes.

## Features

* **Everything is an Environment.** Built-in benchmarks, NeMo Skills, Gym remotes, lm-eval tasks, and VLMEvalKit datasets all resolve through one registry.
* **`@benchmark` + `@scorer`.** Define a complete benchmark in under 10 lines of Python.
* **Pluggable solvers.** `simple`, `harbor`, `tool_calling`, `gym_delegation`, `openclaw` — swap inference strategy per benchmark via config.
* **Cluster backends.** Run locally, in Docker, or on SLURM clusters with automatic model deployment.
* **Resilient suites.** Per-benchmark checkpointing with failure isolation. Resume partially completed suites with `--resume`.
* **Statistical regression.** Compare runs with McNemar's exact test, paired flip analysis, and confidence intervals.  Gate releases across benchmark suites with per-benchmark policy thresholds.
* **17 built-in benchmarks.** MMLU, MMLU-Pro, MATH-500, GPQA, GSM8K, DROP, MGSM, TriviaQA, HumanEval, SimpleQA, HealthBench, PinchBench, XSTest, Terminal-Bench variants, and NMP Harbor.

## Tutorials

#### [Interactive Walkthrough](/tutorials/walkthrough)

Guided tour of all main features through real config examples — start here.

#### [Write Your Own Benchmark](/tutorials/byob)

Define a complete benchmark with `@benchmark` + `@scorer` in under 10 lines.

#### [Gym Integration](/tutorials/gym-integration)

Serve benchmarks for NeMo Gym training and consume remote Gym environments.

#### [NeMo Skills Integration](/tutorials/skills-integration)

Use NeMo Skills benchmarks with full per-request observability.

#### [Distributed Evaluation](/tutorials/distributed-eval)

Scale to thousands of problems with SLURM, Kubernetes, Ray, or manual sharding.

#### [Compare Runs](/tutorials/compare)

Diagnose what changed between two runs of the same benchmark with `nel compare`.

#### [Quality Gates](/tutorials/quality-gate)

Turn benchmark thresholds into a suite-level `GO / NO-GO / INCONCLUSIVE` decision with `nel gate`.

## Architecture & Deployment

#### [Architecture](/architecture)

How the system works: environments, solvers, execution modes, and observability.

#### [Deployment Guide](/deployment)

Deploy on SLURM, Docker, Kubernetes, Ray, and CI/CD pipelines.

#### [Benchmarks](/evaluation/benchmarks)

All runtime-registered built-in benchmarks with scoring details.

#### [API Reference](/api)

Python API and CLI reference.