Home
LLM evaluation framework: benchmark environments, pluggable solvers, multi-format reporting.
Get Started
Installation
Install from source and configure extras (scoring, skills, harbor, proxy).
Quickstart
Run your first evaluation in under 5 minutes.
Features
- Everything is an Environment. Built-in benchmarks, NeMo Skills, Gym remotes, lm-eval tasks, and VLMEvalKit datasets all resolve through one registry.
@benchmark+@scorer. Define a complete benchmark in under 10 lines of Python.- Pluggable solvers.
simple,harbor,tool_calling,gym_delegation,openclaw— swap inference strategy per benchmark via config. - Cluster backends. Run locally, in Docker, or on SLURM clusters with automatic model deployment.
- Resilient suites. Per-benchmark checkpointing with failure isolation. Resume partially completed suites with
--resume. - Statistical regression. Compare runs with McNemar’s exact test, paired flip analysis, and confidence intervals. Gate releases across benchmark suites with per-benchmark policy thresholds.
- 17 built-in benchmarks. MMLU, MMLU-Pro, MATH-500, GPQA, GSM8K, DROP, MGSM, TriviaQA, HumanEval, SimpleQA, HealthBench, PinchBench, XSTest, Terminal-Bench variants, and NMP Harbor.
Tutorials
Interactive Walkthrough
Guided tour of all main features through real config examples — start here.
Write Your Own Benchmark
Define a complete benchmark with @benchmark + @scorer in under 10 lines.
Gym Integration
Serve benchmarks for NeMo Gym training and consume remote Gym environments.
NeMo Skills Integration
Use NeMo Skills benchmarks with full per-request observability.
Distributed Evaluation
Scale to thousands of problems with SLURM, Kubernetes, Ray, or manual sharding.
Compare Runs
Diagnose what changed between two runs of the same benchmark with nel compare.
Quality Gates
Turn benchmark thresholds into a suite-level GO / NO-GO / INCONCLUSIVE decision with nel gate.