Installation | NVIDIA NeMo Evaluator

Requirements

Python 3.12 or 3.13
An OpenAI-compatible model endpoint (NVIDIA API Catalog, vLLM, NIM, etc.)

Install from Source

git clone https://github.com/NVIDIA-NeMo/Evaluator.git
cd Evaluator
pip install -e ".[all]"

Install Extras

Extra	Command	What it adds
`scoring`	`pip install -e ".[scoring]"`	sympy for symbolic math comparison
`stats`	`pip install -e ".[stats]"`	scipy for confidence intervals, McNemar significance testing, and regression analysis
`skills`	`pip install -e ".[skills]"`	NeMo Skills benchmark integration
`harbor`	`pip install -e ".[harbor]"`	Harbor agent integration (OpenHands, Terminus-2, etc.)
`lm-eval`	`pip install -e ".[lm-eval]"`	LM-Evaluation-Harness integration
`inspect`	`pip install -e ".[inspect]"`	Inspect AI log export (`inspect_ai`-compatible `EvalLog` files)
`ray`	`pip install -e ".[ray]"`	Ray distributed launcher
`harnesses`	`pip install -e ".[harnesses]"`	lm-evaluation-harness tasks
`export`	`pip install -e ".[export]"`	WandB and MLflow experiment tracker export
`docs`	`pip install -e ".[docs]"`	Sphinx, NVIDIA theme, mermaid for building docs
`all`	`pip install -e ".[all]"`	Common runtime integrations: scoring, stats, ray, lm-eval, and Harbor
`dev`	`pip install -e ".[dev]"`	pytest, ruff, and common development extras

Verify Installation

nel --version
nel list

Expected output:

nemo-evaluator 0.12.0
Built-in benchmarks:
  drop                           nel eval run --bench drop
  gpqa                           nel eval run --bench gpqa
  gsm8k                          nel eval run --bench gsm8k
  ...

Docker

docker build -t nemo-evaluator .
docker run nemo-evaluator nel list

Next Steps

Proceed to the Quickstart to run your first evaluation.