Evaluation

Built-in Benchmarks

View as Markdown

All 17 runtime-registered built-in benchmarks are defined with @benchmark + @scorer or @register in src/nemo_evaluator/benchmarks/.

Quick Reference

BenchmarkCommandScoringType
MMLUnel eval run --bench mmlumultichoice_regexMultichoice (4-way)
MMLU-Pronel eval run --bench mmlu_promultichoice_regexMultichoice (10-way)
MATH-500nel eval run --bench math500answer_lineMath
GPQA Diamondnel eval run --bench gpqamultichoice_regexMultichoice (shuffled)
GSM8Knel eval run --bench gsm8knumeric_matchMath reasoning
DROPnel eval run --bench dropfuzzy_matchReading comprehension
MGSMnel eval run --bench mgsmnumeric_matchMultilingual math
TriviaQAnel eval run --bench triviaqafuzzy_matchFactual QA
HumanEvalnel eval run --bench humanevalcode_sandboxCode generation (Docker)
SimpleQAnel eval run --bench simpleqaneeds_judgeFactuality (LLM judge)
HealthBenchnel eval run --bench healthbenchneeds_judgeHealth (LLM judge)
PinchBenchnel eval run --bench pinchbenchcode_sandbox / needs_judgeAgentic tasks (code/LLM judge)
XSTestnel eval run --bench xstestneeds_judgeSafety
NMP Harbornel eval run --bench nmp_harborHarbor task testsNMP agentic-use tasks
Terminal-Bench Hardnel eval run --bench terminal-bench-hardTask test harnessTerminal tasks
Terminal-Bench Hard AA Splitnel eval run --bench terminal-bench-hard-aa-splitTask test harnessTerminal tasks
Terminal-Bench v1nel eval run --bench terminal-bench-v1Task test harnessTerminal-Bench v1

Extended Environments

Beyond the 17 built-in benchmarks, NEL resolves additional environment types via URI schemes and namespace prefixes:

SyntaxSourceExample
nel eval run --bench <name>Built-in registrynel eval run --bench mmlu
nel eval run --bench lm-eval://<task>lm-evaluation-harnessnel eval run --bench lm-eval://aime25
nel eval run --bench skills://<name>NeMo Skillsnel eval run --bench skills://mmlu-pro
nel eval run --bench vlmevalkit://<dataset>VLMEvalKitnel eval run --bench vlmevalkit://MMBench_DEV_EN
nel eval run --bench gym://<host:port>Remote Gym servernel eval run --bench gym://localhost:9090
nel eval run --bench harbor://<suite>@<version>Harbor task registrynel eval run --bench harbor://swebench-verified@1.0
nel eval run --bench container://<image>#<task>Legacy containernel eval run --bench container://nvcr.io/image#task

Benchmark Details

MMLU

Massive Multitask Language Understanding — 14K 4-choice questions across 57 subjects.

  • Dataset: hf://cais/mmlu?config=all&split=test
  • Scorer: multichoice_regex — extracts letter (A-D) from “Answer: X” pattern
  • prepare_row: Unpacks choices list into A, B, C, D fields; maps numeric answer to letter

MMLU-Pro

10-choice variant of MMLU with harder distractors.

  • Dataset: hf://TIGER-Lab/MMLU-Pro?split=test
  • Scorer: multichoice_regex with extended pattern [A-J]
  • prepare_row: Pads choices to 10 slots

MATH-500

500 competition-level math problems.

  • Dataset: hf://HuggingFaceH4/MATH-500?split=test
  • Scorer: answer_line — extracts answer after “Answer:” and normalizes

GPQA Diamond

Graduate-level science questions with shuffled answer choices.

  • Dataset: hf://Idavidrein/gpqa?config=gpqa_diamond&split=train
  • Scorer: multichoice_regex
  • prepare_row: Shuffles choices with seeded RNG to prevent position bias

GSM8K

1,319 grade-school math problems requiring multi-step reasoning.

  • Dataset: hf://openai/gsm8k?split=test
  • Scorer: numeric_match — extracts last number from response

DROP

Reading comprehension with discrete reasoning (counting, sorting, arithmetic).

  • Dataset: hf://ucinlp/drop?split=validation
  • Scorer: fuzzy_match with answer aliases

MGSM

Multilingual GSM8K — math problems in 10 languages.

  • Dataset: hf://juletxara/mgsm?split=test
  • Scorer: numeric_match

TriviaQA

Trivia questions with multiple acceptable answer aliases.

  • Dataset: hf://trivia_qa?config=rc.nocontext&split=validation
  • Scorer: fuzzy_match — normalized substring matching against aliases

HumanEval

164 Python function-completion problems with test suites.

  • Dataset: hf://openai/openai_humaneval?split=test
  • Scorer: code_sandbox — extracts code from markdown fences, runs in Docker with network isolation, memory limits, and timeouts
  • Requires: Docker daemon

SimpleQA

Short-form factuality questions requiring LLM-as-judge scoring.

  • Dataset: hf://basicv8vc/SimpleQA?split=test
  • Scorer: needs_judge — flags samples for post-processing by the judge pipeline

HealthBench

Medical accuracy questions requiring LLM-as-judge scoring.

  • Dataset: hf://openai/HealthBench?split=test
  • Scorer: needs_judge

Adding a New Benchmark

See the Write Your Own Benchmark (BYOB) tutorial for a complete walkthrough of the @benchmark + @scorer API.