Browse Environments

NeMo Gym includes 100+ environments covering math, coding, reasoning, knowledge, agentic tool use, instruction following, and safety. An environment includes a dataset, agent harness, verifier, and state. Some resources servers include agent composition and datasets and are runnable environments themselves; others are reusable scoring, tool, or state components and are not separate environment entries. Environments marked ✓ have a corresponding benchmark config with a canonical evaluation split and prepare_script.

Use the CLI to discover what’s available before running:

gym list benchmarks          # all benchmarks with domain and description
gym list benchmarks aime24   # inspect one benchmark, including its agent harness
gym list environments        # unified environment and benchmark catalog
gym search math              # search environments and benchmarks by capability area
gym list benchmarks --json   # machine-readable output for scripting or CI

Math & Science

Environment	Description	Verification	Benchmark
math_with_judge	OpenMathReasoning, DAPO, and MathStackOverflow datasets	math-verify + LLM judge	✓
math_with_autograder	Hard math benchmarks (e.g. IMO AnswerBench)	math-verify + LLM autograder	✓
polymath	Multilingual math across 18 languages and 4 difficulty tiers	LLM judge (weighted)	✓
imo_gradingbench	Four-class IMO proof grading	Last-word extraction	✓
imo_proofbench_judge	IMO ProofBench with 0–7 rubric	LLM judge	✓
math_proof_judgement	Binary proof judgement — model reads a problem and proof, outputs Yes/No	LLM judge	✓
math_formal_lean	Lean4 formal proof verification	Lean4 compiler	✓
physics_judge	Open-ended physics QA	LLM judge + math-verify	✓
ugphysics_judge	Undergraduate physics benchmarks	LLM judge (TRUE/FALSE) + math-verify	✓
critpt	Research-level physics problems scored by the Artificial Analysis API	LLM judge (external)	✓
ether0	Chemistry benchmark verifiers (ether0)	Rule-based + LLM judge	✓
bunsenbench_chemistry_mcq	BunsenBench chemistry multiple-choice benchmark	Exact match	✓
assaybench	AssayBench: rank the hit genes of a CRISPR screen from its description (1,920 screens, 3 cohorts)	Adjusted nDCG@100 via the authors’ `assaybench` package	✓
frontierscience_judge	FrontierScience olympiad and research answer grading	LLM judge	✓
proof_verification	Proof scoring against ground truth	LLM judge + meta-verifier
newton_bench	Scientific law discovery across 12 physics domains	Execution
math_with_code	Competitive math with calculator tools	Boxed answer + numeric match

Coding

Environment	Description	Verification	Benchmark
bigcodebench	BigCodeBench Python solutions against unittest suite	Code execution	✓
evalplus	HumanEval+ and MBPP+ function completion	Code execution	✓
code_gen	Competitive coding problem solving	Code execution	✓
competitive_coding_challenges	Contest-style programming problems	Code execution	✓
code_fim	Code fill-in-the-middle (HumanEval-Infilling)	Code execution	✓
bird_sql	Text-to-SQL on BIRD dev (1,534 SQLite tasks)	SQL result-set equality	✓
spider2_lite	Text-to-SQL on Spider 2.0-Lite (135 enterprise tasks)	SQL result-set equality	✓
text_to_sql	Text-to-SQL across multiple SQL dialects	LLM judge (SQL equivalence)	✓
swerl_gen	SWE patch and test generation in a sandboxed environment	Code execution (pytest)	✓
scicode	Multi-step scientific code generation	Code execution	✓
cvdp	RTL hardware design code generation	Code execution (simulation)	✓
openenv	Python code execution environment via OpenEnv	Execution (stdout/stderr)

Knowledge & Reasoning

Environment	Description	Verification	Benchmark
gpqa_diamond	Graduate-level science multiple choice (GPQA Diamond)	Exact match	✓
mcqa	Multiple-choice QA covering MMLU, GPQA, HLE, LongBench	Exact match	✓
equivalence_llm_judge	Short-answer QA with LLM-as-a-judge equivalence scoring	LLM judge	✓
hotpotqa_qa	Closed-book multi-hop QA (HotPotQA)	SQuAD-style substring match	✓
simpleqa	Short-form factual QA with abstention scoring	LLM judge (3-tier)	✓
omniscience	Factual recall and calibration QA	LLM judge	✓
labbench2_vlm	Scientific VLM QA: figures, tables, lab protocols	LLM judge	✓
litmus_agent	Litmus-Bench v0.1 short-answer chemical reasoning	Answer extraction + numeric/boolean match	✓
arc_agi	Abstract reasoning puzzles (ARC-AGI)	Exact match (grid)	✓
nvarc	ARC-AGI in inductive (Python) and transductive (grid) modes	Code execution / exact match	✓
reasoning_gym	100+ tasks: algebra, logic, geometry, graph theory, games	Exact match
multichallenge	Multi-turn inference memory and instruction retention	LLM judge (rubric)
mrcr	Multi-round coreference resolution	F1 (SequenceMatcher)	✓
abstention	Train models to abstain when unsure (HotPotQA, three-tier reward)	LLM judge

Agentic / Tool Use

Environment	Description	Verification	Benchmark
workplace_assistant	Workplace tasks: 27 tools, 5 databases, 690 tasks	Rule-based (task completion)	✓
aviary	Multi-hop QA with Wikipedia search + GSM8k with calculator	LLM judge + execution	✓
tavily_search	Web search tool use (Tavily API)	Execution + optional LLM judge	✓
calendar	Multi-turn calendar scheduling with constraint satisfaction	Rule-based (constraints)	✓
finance_sec_search	SEC EDGAR filing search for financial analysis	LLM judge + execution	✓
xlam_fc	Function calling from Salesforce xlam-60k	Exact match	✓
single_step_tool_use_with_argument_comparison	Pivot RL for tool use across conversational, SWE, and search domains	Argument comparison	✓
tales	Text-adventure games (AlfWorld, ScienceWorld, Jericho, TextWorld) in Gymnasium API style	Rule-based (task completion)
google_search	MCQA with integrated Google search tool	Exact match

Instruction Following & Safety

Environment	Description	Verification	Benchmark
ifbench	IFBench with 57 instruction types (AllenAI library)	LLM judge	✓
inverse_if	Inverse instruction-following benchmark — instructions that counter conventional expectations	LLM judge (per-task)	✓
structured_outputs	Schema adherence across structured output formats	Rule-based (schema validation)	✓
indirect_prompt_injection	Resistance to injected instructions in tool-use trajectories	Rule-based (attack detection)	✓
jailbreak_detection	Jailbreak resistance with Nemotron judge	LLM judge	✓
xstest	Over-refusal calibration (XSTest)	Rule-based	✓
instruction_following	IFEval and IFBench-style instruction following	LLM judge
verifif	VerifIF instruction-following validators (rule-based + LLM judge)	Rule-based + LLM judge
format_verification	Citation format and freeform text formatting	Regex / rule-based
structeval	StructEval: JSON, YAML, CSV, TOML, XML schema adherence	Rule-based (schema parsing)
over_refusal_detection	Train models to avoid refusing safe prompts	LLM judge

Other

Environment	Description	Verification	Benchmark
asr_with_pc	ASR with WER (standard, case-sensitive, punctuation)	Execution (WER metrics)	✓
wmt_translation	Machine translation with BLEU and xCOMET-XXL	Execution (COMET metrics)	✓
longmt_eval	Document-level translation (SEGALE pipeline + COMETKiwi)	Execution (neural QE)	✓
vlm_eval_kit	VLM benchmarks: MMBench, OCRBench, and others	Execution (VLMEvalKit)	✓
graphwalks	Long-context graph BFS/DFS reasoning	F1 over node sets	✓
arena_judge	Arena Hard v2 pairwise LLM-judge (category-specific, side-swapped)	LLM judge (pairwise)	✓
speed_bench	Speculative-decoding throughput measurement	vLLM Prometheus metrics	✓
genrm_compare	GenRM pairwise comparison for RLHF training	LLM judge (pairwise)
blackjack	Gymnasium-style Blackjack (multi-step)	Win/draw/loss
grl_sokoban	Single-box Sokoban puzzle	Execution (puzzle solved)
grl_tetris	Tetris in Gymnasium API style	Rule-based (score/lines)

Benchmarks

How to choose a benchmark and interpret results.

Add a Benchmark

Use the contribution checklist to add a new benchmark.

Build Verifiers

Learn how environments verify agent behavior and compute rewards.