Browse Environments

NeMo Gym includes 100+ environments covering math, coding, reasoning, knowledge, agentic tool use, instruction following, and safety. An environment combines tasks, verification, and an interaction surface. Some resources servers include agent composition and datasets and are runnable environments themselves; others are reusable scoring, tool, or state components and are not separate environment entries. Environments marked ✓ have a corresponding benchmark config with a canonical evaluation split and prepare_script.

Use the CLI to discover what’s available before running:

$ gym list benchmarks          # all benchmarks with domain and repeat count
$ gym list environments        # unified environment and benchmark catalog
$ gym search math              # search environments and benchmarks by capability area
$ gym list benchmarks --json   # machine-readable output for scripting or CI

Math & Science

Environment	Description	Verification	Benchmark
math_with_judge	OpenMathReasoning, DAPO, and MathStackOverflow datasets	math-verify + LLM judge	✓
math_with_autograder	Hard math benchmarks (e.g. IMO AnswerBench)	math-verify + LLM autograder	✓
polymath	Multilingual math across 18 languages and 4 difficulty tiers	LLM judge (weighted)	✓
imo_gradingbench	Four-class IMO proof grading	Last-word extraction	✓
imo_proofbench_judge	IMO ProofBench with 0–7 rubric	LLM judge	✓
math_proof_judgement	Binary proof judgement — model reads a problem and proof, outputs Yes/No	LLM judge	✓
math_formal_lean	Lean4 formal proof verification	Lean4 compiler	✓
physics_judge	Open-ended physics QA	LLM judge + math-verify	✓
ugphysics_judge	Undergraduate physics benchmarks	LLM judge (TRUE/FALSE) + math-verify	✓
critpt	Research-level physics problems scored by the Artificial Analysis API	LLM judge (external)	✓
ether0	Chemistry benchmark verifiers (ether0)	Rule-based + LLM judge	✓
bunsenbench_chemistry_mcq	BunsenBench chemistry multiple-choice benchmark	Exact match	✓
frontierscience_judge	FrontierScience olympiad and research answer grading	LLM judge	✓
proof_verification	Proof scoring against ground truth	LLM judge + meta-verifier
newton_bench	Scientific law discovery across 12 physics domains	Execution
math_with_code	Competitive math with calculator tools	Boxed answer + numeric match

Coding

Environment	Description	Verification	Benchmark
bigcodebench	BigCodeBench Python solutions against unittest suite	Code execution	✓
evalplus	HumanEval+ and MBPP+ function completion	Code execution	✓
code_gen	Competitive coding problem solving	Code execution	✓
competitive_coding_challenges	Contest-style programming problems	Code execution	✓
code_fim	Code fill-in-the-middle (HumanEval-Infilling)	Code execution	✓
bird_sql	Text-to-SQL on BIRD dev (1,534 SQLite tasks)	SQL result-set equality	✓
spider2_lite	Text-to-SQL on Spider 2.0-Lite (135 enterprise tasks)	SQL result-set equality	✓
text_to_sql	Text-to-SQL across multiple SQL dialects	LLM judge (SQL equivalence)	✓
swerl_gen	SWE patch and test generation in a sandboxed environment	Code execution (pytest)	✓
scicode	Multi-step scientific code generation	Code execution	✓
cvdp	RTL hardware design code generation	Code execution (simulation)	✓
openenv	Python code execution environment via OpenEnv	Execution (stdout/stderr)

Knowledge & Reasoning

Environment	Description	Verification	Benchmark
gpqa_diamond	Graduate-level science multiple choice (GPQA Diamond)	Exact match	✓
mcqa	Multiple-choice QA covering MMLU, GPQA, HLE, LongBench	Exact match	✓
equivalence_llm_judge	Short-answer QA with LLM-as-a-judge equivalence scoring	LLM judge	✓
hotpotqa_qa	Closed-book multi-hop QA (HotPotQA)	SQuAD-style substring match	✓
simpleqa	Short-form factual QA with abstention scoring	LLM judge (3-tier)	✓
omniscience	Factual recall and calibration QA	LLM judge	✓
labbench2_vlm	Scientific VLM QA: figures, tables, lab protocols	LLM judge	✓
litmus_agent	Litmus-Bench v0.1 short-answer chemical reasoning	Answer extraction + numeric/boolean match	✓
arc_agi	Abstract reasoning puzzles (ARC-AGI)	Exact match (grid)	✓
nvarc	ARC-AGI in inductive (Python) and transductive (grid) modes	Code execution / exact match	✓
reasoning_gym	100+ tasks: algebra, logic, geometry, graph theory, games	Exact match
multichallenge	Multi-turn inference memory and instruction retention	LLM judge (rubric)
mrcr	Multi-round coreference resolution	F1 (SequenceMatcher)	✓
abstention	Train models to abstain when unsure (HotPotQA, three-tier reward)	LLM judge

Agentic / Tool Use

Environment	Description	Verification	Benchmark
workplace_assistant	Workplace tasks: 27 tools, 5 databases, 690 tasks	Rule-based (task completion)	✓
aviary	Multi-hop QA with Wikipedia search + GSM8k with calculator	LLM judge + execution	✓
tavily_search	Web search tool use (Tavily API)	Execution + optional LLM judge	✓
calendar	Multi-turn calendar scheduling with constraint satisfaction	Rule-based (constraints)	✓
finance_sec_search	SEC EDGAR filing search for financial analysis	LLM judge + execution	✓
xlam_fc	Function calling from Salesforce xlam-60k	Exact match	✓
single_step_tool_use_with_argument_comparison	Pivot RL for tool use across conversational, SWE, and search domains	Argument comparison	✓
tales	Text-adventure games (AlfWorld, ScienceWorld, Jericho, TextWorld) in Gymnasium API style	Rule-based (task completion)
google_search	MCQA with integrated Google search tool	Exact match

Instruction Following & Safety

Environment	Description	Verification	Benchmark
ifbench	IFBench with 57 instruction types (AllenAI library)	LLM judge	✓
inverse_if	Inverse instruction-following benchmark — instructions that counter conventional expectations	LLM judge (per-task)	✓
structured_outputs	Schema adherence across structured output formats	Rule-based (schema validation)	✓
indirect_prompt_injection	Resistance to injected instructions in tool-use trajectories	Rule-based (attack detection)	✓
jailbreak_detection	Jailbreak resistance with Nemotron judge	LLM judge	✓
xstest	Over-refusal calibration (XSTest)	Rule-based	✓
instruction_following	IFEval and IFBench-style instruction following	LLM judge
verifif	VerifIF instruction-following validators (rule-based + LLM judge)	Rule-based + LLM judge
format_verification	Citation format and freeform text formatting	Regex / rule-based
structeval	StructEval: JSON, YAML, CSV, TOML, XML schema adherence	Rule-based (schema parsing)
over_refusal_detection	Train models to avoid refusing safe prompts	LLM judge

Other

Environment	Description	Verification	Benchmark
asr_with_pc	ASR with WER (standard, case-sensitive, punctuation)	Execution (WER metrics)	✓
wmt_translation	Machine translation with BLEU and xCOMET-XXL	Execution (COMET metrics)	✓
longmt_eval	Document-level translation (SEGALE pipeline + COMETKiwi)	Execution (neural QE)	✓
vlm_eval_kit	VLM benchmarks: MMBench, OCRBench, and others	Execution (VLMEvalKit)	✓
graphwalks	Long-context graph BFS/DFS reasoning	F1 over node sets	✓
arena_judge	Arena Hard v2 pairwise LLM-judge (category-specific, side-swapped)	LLM judge (pairwise)	✓
speed_bench	Speculative-decoding throughput measurement	vLLM Prometheus metrics	✓
genrm_compare	GenRM pairwise comparison for RLHF training	LLM judge (pairwise)
blackjack	Gymnasium-style Blackjack (multi-step)	Win/draw/loss
grl_sokoban	Single-box Sokoban puzzle	Execution (puzzle solved)
grl_tetris	Tetris in Gymnasium API style	Rule-based (score/lines)

Benchmarks

How to choose a benchmark and interpret results.

Add a Benchmark

Use the contribution checklist to add a new benchmark.

Build Verifiers

Learn how environments verify agent behavior and compute rewards.