Benchmarks

A benchmark is a versioned evaluation protocol built on top of an environment’s resources server. It declares a canonical dataset split and prompt configuration, a prepare_script that reproduces the data, and a documented repeat count so models and harnesses can be compared fairly across runs and teams.

All environments can be used for training. A benchmark is the evaluation-configured overlay on top of the same verifier: the same resources server plus a versioned comparison protocol. Benchmarks are not permanently frozen; corrections and improvements may change their data or prompts, but comparison-affecting changes require a new version.

How Benchmarks Relate to Environments

The same resources server powers both. For example, resources_servers/math_with_judge/configs/math_with_judge.yaml provides the OpenMathReasoning training dataset and uses math_with_judge as the verifier for benchmarks like aime24, aime25, and gsm8k. The benchmark config chains to the resources server via config_paths and adds a type: benchmark dataset with its own prepare_script and num_repeats.

# benchmarks/aime24/config.yaml
config_paths:
  - resources_servers/math_with_judge/configs/math_with_judge.yaml
aime24_math_with_judge_simple_agent:
  _inherit_from: math_with_judge_simple_agent
  responses_api_agents:
    simple_agent:
      datasets:
      - name: aime24
        type: benchmark
        prepare_script: benchmarks/aime24/prepare.py
        num_repeats: 32

Discover Available Benchmarks

Use the CLI to browse what’s available before running:

# List all benchmarks with domain and description
gym list benchmarks
# Inspect one benchmark, including its agent harness
gym list benchmarks aime24
# Filter to a capability area
gym search benchmarks math
gym search benchmarks coding
gym search benchmarks safety
# Machine-readable output for scripting or CI
gym list benchmarks --json

Choosing a Benchmark

Pick a benchmark that matches the capability you are actually measuring. Changing the model, harness, prompt, verifier, and dataset at the same time can still produce a useful score as a release gate — but it cannot explain what caused the difference. Vary one thing at a time.

For external benchmarks, reproduce the original reported numbers outside Gym before integrating. This separates benchmark reproduction issues from Gym integration issues.

Benchmark Categories

NeMo Gym includes 80+ benchmarks across six categories. Use gym list benchmarks to browse the full list with domains and descriptions.

Math & Science AIME, IMO, formal proofs (Lean4), physics (critpt, ugphysics), chemistry (ether0, BunsenBench), FrontierScience, and multilingual math. Verification ranges from exact match and math-verify to LLM judges for open-ended proofs.

Coding Function completion (HumanEval, MBPP), competitive programming, text-to-SQL (BIRD, Spider2), SWE-style patch generation, and RTL design. Most use code execution against a test suite.

Knowledge & Reasoning Graduate-level science (GPQA Diamond), MMLU variants, HLE, multi-hop QA (HotPotQA), factual recall (SimpleQA, OmniScience), and abstract reasoning (ARC-AGI).

Agentic / Tool Use Multi-step tool calling, web search (BrowseComp, MM-BrowseComp), calendar scheduling, financial analysis, text-adventure games (AlfWorld, ScienceWorld), and function-calling datasets.

MM-BrowseComp is the multimodal version of BrowseComp: 400 web-research questions, 243 of them with question images. Prepare it with gym eval prepare --benchmark mm_browsecomp, then run it with a vision-capable policy as its README shows. The policy sees the question images, but search and browse return text only. Gym scores answers with the BrowseComp answer judge, not the official MM-BrowseComp checklist judge.

Instruction Following & Safety IFEval, IFBench, inverse instruction following, VerifIF, jailbreak resistance, over-refusal calibration (XSTest), and structured output validation.

Other ASR (LibriSpeech, WER), machine translation (WMT, FLORES-200), VLM benchmarks, Arena Hard pairwise judging, and speculative-decoding throughput.

HealthBench and HealthBench Professional

These clinical-response benchmarks use the shared HealthBench rubric verifier and an independent judge model. They are experimental integrations. See the HealthBench README and HealthBench Professional README for dataset provenance, judge settings, and scoring details.

Prepare each dataset before running it. The smoke config routes Nemotron Lightning policy calls and independent judge calls through the NVIDIA inference gateway. It selects GPT-4.1 for HealthBench; Professional selects GPT-5.4 with low judge reasoning effort in its benchmark config. Set NVIDIA_API_KEY for this example, or provide your own model and judge settings with ++judge_base_url, ++judge_api_key, and ++judge_model_name.

gym eval prepare --benchmark healthbench
gym eval prepare --benchmark healthbench_professional
gym eval run --benchmark healthbench \
  --config benchmarks/healthbench/configs/nemotron-lightning-smoke.yaml \
  --split benchmark --output results/healthbench/rollouts.jsonl
gym eval run --benchmark healthbench_professional \
  --config benchmarks/healthbench/configs/nemotron-lightning-smoke.yaml \
  --split benchmark --output results/healthbench_professional/rollouts.jsonl \
  ++smoke_judge_model_name=azure/openai/gpt-5.4

HealthBench Full is the default. Pass --config benchmarks/healthbench/configs/consensus.yaml or --config benchmarks/healthbench/configs/hard.yaml to both prepare and run for the corresponding subset. The headline aggregate metric is overall_score for HealthBench and overall_score_length_adjusted for Professional. mean/reward is a bounded Gym reward, not the published benchmark score. Judge failures are masked and reduce scored coverage; report that coverage with every score.

Benchmarks Whose Data You Stage

A few benchmarks declare a protocol over a dataset that cannot ship with Gym. Their prepare_script copies a file you stage yourself and fails when you have not given it one, so a run never scores a partial or stale set.

AgentArena (benchmarks/agent_arena) is one. It runs the 433-turn Arena validation split at four repeats over the agentic_turn_simulation environment, which scores a whole agent turn against simulated tools. The rows carry the agent harness’s own system prompt and conversation history, so the benchmark sets no prompt_config of its own. The split is held out from the Arena RL training rows, is NVIDIA internal use only and is not redistributable: obtain it from the dataset’s owners. Its row format is the environment’s own, described in the environment’s data guide.

Stage the file, then prepare and run. The simulator and the judge are the environment’s judge model server, so every run names it:

AGENT_ARENA_TURNS_JSONL=/path/to/arena_turns.jsonl \
  gym eval prepare --benchmark agent_arena
gym eval run --benchmark agent_arena \
  --model-type vllm_model --model-url http://MODEL_HOST:8000/v1 --model MODEL_NAME \
  --output results/agent_arena/rollouts.jsonl \
  ++judge_base_url="$JUDGE_URL" ++judge_api_key="$JUDGE_KEY" ++judge_model_name="$JUDGE_MODEL"

++judge_base_url takes one URL or a list of replicas. ++prepare_script_args.source_jsonl=<path> names the staged file instead of the environment variable, and ++prepare_script_args.limit=<n> prepares only the first n turns for a smoke run. A subset is not the benchmark’s score: the protocol is all 433 turns at four repeats.

Interpreting Results

A score is only meaningful relative to a fixed comparison: same dataset split, same model, same harness, same sampling config. It is also only as reliable as the verifier behind it — before trusting a score, confirm the reward function actually measures the behavior you care about.

Use gym eval profile to compute pass@1, pass@k, and per-task variance from repeated rollouts. Use gym eval aggregate to merge shard outputs and compute summary statistics grouped by agent. Use BLADE to diagnose why a score changed and which tasks drove the shift.

Next Steps