> This page is for version 0.7.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # Add a benchmark > Prepare, declare, and validate a reproducible benchmark using reusable Gym components and a manifest. # Add a benchmark A benchmark contribution should let any user prepare the same tasks, run them under a stated configuration, and understand what the score measures. When onboarding a benchmark, it is important to keep the task contract independent of the harness wherever possible. The task defines the inputs, tools, state, required artifacts, and scoring rules; the harness chooses how to solve it. If the harness itself is part of the subject being measured, this must be stated explicitly. See [Component Ownership](/main/contribute/environments/new-environment#component-ownership) for the agent/environment separation design and the current implementation boundaries. ## Before implementation The measurement contract should be recorded in the benchmark's README: * **Provenance:** upstream repository or dataset, pinned revision, data and software licenses, split, filtering, task count, and any synthetic data generation. * **Task contract:** model-visible inputs, prompt construction, tools, sandbox requirements, and how the harness produces every artifact the verifier reads. Keep reference answers out of model-visible prompts. * **Scoring and run conditions:** reward range and direction, aggregate metric, judge settings, harness and model, sampling settings, token or step budgets, timeouts, repeats, and seeds where supported. ## Onboarding workflow Use a [development checkout](/main/contribute/development-setup), activate its environment, and run commands from the repository root. The example below creates a benchmark named `my_benchmark` using the standard agent loop and a new verifier. ### 1. Reuse components and scaffold Search for an existing task or scorer before adding one: ```bash gym search "task description" gym list resources-servers gym env init --benchmark my_benchmark --profile custom-gym-verifier ``` The scaffold creates these files under `benchmarks/my_benchmark/`: | File | What to complete | | ----------------------------------------- | -------------------------------------------------------------------------------- | | `config.yaml` | Component wiring, dataset declaration, preparation path, and runtime settings | | `manifest.yaml` | Identity, license, reward contract, split, prompt source, and composition mirror | | `prepare.py` | Reproducible source-data conversion into benchmark JSONL | | `prompt.yaml` | Prompt template for the default scaffold's source rows | | `data/source.jsonl`, `data/example.jsonl` | Replace the sample task and retain representative examples | | `README.md` | Measurement contract, preparation and evaluation commands, and limitations | It also creates `resources_servers/my_benchmark/` with a verifier implementation, config, dependencies, unit tests, and verifier fixture. Reuse a harness for a new task; a new dataset or prompt does not by itself require a new Agent Server. See [Single Step Environment](/main/environment-tutorials/single-step-environment) when implementing new task tools or scoring. If an existing scorer exports `VERIFIER_FIXTURE`, use `--reuse-verifier NAME --reward-range 0 1 --higher-is-better` with the scaffold command instead. Supply that scorer's actual reward contract. Other integration shapes are covered in [Integration Profiles](/main/contribute/environments/new-environment#integration-profiles). ### 2. Prepare data and complete the manifest Replace the scaffold's sample logic. The synchronous `prepare()` function must write the dataset declared in the config and return its output path. Pin the source, document transformations, and include representative rows that exercise both successful and failed verification. ```bash gym eval prepare --benchmark my_benchmark ``` Preparation must run before validation when the declared data is absent. Validation reads the prepared JSONL; it does not download data or execute `prepare()`. Complete the manifest's authored metadata and keep its composition mirror consistent with `config.yaml`. For a benchmark: * Set `kind: benchmark`, the source `canonical_split`, and at least one dataset with `type: benchmark` and `prepare_script`. The source split name and Gym's dataset type may differ. * Use `prompt_source: template` (the default) with `standard_prompt_config` for templated rows; use `prepared` when rows already contain `responses_create_params.input`; use `agent` when prompts are constructed during harness or external-engine execution. Follow the [prompt field rules](/main/contribute/environments/new-environment#prompt-sources). * Keep task data with the resources-server instance for new Gym-native tasks, as the standard scaffold does. Existing agent-owned declarations remain supported. Update the runtime config first when wiring changes. * Keep `experimental: true` initially. A manifest is a discovery and validation contract, not a certification of the benchmark's scores. This value moves to *false* with specific approval from the Gym developer team. See [Environment Manifest](/main/contribute/environments/new-environment#environment-manifest) for the example and field reference, including `config_path` for named configurations and `dataset_owner` for ambiguous compositions. ### 3. Check the declaration and scoring ```bash gym env validate my_benchmark gym env test my_benchmark gym env test --resources-server my_benchmark ``` These check the manifest and prepared inputs, the workload's verifier fixture, and the resources server's component tests respectively. Run the component tests when you change its implementation. The fixture must cover both reward endpoints, malformed requests, and reseeding when determinism is claimed. Add workload-specific cases for grading modes or state transitions that a shared scorer fixture does not cover. After an intentional config change, `gym env validate --sync my_benchmark` updates mirrored composition fields only after static checks pass. Review the diff; authored metadata such as split and prompt source still needs your judgment. > **Note** > > The fixture-based `gym env test NAME` and `gym env publish NAME` commands currently require a resources server exporting `VERIFIER_FIXTURE`. External engines with `resources_server: null` can have valid manifests, but must use their own scoring tests and representative rollouts; record this limitation in the PR. ### 4. Run a representative evaluation For an existing benchmark, try to reproduce a published result using your model of choice. Configure your endpoint as described in [Configure Model](/main/model-server). This example uses the OpenAI-compatible adapter; choose the adapter for your endpoint: ```bash gym eval run --benchmark my_benchmark --model-type openai_model \ --split benchmark --limit 5 --num-repeats 1 \ --output results/my_benchmark_smoke.jsonl ``` For workloads with a verifier fixture, finish the local onboarding checks with `gym env publish my_benchmark`. This reruns validation and the fixture, rejects unfinished metadata, and confirms catalog discovery. It does not upload data, register a hosted workload, or push changes.