Add a benchmark

A benchmark contribution should let any user prepare the same tasks, run them under a stated configuration, and understand what the score measures.

When onboarding a benchmark, it is important to keep the task contract independent of the harness wherever possible. The task defines the inputs, tools, state, required artifacts, and scoring rules; the harness chooses how to solve it. If the harness itself is part of the subject being measured, this must be stated explicitly. See Component Ownership for the agent/environment separation design and the current implementation boundaries.

Before implementation

The measurement contract should be recorded in the benchmark’s README:

• Provenance: upstream repository or dataset, pinned revision, data and software licenses, split, filtering, task count, and any synthetic data generation.
• Task contract: model-visible inputs, prompt construction, tools, sandbox requirements, and how the harness produces every artifact the verifier reads. Keep reference answers out of model-visible prompts.
• Scoring and run conditions: reward range and direction, aggregate metric, judge settings, harness and model, sampling settings, token or step budgets, timeouts, repeats, and seeds where supported.

Onboarding workflow

Use a development checkout, activate its environment, and run commands from the repository root. The example below creates a benchmark named my_benchmark using the standard agent loop and a new verifier.

1. Reuse components and scaffold

Search for an existing task or scorer before adding one:

gym search "task description"
gym list resources-servers
gym env init --benchmark my_benchmark --profile custom-gym-verifier

The scaffold creates these files under benchmarks/my_benchmark/:

It also creates resources_servers/my_benchmark/ with a verifier implementation, config, dependencies, unit tests, and verifier fixture. Reuse a harness for a new task; a new dataset or prompt does not by itself require a new Agent Server. See Single Step Environment when implementing new task tools or scoring.

If an existing scorer exports VERIFIER_FIXTURE, use --reuse-verifier NAME --reward-range 0 1 --higher-is-better with the scaffold command instead. Supply that scorer’s actual reward contract. Other integration shapes are covered in Integration Profiles.

2. Prepare data and complete the manifest

Replace the scaffold’s sample logic. The synchronous prepare() function must write the dataset declared in the config and return its output path. Pin the source, document transformations, and include representative rows that exercise both successful and failed verification.

gym eval prepare --benchmark my_benchmark

Preparation must run before validation when the declared data is absent. Validation reads the prepared JSONL; it does not download data or execute prepare().

Complete the manifest’s authored metadata and keep its composition mirror consistent with config.yaml. For a benchmark:

• Set kind: benchmark, the source canonical_split, and at least one dataset with type: benchmark and prepare_script. The source split name and Gym’s dataset type may differ.
• Use prompt_source: template (the default) with standard_prompt_config for templated rows; use prepared when rows already contain responses_create_params.input; use agent when prompts are constructed during harness or external-engine execution. Follow the prompt field rules.
• Keep task data with the resources-server instance for new Gym-native tasks, as the standard scaffold does. Existing agent-owned declarations remain supported. Update the runtime config first when wiring changes.
• Keep experimental: true initially. A manifest is a discovery and validation contract, not a certification of the benchmark’s scores. This value moves to false with specific approval from the Gym developer team.

See Environment Manifest for the example and field reference, including config_path for named configurations and dataset_owner for ambiguous compositions.

3. Check the declaration and scoring

gym env validate my_benchmark
gym env test my_benchmark
gym env test --resources-server my_benchmark

These check the manifest and prepared inputs, the workload’s verifier fixture, and the resources server’s component tests respectively. Run the component tests when you change its implementation. The fixture must cover both reward endpoints, malformed requests, and reseeding when determinism is claimed. Add workload-specific cases for grading modes or state transitions that a shared scorer fixture does not cover.

After an intentional config change, gym env validate --sync my_benchmark updates mirrored composition fields only after static checks pass. Review the diff; authored metadata such as split and prompt source still needs your judgment.

4. Run a representative evaluation

For an existing benchmark, try to reproduce a published result using your model of choice. Configure your endpoint as described in Configure Model. This example uses the OpenAI-compatible adapter; choose the adapter for your endpoint:

gym eval run --benchmark my_benchmark --model-type openai_model \
    --split benchmark --limit 5 --num-repeats 1 \
    --output results/my_benchmark_smoke.jsonl

For workloads with a verifier fixture, finish the local onboarding checks with gym env publish my_benchmark. This reruns validation and the fixture, rejects unfinished metadata, and confirms catalog discovery. It does not upload data, register a hosted workload, or push changes.