Write Your Own Benchmark (BYOB)

Define a complete benchmark with @benchmark + @scorer in under 10 lines.

Minimal Example

Save this as my_bench.py:

1
from nemo_evaluator import benchmark, scorer, ScorerInput, exact_match
2
3
@benchmark(
4
    name="my-bench",
5
    dataset="hf://my-org/my-dataset?split=test",
6
    prompt="Question: {question}\nAnswer:",
7
    target_field="answer",
8
)
9
@scorer
10
def my_scorer(sample: ScorerInput) -> dict:
11
    return exact_match(sample)

Run it:

$
export NVIDIA_API_KEY="your-api-key-here"
$
$
nel eval run --bench ./my_bench.py \
>
  --model-url https://integrate.api.nvidia.com/v1/chat/completions \
>
  --model-id nvidia/nemotron-3-super-120b-a12b \
>
  --api-key $NVIDIA_API_KEY

That is the entire local workflow. Passing the Python file path imports it, lets the @benchmark decorator register the environment, loads the dataset, formats prompts, and wires scoring. No package-internal edits or subclass boilerplate required.

How It Works

1. @benchmark creates a ByobEnvironment (subclass of EvalEnvironment) and registers it by name.
2. On seed(idx), the environment loads the dataset row, formats the prompt template, and returns a SeedResult.
3. On verify(response, expected), it calls your @scorer function with a ScorerInput and converts the result to a VerifyResult.

Step-by-Step

Step 1: Create the benchmark file

Create benchmarks/my_reasoning.py:

1
from nemo_evaluator import benchmark, scorer, ScorerInput, answer_line
2
3
@benchmark(
4
    name="my_reasoning",
5
    dataset="hf://your-org/reasoning-2026?split=test",
6
    prompt=(
7
        "Solve the following problem step by step.\n\n"
8
        "{question}\n\n"
9
        "Put your final answer after 'Answer:'."
10
    ),
11
    target_field="answer",
12
)
13
@scorer
14
def my_reasoning_scorer(sample: ScorerInput) -> dict:
15
    return answer_line(sample)

Step 2: Validate

Set your model endpoint once:

$
export NEMO_MODEL_URL="https://integrate.api.nvidia.com/v1/chat/completions"
$
export NEMO_MODEL_ID="nvidia/nemotron-3-super-120b-a12b"
$
export NVIDIA_API_KEY="your-api-key-here"

$
nel validate -b ./benchmarks/my_reasoning.py --samples 5

Expected output:

my_reasoning: 5 samples
  4/5 correct
  [PASS] p0: expected='42' got='42' (1230ms 156tok)
  [PASS] p1: expected='7/3' got='7/3' (980ms 134tok)
  [FAIL] p2: expected='256' got='512' (1100ms 201tok)
  ...

Step 3: Run full evaluation

$
nel eval run --bench ./benchmarks/my_reasoning.py \
>
  --repeats 4 \
>
  --output-dir ./results/my_reasoning

Step 4: Serve for Gym training

$
nel serve -b ./benchmarks/my_reasoning.py -p 9090

Gym training connects at http://hostname:9090.

Step 5: Package for sharing

Generate a Dockerfile for the benchmark module:

$
nel package \
>
  --module ./benchmarks/my_reasoning.py \
>
  --tag my-reasoning:latest \
>
  --output Dockerfile

Omit --output to build the image directly, and add --push to push it after build.

Decorator Reference

@benchmark parameters

@scorer function signature

1
@scorer
2
def my_scorer(sample: ScorerInput) -> dict:
3
    ...

The function receives a ScorerInput:

Return a dict. The key "correct" (or "reward") is converted to the numeric reward. Optionally include "extracted" for the extracted answer string.

Dataset Specs

HuggingFace URIs support split and config query parameters.

Scoring Primitives

Built-in functions you can call from your scorer:

All are importable from the top-level package:

1
from nemo_evaluator import exact_match, multichoice_regex, numeric_match

Extension Hooks

prepare_row: Transform dataset rows

Use prepare_row when the raw dataset needs restructuring before prompt formatting.

1
def shuffle_choices(row, idx, rng):
2
    """Shuffle GPQA answer choices and track the new correct index."""
3
    choices = [row["Correct Answer"], row["Incorrect Answer 1"],
4
               row["Incorrect Answer 2"], row["Incorrect Answer 3"]]
5
    rng.shuffle(choices)
6
    correct_idx = choices.index(row["Correct Answer"])
7
    return {**row, "A": choices[0], "B": choices[1],
8
            "C": choices[2], "D": choices[3],
9
            "answer": "ABCD"[correct_idx]}
10
11
@benchmark(
12
    name="gpqa",
13
    dataset="hf://Idavidrein/gpqa?config=gpqa_diamond&split=train",
14
    prompt=PROMPT_TEMPLATE,
15
    target_field="answer",
16
    prepare_row=shuffle_choices,
17
)
18
@scorer
19
def gpqa_scorer(sample: ScorerInput) -> dict:
20
    return multichoice_regex(sample)

seed_fn: Fully custom seed

When you need complete control over prompt construction:

1
def custom_seed(row, idx):
2
    from nemo_evaluator import SeedResult
3
    messages = [
4
        {"role": "system", "content": "You are a math tutor."},
5
        {"role": "user", "content": row["problem"]},
6
    ]
7
    return SeedResult(
8
        prompt=row["problem"],
9
        expected_answer=row["answer"],
10
        messages=messages,
11
        system="You are a math tutor.",
12
    )
13
14
@benchmark(name="custom", dataset="hf://...", prompt="", seed_fn=custom_seed)
15
@scorer
16
def custom_scorer(sample: ScorerInput) -> dict:
17
    return numeric_match(sample)

Real-World Examples

Most built-in benchmarks use @benchmark + @scorer. See src/nemo_evaluator/benchmarks/ for reference implementations:

Advanced: Subclass EvalEnvironment Directly

For benchmarks that cannot be expressed with decorators (e.g., multi-turn, stateful), subclass EvalEnvironment:

1
from nemo_evaluator import EvalEnvironment, SeedResult, VerifyResult, register
2
3
@register("my_complex_bench")
4
class MyComplexBenchmark(EvalEnvironment):
5
    def __init__(self):
6
        super().__init__()
7
        self._dataset = [...]
8
9
    async def seed(self, idx: int) -> SeedResult:
10
        row = self._dataset[idx]
11
        return SeedResult(prompt=row["prompt"], expected_answer=row["answer"])
12
13
    async def verify(self, response: str, expected: str, **meta) -> VerifyResult:
14
        correct = response.strip() == expected.strip()
15
        return VerifyResult(reward=1.0 if correct else 0.0)

The decorator path is preferred for single-turn benchmarks. Reserve subclassing for cases that genuinely need it.

Parametrizing a Built-in Benchmark from YAML

Built-in environments registered with @register("<name>") can be tweaked from a config without a new Python module. BenchmarkConfig accepts a params: mapping whose entries are forwarded to the environment constructor, filtered to the arguments it actually declares. Unknown keys raise a clear TypeError at resolution time.

1
benchmarks:
2
  - name: nmp_harbor
3
    params:
4
      task_names: ["workspace-basic-cli"]   # run a single task
5
    solver:
6
      type: harbor
7
      service: anthropic
8
      agent: claude-code
9
    sandbox:
10
      type: docker

Scalar values are accepted for list parameters (task_names: workspace-basic-cli) so the CLI override flag works too: -O benchmarks[0].params.task_names=workspace-basic-cli.

Wrapping a Built-in Benchmark with Extra Pre-Build Steps

When a dataset needs a one-off base image produced from a Dockerfile in a consumer repo, subclass the relevant environment under src/nemo_evaluator/benchmarks/ and prepend your own ImageBuildRequest. See src/nemo_evaluator/benchmarks/nmp_harbor.py for a working reference:

• Subclasses HarborEnvironment and registers under @register("nmp_harbor").
• Reads NMP_REPO (or the nmp_repo param) and points the dataset at $NMP_REPO/tests/agentic-use.
• Overrides image_build_requests() to prepend a single ImageBuildRequest that builds nmp-harbor:latest from $NMP_REPO/Dockerfile.harbor before any per-task image build runs.

1
benchmarks:
2
  - name: nmp_harbor             # reads $NMP_REPO; no manual docker build
3
    solver:
4
      type: harbor
5
      service: anthropic
6
      agent: claude-code
7
    sandbox:
8
      type: docker

The sandbox manager treats the prepended request identically to any other image build, so the same config works against Docker, ECR/ECS Fargate, etc. See examples/configs/10_nmp_harbor.yaml.