Evaluation

Scoring

View as Markdown

Overview

Each benchmark defines its own scorer via the @scorer decorator. The scoring/ package provides reusable scorer implementations:

ModulePurpose
scoring/text.pyexact_match, fuzzy_match — text comparison
scoring/pattern.pymultichoice_regex, answer_line, numeric_match — pattern extraction
scoring/code_execution.pycode_sandbox — Docker-sandboxed code execution
scoring/judge.pyneeds_judge, judge_score — LLM-as-judge pipeline
scoring/json_schema.pyextract_json, validate_json_schema — structured output validation
scoring/types.pyScorerInput — input dataclass for all scorers

Scoring Primitives

These functions are the building blocks for @scorer functions. All are importable from the top-level package.

exact_match(sample)

Normalized string comparison: lowercase, strip whitespace and punctuation, collapse articles.

1from nemo_evaluator import exact_match, ScorerInput
2
3s = ScorerInput(response="  Paris.  ", target="paris")
4exact_match(s)  # {"correct": True}

multichoice_regex(sample)

Extracts a letter (A-D by default) from “Answer: X” patterns.

1from nemo_evaluator import multichoice_regex, ScorerInput
2
3s = ScorerInput(response="The answer is B because...\nAnswer: B", target="B")
4multichoice_regex(s)  # {"correct": True, "extracted": "B"}
5
6# Custom pattern for 10-choice (A-J):
7multichoice_regex(s, pattern=r"(?i)Answer\s*:\s*([A-J])")

answer_line(sample)

Extracts the text after “Answer:” and compares with math normalization.

1from nemo_evaluator import answer_line, ScorerInput
2
3s = ScorerInput(response="Step 1: ...\nAnswer: 42", target="42")
4answer_line(s)  # {"correct": True, "extracted": "42"}

numeric_match(sample)

Extracts the last number in the response.

1from nemo_evaluator import numeric_match, ScorerInput
2
3s = ScorerInput(response="The total is 20 + 22 = 42", target="42")
4numeric_match(s)  # {"correct": True, "extracted": "42"}

fuzzy_match(sample)

Normalized substring containment. Supports multiple correct answers via metadata["correct_answers"].

1from nemo_evaluator import fuzzy_match, ScorerInput
2
3s = ScorerInput(response="The capital is Canberra.", target="Canberra",
4                metadata={"correct_answers": ["Canberra", "canberra"]})
5fuzzy_match(s)  # {"correct": True, "extracted": "The capital is Canberra."}

code_sandbox(sample)

Runs code in a Docker container with network isolation, memory limits, and timeouts. Extracts code from markdown fences, concatenates with prompt code and test harness, and checks the exit code.

1from nemo_evaluator import code_sandbox, ScorerInput
2
3s = ScorerInput(
4    response="```python\ndef add(a, b):\n    return a + b\n```",
5    target="add",
6    metadata={"_prompt": "def add(a, b):\n", "_test": "assert add(1, 2) == 3",
7              "entry_point": "add"},
8)
9code_sandbox(s)  # {"correct": True, "extracted": "def add(a, b):\n    return a + b"}

Requires Docker daemon access.

needs_judge(sample)

Signals that this sample requires LLM-as-judge scoring. Returns {"correct": False, "needs_judge": True} so the eval loop’s judge post-processor handles it.

Used by SimpleQA and HealthBench.

LLM-as-Judge Pipeline

Benchmarks that use needs_judge() are scored in a post-processing step by scoring/judge.py. The judge pipeline:

  1. Collects all samples flagged with needs_judge: True
  2. Constructs judge prompts from the response and expected answer
  3. Calls the judge model (configured via --judge-url / JudgeScoringConfig)
  4. Parses the judge verdict and updates rewards

Configure the judge model:

$nel eval run --bench simpleqa \
>  --model-url https://api.example.com/v1 --model-id my-model \
>  --judge-url https://api.example.com/v1 --judge-id gpt-4o

JSON Schema Scoring

scoring/json_schema.py validates structured model outputs against a JSON schema:

1from nemo_evaluator.scoring import validate_json_schema
2
3result = validate_json_schema(response_text, schema={"type": "object", "required": ["answer"]})
4# {"valid": True, "extracted": {"answer": "42"}, "score": 1.0}

Metrics

pass@k

Standard Codex-style pass@k. Given n attempts per problem with c correct:

pass@k=1−(n−ck)(nk)\text{pass@k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}

1from nemo_evaluator.metrics import pass_at_k
2
3pass_at_k(n=8, c=3, k=1)   # probability at least 1/1 is correct
4pass_at_k(n=8, c=3, k=4)   # probability at least 1/4 is correct

Bootstrap Confidence Intervals

95% CI via bootstrap resampling (10,000 iterations):

1from nemo_evaluator.metrics import bootstrap_ci
2
3ci = bootstrap_ci(scores)
4print(f"pass@1: {ci.value:.4f} [{ci.ci_lower:.4f}, {ci.ci_upper:.4f}]")

Category Breakdown

When problems include category metadata, per-category accuracy is computed automatically:

1from nemo_evaluator.metrics.aggregation import category_breakdown
2
3cats = category_breakdown(results, "category")
4for c in cats:
5    print(f"{c.category}: {c.mean_reward:.3f} ({c.n_samples} samples)")

Failure Analysis

The ArtifactCollector categorizes failures automatically:

CategoryDetection
infra_errorscoring_details.error_category == "infra_error" (set when the solver raises InfraError or returns error_kind=ErrorKind.INFRA)
solve_timeoutscoring_details.error_category == "solve_timeout" (set when the solver returns error_kind=ErrorKind.SOLVE_TIMEOUT after agent.run() exceeded the wall-clock budget; verifier still runs on the partial workspace)
timeoutModel error contains “timeout”
rate_limitModel error contains “429” or “rate”
refusalResponse contains “I cannot”, “I’m sorry”
emptyEmpty response or whitespace-only
format_errorNon-empty response but no answer extracted

Output in failure_analysis.json:

1{
2  "total_failures": 12,
3  "failure_rate": 0.06,
4  "categories": {
5    "refusal": {"count": 5, "rate": 0.025},
6    "format_error": {"count": 4, "rate": 0.02},
7    "timeout": {"count": 3, "rate": 0.015}
8  },
9  "exemplars": [
10    {"category": "refusal", "problem_idx": 42, "response_preview": "I'm sorry, I cannot..."}
11  ]
12}