Equivalence Match

Use an equivalence-match verifier when correctness can be decided deterministically from the agent’s output and task metadata. The verifier:

1. Extracts the answer-bearing part of the response.
2. Converts the prediction and reference to comparable representations.
3. Applies a task-appropriate equality or similarity rule.
4. Maps the result to a reward.

This pattern is usually faster, cheaper, and more reproducible than an LLM-as-Judge. The main design risk is choosing an equivalence rule that is either too strict and rejects valid answers or too permissive and accepts wrong ones.

Start With the Narrowest Correct Rule

Prefer the simplest rule that captures all valid answers for the task:

1. Exact match for closed answer sets such as multiple-choice letters.
2. Normalized match when formatting differences are irrelevant.
3. Numeric tolerance for floating-point results.
4. Symbolic equivalence when different mathematical forms can represent the same value.
5. Continuous similarity only when correctness is naturally graded rather than binary.

Keep extraction separate from comparison. This makes failures observable: you can distinguish “no answer was extracted” from “an answer was extracted but did not match.”

Exact String Match

Exact matching is appropriate after extraction has reduced the output to a canonical value, such as one allowed multiple-choice letter. Do not compare the entire free-form response when reasoning or formatting may vary.

This example is condensed from mcqa:

1
ANSWER_PATTERN = re.compile(r"(?i)answer\s*:\s*([A-Z])\b")
2
3
def extract_choice(text: str, allowed: set[str]) -> str | None:
4
    match = ANSWER_PATTERN.search(text)
5
    if not match:
6
        return None
7
    choice = match.group(1).upper()
8
    return choice if choice in allowed else None
9
10
prediction = extract_choice(response.output_text, {"A", "B", "C", "D"})
11
gold = expected_answer.strip().upper()
12
reward = 1.0 if prediction is not None and prediction == gold else 0.0

Validate extracted values against the allowed answer set. Otherwise an overly broad regular expression can turn incidental prose into a prediction.

Examples:

• mcqa supports boxed, Answer:, Markdown-aware, and dataset-supplied extraction expressions before exact letter comparison.
• gpqa_diamond extracts a choice and compares it with the reference answer.
• circle_count extracts a discrete count and assigns a binary reward.

Normalized Text Match

Normalize only differences that the task declares irrelevant. Common operations include lowercasing, removing punctuation or articles, canonicalizing Unicode, and collapsing whitespace.

This SQuAD-style normalization is condensed from hotpotqa_qa:

1
def normalize_answer(text: str) -> str:
2
    text = text.lower()
3
    text = "".join(ch for ch in text if ch not in string.punctuation)
4
    text = re.sub(r"\b(a|an|the)\b", " ", text)
5
    return " ".join(text.split())
6
7
prediction = extract_answer(response.output_text)
8
reward = float(
9
    any(normalize_answer(prediction) == normalize_answer(answer) for answer in accepted_answers)
10
)

Store legitimate aliases as reference data or generate them with explicit, reviewable rules. Avoid open-ended “synonym” replacement: it can silently make semantically different answers equal.

Examples:

• hotpotqa_qa parses an answer from JSON, applies SQuAD-style normalization, and supports curated surface-form alternatives.
• equivalence_rule lowercases and collapses whitespace before exact or fuzzy comparison.
• ruler uses task-specific deterministic answer matching for long-context retrieval tasks.

Numeric Match With Tolerance

Floating-point strings can differ while representing effectively the same result. First try normalized exact equality, then parse finite numeric values and compare them with an explicit relative and absolute tolerance.

This example follows math_with_code:

1
import math
2
3
def numeric_match(actual: str | None, expected: str) -> bool:
4
    if actual is None:
5
        return False
6
7
    actual_n = normalize_math_text(actual)
8
    expected_n = normalize_math_text(expected)
9
    if actual_n == expected_n:
10
        return True
11
12
    try:
13
        actual_f = float(actual_n)
14
        expected_f = float(expected_n)
15
    except (ValueError, OverflowError):
16
        return False
17
18
    return math.isfinite(actual_f) and math.isfinite(expected_f) and math.isclose(
19
        actual_f,
20
        expected_f,
21
        rel_tol=1e-6,
22
        abs_tol=1e-6,
23
    )

Choose tolerances from the task’s units and expected numerical precision, not as a universal constant. Reject non-finite values unless the benchmark explicitly permits them.

Examples:

• math_with_code normalizes math delimiters, checks exact equality, then applies a relative numeric tolerance.
• litmus_agent supports isclose, absolute-window, and relative-window numeric grading rules.

Regex and Boxed Extraction

Many benchmarks ask the model to mark its final answer with \boxed{...} or Answer:. Extraction is not itself verification; it creates the candidate that a later comparison rule scores.

Use the last final-answer marker so intermediate reasoning does not win. For boxed LaTeX, track brace depth instead of using a regular expression that stops at the first nested }:

1
def extract_last_boxed(text: str) -> str | None:
2
    marker = r"\boxed{"
3
    start = text.rfind(marker)
4
    if start < 0:
5
        return None
6
7
    depth = 1
8
    answer_start = start + len(marker)
9
    for index, char in enumerate(text[answer_start:], start=answer_start):
10
        depth += (char == "{") - (char == "}")
11
        if depth == 0:
12
            answer = text[answer_start:index].strip()
13
            return answer or None
14
    return None
15
16
prediction = extract_last_boxed(response.output_text)
17
reward = float(prediction is not None and answers_match(prediction, expected_answer))

Examples:

• mcqa handles nested boxed answers, Answer: payloads, and custom extraction regexes.
• math_with_code searches the final assistant response and tool output for the last boxed answer.
• format_verification checks task-specific response formatting.

Symbolic Equivalence

String equality is insufficient for mathematics: x(x + 1) and x^2 + x are different strings but equivalent expressions. Use a maintained parser and symbolic verifier with time and concurrency limits.

math_with_judge uses the math-verify library, which parses expressions and uses symbolic math checks:

1
from math_verify import grader
2
from math_verify.metric import math_metric
3
from math_verify.parser import ExprExtractionConfig, LatexExtractionConfig
4
5
verify_math = math_metric(
6
    gold_extraction_target=(LatexExtractionConfig(),),
7
    pred_extraction_target=(ExprExtractionConfig(), LatexExtractionConfig()),
8
)
9
10
score, extracted = verify_math(
11
    [rf"\boxed{{{expected_answer}}}"],
12
    [generated_answer],
13
)
14
reward = float(score)
15
16
# The underlying parsed expressions can also be checked directly when present.
17
if extracted is not None:
18
    gold_candidates, prediction_candidates = extracted
19
    equivalent = any(
20
        grader.verify(gold, prediction)
21
        for gold in gold_candidates
22
        for prediction in prediction_candidates
23
    )

Symbolic simplification can be expensive or hang on adversarial expressions. Run it with a timeout and bounded concurrency. Treat parser failure separately from a valid but non-equivalent expression.

Examples:

• math_with_judge runs library verification in a killable subprocess and optionally falls back to an LLM judge.
• imo_proofbench_judge combines deterministic math handling with proof-oriented judging.

Fuzzy and Continuous Similarity

Use continuous similarity when partial overlap is meaningful, such as translation or long-form retrieval. Return a score in [0, 1] so it can be used directly as a dense reward.

This example follows mrcr:

1
from difflib import SequenceMatcher
2
3
def similarity(response: str, expected: str, required_prefix: str) -> float:
4
    if not response.startswith(required_prefix):
5
        return 0.0
6
7
    prediction = response.removeprefix(required_prefix)
8
    reference = expected.removeprefix(required_prefix)
9
    return float(SequenceMatcher(None, prediction, reference).ratio())
10
11
reward = similarity(response.output_text.strip(), expected_answer, random_prefix)

Different metrics encode different notions of similarity. Pin metric versions and tokenizers, and do not compare scores produced with different configurations.

Examples:

• mrcr uses SequenceMatcher.ratio() after enforcing a required prefix.
• equivalence_rule offers normalized sequence similarity and a weighted prefix variant.
• wmt_translation computes sentence chrF and spBLEU, using normalized chrF as the reward.

Structured Output Match

Parse structured output before comparing it. Depending on the task, correctness can mean either:

• Value equality: the parsed object must equal a reference object.
• Shape validity: the parsed object must satisfy a schema, even when many values are valid.

This grid-equality example is condensed from arc_agi:

1
def parse_grid(candidate: str) -> list[list[int]] | None:
2
    try:
3
        grid = json.loads(candidate)
4
    except json.JSONDecodeError:
5
        return None
6
7
    if not (
8
        isinstance(grid, list)
9
        and grid
10
        and all(isinstance(row, list) and row for row in grid)
11
        and all(isinstance(cell, int) for row in grid for cell in row)
12
    ):
13
        return None
14
    return grid
15
16
prediction = parse_grid(extract_grid_text(response.output_text))
17
reward = float(prediction is not None and prediction == expected_output)

For schema-only tasks, parse the response and validate the resulting object:

1
response_obj = json.loads(response.output_text)
2
validate_against_schema_openapi(response_obj, strict_schema)
3
reward = 1.0

Examples:

• arc_agi extracts a JSON grid, validates integer cells, and compares the nested lists.
• structured_outputs parses JSON, YAML, XML, TOML, CSV, or tool arguments and validates them against a strict OpenAPI schema.

Instruction and Constraint Checking

Some tasks define correctness as satisfying a set of programmatic constraints rather than matching one answer. Build one checker per instruction, retain each boolean result for diagnostics, and then choose binary or fractional aggregation.

This example is condensed from instruction_following:

1
checks: list[bool] = []
2
for instruction_id, kwargs in zip(instruction_ids, instruction_kwargs):
3
    instruction_cls = instructions_registry.INSTRUCTION_DICT[instruction_id]
4
    instruction = instruction_cls(instruction_id)
5
    instruction.build_description(**{k: v for k, v in kwargs.items() if v is not None})
6
    checks.append(bool(instruction.check_following(response.output_text)))
7
8
if grading_mode == "binary":
9
    reward = float(all(checks))
10
else:
11
    reward = sum(checks) / len(checks) if checks else 0.0

Examples:

• instruction_following runs registry-backed checks and supports all-or-nothing or fractional reward.
• ifbench records strict and loose instruction checks and supports binary or fractional grading.

Implement the Pattern in verify()

A resources server should return both the reward and enough structured evidence to debug it:

1
class AnswerVerifyResponse(BaseVerifyResponse):
2
    extracted_answer: str | None
3
    extraction_successful: bool
4
5
async def verify(self, body: AnswerVerifyRequest) -> AnswerVerifyResponse:
6
    prediction = extract_answer(body.response.output_text)
7
    matched = prediction is not None and compare(prediction, body.expected_answer)
8
9
    return AnswerVerifyResponse(
10
        **body.model_dump(),
11
        reward=float(matched),
12
        extracted_answer=prediction,
13
        extraction_successful=prediction is not None,
14
    )

Keep the reference answer in typed request metadata, not in the prompt text alone. Preserve the extracted prediction and any component scores in the verify response so reward profiling can reveal extraction failures, overly strict normalization, and partial-credit behavior.

Test the Decision Boundary

For every equivalence verifier, add tests for:

• A canonical correct answer.
• A valid alternative representation.
• A near miss that must remain incorrect.
• Missing, malformed, and ambiguous answers.
• Multiple final-answer markers.
• Extreme numeric values or deeply nested structured input where applicable.
• Empty output and unexpected response item types.

The most important tests sit at the boundary between accepted variation and false positives. A verifier that accepts the happy path but over-rewards near misses will produce misleading evaluation metrics and a weak training signal.

Related Topics

• Build Verifiers — resources server interfaces and verifier responsibilities
• Verification Patterns — choose another scoring pattern
• LLM-as-Judge — evaluate semantic or rubric-based criteria that deterministic rules cannot express reliably
• Multi-Reward Verification — expose several component scores from one verifier