Accuracy Benchmarking

Run accuracy evaluation alongside performance profiling using the --accuracy-benchmark flag.

Quick Start

$
# MMLU benchmark with 5-shot prompting (chat endpoint, aligned with lighteval)
$
aiperf profile Qwen/Qwen2.5-1.5B-Instruct \
>
  --url http://localhost:8000 \
>
  --endpoint-type chat \
>
  --accuracy-benchmark mmlu \
>
  --accuracy-n-shots 5 \
>
  --num-requests 15000 \
>
  --concurrency 10 \
>
  --extra-inputs '{"temperature": 0, "stop": ["\n"]}'

$
# AIME competition math — defaults match the trt-llm benchmark recipe
$
# (8-shot, chain-of-thought on, sympy-backed math grader)
$
aiperf profile Qwen/Qwen2.5-7B-Instruct \
>
  --url http://localhost:8000 \
>
  --endpoint-type chat \
>
  --accuracy-benchmark aime \
>
  --num-requests 30 \
>
  --concurrency 10 \
>
  --extra-inputs '{"temperature": 0}'

trt-llm reference alignment

The aime benchmark is aligned with the trt-llm benchmark recipe’s DeepEval-backed AIME path (trt-llm-benchmark-recipe/src/accuracy/aime/):

• Dataset: Maxwell-Jia/AIME_2024, train split.
• Defaults: n_shots=8, enable_cot=True (the recipe enforces n_shots <= 8 and aiperf raises ValueError if you exceed it).
• Prompt format: byte-equal to AIMETemplate.generate_output — **Problem**: ... **Solution**: ... **Answer**: ... blocks for few-shots (Solution only when CoT is on), trailing Let's think step-by-step. after the final **Answer**:.
• System prompt (auto-injected): "Please reason step by step, and put your final answer within \\boxed{}." This default lives in plugins.yaml under the aime benchmark’s default_system_prompt metadata. Override it with --accuracy-system-prompt 'your prompt here'. Pass --accuracy-system-prompt '' to disable injection.
• Grader: MathGrader with _math_strip.strip_string + sympy/ latex2sympy2-extended math_equal. Requires the [accuracy] extra: uv pip install 'aiperf[accuracy]'. Without those packages installed the grader falls back to a stdlib normalize+Fraction comparison and emits a one-time warning; reference parity is only achieved with the full sympy stack.

Per-benchmark default system prompts

The CLI’s --accuracy-system-prompt flag always wins; the per-benchmark default is only consulted when the flag is unset. An empty-string default in metadata is treated as no default (aiperf doesn’t inject a zero-length system message).

Available Benchmarks

LiveCodeBench (lcb_codegeneration) version pinning

LiveCodeBench publishes monthly snapshots of livecodebench/code_generation_lite as HuggingFace configs (e.g. v4_v5, v6, …). The loader pins a specific subset so accuracy numbers are reproducible across runs and branches; the default is v4_v5 (the same subset lighteval’s reference LCB task treats as its base). Override at runtime via:

$
export AIPERF_ACCURACY_LCB_RELEASE_TAG=v6   # or any published subset

The env var is read at every load_problems call (no module-reload needed) and is passed as the positional name arg to load_dataset("livecodebench/code_generation_lite", name, split="test", trust_remote_code=True) — the standard HF config-name selector, matching lighteval’s hf_subset= usage. trust_remote_code=True is required because LCB still ships a repository loading script; datasets<4 runs it normally. Nothing is bundled with the aiperf wheel — all subsets are fetched on-demand and cached under ~/.cache/huggingface/datasets/.

Compatibility: livecodebench/code_generation_lite requires datasets<4. datasets>=4 dropped support for repository loading scripts entirely, and the loader surfaces a clear error when it detects this:

lcb_codegeneration: cannot load 'livecodebench/code_generation_lite'
on `datasets>=4` — LCB still ships a repository loading script that
`datasets>=4` no longer executes. Pin to an earlier release:
`uv pip install 'datasets<4'`.

If a future LCB release renames or removes the pinned subset, the loader raises RuntimeError prefixed lcb_codegeneration: failed to load …; recover by bumping the env var:

$
export AIPERF_ACCURACY_LCB_RELEASE_TAG=v6   # or whatever LCB now ships

MMLU chain-of-thought and reasoning models

The mmlu benchmark has two prompting modes, selected by --accuracy-enable-cot:

• Non-CoT (default) — lighteval parity. The prompt ends in a bare Answer: trailer and the generation budget is generation_size=5 (mapped to the turn’s max_tokens), with the ["\n"] stop sequence. This is byte-identical to lighteval’s reference MMLU path: the server is expected to emit a single answer letter immediately. Use this for non-reasoning instruct models where you want reference-comparable scores.

• CoT — --accuracy-enable-cot. The instruction is extended with Think step by step and then output the answer in the format of "The answer is (X)" at the end., the query gets a Let's think step by step. primer, and the generation budget is raised to the full generation_size=4000 so the model has room for a reasoning trace before the final The answer is (X) line. The multiple_choice grader parses the trailing letter.

  $
  aiperf profile my-model --url http://localhost:8000 \
  >
    --endpoint-type chat \
  >
    --accuracy-benchmark mmlu \
  >
    --accuracy-enable-cot \
  >
    --num-requests 15000 \
  >
    --concurrency 10 \
  >
    --extra-inputs '{"temperature": 0}'

For reasoning models whose traces are long enough to exhaust the 4000-token budget before reaching the answer line, raise the budget with --extra-inputs '{"max_completion_tokens": 16000}'. The --extra-inputs value overrides the benchmark’s generation_size (which is what the benchmark maps into the turn max_tokens), so the model can finish its reasoning:

$
aiperf profile my-model --url http://localhost:8000 \
>
  --endpoint-type chat \
>
  --accuracy-benchmark mmlu \
>
  --accuracy-enable-cot \
>
  --num-requests 15000 \
>
  --concurrency 10 \
>
  --extra-inputs '{"temperature": 0, "max_completion_tokens": 16000}'

Troubleshooting: 0% / all-unparsed against a reasoning model

An MMLU run that scores near 0% with (almost) every response flagged unparsed against a reasoning model is expected in non-CoT mode. The non-CoT prompt asks for a single answer letter under a 5-token budget, but a reasoning model emits chain-of-thought that never reaches (or is truncated before) a parseable letter, so extraction falls through every tier. This is not a grader bug. Fix it by giving the model room to reason:

• add --accuracy-enable-cot (MMLU’s CoT mode, full 4000-token budget), or
• switch to the CoT-native mmlu_pro benchmark (below).

MMLU-Pro

The mmlu_pro benchmark ports TIGER-AI-Lab’s MMLU-Pro (evaluate_from_api.py) at parity:

• Dataset: TIGER-Lab/MMLU-Pro. Test split provides the graded questions; the validation split provides the per-category CoT few-shots.
• Categories (14): biology, business, chemistry, computer science, economics, engineering, health, history, law, math, philosophy, physics, psychology, other. Restrict with --accuracy-tasks (e.g. --accuracy-tasks math,physics); omit for all 14.
• Options: up to 10 per question, labeled A-J (N/A placeholder options are filtered out before lettering).
• Defaults: default_n_shots: 5, default_enable_cot: true, default_grader: mmlu_pro. MMLU-Pro is CoT-native — the per-category instruction always requests the "The answer is (X)" format and the generation budget is generation_size=4000.
• Grader (mmlu_pro): extracts the final A-J letter via the upstream 3-tier cascade — answer is (X) -> Answer: X -> the last lone in-range letter. A response parsed by a fallback tier (or not at all) is flagged unparsed. No optional dependencies are required.

Because MMLU-Pro defaults to CoT, it works with reasoning models out of the box; as with MMLU CoT, raise the budget via --extra-inputs '{"max_completion_tokens": 16000}' if long reasoning traces get truncated before the answer line.

A non-CoT variant is available via --accuracy-no-enable-cot, which switches the few-shots and the query to a bare Answer: trailer. This is an AIPerf extension for quick low-latency runs and is not part of upstream MMLU-Pro parity — use the default CoT mode for reference-comparable scores.

$
aiperf profile --model <model> --url <url>/v1 --endpoint-type chat --streaming \
>
  --tokenizer <model> --accuracy-benchmark mmlu_pro --num-requests 200 --concurrency 10 \
>
  --extra-inputs '{"temperature": 0}'

CLI Flags

Endpoint Type: completions vs chat

Both endpoint types are supported. The choice affects prompt format and alignment with reference frameworks:

When --endpoint-type chat is used, MMLU few-shot examples are structured as separate user/assistant message turns (matching lighteval’s PromptManager._prepare_chat_template()). The completions endpoint sends the entire prompt as a single text block.

Temperature: Must be explicitly set to 0 via --extra-inputs '{"temperature": 0}' for deterministic (greedy) decoding. Most LLM servers default to temperature=1.0 when not specified, which introduces random sampling and causes run-to-run variance. lighteval defaults to temperature=0 internally.

Stop sequence: Use --extra-inputs '{"stop": ["\n"]}' to match lighteval’s MMLU behavior (stop at first newline). Can be combined with temperature: --extra-inputs '{"temperature": 0, "stop": ["\n"]}'.

Concurrency: Higher concurrency is faster. --concurrency 10 or above is recommended. Minor run-to-run variance (~0.2% macro) is expected due to GPU floating-point non-determinism; this is independent of concurrency level.

num-requests: Set to at least the total number of benchmark problems (MMLU: 14,042 across 57 subjects).

Examples

$
# Single subject, quick test
$
aiperf profile my-model --url http://localhost:8000 \
>
  --endpoint-type chat \
>
  --accuracy-benchmark mmlu \
>
  --accuracy-n-shots 5 \
>
  --accuracy-tasks abstract_algebra \
>
  --num-requests 100 \
>
  --concurrency 10 \
>
  --extra-inputs '{"temperature": 0, "stop": ["\n"]}'
$
$
# Full MMLU (57 subjects, 14042 problems)
$
aiperf profile my-model --url http://localhost:8000 \
>
  --endpoint-type chat \
>
  --accuracy-benchmark mmlu \
>
  --accuracy-n-shots 5 \
>
  --num-requests 15000 \
>
  --concurrency 50 \
>
  --extra-inputs '{"temperature": 0, "stop": ["\n"]}'
$
$
# Completions endpoint (traditional flat-text format)
$
aiperf profile my-model --url http://localhost:8000 \
>
  --endpoint-type completions \
>
  --accuracy-benchmark mmlu \
>
  --accuracy-n-shots 5 \
>
  --num-requests 15000 \
>
  --concurrency 50 \
>
  --extra-inputs '{"temperature": 0, "stop": ["\n"]}'
$
$
# AIME with explicit math grader and few-shot priming
$
aiperf profile my-model --url http://localhost:8000 \
>
  --endpoint-type chat \
>
  --accuracy-benchmark aime \
>
  --accuracy-grader math \
>
  --accuracy-n-shots 4 \
>
  --num-requests 30 \
>
  --concurrency 10 \
>
  --extra-inputs '{"temperature": 0}'

Graders

The math grader pipeline (aligned with trt-llm-benchmark-recipe/src/accuracy/aime/):

1. Extract the model’s final answer by priority:
   • The contents of the last \boxed{...} in the response (canonical MATH/AIME format).
   • The tail of an “the answer is X” / “answer: X” / “final answer X” phrase, recursively re-parsed for boxed/numeric content.
   • The last numeric literal in the response.
2. Normalize both prediction and gold via the recipe’s strip_string: linebreaks/spacing/quote-style braces collapsed, \dfrac/\tfrac → \frac, \left/\right removed, \text{...} unwrapped, MathQA-derived unit tokens dropped, infinity/percent/months/dollar-sign normalization, trailing .0 decimals trimmed, simple a/b rewritten as \frac{a}{b}.
3. Compare with math_equal (lowercase string equality → choice-prefix unwrap → numerical isclose (abs_tol=1e-4) with percentage variants → brace/paren strip + lowercase compare → equation-form rewrite (f(x) = y ↔ y) → symbolic equivalence via sympy.parsing.sympy_parser.parse_expr and latex2sympy2_extended.latex2sympy).

Symbolic equivalence (e.g. \sqrt{2} ↔ 2^{1/2}, \frac{1}{3} ↔ 0.333333, 1,2,3 ↔ 3,2,1) requires the [accuracy] install:

$
uv pip install 'aiperf[accuracy]'

Without those optional dependencies (sympy, latex2sympy2-extended) the grader falls back to a stdlib normalize + Fraction comparison and emits a single warning the first time it runs. Reference parity with the trt-llm recipe requires the full sympy stack.

When extraction fell back past the \boxed{} step (i.e. the model didn’t follow the boxed-answer instruction), the response is flagged unparsed=True in the per-record output. A correct unparsed response is still scored correct, mirroring multiple_choice’s convention.

Output

Accuracy flows on a dedicated accuracy record-type channel (alongside the metric_records, gpu_telemetry, and server_metrics channels — see Record-Type Channels). Each graded response is routed to two sinks: an accumulator that produces the per-task summary, and a per-record JSONL writer.

Accuracy results are displayed in the console and exported to CSV. The console table and the CSV both carry a per-task Unparsed count (responses where the grader needed a regex fallback because the model output did not match the expected format):

                        Accuracy Benchmark Results
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃ Task                    ┃ Correct ┃ Total ┃ Unparsed ┃ Accuracy ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│ abstract_algebra        │      35 │   100 │        2 │   35.00% │
│ ...                     │     ... │   ... │      ... │      ... │
│ OVERALL                 │    8368 │ 14042 │       61 │   59.59% │
└─────────────────────────┴─────────┴───────┴──────────┴──────────┘

Summary CSV: <artifact_dir>/accuracy_results.csv — one row per task plus a trailing OVERALL row. Columns: task, total, passed, unparsed, accuracy_rate, unparsed_rate.

Per-record accuracy JSONL

Path: <artifact_dir>/accuracy_export.jsonl by default, or <prefix>_accuracy.jsonl when an artifact prefix is configured (see AIPerfConfig.artifacts.accuracy_export_jsonl_file). One JSON object per line, one line per graded response — the full grading detail that the summary CSV and console table roll up. Produced independently by the AccuracyJSONLWriter; it is not affected by the summary/metric bridge that feeds the CSV and console.

Each line is a serialized AccuracyRecordsData (src/aiperf/accuracy/models.py) with these fields, in order:

Three of these fields carry distinct text and are easy to conflate:

• explanation — the grader’s reasoning about the score (why it marked the response right or wrong).
• model_output — the model’s answer content (the answer channel).
• model_thinking — the model’s own chain-of-thought / reasoning_content channel, null when the model emitted no separate reasoning channel.

The full prompt is not embedded in each record: it lives in inputs.json keyed by session_id, which equals this record’s conversation_id. Join on that id to recover the prompt — this avoids duplicating multi-KB prompts on every graded response.

Example line (pretty-printed here; the file emits one compact object per line):

1
{
2
  "session_num": 0,
3
  "conversation_id": "session_000000",
4
  "x_request_id": "de56948f-8736-43e5-b636-303ebee20b20",
5
  "worker_id": "worker_1c12efdd",
6
  "benchmark_phase": "profiling",
7
  "timestamp_ns": 1784176216352916652,
8
  "task": "abstract_algebra",
9
  "grader_name": "multiple_choice",
10
  "passed": false,
11
  "unparsed": false,
12
  "confidence": 0.0,
13
  "expected": "B",
14
  "actual": "D",
15
  "explanation": "first-line-of-response extracted to 'D'; ground_truth stripped to 'B'; match=False",
16
  "model_output": "The answer is (D)",
17
  "model_thinking": "I'll reason about each option in turn. Eliminating the implausible cases narrows it down. Therefore, The answer is (D)"
18
}

Use it for per-response post-hoc analysis — e.g. inspecting exactly what a reasoning model thought before an unparsed answer.

Architecture

All components self-disable when --accuracy-benchmark is not set.