Per-Request Speculative-Decoding Acceptance

When an inference server runs speculative decoding, it can report how well the draft model did on each request: how many draft tokens it proposed, how many were accepted, and the distribution of accepted-draft counts per verify step. AIPerf captures this as an engine-neutral per-request record so the metrics layer can reason about acceptance without knowing which engine produced it.

This page documents the record, the adapter interface that fills it, and the vLLM adapter (the first supported engine). SGLang and TensorRT-LLM adapters are future work and reuse the same record.

The engine-neutral record

SpecDecodeAcceptanceRecord (src/aiperf/common/models/spec_decode_models.py) is one record per request, attached to ParsedResponseRecord.spec_decode_acceptance. It is deliberately tree-agnostic (a histogram, not per-position arrays) and adaptive-safe (no fixed k assumption) so it survives variable-length drafting such as DSpark-style adaptive verification.

Descriptions here are engine-neutral; how a specific engine populates them (field names, which level emits the per-step arrays, counting caveats) lives in that engine’s section below.

The adapter interface

An adapter is the only component that knows an engine’s on-the-wire spec-decode shape. It reads the raw payload captured on the parsed responses (ParsedResponse.spec_decode_stats) and returns a SpecDecodeAcceptanceRecord, so nothing engine-specific leaks into the metrics layer.

Adapters are a plugin category (spec_decode_adapter) and are resolved by auto-detection, mirroring custom-dataset-loader detection: the parser walks registered adapters in priority order and uses the first whose can_adapt recognizes the payload by its engine-specific signature — so an adapter claims only its own payloads and defers on a foreign one. Both methods are classmethods (adapters are stateless).

1
@runtime_checkable
2
class SpecDecodeAdapterProtocol(Protocol):
3
    @classmethod
4
    def can_adapt(cls, responses: list[ParsedResponse]) -> bool: ...
5
    @classmethod
6
    def adapt(cls, responses: list[ParsedResponse]) -> SpecDecodeAcceptanceRecord | None: ...

The vLLM adapter

VLLMSpecDecodeAdapter reads vLLM’s per-choice speculative_decoding_stats object, emitted when the server runs with --per-request-spec-decode-stats (summary or detailed). The field names and shape track vLLM PR #48915; its Per-Request Acceptance Metrics feature doc is the authoritative wire-format reference. It is present on chat and completions, streaming and non-streaming; in streaming it rides the finish-reason chunk’s choice, which AIPerf already parses, so no extra endpoint code is needed.

The wire object maps to the record one-to-one, except:

• Histogram keys are JSON strings and are int-cast into the record.
• completion_tokens comes from the response usage, not the payload. In streaming that usage rides the trailing include_usage chunk, which AIPerf auto-injects only when endpoint.use_server_token_count is enabled; otherwise (or whenever the server omits usage) completion_tokens stays None.
• num_draft_tokens is vLLM’s post-adjustment count: drafts invalidated by structured-output/grammar constraints are already subtracted server-side.
• num_spec_tokens is always present (the configured num_speculative_tokens); vLLM’s DSpark/DFlash drafters are fixed-block, so k stays defined even there.
• The detailed level adds per_step_accepted / per_step_drafted; summary omits them (they stay None).
• mean_acceptance_length / draft_acceptance_rate are taken verbatim (the server already computes them safely, including the zero-step case).

Missing-field and edge cases

• Field absent (spec decode off, or the request had no verify steps): the record is None and dependent metrics simply do not show. This is the common case and is not an error.
• Zero-step / fully-rejected: reported verbatim (empty or {0: N} histogram, mean_acceptance_length == 1.0).
• Malformed payload: the adapter degrades to None rather than raising, so one bad response cannot abort a run. Records whose aggregate counts contradict each other (histogram not summing to num_spec_steps, etc.) are rejected the same way.
• n > 1: when a request produces multiple sequences, each choice carries its own per-sequence stats, but completion_tokens is request-level. Rather than mix one sequence’s acceptance with all sequences’ token count, the record is suppressed (None) for n > 1, mirroring how per-request timing metrics are suppressed for multi-sequence requests.
• Behind Dynamo the custom field is currently stripped, so this path is direct-to-vLLM only.