> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.speculative.bench_vllm

Offline acceptance / speedup benchmark for a trained drafter served by vLLM.

This is the vLLM companion to `bench_sglang`: the same chat-completions
workload machinery (which it imports), with the acceptance statistics read from
vLLM's Prometheus `/metrics` endpoint instead of SGLang's `/server_info`.
It covers every drafter `serve_vllm` can launch -- EAGLE-3, P-EAGLE, and the
DFlash family -- and reports:

* `accept_length` -- mean tokens emitted per target verification step,
  including the one guaranteed bonus token:
  `1 + num_accepted_tokens / num_drafts`.
* `acceptance_rate` -- fraction of proposed draft tokens accepted:
  `num_accepted_tokens / num_draft_tokens`.
* `output_throughput_tok_s` -- measured decode throughput (output tokens per
  wall-clock second).
* `speedup` -- optional: throughput divided by the same workload's throughput
  against a `--baseline-server` running *without* speculation.

The spec-decode counters (`vllm:spec_decode_num_drafts_total`,
`vllm:spec_decode_num_draft_tokens_total`,
`vllm:spec_decode_num_accepted_tokens_total`) are snapshotted before and after
the workload and differenced, so -- unlike SGLang's server-cumulative
`avg_spec_accept_length` -- the numbers cover exactly this benchmark's
requests even on a server that has already handled other traffic.

Typical usage (after `serve_vllm` launches the drafter on port 8000):

python -m nemo\_automodel.components.speculative.bench\_vllm \
\--server [http://localhost:8000](http://localhost:8000) \
\--model Qwen/Qwen3-8B \
\--input-data Aeala/ShareGPT\_Vicuna\_unfiltered \
\--num-prompts 64 --concurrency 16 --max-new-tokens 256

Add `--baseline-server http://localhost:8001` (a second server started
without `--speculative-config`) to also report the end-to-end speedup.

vLLM is intentionally NOT a dependency of this script -- it talks to the server
over HTTP, so only `aiohttp` is required (already pulled in by the project).
The server itself must be running separately; see `serve_vllm`.

## Module Contents

### Classes

| Name                                                                           | Description                                             |
| ------------------------------------------------------------------------------ | ------------------------------------------------------- |
| [`SpecMetrics`](#nemo_automodel-components-speculative-bench_vllm-SpecMetrics) | One snapshot of vLLM's cumulative spec-decode counters. |

### Functions

| Name                                                                                           | Description                                                                         |
| ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| [`_accept_length`](#nemo_automodel-components-speculative-bench_vllm-_accept_length)           | Mean tokens per verify step incl. the bonus token: `1 + accepted / drafts`.         |
| [`_acceptance_rate`](#nemo_automodel-components-speculative-bench_vllm-_acceptance_rate)       | Fraction of proposed draft tokens accepted: `accepted / draft_tokens`.              |
| [`_build_parser`](#nemo_automodel-components-speculative-bench_vllm-_build_parser)             | -                                                                                   |
| [`_fetch_spec_metrics`](#nemo_automodel-components-speculative-bench_vllm-_fetch_spec_metrics) | GET `<server>/metrics` and parse the spec-decode counters; `None` on failure. |
| [`_parse_spec_metrics`](#nemo_automodel-components-speculative-bench_vllm-_parse_spec_metrics) | Parse the spec-decode counters out of a Prometheus exposition payload.              |
| [`_run`](#nemo_automodel-components-speculative-bench_vllm-_run)                               | Async driver: compute the benchmark summary and report it. Returns an exit code.    |
| [`_run_summary`](#nemo_automodel-components-speculative-bench_vllm-_run_summary)               | Validate args, run the workload(s), and return the metrics dict.                    |
| [`_summarize`](#nemo_automodel-components-speculative-bench_vllm-_summarize)                   | Assemble the metrics dict reported to stdout / `--output-json`.                     |
| [`main`](#nemo_automodel-components-speculative-bench_vllm-main)                               | CLI entry point. Parses `argv` and returns the process exit code.                   |

### Data

[`_METRIC_NUM_ACCEPTED_TOKENS`](#nemo_automodel-components-speculative-bench_vllm-_METRIC_NUM_ACCEPTED_TOKENS)

[`_METRIC_NUM_DRAFTS`](#nemo_automodel-components-speculative-bench_vllm-_METRIC_NUM_DRAFTS)

[`_METRIC_NUM_DRAFT_TOKENS`](#nemo_automodel-components-speculative-bench_vllm-_METRIC_NUM_DRAFT_TOKENS)

[`_SPEC_METRIC_NAMES`](#nemo_automodel-components-speculative-bench_vllm-_SPEC_METRIC_NAMES)

[`logger`](#nemo_automodel-components-speculative-bench_vllm-logger)

### API

```python
class nemo_automodel.components.speculative.bench_vllm.SpecMetrics(
    num_drafts: float,
    num_draft_tokens: float,
    num_accepted_tokens: float
)
```

Dataclass

One snapshot of vLLM's cumulative spec-decode counters.

```python
nemo_automodel.components.speculative.bench_vllm.SpecMetrics.delta(
    before: nemo_automodel.components.speculative.bench_vllm.SpecMetrics
) -> nemo_automodel.components.speculative.bench_vllm.SpecMetrics
```

Return the counter increase since `before` (clamped at zero per counter).

```python
nemo_automodel.components.speculative.bench_vllm._accept_length(
    delta: nemo_automodel.components.speculative.bench_vllm.SpecMetrics | None
) -> float | None
```

Mean tokens per verify step incl. the bonus token: `1 + accepted / drafts`.

```python
nemo_automodel.components.speculative.bench_vllm._acceptance_rate(
    delta: nemo_automodel.components.speculative.bench_vllm.SpecMetrics | None
) -> float | None
```

Fraction of proposed draft tokens accepted: `accepted / draft_tokens`.

```python
nemo_automodel.components.speculative.bench_vllm._build_parser() -> argparse.ArgumentParser
```

```python
nemo_automodel.components.speculative.bench_vllm._fetch_spec_metrics(
    server: str,
    timeout_s: float
) -> nemo_automodel.components.speculative.bench_vllm.SpecMetrics | None
```

async

GET `<server>/metrics` and parse the spec-decode counters; `None` on failure.

```python
nemo_automodel.components.speculative.bench_vllm._parse_spec_metrics(
    metrics_text: str
) -> nemo_automodel.components.speculative.bench_vllm.SpecMetrics | None
```

Parse the spec-decode counters out of a Prometheus exposition payload.

Values are summed across label sets (multiple engines / model labels), and
`None` is returned when the drafts counter is absent -- i.e. the server is
not running with speculative decoding.

```python
nemo_automodel.components.speculative.bench_vllm._run(
    args: argparse.Namespace
) -> int
```

async

Async driver: compute the benchmark summary and report it. Returns an exit code.

```python
nemo_automodel.components.speculative.bench_vllm._run_summary(
    args: argparse.Namespace
) -> dict[str, typing.Any] | None
```

async

Validate args, run the workload(s), and return the metrics dict.

Returns `None` when no usable prompts were loaded (the caller's cue to
report a failure without raising -- a bad `--num-prompts`/etc. value is a
real programming error and still raises via `_validate_workload_args`).
Split out of `_run` so `bench_sweep` can drive one dataset at a time
without the printing / `--output-json` side effects below.

```python
nemo_automodel.components.speculative.bench_vllm._summarize(
    gen_cfg: nemo_automodel.components.speculative.regenerate.GenerationConfig,
    spec_result: nemo_automodel.components.speculative.bench_common.WorkloadResult,
    metrics_delta: nemo_automodel.components.speculative.bench_vllm.SpecMetrics | None,
    baseline_result: nemo_automodel.components.speculative.bench_common.WorkloadResult | None
) -> dict[str, typing.Any]
```

Assemble the metrics dict reported to stdout / `--output-json`.

```python
nemo_automodel.components.speculative.bench_vllm.main(
    argv: list[str] | None = None
) -> int
```

CLI entry point. Parses `argv` and returns the process exit code.

```python
nemo_automodel.components.speculative.bench_vllm._METRIC_NUM_ACCEPTED_TOKENS = 'vllm:spec_decode_num_accepted_tokens_total'
```

```python
nemo_automodel.components.speculative.bench_vllm._METRIC_NUM_DRAFTS = 'vllm:spec_decode_num_drafts_total'
```

```python
nemo_automodel.components.speculative.bench_vllm._METRIC_NUM_DRAFT_TOKENS = 'vllm:spec_decode_num_draft_tokens_total'
```

```python
nemo_automodel.components.speculative.bench_vllm._SPEC_METRIC_NAMES = (_METRIC_NUM_DRAFTS, _METRIC_NUM_DRAFT_TOKENS, _METRIC_NUM_ACCEPTED_TOKENS)
```

```python
nemo_automodel.components.speculative.bench_vllm.logger = logging.getLogger(__name__)
```