Is this page helpful?

Speculative Decoding#

Speculative decoding speeds up generation by having a small, fast draft predict several tokens ahead, which the large target model then verifies in a single forward pass. When the draft is right, you get multiple tokens for roughly the cost of one; when it is wrong, the target falls back to normal decoding, so output is never degraded. For a conceptual primer, see NVIDIA’s blog post: An Introduction to Speculative Decoding for Reducing Latency in AI Inference.

Note

This guide documents speculative decoding on the vLLM backend.

Key terms#

  • Target model – the large, accurate model you are serving.

  • Draft – the small, fast predictor. It may be a separate model (EAGLE3), extra heads built into the target (MTP), or a model-free prompt-lookup histogram (n-gram).

  • Acceptance rate – the fraction of drafted tokens the target accepts. Higher is better; it is highly dependent on the workload (coding, math, writing, …).

  • Acceptance length – the average number of tokens accepted per step.

Speculative decoding trades memory and compute for latency. It helps most when a GPU has spare compute (for example, low-to-moderate concurrency). Under saturated load with no spare compute, it may not improve, and can reduce, throughput. Always measure on your own traffic – see Benchmark it yourself.

How a NIM chooses a method#

Each NIM ships with a default speculative-decoding method, chosen in this order of preference:

  1. MTP (multi-token prediction) – used when the base model includes MTP heads. No extra model to download.

  2. EAGLE3 – used when NVIDIA has published an EAGLE3 draft for the model with a useful acceptance rate. The draft downloads through the same path as the checkpoint. See NVIDIA’s published drafts: Speculative Decoding Modules collection.

  3. n-gram – the model-free fallback when neither MTP nor a published EAGLE3 draft is available. No extra model to download.

Important

The shipped method is a supported, validated option – not necessarily the optimal one for your workload. Speculative-decoding performance is workload-specific, and an EAGLE3 draft tuned to your traffic will usually beat the default. You can override the built-in configuration or bring your own draft model at any time – see Bring your own draft model. Memory footprint can also drive that choice: speculative decoding always costs extra GPU memory, and the amount differs sharply by method (see the note below the table).

The methods differ in what they load and how much extra GPU memory they need:

Method

Extra model

Extra GPU memory

n-gram

No

Negligible – model-free (a prompt-lookup histogram).

MTP

No

Low – a few extra prediction heads on the target model.

EAGLE3

Yes (a small draft)

Higher – a separate draft model with its own weights and KV cache.

Important

Speculative decoding needs spare GPU memory. Every method drafts and verifies several tokens per step, so enabling it costs more memory than serving the target alone, and the extra footprint grows EAGLE3 >> MTP > n-gram (see the table above). Confirm the GPU has headroom before turning it on – especially for EAGLE3, whose separate draft model dominates the cost. If you hit out-of-memory, reduce the KV-cache footprint (a shorter NIM_MAX_MODEL_LEN, lower concurrency, or vLLM’s --gpu-memory-utilization) or use a lighter method. See GPU memory troubleshooting.

Enable or disable the built-in method#

Speculative decoding is a runtime toggle, not a separate profile. The same profile serves both modes, so you do not need to pick a different profile to turn it on or off.

Variable

Values

Effect

NIM_SPECDEC_ENABLE

0 (default) / 1

Apply the model’s built-in speculative-decoding config (1) or run without it (0).

NIM_SPECDEC_ARGS

JSON object

Override the built-in config (see below). Consumed only when NIM_SPECDEC_ENABLE=1.

Turn the model’s built-in method on:

export IMG=nvcr.io/nim/<org>/<nim>:<tag>   # your NIM image
docker run --gpus all --shm-size=16GB \
  -e NGC_API_KEY \
  -e NIM_SPECDEC_ENABLE=1 \
  -v /tmp/nim_cache:/opt/nim/.cache \
  -p 8000:8000 \
  $IMG

Leave NIM_SPECDEC_ENABLE unset (or 0) to serve the same model without speculative decoding. For EAGLE3 NIMs, the draft model downloads only when the toggle is on, so disabling it also skips that download.

Override the built-in configuration#

NIM_SPECDEC_ARGS is a JSON object of vLLM runtime-config keys that overrides the model’s default speculative config. Use it to tune parameters or switch methods (for example, force n-gram on a model that ships MTP):

docker run --gpus all --shm-size=16GB \
  -e NGC_API_KEY \
  -e NIM_SPECDEC_ENABLE=1 \
  -e NIM_SPECDEC_ARGS='{"speculative_config": "{\"method\": \"ngram\", \"num_speculative_tokens\": 3, \"prompt_lookup_max\": 4, \"prompt_lookup_min\": 1}"}' \
  -v /tmp/nim_cache:/opt/nim/.cache \
  -p 8000:8000 \
  $IMG

Model-free deployments#

In model-free mode (NIM_MODEL_PATH) the NIM has no built-in speculative config, so you supply one yourself. The simplest path is to pass the vLLM speculative config directly as a backend argument.

n-gram (no draft model)#

docker run --gpus all --shm-size=16GB \
  -e NGC_API_KEY \
  -v /tmp/nim_cache:/opt/nim/.cache \
  -p 8000:8000 \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  $IMG \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 3, "prompt_lookup_max": 4, "prompt_lookup_min": 1}'

MTP (model with built-in MTP heads)#

docker run --gpus all --shm-size=16GB \
  -e NGC_API_KEY \
  -v /tmp/nim_cache:/opt/nim/.cache \
  -p 8000:8000 \
  -e NIM_MODEL_PATH=ngc://<org>/<model>:<tag> \
  $IMG \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

Bring your own draft model#

EAGLE3 (and other external-draft methods) need a separate draft model. Point NIM_DRAFT_MODEL_PATH at it and reference the draft by a bare name in the speculative config; the NIM downloads the draft into that subdirectory and rewrites the reference to the local path before launch.

NIM_DRAFT_MODEL_PATH mirrors NIM_MODEL_PATH: it downloads from the same locations (ngc://, hf://, s3://, gs://, or an absolute local path), is cached, and works in air-gapped deployments.

docker run --gpus all --shm-size=16GB \
  -e NGC_API_KEY \
  -v /tmp/nim_cache:/opt/nim/.cache \
  -p 8000:8000 \
  -e NIM_MODEL_PATH=hf://meta-llama/Llama-3.1-8B-Instruct \
  -e NIM_DRAFT_MODEL_PATH=hf://<org>/<your-eagle3-draft> \
  $IMG \
  --speculative-config '{"method": "eagle3", "model": "eagle3_draft", "num_speculative_tokens": 3}'

Here "model": "eagle3_draft" is the bare subdirectory name the draft is materialized into; it does not need to match the remote repository name. To override the draft on a model-specific NIM instead, set NIM_DRAFT_MODEL_PATH alongside NIM_SPECDEC_ENABLE=1 – an explicit draft source takes precedence over the one shipped with the NIM.

Create your own draft model#

The best draft for your workload is usually an EAGLE3 model trained on it. NVIDIA publishes ready-to-use EAGLE3 drafts for common models in the Speculative Decoding Modules collection, or you can train your own – see the guide to training a speculative-decoding model. Export the trained draft and serve it via NIM_DRAFT_MODEL_PATH as shown above.

Local draft model (air-gapped)#

For air-gapped deployments, point NIM_DRAFT_MODEL_PATH at an absolute local directory holding the draft (its config.json + weights). The draft is linked into the workspace with no network access – the same way a local target model is served. Mount the directory into the container and use its in-container path:

docker run --gpus all --shm-size=16GB \
  -v /tmp/nim_cache:/opt/nim/.cache \
  -v /data/models/llama-3.1-8b:/models/target \
  -v /data/models/eagle3-draft:/models/draft \
  -p 8000:8000 \
  -e NIM_MODEL_PATH=/models/target \
  -e NIM_DRAFT_MODEL_PATH=/models/draft \
  $IMG \
  --speculative-config '{"method": "eagle3", "model": "eagle3_draft", "num_speculative_tokens": 3}'

No NGC_API_KEY or HF_TOKEN is required: the target loads from its local path (or a pre-populated /opt/nim/.cache) and the draft links from /models/draft. Keep the bare "model": "eagle3_draft" reference – NIM materializes the local draft into that workspace subdir and rewrites the reference to the absolute path at launch.

Benchmark it yourself#

Because speculative-decoding gains are workload-specific, measure on representative traffic before committing. NVIDIA AIPerf collects acceptance rate, acceptance length, and throughput directly from the server’s metrics endpoint. NIM exposes Prometheus metrics at /v1/metrics, so pass --server-metrics http://localhost:8000/v1/metrics.

With SPEED-Bench (per-category acceptance)#

SPEED-Bench is NVIDIA’s dataset for evaluating speculative decoding across 11 semantic domains (coding, math, writing, …). Prepare the data, then profile per category and assemble a matrix report:

SPEED_BENCH_DIR="./datasets/speed-bench"
curl -LsSf https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py \
  | python3 - --output_dir $SPEED_BENCH_DIR

CATEGORIES="coding humanities math multilingual qa rag reasoning roleplay stem summarization writing"
for cat in $CATEGORIES; do
  aiperf profile \
    --model <served-model-name> \
    --endpoint-type chat --streaming \
    --url localhost:8000 \
    --custom-dataset-type speed_bench_${cat} \
    --input-file ${SPEED_BENCH_DIR}/qualitative.jsonl \
    --server-metrics http://localhost:8000/v1/metrics \
    --osl 4096 --extra-inputs temperature:0 --concurrency 16 \
    --output-artifact-dir ./artifacts/speed_bench_${cat}
done

aiperf speed-bench-report ./artifacts/ --format both          # acceptance length matrix
aiperf speed-bench-report ./artifacts/ --metric accept_rate   # acceptance rate matrix
aiperf speed-bench-report ./artifacts/ --metric throughput    # throughput matrix

To quantify speedup, run the same matrix with NIM_SPECDEC_ENABLE=0 (acceptance is then 0) and compare the throughput tables.

Replay your own traffic#

To benchmark against your production traffic rather than a public dataset, capture it with NIM’s built-in payload capture and replay it with AIPerf:

  1. Capture real requests by enabling payload capture (off by default; it records exact prompts – see the payload capture reference). With format: mooncake_payload, NIM writes a Mooncake payload-mode trace ({"timestamp": <ms>, "payload": <request>} per line) of the actual requests:

    docker run --gpus all --shm-size=16GB \
  -e NGC_API_KEY \
  -e NIM_SPECDEC_ENABLE=1 \
  -e NIM_CAPTURE_ENABLE=1 \
  -e NIM_CAPTURE_ARGS='{"path": "/captures/traffic.jsonl", "format": "mooncake_payload", "endpoint_pattern": "^/v1/chat/completions$"}' \
  -v /data/captures:/captures \
  -p 8000:8000 \
  $IMG

    The endpoint_pattern scopes capture to one endpoint. AIPerf replays each payload verbatim to the single --endpoint-type you pass below, so a trace must contain one endpoint’s payloads; see the payload capture reference for replaying completion traffic.

  2. Replay the captured trace with AIPerf, collecting server-side spec metrics:

    aiperf profile \
  --model <served-model-name> \
  --endpoint-type chat --streaming \
  --url localhost:8000 \
  --custom-dataset-type mooncake_trace \
  --input-file /data/captures/traffic.jsonl \
  --server-metrics http://localhost:8000/v1/metrics

    AIPerf loads a tokenizer for the model to compute token metrics; if it cannot be fetched (private model name or air-gapped host), pass a local tokenizer directory with --tokenizer <dir> (or --tokenizer builtin).

  3. Read the results. AIPerf’s console report covers latency and throughput (time to first token, request latency, output token throughput); it does not report speculative-decoding acceptance. Those come from the server-side counters that --server-metrics records (written to server_metrics_export.csv/.json, and also served live on /v1/metrics). On vLLM:

    Counter

    Meaning

    vllm:spec_decode_num_draft_tokens_total

    draft tokens proposed

    vllm:spec_decode_num_accepted_tokens_total

    draft tokens accepted

    vllm:spec_decode_num_drafts_total

    draft steps

    Acceptance rate (AR) = accepted / draft tokens; acceptance length (AL) = 1 + accepted / drafts (mean tokens emitted per step). SGLang exposes equivalent acceptance counters under its own metric names. Re-run with NIM_SPECDEC_ENABLE=0 to compare throughput and latency on the same traffic.

Because capture stores the real payloads (not hashed/synthetic prompts), the replay reflects your true workload. For raw_payload capture use --custom-dataset-type raw_payload instead. See the payload capture reference and the AIPerf SPEED-Bench tutorial.

See also#