DFlash2 Speculative Decoding | NVIDIA NIM for Alpamayo Documentation

Alpamayo 2 Super NIM includes opt-in DFlash2 profiles that reduce the text generation portion of trajectory latency. Qualify the other tasks you serve separately: grounding can be slower than on the matching baseline profile (see Qualification). DFlash2 proposes seven draft tokens in parallel, and the full Alpamayo model verifies every proposal before it is accepted. Image preparation, multimodal prefill, and action diffusion still run normally, so end-to-end improvement depends on the reasoning length and draft acceptance rate.

DFlash2 profiles are opt-in and load a draft head in addition to the base model, so they use more GPU memory than the matching baseline profile. fp8-dflash2 fits within the same 80 GB minimum as fp8; allow headroom for larger K. bf16-dflash2 needs a larger GPU (see Support Matrix). The action expert uses the same verified terminal position as the matching baseline profile.

Prepare This Shell

Variables used on this page

Run the commands in Bash. The block below defines IMG_NAME, LOCAL_NIM_CACHE, NIM_CONTAINER (alpamayo2-super-dflash2), NIM_URL, GRPC_TARGET and ALPAMAYO_OUT; ALPAMAYO_OUT keeps a value already set by another page. Online launches also need NGC_API_KEY from Quickstart Guide. The custom-head steps add DFLASH2_HEAD. In a new shell, re-run the block below and, for a custom head, the DFLASH2_HEAD export.

This page covers Alpamayo 2 Super. In a new shell, define the image, cache, container name, client target and an output directory:

export IMG_NAME=nvcr.io/nim/nvidia/alpamayo-2-super:1.0.0

export LOCAL_NIM_CACHE=/tmp/alpamayo2-super-cache
export NIM_CONTAINER=alpamayo2-super-dflash2
export NIM_URL=http://localhost:8000
export GRPC_TARGET=localhost:50051
export ALPAMAYO_OUT="${ALPAMAYO_OUT:-$(mktemp -d "${TMPDIR:-/tmp}/alpamayo.XXXXXX")}"
mkdir -p "$LOCAL_NIM_CACHE"
chmod 1777 "$LOCAL_NIM_CACHE"

For a release candidate, use the exact image supplied for that review. Export NGC_API_KEY using Quickstart Guide before an online launch. Reuse the same cache for subsequent launches. Stop any earlier container using that GPU before starting another profile.

Profiles

Select bf16-dflash2 or fp8-dflash2. Both use the packaged draft head and summary metrics. Memory requirements and recommended K ranges are listed in the Support Matrix.

Run with DFlash2

Select the profile by name with NIM_MODEL_PROFILE. The following example uses FP8 DFlash2 on a supported H100 or H200 GPU with at least 80 GB of memory:

Stop any Alpamayo 2 Super container from the other guides first, because they use the same GPU and host ports. The loop skips names that do not exist and removes a container only after it stops:

for name in alpamayo2-super alpamayo2-super-dflash2 \
    alpamayo2-super-byoc alpamayo2-super-stage; do
  if docker container inspect "$name" >/dev/null 2>&1; then
    docker stop -t 60 "$name" && docker rm "$name"
  fi
done

Each container uses exactly one GPU. Replace 0 in --gpus device=0 with the index of the GPU allocated to you; do not use --gpus all.

docker run -d --name "$NIM_CONTAINER" \
    --runtime=nvidia --gpus device=0 --shm-size=16g \
    -e NGC_API_KEY \
    -e NIM_MODEL_PROFILE=fp8-dflash2 \
    -e NIM_ALPAMAYO_TRAJ_SAMPLES=1,2,3 \
    -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
    -p 8000:8000 -p 50051:50051 \
    "$IMG_NAME"

Use bf16-dflash2 instead only on a supported GPU with at least 120 GB of memory. Setting NIM_PRECISION=bf16 or NIM_PRECISION=fp8 selects the corresponding baseline profile; it does not enable DFlash2. The command intentionally omits NIM_PRECISION: setting both NIM_PRECISION and NIM_MODEL_PROFILE stops startup.

The launch command activates K=1, K=2, and K=3. To use other counts, list them in NIM_ALPAMAYO_TRAJ_SAMPLES. See Support Matrix for the recommended range on your GPU. Each configured value adds startup warmup work. When following the quickstart request examples, replace alpamayo2-super in each docker exec or docker cp command with alpamayo2-super-dflash2.

Wait for Readiness

The DFlash2 profile loads its draft head and warms every configured K before it reports ready, so allow several minutes on the first start. Poll readiness, stop if the container exits, and then confirm the profile and the active K values:

ready=false
for attempt in $(seq 1 180); do
  if curl -sf "$NIM_URL/v1/health/ready"; then
    ready=true
    break
  fi
  state=$(docker inspect -f '{{.State.Status}}' "$NIM_CONTAINER" 2>/dev/null)
  [ "$state" = running ] || break
  sleep 5
done
if [ "$ready" = true ]; then
  curl --fail-with-body -sS "$NIM_URL/v1/models" | python3 -c '
import json, sys
model = json.load(sys.stdin)["data"][0]
print(model["profile"], model["speculative_decoding"], model["k_values"])'
else
  echo "NIM did not become ready; inspect the logs below." >&2
  docker logs --tail 200 "$NIM_CONTAINER"
fi

Expected output after the readiness body:

fp8-dflash2 dflash2 [1, 2, 3]

Send a K=2 Request

num_traj_samples must appear in k_values, and values above 1 require a positive temperature. The builder’s default temperature is 0.6. Build and send a K=2 trajectory request from the bundled scene, then summarize the response:

docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
    --endpoint infer --task trajectory --num-traj-samples 2 \
    --sample-dir /workspace/sample_data > "$ALPAMAYO_OUT/trajectory-k2.json"
curl --fail-with-body -sS "$NIM_URL/v1/infer" \
    -H 'Content-Type: application/json' \
    --data-binary "@$ALPAMAYO_OUT/trajectory-k2.json" \
    --output "$ALPAMAYO_OUT/trajectory-k2-response.json"
python3 -c '
import json, sys
r = json.load(open(sys.argv[1]))
spec = r["metrics"]["speculative_decoding"]
print(r["num_samples"], len(r["pred_trajectories"]), len(r["metrics"]["reasoning_texts"]))
print(spec["num_spec_tokens"], spec["draft_acceptance_rate"] > 0)' \
    "$ALPAMAYO_OUT/trajectory-k2-response.json"

Expected output:

2 2 2
7 True

Each of the two samples has 64 points in pred_trajectories and pred_rotations and its own entry in metrics.reasoning_texts; the speculative-decoding counters sum both samples. The first request for a given K can be slower than later ones; send it once untimed before measuring. The gRPC client takes the same count:

docker exec "$NIM_CONTAINER" python3 /opt/nim/test_grpc_api.py \
    --target localhost:50051 --sample-dir /workspace/sample_data \
    --num-traj-samples 2 --temperature 0.6 --summary-only

Observe Acceptance

DFlash2 profiles default NIM_ALPAMAYO_SPEC_METRICS to summary. With summary or detailed, every trajectory result includes a metrics.speculative_decoding object: in the /v1/infer response for task: trajectory and in the raw_json payload of a gRPC Predict response. VQA, meta-action, auto-labeling, and grounding results do not carry the block, and the baseline bf16 and fp8 profiles never emit it. For K greater than one, counters and histogram bins sum all returned samples; rates are calculated from those totals. Detailed per-step arrays concatenate the steps in sample order rather than averaging per-sample rates.

metrics.speculative_decoding fields

Field	Meaning
`num_spec_tokens`	Draft tokens proposed per verification step; seven for the packaged DFlash2 head.
`num_spec_steps`	Verification steps the request needed.
`num_draft_tokens`	Draft tokens proposed over the whole request.
`num_accepted_draft_tokens`	Draft tokens the full model accepted over the whole request.
`draft_acceptance_rate`	`num_accepted_draft_tokens / num_draft_tokens`, between 0 and 1.
`mean_acceptance_length`	Average tokens produced per verification step, counting the verified token; `1 + num_accepted_draft_tokens / num_spec_steps`.
`acceptance_histogram`	Number of verification steps that accepted 0, 1, …, `num_spec_tokens` draft tokens.
`per_step_accepted`, `per_step_drafted`	Per-step accepted and proposed counts. Present only with `NIM_ALPAMAYO_SPEC_METRICS=detailed`.

The HTTP process also exports the request totals from GET /v1/metrics as Prometheus counters labeled with method="dflash2". The counters accumulate across HTTP and gRPC trajectory requests:

alpamayo_spec_decode_draft_tokens_total;
alpamayo_spec_decode_accepted_tokens_total; and
alpamayo_spec_decode_verify_steps_total.

With NIM_ALPAMAYO_SPEC_METRICS set to summary (the default) or detailed, they are exported at zero with method="dflash2" right after startup and increase with each trajectory request:

curl --silent "$NIM_URL/v1/metrics" | grep '^alpamayo_spec_decode'

Expected output (values abridged):

alpamayo_spec_decode_draft_tokens_total{method="dflash2"} ...
alpamayo_spec_decode_accepted_tokens_total{method="dflash2"} ...
alpamayo_spec_decode_verify_steps_total{method="dflash2"} ...

For acceptance rate over time, use rate(alpamayo_spec_decode_accepted_tokens_total[5m]) / rate(alpamayo_spec_decode_draft_tokens_total[5m]). Mean acceptance length is one plus accepted tokens per verification step. Higher acceptance produces more reasoning tokens per step; compare end-to-end latency with the matching baseline to measure the benefit.

Set NIM_ALPAMAYO_SPEC_METRICS=detailed for per-step diagnostics or none to disable both the response block and the counters. Detailed metrics add response data and are intended for diagnosis, not normal production traffic.

Qualification

Compare the DFlash2 profile with its matching baseline on representative requests; use the baseline if DFlash2 does not improve end-to-end latency for your workload.

DFlash2 targets trajectory reasoning and does not speed up every task. Grounding can be slower on a DFlash2 profile than on the matching baseline profile, and grounding requests do not update the speculative-decoding counters. Qualify every task you serve, not only trajectory. For grounding-heavy or mixed deployments, consider the baseline profile.

Bring Your Own DFlash2 Head

This is an advanced workflow for users experienced in training and exporting DFlash2 heads. Use a custom head only if you can verify its compatibility with the exact Alpamayo base checkpoint and evaluate its output quality. Otherwise, use the packaged head. Passing startup checks does not establish model quality.

The NIM does not train or download custom heads. The exported directory needs:

config.json with architecture DFlash2DraftModel and dflash_config.projector_type="dflash2";
the exported safetensors weights, including the DFlash2 candidate selector;
alpamayo_dflash_metadata.json from the Alpamayo export, with a non-empty base_model_config_sha256 that records the base configuration the head was trained for. This file is required; a head without it, or without that hash, is rejected. Do not invent or edit compatibility hashes.

The block size must equal the speculative-token count plus one (eight for packaged profiles). Preserve training/export provenance and compare output quality with the matching baseline on representative scenes.

To start from the packaged head, for example to check the layout before substituting your own weights, copy it to a host directory. From the cache of a DFlash2 start, locate the head’s snapshot directory by its metadata file and copy it, dereferencing links. The repository and snapshot names in the cache path follow the packaged model artifact and change between releases, so find the directory rather than typing it:

export DFLASH2_HEAD=/tmp/alpamayo2-dflash2-head
HEAD_SNAPSHOT=$(find "$LOCAL_NIM_CACHE/ngc/hub" -path '*/snapshots/*' \
    -name alpamayo_dflash_metadata.json -printf '%h\n' | head -n 1)
echo "Packaged head snapshot: ${HEAD_SNAPSHOT:-not found}"
mkdir -p "$DFLASH2_HEAD"
[ -n "$HEAD_SNAPSHOT" ] && cp -rL "$HEAD_SNAPSHOT/." "$DFLASH2_HEAD"

Or from a running DFlash2 container:

mkdir -p "$DFLASH2_HEAD"
docker exec alpamayo2-super-dflash2 tar -chf - -C /opt/nim/workspace/dflash2 . \
    | tar -xf - -C "$DFLASH2_HEAD"

Set DFLASH2_HEAD to an existing host directory that contains the head files before launching. If the host path does not exist, Docker silently creates an empty, root-owned directory and startup rejects the empty head.

Stop and remove the packaged-head container before running a custom head; both launches use the same name and host ports. Then mount the head read-only and override the speculative-model path while selecting a DFlash2 profile:

test -f "$DFLASH2_HEAD/alpamayo_dflash_metadata.json" || \
    echo "DFLASH2_HEAD does not contain an exported head" >&2
for name in alpamayo2-super alpamayo2-super-dflash2 \
    alpamayo2-super-byoc alpamayo2-super-stage; do
  if docker container inspect "$name" >/dev/null 2>&1; then
    docker stop -t 60 "$name" && docker rm "$name"
  fi
done
docker run -d --name alpamayo2-super-dflash2 \
    --runtime=nvidia --gpus device=0 --shm-size=16g \
    -e NGC_API_KEY \
    -e NIM_MODEL_PROFILE=fp8-dflash2 \
    -e NIM_ALPAMAYO_TRAJ_SAMPLES=1,2,3 \
    -e NIM_ALPAMAYO_SPECULATIVE_MODEL=/byoc-dflash2 \
    -v "$DFLASH2_HEAD:/byoc-dflash2:ro" \
    -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
    -p 8000:8000 -p 50051:50051 \
    "$IMG_NAME"

At startup, the NIM verifies the DFlash2 architecture, token block size, tensor shapes, and the base-configuration hash in alpamayo_dflash_metadata.json. Incompatible or incomplete heads are rejected before the service becomes ready, with an error that names NIM_ALPAMAYO_SPECULATIVE_MODEL and the mounted path. Before any download, the NIM also compares each safetensors file of the head with its header. A truncated or damaged head stops startup with NIM_ALPAMAYO_SPECULATIVE_MODEL=... has a truncated shard model.safetensors: expected X bytes, found Y; copy the complete exported head directory again. A custom head cannot be combined with NIM_FT_CHECKPOINT; that combination stops startup with one cannot be combined with NIM_FT_CHECKPOINT line. Setting NIM_ALPAMAYO_SPECULATIVE_MODEL with a baseline profile fails at startup; select NIM_MODEL_PROFILE=bf16-dflash2 or fp8-dflash2 (and unset NIM_PRECISION). GET /v1/models reports the container path of the active draft head as speculative_model: the custom head’s mount path here, or /opt/nim/workspace/dflash2 for the packaged head. It is null when speculative decoding is off. The relaunch keeps the 1,2,3 K list of the packaged-head launch, so the K=2 example above also runs against the custom head; a launch without NIM_ALPAMAYO_TRAJ_SAMPLES accepts only K=1. Run the quickstart requests with alpamayo2-super-dflash2 as the container name and compare latency and outputs with the matching baseline profile before deployment.

Runtime Discovery and Determinism

Use GET /v1/models and read data[0].k_values for the active sample counts. temperature=0 requires num_traj_samples=1. For multiple trajectories, choose a positive temperature and an explicit seed. Increasing K or switching profiles requires a compatible GPU and a restart.