> This page is for version 2.0.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 2.0.0 (Latest) (default): https://docs.nvidia.com/nim/alpamayo/2.0.0/llms.txt
> - 1.0.0: https://docs.nvidia.com/nim/alpamayo/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/alpamayo/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/alpamayo/_mcp/server.

# DFlash2 Speculative Decoding

Alpamayo 2 Super NIM includes opt-in DFlash2 profiles that reduce the text
generation portion of trajectory latency. Qualify the other tasks you serve
separately: grounding can be slower than on the matching baseline profile (see
[Qualification](#qualification)). DFlash2 proposes seven draft tokens
in parallel, and the full Alpamayo model verifies every proposal before it is
accepted. Image preparation, multimodal prefill, and action diffusion still run
normally, so end-to-end improvement depends on the reasoning length and draft
acceptance rate.

DFlash2 profiles are opt-in and load a draft head in addition to the base
model, so they use more GPU memory than the matching baseline profile.
`fp8-dflash2` fits within the same 80 GB minimum as `fp8`; allow headroom
for larger K. `bf16-dflash2` needs a larger GPU (see
[Support Matrix](/nim/alpamayo/support-matrix)).
The action expert uses the same verified terminal position as the matching
baseline profile.

## Prepare This Shell

<Note title="Variables used on this page">

Run the commands in Bash. The block below defines `IMG_NAME`,
`LOCAL_NIM_CACHE`, `NIM_CONTAINER` (`alpamayo2-super-dflash2`),
`NIM_URL`, `GRPC_TARGET` and `ALPAMAYO_OUT`; `ALPAMAYO_OUT` keeps a
value already set by another page. Online launches also need
`NGC_API_KEY` from [Quickstart Guide](/nim/alpamayo/quickstart-guide). The custom-head steps add
`DFLASH2_HEAD`. In a new shell, re-run the block below and, for a custom
head, the `DFLASH2_HEAD` export.

</Note>

This page covers Alpamayo 2 Super. In a new shell, define the image, cache,
container name, client target and an output directory:

```bash
export IMG_NAME=nvcr.io/nim/nvidia/alpamayo-2-super:1.0.0
```


```bash
export LOCAL_NIM_CACHE=/tmp/alpamayo2-super-cache
export NIM_CONTAINER=alpamayo2-super-dflash2
export NIM_URL=http://localhost:8000
export GRPC_TARGET=localhost:50051
export ALPAMAYO_OUT="${ALPAMAYO_OUT:-$(mktemp -d "${TMPDIR:-/tmp}/alpamayo.XXXXXX")}"
mkdir -p "$LOCAL_NIM_CACHE"
chmod 1777 "$LOCAL_NIM_CACHE"
```

For a release candidate, use the exact image supplied for that review.
Export `NGC_API_KEY` using [Quickstart Guide](/nim/alpamayo/quickstart-guide) before an online launch.
Reuse the same cache for subsequent launches. Stop any earlier container using
that GPU before starting another profile.

## Profiles

Select `bf16-dflash2` or `fp8-dflash2`. Both use the packaged draft head
and summary metrics. Memory requirements and recommended K ranges are listed
in the [Support Matrix](/nim/alpamayo/support-matrix).

## Run with DFlash2

Select the profile by name with `NIM_MODEL_PROFILE`. The following example
uses FP8 DFlash2 on a supported H100 or H200 GPU with at least 80 GB of memory:

Stop any Alpamayo 2 Super container from the other guides first, because they
use the same GPU and host ports. The loop skips names that do not exist and
removes a container only after it stops:

```bash
for name in alpamayo2-super alpamayo2-super-dflash2 \
    alpamayo2-super-byoc alpamayo2-super-stage; do
  if docker container inspect "$name" >/dev/null 2>&1; then
    docker stop -t 60 "$name" && docker rm "$name"
  fi
done
```

Each container uses exactly one GPU. Replace `0` in `--gpus device=0` with
the index of the GPU allocated to you; do not use `--gpus all`.

```bash
docker run -d --name "$NIM_CONTAINER" \
    --runtime=nvidia --gpus device=0 --shm-size=16g \
    -e NGC_API_KEY \
    -e NIM_MODEL_PROFILE=fp8-dflash2 \
    -e NIM_ALPAMAYO_TRAJ_SAMPLES=1,2,3 \
    -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
    -p 8000:8000 -p 50051:50051 \
    "$IMG_NAME"
```

Use `bf16-dflash2` instead only on a supported GPU with at least 120 GB of
memory. Setting `NIM_PRECISION=bf16` or `NIM_PRECISION=fp8` selects the
corresponding baseline profile; it does not enable DFlash2. The command
intentionally omits `NIM_PRECISION`: setting both `NIM_PRECISION` and
`NIM_MODEL_PROFILE` stops startup.

The launch command activates K=1, K=2, and K=3. To use other counts, list them
in `NIM_ALPAMAYO_TRAJ_SAMPLES`. See [Support Matrix](/nim/alpamayo/support-matrix) for the
recommended range on your GPU. Each configured value adds startup warmup work.
When following the quickstart request examples, replace
`alpamayo2-super` in each `docker exec` or `docker cp` command with
`alpamayo2-super-dflash2`.

## Wait for Readiness

The DFlash2 profile loads its draft head and warms every configured K
before it reports ready, so allow several minutes on the first start. Poll
readiness, stop if the container exits, and then confirm the profile and the
active K values:

```bash
ready=false
for attempt in $(seq 1 180); do
  if curl -sf "$NIM_URL/v1/health/ready"; then
    ready=true
    break
  fi
  state=$(docker inspect -f '{{.State.Status}}' "$NIM_CONTAINER" 2>/dev/null)
  [ "$state" = running ] || break
  sleep 5
done
if [ "$ready" = true ]; then
  curl --fail-with-body -sS "$NIM_URL/v1/models" | python3 -c '
import json, sys
model = json.load(sys.stdin)["data"][0]
print(model["profile"], model["speculative_decoding"], model["k_values"])'
else
  echo "NIM did not become ready; inspect the logs below." >&2
  docker logs --tail 200 "$NIM_CONTAINER"
fi
```

Expected output after the readiness body:

```text
fp8-dflash2 dflash2 [1, 2, 3]
```

## Send a K=2 Request

`num_traj_samples` must appear in `k_values`, and values above 1 require a
positive temperature. The builder's default temperature is 0.6. Build and send a K=2
trajectory request from the bundled scene, then summarize the response:

```bash
docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
    --endpoint infer --task trajectory --num-traj-samples 2 \
    --sample-dir /workspace/sample_data > "$ALPAMAYO_OUT/trajectory-k2.json"
curl --fail-with-body -sS "$NIM_URL/v1/infer" \
    -H 'Content-Type: application/json' \
    --data-binary "@$ALPAMAYO_OUT/trajectory-k2.json" \
    --output "$ALPAMAYO_OUT/trajectory-k2-response.json"
python3 -c '
import json, sys
r = json.load(open(sys.argv[1]))
spec = r["metrics"]["speculative_decoding"]
print(r["num_samples"], len(r["pred_trajectories"]), len(r["metrics"]["reasoning_texts"]))
print(spec["num_spec_tokens"], spec["draft_acceptance_rate"] > 0)' \
    "$ALPAMAYO_OUT/trajectory-k2-response.json"
```

Expected output:

```text
2 2 2
7 True
```

Each of the two samples has 64 points in `pred_trajectories` and
`pred_rotations` and its own entry in `metrics.reasoning_texts`; the
speculative-decoding counters sum both samples. The first request for a given
K can be slower than later ones; send it once untimed before measuring. The
gRPC client takes the same count:

```bash
docker exec "$NIM_CONTAINER" python3 /opt/nim/test_grpc_api.py \
    --target localhost:50051 --sample-dir /workspace/sample_data \
    --num-traj-samples 2 --temperature 0.6 --summary-only
```

## Observe Acceptance

DFlash2 profiles default `NIM_ALPAMAYO_SPEC_METRICS` to `summary`. With
`summary` or `detailed`, every trajectory result includes a
`metrics.speculative_decoding` object: in the `/v1/infer` response for
`task: trajectory` and in the `raw_json` payload of a gRPC `Predict`
response. VQA, meta-action, auto-labeling, and grounding results do not carry
the block, and the baseline `bf16` and `fp8` profiles never emit it.
For K greater than one, counters and histogram bins sum all returned samples;
rates are calculated from those totals. Detailed per-step arrays concatenate
the steps in sample order rather than averaging per-sample rates.

**`metrics.speculative_decoding` fields**

| Field | Meaning |
| --- | --- |
| `num_spec_tokens` | Draft tokens proposed per verification step; seven for the packaged DFlash2 head. |
| `num_spec_steps` | Verification steps the request needed. |
| `num_draft_tokens` | Draft tokens proposed over the whole request. |
| `num_accepted_draft_tokens` | Draft tokens the full model accepted over the whole request. |
| `draft_acceptance_rate` | `num_accepted_draft_tokens / num_draft_tokens`, between 0 and 1. |
| `mean_acceptance_length` | Average tokens produced per verification step, counting the verified token; `1 + num_accepted_draft_tokens / num_spec_steps`. |
| `acceptance_histogram` | Number of verification steps that accepted 0, 1, ..., `num_spec_tokens` draft tokens. |
| `per_step_accepted`, `per_step_drafted` | Per-step accepted and proposed counts. Present only with `NIM_ALPAMAYO_SPEC_METRICS=detailed`. |

The HTTP process also exports the request totals from `GET /v1/metrics` as
Prometheus counters labeled with `method="dflash2"`. The counters accumulate
across HTTP and gRPC trajectory requests:

- `alpamayo_spec_decode_draft_tokens_total`;
- `alpamayo_spec_decode_accepted_tokens_total`; and
- `alpamayo_spec_decode_verify_steps_total`.

With `NIM_ALPAMAYO_SPEC_METRICS` set to `summary` (the default) or
`detailed`, they are exported at zero with `method="dflash2"` right after
startup and increase with each trajectory request:

```bash
curl --silent "$NIM_URL/v1/metrics" | grep '^alpamayo_spec_decode'
```

Expected output (values abridged):

```text
alpamayo_spec_decode_draft_tokens_total{method="dflash2"} ...
alpamayo_spec_decode_accepted_tokens_total{method="dflash2"} ...
alpamayo_spec_decode_verify_steps_total{method="dflash2"} ...
```

For acceptance rate over time, use
`rate(alpamayo_spec_decode_accepted_tokens_total[5m]) / rate(alpamayo_spec_decode_draft_tokens_total[5m])`. Mean acceptance length is
one plus accepted tokens per verification step. Higher acceptance produces
more reasoning tokens per step; compare end-to-end latency with the matching
baseline to measure the benefit.

Set `NIM_ALPAMAYO_SPEC_METRICS=detailed` for per-step diagnostics or `none`
to disable both the response block and the counters. Detailed metrics add
response data and are intended for diagnosis, not normal production traffic.

## Qualification

Compare the DFlash2 profile with its matching baseline on representative requests;
use the baseline if DFlash2 does not improve end-to-end latency for your workload.

DFlash2 targets trajectory reasoning and does not speed up every task.
Grounding can be slower on a DFlash2 profile than on the matching baseline
profile, and grounding requests do not update the speculative-decoding
counters. Qualify every task you serve, not only trajectory. For
grounding-heavy or mixed deployments, consider the baseline profile.

## Bring Your Own DFlash2 Head

<Warning>

This is an advanced workflow for users experienced in training and exporting
DFlash2 heads. Use a custom head only if you can verify its compatibility
with the exact Alpamayo base checkpoint and evaluate its output quality.
Otherwise, use the packaged head. Passing startup checks does not establish
model quality.

</Warning>

The NIM does not train or download custom heads. The exported directory needs:

- `config.json` with architecture `DFlash2DraftModel` and
  `dflash_config.projector_type="dflash2"`;
- the exported safetensors weights, including the DFlash2 candidate selector;
- `alpamayo_dflash_metadata.json` from the Alpamayo export, with a
  non-empty `base_model_config_sha256` that records the base configuration
  the head was trained for. This file is required; a head without it, or
  without that hash, is rejected. Do not invent or edit compatibility hashes.

The block size must equal the speculative-token count plus one (eight for
packaged profiles). Preserve training/export provenance and compare output
quality with the matching baseline on representative scenes.

To start from the packaged head, for example to check the layout before
substituting your own weights, copy it to a host directory. From the cache of
a DFlash2 start, locate the head's snapshot directory by its metadata file
and copy it, dereferencing links. The repository and snapshot names in the
cache path follow the packaged model artifact and change between releases, so
find the directory rather than typing it:

```bash
export DFLASH2_HEAD=/tmp/alpamayo2-dflash2-head
HEAD_SNAPSHOT=$(find "$LOCAL_NIM_CACHE/ngc/hub" -path '*/snapshots/*' \
    -name alpamayo_dflash_metadata.json -printf '%h\n' | head -n 1)
echo "Packaged head snapshot: ${HEAD_SNAPSHOT:-not found}"
mkdir -p "$DFLASH2_HEAD"
[ -n "$HEAD_SNAPSHOT" ] && cp -rL "$HEAD_SNAPSHOT/." "$DFLASH2_HEAD"
```

Or from a running DFlash2 container:

```bash
mkdir -p "$DFLASH2_HEAD"
docker exec alpamayo2-super-dflash2 tar -chf - -C /opt/nim/workspace/dflash2 . \
    | tar -xf - -C "$DFLASH2_HEAD"
```

Set `DFLASH2_HEAD` to an existing host directory that contains the head
files before launching. If the host path does not exist, Docker silently
creates an empty, root-owned directory and startup rejects the empty head.

Stop and remove the packaged-head container before running a custom head;
both launches use the same name and host ports. Then mount the head read-only
and override the speculative-model path while selecting a DFlash2 profile:

```bash
test -f "$DFLASH2_HEAD/alpamayo_dflash_metadata.json" || \
    echo "DFLASH2_HEAD does not contain an exported head" >&2
for name in alpamayo2-super alpamayo2-super-dflash2 \
    alpamayo2-super-byoc alpamayo2-super-stage; do
  if docker container inspect "$name" >/dev/null 2>&1; then
    docker stop -t 60 "$name" && docker rm "$name"
  fi
done
docker run -d --name alpamayo2-super-dflash2 \
    --runtime=nvidia --gpus device=0 --shm-size=16g \
    -e NGC_API_KEY \
    -e NIM_MODEL_PROFILE=fp8-dflash2 \
    -e NIM_ALPAMAYO_TRAJ_SAMPLES=1,2,3 \
    -e NIM_ALPAMAYO_SPECULATIVE_MODEL=/byoc-dflash2 \
    -v "$DFLASH2_HEAD:/byoc-dflash2:ro" \
    -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
    -p 8000:8000 -p 50051:50051 \
    "$IMG_NAME"
```

At startup, the NIM verifies the DFlash2 architecture, token block size, tensor
shapes, and the base-configuration hash in `alpamayo_dflash_metadata.json`.
Incompatible or incomplete heads are rejected before the service becomes
ready, with an error that names `NIM_ALPAMAYO_SPECULATIVE_MODEL` and the
mounted path. Before any download, the NIM also compares each safetensors file
of the head with its header. A truncated or damaged head stops startup with
`NIM_ALPAMAYO_SPECULATIVE_MODEL=... has a truncated shard model.safetensors: expected X bytes, found Y`; copy the complete exported head directory again.
A custom head cannot be combined with `NIM_FT_CHECKPOINT`; that combination
stops startup with one `cannot be combined with NIM_FT_CHECKPOINT` line.
Setting `NIM_ALPAMAYO_SPECULATIVE_MODEL` with a baseline
profile fails at startup; select `NIM_MODEL_PROFILE=bf16-dflash2` or
`fp8-dflash2` (and unset `NIM_PRECISION`). `GET /v1/models` reports the container path of the active draft head as
`speculative_model`: the custom head's mount path here, or
`/opt/nim/workspace/dflash2` for the packaged head. It is `null` when
speculative decoding is off. The relaunch keeps the `1,2,3` K list of the
packaged-head launch, so the K=2 example above also runs against the custom
head; a launch without `NIM_ALPAMAYO_TRAJ_SAMPLES` accepts only K=1. Run the
quickstart requests with `alpamayo2-super-dflash2` as the container name and
compare latency and outputs with the matching baseline profile before
deployment.

## Runtime Discovery and Determinism

Use `GET /v1/models` and read `data[0].k_values` for the active sample
counts. `temperature=0` requires `num_traj_samples=1`. For multiple
trajectories, choose a positive temperature and an explicit seed. Increasing
K or switching profiles requires a compatible GPU and a restart.