Skip to navigation

Bring Your Own Checkpoint

View as Markdown

Alpamayo 2 Super. For the previous generation, see Alpamayo 1.5 Bring Your Own Checkpoint.

Alpamayo 2 Super NIM supports Bring Your Own Checkpoint (BYOC) with NIM_FT_CHECKPOINT.

Variables used on this page

Run the commands in Bash. The blocks on this page define every variable they use; in a new shell, re-run the export lines of the download and launch blocks before continuing.

  • ALPAMAYO_CHECKPOINT: absolute host path of the checkpoint directory. The download block sets it from the current directory; to reuse a checkpoint elsewhere, export its absolute path, for example export ALPAMAYO_CHECKPOINT=/data/alpamayo2-super-checkpoint. An unset value makes Docker reject the -v ":/byoc:ro" mount.
  • IMG_NAME, LOCAL_NIM_CACHE, NIM_CONTAINER (alpamayo2-super-byoc) and NIM_URL: set by the launch block.
  • ALPAMAYO_OUT: request output directory. The launch block keeps a value already set by another page and creates one otherwise.

No NGC_API_KEY is needed for a complete checkpoint.

Checkpoint Requirements

Use a complete Alpamayo 2 Super Hugging Face checkpoint directory, or a workspace directory that contains the checkpoint below alpamayo/. Merge adapters or training deltas into a complete checkpoint before using them with the NIM. A complete checkpoint has config.json, model.safetensors.index.json, and every shard that the index names (15 shards, about 72 GB, for nvidia/Alpamayo2-Super).

The checkpoint must be unquantized (BF16) or a ModelOpt FP8 checkpoint. Startup reads the quantization that the checkpoint declares (quantization_config in config.json, or ModelOpt’s hf_quant_config.json), because vLLM loads the checkpoint in that format whatever profile is selected:

  • An unquantized checkpoint runs with NIM_PRECISION=bf16, or with NIM_PRECISION=fp8, which converts it to FP8 on the first start.
  • A ModelOpt FP8 checkpoint (quant_algo FP8, no KV-cache quantization) runs only with NIM_PRECISION=fp8.

Any other combination fails before any download with one line that names the detected quantization, for example a ModelOpt FP8 checkpoint with the bf16 profile, or a W4A16, NVFP4, AWQ, GPTQ or compressed-tensors checkpoint. The precision comes from the selected profile, so NIM_MODEL_PROFILE=fp8 or the fp8 profile ID behaves like NIM_PRECISION=fp8.

Launch with BYOC

Download a checkpoint into a stable local directory. Install the Hugging Face CLI in a virtual environment, like the other host clients: distributions that mark the system Python as externally managed, such as Ubuntu 24.04 and Debian 12, reject a system-wide pip install. The hf command requires huggingface_hub 0.34.0 or later; older releases provide only huggingface-cli.

python3 -m venv .hf-venv
. .hf-venv/bin/activate
python3 -m pip install -U "huggingface_hub>=0.34.0"
export ALPAMAYO_CHECKPOINT="$PWD/alpamayo2-super-checkpoint"
hf download nvidia/Alpamayo2-Super --local-dir "$ALPAMAYO_CHECKPOINT"

pipx install huggingface_hub also provides hf without activating an environment. The virtual environment is needed only for the download; the remaining commands on this page do not use it.

Verify that checkpoint files are at the mount root or under alpamayo/:

test -f "$ALPAMAYO_CHECKPOINT/config.json" || \
test -f "$ALPAMAYO_CHECKPOINT/alpamayo/config.json"
test -f "$ALPAMAYO_CHECKPOINT/model.safetensors.index.json" || \
test -f "$ALPAMAYO_CHECKPOINT/alpamayo/model.safetensors.index.json"

Make the checkpoint readable by the container user (UID 1000, nvs). The checkpoint mount stays read-only; only the cache must be writable:

chmod -R a+rX "$ALPAMAYO_CHECKPOINT"

Use the --local-dir download above instead of capturing CLI output as a snapshot path. If you maintain an existing snapshot separately, its blob links must be dereferenced when copying it for a container mount. Keeping both the snapshot and a complete copy requires roughly twice the checkpoint storage.

Stop and remove any Alpamayo 2 Super container from the other guides before this launch, because they use the same GPU and host ports. The loop skips names that do not exist and removes a container only after it stops:

for name in alpamayo2-super alpamayo2-super-dflash2 \
alpamayo2-super-byoc alpamayo2-super-stage; do
if docker container inspect "$name" >/dev/null 2>&1; then
docker stop -t 60 "$name" && docker rm "$name"
fi
done

Each container uses exactly one GPU. Replace 0 in --gpus device=0 with the index of the GPU allocated to you; do not use --gpus all.

export IMG_NAME=nvcr.io/nim/nvidia/alpamayo-2-super:1.0.0
export NIM_CONTAINER=alpamayo2-super-byoc
export NIM_URL=http://localhost:8000
export ALPAMAYO_OUT="${ALPAMAYO_OUT:-$(mktemp -d "${TMPDIR:-/tmp}/alpamayo.XXXXXX")}"
export LOCAL_NIM_CACHE=/tmp/alpamayo2-super-cache
mkdir -p "$LOCAL_NIM_CACHE"
chmod 1777 "$LOCAL_NIM_CACHE"
docker run -d --name "$NIM_CONTAINER" \
--runtime=nvidia --gpus device=0 --shm-size=16g \
-e NIM_PRECISION=bf16 \
-e NIM_FT_CHECKPOINT=/byoc \
-v "$ALPAMAYO_CHECKPOINT:/byoc:ro" \
-v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
-p 8000:8000 -p 50051:50051 \
"$IMG_NAME"

Wait for readiness before sending any request. A BYOC start validates and fingerprints the checkpoint, and an FP8 profile can also convert it, so the first start takes longer than a packaged-profile start. Poll readiness and stop if the container exits:

ready=false
for attempt in $(seq 1 360); do
if curl -sf "$NIM_URL/v1/health/ready"; then
ready=true
break
fi
state=$(docker inspect -f '{{.State.Status}}' "$NIM_CONTAINER" 2>/dev/null)
[ "$state" = running ] || break
sleep 5
done
if [ "$ready" = true ]; then
curl --fail-with-body -sS "$NIM_URL/v1/models" | python3 -m json.tool
else
echo "NIM did not become ready; inspect the logs below." >&2
docker logs --tail 200 "$NIM_CONTAINER"
fi

Confirm that GET /v1/models reports "checkpoint": "byoc". To run a quickstart request against this container, replace alpamayo2-super in its docker exec or docker cp commands with alpamayo2-super-byoc. For example:

docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
--endpoint infer --task trajectory --sample-dir /workspace/sample_data \
> "$ALPAMAYO_OUT/byoc-trajectory.json"
curl --fail-with-body -sS -H 'Content-Type: application/json' \
--data-binary "@$ALPAMAYO_OUT/byoc-trajectory.json" \
"$NIM_URL/v1/infer"

Before loading, the NIM checks config.json and every indexed shard, or non-empty .safetensors/.bin weights when no index exists. It also compares each safetensors shard’s size with its header, checks that the tokenizer and image-processor files are present (tokenizer_config.json, tokenizer.json or the vocab.json and merges.txt pair, and preprocessor_config.json), and checks that config.json declares architectures: ["Alpamayo2Super"]. These checks read file metadata only, so a checkpoint that cannot load stops startup with one Checkpoint validation failed: ... line before fingerprinting, conversion or model load. After a complete checkpoint passes validation, startup skips the packaged base-weight download, so the BYOC launch needs no NGC_API_KEY and no disk space for the base weights. It logs NIM_FT_CHECKPOINT is complete; skipping the packaged base-weight download. The NIM does this by setting NIM_DISABLE_MODEL_DOWNLOAD=1. BYOC never downloads the packaged base weights: an explicit false value such as 0 is ignored with a warning, so a BYOC start needs no NGC_API_KEY. Use download-to-cache to stage packaged profiles.

The NIM fingerprints the checkpoint for discovery and the FP8 conversion cache. The fingerprint is remembered under $NIM_CACHE_PATH/alpamayo/byoc-fingerprint/ and reused on later starts while the checkpoint files are unchanged; any change to any file causes the whole checkpoint to be hashed again on the next start. Metadata changes count as changes: permission or ownership changes (for example chmod -R) and new hard links also cause a new hash on the next start. Validation, the first fingerprint and a first-time FP8 conversion add startup time.

Readability and conversion space are checked before downloads, even when the profile is selected by manifest hash. A matching cached conversion needs no free conversion workspace; verifying the match may require reading the source checkpoint. Stale or unrelated cache entries do not bypass the space check.

Troubleshooting

The table lists a distinctive part of each startup error; the full log line adds a prefix such as startup error: and can continue with the path and further detail, so search the log for these strings.

Startup logCause and fix
NIM_FT_CHECKPOINT is a ModelOpt FP8 checkpointThe checkpoint is already FP8 but the bf16 profile is selected. Set NIM_PRECISION=fp8, or mount the unquantized BF16 checkpoint.
NIM_FT_CHECKPOINT uses unsupported quantizationThe checkpoint uses a quantization the NIM does not serve (the message names it, for example ModelOpt W4A16_NVFP4). Use the unquantized BF16 checkpoint, which the fp8 profile can convert, or a ModelOpt FP8 checkpoint.
Checkpoint config is missing: /byoc/config.jsonNeither /byoc/config.json nor /byoc/alpamayo/config.json exists. Check the -v source path and the --local-dir passed to hf download; the mount must contain the checkpoint directory itself, not its parent.
Checkpoint is incomplete according to model.safetensors.index.json: missing=...The download was interrupted or shards were removed. Re-run hf download into the same --local-dir; it resumes and verifies existing files. empty=... names zero-byte shards.
Checkpoint has no non-empty .safetensors/.bin weightsThe directory holds only configuration, an adapter, or a single-file delta. Merge or convert it into a complete Hugging Face checkpoint first.
Permission denied (for example Permission denied reading /byoc/config.json or [Errno 13] Permission denied), or Cache directory <path> or Cache path <path> is not writable by UID 1000 / GID 1000The files are not readable by UID 1000 or the cache is not writable by it. Run chmod -R a+rX on the checkpoint and recreate the cache as described in Configuring a NIM. NFS-backed home directories often do not honor these permissions for a container user; prefer local disk.
Truncated shard <name>: expected X bytes, found YThe copy or download was interrupted; copy the checkpoint again.
... is missing tokenizer or processor files: <names>Training scripts that save only weights and config.json omit them; copy tokenizer.json, tokenizer_config.json and preprocessor_config.json unchanged from nvidia/Alpamayo2-Super into the checkpoint directory.
unsupported architecture: ... declares architectures [...]The NIM serves Alpamayo 2 Super checkpoints only; config.json must declare architectures: ["Alpamayo2Super"] with a qwen3_vl vlm_config.
NIM_ALPAMAYO_SPECULATIVE_MODEL cannot be combined with NIM_FT_CHECKPOINTA custom draft head is trained for one base checkpoint, and BYOC runs without speculative decoding. Unset NIM_ALPAMAYO_SPECULATIVE_MODEL to serve the checkpoint, or unset NIM_FT_CHECKPOINT to serve the packaged weights with the head and a DFlash2 profile.
The inference backend failed: <Error>: <message>The checkpoint passed validation but the backend could not load it. The message is the first error the backend printed; the lines after it in the log are follow-on failures.
Not enough free cache space for BYOC FP8 quantizationFP8 conversion needs at least NIM_ALPAMAYO_BYOC_FP8_MIN_FREE_GB (default 45) GiB free in the cache. Free space or point NIM_ALPAMAYO_BYOC_FP8_CACHE_DIR at a larger volume.

Cache Behavior

BYOC supports the baseline bf16 and fp8 profiles. With bf16, the NIM loads the mounted checkpoint. With fp8, the NIM loads a compatible ModelOpt FP8 checkpoint or converts a BF16 checkpoint during the first startup. The converted checkpoint is reused from /opt/nim/.cache on later starts.

To request conversion, allow at least 45 GiB free for a new conversion, then replace the BF16 BYOC container with this FP8 launch:

docker stop -t 60 "$NIM_CONTAINER" && docker rm "$NIM_CONTAINER"
docker run -d --name "$NIM_CONTAINER" \
--runtime=nvidia --gpus device=0 --shm-size=16g \
-e NIM_PRECISION=fp8 \
-e NIM_FT_CHECKPOINT=/byoc \
-v "$ALPAMAYO_CHECKPOINT:/byoc:ro" \
-v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
-p 8000:8000 -p 50051:50051 \
"$IMG_NAME"

Before sending a request, repeat the readiness loop above; the first FP8 start includes the conversion. Then rerun the BYOC request example. The converted checkpoint remains in the persistent cache for later starts.

Keep the same container mount path (/byoc) and NIM_FT_CHECKPOINT value across starts. Changing the mount path changes the checkpoint path seen by the engine, so the next start compiles again even when the files are unchanged.

Prune FP8 Conversion Caches

Each converted checkpoint is a separate entry of about the size of the FP8 weights (roughly 40 GB) under $NIM_CACHE_PATH/alpamayo/byoc-fp8/<cache-key>/, or under NIM_ALPAMAYO_BYOC_FP8_CACHE_DIR when that is set. The key covers the checkpoint fingerprint, the calibration sample, the quantizer and the ModelOpt version, so every fine-tuned checkpoint, and a changed checkpoint, adds an entry; old entries are never removed automatically. The startup log names the entry in use: [alpamayo-byoc-fp8] cache_key=<cache-key> cache_hit=<0 or 1>. Record the key for each checkpoint you keep:

docker logs "$NIM_CONTAINER" 2>&1 | grep -o 'cache_key=[0-9a-f]*'

The entries belong to the container user (UID 1000), so a host user usually cannot delete them. Stop every container that uses the cache, then list the entries with their size and age and remove the ones you no longer need from a container that runs as UID 1000:

docker run --rm --user 1000:1000 --entrypoint sh \
-v "$LOCAL_NIM_CACHE:/opt/nim/.cache" "$IMG_NAME" \
-c 'cd /opt/nim/.cache/alpamayo/byoc-fp8 2>/dev/null && du -sh -- * .[!.]* 2>/dev/null; ls -la'
export PRUNE_KEY='<cache-key-to-remove>'
docker run --rm --user 1000:1000 --entrypoint rm \
-v "$LOCAL_NIM_CACHE:/opt/nim/.cache" "$IMG_NAME" \
-rf -- "/opt/nim/.cache/alpamayo/byoc-fp8/${PRUNE_KEY:?set PRUNE_KEY}"

Replace <cache-key-to-remove> with the 32-character key of the entry to delete.

Hidden .<cache-key>.tmp-<pid> and .<cache-key>.invalid-<time> directories are left by an interrupted or replaced conversion and can also be removed while no container uses the cache; empty .<cache-key>.lock files are harmless. The next FP8 start of a checkpoint whose entry was removed converts it again.

DFlash2

BYOC supports only the baseline bf16 and fp8 profiles. Selecting bf16-dflash2 or fp8-dflash2 together with NIM_FT_CHECKPOINT stops startup with a configuration error, because a DFlash2 head must match the exact base checkpoint. DFlash2 Speculative Decoding describes DFlash2 for the packaged checkpoint.

Metadata

GET /v1/models reports the active profile, precision, checkpoint_quantization (bf16 or modelopt-fp8, the format vLLM serves, which after FP8 conversion is modelopt-fp8), the speculative-decoding method, checkpoint="byoc", and checkpoint_hash.

Under BYOC, /v1/metadata modelInfo describes the mounted checkpoint: modelUrl is file://<NIM_FT_CHECKPOINT path> and shortName is byoc-<checkpoint fingerprint>, the fingerprint that GET /v1/models reports as checkpoint_hash.

A complete checkpoint mount needs no NGC download, so BYOC can start without network access once the image is present. Compiled artifacts are still created on first start; to stage them, follow Air-Gapped Deployment and verify the checkpoint/profile combination with networking disabled.