Bring Your Own Checkpoint
Alpamayo 2 Super. For the previous generation, see Alpamayo 1.5 Bring Your Own Checkpoint.
Alpamayo 2 Super NIM supports Bring Your Own Checkpoint (BYOC) with
NIM_FT_CHECKPOINT.
Run the commands in Bash. The blocks on this page define every variable
they use; in a new shell, re-run the export lines of the download and
launch blocks before continuing.
ALPAMAYO_CHECKPOINT: absolute host path of the checkpoint directory. The download block sets it from the current directory; to reuse a checkpoint elsewhere, export its absolute path, for exampleexport ALPAMAYO_CHECKPOINT=/data/alpamayo2-super-checkpoint. An unset value makes Docker reject the-v ":/byoc:ro"mount.IMG_NAME,LOCAL_NIM_CACHE,NIM_CONTAINER(alpamayo2-super-byoc) andNIM_URL: set by the launch block.ALPAMAYO_OUT: request output directory. The launch block keeps a value already set by another page and creates one otherwise.
No NGC_API_KEY is needed for a complete checkpoint.
Checkpoint Requirements
Use a complete Alpamayo 2 Super Hugging Face checkpoint directory, or a
workspace directory that contains the checkpoint below alpamayo/. Merge
adapters or training deltas into a complete checkpoint before using them with
the NIM. A complete checkpoint has config.json,
model.safetensors.index.json, and every shard that the index names
(15 shards, about 72 GB, for nvidia/Alpamayo2-Super).
The checkpoint must be unquantized (BF16) or a ModelOpt FP8 checkpoint.
Startup reads the quantization that the checkpoint declares
(quantization_config in config.json, or ModelOpt’s
hf_quant_config.json), because vLLM loads the checkpoint in that format
whatever profile is selected:
- An unquantized checkpoint runs with
NIM_PRECISION=bf16, or withNIM_PRECISION=fp8, which converts it to FP8 on the first start. - A ModelOpt FP8 checkpoint (
quant_algoFP8, no KV-cache quantization) runs only withNIM_PRECISION=fp8.
Any other combination fails before any download with one line that names the
detected quantization, for example a ModelOpt FP8 checkpoint with the bf16
profile, or a W4A16, NVFP4, AWQ, GPTQ or compressed-tensors checkpoint.
The precision comes from the selected profile, so NIM_MODEL_PROFILE=fp8
or the fp8 profile ID behaves like NIM_PRECISION=fp8.
Launch with BYOC
Download a checkpoint into a stable local directory. Install the Hugging Face
CLI in a virtual environment, like the other host clients: distributions
that mark the system Python as externally managed, such as Ubuntu 24.04 and
Debian 12, reject a system-wide pip install. The hf command requires
huggingface_hub 0.34.0 or later; older releases provide only
huggingface-cli.
pipx install huggingface_hub also provides hf without activating an
environment. The virtual environment is needed only for the download; the
remaining commands on this page do not use it.
Verify that checkpoint files are at the mount root or under alpamayo/:
Make the checkpoint readable by the container user (UID 1000, nvs).
The checkpoint mount stays read-only; only the cache must be writable:
Use the --local-dir download above instead of capturing CLI output as a
snapshot path. If you maintain an existing snapshot separately, its blob links
must be dereferenced when copying it for a container mount. Keeping both the
snapshot and a complete copy requires roughly twice the checkpoint storage.
Stop and remove any Alpamayo 2 Super container from the other guides before this launch, because they use the same GPU and host ports. The loop skips names that do not exist and removes a container only after it stops:
Each container uses exactly one GPU. Replace 0 in --gpus device=0 with
the index of the GPU allocated to you; do not use --gpus all.
Wait for readiness before sending any request. A BYOC start validates and fingerprints the checkpoint, and an FP8 profile can also convert it, so the first start takes longer than a packaged-profile start. Poll readiness and stop if the container exits:
Confirm that GET /v1/models reports "checkpoint": "byoc". To run a
quickstart request against this container, replace alpamayo2-super in its
docker exec or docker cp commands with alpamayo2-super-byoc. For
example:
Before loading, the NIM checks config.json and every indexed shard, or
non-empty .safetensors/.bin weights when no index exists. It also
compares each safetensors shard’s size with its header, checks that the
tokenizer and image-processor files are present (tokenizer_config.json,
tokenizer.json or the vocab.json and merges.txt pair, and
preprocessor_config.json), and checks that
config.json declares architectures: ["Alpamayo2Super"]. These checks
read file metadata only, so a checkpoint that cannot load stops startup with
one Checkpoint validation failed: ... line before fingerprinting,
conversion or model load. After a
complete checkpoint passes validation, startup skips the packaged base-weight
download, so the BYOC launch needs no NGC_API_KEY and no disk space for
the base weights. It logs NIM_FT_CHECKPOINT is complete; skipping the packaged base-weight download. The NIM does this by setting
NIM_DISABLE_MODEL_DOWNLOAD=1. BYOC never downloads the packaged base
weights: an explicit false value such as 0 is ignored with a warning, so a
BYOC start needs no NGC_API_KEY. Use download-to-cache to stage
packaged profiles.
The NIM fingerprints the checkpoint for discovery and the FP8 conversion
cache. The fingerprint is remembered under
$NIM_CACHE_PATH/alpamayo/byoc-fingerprint/ and reused on later starts
while the checkpoint files are unchanged; any change to any file causes the
whole checkpoint to be hashed again on the next start.
Metadata changes count as changes: permission or ownership changes (for
example chmod -R) and new hard links also cause a new hash on the next
start.
Validation, the first fingerprint and a first-time FP8 conversion add startup
time.
Readability and conversion space are checked before downloads, even when the profile is selected by manifest hash. A matching cached conversion needs no free conversion workspace; verifying the match may require reading the source checkpoint. Stale or unrelated cache entries do not bypass the space check.
Troubleshooting
The table lists a distinctive part of each startup error; the full log line
adds a prefix such as startup error: and can continue with the path and
further detail, so search the log for these strings.
Cache Behavior
BYOC supports the baseline bf16 and fp8 profiles. With bf16, the
NIM loads the mounted checkpoint. With fp8, the NIM loads a compatible
ModelOpt FP8 checkpoint or converts a BF16 checkpoint during the first startup.
The converted checkpoint is reused from /opt/nim/.cache on later starts.
To request conversion, allow at least 45 GiB free for a new conversion, then replace the BF16 BYOC container with this FP8 launch:
Before sending a request, repeat the readiness loop above; the first FP8 start includes the conversion. Then rerun the BYOC request example. The converted checkpoint remains in the persistent cache for later starts.
Keep the same container mount path (/byoc) and NIM_FT_CHECKPOINT
value across starts. Changing the mount path changes the checkpoint path seen
by the engine, so the next start compiles again even when the files are
unchanged.
Prune FP8 Conversion Caches
Each converted checkpoint is a separate entry of about the size of the FP8
weights (roughly 40 GB) under
$NIM_CACHE_PATH/alpamayo/byoc-fp8/<cache-key>/, or under
NIM_ALPAMAYO_BYOC_FP8_CACHE_DIR when that is set. The key covers the
checkpoint fingerprint, the calibration sample, the quantizer and the
ModelOpt version, so every fine-tuned checkpoint, and a changed checkpoint,
adds an entry; old entries are never removed automatically. The startup log
names the entry in use: [alpamayo-byoc-fp8] cache_key=<cache-key> cache_hit=<0 or 1>. Record the key for each checkpoint you keep:
The entries belong to the container user (UID 1000), so a host user
usually cannot delete them. Stop every container that uses the cache, then
list the entries with their size and age and remove the ones you no longer
need from a container that runs as UID 1000:
Replace <cache-key-to-remove> with the 32-character key of the entry to
delete.
Hidden .<cache-key>.tmp-<pid> and .<cache-key>.invalid-<time>
directories are left by an interrupted or replaced conversion and can also be
removed while no container uses the cache; empty .<cache-key>.lock files
are harmless. The next FP8 start of a checkpoint
whose entry was removed converts it again.
DFlash2
BYOC supports only the baseline bf16 and fp8 profiles. Selecting
bf16-dflash2 or fp8-dflash2 together with NIM_FT_CHECKPOINT stops
startup with a configuration error, because a DFlash2 head must match the
exact base checkpoint. DFlash2 Speculative Decoding describes DFlash2
for the packaged checkpoint.
Metadata
GET /v1/models reports the active profile, precision,
checkpoint_quantization (bf16 or modelopt-fp8, the format vLLM
serves, which after FP8 conversion is modelopt-fp8), the
speculative-decoding method, checkpoint="byoc", and checkpoint_hash.
Under BYOC, /v1/metadata modelInfo describes the mounted checkpoint:
modelUrl is file://<NIM_FT_CHECKPOINT path> and shortName is
byoc-<checkpoint fingerprint>, the fingerprint that GET /v1/models
reports as checkpoint_hash.
A complete checkpoint mount needs no NGC download, so BYOC can start without network access once the image is present. Compiled artifacts are still created on first start; to stage them, follow Air-Gapped Deployment and verify the checkpoint/profile combination with networking disabled.