Fast Model Weight Loading | NVIDIA Dynamo Documentation

Loading a large checkpoint from a shared PVC into GPU memory can dominate cold start. A 1T-parameter model is hundreds of GiB on disk; with the wrong loader and the wrong filesystem layout it can take over an hour, even on storage that advertises multiple GB/s. This page explains the two independent bottlenecks and how to fix both, with a measured end-to-end example.

This is complementary to Model Caching: caching gets the weights onto a shared PVC once; this page makes each worker read them fast. It applies to the PVC + Download Job path, not the ModelExpress object-storage path (which streams from S3/GCS/Azure Blob instead — see ModelExpress).

TL;DR

Two stacked bottlenecks slow a cold weight load off a shared filesystem:

The loader’s access pattern (usually dominant). The default safetensors loader reads single-stream with small effective I/O. On a parallel filesystem this leaves most of the bandwidth idle.
The filesystem layout. On a striped/parallel filesystem (Lustre, and similar), a file that lives on a single storage target is capped at that one target’s bandwidth no matter how much aggregate the filesystem has.

Fix both:

Loader: add --load-format runai_streamer to the vLLM worker (concurrent, multi-threaded, large-block reader; already shipped in Dynamo vLLM runtime images).
Layout (Lustre): stripe the model files across all storage targets — set the directory default before download so files are born wide, or restripe existing files in place with lfs migrate.

In a measured 554 GiB / TP16 deployment these two changes together cut weight load from ~76 min to ~5 min (~14×) with no re-download. See Case study below.

Background: why a cold load is slow

Bottleneck 1 — loader access pattern

vLLM’s default loader (--load-format auto) memory-maps each safetensors shard and copies tensors out largely single-stream. The effective read is latency-bound against remote storage — small requests at shallow queue depth — so it never fills the pipe. Concurrency and block size both matter: on one Azure Managed Lustre (AMLFS) filesystem, a cache-bypassed dd of a single shard measured:

Block size	Throughput (single storage target)
128 KiB	~228 MB/s
1 MiB	~854 MB/s
16 MiB	~1.6 GB/s

The default loader sits near the top of that table (small reads), which is why a real load measured only ~124 MB/s effective — over an order of magnitude below what the same storage delivers to a large-block reader.

runai-model-streamer (the runai_streamer load format) replaces this with a pool of worker threads issuing large, concurrent reads, saturating the storage instead of waiting on it. The package ships in Dynamo’s vLLM runtime images.

Bottleneck 2 — filesystem striping (Lustre)

A parallel filesystem spreads data across multiple OSTs (Object Storage Targets — the backing storage volumes). The filesystem’s advertised throughput is the sum across all OSTs; you only reach it when many OSTs work in parallel. A file’s layout has two knobs:

stripe_count — how many OSTs the file’s bytes are spread across.
stripe_size — the chunk written to one OST before moving to the next.

The default stripe_count is often 1 — each file lives entirely on one OST. A single-file read is then capped at one OST’s bandwidth, regardless of the filesystem’s aggregate. Striping a file across all OSTs lets even one sequential reader pull from all of them at once.

On the same AMLFS (4 OSTs, ~2 GB/s each, 8 GB/s aggregate), large-block read of one shard:

Layout	Throughput
`stripe_count=1` (default)	~2.0 GB/s (one OST)
`stripe_count=4` (all OSTs)	~6.4 GB/s (~3.2×)

Striping only helps once the reader issues large, concurrent I/O. Wide striping with the default single-stream loader barely moves the needle, and a fast loader on stripe-1 files is still capped at one OST. You need both fixes — they multiply.

How many stripes? (OSTs and capacity)

The maximum useful stripe_count is the number of OSTs in the filesystem — you cannot stripe a file across more OSTs than exist. Use stripe_count=-1 to mean “all available OSTs.”

The OST count scales with provisioned capacity, not the SKU alone. AMLFS provisions roughly 1 OST per 4 TiB (this matches the 500-SKU’s 4 TB capacity increment). So aggregate bandwidth and max stripe width grow only as you buy more TiB:

Capacity (AMLFS-500)	~OSTs	Max stripe	Aggregate throughput
16 TiB	4	4	~8 GB/s
32 TiB	8	8	~16 GB/s
128 TiB	~32	~32	~64 GB/s

Check the live OST count on any Lustre mount with lfs df (see Inspecting and setting stripe layout).

Backend support

runai_streamer is a vLLM load format. Striping is filesystem-level and helps every backend, because they all read the same safetensors off the same mount.

Backend	Concurrent streaming loader	Benefits from striping
vLLM	✅ `--load-format runai_streamer` (in-image)	✅
SGLang	❌ not in SGLang’s `--load-format` choices	✅
TensorRT-LLM	❌ no streaming loader, but parallel weight loading is on by default	✅

For SGLang and TensorRT-LLM, the striping fix alone still helps; there is no runai_streamer equivalent to add. (SGLang’s fast-distribution path is ModelExpress remote_instance — a P2P pull from an already-loaded worker, not a faster first read from storage. See Model Caching.)

Runbook: a new model on Lustre storage

This is the end-to-end procedure for bringing up a new large model on a Lustre-backed PVC with both optimizations from the start.

Stripe the cache directory before downloading

Lustre sets a file’s layout at creation time, so the cheapest approach is to widen the target directory first — every file the download writes then inherits wide striping, and you never have to rewrite anything.

From any pod that has the lfs client and mounts the PVC:

$ # -1 = stripe across all OSTs; 4 MiB stripe chunk suits large sequential shards
$ lfs setstripe -c -1 -S 4M /mnt/model-cache/hub

lfs setstripe on a directory sets the default layout for new files; it does not move existing data. Run it before the download Job. If the hub/ subdirectory does not exist yet, create it (or set the layout on the PVC root) so the default is inherited.

If your downloader image lacks the lfs client (for example python:3.12-slim), either add lustre-client-utils to it, or run the lfs setstripe from a separate short-lived pod that has the client before you start the download Job.

Download the model

Run the normal PVC + Download Job. Because the directory default is now wide, every shard lands striped across all OSTs.

Enable the streaming loader in the DGD

Add the load format and (optionally) a concurrency hint to the vLLM worker:

1 services:
2   VllmWorker:
3     extraPodSpec:
4       mainContainer:
5         args:
6           - --model
7           - <org>/<model>
8           - --load-format
9           - runai_streamer
10         env:
11           # Parallel read streams per rank. Raise to keep all OSTs saturated;
12           # lower if you see host-RAM pressure. Docs default is 8.
13           - name: RUNAI_STREAMER_CONCURRENCY
14             value: "16"

Unlike --load-format modelexpress, runai_streamer still expects the weights present on the mounted path — Dynamo prefetches/uses the local cache as usual. It changes how the files are read, not where they come from. No ModelExpress server or VLLM_PLUGINS setting is required.

Verify the load time

Watch the worker log for the streamer and the load-time line:

$ kubectl logs <worker-leader-pod> | grep -iE "Runai Model Streamer|Loading weights took|Model loading took"

You should see Loading safetensors using Runai Model Streamer: ... during load and a final Model loading took N seconds. Compare against a baseline run without the flag to confirm the speedup.

Fixing an already-downloaded model (no re-download)

If the weights are already on the PVC with the default stripe_count=1, you do not need to re-download. lfs migrate rewrites each file’s data across OSTs in place, preserving the HuggingFace cache symlinks (the blobs/ files are the real data; snapshots/ are symlinks into them).

$ # Set the directory default so any FUTURE writes land wide too
$ lfs setstripe -c -1 -S 4M /mnt/model-cache/hub/models--<org>--<model>
$ lfs setstripe -c -1 -S 4M /mnt/model-cache/hub/models--<org>--<model>/blobs
$ 
$ # Restripe the existing blob data in place (idempotent: skips already-wide files)
$ osts=$(lfs df /mnt/model-cache | awk '/OST:/ {n++} END {print n+0}')
$ for f in /mnt/model-cache/hub/models--<org>--<model>/blobs/*; do
$   c=$(lfs getstripe -c "$f")
$   [ "$c" -lt "$osts" ] && lfs migrate -c -1 -S 4M "$f"
$ done

Verify integrity for free

HuggingFace LFS blob filenames are the file’s SHA-256. After migrating you can confirm the data is bit-identical at zero extra bookkeeping cost:

$ bad=0
$ for f in /mnt/model-cache/hub/models--<org>--<model>/blobs/*; do
$   name=$(basename "$f")
$   # Only LFS blobs are named by sha256 (64 hex chars); skip small git-hash metadata blobs
$   echo "$name" | grep -qE '^[0-9a-f]{64}$' || continue
$   calc=$(sha256sum "$f" | awk '{print $1}')
$   if [ "$calc" = "$name" ]; then
$     echo "OK   $name"
$   else
$     echo "BAD  $name"
$     bad=1
$   fi
$ done
$ exit "$bad"

lfs migrate is non-destructive (it writes a new layout and swaps only on success), and the mount typically carries a client-side checksum option, so corruption is already unlikely — the SHA-256 sweep is a cheap final gate before you reload the model on top of the restriped data.

Inspecting and setting stripe layout

You need the lfs client (from lustre-client-utils) on a pod that mounts the PVC. The Azure Lustre CSI node image (csi-azurelustre-node) already contains lfs.

$ # OST count and per-OST capacity/usage
$ lfs df -h /mnt/model-cache
$ 
$ # Directory default layout (applies to new files)
$ lfs getstripe -d /mnt/model-cache/hub
$ 
$ # A specific file's layout
> lfs getstripe -c -S -i /mnt/model-cache/hub/.../blobs/<sha>   # count, size, OST index

Case study: Kimi-K2.7-Code, 554 GiB on AMLFS

A real bring-up: moonshotai/Kimi-K2.7-Code (1T-param / 32B-active MoE, native INT4), served aggregated at TP16 across two 8×H100 nodes, weights on an AMLFS-Durable-Premium-500 filesystem (16 TiB, 4 OSTs, 8 GB/s aggregate).

Before. Default loader, files at stripe_count=1: weight load took 4575 s (~76 min) — an effective ~124 MB/s, ~1/60th of the filesystem’s aggregate. Root cause split cleanly into the two bottlenecks above (loader access pattern dominant; single-OST striping secondary).

Change. lfs migrate -c -1 -S 4M across all 87 blobs (555 GiB rewritten in place in ~10 min, SHA-256 verified bit-identical, no re-download), plus --load-format runai_streamer with RUNAI_STREAMER_CONCURRENCY=16 in the DGD.

After. On reload, weight load took 318 s (~5.3 min) — a ~14.4× speedup — for the identical 36.96 GiB/GPU (layout-only change; weights unchanged). The model served correctly afterward.

The larger share of the win came from the loader flag (the access pattern was the dominant bottleneck); striping stacked on top by letting those concurrent streams hit all four OSTs at once.