Offload KV Cache Locally | NVIDIA Dynamo Documentation

KV cache offloading moves reusable KV blocks from GPU memory to host memory or storage. This tutorial starts an aggregated Dynamo deployment with one offloading backend, sends requests through it, and checks that the backend is running.

Prepare Your Environment

Complete the local installation and start Dynamo’s NATS and etcd services before continuing. Run the commands from the root of the cloned Dynamo repository.

If docker run opened a shell in a prebuilt runtime container, you are already in the correct environment and can skip package installation when the table below marks the backend as included. For a container running in the background, open a shell with docker exec; the repository is available at /workspace:

$ docker exec -it <container-name> bash
$ cd /workspace

If you installed Dynamo into a Python environment, activate that environment before continuing.

Choose a Backend

Select one offloading backend for each worker. KVBM, LMCache, FlexKV, and HiCache are alternatives, not cache layers to stack together. KV-aware routing and NIXL-based prefill/decode transfer are separate features that can operate alongside the selected backend.

Backend	Engine	Cache tiers	Prebuilt Dynamo runtime container	Python environment
KVBM	vLLM, TensorRT-LLM	Host, disk, object storage	Included in current vLLM and TensorRT-LLM runtime images	Install `kvbm`
LMCache	vLLM	Host and persistent L2 adapters	Inherited from the upstream vLLM image; verify compatibility for the image tag and platform	Install `lmcache`
HiCache	SGLang	Host and optional external storage	Included with SGLang	Included with `ai-dynamo[sglang]`
FlexKV	vLLM in this tutorial	Host, SSD, cloud storage	Requires a custom image or a source build inside the container	Build FlexKV from source

The Dynamo Operator does not install these packages. Local containers and Kubernetes deployments get them from the selected runtime image. The Python environment path installs packages on the host instead.

Start Offloading

Choose a backend and follow its tab. The primary examples use Qwen/Qwen3-0.6B and one GPU; the optional disaggregated FlexKV example uses two GPUs. The vLLM launch scripts start the Dynamo frontend and workers, and stop all processes when you press Ctrl+C.

KVBM

LMCache

HiCache

FlexKV

KVBM is Dynamo’s native KV block manager and the recommended starting point for vLLM. The current Dynamo vLLM runtime container includes it. For a Python environment, install its wheel first:

$ uv pip install kvbm

Start an aggregated vLLM worker with a 20 GB host-memory cache:

$ cd examples/backends/vllm
$ DYN_KVBM_CPU_CACHE_GB=20 ./launch/agg_kvbm.sh

The script selects DynamoConnector. To add a disk tier later, set DYN_KVBM_DISK_CACHE_GB and DYN_KVBM_DISK_CACHE_DIR. Keep every lower cache tier at least as large as the tier above it to avoid repeatedly evicting and reloading the same blocks.

For all KVBM settings, see the KVBM Configuration Reference.

Send a Request

After the frontend listens on port 8000, send a request from another terminal:

$ curl localhost:8000/v1/chat/completions \
>   -H "Content-Type: application/json" \
>   -d '{
>     "model": "Qwen/Qwen3-0.6B",
>     "messages": [{"role": "user", "content": "Explain how a KV cache reduces repeated computation."}],
>     "stream": false,
>     "max_tokens": 30
>   }'

Send the request again to create an opportunity to reuse its prompt prefix. A short request confirms the deployment path but does not establish a performance improvement. Use a representative workload with long, repeated prefixes to measure Time To First Token (TTFT) and cache hit rate.

Next Steps

Add KV-aware routing when multiple workers can reuse cached prefixes.
Combine one offloading backend with disaggregated serving. In that topology, the offloading connector manages storage while NIXL transfers blocks from prefill to decode workers.
Review KVBM architecture for Dynamo’s cache-tier and transfer design.
To make a custom inference engine visible to the KV router, implement KV event publishing.