Model Deployment

Set up KV Cache Offloading

Add a KV cache offloading connector to a worker in a DynamoGraphDeployment so KV blocks spill to host memory or disk.
View as Markdown

KV cache offloading lets a worker keep more KV cache than fits in GPU memory by spilling blocks to host (CPU) memory or local disk. This serves longer contexts and reuses cached prefixes across requests. This page shows how to turn it on inside a DynamoGraphDeployment (DGD) — the engine-internals and local-CLI details live in the per-backend pages linked at the end.

This is a how-to for an existing deployment. If you have not authored a DGD yet, start with the DGD Guide.

The pattern

Offloading is configured on the worker, not the Frontend. For vLLM workers, you pass a --kv-transfer-config JSON argument that names a connector, then size the offload tiers with environment variables. The shape is always the same:

1spec:
2  components:
3  - name: VllmDecodeWorker
4    type: worker
5    podTemplate:
6      spec:
7        containers:
8        - name: main
9          command:
10          - python3
11          - -m
12          - dynamo.vllm
13          args:
14          - --model
15          - Qwen/Qwen3-32B
16          - --kv-transfer-config
17          - '{"kv_connector":"<Connector>","kv_role":"kv_both", ...}'
18          env:
19          - name: <TIER_SIZE_ENV>
20            value: "<gb>"
1

Choose a connector

Each offloading backend uses the same --kv-transfer-config hook with a different kv_connector. Pick one — they are alternatives, not layers.

Backendkv_connectorSizingBest for
KVBM (KV Block Manager)DynamoConnector (with kv_connector_module_path: kvbm.vllm_integration.connector)DYN_KVBM_CPU_CACHE_GB, DYN_KVBM_DISK_CACHE_GB (env)Dynamo-native CPU/disk tiering across vLLM and TensorRT-LLM
LMCacheLMCacheMPConnectorLMCache server configPrefill-once / reuse-everywhere across a fleet
FlexKVFlexKVConnectorV1DYNAMO_USE_FLEXKV=1, FLEXKV_CPU_CACHE_GB (env)Distributed KV offloading runtime
SGLang HiCachen/a — use --enable-hierarchical-cache and DYN_SHARED_CACHE_TYPE insteadDYN_SHARED_CACHE_MULTIPLIERSGLang hierarchical cache with a tier-aware router

SGLang HiCache does not use the --kv-transfer-config connector mechanism. On an SGLang worker, set --enable-hierarchical-cache in args and DYN_SHARED_CACHE_TYPE in the container env. See Using HiCache.

2

Worked example: KVBM on a vLLM worker

This aggregated Qwen3-32B worker offloads KV blocks to 100 GB of host memory. It is the worker section of the agg-kvbm recipe:

1spec:
2  components:
3  - name: VllmDecodeWorker
4    type: worker
5    replicas: 1
6    podTemplate:
7      spec:
8        containers:
9        - name: main
10          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1
11          command:
12          - python3
13          - -m
14          - dynamo.vllm
15          args:
16          - --model
17          - Qwen/Qwen3-32B
18          - --kv-transfer-config
19          - '{"kv_connector":"DynamoConnector","kv_role":"kv_both","kv_connector_module_path":"kvbm.vllm_integration.connector"}'
20          envFrom:
21          - secretRef:
22              name: hf-token-secret
23          env:
24          # Tune the CPU cache size for your node's available host memory.
25          - name: DYN_KVBM_CPU_CACHE_GB
26            value: "100"
27          resources:
28            limits:
29              nvidia.com/gpu: "1"
30              memory: 200Gi        # host memory must hold the CPU cache tier
31            requests:
32              nvidia.com/gpu: "1"
33              memory: 150Gi

Two things to size together:

  • DYN_KVBM_CPU_CACHE_GB (and optionally DYN_KVBM_DISK_CACHE_GB) set the offload tier sizes.
  • resources.limits.memory must be large enough to hold the CPU cache tier plus the engine’s normal footprint. The recipe requests 150–200 GB for a 100 GB CPU cache.

To add a disk tier, set DYN_KVBM_DISK_CACHE_GB and mount a volume for it. To export offloading metrics to Prometheus, set DYN_KVBM_METRICS=true.

3

Verify

After the worker is Running, send repeated requests that share a long prefix and confirm the second request has lower TTFT. With DYN_KVBM_METRICS=true, check the offload hit-rate metrics — see Metrics.

These cover engine internals, the local-CLI workflow, and tuning for each backend: