KV Cache Offloading

Dynamo supports multiple KV cache offloading backends for vLLM, allowing you to extend effective KV cache capacity beyond GPU memory using CPU RAM and disk storage. Each backend integrates through vLLM’s connector interface and works with both aggregated and disaggregated serving.

LMCache

LMCache is an open-source KV cache engine that provides prefill-once, reuse-everywhere caching with multi-level storage backends (CPU RAM, local storage, Redis, GDS, InfiniStore/Mooncake).

For local setup, see KV Cache Offloading.

For Kubernetes deployment, see Deploy LMCache MP.

FlexKV

FlexKV is a scalable, distributed KV cache runtime developed by Tencent Cloud’s TACO team. It supports multi-level caching (GPU, CPU, SSD), distributed KV cache reuse across nodes, and high-performance I/O via io_uring and GPUDirect Storage.

For local setup, see KV Cache Offloading.

See Also

• Routing Concepts: Routing requests based on KV cache state
• Disaggregated Serving: Prefill/decode separation architecture