Multimodal KV Routing
Overview
Multimodal KV routing extends Dynamo’s KV-aware router to account for image content when calculating cache overlap. The frontend assigns each image a stable hash and includes its identity in the routing view of the prompt.
When an image appears again, the KV router sends the request to the worker with the most matching KV cache blocks. This increases prefix-cache reuse and avoids repeating prefill work for cached multimodal content.
The KV cache stores attention key/value state so a worker can skip repeated prefill work. The embedding cache stores vision encoder outputs so the encoder can skip repeated image processing. You can use both features together. See Embedding Cache.
When to Use
Use multimodal KV routing when:
- Multiple backend workers serve multimodal requests.
- Images repeat across requests, such as product photos or shared reference images.
- You want to maximize KV cache reuse for multimodal content.
Single-worker deployments do not need routing, and workloads with entirely unique images receive little image-specific cache benefit.
How It Works
The routing flow in general has three steps:
- The frontend computes a stable identity for each image.
- The frontend represents the image in a routing-only token view that matches the backend’s cache identity.
- The KV router selects the worker with the highest block overlap and forwards the same image identity to that worker.
vLLM
SGLang
TensorRT-LLM
vLLM provides two routing paths. Use the default Rust frontend for supported model families when you want minimal frontend processing. Use the Python chat processor when you need vLLM’s broader model support or want the frontend to preprocess images and transfer the processed inputs to workers.
Default Rust frontend
The frontend calculates only the image identity and routing token layout. The selected worker still runs the model’s multimodal processor. This path has lower frontend overhead, but multimodal routing depends on the model being registered with Dynamo’s Rust processor registry.
Alternative: Python chat processor
With --dyn-chat-processor vllm, the frontend runs vLLM’s full multimodal processor. It supports models known to vLLM without requiring a Dynamo Rust processor specification and can transfer processed inputs through shared memory or NIXL. This shifts preprocessing and transfer work to the frontend.
Launch
vLLM
SGLang
TensorRT-LLM
Use the Python chat processor when you need vLLM’s model-native multimodal processor or want to transfer processed multimodal inputs from the frontend. Otherwise, use the default Rust frontend.
Default Rust frontend
Alternative: Python chat processor
See vLLM Multimodal for model support, hashing behavior, transfer modes, and configuration.