> This page is for version dev. > For other versions, use one of these documentation indexes: > - Latest (v1.6.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt > - dev: https://docs.nvidia.com/dynamo/dev/llms.txt > - v1.6.1: https://docs.nvidia.com/dynamo/v1.6.1/llms.txt > - v1.6.0: https://docs.nvidia.com/dynamo/v1.6.0/llms.txt > - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt > - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt > - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt > - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt > - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt > - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt > - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt > - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt > - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt > - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt > - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt > - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt > - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt > - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt > - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt > - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt > - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt > - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt > - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt > For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Multimodal KV Routing ## Overview Multimodal KV routing extends Dynamo's KV-aware router to account for image or video content when calculating cache overlap. The frontend assigns each supported media object a stable hash and includes its identity in the routing view of the prompt. When media appears again, its identity contributes to each worker's KV overlap. The router balances that cache credit against projected prefill and decode load, so the worker with the largest overlap does not necessarily win when it is busy. Cache-aware placement increases prefix reuse without abandoning load balancing. The KV cache stores attention key/value state so a worker can skip repeated prefill work. The embedding cache stores vision encoder outputs so the encoder can skip repeated image processing. You can use both features together. See [Embedding Cache](/dynamo/dev/image-understanding/vlm-performance-optimization/embedding-cache). ## When to Use Use multimodal KV routing when: - Multiple backend workers serve multimodal requests. - Images or videos repeat across requests, such as product photos, shared reference images, or several questions about the same video. - You want to maximize KV cache reuse for multimodal content. Single-worker deployments do not need routing, and workloads with entirely unique media receive little media-specific cache benefit. This section covers image routing. For video, see [Exact Video KV Routing](/dynamo/dev/video-understanding/exact-video-kv-routing). ## How It Works The routing flow in general has three steps: 1. The frontend computes a stable identity for each supported media object. 2. The frontend represents the media in a routing-only token view that matches the backend's cache identity and token layout. 3. The KV router includes that overlap in its combined cache-and-load score, selects the lowest-cost eligible worker, and forwards the same media identity. ![A request with a repeated image reaches the frontend. The frontend hashes the image URL, reads the image size, counts image tokens, builds a routing-only prompt, and scores workers. Worker A, which holds the image's KV blocks, is selected; workers B and C are considered but not chosen.](/dynamo/_fern-files/dynamo.docs.buildwithfern.com/ebdcbc8ed16c05c2609e91fed50a3fab4fef619406599f485b95c8926006c137/assets/img/mm-kv-routing.svg) ## Image Identity By default, an HTTP image's identity hashes the exact URL bytes, including its query string. Reusing the identical URL produces the same identity, but two URLs for the same image bytes do not. Enable `--frontend-decoding` when you need content-stable identity across URLs; the frontend then hashes decoded image content. Data URLs follow the same rule: the default path hashes the full data URI string, while frontend decoding hashes the decoded bytes. vLLM provides two routing paths. Use the default Rust frontend for supported model families when you want minimal frontend processing. Use the Python chat processor when you need vLLM's broader model support or want the frontend to preprocess media and transfer the processed inputs to workers. **Default Rust frontend** For images, the frontend calculates only the identity and routing token layout unless frontend decoding is enabled. The selected worker still runs the model's multimodal processor. This path depends on the model being registered with Dynamo's Rust processor registry. **Alternative: Python chat processor** With `--dyn-chat-processor vllm`, the frontend runs vLLM's full multimodal processor. It derives routing positions from vLLM's processed image or video features without requiring a Dynamo Rust processor specification and can transfer processed inputs through shared memory or NIXL. This shifts preprocessing and transfer work to the frontend. On compatible SGLang builds, the frontend hashes each image and converts that hash into the same `pad_value` token that SGLang RadixAttention uses for its prefix-cache key. The matching token view lets the router measure image overlap before selecting a worker. Dynamo's SGLang 0.5.23 image includes hash-forwarding support but is a [wide-token build](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/sglang-multimodal#wide-token-sglang-builds), so it uses text-prefix routing for images and videos, including with `--frontend-decoding`. For video, see [Exact Video KV Routing](/dynamo/dev/video-understanding/exact-video-kv-routing). The frontend hashes each image, represents that identity in its routing token view, and forwards the hash as `multi_modal_uuids`. TensorRT-LLM workers publish matching KV events so the router can identify cached image blocks. Exact video routing is not available on this path; video requests use text-prefix routing. ## Launch Use the Python chat processor when you need vLLM's model-native multimodal processor or want to transfer processed multimodal inputs from the frontend. Otherwise, use the default Rust frontend. **Default Rust frontend** ```bash cd $DYNAMO_HOME bash examples/backends/vllm/launch/agg_multimodal_router.sh ``` **Alternative: Python chat processor** ```bash cd $DYNAMO_HOME bash examples/backends/vllm/launch/agg_multimodal_router_chat_processor.sh ``` See [vLLM Multimodal](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/vllm-multimodal#multimodal-kv-routing) for model support, hashing behavior, transfer modes, and configuration. ```bash cd $DYNAMO_HOME bash examples/backends/sglang/launch/agg_multimodal_router.sh ``` The launcher enables `--frontend-decoding`, configures KV events, and sets matching frontend and worker block sizes. See [SGLang Multimodal](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/sglang-multimodal#multimodal-kv-routing) for prerequisites, configuration, fallback behavior, and verification. ```bash cd $DYNAMO_HOME bash examples/backends/trtllm/launch/agg_multimodal_router.sh ``` The launcher enables multimodal serving, KV event publishing, block reuse, and KV-aware routing. See [TensorRT-LLM Multimodal](/dynamo/dev/knowledge-base/modular-components/backends/tensor-rt-llm/tensorrt-llm-multimodal#multimodal-kv-routing) for worker requirements, supported models, and limitations. ## Send an Image Request Send the same image twice. The first request populates the selected worker's KV cache, and the second should be routed back to it while workers are idle: ```bash for i in 1 2; do curl -s http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen3-VL-2B-Instruct", "messages": [{ "role": "user", "content": [ {"type": "text", "text": "Describe this image."}, {"type": "image_url", "image_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"}} ] }], "max_tokens": 32 }' > /dev/null done ``` Set `"model"` to the model your launcher serves. With `DYN_LOG=info,dynamo_kv_router::scheduling=debug,dynamo_llm::kv_router=debug`, the frontend logs a `[ROUTING] Best: worker_ … with / blocks overlap` line for each request; the second request should report more overlap than the first. ## Support Matrix | Backend | Routing Path | Status | Notes | |---------|--------------|--------|-------| | [vLLM](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/vllm-multimodal#multimodal-kv-routing) | Rust frontend (default) | Yes | Exact image routing supports Qwen2-VL, Qwen2.5-VL, Qwen3-VL, LLaVA 1.5, LLaVA-NeXT, Llama 4, Kimi K2.5/K2.6, Qwen3.5, Qwen3.6, and Nemotron 3 Nano Omni. Exact video routing requires frontend decoding and supports Qwen3-VL, Qwen3.5, and Nemotron 3 Nano Omni. Other media layouts use text-prefix routing. | | [vLLM](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/vllm-multimodal#multimodal-kv-routing) | Python chat processor | Yes | Uses vLLM's multimodal processor to derive image and video feature hashes and positions for models supported by vLLM. | | [SGLang](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/sglang-multimodal#multimodal-kv-routing) | Rust frontend (default) | Text prefix | Dynamo's SGLang 0.5.23 image is a [wide-token build](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/sglang-multimodal#wide-token-sglang-builds), so image and video requests use text-prefix routing. Builds before [sgl-project/sglang#43043](https://github.com/sgl-project/sglang/pull/43043) support exact image routing through hash forwarding, and exact video routing on aggregated Qwen3-VL and Qwen3.5 CUDA workers with frontend decoding. | | [TensorRT-LLM](/dynamo/dev/knowledge-base/modular-components/backends/tensor-rt-llm/tensorrt-llm-multimodal#multimodal-kv-routing) | Rust frontend (default) | Images | Exact image routing supports Qwen2-VL, Qwen2.5-VL, Qwen3-VL, and Kimi K2.5/K2.6. Video requests and other models use text-prefix routing. | Nemotron 3 Nano Omni exact image and video routing currently requires the vLLM backend. Other backends fall back to text-prefix routing for this model.