> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Multimodal KV Routing

## Overview

Multimodal KV routing extends Dynamo's KV-aware router to account for image content when calculating cache overlap. The frontend assigns each image a stable hash and includes its identity in the routing view of the prompt.

When an image appears again, its identity contributes to each worker's KV overlap. The router balances that cache credit against projected prefill and decode load, so the worker with the largest overlap does not necessarily win when it is busy. Cache-aware placement increases prefix reuse without abandoning load balancing.

<Info>
The KV cache stores attention key/value state so a worker can skip repeated prefill work. The embedding cache stores vision encoder outputs so the encoder can skip repeated image processing. You can use both features together. See [Embedding Cache](/dynamo/dev/multimodal/embedding-cache).
</Info>

## When to Use

Use multimodal KV routing when:

- Multiple backend workers serve multimodal requests.
- Images repeat across requests, such as product photos or shared reference images.
- You want to maximize KV cache reuse for multimodal content.

Single-worker deployments do not need routing, and workloads with entirely unique images receive little image-specific cache benefit.

## How It Works

The routing flow in general has three steps:

1. The frontend computes a stable identity for each image.
2. The frontend represents the image in a routing-only token view that matches the backend's cache identity.
3. The KV router includes that overlap in its combined cache-and-load score, selects the lowest-cost eligible worker, and forwards the same image identity.

By default, an HTTP image's identity hashes the exact URL bytes, including its
query string. Reusing the identical URL produces the same identity, but two URLs
for the same image bytes do not. Enable `--frontend-decoding` when you need
content-stable identity across URLs; the frontend then hashes decoded image
content. Data URLs follow the same rule: the default path hashes the full data
URI string, while frontend decoding hashes the decoded bytes.

<Tabs>
  <Tab title="vLLM" language="vllm">
    vLLM provides two routing paths. Use the default Rust frontend for supported model families when you want minimal frontend processing. Use the Python chat processor when you need vLLM's broader model support or want the frontend to preprocess images and transfer the processed inputs to workers.

    **Default Rust frontend**

    The frontend calculates only the image identity and routing token layout. The selected worker still runs the model's multimodal processor. This path has lower frontend overhead, but multimodal routing depends on the model being registered with Dynamo's Rust processor registry.

    **Alternative: Python chat processor**

    With `--dyn-chat-processor vllm`, the frontend runs vLLM's full multimodal processor. It supports models known to vLLM without requiring a Dynamo Rust processor specification and can transfer processed inputs through shared memory or NIXL. This shifts preprocessing and transfer work to the frontend.

  </Tab>
  <Tab title="SGLang" language="sglang">
    The frontend hashes each image and converts that hash into the same `pad_value` token that SGLang RadixAttention uses for its prefix-cache key. The matching token view lets the router measure image overlap before selecting a worker.

    Dynamo's SGLang image includes the required hash-forwarding support.
  </Tab>
  <Tab title="TensorRT-LLM" language="trtllm">
    The frontend hashes each image, represents that identity in its routing token view, and forwards the hash as `multi_modal_uuids`. TensorRT-LLM workers publish matching KV events so the router can identify cached image blocks.

  </Tab>
</Tabs>

## Launch

<Tabs>
  <Tab title="vLLM" language="vllm">
    Use the Python chat processor when you need vLLM's model-native multimodal processor or want to transfer processed multimodal inputs from the frontend. Otherwise, use the default Rust frontend.

    **Default Rust frontend**

    ```bash
    cd $DYNAMO_HOME
    bash examples/backends/vllm/launch/agg_multimodal_router.sh
    ```

    **Alternative: Python chat processor**

    ```bash
    cd $DYNAMO_HOME
    bash examples/backends/vllm/launch/agg_multimodal_router_chat_processor.sh
    ```

    See [vLLM Multimodal](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/vllm-multimodal#multimodal-kv-routing) for model support, hashing behavior, transfer modes, and configuration.
  </Tab>
  <Tab title="SGLang" language="sglang">
    ```bash
    cd $DYNAMO_HOME
    bash examples/backends/sglang/launch/agg_multimodal_router.sh
    ```

    The launcher configures KV events and matching frontend and worker block sizes.

    See [SGLang Multimodal](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/sglang-multimodal#multimodal-kv-routing) for prerequisites, configuration, fallback behavior, and verification.
  </Tab>
  <Tab title="TensorRT-LLM" language="trtllm">
    ```bash
    cd $DYNAMO_HOME
    bash examples/backends/trtllm/launch/agg_multimodal_router.sh
    ```

    The launcher enables multimodal serving, KV event publishing, block reuse, and KV-aware routing.

    See [TensorRT-LLM Multimodal](/dynamo/dev/knowledge-base/modular-components/backends/tensor-rt-llm/tensorrt-llm-multimodal#multimodal-kv-routing) for worker requirements, supported models, and limitations.
  </Tab>
</Tabs>

## Support Matrix

| Backend | Routing Path | Status | Notes |
|---------|--------------|--------|-------|
| [vLLM](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/vllm-multimodal#multimodal-kv-routing) | Rust frontend (default) | <Badge intent="success" minimal>Yes</Badge> | Supported families include Qwen2-VL, Qwen2.5-VL, Qwen3-VL, LLaVA 1.5, LLaVA-NeXT, Llama 4, Kimi K2.5/K2.6, Qwen3.5, and Qwen3.6. The rest use text-prefix-only routing. |
| [vLLM](/dynamo/dev/knowledge-base/modular-components/backends/v-llm/vllm-multimodal#multimodal-kv-routing) | Python chat processor | <Badge intent="success" minimal>Yes</Badge> | Uses vLLM’s own multimodal processor — supports any VLM that vLLM supports. |
| [SGLang](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/sglang-multimodal#multimodal-kv-routing) | Rust frontend (default) | <Badge intent="success" minimal>Yes</Badge> | Hash forwarding is upstream in SGLang 0.5.13+; Dynamo pins 0.5.16. Older custom installations need the upstream patch. |
| [TensorRT-LLM](/dynamo/dev/knowledge-base/modular-components/backends/tensor-rt-llm/tensorrt-llm-multimodal#multimodal-kv-routing) | Rust frontend (default) | <Badge intent="success" minimal>Yes</Badge> | Supported model scope is the Qwen2-VL family (Qwen2-VL / Qwen2.5-VL / Qwen3-VL) and Kimi (Kimi-K2.5 / Kimi-K2.6). Other multimodal models fall back to text-prefix routing. |