Knowledge BaseModular ComponentsRouter

Router

KV cache-aware router that picks workers by combined prefill and decode cost to maximize throughput and minimize latency.

View as Markdown

The Dynamo KV Router intelligently routes requests by evaluating their computational costs across different workers. It considers both decoding costs (from active blocks) and prefill costs (from newly computed blocks), using KV cache overlap to minimize redundant computation. Optimizing the KV Router is critical for achieving maximum throughput and minimum latency in distributed inference setups.

Quick Start

To launch the Dynamo frontend with the KV Router:

$python -m dynamo.frontend --router-mode kv --http-port 8000

The Frontend Configuration Reference is the canonical reference for embedded-router flags, environment variables, defaults, and boolean forms. See Configuration and Tuning for behavioral guidance.

--router-mode kv (or DYN_ROUTER_MODE=kv) enables KV routing on the frontend, but it does not enable KV event publishing on backend workers. With the default --router-kv-events setting, missing publishers leave the router in event-driven mode without real cache state; the router does not automatically switch to approximate prediction. Configure the backend-specific publishing flags in Router Operations. If workers will not publish events, use --no-router-kv-events for approximate cache prediction or --load-aware for load-only routing.

For Kubernetes, set DYN_ROUTER_MODE=kv on the Frontend service.

Standalone Router

You can also run the KV router as a standalone service (without the Dynamo frontend). See the Standalone Router component for more details.

For deployment modes and quick start steps, see the Router Guide. For tuning guidelines, see Configuration and Tuning. For A/B benchmarking, see the KV Router A/B Benchmarking Guide.

Prerequisites and Limitations

Requirements:

  • Dynamic endpoints only: KV router requires register_model() with model_input=ModelInput.Tokens. Your backend handler receives pre-tokenized requests with token_ids instead of raw text.
  • Backend workers must call register_model() with model_input=ModelInput.Tokens (see Backend Guide)
  • Use dynamic discovery with KV routing so the router can track worker instances and KV cache state

Multimodal Support:

  • Image routing via multimodal hashes: Supported in the documented TRT-LLM and vLLM router paths.
  • Other backend or modality combinations: Check the backend-specific multimodal docs before relying on multimodal hash routing.

Limitations:

  • Static endpoints are not supported with KV routing; use dynamic discovery so the router can track worker instances and KV cache state

For basic model registration without KV routing, use --router-mode round-robin, --router-mode random, --router-mode power-of-two, --router-mode least-loaded, or --router-mode device-aware-weighted with both static and dynamic endpoints.

Next Steps