For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Digest
  • Getting Started
    • Quickstart
    • Introduction
    • Local Installation
    • Building from Source
    • Contribution Guide
  • Resources
    • Support Matrix
    • Feature Matrix
    • Release Artifacts
    • Examples
  • Kubernetes Deployment
    • Deployment Guide
  • User Guides
    • KV Cache Aware Routing
    • Disaggregated Serving
    • KV Cache Offloading
    • Dynamo Benchmarking
    • Multimodal
    • Diffusion (Preview)
    • Tool Calling
    • LoRA Adapters
    • Agents
    • Observability (Local)
    • Fault Tolerance
    • Writing Python Workers
  • Backends
    • SGLang
    • TensorRT-LLM
    • vLLM
  • Components
    • Frontend
    • Router
      • Router Guide
      • Router Examples
      • KV Event Replay — Dynamo vs vLLM
    • Planner
    • Profiler
    • KVBM
  • Integrations
    • LMCache
    • SGLang HiCache
    • FlexKV
    • KV Events for Custom Engines
  • Design Docs
    • Overall Architecture
    • Architecture Flow
    • Disaggregated Serving
    • Distributed Runtime
    • Blog
  • Documentation
    • Dynamo Docs Guide
NVIDIANVIDIA
Developer-friendly docs for your API
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2026, NVIDIA Corporation.

LogoLogoDocumentation
Digest
On this page
  • Quick Start
  • Standalone Router
  • Prerequisites and Limitations
  • Next Steps
Components

Router

||View as Markdown|
Edit this page
Previous

Frontend Guide

Next

Router Guide

The Dynamo KV Router intelligently routes requests by evaluating their computational costs across different workers. It considers both decoding costs (from active blocks) and prefill costs (from newly computed blocks), using KV cache overlap to minimize redundant computation. Optimizing the KV Router is critical for achieving maximum throughput and minimum latency in distributed inference setups.

Quick Start

To launch the Dynamo frontend with the KV Router:

$python -m dynamo.frontend --router-mode kv --http-port 8000

For Kubernetes, set DYN_ROUTER_MODE=kv on the Frontend service. Workers automatically report KV cache events — no worker-side configuration changes needed.

ArgumentDefaultDescription
--router-mode kvround_robinEnable KV cache-aware routing
--router-kv-overlap-score-weight1.0Balance prefill vs decode optimization (higher = better TTFT)
--no-router-kv-eventsenabledFall back to approximate routing (no event consumption from workers)
--router-queue-threshold2.0Backpressure queue threshold; enables priority scheduling via nvext.agent_hints.latency_sensitivity
--router-queue-policyfcfsQueue scheduling policy: fcfs (tail TTFT) or wspt (avg TTFT)

Standalone Router

You can also run the KV router as a standalone service (without the Dynamo frontend). See the Standalone Router component for more details.

For all CLI arguments, environment variables, K8s deployment examples, and tuning guidelines, see the Router Guide. For A/B benchmarking, see the KV Router A/B Benchmarking Guide.

Prerequisites and Limitations

Requirements:

  • Dynamic endpoints only: KV router requires register_model() with model_input=ModelInput.Tokens. Your backend handler receives pre-tokenized requests with token_ids instead of raw text.
  • Backend workers must call register_model() with model_input=ModelInput.Tokens (see Backend Guide)
  • You cannot use --static-endpoint mode with KV routing (use dynamic discovery instead)

Multimodal Support:

  • TRT-LLM and vLLM: Multimodal routing supported for images via multimodal hashes
  • SGLang: Image routing not yet supported
  • Other modalities (audio, video, etc.): Not yet supported

Limitations:

  • Static endpoints not supported—KV router requires dynamic model discovery via etcd to track worker instances and their KV cache states

For basic model registration without KV routing, use --router-mode round-robin or --router-mode random with both static and dynamic endpoints.

Next Steps

  • Router Guide: Deep dive into KV cache routing, configuration, disaggregated serving, and tuning
  • Router Examples: Python API usage, K8s examples, and custom routing patterns
  • Standalone Indexer: Run the KV indexer as a separate service for independent scaling
  • Router Design: Architecture details, algorithms, and event transport modes