Router Guide

Overview

The Dynamo KV Router intelligently routes requests by evaluating their computational costs across different workers. It considers both decoding costs (from active blocks) and prefill costs (from newly computed blocks), using KV cache overlap to minimize redundant computation. Optimizing the KV Router is critical for achieving maximum throughput and minimum latency in distributed inference setups. This guide helps you get started with using the Dynamo router and points to the pages that cover routing concepts, configuration, disaggregated serving, and operations in more detail.

Quick Start

The router can be deployed using Python / CLI, Kubernetes, or as a standalone component.

Python / CLI Deployment

To launch the Dynamo frontend with the KV Router:

$
python -m dynamo.frontend --router-mode kv --http-port 8000

This command:

• Launches the Dynamo frontend service with KV routing enabled
• Exposes the service on port 8000 (configurable)
• Automatically handles all backend workers registered to the Dynamo endpoint

Backend workers register themselves using the register_model API. For accurate prefix-cache state, workers must also publish KV cache events with the backend-specific event flags; otherwise the router can run in approximate mode with --no-router-kv-events.

The Frontend Configuration Reference is the canonical list of embedded-router CLI arguments, environment variables, defaults, and boolean forms. Use Configuration and Tuning for workload-specific guidance, Router Filtering for candidate eligibility, and Routing Concepts for the prefill and decode cost model.

Kubernetes Deployment

To enable the KV Router in Kubernetes, add the DYN_ROUTER_MODE environment variable to your frontend service:

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeployment
3
metadata:
4
  name: my-deployment
5
spec:
6
  services:
7
    Frontend:
8
      componentType: frontend
9
      replicas: 1
10
      envs:
11
        - name: DYN_ROUTER_MODE
12
          value: kv  # Enable KV Smart Router

Key Points:

• Set DYN_ROUTER_MODE=kv on the Frontend service only
• Configure worker-side KV event publishing when you want event-driven prefix-cache state
• Use --no-router-kv-events for approximate cache-state prediction when workers are not publishing events

For exact environment-variable mappings, see the Frontend Configuration Reference. For complete Kubernetes examples and tuning guidance, see Kubernetes Examples and Configuration and Tuning. For A/B testing and advanced K8s setup, see the KV Router A/B Benchmarking Guide.

Standalone Router

You can also run the KV router as a standalone service (without the Dynamo frontend) for disaggregated serving (e.g., routing to prefill workers), multi-tier architectures, or any scenario requiring intelligent KV cache-aware routing decisions. See the Standalone Router component for more details.

Frontend-Embedded vs. Standalone Router

The standalone router does not include the HTTP frontend (no /v1/chat/completions endpoint). It exposes only the RouterRequestMetrics via the system status server. See the Standalone Router README.

Deployment Modes

The Dynamo router can be deployed in several configurations. The table below shows every combination and when to use it:

Routing Modes (--router-mode)

Device-Aware Weighted Routing

device-aware-weighted is designed for heterogeneous fleets where workers of different compute capability, for example CPU embedding encoders alongside GPU embedding encoders, share the same endpoint.

Workers are split into CPU and non-CPU groups. The router compares a capability-normalized load across the two groups:

normalized_load = total_inflight(group) / (instance_count(group) x throughput_weight)

The throughput weight is 1 for CPU workers and DYN_ENCODER_CUDA_TO_CPU_RATIO for non-CPU workers. The next request is routed to the group with the lower normalized load, then to the least-loaded worker inside that group.

For multimodal requests, a full embedding-cache hit on one or more workers bypasses the CPU-to-non-CPU ratio. The router selects the least-loaded worker among those that hold every distinct embedding-cache key in the request. Partial hits continue through the normal weighted group selection. See Embedding Cache.

Use DYN_ENCODER_CUDA_TO_CPU_RATIO to approximate the throughput ratio of a non-CPU worker relative to one CPU worker. The default is 8.

When only one device class is present, the policy degenerates to standard least-loaded routing.

KV Event Transport Modes (within --router-mode kv)

When using KV routing, the router needs to know what each worker has cached. There are three ways to get this information:

Aggregated vs. Disaggregated Topology

Disaggregated mode is activated automatically when prefill workers register alongside decode workers. See Disaggregated Serving for details.

More Router Docs

• Routing Concepts: Cost model, worker selection, and routing primitives
• Multi-DC KV Routing and the DC Relay: DC-local exact ownership, pool publication, and cross-DC CKF search
• Frontend Configuration Reference: Canonical router flags, environment variables, defaults, and boolean forms
• Configuration and Tuning: Router behavior, transport modes, load tracking, and tuning guidance
• Disaggregated Serving: Prefill and decode routing setups
• Topology-Aware KV Transfer: Runtime metadata and decode routing constraints for topology-aware prefill/decode handoff
• Router Operations: Replicas, remote indexers, persistence, and recovery
• Router Examples: Python API usage, K8s examples, and custom routing patterns
• Router Testing: Recommended test layers for non-trivial router changes
• Standalone Indexer: Run the KV indexer as a separate service
• Standalone Selection Service: Select workers and account for reservations without forwarding requests
• Standalone Slot Tracker: Run active-request accounting as a separate service
• KV Event Replay — Dynamo vs vLLM: Gap detection and replay behavior