> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Routing Concepts This page explains how the Dynamo router evaluates workers, chooses a target, and fits into the request path. For CLI flags and tuning knobs, see [Configuration and Tuning](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning). ## KV Cache Routing KV cache routing optimizes large language model inference by directing requests using both reusable cache state and projected active load. Cache reuse reduces redundant prompt computation, while live prefill and decode accounting prevents cache-rich workers from becoming overloaded. KV cache reuse introduces complexity to LLM serving load balancing. While it can significantly reduce computation costs, routing strategies that ignore worker-specific KV states can lead to: - Missed cache reuse opportunities due to suboptimal worker selection - System throughput degradation from uneven request distribution across workers ## Cost Calculation ![Request tokens are hashed and evaluated using KV indexer prefix hits and slot tracker active load before the router selects the lowest-cost worker. Worker KV events update the indexer through the event plane.](https://fdr-prod-docs-files-public.s3.us-east-1.amazonaws.com/dynamo.docs.buildwithfern.com/c9a2f2869e7ba337dda3f07db1c73c2d0390ecb01fd9df362b4516354a874e10/pages-dev/assets/img/router-kv-routing-overview.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA6KXJSKKNFOCF7G4B%2F20260727%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260727T184400Z&X-Amz-Expires=604800&X-Amz-Signature=291332d5281748addce3ebe65cf4d2c62b45229369229c54212e0054dc97e05d&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject) The cost function combines three worker-specific cost terms: - **Prefill cost**: Active prompt work already assigned to the worker plus the incoming request's uncached prompt work. Device, host, disk, and shared-cache hits reduce this cost according to their configured credits. - **Decode cost**: Active KV blocks already assigned to the worker plus the blocks projected for the incoming request. - **Active-request cost**: An optional block-equivalent charge for each request already active on the worker. This term can represent decode regimes where step latency follows batch size more closely than resident KV footprint. ```text raw_prefill_blocks = active_prefill_blocks + incoming_prompt_blocks adjusted_prefill_blocks = raw_prefill_blocks - overlap_credit_blocks potential_decode_blocks = active_decode_blocks + incoming_active_blocks active_request_blocks = decode_active_request_weight * active_requests cost = prefill_load_scale * adjusted_prefill_blocks + potential_decode_blocks + active_request_blocks ``` `overlap_credit_blocks` combines the configured device, host, disk, and shared-cache credits. `overlap_score_credit_decay` can reduce the device-local portion when a cache-rich worker has excess active prefill load. `decode_active_request_weight` defaults to `0`, so the active-request term is opt-in. The router selects the lowest-cost eligible worker. For exact tuning behavior, see [Configuration and Tuning](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning#tuning-guidelines). ### Active Load Modeling The prefill and decode projections include load from the incoming request plus active load already assigned to each worker. #### Prefill Load Modeling For prefill load, the router estimates each candidate worker's uncached prompt work by subtracting its cached prefix tokens from the request's input tokens. By default, that effective prefill load remains charged at full value until the first output token marks prefill complete. With `--router-prefill-load-model aic`, the router also asks [AIConfigurator (AIC)](/dynamo/dev/additional-resources/ai-configurator-reference) for an expected prefill duration using the effective ISL and cached prefix length. The active load tracker uses the oldest active prefill as a time anchor and applies elapsed time to the worker's aggregate modeled prefill backlog. When the oldest request completes, the next active prefill becomes the anchor. If a modeled non-anchor request completes first, the tracker adjusts the anchor to keep the reported load continuous. ![Timeline showing how AIC decays active prefill load across an aggregate modeled backlog, compared with static accounting until the first output token.](https://fdr-prod-docs-files-public.s3.us-east-1.amazonaws.com/dynamo.docs.buildwithfern.com/6321ccb42e793296218f385a550bf37372d171fa3af81efd8779b60068090f9c/pages-dev/assets/img/router-active-prefill-timeline.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA6KXJSKKNFOCF7G4B%2F20260727%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260727T184400Z&X-Amz-Expires=604800&X-Amz-Signature=e5ee74c3896564f4bd5d3f03fb32041fbadf28c65292c0ca0fbc418627e67381&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject) This model changes router-side prompt load accounting only; it does not change backend batching or execution. #### Decode Load Modeling For decode load, the router tracks active KV blocks and active-request count for each worker. It captures both values in the same request-specific worker-load projection. By default, the cost covers the prompt-side blocks that are already assigned to active requests and frees them when each request finishes. When `--router-track-output-blocks` is enabled, the router also adds placeholder output blocks as generation crosses block boundaries. If the request includes `nvext.agent_hints.osl`, those output blocks receive a fractional weight based on progress toward the expected output length. This expected OSL proxy lets requests near completion contribute less future decode load. Without an expected OSL, tracked output blocks count at full weight until the request finishes. When `--router-decode-active-request-weight` is positive, every active request also adds the configured block-equivalent cost. This can improve data-parallel batch balance when decode latency is sensitive to request count, but it can reduce cache locality or over-penalize batches when KV-memory traffic remains the dominant cost. The default is `0`. For the flags that enable these models, see [Configuration and Tuning](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning). ## Worker Selection The router selects the worker with the lowest cost. When `router_temperature` is set to a non-zero value, the router uses softmax sampling on the normalized cost logits to introduce randomness in the selection, which can help with load distribution. Before scoring, the router filters candidates by request allow-lists, exact pins, DP-rank bounds, required taints, and busy-threshold overload state. For those hard eligibility rules, see [Router Filtering](/dynamo/dev/knowledge-base/modular-components/router/worker-filtering). When requests wait in policy-class queues, weighted [Deficit Round Robin Queue Scheduling](/dynamo/dev/knowledge-base/modular-components/router/deficit-round-robin) selects the physical class to dispatch before worker scoring runs. ## Using the KV Cache Router To enable KV cache-aware routing, start the frontend node like this: ```bash python -m dynamo.frontend --router-mode kv ``` When KV blocks are created or removed, the engine notifies the Dynamo router, which then identifies the worker with the best matching blocks and routes traffic accordingly. To evaluate the benefits of KV-aware routing, compare your workload's performance using `--router-mode random|round-robin` against KV-aware routing. For detailed CLI arguments and advanced configuration options, see [Configuration and Tuning](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning). ## Basic Routing Dynamo supports several routing strategies when sending requests from one component to another component's endpoint. First, create a client tied to a component endpoint. Here we get a client tied to the `generate` endpoint of the `VllmWorker` component. ```python client = runtime.endpoint("dynamo.VllmWorker.generate").client() ``` You can then use the default routing methods exposed by the client class to send requests to the `VllmWorker` component. - **Random routing**: Default strategy, available via `client.generate()` or `client.random()` - **Round-robin routing**: Cycles through available workers via `client.round_robin()` - **Direct routing**: Explicitly targets a specific worker via `client.direct(input, component_id)` - **Least-loaded routing**: Routes to the worker with fewest active connections via `--router-mode least-loaded` - **Device-aware weighted routing**: Routes using CPU/non-CPU ratio budgeting plus least-loaded selection within the selected device group via `--router-mode device-aware-weighted` KV cache routing uses direct routing with a special worker selection algorithm. For benchmarking KV router performance, see the [KV Router A/B Benchmarking Guide](/dynamo/dev/recipes/benchmarks/kv-router-ab-testing). For custom routing logic and advanced patterns, see [Routing Patterns](/dynamo/dev/knowledge-base/modular-components/router/router-examples#routing-patterns). ## Device-Aware Weighted Routing `device-aware-weighted` is designed for heterogeneous fleets where CPU and non-CPU workers share the same endpoint. Instead of comparing raw in-flight counts, the router compares a capability-normalized load across the CPU and non-CPU groups, then selects the least-loaded worker within the winning group. ```text normalized_load = total_inflight(group) / (instance_count(group) x throughput_weight) ``` The throughput weight is `1` for CPU workers and `DYN_ENCODER_CUDA_TO_CPU_RATIO` for non-CPU workers. This lets the router route proportionally to device capability instead of permanently starving slower devices. A full multimodal embedding-cache hit bypasses the CPU-to-non-CPU ratio. The router instead selects the least-loaded worker among those that hold every distinct cache key in the request. Partial hits continue through the weighted group selection. See [Embedding Cache](/dynamo/dev/multimodal/embedding-cache#cache-aware-routing). When only one device class is present, the behavior degenerates to standard least-loaded routing.