Router Testing

Overview

The router has three useful test layers. When you add a non-trivial or potentially breaking feature, do not stop at the smallest local test by default. Consider extending the relevant layer or layers below so the change is covered at the same level where it can fail.

1. Rust Unit and Integration Tests

Use Rust tests in lib/kv-router and lib/llm for local correctness:

• cost-model math
• indexer behavior
• event application
• recovery and persistence logic
• sequence-tracking invariants
• remote-indexer query behavior

These tests should be the first line of defense for narrowly scoped logic. They are the right place to pin exact edge cases and regressions close to the implementation.

Examples:

• lib/kv-router/src/indexer/tests.rs
• lib/kv-router/src/sequences/*
• lib/llm/src/kv_router/indexer/worker_query.rs

Typical commands:

$
cargo test -p dynamo-kv-router
$
cargo test -p dynamo-llm --no-default-features

2. Bench-Backed E2E Invariant Tests

Use the fixture-backed tests in lib/bench/tests when you want a realistic replay path without launching the full router stack. These tests share the same replay machinery as the Mooncake and active-sequences benches, but run in the Rust test profile and assert invariants instead of reporting benchmark numbers.

Current coverage uses the checked-in 1000-line Mooncake trace fixture:

• active_sequences_trace.rs
• mooncake_trace.rs
• mooncake_trace_1000.jsonl

These tests are useful for catching regressions such as:

• state not draining at the end of replay
• unexpected WARN or ERROR logs on hot paths
• duplicate-store or similar warning metrics
• indexer-specific replay behavior differences across implementations

Typical command:

$
cargo test --package dynamo-bench --all-targets

Use this layer when a feature changes router behavior over time, depends on realistic event orderings, or should hold across multiple indexer implementations.

3. Full Router E2E Process Tests

Use the Python tests in tests/router when you need the full request plane and event plane in play. These tests launch router and mocker or backend processes and exercise cross-process behavior that bench-backed replay cannot cover.

Current entry points include:

• test_router_e2e_with_mockers.py
• test_router_e2e_with_vllm.py
• test_router_e2e_with_trtllm.py
• test_router_e2e_with_sglang.py

Use this layer for changes involving:

• process boundaries
• request routing through the Dynamo frontend or router service
• worker registration and discovery
• event-plane transport and delivery
• backend integration behavior
• startup, recovery, or lifecycle flows

Typical command:

$
.venv/bin/python -m pytest tests/router/test_router_e2e_with_mockers.py

Recommended Usage

When a router change is non-trivial or potentially breaking, consider the following default progression:

• Add or update Rust unit tests for the local logic.
• Add or update a bench-backed invariant test if the change affects replay ordering, indexer behavior, cache-event handling, or state-drain assumptions.
• Add or update a full tests/router E2E test if the change depends on real processes, transport, registration, or backend interaction.

Not every change needs all three layers. But if a change can break behavior outside a single module boundary, it usually deserves more than a unit test.