Router End-to-End Latency Sweep

Reproduce the DynoSim router experiment, then use Spica’s smart sweep to find the KV-router configuration that minimizes mean end-to-end latency on the Mooncake toolagent trace, and validate the winner against the default-router baseline on the full trace.

Setup

The deployment is fixed (pinned parallel_configs, fixed engine batching, fixed concurrency) so any change in e2e latency is attributable to the router alone.

Sweep config

Searched with run_smart_search (Vizier GP-bandit), goal.target = e2e_latency. Search runs on a 2k-request subset (fast, representative — same long-context mix); the winner is then validated on the full trace.

1
search_space:
2
  model_name: MiniMaxAI/MiniMax-M2.5
3
  hardware_sku: b200_sxm
4
  backend: [vllm]
5
  deployment_mode: [agg]
6
  gpu_budget: 32
7
  context_length: 131072                       # covers the trace's 126k-token max
8
  parallel_configs:
9
    - {tp: 4, moe_ep: 4, replicas: 8}           # fixed: TEP, 8 workers x TP4
10
  agg_max_num_batched_tokens: [16384]
11
  agg_max_num_seqs: [512]
12
  planner_scaling_policy: [disabled]            # static — no autoscaling
13
  # swept: the router
14
  router_mode: [kv_router, round_robin]
15
  overlap_score_credit: [0.0, 0.5, 1.0]         # kv-router hard-caps this at 1.0
16
  prefill_load_scale: [0.0, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0, 16.0, 32.0]
17
  router_temperature: [0.0, 0.2, 0.5, 1.0]
18
workload:
19
  trace_path: <toolagent_trace.jsonl>           # 2k subset for the search; full trace to validate
20
  trace_format: mooncake
21
  replay_concurrency: 32
22
goal:
23
  target: e2e_latency
24
sweep:
25
  max_rounds: 6
26
  candidates_per_round: 10
27
  parallel_evals: 10

Result — full trace, c=32 (the validation)

* round_robin numbers are from the 2k subset (worst on every latency metric; not re-run full).

Optimized vs default kv_router (full trace): e2e −3.6%, TTFT −14%, TPOT −27%, throughput +3.6%, prefix reuse +12% (0.494 → 0.554).

Winning config: kv_router, overlap_score_credit=1.0, prefill_load_scale=32, router_temperature=0.0.

Why prefill_load_scale matters most (and where it saturates)

prefill_load_scale was best at the original ceiling (4.0), so the search space was extended to 32. Higher values keep helping but saturate around 16–32 (2k subset, overlap=1.0, temp=0):

Going beyond 32 (64/128) is expected to give negligible further gain.

All searched configurations (2k-subset search)

Every distinct config the Vizier sweep evaluated (2k subset, c=32), best mean-e2e first. n = how many of the 48 trials landed on it (Vizier concentrates samples on the optimum). round_robin ignores the kv-router weights (shown as —).

Takeaways

• kv_router ≫ round_robin for e2e/TTFT (cache-affine placement → higher prefix reuse → less prefill recompute → lower TTFT), matching the DynoSim blog direction.
• prefill_load_scale is the dominant router knob here: pushing it up (to ~16–32) spreads prefill load off hot workers, cutting both TTFT and (notably) TPOT. It saturates ~16–32.
• overlap_score_credit=1.0 (its hard max) and router_temperature=0.0 (fully deterministic, cache-affine routing) are best.
• The smart sweep cut mean e2e −3.6% and TPOT −27% over the default kv_router with no change to the deployment — purely router tuning.

Reproduce

$
python -m aisimulate.spica --config path/to/router-sweep.yaml

Run the winning router configuration against the full trace with dynamo.replay.run_trace_replay. The AI Configurator performance model needs the aic-forward-pass binding. To suppress router logs, set RUST_LOG=error. prefill_load_scale values up to 32 require the extended SEARCH_CHOICES in config.py.