Planner Goodput-per-GPU Sweep

Enable the planner (SLA mode) on the same deployment as the router experiment, then use Spica’s smart sweep to find the planner configuration that maximizes goodput-per-GPU on the Mooncake toolagent trace, and compare against the static deployment and the default planner.

Setup

Why open-loop (not closed-loop)

A planner experiment must be open-loop. In closed-loop (fixed N in flight) the throughput is concurrency-bound — fewer workers serve the same N concurrent requests slower, so scaling down lowers throughput (hence goodput) roughly as fast as it lowers GPU count, and goodput-per-GPU does not improve (the optimum degenerates to “stay at max workers”). In open-loop the throughput is arrival-bound (fixed total tokens over the trace’s ~59 min), so adding workers does not raise throughput — it only lowers latency. The planner can then scale down in lulls without losing goodput, and goodput-per-GPU improves. That is the regime where the planner has value.

Baselines (open-loop, full trace)

The default planner already gives 2.3× the static goodput-per-GPU — same goodput, but it scales the fleet to an average of ~13.5 GPUs instead of pinning 32.

Sweep config

run_smart_search, goal.target = goodput_per_gpu. Deployment + router + workload + SLA fixed; only the planner knobs swept (full trace, open-loop):

1
search_space:
2
  model_name: MiniMaxAI/MiniMax-M2.5
3
  hardware_sku: b200_sxm
4
  backend: [vllm]
5
  deployment_mode: [agg]
6
  gpu_budget: 32
7
  context_length: 131072
8
  parallel_configs:
9
    - {tp: 4, moe_ep: 4, replicas: 8}           # fixed; planner scales replicas 1..8
10
  agg_max_num_batched_tokens: [16384]
11
  agg_max_num_seqs: [512]
12
  router_mode: [kv_router]                       # fixed tuned router
13
  overlap_score_credit: [1.0]
14
  prefill_load_scale: [4.0]
15
  router_temperature: [0.0]
16
  # swept: the planner
17
  planner_scaling_policy: [throughput_180_5, throughput_600_5, load_180_5,
18
                           load_180_10, hybrid_180_5, hybrid_600_5]
19
  planner_load_sensitivity: [aggressive, default, conservative]
20
  planner_fpm_sampling: [small, default, large, fine]
21
workload:
22
  trace_path: <toolagent_trace.jsonl>            # open-loop: no replay_concurrency
23
  trace_format: mooncake
24
goal:
25
  target: goodput_per_gpu
26
  sla: {ttft_ms: 2000, itl_ms: 50}
27
sweep:
28
  max_rounds: 6
29
  candidates_per_round: 8
30
  parallel_evals: 8

Result — goodput-per-GPU by planner-policy family

The dominant factor is the scaling-policy family (48 trials → 19 distinct configs):

Best: load_180_10 → goodput/gpu ≈ 121 (avg_gpu 8.3, goodput 999).

Static 37.6 → default planner 87.7 → optimized planner ≈ 121 — the tuned planner is 3.2× static and +38% over the default planner.

All searched configurations (full-trace sweep)

Every distinct planner config the Vizier sweep evaluated (48 trials → 19 distinct), best goodput-per-GPU first. load_scaling_down_sensitivity 70/80/90 = aggressive/default/conservative; max_num_fpm_samples is the FPM budget (small=32, default=64, large/fine=128) and only affects the predictive throughput-scaling policies — for the load_* policies it is inert (so its spread within the load_180_10 family is noise).

Takeaways

• Reactive load scaling ≫ hybrid ≫ predictive throughput scaling for goodput-per-GPU. Predictive throughput scaling over-provisions (it pre-scales to avoid SLA misses → avg_gpu ~20 → worst goodput/gpu); reactive load scaling tracks the actual load and scales down hard (avg_gpu ~8).
• The optimizer drives latency to the SLA edge to minimize GPUs: the best load_* config runs TPOT ~78 ms (near the 50 ms ITL bound) and TTFT ~1100 ms (under the 2000 ms bound), letting goodput dip to ~1000 while avg_gpu drops to ~8 → goodput/gpu peaks. The conservative predictive policies keep latency low but waste GPUs.
• Caveat: load_* policies don’t use planner_fpm_sampling (it only affects predictive throughput scaling), so the sens/fpm spread within the load_180_10 family (117–121) is mostly mocker noise (~5%). The robust conclusion is the policy family (load_180_10), not a precise sens/fpm setting.

Reproduce

$
python -m aisimulate.spica --config path/to/planner-sweep.yaml

Notes: the planner path needs the aic-forward-pass binding; a per-throughput-interval load-predictor sub-sweep runs first (forecast-loss winner pinned per interval). Static and default-planner baselines are run via dynamo.replay.run_trace_replay (static: no planner_config; planner: planner_config + the SLA). Same deployment as Router End-to-End Latency Sweep.