Planner Goodput-per-GPU Sweep

This experiment enabled the planner (SLA mode) on the same deployment as the router experiment, then used Sweeper’s smart sweep to find the planner configuration that maximized goodput-per-GPU on the Mooncake toolagent trace and compared it against the static deployment and the default planner.

Setup

Why open-loop (not closed-loop)

A planner experiment must be open-loop. In closed-loop (fixed N in flight) the throughput is concurrency-bound — fewer workers serve the same N concurrent requests slower, so scaling down lowers throughput (hence goodput) roughly as fast as it lowers GPU count, and goodput-per-GPU does not improve (the optimum degenerates to “stay at max workers”). In open-loop the throughput is arrival-bound (fixed total tokens over the trace’s ~59 min), so adding workers does not raise throughput — it only lowers latency. The planner can then scale down in lulls without losing goodput, and goodput-per-GPU improves. That is the regime where the planner has value.

Baselines (open-loop, full trace)

The default planner already gives 2.3× the static goodput-per-GPU — same goodput, but it scales the fleet to an average of ~13.5 GPUs instead of pinning 32.

Sweep config

The experiment used Sweeper.run with goal.target = goodput_per_gpu. It fixed the deployment, router, workload, and SLA, and swept only the planner settings using the full open-loop trace. The configuration below documents the retired aisimulate==0.1.0.dev1 API and is not accepted by AISimulate 0.12.0:

search_space:
  model_name: MiniMaxAI/MiniMax-M2.5
  hardware_sku: b200_sxm
  backend: [vllm]
  deployment_mode: [agg]
  gpu_budget: 32
  context_length: 131072
  parallel_configs:
    - {tp: 4, moe_ep: 4, replicas: 8}           # fixed; planner scales replicas 1..8
  agg_max_num_batched_tokens: [16384]
  agg_max_num_seqs: [512]
adapters:
  dynamo.router:
    search_space:
      mode: [kv_router]
      overlap_score_credit: [1.0]
      prefill_load_scale: [4.0]
      temperature: [0.0]
  dynamo.planner:
    search_space:
      scaling_policy:
        preset: [throughput_180_5, throughput_600_5, load_180_5,
                 load_180_10, hybrid_180_5, hybrid_600_5]
      load_sensitivity:
        preset: [aggressive, default, conservative]
      fpm_sampling:
        preset: [small, default, large, fine]
workload:
  trace_path: <toolagent_trace.jsonl>            # open-loop: no replay_concurrency
  trace_format: mooncake
goal:
  target: goodput_per_gpu
  sla: {ttft_ms: 2000, itl_ms: 50}
sweep:
  max_rounds: 6
  candidates_per_round: 8
  parallel_evals: 8

Result — goodput-per-GPU by planner-policy family

The dominant factor is the scaling-policy family (48 trials → 19 distinct configs):

Best: load_180_10 → goodput/gpu ≈ 121 (avg_gpu 8.3, goodput 999).

Static 37.6 → default planner 87.7 → optimized planner ≈ 121 — the tuned planner is 3.2× static and +38% over the default planner.

All searched configurations (full-trace sweep)

Every distinct planner config the Vizier sweep evaluated (48 trials → 19 distinct), best goodput-per-GPU first. load_scaling_down_sensitivity 70/80/90 = aggressive/default/conservative; max_num_fpm_samples is the FPM budget (small=32, default=64, large/fine=128) and only affects the predictive throughput-scaling policies — for the load_* policies it is inert (so its spread within the load_180_10 family is noise).

Takeaways

• Reactive load scaling ≫ hybrid ≫ predictive throughput scaling for goodput-per-GPU. Predictive throughput scaling over-provisions (it pre-scales to avoid SLA misses → avg_gpu ~20 → worst goodput/gpu); reactive load scaling tracks the actual load and scales down hard (avg_gpu ~8).
• The optimizer drives latency to the SLA edge to minimize GPUs: the best load_* config runs TPOT ~78 ms (near the 50 ms ITL bound) and TTFT ~1100 ms (under the 2000 ms bound), letting goodput dip to ~1000 while avg_gpu drops to ~8 → goodput/gpu peaks. The conservative predictive policies keep latency low but waste GPUs.
• Caveat: load_* policies don’t use fpm_sampling (it only affects predictive throughput scaling), so the sens/fpm spread within the load_180_10 family (117–121) is mostly mocker noise (~5%). The robust conclusion is the policy family (load_180_10), not a precise sens/fpm setting.

Reproduction Status

These results are historical and cannot be reproduced exactly. The experiment used aisimulate==0.1.0.dev1; that wheel did not include the example runner, and the experiment configuration was not published.

The original planner path required the aic-forward-pass binding; a per-throughput-interval load-predictor sub-sweep ran first (forecast-loss winner pinned per interval). Static and default-planner baselines ran via dynamo.replay.run_trace_replay (static: no planner_config; planner: planner_config + the SLA). Same deployment as Router End-to-End Latency Sweep.