GLM-5-FP8 Disaggregated Pareto Sweep

This replay-backed sweep targets the InferenceX/SemiAnalysis (SA) GLM-5-FP8 / B200 / Dynamo with SGLang / 1k-1k / disaggregated frontier. It tests whether Sweeper can discover competitive deployment mappings without pinning the parallel shape or replica count.

Setup

kv_load_ratio=1 means the candidate’s estimated decode KV capacity divided by isl + floor(osl / 2); 0 maps to concurrency 1. This makes traffic pressure comparable across candidates with different decode shapes and replica counts.

Latest replay baseline

The replay baseline uses SA’s per-concurrency deployment mappings with the fixed AI Configurator and Dynamo mocker stack, block size 64, FP8 KV cache, and 10 requests per concurrency unit. It is the correct software baseline for this sweep; the earlier pre-fix replay curve is no longer used.

Hypervolume uses reference point (0, 0). Sweeper reaches 113.5% of the latest replay baseline’s hypervolume by finding deployment mappings beyond the fixed SA mapping schedule.

Search encoding

Parallel search uses the default structured projection. Vizier models:

• total used-GPU ratio;
• prefill GPU share;
• prefill and decode GPUs-per-engine targets;
• per-role attention mode (tp or dp);
• per-role MoE FFN mode (tp or ep).

Each suggestion is projected onto the nearest backend-compatible, KV-feasible enumerated configuration. Replica counts and concrete TP / DP / MoE-TP / MoE-EP fields are derived from that valid target. parallel_configs: [] requests the full enumerated pool; a user can still provide one config as a strict pin or several configs as a custom projection pool.

Configuration

The reference configuration is aisimulate/examples/sweeper/configs/glm5-disagg-pareto-frontier.yaml:

1
search_space:
2
  deployment_mode: [disagg]
3
  backend: [sglang]
4
  parallel_configs: []
5
  model_name: zai-org/GLM-5-FP8
6
  hardware_sku: b200_sxm
7
  gpu_budget: 72
8
  context_length: 2048
9
workload:
10
  isl: 1024
11
  osl: 1024
12
  kv_load_ratio: [0.0, 1.0]
13
  num_request_ratio: 10
14
goal:
15
  target: pareto
16
  pareto_objectives: [throughput_per_gpu, throughput_per_user]
17
sweep:
18
  max_rounds: 80
19
  parallel_evals: 8
20
  candidates_per_round: 8
21
  max_eval_seconds: 600

Result

The 80-round run completed 640 successful unique samples from 670 attempts in 12:06:56:

• 22 candidates were rejected by KV-capacity prechecks;
• 8 replay evaluations exceeded the 600-second limit;
• 0 replay evaluations crashed;
• 155 distinct concrete parallel mappings were evaluated.

The highest-throughput point reached 1788.4 output tok/s/GPU at 23.56 tok/s/user:

prefill: 2xDEP8   (tp=1, attention_dp=8, moe_tp=1, moe_ep=8)
decode:  6xDTP8  (tp=1, attention_dp=8, moe_tp=8, moe_ep=1)
total:   64 GPUs
load:    kv_load_ratio=0.590, concurrency=11,940, requests=119,400
batch:   prefill=(32768 tokens, 256 seqs), decode=(8192 tokens, 1024 seqs)

Representative non-dominated points:

The exact Pareto filter returns 204 points because continuous load values produce many tiny tradeoffs along otherwise flat regions. The representative table and plot retain the useful shape of the frontier without treating those near-duplicates as different deployment choices.

Convergence

For this workload, 50-55 rounds is the practical quality/runtime point. The final 27 rounds after round 53 consumed about 4 hours 39 minutes for 1.05% additional hypervolume.

Reproduction Status

The packaged dependencies support this configuration’s KV-capacity and candidate-concurrency calculations. Reproducing the historical frontier still depends on AI Configurator performance database coverage for B200/SGLang/GLM-5-FP8 and on Replay behavior in the runtime snapshot.

Reproduce

$
python aisimulate/examples/sweeper/tools/run_sweep.py \
>
  --config aisimulate/examples/sweeper/configs/glm5-disagg-pareto-frontier.yaml

The AI Configurator performance model needs the aic-forward-pass binding. Dynamo’s attention-DP KV-capacity calculation multiplies per-rank capacity by attention DP and replicas before passing engine capacity to Replay.