Sweeper Traffic

<!— Generated from aisimulate/docs/sweeper/traffic.md by docs/fern/scripts/sync_aisimulate_docs.py. Edit the canonical source instead of this Fern copy. —>

The workload: block of a SmartSearchConfig YAML is the traffic every candidate is replayed against. Most workload fields are pinned. The exception is a ranged kv_load_ratio under a pareto goal: Vizier searches it as a continuous load dimension. The block maps to Workload in aisimulate/src/aisimulate/sweeper/config.py.

A workload is exactly one of four load shapes. The shape is inferred from which field is set (Workload._validate_workload), and each shape is either open-loop (requests arrive on a clock, independent of how fast the system drains them) or closed-loop (a fixed number of requests are kept in flight; a new one starts as one finishes).

The four load shapes

is_trace_based is trace_path is not None; is_synthetic is its complement. Setting trace_path selects shape 1; otherwise it is synthetic and exactly one of request_rate (shape 2), concurrency (shape 3), or kv_load_ratio (shape 4) selects the sub-shape.

The closed-loop in-flight cap is resolved by effective_in_flight_cap() (None = open-loop):

• trace -> replay_concurrency (so a trace is open-loop unless replay_concurrency is set);
• synthetic KV load -> the candidate-derived concurrency_override;
• synthetic fixed concurrency -> that value;
• request-rate -> None (open-loop).

Fields

Every Workload field:

The synthetic fields are isl, osl, request_rate, concurrency, kv_load_ratio, num_request_ratio; shared_prefix_ratio, num_prefix_groups, turns_per_session, inter_turn_delay_ms are shared synthetic knobs carried by ReplaySpec.workload.

kv_load_ratio (candidate-relative concurrency)

Sweeper resolves a KV-load trial after the backend, parallel shape, replicas, and batching knobs have been selected. For every active role, it asks AI Configurator for the per-rank KV token capacity using that candidate’s max_num_batched_tokens, max_num_seqs, memory fraction, parallel shape, and MTP setting. The scheduler-visible role capacity is:

per_rank_usable_tokens = floor(per_rank_tokens / block_size) * block_size
role_capacity_tokens = per_rank_usable_tokens * attention_dp * replicas

Attention-DP ranks own independent sequence pools, so capacity is multiplied by attention_dp; TP/EP ranks shard the same sequences and are not multipliers. For disagg, both prefill and decode are checked for candidate-specific memory feasibility, but only decode capacity drives load. For agg, agg capacity drives load.

The concrete closed-loop cap is:

average_tokens_per_request = isl + floor(osl / 2)
capacity_concurrency = floor(role_capacity_tokens / average_tokens_per_request)
concurrency = max(1, floor(kv_load_ratio * capacity_concurrency))

kv_load_ratio = 0 therefore maps to the minimum concurrency 1; 1 means estimated 100% steady-state KV occupancy. It is an estimate, not a guarantee that replay sees no temporary KV pressure or request retraction. Batching combinations that leave no KV budget are reported to Vizier as infeasible before replay.

Every resulting candidate records kv_load_ratio, the derived concurrency, kv_load_concurrency_capacity, kv_load_capacity_tokens, and per-role *_kv_capacity_tokens for traceability.

num_request_ratio (synthetic length scales with the load)

resolved_request_count(concurrency_override=None) computes the synthetic request count as

num_requests = max(1, round(num_request_ratio * load))

where load is, in precedence order: the candidate-derived concurrency_override (KV-load mode), else fixed concurrency (closed-loop), else request_rate (open-loop).

So the synthetic trace length scales with the swept load automatically: with num_request_ratio = 10, concurrency 256 yields 2560 requests, concurrency 512 yields 5120. Result is floored at 1; num_request_ratio itself is treated as 0.0 when unset (max(1, …) keeps at least one request).

Validation (Workload._validate_workload)

• Trace workload (trace_path set): must not set any synthetic field (isl, osl, request_rate, concurrency, kv_load_ratio, num_request_ratio) — error lists the offenders. replay_concurrency, if set, must be a positive int.
• Synthetic workload (no trace_path): exactly one of request_rate, concurrency, or kv_load_ratio (none / multiple -> error); isl, osl, num_request_ratio are all required; replay_concurrency is rejected (it is trace-only — use concurrency). concurrency must be one positive int; request_rate, isl, osl, and num_request_ratio must be positive. KV-load values must be finite and non-negative.
• Ranged KV load only under Pareto — [min, max] must contain exactly two values with min \< max; SmartSearchConfig._validate_kv_load_ratio_range rejects it for scalar goals. A scalar kv_load_ratio is valid for every goal. A synthetic Pareto config that omits all three load fields defaults to kv_load_ratio: [0.0, 1.0].

Replay Routing

Sweeper places the validated workload and concrete concurrency in ReplaySpec. The injected runner owns traffic execution and declares which backend/topology and runtime-hook combinations it supports.

For closed-loop traffic, ReplaySpec.concurrency is the trace’s replay_concurrency, the fixed synthetic concurrency, or the KV-load-derived candidate value. For open-loop traffic it is None. The complete goal, including an optional goodput SLA, is also preserved in ReplaySpec; the runner decides how to apply it and which normalized metrics to return.