Adaptive Scale

Adaptive scale is a single-run load controller for finding and sustaining an SLA boundary. Instead of launching many independent sweep or search runs, AIPerf starts at a low control value, evaluates SLA windows, increases load while every SLA filter passes, and then sustains near the last passing boundary.

Use adaptive scale when you want one benchmark invocation to push a service until a latency or throughput constraint starts failing, then keep pressure near that edge. Use adaptive_search when you want offline Bayesian optimization across multiple runs, sweeps when you want a fixed experiment grid, fixed ramps when you already know the schedule you want to replay, and static concurrency or request rate when you only need one fixed load point.

YAML example

YAML is the preferred way to configure adaptive scale because it keeps the control variable, assessment windows, sustain period, and SLA filters together. The canonical shape uses a nested adaptive_scale.control block:

1
schemaVersion: "2.0"
2
3
benchmark:
4
  model: meta-llama/Llama-3.1-8B-Instruct
5
  endpoint:
6
    url: http://localhost:8000/v1/chat/completions
7
    type: chat
8
    streaming: true
9
  dataset:
10
    type: synthetic
11
    entries: 1000
12
    prompts: {isl: 512, osl: 128}
13
  phases:
14
    - name: profiling
15
      type: concurrency
16
      concurrency: 200
17
      prefill_concurrency: 64
18
      duration: 3600
19
      adaptive_scale:
20
        enabled: true
21
        control:
22
          variable: prefill_concurrency
23
          min: 1
24
          max: 64
25
        assessment_period: 60
26
        min_completed_requests: 20
27
        sustain_duration: 1800
28
        strategy:
29
          type: ramp_until_fail
30
          step_policy: sla_margin
31
          base_step: 10
32
          max_step_multiplier: 4
33
      sla:
34
        request_latency:
35
          p95:
36
            le: 30000
37
        request_throughput:
38
          avg:
39
            ge: 1

A window passes only when every SLA filter passes. Latency thresholds use milliseconds. Lower-is-better metrics usually use lt or le; higher-is-better metrics such as request throughput and goodput ratio usually use gt or ge.

The canonical adaptive shape uses adaptive_scale.control for the load-control variable and phase-level sla for the filters. Existing flat concurrency aliases such as min_concurrency and max_concurrency are still accepted for compatibility, but new configs should use control.min and control.max.

Do not combine adaptive scale with a fixed ramp on the same variable. For example, control.variable: prefill_concurrency cannot be used with prefill_ramp, and control.variable: request_rate cannot be used with rate_ramp.

Boundary Example

This example demonstrates adaptive scale discovering a latency boundary against an OpenAI-compatible chat endpoint. Make sure the endpoint is reachable at localhost:8000, then tune the model and SLA threshold for your service.

1
schemaVersion: "2.0"
2
3
benchmark:
4
  model: Qwen/Qwen3-0.6B
5
  endpoint:
6
    url: http://localhost:8000/v1/chat/completions
7
    type: chat
8
    streaming: true
9
  dataset:
10
    type: synthetic
11
    entries: 1000
12
    prompts:
13
      isl: 128
14
      osl: 16
15
  phases:
16
    - name: profiling
17
      type: concurrency
18
      concurrency: 4
19
      duration: 45
20
      adaptive_scale:
21
        enabled: true
22
        control:
23
          variable: concurrency
24
          min: 4
25
          max: 64
26
        assessment_period: 5
27
        min_completed_requests: 5
28
        sustain_duration: 10
29
        strategy:
30
          type: ramp_until_fail
31
          step_policy: fixed_percent_step
32
          step_percent: 100
33
      sla:
34
        request_latency:
35
          p95:
36
            le: 80

Run it with:

$
aiperf profile \
>
  --config adaptive-scale-boundary.yaml \
>
  --tokenizer builtin \
>
  --extra-inputs ignore_eos:true \
>
  --ui none \
>
  --output-artifact-dir ./adaptive-scale-boundary-artifacts

Sample Output (Successful Run):

concurrency 4  -> request_latency:p95 = 54.1 ms, pass
concurrency 8  -> request_latency:p95 = 58.7 ms, pass
concurrency 16 -> request_latency:p95 = 56.1 ms, pass
concurrency 32 -> request_latency:p95 = 73.7 ms, pass
concurrency 64 -> request_latency:p95 = 98.2 ms, fail
sustain at 32  -> request_latency:p95 = 72.0 ms, then 74.4 ms, pass

The corresponding adaptive_scale_summary.json included:

1
{
2
  "status": "completed",
3
  "completed_reason": "sustain_duration_completed",
4
  "control_variable": "concurrency",
5
  "control_value": 32,
6
  "boundary_value": 32,
7
  "last_passing_value": 32,
8
  "first_failing_value": 64,
9
  "sustain_windows": 2,
10
  "sustain_passed_windows": 2
11
}

Exact numbers depend on your server and hardware. If the run passes at max, lower the latency threshold or raise adaptive_scale.control.max. If it fails at min, loosen the latency threshold or lower adaptive_scale.control.min.

Control variables

For users, adaptive scale changes population pressure rather than acting as another spelling of request rate.

CLI quick start

The CLI is useful for scripts and simple one-phase runs. Prefer YAML for reusable benchmark definitions.

$
aiperf profile \
>
  --url http://localhost:8000/v1/chat/completions \
>
  --model meta-llama/Llama-3.1-8B-Instruct \
>
  --endpoint-type chat \
>
  --streaming \
>
  --concurrency 400 \
>
  --benchmark-duration 3600 \
>
  --adaptive-scale \
>
  --adaptive-scale-control concurrency:1,400:int \
>
  --adaptive-scale-assessment-period 60 \
>
  --adaptive-sustain-duration 1800 \
>
  --adaptive-scale-sla request_latency:p95:le:30000

The compact control form is --adaptive-scale-control variable:min,max:type. Use int for concurrency, prefill_concurrency, and users; use float for request_rate. Do not mix compact control with expanded --adaptive-control-* flags.

Artifacts

Adaptive scale writes these timing-owned artifacts into the run artifact directory:

adaptive_scale_events.jsonl
adaptive_scale_summary.json

adaptive_scale_events.jsonl is an event stream for orchestration and post-processing. Each line includes schema_version, timestamps, event, control_variable, control_value_before, control_value_after, boundary_value, last_passing_value, first_failing_value, sla_values, and binding_sla fields. Pollers should key off explicit events such as sustain_started rather than sleeping for a fixed amount of time.

adaptive_scale_summary.json is the final controller summary. It records the discovered boundary, final control value, last passing value, first failing value, sustain status, throughput, sample counts, error counts, cancellation counts, and the evaluated candidate windows.

Artifact fields are intended for orchestration-facing consumers, so treat schema changes and field renames as compatibility events.

Metric semantics

Use request_latency for simple latency-boundary examples. Request throughput aliases and goodput_ratio can also be used as adaptive SLA filters. See Adaptive SLA metric support for the full metric and statistic matrix.