Tune the Planner

Choose an optimization target

optimization_target is the knob to set first. It has four values, in rough order of how much setup they need:

Reach for each when:

• throughput — Start here. It works out of the box with no SLA targets and no profiling data, and it’s the right default when you don’t yet have latency requirements to hit.
• latency — Switch when your workload is latency-sensitive and you’d rather over-provision than let requests queue. It uses the same thresholds as throughput with a more aggressive trigger.
• load — Use when you want to drive scaling directly from thresholds you set — prefill queue tokens and decode KV-cache utilization — instead of the built-in ones. Good when you understand your engine’s saturation points and want explicit control without committing to full SLA modeling.
• sla — Use when you need to hold a specific TTFT and ITL. This is the only target that models engine performance to translate “keep P95 TTFT under 500 ms” into replica counts. It’s the most powerful and the most involved — the rest of this page is mostly about tuning it.

1
spec:
2
  features:
3
    planner:
4
      mode: disagg
5
      backend: vllm
6
      optimization_target: latency   # throughput | latency | load | sla

Tune SLA mode: throughput vs. load scaling

Under optimization_target: sla, two scaling mechanisms decide when to change replica counts. You enable them independently, and at least one must be on.

Reach for each when:

• Throughput-based scaling gives you stable, prediction-driven capacity planning. Keep it on (the default) when traffic is reasonably predictable and you want the Planner to hold a sensible baseline rather than chase every spike.
• Load-based scaling reacts fast — it reads per-iteration engine metrics and adjusts on a short interval, so it catches bursts that prediction misses. Turn it on when traffic is spiky.
• Both together is the recommended production setup: throughput-based scaling sets the long-term floor, and load-based scaling adjusts above it in real time. When you enable both, keep throughput_adjustment_interval_seconds well above load_adjustment_interval_seconds so the two mechanisms don’t fight.

1
spec:
2
  features:
3
    planner:
4
      optimization_target: sla
5
      enable_throughput_scaling: true
6
      enable_load_scaling: true
7
      ttft_ms: 500.0
8
      itl_ms: 50.0
9
      pre_deployment_sweeping_mode: rapid

Warm up the performance model

SLA mode relies on a performance model to translate latency targets into replica counts. How fast it becomes accurate depends on pre_deployment_sweeping_mode:

Use rapid for most cases. Reach for thorough when you need measured accuracy for a production deployment or your hardware isn’t covered by AIC. With none, the Planner still starts, but throughput decisions report model_not_ready until enough live metrics accumulate — acceptable when load-based scaling is carrying real-time response and you’re comfortable with a slower warmup.

Set the guardrails

Whatever target you choose, a few knobs bound what the Planner is allowed to do. Set these before you let it scale unattended:

min_endpoint and load_scaling_down_sensitivity matter most because of how scale-down works: when the Planner removes a worker, it terminates it without draining in-flight requests, so requests mid-prefill on that worker fail. In disaggregated deployments this can also fail decode workers waiting on a KV-cache transfer from the terminated prefill worker. To reduce the blast radius, set min_endpoint above your steady-state floor and lower load_scaling_down_sensitivity so scale-down fires less often. See Current Limitations for the full caveat.

Validate before you scale for real

Before letting a new configuration move GPUs, run it in advisory mode. The Planner observes traffic, computes recommended replica counts, logs them, and exports them as metrics — but never actually scales.

1
spec:
2
  features:
3
    planner:
4
      optimization_target: sla
5
      enable_throughput_scaling: true
6
      ttft_ms: 500.0
7
      itl_ms: 50.0
8
      advisory: true

Watch the recommended counts against your actual traffic — dynamo_planner_predicted_num_prefill_replicas and dynamo_planner_predicted_num_decode_replicas, or the Replica Counts plot in the diagnostics reports, which overlays recommendations on actual replicas. Use advisory mode when validating new SLA targets, evaluating a config change, or confirming how the Planner would react to production traffic. Once the recommendations look right, drop advisory: true and let it scale.