Planner

Why LLM Inference Needs a Different Autoscaler

Scaling a traditional web service is straightforward: watch CPU or request rate, add replicas when load is high, remove them when it’s low. Tools like HPA and KEDA work well for this because the relationship between load and latency is roughly linear — twice the requests means roughly twice the CPU, so a simple threshold policy keeps response times stable.

LLM inference breaks these assumptions:

• Latency depends on request content, not just request count. A single request with a 32K-token prompt consumes orders of magnitude more compute than a short one. Two requests per second can mean completely different GPU loads depending on input/output sequence lengths.
• Prefill and decode have different scaling characteristics. In disaggregated serving, prefill is compute-bound (scales with input length) while decode is memory-bound (scales with concurrent sequences and KV cache usage). A single replica count doesn’t capture both.
• The metrics that matter aren’t standard. The SLAs users care about — Time to First Token (TTFT) and Inter-Token Latency (ITL) — don’t map cleanly to CPU utilization or request throughput. HPA can’t target “keep P95 TTFT under 500ms” because that requires understanding the relationship between sequence lengths, GPU memory pressure, and latency.
• Scaling decisions are expensive. Spinning up a GPU worker takes minutes, not seconds. Overscaling wastes GPU-hours at cloud prices; underscaling violates SLAs. The autoscaler needs to predict demand, not just react to it.

The Dynamo Planner is an autoscaler purpose-built for these constraints. It understands engine profiling data, tracks per-worker GPU utilization, predicts traffic patterns, and makes scaling decisions that directly target TTFT and ITL SLAs — not proxy metrics.

Feature Matrix

Legend: ✅ Supported; — Not applicable.

Router mode does not constrain either scaling method. Load-based scaling consumes FPM directly from the engines through the Dynamo event plane, so it does not require the KV router. When runtime metrics are available, both scaling methods account for KV cache reuse through KV hit rate and speculative decoding through accepted tokens per forward pass. The Planner uses these signals for capacity estimates; it does not enable the engine features.

Optimization Targets and Scaling Methods

Planner configuration has two separate layers:

An optimization target is not a scaling method. In particular, the throughput optimization target uses load-based scaling with built-in thresholds; it does not enable the throughput-based scaling method.

Optimization Targets

The Planner offers four optimization targets:

Scaling Methods

The Planner implements two scaling methods:

• Throughput-based scaling (sla target only): Uses the Planner engine-query layer and traffic prediction to compute the replica count needed to meet TTFT and ITL targets. Forward-pass estimates come from the AIConfigurator compatibility API in the aisimulate wheel, with self-benchmark or profiler FPM bootstrap data and live FPM tuning. Adjusts on a longer interval (default 180s). The default throughput target instead uses load-based queue and KV-utilization thresholds.
• Load-based scaling: Uses ForwardPassMetrics (FPM) from the Dynamo event plane and queries the same Planner engine-query layer for short-term TTFT/ITL estimates. No pre-deployment data or KV Router required. Adjusts on a short interval (default 5s) to respond quickly to bursts.

When both methods are enabled for the sla target, throughput-based scaling provides a capacity floor for long-term planning while load-based scaling handles real-time adjustments above that floor.

Target and Scaling Method Compatibility

For throughput, latency, and load, the Planner enables load-based scaling, disables throughput-based scaling, and ignores both enable_*_scaling fields. The sla target is the only target that lets you select either scaling method or combine them.

We recommend starting with the default throughput target because it requires no configuration. Switch to latency for latency-sensitive workloads, load for explicit prefill queue token and decode KV cache utilization thresholds, or sla for precise SLA targeting with native AIC or FPM-based performance modeling.

New to the Planner? Start with the Planner Guide for a complete workflow including profiling and deployment.

Need multi-DGD coordination? See the Global Planner Guide for shared-policy coordination across multiple DGDs and single-endpoint multi-pool deployments.

Choose an Optimization Target

• Without a specific SLA: Choose the default throughput target for a balance of throughput and GPU use, or choose latency to scale up earlier and keep queues shorter. Both targets use load-based scaling automatically, with no SLA values or profiling data required.
• With a specific SLA: Choose the sla target and enable throughput-based and load-based scaling together. Throughput-based scaling provides a stable capacity floor, while load-based scaling responds to bursts above that floor. Native AIC or bootstrap FPMs make the performance model ready sooner; otherwise it warms from live FPMs.

For topology, target, runtime environment, dependencies, and a recommended SLA configuration, see Choose a Planner Mode.

Quick Start

Prerequisites

• Dynamo platform installed on Kubernetes (Installation Guide)
• For the sla target, kube-prometheus-stack installed (Metrics Setup)

Default Target (zero config)

The planner works out of the box with no configuration needed. By default, optimization_target is set to throughput, which uses static thresholds on queue depth and KV cache utilization — no SLAs or profiling required:

1
# Minimal planner config — uses throughput optimization by default
2
features:
3
  planner:
4
    mode: disagg
5
    backend: vllm

For latency-sensitive workloads:

1
features:
2
  planner:
3
    mode: disagg
4
    backend: vllm
5
    optimization_target: latency

SLA-Based Scaling (advanced)

For precise SLA targeting with native AIC estimates, optional bootstrap profiling data, or live FPM warmup, set optimization_target: sla:

1
features:
2
  planner:
3
    optimization_target: sla
4
    enable_throughput_scaling: true
5
    enable_load_scaling: true
6
    ttft_ms: 500.0
7
    itl_ms: 50.0
8
    pre_deployment_sweeping_mode: rapid

The fastest path to SLA-based scaling is through a DynamoGraphDeploymentRequest, which automatically profiles your model. See DGDR Templates for copyable DGDR manifests.

See Planner Guide for the full workflow.

Current Limitations

Load-based scaling

Load-based scaling has the following known limitations. Throughput-based scaling is not affected by any of these.

Requires ForwardPassMetrics (FPM). Load-based scaling uses per-engine per-iteration metrics delivered via the Dynamo event plane (ForwardPassMetrics). The KV Router is not required for load-based scaling. FPM availability by backend:

• vLLM — supported. Automatically enabled when the engine uses InstrumentedScheduler and DYN_FORWARDPASS_METRIC_PORT is set.
• TensorRT-LLM — supported, including attention-DP when the runtime emits the required iteration statistics. Dynamo publishes one FPM channel per attention-DP rank and disables FPM emission if the runtime schema is missing required fields.
• SGLang — supported. Enabled when DYN_FORWARDPASS_METRIC_PORT is set; requires the upstream FPM module, which ships in the SGLang runtime as of sglang==0.5.13.post1 (SGLang >= v0.5.13). See the SGLang FPM section.

General

In-flight requests during scale-down. When the Planner scales down a worker, the worker is terminated without waiting for in-flight requests to complete. Requests that were mid-prefill on the terminated worker will fail. In disaggregated deployments, this can also affect decode workers that were waiting on KV cache transfers from the terminated prefill worker. Workaround: For an aggregated deployment, set min_endpoint. For a disaggregated deployment, set min_endpoint to apply the same floor to prefill and decode, or set prefill_min_endpoint and decode_min_endpoint when the components need different floors. For a single-component deployment, set the active role’s field; when that field is unset, min_endpoint supplies the active role. Use a lower load_scaling_down_sensitivity value to reduce the frequency of scale-down events.

Documentation

Configuration Reference

Key PlannerConfig Fields

The planner process is launched with --config /path/to/planner_config.json. DGDR planner features and generated ConfigMaps are materialized into these PlannerConfig fields.

Environment Variables

Monitoring

Grafana Dashboard

Deploy the planner dashboard:

$
kubectl apply -n monitoring -f deploy/observability/grafana-planner-dashboard-configmap.yaml

The dashboard shows:

• Worker counts and GPU usage over time
• Observed TTFT, ITL, request rate, sequence lengths
• Predicted load and recommended replica counts
• Engine perf model status

Prometheus Metrics

When PLANNER_PROMETHEUS_PORT is set, the planner serves its own metrics endpoint. Exported series use the dynamo_planner_* naming convention (underscores and standard unit suffixes), replacing older planner:*-style names.

Throughput-based scaling pulls traffic metrics from the cluster-wide Prometheus server:

• Request count and duration
• TTFT and ITL distributions
• Input/output sequence lengths

Planner can read these traffic signals from either the public Frontend or a pool-local LocalRouter. Use throughput_metrics_source: "frontend" for a single-DGD deployment. Use throughput_metrics_source: "router" for GlobalPlanner / multi-pool deployments so each pool Planner reads its own router traffic instead of the shared public endpoint.

The throughput planner uses request count, ISL, OSL, and optional KV hit rate as the core traffic forecast inputs. The router component metric supplies KV hit rate for both traffic metric sources. TTFT, ITL, and request duration are also scraped and exported as observed diagnostics.

The request-count metrics increase when the Frontend accepts a request or the router scheduler admits it, before the response completes. Planner falls back to the corresponding completed-request counter when an older Frontend or router does not expose the started counter. This compatibility fallback can underestimate demand during backpressure.

Load-based scaling uses ForwardPassMetrics (FPM) from the Dynamo event plane:

• Per-iteration wall time, scheduled prefill/decode tokens, and queued request status
• Delivered via FpmEventSubscriber with automatic engine discovery and lifecycle tracking
• No router /metrics scraping required

FPM observes engine-side scheduled and queued work. It does not include requests still queued in the LocalRouter before engine assignment.

Core gauges on the planner port include replica counts (dynamo_planner_num_prefill_replicas, dynamo_planner_num_decode_replicas), observed traffic (dynamo_planner_observed_*), replica recommendations (dynamo_planner_predicted_num_prefill_replicas, dynamo_planner_predicted_num_decode_replicas), and cumulative dynamo_planner_gpu_hours.

Throughput prediction gauges dynamo_planner_predicted_requests_per_second, dynamo_planner_predicted_input_sequence_tokens, and dynamo_planner_predicted_output_sequence_tokens are wired from throughput-scaling traffic prediction and exposed alongside observed sequence-length metrics.

Advisory mode

Set advisory: true to run the local Planner in suggestion-only mode. This is recommended when you are evaluating a new Planner configuration, validating SLA targets, or reviewing how the Planner would react to production traffic before allowing it to scale workers.

In advisory mode, the Planner still observes traffic and FPM data, computes recommended prefill and decode replica counts, logs recommendation summaries, exports predicted replica metrics, and includes recommendations in diagnostics reports. The recommendations are not applied as scaling decisions: the Planner does not execute scaling actions, send replica changes to Kubernetes or GlobalPlanner, or mutate the deployment.

Diagnostics metrics

Additional series support dashboards and offline analysis:

• Perf-model latency estimates: dynamo_planner_estimated_ttft_ms and dynamo_planner_estimated_itl_ms reflect the maximum estimated TTFT and ITL from the engine perf model across engines.
• Engine capacity: dynamo_planner_engine_prefill_capacity_requests_per_second and dynamo_planner_engine_decode_capacity_requests_per_second report single-engine prefill and decode capacity under the configured SLA.
• Scaling decision reasons: dynamo_planner_load_scaling_decision and dynamo_planner_throughput_scaling_decision are Enum gauges whose state labels encode why each mode chose to scale, hold, or skip (for example scale_up, no_fpm_data, set_lower_bound).
• Per-engine FPM queue depths: dynamo_planner_engine_queued_prefill_tokens, dynamo_planner_engine_queued_decode_kv_tokens, and dynamo_planner_engine_inflight_decode_kv_tokens are labeled with worker_id and dp_rank for each engine.

HTML diagnostics reports

The planner can emit periodic, self-contained HTML diagnostics files with interactive Plotly charts.

Configure this in PlannerConfig (or the equivalent YAML / constructor wiring your deployment uses):

• report_interval_hours: interval in simulated time between reports (default 24.0 hours); set to None to disable.
• report_output_dir: directory where HTML files are written (default ./planner_reports).
• live_dashboard_port: port for a real-time HTTP dashboard (default 8080). Set to 0 to disable. An aiohttp server starts on the given port and serves the current accumulated snapshot data as an interactive Plotly report at http://<host>:<port>/. Unlike periodic reports, the live dashboard does not clear snapshots — it always shows all data accumulated since the last periodic report (or since startup if periodic reports are disabled).

Reports aggregate per-tick snapshots and use TickInput.now_s for timestamps, so they behave the same in live runs (wall clock) and in replay with a simulated clock. Typical charts cover worker counts, recommended replica counts, observed versus estimated latencies versus SLA targets, request rate, engine capacity, scaling decision timelines, and input/output sequence lengths. In the Replica Counts plot, actual replicas are shown as lines and the Planner’s recommended prefill and decode replica counts are shown as discrete markers at the ticks where recommendations were produced. This is especially useful with advisory: true because those recommendations are suggestions only.