Planner Guide

The Dynamo Planner is an autoscaling controller that adjusts prefill and decode engine replica counts at runtime to meet latency SLAs. It reads traffic signals (Prometheus metrics or load predictor output) and engine performance profiles to decide when to scale up or down.

For a quick overview, see the Planner overview. For architecture internals, see Planner Design.

Scaling Modes

The planner supports two scaling modes that can be used independently or together:

• Throughput-based scaling (enable_throughput_scaling: true): Uses pre-deployment engine interpolation data and traffic prediction to plan capacity. Best for stable, predictable workloads. Requires profiling data generated by the Profiler.
• Load-based scaling (enable_load_scaling: true): Uses real-time per-worker engine metrics and online regression. Best for bursty or unpredictable traffic. Does not require profiling data. Requires the KV Router — see Current Limitations.

When to use which:

• Enable throughput-based scaling whenever profiling data is available. It provides stable, prediction-based capacity planning.
• Enable load-based scaling when traffic is bursty. It reacts quickly to real-time load changes.
• Enable both for the best of both worlds: throughput-based provides a capacity floor, load-based handles bursts above it. When both are enabled, use a longer throughput_adjustment_interval.

PlannerConfig Reference

The planner is configured via a PlannerConfig JSON/YAML object. When using the profiler, this is placed under the features.planner section of the DGDR spec:

1
features:
2
  planner:
3
    enable_throughput_scaling: true
4
    enable_load_scaling: false
5
    pre_deployment_sweeping_mode: rapid
6
    mode: disagg
7
    backend: vllm

Scaling Mode Fields

At least one scaling mode must be enabled.

Pre-Deployment Sweeping

When throughput-based scaling is enabled, the planner needs interpolation curves that map ISL to TTFT (prefill) and KV-cache utilization to ITL (decode). The profiler generates this data based on the pre_deployment_sweeping_mode setting. See the Profiler Guide for details on how this data is produced.

Throughput-Based Scaling Settings

Load-Based Scaling Settings

General Settings

Traffic Prediction Settings

Kalman Filter Settings

Integration with Profiler

When the profiler runs with planner enabled, it:

1. Selects the best prefill and decode engine configurations
2. Generates interpolation curves (TTFT vs ISL, ITL vs KV-cache utilization)
3. Saves the PlannerConfig and profiling data into separate Kubernetes ConfigMaps
4. Adds the planner service to the generated DGD, configured to read from those ConfigMaps

The planner receives its config via --config /path/to/planner_config.json which is mounted from the planner-config-XXXX ConfigMap. Profiling data is mounted from the planner-profile-data-XXXX ConfigMap.

See the Profiler Guide for the full profiling workflow and how to configure pre-deployment sweeping.

Hierarchical Deployments

If you want one public endpoint for a model but multiple private DGDs optimized for different request classes, use a hierarchical deployment:

• one control DGD with Frontend, GlobalRouter, and GlobalPlanner
• one or more prefill pool DGDs
• one or more decode pool DGDs

In the current workflow, run profiling independently for each intended pool, then compose the final control DGD plus pool DGDs manually. See the Global Planner Guide.

See Also

• Planner overview — Why LLM inference needs a different autoscaler
• Planner Design — Architecture and algorithm internals
• Planner Examples — DGDR YAML examples, sample configurations, advanced patterns
• Global Planner Guide — Multi-DGD coordination, shared GPU budgets, single-endpoint multi-pool deployments
• Profiler Guide — How profiling data is generated