Planner Configuration (PlannerConfig)

PlannerConfig is the configuration object for the Dynamo Planner. The Planner service parses it with Pydantic, so every field, type, default, and constraint on this page comes from the PlannerConfig model. For the deployment workflow and scaling-mode concepts, see the Planner Guide; for the autoscaler overview, see the Planner overview.

How the config is loaded

The Planner service loads PlannerConfig from a single required --config argument — a path to a .json, .yaml, or .yml file, or an inline JSON string. The service auto-detects which, loads it, and validates it against the PlannerConfig model. A value that is neither a readable file nor valid JSON fails at startup. How you supply it depends on where you run the Planner.

spec:
  features:
    planner:
      mode: disagg
      backend: vllm
      # optimization_target defaults to "throughput" — works with no further config

Core settings

Optimization target

When optimization_target is throughput, latency, or load, the Planner forces load-based scaling on and throughput-based scaling off, and it ignores ttft_ms / itl_ms. The two scaling-mode flags below apply only when optimization_target is sla.

Performance model and pre-deployment sweeping

Local execution-profile datasets

Download the AISimulate dataset snapshot into the container image during the image build, or mount a local copy before starting Planner. Use the generic AISimulate wheel; measurements do not belong inside the wheel. Use matching AISimulate Python and Rust releases that support execution profiles through the common forward-pass query API. This integration pins Python 0.13.0.dev202610290000000087 and Rust 0.13.0-dev.202610290000000087, which include that support. Use version-2 execution profiles; version 1 retains its legacy direct-phase queries and must be explicitly migrated and repinned.

The following configuration illustrates the paths to supply for a prefill role:

mode: prefill
pre_deployment_sweeping_mode: none
ais_perf_model:
  dataset_profiles:
    prefill:
      root: /opt/aisim-data
      manifest: metadata/prefill-manifest.json

Set the paths to the installed snapshot and its selected manifest. For disaggregated serving, supply separate prefill and decode sources. For mode: agg, use an aggregated source whose profile contains both phase graphs. Canonical prepared configs containing estimator_config.op_level.execution_profile also work through roles.

Planner applies its existing workload estimates, including queue scaling, prefix-cache discounts, and next-request estimates, before calling AISimulate’s common forward-pass query API. The prediction path is the same for built-in and external inputs, including mixed steps. If it cannot estimate the resulting shape, Planner skips scaling decisions that depend on that estimate.

GPU budget

The Planner does not have per-component maximum endpoint fields. max_gpu_budget, the power budget when enabled, Global Planner allocation, and cluster capacity continue to bound scale-up. At startup and for runtime updates, the Planner rejects minimum endpoint combinations that cannot fit the configured GPU or power budget. Send min_gpu_budget and max_gpu_budget with the endpoint fields to PATCH /v1/min-endpoints to update them atomically at runtime. Set either budget to -1 to disable that bound. When both budgets are enabled, min_gpu_budget must not exceed max_gpu_budget. The response from GET or PATCH includes both budgets.

Throughput-based scaling

Load-based scaling

Load prediction

Prometheus metrics

These fields default from environment variables and are excluded from serialized output. Set them in the config object or through the noted environment variable.

Advisory mode

Diagnostics and reporting

Runtime configuration

Runtime minimum endpoint API

The Planner exposes GET /v1/min-endpoints and partial PATCH /v1/min-endpoints. The request and response fields depend on the active mode:

The API has no authentication and is not exposed by a Kubernetes Service. For Kubernetes deployments, port-forward directly to the Planner pod:

kubectl port-forward pod/<planner-pod> 9086:9086
curl http://127.0.0.1:9086/v1/min-endpoints
curl --request PATCH http://127.0.0.1:9086/v1/min-endpoints \
  --header 'Content-Type: application/json' \
  --data '{"prefill_min_endpoint": 2}'

PATCH is atomic. Malformed JSON, unknown fields, and empty patches return HTTP 400. null, a negative min_endpoint, component-specific minimums below 1, fields that are inactive for the current mode, and minimums that exceed the configured GPU or power budget return HTTP 422 without changing the active configuration.

System-port reconfiguration API

The supported configuration groups are:

Fields retain their normal mode-specific meaning. For example, the explicit queue/KV thresholds are used by optimization_target: load. Predictor replacement requires optimization_target: sla with throughput scaling enabled. Scheduling intervals, AIS performance models, deployment topology, and other fields are rejected by this API.

Use the instance ID and revision returned by GET when submitting an update:

{
  "instance_id": "<instance_id from GET>",
  "expected_revision": 0,
  "request_id": "sweep-42",
  "config": {
    "ttft_ms": 800,
    "load_predictor": "kalman",
    "kalman_min_points": 5
  },
  "warmup_trace": "recent-traffic.jsonl"
}

config is a nonempty partial update. A predictor change requires either warmup_trace or explicit cold_start: true. The trace must be a single regular Dynamo/Mooncake JSONL or JSONL.GZ file inside runtime_warmup_trace_dir. Relative paths are resolved from that directory; absolute paths must also be inside it. Runtime warmup accepts at most 16 MiB of stored data and 16 MiB after decompression, and at most 4,096 aggregation intervals (including idle gaps). Directories and globs remain supported by the startup load_predictor_warmup_trace option but are rejected by runtime reconfiguration. Observations use the unchanged throughput adjustment interval. Warmup must provide enough effective samples for request count, ISL, and OSL and complete a successful forecast. SLA/budget-only updates do not require warmup. Supplying a trace explicitly also allows rewarming an unchanged predictor.

The active configuration continues serving while a candidate is prepared. Preparation has a 120-second deadline and only one preparation can run at a time. If a background fit outlives its deadline, its result cannot activate and further preparations are rejected until that work exits; updates that do not replace the predictor remain available. Invalid data, failed fitting, or a changed revision leave the active settings unchanged. Successful activation occurs between decisions and increments the revision. Updates preserve the existing prediction and scaling cadence, including changes to SLA targets, max_throughput_scaling_replicas, and predictor settings. The next scheduled throughput decision uses the active settings and the full trailing traffic window; predictor replacement does not delay that decision. Performance models, worker state, and already-admitted scaling operations are retained. applied means the configuration is active; it does not mean the deployment has finished scaling.

A successful write through the legacy minimum-endpoint API also changes the revision reported here. If it happens during preparation, the pending update fails with error_type: conflict. Updates are process-local and are not written to the Planner ConfigMap. A restart creates a new instance_id and restores startup settings. Address the intended Planner instance directly rather than distributing a conditional write across replicas.

Retry the identical request with the same request_id to retrieve its retained operation without applying it twice. The last 32 operations are retained in memory. Reusing an ID for different content returns 409; an evicted operation returns 404, so inspect the applied revision before submitting another update.

Malformed JSON returns 400; invalid or unsupported updates return 422; instance, revision, idempotency, and concurrent-update conflicts return 409; unavailable control returns 503. Requests exceeding 64 KiB return 413. Preparation failures after 202 are reported by the operation’s state, error, and error_type fields.

Scheduling and plugin pipeline

The Planner runs the builtin plugin pipeline by default. scheduling controls pipeline cadence; plugin_registration controls how plugins register, authenticate, and communicate. For the pipeline model, see the Planner Guide.

Retune (Experimental, Simulation Only)

retune makes the Planner periodically replay the last window of recorded traffic against a grid of router and Planner parameters and apply the best point by objective, SLA goodput per GPU by default. This section is the field contract. For the concept, see Periodic Retune; for windows, scoring, and decision timing, see Planner Design; to run it, see Simulate Retune.

Objectives

retune.objective takes the values of the DGDR v1beta2 spec.objective.metric enum, spelled the same, and scores each as the AISimulate optimization target the Dynamo operator maps it to. Scores are in the metric’s units, and decisions record them as baseline_score and best_score.

• SLA. Only Goodput and GoodputPerGPU count SLA-good tokens, against sla_ttft_ms and sla_itl_ms. The other objectives ignore both fields.
• Time base. A candidate replay runs until every request of the window completes, so its duration is the window plus a drain that differs by candidate. Goodput and GoodputPerGPU are rescaled to the window length; the throughput objectives are not, so they rank the candidate that finishes the window’s tokens sooner. Scoring and Outcomes gives the formulas and the reasons.
• Missing inputs. A replay that lacks an input of the objective fails that candidate instead of scoring it, for example TTFT with no first-token samples, or a per-GPU objective without GPU-hours.
• Scaling. With load or throughput scaling enabled, each candidate replay runs a Planner that sizes its own replicas, and retune.planner can search min_workers and the GPU budgets. A metric that is not divided by GPUs improves with more GPUs and would apply the candidate that provisions the most, so only GoodputPerGPU and ThroughputPerGPU are accepted. ThroughputPerGPU ignores the SLA, so with scaling it favors the candidate that serves the window on the fewest GPUs, whatever its latency.

Search domains

retune.router and retune.planner use the knob names, value syntax, and default domains of the Dynamo router and Planner adapters of aisimulate recommend --stack dynamo:

• Values. A knob takes the domain syntax of aisimulate recommend: a scalar, {choices: [...]}, {range: {min, max, step}}, or, for the load_sensitivity and load_predictor groups, {preset: ...}. Retune adds these rules:
  • A range must be enumerable: a range without step and a scale: log range are rejected. A range with min equal to max is one value. Stepped ranges are computed in decimal, so {min: 0.0, max: 0.3, step: 0.1} ends at 0.3.
  • Each knob may expand to at most max_candidates values.
  • A bare list ([0.5, 1.0]) is rejected; write {choices: [0.5, 1.0]}.
  • In YAML, write floats with a decimal point and a signed exponent (1.0e-3): YAML 1.1 reads 1e-3 as a string, which the schema rejects.
• Infeasible points. When the Planner starts, a Planner grid point that fails PlannerConfig validation or does not fit the deployment, such as a min_workers above what max_num_gpus allows, is dropped, and one warning reports how many points were dropped. Startup fails only when some value of a searched Planner knob fits in no grid point, or when a load-predictor candidate is invalid. A decision that would apply Planner fields that no longer validate or fit, for example after a budget change, is failed.
• Omitted knobs. A knob that has an AISimulate default domain searches it when omitted, as in aisimulate recommend. The retune-only knobs in the tables below keep their current value unless written. Knobs fixed at startup always keep their running value.
• Names. Write the public names below. A live name, such as router_temperature, min_endpoint, or max_gpu_budget, is rejected with the public name to use. Decisions and report.planner.retune_events record the live names: the PATCH /router_policy parameter names and the PlannerConfig field names.

Router knobs:

router: default is 3 × 9 × 4 = 108 points.

The other router policy parameters are not retunable and are rejected:

• host_cache_hit_weight and disk_cache_hit_weight: the startup router configuration also uses them to count a request’s cached tokens, which a runtime update does not change. A candidate replay would start its router with the candidate’s weights and score a router that no applied update can produce.
• shared_cache_multiplier: replay reports no shared cache hits, so every value would score the same.

Planner knobs:

The group rules are those of aisimulate recommend: to write a knob of a group, set the group to {preset: false}; a group that is omitted or {preset: default} cannot be combined with its knobs. Within these groups:

• Load sensitivity. Retune searches only load_scaling_down_sensitivity; load_min_observations keeps its running value. A preset name contributes its sensitivity alone, so with a running load_min_observations other than 5 the default preset is applied only in part. A preset mapping must set load_min_observations to the running value.
• Load predictor. The predictor is never a grid dimension: a pre-sweep picks one candidate per sweep. Candidates that give the same predictor are evaluated once ({preset: false} with the default knob domains is 256 combinations and 40 predictors). Records label a candidate with its preset name, or custom_<i> for the i-th entry. The number of candidates must not exceed max_candidates.
• Planner defaults. planner: default searches load_scaling_down_sensitivity ∈ {70, 80, 90} with load scaling and the 11 predictor presets with optimization_target: sla and throughput scaling. Without scaling it expands to nothing.

An explicit domain for a disaggregated SLA Planner with load and throughput scaling. The router grid is 3 × 1 × 3 × 2 = 18 points and the Planner grid 3 × 2 = 6, so the sweep replays 108 grid points, after a pre-sweep over three predictors:

retune:
  interval_seconds: 21600
  router:
    overlap_score_credit: {choices: [0.5, 1.0, 2.0]}
    prefill_load_scale: 1.0                               # pinned
    temperature: {range: {min: 0.0, max: 0.2, step: 0.1}} # 0, 0.1, 0.2
    decode_active_request_weight: {choices: [0.5, 1.0]}   # retune-only knob
  planner:
    load_sensitivity: {preset: false}
    load_scaling_down_sensitivity: {choices: [70, 80, 90]}
    decode_min_workers: {choices: [1, 2]}
    load_predictor: {preset: [arima_log1p, kalman_default_raw, kalman_reactive_log1p]}

Additional types

Nested object types referenced above, broken out to keep the field lists shallow.

PickedParallelConfig

A parallelism pick emitted by AIConfigurator. Referenced by AICInterpolationSpec.

AICInterpolationSpec

Everything the Planner needs to reproduce a rapid-mode AIC sweep. Written by the profiler onto the Planner ConfigMap; not normally hand-authored.

ExternalPluginEntry

One static external-plugin registration under scheduling.external_plugins. Rejects unknown fields.

InProcessPluginSpec

One in-process plugin under plugin_registration.in_process_plugins. Rejects unknown fields.

Environment variables

The Planner reads these environment variables. The Prometheus and namespace variables set the defaults for the config fields noted above; the rest control runtime behavior for specific environments.

Validation rules

The Planner enforces these cross-field rules at startup and rejects a config that violates any of them:

• ttft_ms must be greater than 0.
• report_interval_hours must be a positive finite number or null.
• fpm_sample_bucket_size must be a perfect square.
• global_planner_namespace is required when environment is global-planner.
• Under optimization_target: load, prefill modes require prefill_scale_up_queue_tokens > prefill_scale_down_queue_tokens, and decode modes require decode_scale_up_kv_rate > decode_scale_down_kv_rate.
• At least one scaling mode must be enabled. Under any optimization_target other than sla, the Planner forces load scaling on and throughput scaling off and ignores ttft_ms / itl_ms. The only exception is retune-only mode: retune set with both enable_load_scaling and enable_throughput_scaling explicitly false (see Retune).
• When ais_perf_model is set, its roles must cover the deployment: prefill and decode for disagg, the matching role for prefill/decode, or aggregated for agg. Role keys must agree with worker_type. Unknown canonical fields and the retired aic_perf_model input are rejected.
• scheduling.scale_interval_seconds must evenly divide load_adjustment_interval_seconds and, when throughput scaling is enabled, throughput_adjustment_interval_seconds.
• When both scaling modes are enabled, load_adjustment_interval_seconds must be shorter than throughput_adjustment_interval_seconds.
• retune has its own cross-field rules, listed with its fields in Retune. A live Planner rejects any config that sets retune.

Related pages