Planner Configuration (PlannerConfig)
Field reference for the PlannerConfig JSON/YAML object consumed by the Dynamo Planner service.
PlannerConfig is the configuration object for the Dynamo Planner. The Planner service parses it with Pydantic, so every field, type, default, and constraint on this page comes from the PlannerConfig model. For the deployment workflow and scaling-mode concepts, see the Planner Guide; for the autoscaler overview, see the Planner overview.
How the config is loaded
The Planner service loads PlannerConfig from a single required --config argument — a path to a .json, .yaml, or .yml file, or an inline JSON string. The service auto-detects which, loads it, and validates it against the PlannerConfig model. A value that is neither a readable file nor valid JSON fails at startup. How you supply it depends on where you run the Planner.
Several Prometheus fields default from environment variables and are excluded when the config is serialized back out. Set them either in the config object or through the environment variable noted on each field. See Environment variables.
Core settings
Planner operating mode, matching the deployment topology.
Allowed values: disagg prefill decode aggInference backend the Planner scales.
Allowed values: vllm sglang trtllm mockerRuntime environment that determines how the Planner applies scaling actions.
Allowed values: kubernetes virtual global-plannerDynamo namespace of the deployment the Planner manages. Defaults from DYN_NAMESPACE, which the operator injects as {k8s_namespace}-{dgd_name}. Excluded from serialized output.
Optional model name override. Auto-detected from the deployment when unset.
Namespace where the GlobalPlanner runs. Required when environment is global-planner. See the Global Planner Guide.
Optional directory for Planner log output.
Optimization target
Scaling strategy. throughput and latency use static thresholds on queue depth and KV cache utilization — no SLA targets or profiling required. load uses user-defined prefill queue-token and decode KV-utilization thresholds. sla uses the Planner engine-query layer and AIC core performance model to target specific ttft_ms / itl_ms values.
When optimization_target is throughput, latency, or load, the Planner forces load-based scaling on and throughput-based scaling off, and it ignores ttft_ms / itl_ms. The two scaling-mode flags below apply only when optimization_target is sla.
Enable predictive, traffic-based scaling. Only honored when optimization_target is sla.
Enable reactive, load-based scaling. Only honored when optimization_target is sla. At least one scaling mode must be enabled.
Time To First Token SLA target, in milliseconds. Also accepts the alias ttft. Must be greater than 0. Used only under optimization_target: sla.
Inter-Token Latency SLA target, in milliseconds. Also accepts the alias itl. Used only under optimization_target: sla.
Performance model and pre-deployment sweeping
How to generate optional bootstrap performance data. none skips bootstrap data; rapid uses AIConfigurator to simulate engine performance (~30s); thorough measures on real GPUs (several hours).
Directory holding profiler-generated performance data (npz or JSON), used to bootstrap or tune the performance model when the get_perf_metrics endpoint is unavailable.
Role-indexed AISimulate forward-pass configurations. Constructs each model through
aisimulate_core.sdk.RustForwardPassPerfModel.best_available(config) without an
interpolation sweep. Configure model identity and estimator controls through
ais_perf_model.roles, or select local dataset manifests with dataset_profiles;
the retired aic_perf_model and hf_id / pick shape
are rejected.
Maps deployment roles to complete upstream ForwardPassPerfModelConfig objects.
Supply prefill and decode for disagg, the matching single role for prefill
or decode, and aggregated for agg. Each key must match worker_type.
Each required role must appear in exactly one of roles and dataset_profiles.
Maps prefill, decode, or aggregated to an execution-profile manifest in a
local AISimulate dataset. The manifest supplies the complete
canonical configuration; Planner does not override its model, runtime, or pin.
Required absolute path to the dataset snapshot already installed in the container. Planner performs no downloads and needs no Hugging Face credentials.
Required path relative to root, selecting an execution-profile manifest for that
role. AISimulate validates artifact hashes, the compiled graph, and retained inputs.
Planner prepares a private temporary directory once per role/source and retains it
across worker-limit changes. Provide writable temporary storage and bundled or
existing local model metadata. Restart Planner to adopt changed dataset contents.
Model identifier, AISimulate system identifier, and inference backend
(vllm, sglang, or trtllm). Optional backend_version, tp, pp,
attention_dp, MoE axes, quantization, KV block size, and speculation fields
use the upstream schema. Published worker capabilities must agree with the
configured identity.
auto searches op_level, fpm_interpolation, then fpm_regression.
Explicit modes select the named estimator.
deny rejects failure of an explicitly selected mode. allow tries the other modes in
op_level, fpm_interpolation, fpm_regression priority order; regression permits a regression fallback.
Auto selection searches the full priority order even with deny.
Ordered performance-data roots. The SDK resolves its packaged root when unset.
database_mode and transfer_policy are also passed through unchanged.
Complete upstream estimator controls, including independent regression and
correction sampling, fit settings, feature weights, and correction bounds.
Explicit settings override Planner sampling defaults. A legacy total bucket
count of 16 maps to bins_per_axis: [4, 4]. Unknown fields are rejected by
AISimulate. See the AISimulate core API.
Local execution-profile datasets
Download the AISimulate dataset snapshot into the container image during the image
build, or mount a local copy before starting Planner. Use the
generic AISimulate wheel; measurements do not belong inside the wheel. Use matching
AISimulate Python and Rust releases that support execution profiles through the
common forward-pass query API. This integration pins Python
0.13.0.dev202610290000000087 and Rust 0.13.0-dev.202610290000000087,
which include that support. Use version-2 execution profiles; version 1 retains
its legacy direct-phase queries and must be explicitly migrated and repinned.
The following configuration illustrates the paths to supply for a prefill role:
Set the paths to the installed snapshot and its selected manifest. For disaggregated
serving, supply separate prefill and decode sources. For mode: agg, use an
aggregated source whose profile contains both phase graphs. Canonical prepared configs
containing estimator_config.op_level.execution_profile also work through roles.
Planner applies its existing workload estimates, including queue scaling, prefix-cache discounts, and next-request estimates, before calling AISimulate’s common forward-pass query API. The prediction path is the same for built-in and external inputs, including mixed steps. If it cannot estimate the resulting shape, Planner skips scaling decisions that depend on that estimate.
AIConfigurator interpolation spec. Populated by the profiler in rapid mode and written onto the Planner ConfigMap; you do not normally set this by hand. When present, the Planner runs the AIC sweep in-process at bootstrap to seed the performance model. See AICInterpolationSpec.
GPU budget
Maximum total GPUs the Planner may allocate across worker types.
Per-DGD GPU floor enforced by the local Planner. -1 disables it. When set with max_gpu_budget such that min == max, the Planner pins the per-DGD total and only redistributes replicas between prefill and decode.
Minimum engine endpoints (replicas) for aggregated deployments. In disaggregated deployments, this value applies to both prefill and decode unless a role-specific value is set. In prefill-only or decode-only deployments, it supplies the active role when the corresponding role-specific value is unset. Must be nonnegative; use 0 for scale-to-zero compatibility.
Minimum prefill endpoints in disagg and prefill modes. When set, replaces the prefill value supplied by min_endpoint. Must be at least 1.
Minimum decode endpoints in disagg and decode modes. When set, replaces the decode value supplied by min_endpoint. Must be at least 1.
The Planner does not have per-component maximum endpoint fields. max_gpu_budget, the power budget when enabled, Global Planner allocation, and cluster capacity continue to bound scale-up. At startup and for runtime updates, the Planner rejects minimum endpoint combinations that cannot fit the configured GPU or power budget. Send min_gpu_budget and max_gpu_budget with the endpoint fields to PATCH /v1/min-endpoints to update them atomically at runtime. Set either budget to -1 to disable that bound. When both budgets are enabled, min_gpu_budget must not exceed max_gpu_budget. The response from GET or PATCH includes both budgets.
GPUs per decode engine replica. Auto-detected from the deployment when unset.
GPUs per prefill engine replica. Auto-detected from the deployment when unset.
Throughput-based scaling
Seconds between throughput-based scaling decisions. Also accepts the alias throughput_adjustment_interval.
Maximum replica-count change per component from one throughput observation. The Planner caps and stores the throughput lower bound, so load-scaling ticks cannot repeatedly apply the same uncapped prediction. GPU and power budgets can reduce the change further. At the nominal minimum GPU budget, scale-down-only requests hold. At a full GPU budget, two scale-up requests hold; an opposing prefill/decode proposal applies only the largest atomic pair that fits the budget. Endpoint and out-of-band budget recovery take precedence over this limit. Must be greater than 0.
Prometheus traffic source. frontend reads dynamo_frontend_* metrics from the public Frontend; router reads dynamo_component_router_* from a LocalRouter (use for a pool-local Planner in GlobalPlanner deployments).
Load-based scaling
Seconds between FPM tuning updates and load-based scaling decisions. Even when only throughput scaling is enabled, live FPM observations feed the performance model at this interval. Also accepts the alias load_adjustment_interval. Must be greater than 0 and, when both scaling modes are on, shorter than throughput_adjustment_interval_seconds.
Maximum retained ForwardPassMetrics observations for online tuning and regression fallback.
Number of buckets for observation retirement. Must be a perfect square.
Scale-down sensitivity from 0 to 100 for SLA-mode load scaling (100 = most aggressive). It sets how much
latency headroom the performance model must predict before load-based scaling removes a replica; 0 blocks
these model-based scale-downs. It does not apply when every fresh FPM observation for a pool is idle: the
Planner then removes one replica per decision, down to the larger of 1 and the endpoint floor. Set
min_endpoint (or the per-component floors) to keep idle capacity. Ignored unless optimization_target
is sla.
Minimum observations before load-based scaling decisions begin (cold-start threshold).
Prefill queue-token count that triggers scale-up. Required (with the scale-down field) when optimization_target is load and mode includes prefill. Must be greater than 0 and greater than prefill_scale_down_queue_tokens.
Prefill queue-token count that allows scale-down. Required alongside prefill_scale_up_queue_tokens.
Decode KV-utilization percentage (0–100) that triggers scale-up. Required (with the scale-down field) when optimization_target is load and mode includes decode. Must be greater than decode_scale_down_kv_rate.
Decode KV-utilization percentage (0–100) that allows scale-down. Required alongside decode_scale_up_kv_rate.
Manual fallback speculative-decoding depth. A worker’s published runtime_config.runtime_data.spec_decode.nextn takes precedence when present. Must be 0 or greater.
Load prediction
Prediction method for request count, ISL, and OSL.
Allowed values: constant arima kalman prophetApply a log1p transform to predicted request count, ISL, and OSL.
Window size (seconds) for the Prophet predictor.
Path to a warmup trace file for bootstrapping predictions.
Kalman process noise for the level component.
Kalman process noise for the trend component.
Kalman measurement noise.
Minimum data points before Kalman predictions activate.
Prometheus metrics
These fields default from environment variables and are excluded from serialized output. Set them in the config object or through the noted environment variable.
Prometheus endpoint the Planner queries for traffic metrics. Defaults to PROMETHEUS_ENDPOINT, else http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090.
Optional bearer token sent as Authorization: Bearer <token> on every PromQL request. Read once at startup.
Path to a file containing a bearer token, re-read before every request so rotated tokens are picked up without a restart.
Verify the upstream Prometheus TLS certificate. Defaults from PROMETHEUS_SSL_VERIFY (1/true/yes enable it). Pair with a CA bundle for a private CA.
Fixed key/value pairs appended as URL query parameters on every PromQL request. Set PROMETHEUS_EXTRA_QUERY_PARAMS as a URL query string, for example namespace=my-ns&tenant=foo.
Path to a CA bundle for verifying the upstream Prometheus TLS certificate. When set, the bundle is used for verification regardless of metric_pulling_prometheus_ssl_verify. Must point to an existing file.
Connection and read inactivity timeout in seconds for each Prometheus API request. Defaults from DYN_PLANNER_PROMETHEUS_REQUEST_TIMEOUT_SECONDS and must be greater than 0. This is not a total wall-clock deadline: a response that continues delivering data can run longer. The Planner does not retry failed requests within the same collection cycle.
Port on which the Planner exposes its own dynamo_planner_* metrics. Defaults from PLANNER_PROMETHEUS_PORT. 0 disables.
Advisory mode
Suggestion-only mode. The Planner computes, logs, exports, and reports recommended replica counts and retune decisions without executing any of them or changing the deployment. Use it to evaluate a new configuration or validate SLA targets against production traffic. The runtime configuration APIs, such as PATCH /v1/min-endpoints, still change the Planner’s own settings.
Planner-in-the-loop replay honors it too: an advisory replay keeps the simulated deployment at its starting replicas and router parameters, reports the recommendations in planner.recommended_scaling_events, and records a retune decision that would apply with outcome advisory. See Replay in Advisory Mode.
Diagnostics and reporting
Generate an HTML diagnostics report every N hours (simulated time). Set to null to disable. Must be a positive finite number or null.
Directory for HTML diagnostics reports.
Fixed report filename written under report_output_dir. When unset, a timestamped name is used.
Write a compressed JSONL diagnostics log (.log.jsonl.gz) next to each HTML report.
Port for the live diagnostics dashboard HTTP server. 0 disables. When enabled, visit http://host:port/ for a real-time Plotly report.
Runtime configuration
Port for the runtime minimum-endpoint API. The server listens only on 127.0.0.1; 0 disables it. Updates are process-local, are not persisted to the Planner ConfigMap, and take effect on the next planner tick.
Runtime minimum endpoint API
The Planner exposes GET /v1/min-endpoints and partial PATCH /v1/min-endpoints. The request and response fields depend on the active mode:
The API has no authentication and is not exposed by a Kubernetes Service. For Kubernetes deployments, port-forward directly to the Planner pod:
PATCH is atomic. Malformed JSON, unknown fields, and empty patches return HTTP 400. null, a negative min_endpoint, component-specific minimums below 1, fields that are inactive for the current mode, and minimums that exceed the configured GPU or power budget return HTTP 422 without changing the active configuration.
System-port reconfiguration API
Enable the Planner control API on the existing DYN_SYSTEM_PORT listener.
The runtime must have a system listener enabled. Kubernetes Planner deployments
configure this port as 9090.
Directory containing permitted runtime warmup traces. Required when submitting
warmup_trace; paths must remain inside this directory, and symlinks are rejected.
This setting cannot be changed through the runtime API.
This API changes scaling behavior and has no application authentication. Restrict
access to the system listener to trusted controllers and operators before enabling it.
The existing localhost 9086 API remains available with its original configuration.
The supported configuration groups are:
Fields retain their normal mode-specific meaning. For example, the explicit queue/KV
thresholds are used by optimization_target: load, and under any other target than sla
the Planner keeps scaling on load only, whatever the flags say. An update is validated as a
whole configuration, so it cannot turn both scaling modes off, and with both on the load
interval must stay shorter than the throughput interval. Both intervals must remain
multiples of scheduling.scale_interval_seconds, which is fixed at startup.
A new load predictor needs optimization_target: sla with throughput scaling enabled,
and warmup_trace or cold_start: true. An update needs one when it changes a predictor
field, changes throughput_adjustment_interval_seconds (the predictor’s step and the
bucket length of its history), or turns throughput scaling on. Other scaling-mode changes
keep the running predictor and the live FPM samples:
- Turning a mode on registers its builtin plugins as at startup, so they first run one interval later; turning it off unregisters them. Turning throughput scaling off, or load scaling on, also drops the last throughput targets, which would otherwise floor load scaling at a stale target.
- A mode that stays on keeps its builtin plugins and their state. After an interval change, it next runs one new interval after its last run.
- A new load interval also sets the FPM freshness window to three load intervals.
- A throughput-only SLA Planner keeps pulling FPM samples at the load interval to refit its performance model, so turning load scaling on later does not start that model cold.
scheduling.scale_interval_seconds, AIS performance models, deployment topology, and
other fields are rejected by this API.
With live retune (retune.executor: dgdr),
an update of ttft_ms or itl_ms that changes the retune scoring SLA, because retune.sla_ttft_ms
or retune.sla_itl_ms is unset, is rejected with HTTP 422. To change the SLA, restart the
Planner with the new values; to keep ttft_ms and itl_ms changeable at runtime, start it with
both retune SLA fields set.
Use the instance ID and revision returned by GET when submitting an update:
config is a nonempty partial update. A predictor change requires either warmup_trace
or explicit cold_start: true. The trace must be a single regular Dynamo/Mooncake JSONL
or JSONL.GZ file inside runtime_warmup_trace_dir. Relative paths are resolved from
that directory; absolute paths must also be inside it. Runtime warmup accepts at most
16 MiB of stored data and 16 MiB after decompression, and at most 4,096 aggregation
intervals (including idle gaps). Directories and globs remain supported by the startup
load_predictor_warmup_trace option but are rejected by runtime reconfiguration.
Observations are bucketed at the candidate’s throughput_adjustment_interval_seconds. Warmup must provide enough
effective samples for request count, ISL, and OSL and complete a successful forecast.
SLA/budget-only updates do not require warmup. Supplying a trace explicitly also allows rewarming an unchanged predictor.
The active configuration continues serving while a candidate is prepared. Preparation
has a 120-second deadline and only one preparation can run at a time. If a background
fit outlives its deadline, its result cannot activate and further preparations are
rejected until that work exits; updates that do not replace the predictor remain available.
Invalid data, failed fitting, or a changed revision leave the active settings unchanged.
Successful activation occurs between decisions and increments the revision. An update that
leaves the scaling modes and intervals unchanged preserves the existing prediction and scaling
cadence, including changes to SLA targets, max_throughput_scaling_replicas, and predictor
settings. A scaling-mode or interval change sets the new cadence as described above. The next scheduled throughput
decision uses the active settings and the full trailing traffic window; predictor replacement
does not delay that decision. Performance models, worker state, and already-admitted
scaling operations are retained. applied means the
configuration is active; it does not mean the deployment has finished scaling.
A successful write through the legacy minimum-endpoint API also changes the revision
reported here. If it happens during preparation, the pending update fails with
error_type: conflict.
Updates are process-local and are not written to the Planner ConfigMap. A restart creates
a new instance_id and restores startup settings. Address the intended Planner instance
directly rather than distributing a conditional write across replicas.
Retry the identical request with the same request_id to retrieve its retained operation
without applying it twice. The last 32 operations are retained in memory. Reusing an ID
for different content returns 409; an evicted operation returns 404, so inspect the applied revision before submitting another update.
Malformed JSON returns 400; invalid or unsupported updates return 422; instance,
revision, idempotency, and concurrent-update conflicts return 409; unavailable control
returns 503. Requests exceeding 64 KiB return 413. Preparation failures after 202
are reported by the operation’s state, error, and error_type fields.
Scheduling and plugin pipeline
The Planner runs the builtin plugin pipeline by default. scheduling controls pipeline cadence; plugin_registration controls how plugins register, authenticate, and communicate. For the pipeline model, see the Planner Guide.
Plugin-pipeline scheduling. Rejects unknown fields.
Base pipeline cadence. The pipeline wakes once per interval; each plugin’s execution_interval_seconds decides whether it fires. When unset, computed as the gcd of load_adjustment_interval_seconds and, when throughput scaling is enabled, throughput_adjustment_interval_seconds. Must be greater than 0 and evenly divide both adjustment intervals.
Outer wall-clock deadline wrapping the full 4-stage pipeline. Exceeding it aborts the tick; the next tick runs from a clean state. Replay does not apply it: a replayed tick runs to completion, however long it takes on the host. Must be greater than 0.
Static external-plugin registrations applied at startup. Per-entry failures are logged but do not crash the Planner. See ExternalPluginEntry.
gRPC registration gateway for self-registering plugins. Disabled by default.
Open the gRPC gateway at listen. Not needed for static external_plugins.
Bind address. A unix: socket path for in-Pod registration, or host:port for TCP.
Permit binding a plaintext (no-TLS) gateway on a TCP listen. Fails closed by default because the gateway receives plugins’ shared-secret tokens. unix: listens are always allowed.
Plugin registry configuration — auth, transport, heartbeat, and in-process plugins. Rejects unknown fields.
Registry authentication. In the Planner process, an empty trusted_sources activates a legacy compatibility fallback to unauthenticated access and logs a warning. This fallback is DEV ONLY; do not rely on it as an intentional configuration. Select allow_unauthenticated explicitly for development. Production deployments must configure static_secret.
Accepted auth sources. Use ['allow_unauthenticated'] for development, or ['static_secret'] with static_secrets for production.
Map of secret_value to subject_label. Populate from a mounted Kubernetes Secret rather than hard-coding in the ConfigMap.
Outbound gRPC transport settings for calling plugins.
Allow plaintext grpc:// plugin endpoints. Required to use any grpc:// endpoint; mTLS support is not yet shipped.
Per-RPC timeout applied to every plugin call. Replay does not apply it to in-process plugins: a replayed tick runs to completion, however long it takes on the host. Must be greater than 0.
gRPC keepalive time, in milliseconds.
Maximum gRPC message size (10 MB).
Minimum accepted plugin protocol version.
Maximum accepted plugin protocol version.
Reserved liveness setting. Heartbeats update registry timestamps, but no monitor currently applies this timeout or evicts plugins automatically.
Reserved liveness setting. Automatic missed-heartbeat eviction is not currently wired.
In-process plugins loaded from Python modules at startup. See InProcessPluginSpec.
Admin (ListPlugins) RBAC config. Parsed but currently inert — the gateway default-denies ListPlugins regardless of mode.
Retune (Experimental)
retune makes the Planner periodically replay the last window of recorded traffic against a grid of router and Planner parameters and apply the best point by objective, SLA goodput per GPU by default. This section is the field contract. For the concept, see Periodic Retune; for windows, scoring, and decision timing, see Planner Design; to run it in replay, see Simulate Retune; to run it on a live deployment, see Planner Live Retune.
Experimental. Where retune runs depends on executor:
local, the default, runs only inside planner-in-the-loop offline replay:dynamo.replay.run_trace_replaywithtrace_format="dynamo", oraisimulate predict --stack dynamoon a Dynamo request trace. A live Planner rejects it at startup.dgdrruns on a live Planner on Kubernetes, through DGDR v1beta2 Requests. It retunes three KV router knobs, with fixed replicas or with scaling, and with scaling the Planner fields ofretune.plannertoo. It is meant for development clusters only; see Planner Live Retune.
Periodic replay-based retune. Rejects unknown fields. Requires mode agg or disagg, and a non-empty router or planner search domain.
To retune with fixed replicas (retune-only mode), set both enable_load_scaling: false and enable_throughput_scaling: false explicitly. Only that combination skips the easy-mode forcing and the “at least one scaling mode” rule; every other combination keeps them. A retune-only Planner never scales, not even to fit min_gpu_budget/max_gpu_budget or a minimum endpoint count, runs no load loop, and pulls no forward-pass metrics. It still ticks every scheduling.scale_interval_seconds (by default the 5-second load interval); to tick less often, set scheduling.scale_interval_seconds and load_adjustment_interval_seconds together.
With load or throughput scaling enabled, scheduling.external_plugins must be empty and scheduling.gateway.enabled must be false, because every candidate replay runs an inner Planner. That inner Planner always scales and never runs batch scheduling, even when the Planner is advisory.
With advisory: true, retune keeps sweeping and scoring, but a decision that would apply is recorded with outcome advisory and the point it recommends, and nothing changes.
Window length and decision cadence T. Must be greater than 0 and a multiple of scheduling.scale_interval_seconds.
Delay after a window ends before its sweep launches. Requests that complete after the launch time are excluded from the window. Must be 0 or greater.
Trace history before each window that warms the load predictors of candidate planners and of a newly selected predictor. With throughput scaling, replay reads this history from the replayed trace and, before its first request, from load_predictor_warmup_trace, which must then be a Dynamo request trace that ends at or before the first replayed request; without throughput scaling no history is read. null resolves to the minimum the predictors need: throughput_adjustment_interval_seconds × max(prophet_window_size, kalman_min_points) over the current predictor and the load-predictor candidates, or 0 without throughput scaling. An explicit value must be 0 or greater, and a value smaller than that minimum is rejected.
Delay from sweep launch to decision. A number is a fixed delay, 0 or greater. measured uses the sweep’s own wall-clock time, capped at timeout_seconds, times decision_latency_scale; it depends on the replay host, so it is not reproducible across hosts or runs. The decision lands no earlier than one Planner tick after launch. See Decision Timing for the mechanics and Decision Latency for choosing a value. With executor dgdr, the decision lands when the sweep returns, so this must be 0, or measured with decision_latency_scale: 1.
Multiplier on the measured sweep time, for example to model a production sweep host that is slower or faster than the replay host. Must be greater than 0. A value other than 1 requires decision_latency_seconds: measured. Above 1, set timeout_seconds ≤ interval_seconds / scale (the default, interval_seconds, is rejected); the modeled deadline is then scale × timeout_seconds.
Wall-clock deadline for one sweep; its workers are killed after it and the decision is timeout. null means interval_seconds. Must be greater than 0. Requires timeout_seconds ≤ interval_seconds. A numeric decision_latency_seconds must also be ≤ timeout_seconds; with measured, decision_latency_scale × timeout_seconds must also be ≤ interval_seconds. This keeps at most one sweep in flight. With executor dgdr, it is required, must be ≤ interval_seconds − 600, and includes the window cut.
Metric each candidate replay is scored on, written as the DGDR v1beta2 spec.objective.metric value. TTFT and E2ELatency are minimized; the others are maximized. See Objectives for what each one scores. With load or throughput scaling enabled, only GoodputPerGPU and ThroughputPerGPU are accepted; retune-only mode accepts every value.
TTFT SLA of the Goodput and GoodputPerGPU objectives; other objectives ignore it. Must be greater than 0. null uses the Planner’s ttft_ms; in aisimulate predict, null takes evaluation.sla.ttft_ms when it is set (see Requirements).
ITL SLA of the Goodput and GoodputPerGPU objectives; other objectives ignore it. Must be greater than 0. null uses the Planner’s itl_ms; in aisimulate predict, null takes evaluation.sla.itl_ms when it is set (see Requirements).
Apply the best candidate only if it beats the current point’s score on the same window by more than this percentage of that score: a higher score for a maximized objective, a lower one for TTFT and E2ELatency. Must be in [0, 100). Candidate replays share one random seed, so a candidate that changes no routing or scaling decision scores exactly the current point and is kept; see Candidates and the Sweep. To choose a value, see Measure the Gain.
Upper bound on the expanded grid size (the product of all domain sizes), on the values of each knob, and separately on the number of load-predictor candidates. Must be at least 1. A grid that exceeds it is rejected with the size of each dimension. For the candidate set a sweep replays, see Candidates and the Sweep.
Number of sweep worker processes of executor local. Must be at least 1. Each worker replays one candidate at a time, so memory grows with this value and the window size. Executor dgdr ignores it and uses dgdr.run.parallelism.
Router search domain, written like the router section of aisimulate recommend --stack dynamo (see Search domains). A knob omitted from the map searches its AISimulate default domain; write a scalar to pin it. default searches the whole AISimulate default domain. null (or no router field) keeps every router parameter at its startup value; an empty map {} is rejected. A searched router requires router_mode="kv_router" (router.policy: kv_router in aisimulate predict) with the default worker-selection policy, no overlap_score_weight, and conditional disaggregation disabled.
Planner search domain, written like the planner section of aisimulate recommend --stack dynamo and limited to what a running Planner can change (see Search domains). default searches the parts of the AISimulate default domain that apply to this Planner. null (or no planner field) keeps every Planner field unchanged; an empty map {} is rejected. A written knob or group requires load or throughput scaling, and the scaling mode that uses it: for example, load_sensitivity needs load scaling, and load_predictor needs optimization_target: sla with throughput scaling. A searched field must have a value in the Planner config. ttft_ms and itl_ms are never searched.
Where sweeps run. local replays the candidates in worker processes of the Planner and runs only inside planner-in-the-loop replay; a live Planner rejects it at startup with planner.retune is replay-only with executor 'local'; use executor: dgdr. dgdr runs each sweep as DGDR v1beta2 Requests from a live Planner on Kubernetes and requires the dgdr section below. Experimental, development clusters only: see Planner Live Retune.
With dgdr, the Planner also rejects at startup a config that breaks a configuration rule of live retune: it requires environment: kubernetes, a router domain (pin its knobs with scalars to retune only the Planner) that writes only overlap_score_credit, prefill_load_scale, temperature, policy: kv_router and prefill_load_model: {type: none}, the timeout_seconds and decision_latency_seconds limits above, a dgdr.engine that follows the rules below, and a largest grid shard of at most 100 points that fits the time budget. With scaling, the Planner points are one more shard dimension, the history read takes another cut_budget_seconds, and the history before a window may hold at most 1024 buckets of the shortest throughput interval.
Settings of executor dgdr. Required when executor is dgdr, and rejected otherwise. Rejects unknown fields. Planner Live Retune shows a complete example.
ReadWriteMany PersistentVolumeClaim, in the DGD namespace, that holds the frontend traces and the window files. The frontends and the Planner mount it.
Where the Planner container mounts claim_name. Must be absolute.
Claim-relative directory of the frontend trace segments, written as <segments_dir>/<dgd>/<pod>/. /-separated names without empty, . or .. segments. segments_dir and windows_dir must not contain one another.
Claim-relative directory of the window files, written as <windows_dir>/<dgd>/<window_end_ms>-<selection>.jsonl.gz, where <selection> is a digest of the window’s start, end and cutoff. Same rules as segments_dir.
Window phase. Window k is [phase_seconds + k × interval_seconds, phase_seconds + (k+1) × interval_seconds) in epoch seconds, so a restarted Planner continues the same windows. Must be 0 or greater.
Hugging Face repository id the workers serve, written as each Request’s spec.model.name. The Run Job has no Hugging Face credentials or model mount, so it must be a public repository that AISimulate can resolve.
GPU SKU, written as each Request’s spec.hardware.gpu.sku.
The pinned AISimulate engine section, written as each Request’s spec.aisim.inputOverrides.engine. Every value must be a scalar, each worker role of mode sets one parallelism.preset entry and both scheduler knobs, and replicas must not be set: the Planner fills them in from the DGD spec at each launch. See Engine section.
Deadline of the subprocess that cuts a window file from the segments. Must be greater than 0.
Run Job start-up time reserved when deciding whether a shard can still finish before the window’s deadline. Must be 0 or greater.
How each DGDR Run evaluates its points. Rejects unknown fields.
AISimulate optimizer.parallelism of each Run. Must be at least 1. The Run Job’s CPU and memory may lower the effective value.
AISimulate optimizer.candidate_timeout_seconds of each Run. Must be greater than 0.
Most grid shards, one Request each, per window. A shard is a Cartesian sub-grid of at most 100 points. The common-random-numbers probe and off-grid points are extra single-point Requests and do not count. Must be in [1, 16].
Windows whose finished Requests are kept for audit; older ones are deleted. Must be 0 or greater.
A window with more requests fails with window_too_large. Must be at least 1.
On switching from advisory: true to false, apply the last advisory recommendation if it is at most one interval old and the configuration is unchanged.
Duration of the leadership Lease, renewed every third of it or sooner. Must be at least 37: the Planner stops acting as leader 15 seconds (three 5-second Kubernetes request timeouts) before the Lease expires, and two renewals of up to 11 seconds each must fit before then.
Period of the router and clean-up rounds. Must be greater than 0.
Objectives
retune.objective takes the values of the DGDR v1beta2 spec.objective.metric enum, spelled the same, and scores each as the AISimulate optimization target the Dynamo operator maps it to. Scores are in the metric’s units, and decisions record them as baseline_score and best_score.
- SLA. Only
GoodputandGoodputPerGPUcount SLA-good tokens, againstsla_ttft_msandsla_itl_ms. The other objectives ignore both fields. - Time base. A candidate replay runs until every request of the window completes, so its duration is the window plus a drain that differs by candidate.
GoodputandGoodputPerGPUare rescaled to the window length; the throughput objectives are not, so they rank the candidate that finishes the window’s tokens sooner. Scoring and Outcomes gives the formulas and the reasons. - Missing inputs. A replay that lacks an input of the objective fails that candidate instead of scoring it, for example
TTFTwith no first-token samples, or a per-GPU objective without GPU-hours. - Scaling. With load or throughput scaling enabled, each candidate replay runs a Planner that sizes its own replicas, and
retune.plannercan search the scaling policy. A metric that is not divided by GPUs improves with more GPUs and would apply the candidate that provisions the most, so onlyGoodputPerGPUandThroughputPerGPUare accepted.ThroughputPerGPUignores the SLA, so with scaling it favors the candidate that serves the window on the fewest GPUs, whatever its latency.GoodputPerGPUcounts only SLA-good tokens, but a candidate that provisions fewer GPUs can still score higher while it meets the SLA for fewer requests; retune has no SLA-attainment floor, so checkcompleted_requestsandgoodput_completed_requestsin the decision metrics when the scaling policy is searched.
Search domains
retune.router and retune.planner use the knob names, value syntax, and default domains of the Dynamo router and Planner adapters of aisimulate recommend --stack dynamo:
- Values. A knob takes the domain syntax of
aisimulate recommend: a scalar,{choices: [...]},{range: {min, max, step}}, or, for thescaling_policy,load_sensitivityandload_predictorgroups,{preset: ...}. Retune adds these rules:- A range must be enumerable: a range without
stepand ascale: logrange are rejected. A range withminequal tomaxis one value. Stepped ranges are computed in decimal, so{min: 0.0, max: 0.3, step: 0.1}ends at 0.3. - Each knob may expand to at most
max_candidatesvalues. - A bare list (
[0.5, 1.0]) is rejected; write{choices: [0.5, 1.0]}. - In YAML, write floats with a decimal point and a signed exponent (
1.0e-3): YAML 1.1 reads1e-3as a string, which the schema rejects.
- A range must be enumerable: a range without
- Infeasible points. When the Planner starts, a Planner grid point that fails
PlannerConfigvalidation or does not fit the deployment, such as adecode_scale_up_kv_ratenot above thedecode_scale_down_kv_rateit is combined with, or a throughput-scaling policy the configuration cannot run, is dropped, and one warning reports how many points were dropped. Startup fails only when some value of a searched Planner knob other than a scaling-policy field fits in no grid point, or when a load-predictor candidate is invalid. A decision that would apply Planner fields that no longer validate or fit, for example after a runtime threshold change, isfailed. - Omitted knobs. A knob that has an AISimulate default domain searches it when omitted, as in
aisimulate recommend. The retune-only knobs in the tables below and thescaling_policygroup keep their current value unless written. Knobs fixed at startup always keep their running value. - User constraints. Worker minimums and GPU budgets (
min_workers,prefill_min_workers,decode_min_workers,max_num_gpus,min_num_gpus, and their live names) are never searched: they bound what the Planner may provision, and every candidate replays within them. Change them through the Planner config or the runtime configuration API. - Names. Write the public names below. A live name, such as
router_temperature, is rejected with the public name to use. Decisions andreport.planner.retune_eventsrecord the live names: the PATCH/router_policyparameter names and thePlannerConfigfield names.
Router knobs:
router: default is 3 × 9 × 4 = 108 points.
The other router policy parameters are not retunable and are rejected:
host_cache_hit_weightanddisk_cache_hit_weight: the startup router configuration also uses them to count a request’s cached tokens, which a runtime update does not change. A candidate replay would start its router with the candidate’s weights and score a router that no applied update can produce.shared_cache_multiplier: a router started with it at 0 builds no KV fetch inputs, and a runtime update cannot add inputs, so a candidate that raises it could not be applied. Retune keeps its startup value.
Planner knobs:
The group rules are those of aisimulate recommend: to write a knob of a group, set the group to {preset: false}; a group that is omitted or {preset: default} cannot be combined with its knobs. Within these groups:
- Scaling policy. Each policy is one correlated choice of its four fields, so the Planner grid is a list of points rather than a full product. Retune turns a mode on or off and moves its interval at runtime (runtime configuration), with these rules:
- Retune never turns scaling off: a point with both modes off is dropped, and a written
disabledpreset is rejected. A retune-only Planner (both flagsfalse) rejectsretune.planneraltogether. - With both modes on, the load interval must be shorter than the throughput interval, and both intervals must be multiples of
scheduling.scale_interval_seconds, which is fixed at startup. A written preset entry that breaks either rule is rejected; a point of{preset: default}or{preset: false}that breaks one is dropped. A null interval in a preset is the running value, except for an enabled mode. - Under any
optimization_targetother thansla, the Planner always scales on load only:{preset: default}keeps the twoload_*presets, and a written entry or knob that enables throughput scaling is rejected. - Fields that a point’s own modes make inactive take the running value, so equivalent points are replayed once:
load_scaling_down_sensitivitywithout load scaling, and the throughput interval,max_throughput_scaling_replicasand the predictor without throughput scaling. The load interval stays active either way: it is also the cadence at which live FPM samples refit the performance model. - The predictor pre-sweep runs per throughput interval, and a point carries the winner of its own interval, or the running predictor where the pre-sweep picks none.
history_secondsmust cover the largest throughput interval of any point. - Applying a point that changes the throughput interval or turns throughput scaling on swaps in a new predictor, warmed on the sweep’s history bucketed at the new interval. Other scaling-policy changes keep the running predictor.
- Retune never turns scaling off: a point with both modes off is dropped, and a written
- Load sensitivity. Retune searches only
load_scaling_down_sensitivity;load_min_observationskeeps its running value. A preset name contributes its sensitivity alone, so with a runningload_min_observationsother than 5 thedefaultpreset is applied only in part. A preset mapping must setload_min_observationsto the running value. - Load predictor. The predictor is never a grid dimension: a pre-sweep picks one candidate per sweep. Candidates that give the same predictor are evaluated once (
{preset: false}with the default knob domains is 256 combinations and 40 predictors). Records label a candidate with its preset name, orcustom_<i>for thei-th entry. The number of candidates must not exceedmax_candidates. - Planner defaults.
planner: defaultsearchesload_scaling_down_sensitivity∈{70, 80, 90}with load scaling and the 11 predictor presets withoptimization_target: slaand throughput scaling. It never searches the scaling policy. Without scaling it expands to nothing.
An explicit domain for a disaggregated SLA Planner with load and throughput scaling. The router grid is 3 × 1 × 3 × 2 = 18 points and the Planner grid 2 scaling policies × 3 sensitivities = 6 points, so the sweep replays 108 grid points, after a pre-sweep over three predictors at each of the two throughput intervals:
Additional types
Nested object types referenced above, broken out to keep the field lists shallow.
PickedParallelConfig
A parallelism pick emitted by AIConfigurator. Referenced by AICInterpolationSpec.
Tensor-parallel size.
Pipeline-parallel size.
Data-parallel (attention-DP) size.
Mixture-of-Experts tensor-parallel size.
Mixture-of-Experts expert-parallel size.
AICInterpolationSpec
Everything the Planner needs to reproduce a rapid-mode AIC sweep. Written by the profiler onto the Planner ConfigMap; not normally hand-authored.
HuggingFace model id.
AIC system identifier.
Input sequence length for the sweep. Must be greater than 0.
Output sequence length for the sweep. Must be greater than 0.
Maximum context length swept. Must be greater than 0.
Number of prefill interpolation points. Must be greater than 0.
Number of decode interpolation points. Must be greater than 0.
Prefill parallelism pick. See PickedParallelConfig.
Decode parallelism pick. See PickedParallelConfig.
ExternalPluginEntry
One static external-plugin registration under scheduling.external_plugins. Rejects unknown fields.
Unique identifier. Must not collide with a builtin plugin id.
Pipeline stage the plugin participates in.
Allowed values: predict propose reconcile constrainStage priority. Smaller number is more authoritative.
Wire endpoint. Must start with grpc://host:port; inproc:// is rejected here.
Bearer token validated by the registry. Populate from a mounted Secret.
Plugin protocol version. Must fall within the registry’s supported range.
Plugin’s own version string, surfaced in ListPlugins.
0.0 runs every tick; a positive value throttles to every N seconds. Must be 0 or greater.
Behavior when throttled. HOLD_LAST reuses the cached result; ACCEPT_WHEN_IDLE treats the plugin as no-opinion when not due. Accepts the name (case-insensitive) or its integer value.
Capability list consumed by the type-aware merge.
Dot-paths into the pipeline context (for example predictions, observations.traffic) that must be set for the plugin to fire on a tick. Empty means no gating.
Aggregation window for windowed observation types in needs. 0.0 uses scale_interval freshness; a positive value aggregates over the last N seconds and must be an integer multiple of scale_interval_seconds.
InProcessPluginSpec
One in-process plugin under plugin_registration.in_process_plugins. Rejects unknown fields.
Python module path containing the plugin class.
Plugin class name within module.
Unique plugin identifier.
Stage priority. Smaller number is more authoritative.
0.0 runs every tick; a positive value throttles to every N seconds.
Behavior when throttled.
Allowed values: ACCEPT_WHEN_IDLE HOLD_LASTCapability list consumed by the type-aware merge.
Dot-paths into the pipeline context that must be set for the plugin to fire on a tick.
Aggregation window for windowed observation types in needs. Must be 0.0 or a positive integer multiple of scale_interval_seconds.
Keyword arguments passed to the plugin class constructor.
Environment variables
The Planner reads these environment variables. The Prometheus and namespace variables set the defaults for the config fields noted above; the rest control runtime behavior for specific environments.
Validation rules
The Planner enforces these cross-field rules at startup and rejects a config that violates any of them:
ttft_msmust be greater than 0.report_interval_hoursmust be a positive finite number ornull.fpm_sample_bucket_sizemust be a perfect square.global_planner_namespaceis required whenenvironmentisglobal-planner.- Under
optimization_target: load, prefill modes requireprefill_scale_up_queue_tokens>prefill_scale_down_queue_tokens, and decode modes requiredecode_scale_up_kv_rate>decode_scale_down_kv_rate. - At least one scaling mode must be enabled. Under any
optimization_targetother thansla, the Planner forces load scaling on and throughput scaling off and ignoresttft_ms/itl_ms. The only exception is retune-only mode:retuneset with bothenable_load_scalingandenable_throughput_scalingexplicitlyfalse(see Retune). - When
ais_perf_modelis set, itsrolesmust cover the deployment:prefillanddecodefordisagg, the matching role forprefill/decode, oraggregatedforagg. Role keys must agree withworker_type. Unknown canonical fields and the retiredaic_perf_modelinput are rejected. scheduling.scale_interval_secondsmust evenly divideload_adjustment_interval_secondsand, when throughput scaling is enabled,throughput_adjustment_interval_seconds.- When both scaling modes are enabled,
load_adjustment_interval_secondsmust be shorter thanthroughput_adjustment_interval_seconds. retunehas its own cross-field rules, listed with its fields in Retune. A live Planner rejects aretunewith executorlocal; executordgdradds the configuration rules of live retune.