Planner Configuration (PlannerConfig) | NVIDIA Dynamo Documentation

PlannerConfig is the configuration object for the Dynamo Planner. The Planner service parses it with Pydantic, so every field, type, default, and constraint on this page comes from the PlannerConfig model. For the deployment workflow and scaling-mode concepts, see the Planner Guide; for the autoscaler overview, see the Planner overview.

How the config is loaded

The Planner service loads PlannerConfig from a single required --config argument — a path to a .json, .yaml, or .yml file, or an inline JSON string. The service auto-detects which, loads it, and validates it against the PlannerConfig model. A value that is neither a readable file nor valid JSON fails at startup. How you supply it depends on where you run the Planner.

Kubernetes

Local

Set the object under spec.features.planner in a DynamoGraphDeploymentRequest (DGDR). DGDR passes it through without field-level validation, mounts it as a planner-config-* ConfigMap, and launches the Planner with --config pointing at the mounted file. The Planner validates it at startup. Every field below can be set here.

1 spec:
2   features:
3     planner:
4       mode: disagg
5       backend: vllm
6       # optimization_target defaults to "throughput" — works with no further config

Several Prometheus fields default from environment variables and are excluded when the config is serialized back out. Set them either in the config object or through the environment variable noted on each field. See Environment variables.

Core settings

mode

stringDefaults to disagg

Planner operating mode, matching the deployment topology.

Allowed values: disagg prefill decode agg

backend

stringDefaults to vllm

Inference backend the Planner scales.

Allowed values: vllm sglang trtllm mocker

environment

stringDefaults to kubernetes

Runtime environment that determines how the Planner applies scaling actions.

Allowed values: kubernetes virtual global-planner

namespace

stringDefaults to env DYN_NAMESPACE, else dynamo

Dynamo namespace of the deployment the Planner manages. Defaults from DYN_NAMESPACE, which the operator injects as {k8s_namespace}-{dgd_name}. Excluded from serialized output.

model_name

stringDefaults to null

Optional model name override. Auto-detected from the deployment when unset.

global_planner_namespace

stringDefaults to null

Namespace where the GlobalPlanner runs. Required when environment is global-planner. See the Global Planner Guide.

log_dir

stringDefaults to null

Optional directory for Planner log output.

Optimization target

optimization_target

stringDefaults to throughput

Scaling strategy. throughput and latency use static thresholds on queue depth and KV cache utilization — no SLA targets or profiling required. load uses user-defined prefill queue-token and decode KV-utilization thresholds. sla uses the Rust engine performance model to target specific ttft_ms / itl_ms values.

Allowed values: throughput latency load sla

When optimization_target is throughput, latency, or load, the Planner forces load-based scaling on and throughput-based scaling off, and it ignores ttft_ms / itl_ms. The two scaling-mode flags below apply only when optimization_target is sla.

enable_throughput_scaling

booleanDefaults to true

Enable predictive, traffic-based scaling. Only honored when optimization_target is sla.

enable_load_scaling

booleanDefaults to false

Enable reactive, load-based scaling. Only honored when optimization_target is sla. At least one scaling mode must be enabled.

ttft_ms

numberDefaults to 500.0

Time To First Token SLA target, in milliseconds. Also accepts the alias ttft. Must be greater than 0. Used only under optimization_target: sla.

itl_ms

numberDefaults to 50.0

Inter-Token Latency SLA target, in milliseconds. Also accepts the alias itl. Used only under optimization_target: sla.

Performance model and pre-deployment sweeping

pre_deployment_sweeping_mode

stringDefaults to rapid

How to generate optional bootstrap performance data. none skips bootstrap data; rapid uses AIConfigurator to simulate engine performance (~30s); thorough measures on real GPUs (several hours).

Allowed values: none rapid thorough

profile_results_dir

stringDefaults to profiling_results

Directory holding profiler-generated performance data (npz or JSON), used to bootstrap or tune the performance model when the get_perf_metrics endpoint is unavailable.

aic_perf_model

AICPerfModelSpecDefaults to null

Native AIConfigurator forward-pass model identity for the Rust engine performance shim, enabling real-time AIC estimates with online correction. Unsupported native configs fall back to FPM regression. Does not trigger an AIC interpolation sweep.

hf_id

stringRequired

HuggingFace model id, for example Qwen/Qwen3-32B.

system

stringRequired

AIC system identifier, for example h200_sxm.

backend

stringRequired

Allowed values: trtllm vllm sglang

backend_version

stringDefaults to null

Optional backend version string.

prefill_pick

PickedParallelConfigDefaults to null

Parallelism pick for the prefill engine. Required for mode disagg or prefill. See PickedParallelConfig.

decode_pick

PickedParallelConfigDefaults to null

Parallelism pick for the decode engine. Required for mode disagg, decode, or agg. See PickedParallelConfig.

model_arch

stringDefaults to null

Optional model architecture override.

weight_dtype

stringDefaults to null

Optional weight dtype.

moe_dtype

stringDefaults to null

Optional Mixture-of-Experts dtype.

activation_dtype

stringDefaults to null

Optional activation dtype.

kv_cache_dtype

stringDefaults to null

Optional KV cache dtype.

aic_interpolation

AICInterpolationSpecDefaults to null

AIConfigurator interpolation spec. Populated by the profiler in rapid mode and written onto the Planner ConfigMap; you do not normally set this by hand. When present, the Planner runs the AIC sweep in-process at bootstrap to seed the performance model. See AICInterpolationSpec.

GPU budget

max_gpu_budget

integerDefaults to 8

Maximum total GPUs the Planner may allocate across worker types.

min_gpu_budget

integerDefaults to -1

Per-DGD GPU floor enforced by the local Planner. -1 disables it. When set with max_gpu_budget such that min == max, the Planner pins the per-DGD total and only redistributes replicas between prefill and decode.

min_endpoint

integerDefaults to 1

Minimum number of engine endpoints (replicas) to maintain per worker type.

decode_engine_num_gpu

integerDefaults to null

GPUs per decode engine replica. Auto-detected from the deployment when unset.

prefill_engine_num_gpu

integerDefaults to null

GPUs per prefill engine replica. Auto-detected from the deployment when unset.

Throughput-based scaling

throughput_adjustment_interval_seconds

integerDefaults to 180

Seconds between throughput-based scaling decisions. Also accepts the alias throughput_adjustment_interval.

throughput_metrics_source

stringDefaults to frontend

Prometheus traffic source. frontend reads dynamo_frontend_* metrics from the public Frontend; router reads dynamo_component_router_* from a LocalRouter (use for a pool-local Planner in GlobalPlanner deployments).

Allowed values: frontend router

Load-based scaling

load_adjustment_interval_seconds

integerDefaults to 5

Seconds between FPM tuning updates and load-based scaling decisions. Even when only throughput scaling is enabled, live FPM observations feed the performance model at this interval. Also accepts the alias load_adjustment_interval. Must be greater than 0 and, when both scaling modes are on, shorter than throughput_adjustment_interval_seconds.

max_num_fpm_samples

integerDefaults to 64

Maximum retained ForwardPassMetrics observations for online tuning and regression fallback.

fpm_sample_bucket_size

integerDefaults to 16

Number of buckets for observation retirement. Must be a perfect square.

load_scaling_down_sensitivity

integerDefaults to 80

Scale-down sensitivity from 0 to 100 (0 = never scale down, 100 = aggressive).

load_min_observations

integerDefaults to 5

Minimum observations before load-based scaling decisions begin (cold-start threshold).

prefill_scale_up_queue_tokens

integerDefaults to null

Prefill queue-token count that triggers scale-up. Required (with the scale-down field) when optimization_target is load and mode includes prefill. Must be greater than 0 and greater than prefill_scale_down_queue_tokens.

prefill_scale_down_queue_tokens

integerDefaults to null

Prefill queue-token count that allows scale-down. Required alongside prefill_scale_up_queue_tokens.

decode_scale_up_kv_rate

numberDefaults to null

Decode KV-utilization percentage (0–100) that triggers scale-up. Required (with the scale-down field) when optimization_target is load and mode includes decode. Must be greater than decode_scale_down_kv_rate.

decode_scale_down_kv_rate

numberDefaults to null

Decode KV-utilization percentage (0–100) that allows scale-down. Required alongside decode_scale_up_kv_rate.

speculative_nextn

integerDefaults to 0

Manual fallback speculative-decoding depth. A worker’s published runtime_config.runtime_data.spec_decode.nextn takes precedence when present. Must be 0 or greater.

Load prediction

load_predictor

stringDefaults to arima

Prediction method for request count, ISL, and OSL.

Allowed values: constant arima kalman prophet

load_predictor_log1p

booleanDefaults to false

Apply a log1p transform to predicted request count, ISL, and OSL.

prophet_window_size

integerDefaults to 50

Window size (seconds) for the Prophet predictor.

load_predictor_warmup_trace

stringDefaults to null

Path to a warmup trace file for bootstrapping predictions.

kalman_q_level

numberDefaults to 1.0

Kalman process noise for the level component.

kalman_q_trend

numberDefaults to 0.1

Kalman process noise for the trend component.

kalman_r

numberDefaults to 10.0

Kalman measurement noise.

kalman_min_points

integerDefaults to 5

Minimum data points before Kalman predictions activate.

Prometheus metrics

These fields default from environment variables and are excluded from serialized output. Set them in the config object or through the noted environment variable.

metric_pulling_prometheus_endpoint

stringDefaults to env PROMETHEUS_ENDPOINT

Prometheus endpoint the Planner queries for traffic metrics. Defaults to PROMETHEUS_ENDPOINT, else http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090.

metric_pulling_prometheus_token

stringDefaults to env PROMETHEUS_TOKEN

Optional bearer token sent as Authorization: Bearer <token> on every PromQL request. Read once at startup.

metric_pulling_prometheus_token_file

stringDefaults to env PROMETHEUS_TOKEN_FILE

Path to a file containing a bearer token, re-read before every request so rotated tokens are picked up without a restart.

metric_pulling_prometheus_ssl_verify

booleanDefaults to false

Verify the upstream Prometheus TLS certificate. Defaults from PROMETHEUS_SSL_VERIFY (1/true/yes enable it). Pair with a CA bundle for a private CA.

metric_pulling_prometheus_extra_query_params

objectDefaults to env PROMETHEUS_EXTRA_QUERY_PARAMS

Fixed key/value pairs appended as URL query parameters on every PromQL request. Set PROMETHEUS_EXTRA_QUERY_PARAMS as a URL query string, for example namespace=my-ns&tenant=foo.

metric_pulling_prometheus_ca_bundle

stringDefaults to env PROMETHEUS_CA_BUNDLE

Path to a CA bundle for verifying the upstream Prometheus TLS certificate. No-op unless SSL verification is enabled. Must point to an existing file when set.

metric_reporting_prometheus_port

integerDefaults to 0

Port on which the Planner exposes its own dynamo_planner_* metrics. Defaults from PLANNER_PROMETHEUS_PORT. 0 disables.

Advisory mode

advisory

booleanDefaults to false

Suggestion-only mode. The Planner computes, logs, exports, and reports recommended replica counts without executing scaling actions or changing the deployment. Use it to evaluate a new configuration or validate SLA targets against production traffic.

Diagnostics and reporting

report_interval_hours

numberDefaults to 24.0

Generate an HTML diagnostics report every N hours (simulated time). Set to null to disable. Must be a positive finite number or null.

report_output_dir

stringDefaults to ./planner_reports

Directory for HTML diagnostics reports.

report_filename

stringDefaults to null

Fixed report filename written under report_output_dir. When unset, a timestamped name is used.

report_write_gzip_log

booleanDefaults to true

Write a compressed JSONL diagnostics log (.log.jsonl.gz) next to each HTML report.

live_dashboard_port

integerDefaults to 8080

Port for the live diagnostics dashboard HTTP server. 0 disables. When enabled, visit http://host:port/ for a real-time Plotly report.

Scheduling and plugin pipeline

The Planner runs the builtin plugin pipeline by default. scheduling controls pipeline cadence; plugin_registration controls how plugins register, authenticate, and communicate. For the pipeline model, see the Planner Guide.

scheduling

SchedulingConfig

Plugin-pipeline scheduling. Rejects unknown fields.

scale_interval_seconds

numberDefaults to gcd of enabled builtin intervals

Base pipeline cadence. The pipeline wakes once per interval; each plugin’s execution_interval_seconds decides whether it fires. When unset, computed as the gcd of load_adjustment_interval_seconds and, when throughput scaling is enabled, throughput_adjustment_interval_seconds. Must be greater than 0 and evenly divide both adjustment intervals.

tick_max_duration_seconds

numberDefaults to 30.0

Outer deadline wrapping the full 4-stage pipeline. Exceeding it aborts the tick; the next tick runs from a clean state. Must be greater than 0.

external_plugins

[]ExternalPluginEntryDefaults to []

Static external-plugin registrations applied at startup. Per-entry failures are logged but do not crash the Planner. See ExternalPluginEntry.

gateway

GatewayConfig

gRPC registration gateway for self-registering plugins. Disabled by default.

enabled

booleanDefaults to false

Open the gRPC gateway at listen. Not needed for static external_plugins.

listen

stringDefaults to unix:///var/run/dynamo/planner/registry.sock

Bind address. A unix: socket path for in-Pod registration, or host:port for TCP.

allow_insecure

booleanDefaults to false

Permit binding a plaintext (no-TLS) gateway on a TCP listen. Fails closed by default because the gateway receives plugins’ shared-secret tokens. unix: listens are always allowed.

plugin_registration

PluginRegistrationConfig

Plugin registry configuration — auth, transport, heartbeat, and in-process plugins. Rejects unknown fields.

auth

AuthConfig

Registry authentication. An empty trusted_sources fails closed (every token rejected).

trusted_sources

[]stringDefaults to []

Accepted auth sources. Use ['allow_unauthenticated'] for development, or ['static_secret'] with static_secrets for production.

Allowed values: static_secret allow_unauthenticated

static_secrets

objectDefaults to {}

Map of secret_value to subject_label. Populate from a mounted Kubernetes Secret rather than hard-coding in the ConfigMap.

transport

TransportConfig

Outbound gRPC transport settings for calling plugins.

allow_insecure_grpc

booleanDefaults to false

Allow plaintext grpc:// plugin endpoints. Required to use any grpc:// endpoint; mTLS support is not yet shipped.

request_timeout_seconds

numberDefaults to 5.0

Per-RPC timeout applied to every plugin call. Must be greater than 0.

keepalive_time_ms

integerDefaults to 30000

gRPC keepalive time, in milliseconds.

max_message_size_bytes

integerDefaults to 10485760

Maximum gRPC message size (10 MB).

protocol_version_min

stringDefaults to 1.0

Minimum accepted plugin protocol version.

protocol_version_max

stringDefaults to 1.0

Maximum accepted plugin protocol version.

heartbeat_timeout_seconds

numberDefaults to 15.0

Seconds without a heartbeat before a plugin is considered missed.

heartbeat_missed_threshold

integerDefaults to 2

Consecutive missed heartbeats before a plugin is dropped.

in_process_plugins

[]InProcessPluginSpecDefaults to []

In-process plugins loaded from Python modules at startup. See InProcessPluginSpec.

admin

AdminAuthConfig

Admin (ListPlugins) RBAC config. Parsed but currently inert — the gateway default-denies ListPlugins regardless of mode.

mode

stringDefaults to allow_all

Allowed values: allow_all k8s_rbac

Additional types

Nested object types referenced above, broken out to keep the field lists shallow.

PickedParallelConfig

A parallelism pick emitted by AIConfigurator. Referenced by aic_perf_model.prefill_pick / decode_pick and by AICInterpolationSpec.

integerDefaults to 1

Tensor-parallel size.

integerDefaults to 1

Pipeline-parallel size.

integerDefaults to 1

Data-parallel (attention-DP) size.

moe_tp

integerDefaults to 1

Mixture-of-Experts tensor-parallel size.

moe_ep

integerDefaults to 1

Mixture-of-Experts expert-parallel size.

AICInterpolationSpec

Everything the Planner needs to reproduce a rapid-mode AIC sweep. Written by the profiler onto the Planner ConfigMap; not normally hand-authored.

hf_id

stringRequired

HuggingFace model id.

system

stringRequired

AIC system identifier.

backend

stringRequired

Allowed values: trtllm vllm sglang

isl

integerRequired

Input sequence length for the sweep. Must be greater than 0.

osl

integerRequired

Output sequence length for the sweep. Must be greater than 0.

sweep_max_context_length

integerRequired

Maximum context length swept. Must be greater than 0.

prefill_interpolation_granularity

integerRequired

Number of prefill interpolation points. Must be greater than 0.

decode_interpolation_granularity

integerRequired

Number of decode interpolation points. Must be greater than 0.

prefill_pick

PickedParallelConfigRequired

Prefill parallelism pick. See PickedParallelConfig.

decode_pick

PickedParallelConfigRequired

Decode parallelism pick. See PickedParallelConfig.

ExternalPluginEntry

One static external-plugin registration under scheduling.external_plugins. Rejects unknown fields.

plugin_id

stringRequired

Unique identifier. Must not collide with a builtin plugin id.

plugin_type

stringRequired

Pipeline stage the plugin participates in.

Allowed values: predict propose reconcile constrain

priority

integerRequired

Stage priority. Smaller number is more authoritative.

endpoint

stringRequired

Wire endpoint. Must start with grpc://host:port; inproc:// is rejected here.

auth_token

string

Bearer token validated by the registry. Populate from a mounted Secret.

protocol_version

stringDefaults to 1.0

Plugin protocol version. Must fall within the registry’s supported range.

version

stringDefaults to v1

Plugin’s own version string, surfaced in ListPlugins.

execution_interval_seconds

numberDefaults to 0.0

0.0 runs every tick; a positive value throttles to every N seconds. Must be 0 or greater.

hold_policy

stringDefaults to HOLD_LAST

Behavior when throttled. HOLD_LAST reuses the cached result; ACCEPT_WHEN_IDLE treats the plugin as no-opinion when not due. Accepts the name (case-insensitive) or its integer value.

Allowed values: HOLD_LAST ACCEPT_WHEN_IDLE

needs

[]stringDefaults to []

Capability list consumed by the type-aware merge.

requires_produced_fields

[]stringDefaults to []

Dot-paths into the pipeline context (for example predictions, observations.traffic) that must be set for the plugin to fire on a tick. Empty means no gating.

observation_window_seconds

numberDefaults to 0.0

Aggregation window for windowed observation types in needs. 0.0 uses scale_interval freshness; a positive value aggregates over the last N seconds. Enforced to be greater than or equal to scale_interval_seconds.

InProcessPluginSpec

One in-process plugin under plugin_registration.in_process_plugins. Rejects unknown fields.

module

stringRequired

Python module path containing the plugin class.

class

stringRequired

Plugin class name within module.

plugin_id

stringRequired

Unique plugin identifier.

plugin_type

stringRequired

Allowed values: predict propose reconcile constrain

priority

integerRequired

Stage priority. Smaller number is more authoritative.

execution_interval_seconds

numberDefaults to 0.0

0.0 runs every tick; a positive value throttles to every N seconds.

hold_policy

stringDefaults to ACCEPT_WHEN_IDLE

Behavior when throttled.

Allowed values: ACCEPT_WHEN_IDLE HOLD_LAST

needs

[]stringDefaults to []

Capability list consumed by the type-aware merge.

requires_produced_fields

[]stringDefaults to []

Dot-paths into the pipeline context that must be set for the plugin to fire on a tick.

observation_window_seconds

numberDefaults to 0.0

Aggregation window for windowed observation types in needs. Must be 0 or greater.

kwargs

objectDefaults to {}

Keyword arguments passed to the plugin class constructor.

Environment variables

The Planner reads these environment variables. The Prometheus and namespace variables set the defaults for the config fields noted above; the rest control runtime behavior for specific environments.

Variable	Effect	Default
`DYN_NAMESPACE`	Default for `namespace`. Operator injects `{k8s_namespace}-{dgd_name}`.	`dynamo`
`PROMETHEUS_ENDPOINT`	Default for `metric_pulling_prometheus_endpoint`.	in-cluster URL
`PROMETHEUS_TOKEN`	Default for `metric_pulling_prometheus_token`.	unset
`PROMETHEUS_TOKEN_FILE`	Default for `metric_pulling_prometheus_token_file`.	unset
`PROMETHEUS_SSL_VERIFY`	Default for `metric_pulling_prometheus_ssl_verify` (`1`/`true`/`yes`).	`false`
`PROMETHEUS_EXTRA_QUERY_PARAMS`	Default for `metric_pulling_prometheus_extra_query_params` (URL query string).	unset
`PROMETHEUS_CA_BUNDLE`	Default for `metric_pulling_prometheus_ca_bundle`.	unset
`PLANNER_PROMETHEUS_PORT`	Default for `metric_reporting_prometheus_port`.	`0`
`DYN_PARENT_DGD_K8S_NAME`	Parent DGD name for the GlobalPlanner connector (`environment: global-planner`).	required
`POD_NAMESPACE`	Kubernetes namespace for the GlobalPlanner connector.	required
`SCALING_CHECK_INTERVAL`	Poll interval (seconds) for the virtual environment connector (`environment: virtual`).	`10`
`SCALING_MAX_WAIT_TIME`	Maximum wait (seconds) for a scaling action in the virtual environment.	`1800`
`DYNAMO_PLANNER_TEST`	Test-only. Must be `1` to allow the virtual clock. Not for production.	unset

Validation rules

The Planner enforces these cross-field rules at startup and rejects a config that violates any of them:

ttft_ms must be greater than 0.
report_interval_hours must be a positive finite number or null.
fpm_sample_bucket_size must be a perfect square.
global_planner_namespace is required when environment is global-planner.
Under optimization_target: load, prefill modes require prefill_scale_up_queue_tokens > prefill_scale_down_queue_tokens, and decode modes require decode_scale_up_kv_rate > decode_scale_down_kv_rate.
At least one scaling mode must be enabled. Under any optimization_target other than sla, the Planner forces load scaling on and throughput scaling off and ignores ttft_ms / itl_ms.
When aic_perf_model is set, prefill_pick is required for mode disagg/prefill and decode_pick for mode disagg/decode/agg.
scheduling.scale_interval_seconds must evenly divide load_adjustment_interval_seconds and, when throughput scaling is enabled, throughput_adjustment_interval_seconds.
When both scaling modes are enabled, load_adjustment_interval_seconds must be shorter than throughput_adjustment_interval_seconds.

Planner Guide

Deployment workflow, scaling modes, and profiler integration.

Planner Overview

Why LLM inference needs a different autoscaler.

DGDR Reference

The custom resource whose spec.features.planner carries this config.

Environment Variables

Observability environment variables shared across Dynamo processes.

How the config is loaded

Kubernetes

Local

Core settings

Optimization target

Performance model and pre-deployment sweeping

GPU budget

Throughput-based scaling

Load-based scaling

Load prediction

Prometheus metrics

Advisory mode

Diagnostics and reporting

Scheduling and plugin pipeline

Additional types

PickedParallelConfig

AICInterpolationSpec

ExternalPluginEntry

InProcessPluginSpec

Environment variables

Validation rules

Related pages