Simulate Planner Decisions with AISimulate
aisimulate predict --stack dynamo replays a request trace through simulated engines with the
Dynamo Planner in the loop. Use it to compare aggregated and disaggregated topologies, service-level
objective (SLO) targets, scaling settings, worker startup delays, and
retune without a cluster. The dynamo.replay Python API runs the
same simulation from a script.
The production Planner scales Kubernetes or Global Planner deployments; it does not autoscale a
local deployment. aisimulate predict --stack dynamo runs locally but evaluates Planner decisions
for the deployment described by its YAML input.
For the general simulation workflow, see
Run a Simulation. For every
aisimulate predict option and configuration section, see the
replay CLI reference. For how
replay feeds the Planner its observations, see the
Planner simulation adapter.
Prerequisites
- Install Dynamo, AISimulate, and the Planner dependencies from one checkout, as described in Install. AISimulate supports Python 3.11 through 3.13. That procedure builds the bindings in release mode, which matters because simulation runs are CPU-bound.
- AISimulate reads each model’s Hugging Face configuration on first use, for
engine.context_length: maxand the default timing model. The host needs Hugging Face Hub access or the configuration in its local Hugging Face cache.
Configure the Planner Section
Add a planner section to the prediction config. This section runs the Planner with its
production defaults:
policy defaults to disabled, which replays the deployment with fixed replicas and writes
planner: null to the report. With enabled, the Dynamo Planner adapter builds a PlannerConfig
from the section and validates it as a live Planner would. Omitted keys take the defaults listed in
the Planner Configuration reference.
The section uses AISimulate key names. These differ from the PlannerConfig field names:
The other accepted keys keep their PlannerConfig names: enable_throughput_scaling,
enable_load_scaling, throughput_adjustment_interval_seconds,
load_adjustment_interval_seconds, max_throughput_scaling_replicas, max_num_fpm_samples,
fpm_sample_bucket_size, load_scaling_down_sensitivity, load_min_observations,
load_predictor, load_predictor_log1p, prophet_window_size, kalman_q_level,
kalman_q_trend, kalman_r, kalman_min_points, load_predictor_warmup_trace,
advisory, and retune. Any other
key fails with Extra inputs are not permitted. load_predictor_warmup_trace names a file, a
directory of JSONL or JSONL.GZ shards, or a glob; a relative path resolves against the working
directory.
The adapter sets the remaining fields from the rest of the config:
modefollowsengine.mode:aggordisagg.prefill_engine_num_gpuanddecode_engine_num_gpuaretensor×attention_dataof each role’sparallelism.- With
target: sla,ttft_msanditl_mscome fromevaluation.sla. - With
target: load, the scaling thresholds are fixed:decode_scale_up_kv_rate: 90anddecode_scale_down_kv_rate: 70, plusprefill_scale_up_queue_tokens: 1andprefill_scale_down_queue_tokens: 0for a disaggregated engine. The section cannot change them. - The Planner writes no HTML diagnostics report, starts no live dashboard, and connects to no Prometheus server.
Set max_num_gpus above the starting GPU count: when the starting replicas already use the whole
budget, the Planner holds every scale-up it proposes.
A planner section requires --stack dynamo; the default engine stack rejects it as an unknown
section. aisimulate predict --stack dynamo runs offline only, so --online is rejected.
Targets and Scaling Modes
throughput(the default),latency, andloadrun load-based scaling only and ignoreenable_throughput_scalingandenable_load_scaling, as a live Planner does. Retune-only mode is the one exception; see Simulate Retune.target: slauses the two flags: throughput scaling is on and load scaling is off unless you set them. Throughput scaling requiresevaluation.sla.ttft_msandevaluation.sla.itl_ms; an SLA with onlye2e_msis rejected.- Omitted and
nulladjustment intervals follow the Engine and Adapter Rules; explicit intervals follow thePlannerConfigconstraints in the Planner Configuration reference.
Planner Performance Model
Throughput scaling with target: sla needs a performance model of the engines. Predict
bootstraps it from AISimulate forward-pass estimates for engine.model, engine.hardware, and
engine.backend, and records the result in planner.metadata.bootstrap: status: installed, the
number of prefill and decode samples (prefill_fpm_count, decode_fpm_count), and an
fpm_sha256 digest. Other targets do not need the model and record status: not_required.
The bootstrap uses AISimulate estimates even when engine.workers.<role>.timing.type is
polynomial or fixed. With those timing models, the Planner’s model and the simulated engines
disagree until the Planner has tuned its model on the forward passes it observes. Keep
timing.type: default when the Planner’s throughput decisions matter.
Compare Topologies and Startup Delays
The steps below replay one hour of the FAST’25 tool-agent trace with a Planner that targets a 1,500 ms TTFT and a 50 ms ITL and may use up to four GPUs.
Run the aggregated baseline
Download the trace:
Save this configuration as planner-aggregated.yaml. It starts two aggregated workers and lets the
Planner scale to four:
Run the prediction:
The command prints a summary table and writes planner-reports/aggregated/prediction.json. The run
replays 23,608 requests in about 20 seconds. The table’s last line,
Timing (runner): unavailable, is expected with --stack dynamo.
Run the disaggregated comparison
Copy the baseline, then replace its engine mapping with separate prefill and decode roles:
Also replace the planner mapping with role-specific minimums:
Save the result as planner-disaggregated.yaml, then run it:
Compare GPU time, goodput, latency, and the number of scaling events of the two runs:
Sweep aggregated startup time
startup_seconds delays only the workers the Planner adds; the initial replicas are ready at time
zero. Run the aggregated prediction with a sequence of overrides. Each run takes about 20 seconds:
With startup_seconds: 0, an added worker appears in planner.lifecycle_operations directly as a
worker_ready transition. With a positive delay, it first appears as worker_starting and becomes
worker_ready startup_seconds later. Count added workers from planner.scaling_events, which
does not depend on the delay, and compare GPU time and TTFT as the delay grows. The starting
column counts the added workers that went through a startup delay:
Any traffic that aisimulate predict accepts works with a Planner, including synthetic traffic and
the other trace formats in Traffic Rules.
Retune requires a Dynamo request trace.
Read the Results
prediction.json has these top-level keys:
summary.gpu_hours is the provisioned GPU time: (prefill_worker_seconds ×
prefill_gpus_per_worker + decode_worker_seconds × decode_gpus_per_worker) / 3600, with GPUs
per worker taken from each role’s parallelism.
The planner object holds:
Each ticks entry has ordinal, at_ms, scheduled_tick (which scaling loops ran),
input (traffic, worker counts, and forward-pass observations), topology (active, starting, and
draining workers per role), effects (the Planner’s output, including scale_to and
diagnostics), and runtime_decision (target_prefill, target_decode, next_tick_ms, and
router_parameters). Aggregated workers appear under topology.decode and
runtime_decision.target_decode. Predict always records ticks: the one-hour aggregated example
writes about 2 MB.
For the other files in the output directory and the exit codes, see Output in the replay CLI reference.
Run from Python
dynamo.replay.run_trace_replay runs the same Planner-in-the-loop replay from a script and returns
a ReplayReport with summary, per_request, coverage, planner, and to_dict(). The Planner
input is a PlannerConfig, not the AISimulate planner section: use PlannerConfig field names
(optimization_target, max_gpu_budget, min_endpoint, ttft_ms, itl_ms), and set mode to
agg with extra_engine_args or disagg with prefill_engine_args and decode_engine_args.
This script mirrors the aggregated baseline:
planner_configaccepts a mapping, an inline JSON string, or the path of a JSON or YAML file.- Planner replay is offline only. Trace formats other than
dynamotake exactly one file. sla_ttft_ms,sla_itl_ms, andsla_e2e_msset the SLA for the report’s goodput metrics. The Planner’s own targets arettft_msanditl_msinplanner_config.- Engine arguments use the
canonical engine mapping.
An AISimulate
timing_modelwithworker_typegives the same performance-model bootstrap as predict. Engine arguments without one, such as the defaultpolynomialtiming, leave the bootstrap atnot_configured_load_only, and the log warns that throughput-based scaling falls back to load-based scaling. The engine argumentstartup_time, in seconds, delays added workers likestartup_seconds. - With the
PlannerConfigdefaults, the replay writes an HTML diagnostics report to./planner_reports/and returns its path inreport.planner.html_report_path; predict never writes one. Setreport_interval_hours: Noneandlive_dashboard_port: 0to skip it. capture_planner_details=Falsedropsticks,lifecycle_operations, and the HTML report, and keepsscaling_events,total_ticks,metadata, andretune_events.report.planner.scaling_eventsholdsScalingEventobjects;report.to_dict()returns the same JSON asprediction.json.
Replay in Advisory Mode
Set advisory: true to replay the Planner as it runs in advisory mode
in production: it makes every decision and changes nothing.
In Python, set "advisory": True in planner_config. A Planner configuration copied from a
deployment that runs in advisory mode replays the same way; set advisory: false, or pass
--set planner.advisory=false to predict, to simulate the scaling it would do. If that
configuration enables batch_scheduling, also set batch_scheduling.enabled: false: replay does not
run batch scheduling, and non-advisory batch scheduling requires Redis settings. An advisory replay
logs an [ADVISORY] warning at startup. policy: disabled rejects advisory: true, and
aisimulate recommend rejects the advisory key.
What runs:
- Every Planner tick, with the same observations, load predictions, and load-based and throughput-based decisions as an active Planner.
- Retune sweeps and scoring. With load-based or throughput-based scaling, the candidate replays run an active inner Planner, so a candidate is scored on the replicas it would drive; retune-only candidates keep the replicas fixed.
What stays fixed:
- The replica counts and router parameters keep their starting values, and
lifecycle_operationsis empty. The Planner sees no scaling in progress and its starting replicas as the expected counts, as a live advisory Planner sees a deployment that never changes. summaryreports the starting deployment serving the whole trace.
What is reported:
planner.metadata.advisoryistrueandplanner.scaling_eventsis empty.planner.recommended_scaling_eventslists the recommendations with the fields ofscaling_events. An entry is added each time a component’s recommended count changes to a value other than its current count, which isfrom_count. A scaling decision that keeps a component at its current count resets that component’s recommendation, so the same recommendation after a hold is listed again. The recommendation of every tick stays inticks[].effects.scale_to.- A retune decision that would apply has outcome
advisory, with thechosenpoint and theplanner_changesit recommends, andpredictor_warmup: null. Every tick’sruntime_decision.router_parametersisnull, so the recommended router parameters appear only inretune_events[].chosen.router. - With
run_trace_replayand thePlannerConfigreport defaults, the HTML diagnostics report shows the fixed replica counts with the recommendation markers. Predict writes no HTML report.
Each recommendation is one step from the starting deployment, decided on traffic that the starting
deployment served. It is not the path an active Planner would take: after its first scale-up, an
active Planner observes the larger deployment and decides from there, and every later retune sweep
searches from the point it applied. To predict what an active Planner does, replay with
advisory: false.
Simulate Retune (Experimental)
Experimental. Retune runs only in offline simulation in this release. A live Planner rejects a
configuration that sets retune at startup.
Retune makes the Planner retune its router and Planner runtime parameters from the traffic of the
last window. For the concept and the two scenarios, see
Periodic Retune. For windows, scoring, and decision
timing, see Planner Design. For every
retune field, its default, and the search-domain syntax, see the
retune reference.
Requirements
With aisimulate predict --stack dynamo:
planner.policy: enabled. To retune with fixed replicas, also set bothenable_load_scaling: falseandenable_throughput_scaling: false(retune-only mode).traffic.sourcewithtype: traceandformat: dynamo, andtraffic.load.type: trace_timestamps;speedupis allowed. Theagentic_*load options andtraffic.stop.max_virtual_time_secondsare rejected.router.policy: kv_routerwhenretune.routeris set.aisimulate recommendrejectsretune: each recommendation candidate is a static configuration.
Recommended: set evaluation.sla with ttft_ms and itl_ms. Predict fills retune.sla_ttft_ms
and retune.sla_itl_ms from it when they are unset, so retune scores candidates against the SLA
that AISimulate reports goodput for. Without it, scoring uses the Planner’s ttft_ms and itl_ms,
which then keep their PlannerConfig defaults.
With dynamo.replay.run_trace_replay:
trace_format="dynamo". Other trace formats andrun_synthetic_trace_replayreject retune.- No
replay_concurrency,agentic_*options, ormax_sim_time_ms, and no requests withagent_context(an agentic Dynamo trace). Retune windows drop tool rows.execution_modelis accepted, because it applies only to agentic requests. router_mode="kv_router"and a router configuration that meets theretune.routerrequirements whenretune.routeris set.- With throughput scaling, retune reads
load_predictor_warmup_traceas recorded history. It must then be a Dynamo request trace that ends at or before the first replayed request, on the same absolute timestamps. Without throughput scaling, retune does not read it.
Prepare a Dynamo Request Trace
Retune replays windows of a Dynamo request trace (dynamo.request.trace.v1). Record one from a
deployment with request tracing;
the record schema describes
its rows. To try retune without a deployment, convert the Mooncake trace that
Run the aggregated baseline downloads. This script writes the first 20 minutes to traces/toolagent-history.jsonl, the predictor history in
Scenario 2, and the remaining 39 minutes to traces/toolagent-replay.jsonl:
Save it as mooncake_to_dynamo.py and run it:
A converted row has no total_time_ms, so retune cannot exclude requests that completed after a
window’s sweep launched. A recorded trace carries each request’s latency.
Scenario 1: Fixed Replicas, Router Only
Save this configuration as retune-router.yaml. Three aggregated workers start with router
temperature 1.0, and retune searches prefill_load_scale and temperature every 10 minutes of
trace time:
- The router domain is 1 × 3 × 2 = 6 grid points. The scalar
overlap_score_credit: 1.0pins that knob; an omitted knob would search its default domain. - Both scaling flags are
false, so the Planner never scales. It ticks everyload_adjustment_interval_seconds(5 seconds by default), andinterval_secondsmust be a multiple of that tick. - Window
kcovers[600k, 600(k+1))seconds after the first request, and its sweep launches 60 seconds (grace_seconds) after the window ends.
Run the prediction:
The 39-minute replay makes three decisions. Each sweep replays about 4,100 requests once per candidate, and the whole run takes about a minute.
Scenario 2: Autoscaling, Router and Planner
For a Planner that scales and is retuned, copy retune-router.yaml to retune-joint.yaml, set
replicas: 2, and replace the planner section:
planner: defaultsearchesload_scaling_down_sensitivityin{70, 80, 90}, because load scaling is on, and runs a load-predictor pre-sweep over the 11 predictor presets, becausetarget: slauses throughput scaling. With two temperatures, the grid has 6 points.- Each candidate replay runs the current Planner with the candidate’s Planner fields, starting from the replica counts at the window start.
load_predictor_warmup_tracewarms the Planner’s load predictor and, becausetarget: slascales on throughput, is also retune’s trace history.history_secondsresolves to the throughput interval (180 seconds by default) × 50, the largestprophet_window_sizeamong the presets: 9,000 seconds. The history file supplies the 20 minutes it has.
Run it:
Read Retune Decisions
planner.retune_events lists every decision whose decision tick falls within the replay. It is
empty when retune made no decision, for example in a replay shorter than one window plus
grace_seconds:
Each record has these fields. Times are Planner-clock seconds from the start of the replay:
Each tick also records the router state. runtime_decision.router_parameters is the desired
parameter map per worker type from that tick on. router_policies is the replay router’s state,
revision and parameters, before the tick’s update. A tick does not repeat the decisions; match
a decision’s decided_at_s × 1000 to the tick’s at_ms. To see the switch around the first
decision of Scenario 1:
Decision Latency
A sweep takes time, and the old parameters keep serving until the decision. In replay, virtual time
stops while the sweep runs, so decision_latency_seconds sets how much trace time passes between
launch and decision; Decision Timing describes when the
decision lands. Choose the value by what the run is for:
- The default,
0, decides one tick after launch. It gives the best case. - A fixed number of seconds gives the same decision ticks on every run. Use it to compare configurations, and set it to the sweep time you expect on the host that will run sweeps.
measuredcharges the sweep’s own wall-clock time on the replay host, as thedecision_latency_secondsfield defines it. Decision ticks then depend on the speed and load of the replay host, so two runs of the same config can switch at different ticks and report different results. Setdecision_latency_scaleabove 1 to model a slower sweep host, or below 1 for a faster one. Above 1, also lowertimeout_secondsto at mostinterval_seconds/decision_latency_scale; the default timeout is rejected.
For example, on a host where Scenario 1’s sweeps take about 11 seconds, measured moves each
decision from one tick after launch (665 seconds for window 0) to the third (675 seconds):
Measure the Gain
Run the same config without retune and compare summary.goodput_output_throughput_tok_s,
summary.gpu_hours, and the latency percentiles:
- For Scenario 1, replace the whole
plannersection with--set 'planner={policy: disabled}'. Do not only removeretune: without it, the twofalsescaling flags no longer select retune-only mode, and the Planner autoscales with its target’s defaults. - For Scenario 2, remove retune with
--set planner.retune=null.
--set planner.advisory=true keeps the sweeps and records a decision that would apply as
advisory, with the point it would choose, without applying it. Other decisions keep their
outcomes, such as kept when no candidate clears the threshold. Advisory mode also keeps the
replica counts fixed; see Replay in Advisory Mode.
Each candidate is replayed once. All candidate replays of a window, the current point’s included,
use the same router seed and request ids, so a candidate that changes no routing or scaling
decision scores exactly the current point and is kept. A candidate that changes decisions is still
judged on one sample path of one window. With the pinned AISimulate release, an inner Planner with
optimization_target: sla can still decide differently in two replays of the same candidate, so
such a candidate can miss an exact tie; see
Candidates and the Sweep.
The outer replay keeps its default, unseeded routing, so two runs of the same config can differ in
the outer replay’s results and, with autoscaling, in the replica counts a window starts from. Set
min_improvement_percentage (3 by default) to the smallest gain worth a switch, and check a
decision across windows or runs before trusting a small margin.
Run Retune from Python
run_trace_replay takes the same retune block inside a PlannerConfig mapping. Sweeps start
worker processes with the spawn method, so guard the script with if __name__ == "__main__":.
This script runs Scenario 1:
router_config is the startup router. Without evaluation.sla, the scoring SLA defaults to the
Planner’s ttft_ms and itl_ms, so the script sets retune.sla_ttft_ms and retune.sla_itl_ms.
With retune set, report.planner.retune_events is a list, empty when retune made no decision;
without retune, it is None.
For Scenario 2, pass num_workers=2 and write the Scenario 2 planner section with
PlannerConfig field names. Python does not take the SLA from evaluation.sla, so set the
Planner’s own ttft_ms and itl_ms, which target: sla scales against and retune scores with:
Keep the AISimulate timing_model with worker_type in ENGINE. Without it, the
performance-model bootstrap is not_configured_load_only, and throughput scaling falls back to
load scaling, as described in Run from Python.
Size the Run
Each window costs one sweep: one replay of the window per distinct candidate, plus one forecast task
per predictor candidate when the load predictor is searched, and one task that prepares a winning
predictor that differs from the current one. A sweep runs in max_parallel worker processes that
it starts and stops, and the replay waits for it. A run therefore takes about the replay without
retune plus the sum of sweep_wall_s over its decisions. For the size of the
candidate set, see Candidates and the Sweep. In
Scenario 2, every candidate replay also runs a Planner, including its performance-model bootstrap
for target: sla.
timeout_seconds (by default interval_seconds) bounds each sweep in wall-clock seconds. A sweep
that exceeds it is killed, the decision is timeout, and nothing changes. If sweeps time out,
narrow the domain, raise max_parallel, or lengthen interval_seconds.
aisimulate predict runs the replay in a supervised execution process and sets no wall-clock limit
on the replay itself. Sweep workers are child processes of the execution process, so they count
against its limits under execution.resources:
- Memory.
autois the smaller of 90% of the memory available at start and the available memory less the reserve. The resident memory of the whole process tree, sweep workers included, must stay below the limit, and the host’s available memory must stay above the reserve; otherwise the run stops asresource_limitedwith exit code 3. Size the limit as the outer replay plusmax_paralleltimes the peak of one window replay, measured on a short run:resource-runtime.jsonreports the tree’speak_observed_rss_bytes. - CPUs. The execution process and its sweep workers are pinned to
cpu_limitCPUs. AISimulate also limits math libraries to one thread (OMP_NUM_THREADS=1and similar), which sweep workers inherit. - Timeouts. The initialization timeout covers start-up until the replay begins, including a pass over the whole trace that sizes the memory plan, about 20 seconds per million requests. The shutdown timeout covers process exit after the replay; sweeps finish before the replay returns.
For scale, on a 24-core host, Scenario 1 above peaks at about 2 GB with max_parallel: 4, and
Scenario 2 at about 4.5 GB with max_parallel: 8.
Limitations
- Simulation only. A live Planner rejects
retuneat startup. - Open-loop Dynamo request traces only. See Requirements.
- Cold window replays. Each candidate replay starts with an empty KV cache and the replica counts at the window start, so prefix-cache reuse from before the window does not carry over.
- Recorded completion times. A window includes a request only if it completed before the sweep launched, judged by the latency recorded in the trace rather than the simulated one.
- One replay per candidate. Candidate replays share one random seed, but a score is still one sample path of one window; see Measure the Gain.
- Router cache weights are not retunable.
host_cache_hit_weightanddisk_cache_hit_weightalso set how the startup router counts cached tokens, which a runtime update does not change, and replay reports no shared cache hits forshared_cache_multiplier. Retune rejects all three and its error says why.