DynoSim Architecture | NVIDIA Dynamo Documentation

DynoSim connects a workload driver to one or more Mocker engine cores and records request and token timing for analysis. The unified aisimulate predict and aisimulate recommend commands run this simulation offline. The former public Replay online CLI is unavailable, although the Python replay SDK retains online mode. The separate python3 -m dynamo.mocker command launches live Mocker workers without replay orchestration.

For task-oriented instructions, see Run a DynoSim Simulation, Sweep DynoSim Configurations, and Benchmark Planner Decisions. For engine-core details, see Mocker Engine Architecture.

Replay harness

The load driver supplies either a trace or a generated workload. The harness admits requests into a simulated configuration, advances the simulation, and passes lifecycle timing to the trace collector. The collector produces the AIPerf-style terminal summary and JSON report.

Aggregated simulation uses one event loop for single-worker, multi-worker, and attention-DP deployments. Disaggregated simulation uses separate prefill and decode pools. The same replay boundary also supports KV routing and Planner-in-the-loop experiments.

Trace ingestion and session reconstruction

Mooncake-compatible formats carry request timing, token lengths, prefix hashes, and optional session or dependency fields. Dynamo request traces are loaded directly from one or more JSONL or JSONL.GZ shards. The loader maps Dynamo’s sequence-aware hashes to compact replay IDs without writing an intermediate Mooncake file and validates that every shard uses the same embedded trace block size.

Context-free Dynamo records become independent requests. When every request contains agent_context, the loader reconstructs session dependencies and tool waits. It rejects mixed traces instead of silently dropping agent relationships. This path preserves session identity exported by supported agent harnesses, including parent and child sessions.

Component composition

The engine core owns scheduling, KV allocation, prefix caching, preemption, and forward-pass timing. The multi-engine layer adds behavior that requires coordination across engine instances.

Execution model

Offline execution drives Mocker engine cores directly. It uses a logical clock and does not require a frontend, worker registration, etcd, NATS, or HTTP traffic. This path is appropriate for fast, repeatable configuration comparisons and continuous-integration tests.

Run python3 -m dynamo.mocker for the supported live worker CLI. The Python replay SDK retains online mode for programmatic callers, but no public Replay CLI currently exposes online replay orchestration.

Routing simulation

Round-robin simulation assigns requests without KV-aware scoring. KV-router simulation layers an in-process indexer, worker queues, and routing lifecycle events over the engine cores. Router queueing uses simulation time in offline mode.

The router observes request admission, prefill completion, and sequence release. It can estimate prompt-side load from token counts or an AIConfigurator timing model. These estimates influence worker selection but do not replace the engine scheduler’s own queue and KV-cache behavior.

Policy-class replay uses the same policy-family and cache-bucket model as the live router:

The trace loader preserves policy_class metadata for the replay runtime. The unified public YAML does not expose the former startup policy-file and --model-name CLI controls.

Planner simulation adapter

Planner-in-the-loop simulation supplies traffic observations from the replay harness instead of Prometheus. On each Planner traffic tick, the adapter reports:

Replay metric	Planner meaning
`num_req`	Completed requests in the observation window
`avg_isl` / `avg_osl`	Mean raw input and output lengths
`avg_kv_hit_rate`	Mean router prefix-cache hit rate at admission
`avg_accept_length`	Mean visible output tokens per decode request-forward

KV hit rate and speculative accept length use last-value semantics in the Planner. Missing accept length samples preserve the previous valid value. Without valid speculative-decoding metadata, the effective accept length is 1.0.

Speculative decoding changes the Planner’s effective decode latency and capacity calculations. It does not rewrite raw output length, which remains the input for KV residency, context-length, and request-length calculations.

The simulation adapter cannot auto-detect the GPU count from a deployment. Planner experiments must set prefill_engine_num_gpu and decode_engine_num_gpu explicitly when cumulative GPU-hours are part of the analysis.

Timing models

DynoSim can use the default AIConfigurator-backed timing model or explicit fixed and polynomial timing from engine.workers.<role>.timing. The timing model predicts prefill and decode duration. The Mocker engine still owns batching, KV-cache state, prefix reuse, preemption, and request progression.

AIConfigurator compatibility APIs from the aisimulate wheel are used in two distinct places:

engine.workers.<role>.timing.type: default configures the Mocker forward-pass timing model
router.prefill_load_model.type: aic configures router-side prompt-load estimation

Keeping these paths separate makes it possible to test router estimates independently from engine timing.