Dynamo v1.3.0

Dynamo v1.3.0 is the 16th feature release of the open-source distributed inference platform. It delivers the largest Dynamo Router buildout to date (a standalone selection service, the Branch-Sharded KV Indexer, compressed-radix-tree hot-path speedups, and topology-aware routing), overhauls tool-calling and reasoning into a configurable parser layer aligned to vLLM and SGLang across the model fleet, and deepens Kubernetes deployment around a production GPU Memory Service and DynamoGraphDeployment (DGD) v1beta1 admission. It also expands performance modeling and replay with AIC latency prediction and Mooncake trace replay, and moves the platform to CUDA 13 with refreshed vLLM, SGLang, and TensorRT-LLM engines and NIXL v1.x.

Get v1.3.0

Pull, deploy, and install with every artifact pinned to the v1.3.0 release set.

docker pull nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0
docker pull nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.3.0
docker pull nvcr.io/nvidia/ai-dynamo/kubernetes-operator:1.3.0
helm install dynamo-platform \
  oci://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform --version 1.3.0
uv pip install "ai-dynamo[sglang]==1.3.0.post1"

docker pull nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.3.0
docker pull nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.3.0
docker pull nvcr.io/nvidia/ai-dynamo/kubernetes-operator:1.3.0
helm install dynamo-platform \
  oci://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform --version 1.3.0
# TensorRT-LLM ships via the NGC container

docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.3.0
docker pull nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.3.0
docker pull nvcr.io/nvidia/ai-dynamo/kubernetes-operator:1.3.0
helm install dynamo-platform \
  oci://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform --version 1.3.0
uv pip install "ai-dynamo[vllm]==1.3.0.post1"

Highlights

Features & Improvements

Frontend

• GPT-OSS Harmony Parser Fixes: Corrected the Harmony parser to preserve analysis text as normal content when reasoning parsing is unconfigured (#9729), enabled GPT-OSS tool calling from Codex by retaining the <|call|> terminator and resolving Jinja .items() schema conflicts (#9778), corrected thinking behavior (#9844), handled bare Harmony stream markers starting directly at the commentary channel (#9897), and routed Harmony reasoning, visible text, and tool-call handoff into the correct Dynamo fields by channel and recipient (#10054). Preserved the directed commentary tool-call handoff to the downstream Harmony tool parser (#10111), and recovered tool calls emitted on the analysis channel using the recipient as the definitive signal (#10366).
• Experimental Parsers V2 Routing: Routed Qwen3-Coder and DeepSeek-V4 tool calls through the dynamo-parsers-v2 streaming parser behind a new DYN_ENABLE_EXPERIMENTAL_PARSERS_V2 flag (off by default); the v2 parser owns incremental tool-call emission and drops parameter values truncated at EOF (#10853). Added reasoning-parser parity cases for the “1 open + 2 close” pattern and fixed BasicReasoningParser and Gemma4ReasoningParser to keep stray close markers out of normal_text (#10250).
• Structural Tag Tool Calls: Added automatic xgrammar structural tag generation for tool-calling requests, constraining decoding in each model’s native tool-call format to enforce required/named tool choice, strict function schemas, parallel_tool_calls=false, and tool_choice="none" token bans. Disabled by default, configurable via --dyn-enable-structural-tag, supporting the hermes, qwen3_coder, deepseek_v3_2, and deepseek_v4 parsers (#9711).
• Skip Special Tokens Warning: Added a one-time diagnostic warning in the OpenAIPreprocessor when a request forces skip_special_tokens=true while a special-token-dependent parser is active, a combination that silently strips parser markers and leaks tool-call or reasoning markup into content (#10225).
• Realtime API Endpoint Build-Out: Wired the /v1/realtime WebSocket endpoint through ModelManager with a first-class ModelType::Realtime endpoint kind and bidirectional PushRouter transport (#9308), and replaced the placeholder chat-completion shape with OpenAI Realtime API event types, exchanging RealtimeClientEvent and RealtimeServerEvent frames and synthesizing a spec-compliant session.created event on connect (#9205).
• OpenAI Embeddings Format Support: Added dimensions Matryoshka truncation to the vLLM /v1/embeddings worker, validating and slicing each embedding (#9751), then added end-to-end encoding_format=base64 support: the Rust frontend accepts array and base64 shapes and both SGLang and vLLM backends emit the requested form (#9887).
• MiniMax M3 Frontend Support: Added MiniMax M3 support to the OpenAI frontend: thinking/thinking_mode handling for enabled, disabled, and adaptive modes, parser alias normalization, special-token and trailing EOS handling, prior tool-call argument validation, and MiniMax-M3-VL multimodal context-length detection (#10983).
• Unified Backend Logits and LogProbs: Added dynamo.common.backend.logprobs as the single source of truth for logprob option parsing and per-chunk extraction, wiring logprobs end-to-end on the unified path so output_options reaches sampling_params and chunks carry log_probs and top_logprobs keys (#10149), and brought the custom logits-processor smoke path and shared serializable wire format to the unified vLLM and SGLang backends (#10224).
• Frontend nvext Switches and Relocation: Added frontend master switches for the nvext and admin-API surfaces, letting operators close off non-OpenAI-spec surfaces on the frontend HTTP service without affecting inference, metrics, models, or health probes (both on by default) (#10556), and moved typed NvExt handling out of the OpenAI module into the LLM common extension layer, keeping dynamo-protocols extension-agnostic and adding Anthropic Messages ingress support for nvext.agent_context (#10784).
• Dynamo Routing Header Standardization: Added x-dynamo-request-priority and x-dynamo-request-strict-priority HTTP headers as alternatives to nvext.agent_hints, letting OpenAI and Anthropic clients set scheduling priority without modifying request bodies, behind DYN_DISABLE_FRONTEND_NVEXT (#10871), then standardized every Dynamo-owned public routing header on the x-dynamo-* namespace while keeping the original unprefixed worker and DP-rank names as compatibility aliases across EPP, ext-proc, frontend, tests, and docs (#10873).
• HTTP Header Metadata Propagation: Extracted HTTP request headers matching a configurable prefix into the context metadata map, so tenants and routing systems can inject key-value metadata at the HTTP boundary and flow it end-to-end across HTTP, streaming, and gRPC endpoints. Oversized or invalid header metadata now returns a clear “request headers too large” error (#9726).
• LoRA Controller and Routing: Added a LoRA controller with HRW/MCF placement, filtered request routing, metrics, discovery, and configuration wiring across the vLLM and SGLang backends. Preserved session affinity when LoRA is disabled, rejected incompatible direct and advanced routes, and returned a resource-exhausted response when every worker in a selected replica set is saturated (#8180).
• Admission Control Flag Replacement: Replaced the boolean --no-admission-control opt-out flag with an enum-valued --admission-control {token-capacity,none} flag defaulting to none, so operators must now explicitly opt in to busy-worker rejection. DYN_NO_ADMISSION_CONTROL gave way to DYN_ADMISSION_CONTROL (#9694).
• Configurable Overload Status Code: Added the DYN_HTTP_OVERLOAD_STATUS_CODE environment variable to set the frontend’s HTTP status code for overload rejections, with the default 529 unchanged. Deployments behind proxies or clients that only understand 503 can now select that code without a code change (#11420).
• L1 Tokenizer Prefix Cache: Added an opt-in CachedTokenizer that records prefix tokenizations at special-token boundaries and re-encodes only the diverging suffix, enabled per-frontend via DYN_TOKENIZER_CACHE=1 with a configurable budget and /metrics hit and miss counters (#9742), then extended it with partial-hit handling for growing multi-turn conversations, a moka W-TinyLFU backend, single-pass Aho-Corasick boundary detection, and broadened tests across tokenizer families (#10201).
• Streaming Hot Path Optimization: Optimized the OpenAI chat SSE streaming path by replacing the per-chunk flat_map to Vec to stream::iter plumbing with a single async_stream adapter that reuses event storage and serializes JSON directly (#10433), applied the same rework to the Responses streaming adapter so it owns the converter and serializes events with borrowed Serialize wrappers (#10498), and to the Anthropic Messages streaming adapter to drain a reusable event buffer from a single async stream while preserving error handling, disconnect monitoring, and keepalive behavior (#10499).
• Frontend Hot-Path Optimizations: Applied three profile-guided optimizations to the frontend hot path: returning the shared Arc instead of cloning the cached prefix on a tokenizer L1 cache hit, caching per-model Prometheus metric handles in ResponseMetricCollector rather than resolving them per chunk and per token, and capping the log crate at compile-time debug so per-character trace! events from the HF tokenizers crate compile to no-ops in release. Stock-tokenizer-path throughput improved roughly 1.45x, from about 189 to about 275 requests per second (#10273).
• Self-Host Sibling Metadata Harvesting: Harvested non-weight sibling files (preprocessor_config.json, special_tokens_map.json, added_tokens.json, tokenizer.model) into slug_dir at the end of resolve_metadata_files to fix KV routing regressions for from_pretrained consumers (#9610), had the worker advertise non-typed siblings through a new ModelDeploymentCard.extra_files field for custom directories without an HF source (#9707), made the vLLM and SGLang native preprocessors consume the resolved slug_dir instead of re-running fetch_model (#10037), and had the MetadataArtifactRegistry auto-clean entries on detach to stop leaks on LoRA detach or model reload (#10351).
• Disagg Processor Engine Support: Routed the vLLM preprocessor through a new Rust RoutedEngine that wraps Client or KV router with PrefillRouter for disaggregated serving (#9503), extended the same path to the SGLang processor and added structural_tag for guided decoding on the Rust side (#9577), and added request migration failover to the vLLM and SGLang processor paths (#9617).
• Disaggregated Topology Readiness: Migrated the worker set key to include worker_type, removed the legacy ModelType::Prefill bit, registered encode workers, and added per-model topology gating in the frontend (#9815), plus a read-only GET /v1/models/{model}/ready endpoint exposing structured per-namespace worker readiness for a model (#10383).
• Worker Type Topology Scaffolding: Added a WorkerType enum with Prefill, Decode, Encode, and Aggregated roles, plus worker_type and needs fields on the model deployment card and live readiness methods on Model, a first step toward making the frontend aware of disaggregated serving topology before serving requests (#8626).
• Python-Rust PyO3 Bridge: Exposed the runtime’s bidirectional engine path to Python so an async generate worker can be registered and driven end-to-end via PyAsyncRequestStream and serve_bidirectional_endpoint (#10135), propagated HTTP-status-carrying exceptions through the boundary so unsupported image formats return their intended status instead of HTTP 500 (#10364), and removed the cyclic Python to env to Rust config side channels in the frontend by passing resolved config through the PyO3 contract as explicit Rust fields (#10446).

Agents

• Agent Request Trace Capture: Added agent traces to the v1/completions endpoint and refactored the internal streaming trace infrastructure (#9125), plus optional finish_reason_metadata recording backend stop metadata, final chat and completions finish metadata, and tool-call metadata without storing arguments, surfaced in the Perfetto conversion (#9817).
• Header-Only Agent Trajectory Identity: Added a deterministic program-close signal via nvext.agent_context.trajectory_final, with a symmetric resume INFO log and scheduler docs for dynamo.thunderagent_router (#10172), and mapped trajectory identity into SGLang as a radix-cache session, normalizing an explicit final header into kv_hints.evict_trajectory (#10214). Made trajectory_id the only required identity field and added the generic x-dynamo-trajectory-id HTTP fallback (#10800), removed nvext.agent_context from public request bodies for a header-only contract parsed at the HTTP boundary (#10808), and preserved the immediate Claude parent for nested subagents via x-claude-code-parent-agent-id (#10942).

Reinforcement Learning

• RL Tokens-In-Tokens-Out Support: Added the Rust lib/llm nvext Tokens-in-Tokens-Out protocol, preprocessor, and response plumbing, carrying nvext.token_data, cache_salt, extra_fields, token constraints, prompt_logprobs, detokenize, and stop token IDs through the request and response path, validating passthrough cache_salt and stop_token_ids while rejecting unsupported truncate_prompt_tokens (#9649), plus a read-only GET /v1/rl/workers discovery endpoint in the new dynamo-rl crate gated behind DYN_ENABLE_RL and DYN_RL_PORT (#9681).
• vLLM RL Token-In/Token-Out Support: Added vLLM worker-side support for nvext Tokens-in-Tokens-Out requests, forwarding RL/TITO flags through the CLI and runtime config, preserving cache salt on decode and disaggregated prefill prompts, and exposing completion token IDs, logprobs, and engine data parity fields (#9651), registered worker RL admin routes under /engine/<route> with LoRA request validation (#9680), and added NeMo-RL NCCL weight updates (#11034).
• SGLang RL Metadata Uploads: Added an opt-in path to upload large reinforcement-learning serving metadata (routed expert captures, top-logprobs, tensors, NumPy arrays) as zstd-compressed msgpack objects through fsspec storage backends. Gated by SGLang RL mode and a per-request nvext.metadata_upload.url field, so callers can scope rollout metadata to any installed fsspec target (#10034).

Multimodal & Diffusion

• Multimodal KV Routing Support: Added MM-aware KV routing on the SGLang backend via per-image pad_value substitution with auto-detected backend identity (#9561), tightened post-merge tests to fail closed when routing degrades and fixed the Qwen2-VL, Qwen2.5-VL, and fastokens paths (#9441), dropped Phi-3 from MM-aware routing since its sliding-window KV events are not admitted to the contiguous-prefix index (#10441), and added the Qwen3.5/3.6 and Kimi K2 families with a tightened pre-merge gate (#10999).
• SGLang Multimodal Disaggregated Support: Forwarded image and video inputs to both SGLang disaggregated prefill and decode workers so media is encoded and KV-aligned over NIXL instead of dropped and decoded as text (#10643), normalized multimodal content shapes on the SGLang chat-processor path so chat templates bind media placeholders (#10673), rejected multimodal requests when a PD worker lacks the multimodal-worker flag (#10680), and unified multimodal E/P/D configuration flags across backends with deprecation warnings for the prior backend-specific flags (#10690).
• Encoder-Result Handoff Contract: Added an engine-opaque, object-only encoder_result payload to PreprocessedRequest, LLMEngineOutput, and BackendOutput, with producer helpers in Rust and a new dynamo.common.backend.multimodal Python module, the foundation for multimodal Encode worker to Prefill peer handoff (#10969).
• Cache-Aware Embedding Device Routing: Integrated EmbeddingCache awareness into the heterogeneous EPD weighted device routing so cache-hit requests are excluded from the CUDA-to-CPU ratio accounting and fall back to round-robin (#8750), and added video embedding cache support to the SGLang multimodal encode worker, storing video-specific metadata and config-aware video cache keys while preserving image cache behavior (#8298).
• OpenAI Embeddings Wire Performance: Added the OpenAI /v1/embeddings dimensions field with Matryoshka truncation in the SGLang handler (#9722), submitted batched prompts concurrently via asyncio.gather so vLLM’s scheduler coalesces them into a single forward pass (#10117), and switched the internal worker-to-frontend wire format to base64-encoded bytes, decoded back to floats at the HTTP boundary, dropping latency at --batch-size 15 from 215 ms to 124 ms on arm64 GB200 (#10139).
• Diffusion Unified Backend Onboarding: Onboarded diffusion image and video generation onto the unified backend, so raw-media engines for TensorRT-LLM and SGLang share the same Worker lifecycle as token LLM engines: signal handling, graceful drain, 3-phase shutdown, and health checks (#10371).

Scheduling

KV Router

• Branch-Sharded KV Indexer: Added the AsyncShardHandle trait and generalized BranchShardedIndexer over in-process and future remote shard backends (#9763); removed the scheduler hop from in-process anchored reads (#10200); added O(1) anchor cleanup and worker-id lookup via per-worker indexes (#10302); and borrowed shard-read suffix slices instead of per-lookup allocations (#10495).
• Standalone KV-Router Selection Service: Added a runtime-independent slot-tracker microservice with HTTP worker registration, hash-based lifecycle accounting, and advisory load reads (#10291), plus optional ZMQ replica sync for lifecycle events and dynamic peer management (#10394). Added a runtime-free selection service with worker-catalog reconciliation, selection and reservation APIs, and scoring endpoints (#10641), extended it with startup recovery from HTTP indexer peers, best-effort replica sync, and Mooncake-style token-overlap summaries (#10745), and exposed the selection core as a feature-gated Python class for in-process use (#10766).
• Router Overlap Score Handling: Added a KvRouter overlap-scores API and standalone router endpoint returning per-worker tiered KV overlap with optional shared-cache hit context (#9538); refreshed overlap scores at dequeue time for requests whose scheduler queue wait exceeds a threshold so stale scores no longer misroute workers (#9663); and added opt-in decay of device-local overlap credit under active prefill load, defaulting to off and configurable through Rust, Python, CLI, and environment (#10574).
• Topology-Aware Routing Constraints: Added worker taints and request-side RoutingConstraints for KV router selection, enforcing required_taints and biasing traffic with preferred_taints (#9558); added typed topology metadata and KV-transfer enforcement fields in ModelRuntimeConfig plus canonical topology-taint generation (#9767); and propagated topology constraints to decode from the selected prefill worker, failing closed only for required KV-transfer topology policy when prefill-worker attribution is unavailable (#9893).
• Global Router SLA Retries: Added priority-retry configuration for global-router pool failures, retrying prefill, decode, and aggregated requests on faster pools before streaming starts, with documented pool priorities, defaults, and unit coverage (#9460), and moved per-request target TTFT and ITL fields into a schema-checked router sub-field under nvext (#9845).
• KV-Router Recovery Hardening: Added a bounded local pending-live-event buffer for KV-router recovery and split the local-indexer recovery code into focused modules (#9881); added batched replica-sync side effects, cancellation-aware push routing, and ghost-booking prevention (#10331); and added per-rank cancellation tokens that abort in-flight recovery when a worker is removed (#10616).
• Lower-Tier Cache-Hit Scoring: Fixed the KV-router lower-tier walk to seed query_lower_tiers from primary-only MatchDetails, then merge side-indexer scores into the device output, so side-only worker-block credit is no longer dropped from host and disk cache-hit math (#9797); exposed host_cache_hit_weight and disk_cache_hit_weight through the dynamo.frontend and dynamo.router CLI and DYN_ROUTER_* env vars at existing defaults of 0.75 and 0.25 (#10157); and routed vLLM native CPU-offload KV events to the HostPinned tier with lower-tier applies counted in kv_cache_events_applied (#10368).
• KV-Router Hot-Path Optimizations: Reduced block-hash cloning in router-lookup and route-time recording paths (#9960); replaced per-block approximate-prune heap entries with 100ms expiry buckets for a 13.7% throughput gain on the bs64 c384 multi-frontend AgentX benchmark (#10521); and reused caller-owned child-traversal storage during CRTC lookup repair while skipping redundant same-node worker updates (#10540). Changed compressed-edge remove bookkeeping to return only the newly uncovered hash range for a ~28x speedup in store/remove event processing (#10676), and batched CRTC remove handling for hashes on the same compressed edge with a split-safe per-hash fallback (#10858).
• Active Sequence Load Tracking: Optimized scheduler load projection by replacing the prompt registry’s DashMap load table with a dense worker load table and updating active_sequences_bench to measure the production project_worker_loads path (#10935); added incremental worker-overload tracking in KvWorkerMonitor via a dedicated OverloadedWorkerTracker that updates only the touched worker per ActiveLoad event (#10645); and reduced hot-path locking by switching per-worker load entries to SeqLock and the worker load table to IndexMap (#10967).
• DP-Rank Session Affinity: Promoted sticky-session affinity from worker to (worker, dp_rank) granularity so a session pinned to a multi-DP-rank engine stayed on the rank where its prefix was warm (#9920); added header-based session identity via X-Dynamo-Session-ID with Claude Code, Codex, and OpenCode fallbacks, router-local atomic first-dispatch affinity with idle expiry and active-request leases, and an independent --router-session-affinity-ttl-secs setting (#10875).
• Unified Routing Log Output: Standardized worker selection across all router modes into one structured info-level “Selected worker” log with consistent fields (#10554), and reclassified prefill capacity rejections from error to warn so routine load-shed backpressure is no longer logged as a failure (#10561).
• Strict Priority Tiers in KV-Router: Added nvext.agent_hints.strict_priority as a per-request pending-queue tier ordering higher values before lower, preserving FCFS, LCFS, and WSPT as secondary ordering within each tier and propagating the hint through push and picker routing, standalone requests, EPP, C JSON routing, and Python KvRouter.best_worker (#10638), and fixed the strict_priority call sites in the selection service that broke the build on main (#10689).
• Tiered Router Queue Backpressure: Added a configurable KV Router queue-depth backpressure option that caps per-worker queue depth by request cost (missing prefill tokens) and returns immediate rejections when caps are exceeded, shedding expensive low-cache-hit requests first under load. A new Prometheus counter tracks router backpressure by worker type and reason (#8144).
• Min-Cost Flow LoRA Placement: Added a churn-minimizing LoRA-placement solver that assigns LoRA adapters to workers via a min-cost flow algorithm with top-M candidate generation, delta freezing, and a configurable cost function (#8179).
• Router Predict-On-Route Colocation: Added --router-predict-on-route and --router-predicted-ttl-secs options so the KV Router co-locates burst-arrival sibling requests (parallel sampling, best-of-N, agent fan-out) on the worker that picked the first sibling, closing the window between the routing decision and the engine’s first block-stored event. When KV events are enabled, the Router runs a secondary short-TTL approximate indexer and scores each worker with the per-worker max overlap of both trees (#8276).
• Experimental ThunderAgent Router: Added the thunderagent_router service that wraps the native KvRouter with program-level pause/resume scheduling for agentic workloads, applying per-worker capacity accounting and request-boundary admission control. It is experimental and opt-in; requests lacking nvext.agent_context.trajectory_id pass through unchanged (#9448).
• KV-Aware Routing Unification: Added KV event and metrics source descriptors to the unified backend abstraction so engines declare publishers through a shared interface, with the Rust Worker owning publisher construction, the metrics poll loop, and shutdown ordering. Migrated vLLM, SGLang, and TensorRT-LLM onto one component naming convention for KV-aware routing (#9493).
• Stable Worker Routing ID: Added a stable_routing_id field on ModelRuntimeConfig that survives process restarts, sourced from the DYN_STABLE_ROUTING_ID or HOSTNAME environment variables and published through the worker’s existing etcd discovery entry, giving caching layers a persistent worker identity so Kubernetes StatefulSet pod restarts no longer trigger cache rebalancing and cold-start misses (#9665).
• Workers Endpoint Filtering: Enriched the GET /workers response with model name, tenant ID, and block size fields, added optional model_name and tenant_id query parameters for filtering, and fixed a block_size=0 time-of-check-to-time-of-use race by skipping mid-deregistration workers (#9983).
• Optional Binary-Search Indexer Mode: Added an optional binary-search algorithm to the KV Router’s PositionalIndexer::find_matches, selectable via the DYN_ROUTER_POSITIONAL_SEARCH_MODE environment variable or API, cutting index probes for long query-block lengths while producing byte-identical results to the default strided mode (#10181).
• Standalone Router Timing Propagation: Forwarded per-request timing from a standalone KV-router to the frontend on the data payload, so prefill_time_ms, ttft_ms, prefill_wait_time_ms, kv_hit_rate, and queue_depth reach the nvext timing field, Prometheus metrics, and agent traces in split deployments (#10182).
• Modeled Prefill Time Reads: Added a Rust-internal slot-tracker read for modeled remaining prefill time in the KV Router, exposing per-worker projected prefill consumption in signed milliseconds for load visibility while keeping unmodeled workers on the fast error path (#10190).
• Router Load and Request APIs: Added a potential-loads API, exposed active request counts to external APIs, and allowed markPrefill and markFree to execute without a context ID for uniform behavior, aiding cross-region balancing and detection of overloaded or queueing workers (#10412).
• KV Indexer Access Logging: Added structured JSONL access logging to the standalone KV indexer via a --access-log option recording timestamp, trace-id, method, path, model, status code, and duration per request, plus --trace-id-header and --access-log-local-time flags and a /reopen_logs endpoint for logrotate integration, improving production observability and request tracing (#10700).
• Standalone Indexer Metrics Export: Registered the core KvIndexerMetrics collectors in the standalone indexer’s Prometheus registry, so event warnings and errors now export through /metrics, matching the native Dynamo path (#10768).

Planner & Profiler

• Configurable Prometheus Client Auth: Made Planner’s Prometheus client TLS verification configurable via PROMETHEUS_SSL_VERIFY, defaulting to off to preserve existing behavior (#9510); added a custom CA-bundle path for private and internal roots (#9511), a static bearer token (#9512), and a bearer-token file re-read on every request for rotating credentials (#9513); and fixed extra URL query parameters on every PromQL request for tenancy-enforcing front-ends (#9557).
• MTP Accept Length Scaling: Added MTP accept-length scaling to the Planner, publishing backend spec-decode metadata in the ModelDeploymentCard and scraping speculative accept length from worker Prometheus metrics to discount decode ITL without altering raw OSL, KV, or context estimates, and rebuilding AIC perf models when capabilities change (#10435); added replay support carrying accept length through mocker engine passes, PSM replay, orchestrator plugin metrics, and the plugin proto schema, and fixed PSM replay tick time to use the scheduled replay clock (#10501).
• Attention-DP Forward Pass Metrics: Enabled TensorRT-LLM ForwardPassMetrics under Attention-DP by removing the Dynamo-side gate that disabled them when attention_dp_size exceeded one, so the Planner now receives one metrics stream per Attention-DP rank and can detect load imbalance (#9059).
• Planner Load Optimization Target: Added an optimization_target=‘load’ option for the Planner with configurable prefill queue token and decode KV utilization thresholds, forcing reactive load scaling and disabling throughput scaling with a warning (#9590).
• Planner Gzip Diagnostics Logs: Added a compressed JSONL diagnostics sidecar written next to each Planner HTML report by default, capturing per-tick snapshots; a report_write_gzip_log setting disables it, and writes stay best-effort so gzip failures warn instead of breaking the HTML report (#9623).
• Planner Recommended Replica Markers: Added recommended prefill and decode replica markers to the Planner HTML diagnostics Replica Counts plot, and documented local Planner advisory mode as suggestion-only for evaluating configurations without scaling (#9644).
• SLA Target Metrics Exposure: Added sla_target_ttft_ms and sla_target_itl_ms Gauges to the Planner’s Prometheus metrics, publishing operator-configured SLA targets so Grafana dashboards can compare observed and estimated latency against the SLA boundary (#10032).
• Planner Plugin Framework Infrastructure: Added an opt-in tick-engine path for the Planner that runs decisions through a PREDICT, PROPOSE, RECONCILE, CONSTRAIN pipeline driven by in-process or gRPC plugins, enabled per planner via scheduling.use_orchestrator=true while the legacy PlannerStateMachine path stays the default. This infrastructure-only change ships proto and Pydantic types, transport clients, a plugin registry and scheduler, merge algorithms, and static-config and gRPC self-register plugin registration, with builtin plugins to follow (#10124).
• Rust Engine Perf Shim: Routed SLA Planner performance queries through a new PlannerEnginePerfModel adapter backed by the Rust engine perf model, with legacy Python regression fallback when the optional Rust shim is unavailable, and relaxed planner bootstrap requirements so missing pre-deployment data no longer blocks throughput mode (#10229).

Performance Modeling & Replay

Mocker & Simulation

• KVBM Offload Replay Simulation: Added mocker support for KVBM G3 offload simulation following the G1 to G2 to G3 tiering path with staged promotion, and a first-admission prefix-cache reused-ratio metric in replay reports (#9337); modeled G4 object-storage offload as shared infinite-capacity storage with G2-to-G4 pipeline and bandwidth configs (#9939); and extended the Rust-native offline-replay benchmark with aggregated and disaggregated topology selection, KVBM capacity and bandwidth options, worker-initialization error propagation, and benchmark documentation (#11010).
• AIC Latency Prediction Integration: Added a feature-gated AIC forward-pass engine perf shim to the mocker, exposing RustEnginePerfModel through PyO3 (#10150), and used AIC memory estimates to size KV cache blocks while preserving the legacy 16384 fallback when AIC is disabled (#9598). Switched latency prediction to the AIC Rust crate as a GIL-free callback on the hot path (#10615), forwarded AIC quantization-mode overrides so replay latency and capacity track quantized deployments (#10914), scaled num_gpu_blocks by attention_dp_size for DP-attention KV provisioning (#10964), and divided the scheduled batch by attention_dp_size for correct per-rank perf queries in offline replay (#11002).
• TensorRT-LLM Scheduler Simulation: Added TensorRT-LLM scheduler simulation to the mocker via engine_type=trtllm, modeled as a scheduling-policy variant of the vLLM scheduler core with a GUARANTEED_NO_EVICT policy and AIC perf-model integration (#10193), and terminal rejection of unschedulable requests whose footprint exceeds the KV pool, propagated through every replay driver via a new rejected flag on OutputSignal (#10287).
• Mocker Replay Performance Tuning: Routed one-worker aggregated offline replay through SingleRuntime for vLLM, SGLang, and TRT-LLM across flat, workload, concurrency, and agentic entrypoints while retaining AggRuntime for multi-worker, KV-router, planner-driven, and disaggregated replay, and added cross-runtime parity coverage and an --engine-type option to offline_replay_bench (#10629); and reduced scheduler and FPM publish overhead by skipping timerfd construction for consumed deadlines, reusing the publisher task’s MessagePack buffer, and caching the event-plane subject (#11025).
• Eagle Speculative-Decoding Perf Model: Added MTP/Eagle speculative-decoding support to the AIC perf model used by mocker and replay, exposing the aic_nextn draft-token count and aic_nextn_accept_rates per-position accept rates through the CLI, Python bindings, and mocker serde so AIC applies the speculative-decode decode speedup during latency modeling (#10197).
• MTP Burst Sampling: Added speculative-decoding (Multi-Token Prediction) support to the mocker engines with conditional acceptance sampling and atomic burst reservations, exposed through new options --aic-nextn, --aic-nextn-accept-rates, and --aic-mtp-seed across live and replay paths (#10436).
• Atomic Source Hold Primitive: Added a scheduler-owned source-hold registry with a stable HandoffId to the mocker, letting vLLM and SGLang retain completed request state and defer terminal cleanup so release and cancel clean up exactly once, even for early or duplicate commands (#10831).
• Disaggregated Handoff Lifecycle: Replaced the mocker’s synthetic second-request approximation of disaggregated serving with one explicit handoff lifecycle covering source hold, destination reservation, transfer, activation, release, completion, cancellation, and failure, shared across offline replay and live bootstrap. It preserves vLLM source-first and SGLang destination-first admission semantics and surfaced latent fixes in virtual-time visibility, offload wakeups, cancellation cleanup, worker retirement, and queue accounting (#10915).
• Unified KV-Cache Estimator: Routed the mocker and replay num_gpu_blocks estimator through aiconfigurator’s unified sdk.memory.estimate_num_gpu_blocks API instead of hand-rolled per-backend budget math, making aiconfigurator the single source of truth. SGLang block counts grew slightly more conservative by subtracting full non-KV memory; vLLM and TensorRT-LLM counts were unchanged (#10686).

Trace Replay

• Mooncake Trace Capture and Replay: Added per-request Mooncake trace capture for Rust OpenAI chat and completion serving via the DYN_REQUEST_TRACE environment variable, with rolling sequence hashes, partial output length on cancellation, and configurable trace sinks (#10381), introduced a mooncake-delta trace format that accumulates session input deltas into cumulative prompts before recomputing engine block hashes (#9653), added an experimental agentic Mooncake replay path with dependency edges, branch markers, and tool wait timing (#9728), and fixed mooncake-delta prompt reconstruction to include the previous prompt and generated output in each follow-up turn (#11060).
• Planner Replay Goodput Metrics: Added SLA-satisfying goodput and provisioned GPU-hours to the planner-replay report, classifying each completed request against an optional SLA using aiperf-matched ITL, TTFT, and e2e bounds independent of the planner scaling SLA (#10739); exposed the goodput SLA on the plain trace-replay binding so static non-autoscaling replays also report the goodput keys (#10849); and supported concurrency-capped and synthetic workloads in the planner-in-the-loop path beyond arrival-timestamp traces (#10888).
• Offline Replay Event-Driven Path: Unified offline replay onto one event-driven run loop where a planner tick is a first-class PlannerTick event, fixing an unbounded FPM-snapshot memory leak on the plain run path (#10940), added a --max-sim-time-minutes flag that ends the benchmark at the specified simulated wall time across disagg and agg paths (#9699), and changed --replay-concurrency to schedule multi-turn sessions depth-first as a session cap, admitting a new session only when an active one finishes (#10516).
• Replay Per-Request Metrics: Added a --report-jsonl option to offline dynamo.replay that emits one JSON object per completed request with TTFT, ITL, ISL/OSL, cache hit rate, and worker index (#9720); added native vLLM/SGLang-style Prometheus metrics for the live mocker covering request timing, cache hit rates, queue status, and preemption counts (#10056); and trimmed online-replay diagnostics by demoting replay_diag breadcrumbs to debug level and removing test timing prints (#9565).
• Output Token Replay: Added deterministic output-token replay for Dynamo mockers, where enriched Mooncake traces supply exact output_token_ids and live requests select a replay row via an output_replay_id:<key> annotation, retaining the random-token fallback when no replay plan is present (#10877).
• Claude Session Trace Export: Added export of local Claude Code root and subagent sessions as canonical dynamo.request.trace.v1 request and tool events, with explicit source-to-tool-to-child causality, making historical sessions replayable while an always-on verifier fails on fidelity drift (#10971).

Kubernetes Deployment

• GMS Weight Loading Integration: Integrated ModelExpress P2P weight transfer into the GMS loader and worker via a LoadStrategyChain for automatic weight-source detection (#8218), and cut GMS load overhead by skipping torch import on the server startup path and bounding NIXL staging workers (#9635). Auto-enabled the SGLang memory saver path for GMS startup (#9647), used the DP-adjusted device for the early vLLM GMS connection (#9840), refreshed vLLM KV cache wake state after GMS remaps VAs (#10319), set a 512 MiB default scratch alias size for shadow-engine scratch mappings (#10329), and dropped the implicit failover requirement from standalone inter-pod GMS in the Operator (#10378).
• GMS Sidecar Lifecycle Rework: Coordinated GMS sidecar lifecycles through the lock state machine instead of kubelet probes and init-phase ordering, dropping the gms-server StartupProbe and fixing a saver Job-completion race and a too-short 30 second read-only reconnect timeout (#9514). Reworked the checkpoint-client API around user-declared Kubernetes containers so the Operator no longer creates gms-loader or gms-saver containers (#9641), and made the _allocator_ext C++ extension PEP 703 free-threading ready by replacing torn-pointer callback globals with a magic-statics singleton (#9575).
• Snapshot Restore Probing and Runtime: Reworked snapshot checkpoint and restore handling to gate checkpoints on target-container readiness with an OCI runtime container-ID fallback (#9534), tightened restore-target StartupProbe cadence so containers flip Ready as soon as CRIU restore completes (#9627), and let restore pods start container-ID polling before Kubernetes marks the pod Running (#9984). Refreshed restore-time runtime env before vLLM and SGLang create the DistributedRuntime and split the snapshot utils into focused modules (#10376), added a CRIU UPDATE_INETSK plugin that remaps IPv4 socket addresses across pod IPs and defaulted established TCP socket restore (#10727), and applied fast 1s worker probes only once checkpointInfo is Ready, bounding restore startup probes to 30 minutes (#10859).
• DGD Auto-Checkpoint Lifecycle Management: Scoped DynamoGraphDeployment automatic checkpoints to the owning DGD component worker generation instead of legacy identity-hash reuse, removing the need for checkpoint.identity in mode: Auto (#10177). Added a checkpoint.startupPolicy and Pod CREATE mutating webhook that defaults workers to Immediate cold start while the checkpoint job runs in the background (#10179), and a checkpoint.deletionPolicy with finalizer-based artifact cleanup that defaults to deleting DGD-managed checkpoint CRs and stored artifacts on DGD deletion (#10183).
• TensorRT-LLM Snapshot Support: Added TensorRT-LLM support for Dynamo Snapshot restore, delaying DistributedRuntime creation until after restore and reloading the Kubernetes restore identity before endpoint registration, plus a Qwen3-0.6B single-GPU example (#10432). Introduced the namespaced PodSnapshot and cluster-scoped PodSnapshotContent CRDs (nvidia.com/v1alpha1) with a PodSnapshotReconciler modeled on the CSI VolumeSnapshot pattern (#10820).
• Grove DGD Integration: Preserved existing Grove PodCliqueSet replica fields during DGD reconciliation so live scaling stays driven through Grove scale subresources instead of PCS template updates (#9773). Added a DynamoGraphDeployment priorityClassName field passed through to the Grove PodCliqueSet and rejected by the webhook unless the Grove pathway is active (#10217), and a component-level minAvailable knob for Grove-backed DGDs controlling gang scheduling and termination (#10532).
• Worker Topology Label Injection: Injected topology metadata into worker pod templates when spec.experimental.kvTransferPolicy is set, adding a Downward API volume and runtime topology env vars while leaving frontend pods and policy-less workers unchanged (#9792). Added a TopologyLabelReconciler that watches annotated scheduled pods and copies the configured topology label from their node, marks worker DCDs with nvidia.com/topology-label-key, and adds node-read RBAC for cluster-wide and namespace-restricted installs (#9879).
• Volcano Scheduler Support: Added Volcano scheduler support for Grove in the Operator (#11348).
• Rust ext_proc EPP Server: Added a Rust ext_proc service for the Endpoint Picker (EPP) that Envoy calls to make routing decisions, replacing the llm-d Go implementation and cutting the call path from three language boundaries to one (Envoy C++ to Rust gRPC). The server calls Dynamo Router, tokenizes the request body for KV-aware routing, and locates workers via DistributedRuntime discovery plus Kubernetes pod watches (#8783).
• KV Transfer Policy API: Added an experimental topology-aware KV transfer policy under spec.experimental.kvTransferPolicy on DynamoGraphDeployment, with labelKey, domain, required or preferred enforcement, and worker-side DYN_KV_TRANSFER_* env injection (#9768). Added Grove ClusterTopology as an alternative topology source via clusterTopologyName, projecting topology levels as nvidia.com/dynamo-topology.<domain> pod labels (#10041).
• Custom Init Container Image: Added a Helm value to configure the snapshot DaemonSet init container image, letting users override the default in their charts (#8432).
• Controller-Manager Pod Metadata: Added controllerManager.podLabels and controllerManager.podAnnotations values to the Operator chart, letting users inject pod-level labels and annotations onto the controller-manager Deployment without forking the chart for observability integrations such as Datadog and Prometheus (#9195).
• Power Agent DaemonSet: Added a standalone Power Agent DaemonSet for per-node GPU power-cap enforcement, packaged as a Helm chart with DaemonSet, ServiceAccount, and RBAC for cluster-scoped or namespace-restricted mode. Power caps apply through the dynamo.nvidia.com/gpu-power-limit pod annotation, with NVML clamping, multi-pod policy, SIGTERM restore, and fail-safe reconcile (#9682).
• GB10 GPU SKU Support: Added support for the GB10 GPU SKU type, letting users select and deploy on GB10 hardware in deployment specs (#9976).

Fault Tolerance & Observability

• Unified Backend Observability Surface: Added end-to-end OpenTelemetry tracing across the unified backend, forwarding W3C traceparent to vLLM, TRT-LLM, and SGLang with cross-process trace linking for disaggregated serving (#9543). Added Prometheus metric-name parity via a new register_prometheus engine hook and Rust EngineMetrics handle (#9586), and health-check canary support through LLMEngine.health_check_payload() registered with the runtime HealthCheckManager, overridable by the --health-check-payload flag and DYN_HEALTH_CHECK_PAYLOAD env var (#9642).
• OTLP HTTP and Sampling: Added OTLP HTTP/protobuf export for traces and logs with protocol and endpoint selection via OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_EXPORTER_OTLP_ENDPOINT, plus optional trace sampling through OTEL_TRACES_SAMPLE_RATIO while keeping the default sampler (#10576), fixed a startup crash when the http/protobuf transport selected no default HTTP client (#10829), and preserved W3C traceparent sampling flags so inbound sampler decisions propagate to downstream workers (#10980).
• Embedding Workload Metrics Gating: Added embedding-shaped Prometheus histograms for batch size and per-request input tokens on the SGLang embedding worker (#9753), plus a dynamo_embedding_latency_seconds histogram for end-to-end /v1/embeddings latency in the Rust frontend (#9758), and gated the chat-shaped collectors (KV gauges, prefill/decode counters, LLMBackendMetrics) so they no longer emit zeros on embedding workers in SGLang (#9830) and vLLM (#9886).
• Request Trace Unification: Removed the separate DYN_AGENT_TRACE path and made DYN_REQUEST_TRACE the single trace switch, emitting enriched dynamo.request.trace.v1 rows with agent context, replay hashes, timing and token metrics, worker attribution, and finish-reason metadata for requests carrying nvext.agent_context (#10701). The Perfetto converter now infers tool slices from request.finish_reason_metadata.tool_calls when explicit harness tool events are absent (#10802).
• Log Crate Trace Capping: Capped the log crate’s trace level at compile time via the release_max_level_debug feature so log::trace! compiles to a no-op in release builds, removing per-character trace spam from the HF tokenizers normalization path (#10286), and removed the DYNAMO_SKIP_PYTHON_LOG_INIT env var from the replay and indexer launchers so logging state no longer leaks into child pytest processes (#9724).
• Chat Stream Metrics Annotations: Emitted the llm_metrics annotation from the vLLM chat processor so per-chunk TTFT and ITL histograms record samples under --dyn-chat-processor vllm (#10019), added a typed LLMMetricAnnotation fast path that attaches metrics directly to chat stream responses and skips the per-chunk JSON serialize/parse round trip while keeping the legacy path as a fallback (#10512), and emitted token, TTFT, ITL, and cached-token annotations from the SGLang chat processor (#10779).
• Worker Request Admission Control: Added worker-side defensive request admission, disabled by default, via --engine-request-limit and --dynamo-request-queue-limit flags that bound concurrent in-engine and Dynamo-queued requests with fast HTTP 503 shedding on overflow (#10509). Added a --no-admission-control escape hatch that clears busy-worker rejection thresholds while keeping router queueing independent, and updated the default queue and prefill-fraction thresholds plus the 503 rejection hint (#9547).
• Planner SLA Dashboard Updates: Added SLA target reference lines for TTFT and ITL to the planner Grafana dashboard’s Observed Latency panel, matching each target to its observed counterpart’s color and axis with a dashed style (#10712), and removed the dead Correction Factors panels that rendered permanent No data once the correction-factor gauges gave way to online FPM regression (#10824).
• Configurable SGLang Trace Level: Added a default SGLang trace level of 2 when --enable-trace is set, suppressing high-volume scheduler spans such as decode_loop while keeping per-request spans, and made the level overridable via SGLANG_TRACE_LEVEL (#9327).
• Standalone Resource Observability Dashboard: Added a standalone optional WebSocket dashboard served at / alongside the /metrics endpoint, giving lightweight host-resource debugging without standing up Grafana or Docker Compose. If dashboard packages are missing, / shows the install command and /metrics still works (#9780).
• Per-Model Grafana Dashboard: Added an engine-agnostic per-model Grafana dashboard with 24 panels across Overview, Frontend, KV Routing, and Workers rows, reading the standard dynamo_frontend_*, dynamo_component_*, and dynamo_router_* metrics filtered by a templated model selector. It preserves the existing UID so bookmarks survive (#9811).
• TensorRT-LLM Engine Health Monitor: Added TrtllmEngineMonitor, which polls TensorRT-LLM’s native check_health API and shuts down the engine, distributed runtime, and worker process on a fatal health state so Kubernetes can restart the pod (#10266).
• Registered Worker Metric: Added a Router component gauge, dynamo_component_router_worker_registered, that tracks registered backend workers and updates as topology changes or workers are removed, labeled by router worker ID, data-parallel rank, and worker type for worker-level monitoring (#10587).

vLLM

• Native vLLM Rust Backend: Added a native vLLM backend built on the vLLM Rust crates that act as clients of the Python vLLM engine, replacing the older AsyncLLM wrapper and leaving the Dynamo frontend untouched (#9206).
• Unified vLLM LoRA Support: Added dynamic LoRA adapter load, unload, and list operations to the unified vLLM engine, plus discovery publishing and per-request LoRA routing, closing the parity gap with the legacy vLLM path. Controls gate on --enable-lora and DYN_LORA_ENABLED, with a /v1/loras compatibility shim preserving the legacy HTTP surface (#10347).
• vLLM Worker Topology Registration: Added worker_type and peer-dependency declarations to each vLLM model registration call so aggregated, decode, and prefill workers declare their disaggregation role and dependencies. Re-exported WorkerType through dynamo.llm for backend imports (#9395).
• Mooncake PD Disaggregation Support: Let vLLM workers running MooncakeConnector participate in prefill/decode disaggregation by branching the prefill handler to set push-based kv_transfer_params and synthesize the decode-side disaggregated_params payload. No Rust router changes were needed, since the router treats disaggregated_params as opaque (#9414).
• Aggregated Text-Embedding Worker: Added an aggregated text-embedding worker shape to the vLLM backend, selected with a new --embedding-worker flag, so existing embedding deployments such as Qwen3-Embedding-0.6B can move onto Dynamo and vLLM without changing engine args. The worker serves the /v1/embeddings route and rejects incompatible combinations like prefill or decode disaggregation, multimodal flags, and benchmark mode at parse time (#9713).
• Aggregated LMCache MP Mode: Added a launch script for vLLM aggregated serving that starts an LMCache server sidecar and runs the LMCacheMPConnector, replacing the in-process LMCache path on non-Kubernetes environments, and realigned the LMCache integration and observability docs to the multi-process sidecar architecture (#9982).

SGLang

• SGLang Worker Topology Fields: Added worker-type and dependency declarations at SGLang LLM registration sites, deriving Decode, Prefill, and Aggregated roles from the serving mode so each worker reports its topology role at registration. Embedding, diffusion, and multimodal-encode workers default to Aggregated, preserving current behavior (#9397).

TensorRT-LLM

• Worker Type Registration: Populated worker_type and peer dependencies at the TensorRT-LLM register_model call site, deriving the disaggregation role (Prefill, Decode, or Aggregated) and required peers per branch of the configured disaggregation mode. Appended Encode to the dependency set when an encode endpoint is configured (#9396).
• Unified Backend Logits Processors: Ported the logits-processor pipeline to the unified TensorRT-LLM backend and added a backend-agnostic LogitsProcessorSpec layer that expresses per-engine logits-processor activation as serializable spec entries and applies cross-backend policies for per-request runtime state and PREFILL skip in disaggregated serving (#10080).

Infrastructure Modernization

• Call-Home Request-Plane Transport: Implemented the request-stream half of the call-home TCP transport so an upstream server can stream frames to a downstream client over a dedicated socket, settling on a unidirectional-after-handshake design (#9991). Wired end-to-end bidirectional dispatch through PushRouter::generate(ManyIn<T>) and AddressedPushRouter::generate_bidirectional with a new runtime-side ingress handler (#9674), and added an opt-in DYN_REQUEST_PLANE_CODEC=json|msgpack setting carried in the request control message while keeping JSON as the default (#10437).
• Runtime Event Plane Selection: Added an explicit event_plane parameter to the Python DistributedRuntime so callers can request ZMQ and avoid ambient NATS (#10021), then defaulted ZMQ across all discovery backends and kept NATS as an opt-in via DYN_EVENT_PLANE=nats (#10902).
• Configurable ETCD Lease TTL: Added the ETCD_LEASE_TTL environment variable to control the etcd lease time-to-live, previously hardcoded to 10 seconds, letting users tune lease duration at runtime (#8076).
• Context Metadata Propagation: Added a string-keyed metadata field to Context that propagates end-to-end through the Dynamo pipeline, from the Python caller through the network into the backend engine, and survives cancellation, retries, and prefill handoffs. Python callers can attach metadata such as tenant or region to requests and read or mutate it on the backend (#9662).

Hardware

Intel XPU

• XPU Multimodal Nixl Support: Added Intel XPU aggregated multimodal examples, router launchers, and test profiles using the shared MultimodalModelProfile pattern (#8112), with NIXL canonical VRAM/DRAM memory-type classification, Intel XPU device support for disaggregated E/PD, and XPU encode-worker kernel synchronization (#9073), and aligned DYN_SYSTEM_PORT defaults plus ZE_AFFINITY validation in the XPU launch scripts (#10488).

KV Block Manager

• KVBM-Logical Backend Rework: Restructured block lifecycle management into a unified store architecture, replacing the separate pool-based designs and simplifying block state transitions and registration flows under concurrent operations (#8793). Added single-lock prefix matching that resolves an N-hash active-or-inactive prefix under one store-mutex acquisition, plus a Criterion benchmark scaffold covering the lru, multi_lru, and lineage backends (#9551).
• Universal Hashing Consolidator: Added a KV cache event consolidator that uses universal hashing to process KVBM v2 event streams and vLLM KV events, deduplicating across sources and publishing a unified stream (#9480).
• Configurable Block Reset Toggles: Added pool-level and per-block controls to disable caching of inactive blocks in the KVBM block manager, keeping the reset-on-release preference across cache resurrections (#9504).
• Python Topology Config Reading: Added a Python topology environment and file reader for worker startup, and wired topology plus KV-transfer-policy fields into vLLM, SGLang, and TensorRT-LLM runtime config publication through the ModelRuntimeConfig binding (#9878).
• ModelExpress Engine Integration: Added ModelExpress engine-side weight loading for the vLLM and SGLang runtimes, installing the modelexpress client in both images by default and adding an ignore_weights option to register_model() so registration fetches tokenizer and config metadata without a full weight download. Deprecated the legacy Dynamo-owned vLLM --model-express-url wrapper in favor of the plugin-owned --load-format modelexpress path (#10049).

General

• LoRA Load Estimation Pipeline: Added a self-contained load-tracking pipeline for LoRA adapter allocation, with environment-variable configuration, an exponential-moving-average predictor, and a lock-free bucketed arrival-rate counter for per-LoRA request-rate estimation (#8178).
• Worker Topology in register_model: Extended register_model and LocalModel::attach with optional worker_type and needs parameters so backends can attach disaggregation role and peer dependencies to a worker’s ModelDeploymentCard, rejecting non-canonical combinations at the binding. The card checksum now covers these fields, so a rolling update that changes only topology metadata is rejected as incompatible rather than joining a WorkerSet with a stale card (#8700).
• Engine Management Routes: Added unified-backend engine-management plumbing exposed through Dynamo’s /engine/{route} runtime endpoint, with per-backend callbacks for profiling, sleep/wake and memory release/resume, weight updates, and elastic scaling across vLLM, SGLang, and TensorRT-LLM (#10094).
• Hugging Face Hub LoRA Sources: Added native hf://<repo>[@revision] support for dynamic LoRA loading, downloading adapters into the standard Hugging Face Hub cache and returning the immutable commit snapshot without a second copy under DYN_LORA_PATH. Revision-pinned, timeout-bounded snapshot downloads honor Hugging Face cache, token, endpoint, and offline semantics (#11816).

Recipes

• Kimi K2 Agentic Recipes: Replaced outdated Kimi K2.5 recipes with disaggregated TensorRT-LLM deployment using KV-aware routing and Eagle3 speculative decoding, aggregated round-robin variants, and ComputeDomain multi-node configuration tuned on the agentic code dataset (#9621), and added Kimi K2.6 Turbo recipes with Kubernetes deployments for chat and agentic workloads on B200 and H200 GPUs plus a trace-replay benchmarking guide (#10187).
• Qwen3 vLLM Serving Recipes: Added an aggregated serving recipe for Qwen3-0.6B with Gateway API Inference Extension (GAIE) integration, including an Istio DestinationRule required for the gateway proxy to reach the EPP service, plus a Qwen3.6-35B-A3B aggregated serving recipe with a standalone frontend (#8355).
• Qwen3-VL-32B-FP8 Recipe: Added a production-ready deployment recipe for the Qwen3-VL-32B-Instruct-FP8 vision-language model with aggregated single-GPU and disaggregated cross-vendor modes, the latter using Intel XPU encode and NVIDIA GPU decode with NIXL embedding transfer over RDMA. The recipe includes model-cache jobs and QPS-sweep multimodal benchmarks (#9252).
• GLM-5-NVFP4 EFA Variant: Added an EFA variant of the GLM-5-NVFP4 SGLang disaggregated recipe for GB200 clusters on AWS, routing KV transfer over AWS EFA RDMA via NIXL’s LIBFABRIC backend instead of UCX for p6e-gb200.36xlarge deployments (#9712).
• Nemotron-3-Super Turbo Recipes: Added NIM Turbo deployment recipes for NVIDIA-Nemotron-3-Super-120B-A12B using Dynamo and vLLM, covering NVFP4 on B200 and FP8 on H200 across chat and agentic serving modes, plus model-cache manifests and an AIPerf trace-replay performance job (#10223).
• Speculative Decoding Hardcoding Fix: Removed a hardcoded speculative decoding variable from the Kimi K2.5 TensorRT-LLM recipes and corrected the README. (#9637).
• Model-Cache Memory Limits: Added per-recipe memory and CPU resource requests and limits plus an absolute 16GB HF XET reconstruction buffer cap to all 15 model-download Jobs, preventing kubelet OOM eviction during downloads on Kubernetes (#9884).
• DeepSeek V4 SGLang Dependency: Added an explicit distro install to the DeepSeek V4 SGLang B200 runtime image, fixing a ModuleNotFoundError that caused the worker to exit during startup when the OpenAI SDK imported the missing package (#10085).
• gpt-oss-120b Recipe Revert: Reverted the gpt-oss-120b TensorRT-LLM aggregated recipe runtime image from 1.1.1 to 1.0.0, since the 1.1.1 runtime regressed the recipe on GB200 (#10147).
• Hugging Face Hub Pins: Bumped the Hugging Face Hub CLI pin from 1.11.0 to 1.16.4 across 17 model-download recipes and two vLLM LoRA sync jobs, so helper jobs install the click dependency required by hf download (#11273).

Bug Fixes

Frontend

• Tool-Call & Reasoning Parser Hardening: Aligned tool-call and reasoning parsers for GLM-4.7, DeepSeek V3/V3.2/V4, Kimi K2, Qwen, Gemma4, Mistral, MiniMax, and Nemotron with the vLLM/SGLang reference contracts, suppressing streaming markup leaks, recovering truncated and EOF-terminated tool calls, restoring chat-template fidelity, and routing structured output correctly (#9355, #9524, #9594, #9462, #9411, #10833) and related fixes.
• Worker-Set Readiness Routing: Confined routing to complete worker sets and added 529/503 codes separating capacity overload from backend unavailability (#10503, #10941).
• KServe gRPC Inference Dispatch: Fixed a KServe model_ready/server_ready race that reported ready before engines attached, and centralized dispatch to return clear model-not-found errors (#9619, #10621).
• EOS/BOS Token & Model Discovery: Resolved eos_token_id/bos_token_id from tokenizer and special-tokens files when absent from model config (#10672), and accepted the deepseek_v32 model_type spelling so V3.2 workers register (#10332).
• Sanitized Errors & API Compatibility: Stopped HTTP errors from leaking paths, versions, and stack text (CWE-209) (#10151), mapped backend InvalidArgument to HTTP 400 (#11591), and relaxed deserialization for multimodal EasyMessage and Anthropic message-level system roles so agent clients like Claude Code pass validation (#9470, #10108).
• Request Draining & Context Length: Drained inflight requests on shutdown so terminating frontend pods finish streaming instead of returning HTTP 500 (#10862), and propagated the backend context_length as max_model_len so long-context requests reach the backend (#10827).

Agents

• Coding Agent Header Handling: Merged runs of leading system messages when converting /v1/responses to chat completions, fixing Codex CLI first-turn failures on templates that reject multiple system messages (#10283), and normalized Codex, Claude Code, and OpenCode session headers into nvext.agent_context, mapping x-claude-code-agent-id to a child trajectory_id (#10782).

Reinforcement Learning

• Routed-Experts Capture for Log-Prob Recompute: Base64-encoded MoE routed-experts capture with a start offset so consumers align routing data to the completion (#10529), and fixed a first-token crash from double re-encoding routed_experts on non-DSv4 MoE models under SGLang v0.5.11+ (#9657).
• Rollout Serving Robustness: Rejected unsupported RL output fields completion_token_ids and prompt_logprobs on /v1/responses with a field-specific 501 (#11265), added a configurable weight-update init watchdog DYN_RL_INIT_WEIGHTS_TIMEOUT_S (#11692), and pinned zstandard==0.23.0 in the SGLang image to restore the RL metadata upload path (#11379).

Multimodal & Diffusion

• Diffusion & Omni Model Coverage: Forced eager mode in the Wan2.2 video and image-to-video launchers to bypass a CUDA illegal memory access (#9563), raised Qwen3-VL max_seq_len to 8192 restoring disaggregated multimodal coverage (#9764), and emitted one image per diffusion output instead of capping at n=1 (#9822).
• Multimodal Request Safety & Limits: Returned actionable 4xx for SSRF-blocked media URLs instead of 500 (#11360), derived the max_tokens default from the image-expanded prompt length so requests no longer overshoot context (#10757), and threaded trust_remote_code from engine settings (default False) across vLLM, SGLang, and TensorRT-LLM, closing CWE-829 (#11366).
• Multimodal Rendering & Tensor Path: Derived CHW tensor image dimensions from shape[-2:] to keep SGLang prefill/decode on the exact-token path (#11299), and flattened mixed text and image content in the native Rust renderer for verbatim-echo templates like NVIDIA-Nemotron-Parse (#10380).

Scheduling

• Router Admission & Backpressure: Gated prefill admission on overloaded-worker sets, excluded over-threshold workers across routing modes, randomized tie-breaks, and bounded event-subscriber channels to cap frontend memory (#10631, #9688, #9516).
• KV-Router Indexing & Attribution: Fixed prefill-router hash-mode mismatches, invalidated cached hashes on LoRA/multimodal metadata, and reconciled instance state during worker churn (#9857, #10500, #9631).
• Planner Load & FPM Accounting: Counted started requests for throughput load, classified decode-sweep rows in Forward-Pass Metrics, and guarded overlapping disagg scale operations (#9526, #9528, #9774).

KV Router

• KV Event & Cancellation Correctness: Stamped SGLang multinode KV events with routable leader worker IDs so remote DP ranks update Router trees, shared KV hashing via the dynamo-kv-hashing contract, attributed workers only after dispatch, and scoped cancellation to a child token (#9505, #10095, #11324, #11455).

Planner & Profiler

• Planner Scaling & SLA Correctness: Scaled down throughput-only SLA deployments after traffic stops, propagated user SLA targets into PlannerConfig, and kept Planner bound to the serving worker namespace through rollouts (#11296, #9537, #11622).
• Profiler Sweep & DGDR Fixes: Preserved DGDR served-model identity across sweeps, made parallel-config labels injective, resolved local PVC model paths (no HF refetch), and normalized SGLang/vLLM/TRT-LLM CUDA-graph arguments (#11196, #10086, #9877).
• AIConfigurator Version Fixes: Capped aiconfigurator below v0.10 to restore the v0.9 API the profiler targets and removed the deprecated profiler WebApp and its Gradio dependency (#11732, #11361).

Performance Modeling & Replay

• Replay & Mocker Correctness: Preserved zero-duration completions and single-rank KV capture in disagg replay, fixed cached chunked-prefill accounting, gated mocker admission on physically available KV, and hardened KVBM offload simulation to fail fast on invalid states (#9771, #10919, #10439, #11012).

Kubernetes Deployment

• Inference Gateway & Service Mesh: Switched the GAIE install and validation flow from kgateway to agentgateway (#7607), fixed EPP DestinationRule generation for tlsMode=MUTUAL (#10068), and added serviceMesh.enabled to disable Istio where the DestinationRule API is absent (#11344).
• GMS, DRA & Snapshot/CRIU Restore: Gated GMS ResourceClaimTemplate sync on DRA v1 and derived snapshot-restore CUDA order from nvidia-smi (#9454, #11325) and related fixes.
• DGD/DGDR/DCD Admission & CRD Versioning: Ported DynamoGraphDeployment webhooks to v1beta1 with legacy v1alpha1 conversion and moved CRDs into a crd-apply init container to fit the 1 MiB Secret limit (#10934, #11689) and related fixes.
• DGDR & Rollout Lifecycle: Surfaced specific DGDR failure reasons, fixed a profiling race that stranded DGDRs in Deploying, and preserved availability during overlapping worker rollouts (#8227, #11340, #10184) and related fixes.
• Operator Config, Secrets, Volumes & Services: Fixed spurious rollouts from non-deterministic pull-secret discovery, restored the LWS <dcd-name>-0 service name for worker DNS, and gated vLLM MP-init to the multiprocessing backend (#9826, #9612) and related fixes.

Fault Tolerance & Observability

• Frontend Metrics & Shutdown Hardening: Normalized model metric label casing, collapsed unregistered names to a bounded unknown_model sentinel, added a configurable graceful-shutdown drain timeout, and migrated Grafana dashboards to the replacement request gauges (#9775, #9836, #10705, #11690).

vLLM

• Config & Correctness Fixes: Stopped overwriting the user --runner flag that broke embedding models, removed per-run torch-compile cache regeneration, and aligned the vLLM-Omni pin to fix an import-time crash that left workers unregistered (#9710, #11374).
• Embedding & Multimodal Fixes: Fixed the text-embedding worker to return one pooled, L2-normalized vector instead of the full per-token matrix, and backported a deepstack fix that crashed EngineCore on disaggregated multimodal Qwen3-VL requests (#10248, #9491).

SGLang

• Correctness & Routing Fixes: Forwarded dropped OpenAI sampling controls that caused degenerate token repetition, and advertised the global data-parallel rank range so distributed routing targets the correct ranks (#10310, #10196).
• Stability & Backend Fixes: Fixed an asyncio Future/Task leak in the cancellation monitor causing linear RSS growth and TTFT spikes under load, and resolved NIXL libraries from nixl_cu13 on cuda13 images (#10082, #10325).

TensorRT-LLM

• Crash & Compatibility Fixes: Skipped unused NIXL connector creation in aggregated mode, and capped mpmath<1.4 to prevent an import-time crash from a missing sympy symbol (#9501, #11404).
• Performance & Telemetry Fixes: Enabled block reuse and CUDA graphs in the qwen2-vl-7b example to close a multimodal TTFT regression, and replaced multi-megabyte KV event dumps with compact summaries plus Prometheus telemetry (#9942, #10298).

Infrastructure Modernization

• Request-Plane & Discovery Hardening: Rejected oversized NATS payloads with a sanitized InvalidArgument, aborted the TCP writer when its reader fails to end a connection-monitor hang, and gave each publisher a fork-safe ID to stop discovery-key collisions from dropping subscribers (#9769, #9716, #11311).

Hardware

• Intel XPU & AMD ROCm Import Compatibility: Added a MultiModalUUIDDict import fallback for XPU builds on vLLM v0.16.0, and fixed ROCm import failures on Python 3.10 by deferring the nixl_connect import (#8106, #9929).

KV Block Manager

• Leak & Lifecycle Fixes: Added a prefill-drain hook to prevent a decode-side use-after-free during NIXL transfers (#9937).
• Compatibility & Accounting Fixes: Defaulted NIXL install extras to nixl[cu13], and normalized live KV metrics to per-data-parallel-rank units (#10413, #9932).

General

• Model Loading & Storage Fixes: Made register_model/hub::from_hf cache-first, and prepended the hidden NIXL wheel directory to LD_LIBRARY_PATH so the frontend loads libnixl.so (#10102, #9911).
• Concurrency & Store Consistency Fixes: Centralized get_lora_manager() behind a thread-safe OnceLock to stop duplicate LoRAManager construction, and fixed FileStore to honor revision = 0 as create-if-absent instead of overwriting existing files (#10110, #10968).

Documentation

• Parser & Tool-Calling Docs: Reorganized tool-call and reasoning parser docs into a single Tool Calling section with a top-level Parser Configuration reference page, split Dynamo-native from engine-fallback paths, and added a logprobs-based troubleshooting guide (#10236, #9400, #9658) and related updates.
• Recipes & Feature Benchmarks: Added Recipes and Feature Benchmarks doc surfaces with model-family filters and deployable targets, a DynoSim simulation section, and Nemotron-3-Ultra NIM Turbo recipes (#10628, #10156, #10303) and related updates.
• Deployment & Configuration Docs: Documented Standalone and Gateway (GAIE) deployment modes, consolidated EKS/GKE/AKS/ECS guides under cloud-providers/, and expanded DGDR, Planner, and router configuration and KV-transfer guides (#10059, #10348, #9428, #10123) and related updates.

Looking Ahead

The following is planned for the next release, v1.4.0 (targeted 2026-08-12). See the public Dynamo roadmap for the full plan.

Self-Tuning Planner & Live Autoscaling

Dynamo Planner moves from configured scaling to live, self-tuning scaling. Forward-Pass Metrics streamed from the vLLM, SGLang, and TensorRT-LLM engines feed an AIConfigurator performance model that the Planner continuously fine-tunes against real traffic, so scaling decisions track your actual workload rather than a static profile. This release line adds bring-your-own-trace warmup from production traffic replay, a self-tuning remote-prefill policy that shifts local-versus-remote prefill live from prefill-worker load, and a plugin interface for custom scaling logic such as cost-aware or spot-instance strategies.

Fault Tolerance & Fast Recovery

Production resiliency hardens across the stack. GPU Memory Service targets sub-five-second recovery for failed workers through shadow failover backed by a warm compile cache, CRIU-based GPU process snapshots make multi-GPU checkpoint and restore practical, and in-flight requests migrate or cancel cleanly under SLA pressure instead of failing outright. That foundation extends toward WideEP fail-continue, where a single GPU failure lets the healthy ranks keep serving while a replacement rejoins at rank granularity.

Voice & Multimodal Pipelining

Dynamo extends beyond text to orchestrate multi-model pipelines as a single request. Transparent pipelining chains stages such as speech-to-text, an LLM, and text-to-speech (STT → LLM → TTS) into one logical call, with Dynamo handling scheduling, colocation, streaming, and latency budgeting across stages — so real-time voice assistants and audio-to-audio translation no longer need custom orchestration outside the inference stack. Landing alongside are use-case-optimized Omni and vision-language recipes tuned for VLM multi-turn chat and streaming audio.