> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Dynamo v1.3.0

Dynamo v1.3.0 is the 16th feature release of the open-source distributed inference platform. It delivers the largest **Dynamo Router** buildout to date (a standalone selection service, the **Branch-Sharded KV Indexer**, compressed-radix-tree hot-path speedups, and topology-aware routing), overhauls **tool-calling and reasoning** into a configurable parser layer aligned to vLLM and SGLang across the model fleet, and deepens **Kubernetes deployment** around a production **GPU Memory Service** and DynamoGraphDeployment (DGD) `v1beta1` admission. It also expands **performance modeling and replay** with AIC latency prediction and Mooncake trace replay, and moves the platform to **CUDA 13** with refreshed vLLM, SGLang, and TensorRT-LLM engines and **NIXL v1.x**.

Breaking changes and deprecations for this release are tracked on the [Deprecations](/dynamo/dev/reference/releases/deprecations#v130) ledger; known issues on the [Known Issues](/dynamo/dev/reference/releases/known-issues#v130) page. Key dependency pins live on [Compatibility](/dynamo/dev/reference/compatibility); shipped artifacts on [Release Artifacts](/dynamo/dev/reference/release-artifacts). Model early access builds (`vX.Y.Z-<model>-dev.N`) are tracked in [Model Early Access Builds](/dynamo/dev/reference/model-early-access-builds).

## Get v1.3.0

Pull, deploy, and install with every artifact pinned to the v1.3.0 release set.

## Highlights

## Features & Improvements

### Frontend

* **GPT-OSS Harmony Parser Fixes:** Corrected the Harmony parser to preserve analysis text as normal content when reasoning parsing is unconfigured ([#9729](https://github.com/ai-dynamo/dynamo/pull/9729)), enabled GPT-OSS tool calling from Codex by retaining the `<|call|>` terminator and resolving Jinja `.items()` schema conflicts ([#9778](https://github.com/ai-dynamo/dynamo/pull/9778)), corrected thinking behavior ([#9844](https://github.com/ai-dynamo/dynamo/pull/9844)), handled bare Harmony stream markers starting directly at the commentary channel ([#9897](https://github.com/ai-dynamo/dynamo/pull/9897)), and routed Harmony reasoning, visible text, and tool-call handoff into the correct Dynamo fields by channel and recipient ([#10054](https://github.com/ai-dynamo/dynamo/pull/10054)). Preserved the directed commentary tool-call handoff to the downstream Harmony tool parser ([#10111](https://github.com/ai-dynamo/dynamo/pull/10111)), and recovered tool calls emitted on the analysis channel using the recipient as the definitive signal ([#10366](https://github.com/ai-dynamo/dynamo/pull/10366)).
* **Experimental Parsers V2 Routing:** Routed Qwen3-Coder and DeepSeek-V4 tool calls through the dynamo-parsers-v2 streaming parser behind a new `DYN_ENABLE_EXPERIMENTAL_PARSERS_V2` flag (off by default); the v2 parser owns incremental tool-call emission and drops parameter values truncated at EOF ([#10853](https://github.com/ai-dynamo/dynamo/pull/10853)). Added reasoning-parser parity cases for the "1 open + 2 close" pattern and fixed BasicReasoningParser and Gemma4ReasoningParser to keep stray close markers out of normal\_text ([#10250](https://github.com/ai-dynamo/dynamo/pull/10250)).
* **Structural Tag Tool Calls:** Added automatic xgrammar structural tag generation for tool-calling requests, constraining decoding in each model's native tool-call format to enforce `required`/`named` tool choice, strict function schemas, `parallel_tool_calls=false`, and `tool_choice="none"` token bans. Disabled by default, configurable via `--dyn-enable-structural-tag`, supporting the hermes, qwen3\_coder, deepseek\_v3\_2, and deepseek\_v4 parsers ([#9711](https://github.com/ai-dynamo/dynamo/pull/9711)).
* **Skip Special Tokens Warning:** Added a one-time diagnostic warning in the OpenAIPreprocessor when a request forces `skip_special_tokens=true` while a special-token-dependent parser is active, a combination that silently strips parser markers and leaks tool-call or reasoning markup into content ([#10225](https://github.com/ai-dynamo/dynamo/pull/10225)).
* **Realtime API Endpoint Build-Out:** Wired the `/v1/realtime` WebSocket endpoint through ModelManager with a first-class `ModelType::Realtime` endpoint kind and bidirectional PushRouter transport ([#9308](https://github.com/ai-dynamo/dynamo/pull/9308)), and replaced the placeholder chat-completion shape with OpenAI Realtime API event types, exchanging `RealtimeClientEvent` and `RealtimeServerEvent` frames and synthesizing a spec-compliant `session.created` event on connect ([#9205](https://github.com/ai-dynamo/dynamo/pull/9205)).
* **OpenAI Embeddings Format Support:** Added `dimensions` Matryoshka truncation to the vLLM `/v1/embeddings` worker, validating and slicing each embedding ([#9751](https://github.com/ai-dynamo/dynamo/pull/9751)), then added end-to-end `encoding_format=base64` support: the Rust frontend accepts array and base64 shapes and both SGLang and vLLM backends emit the requested form ([#9887](https://github.com/ai-dynamo/dynamo/pull/9887)).
* **MiniMax M3 Frontend Support:** Added MiniMax M3 support to the OpenAI frontend: `thinking`/`thinking_mode` handling for `enabled`, `disabled`, and `adaptive` modes, parser alias normalization, special-token and trailing EOS handling, prior tool-call argument validation, and MiniMax-M3-VL multimodal context-length detection ([#10983](https://github.com/ai-dynamo/dynamo/pull/10983)).
* **Unified Backend Logits and LogProbs:** Added `dynamo.common.backend.logprobs` as the single source of truth for logprob option parsing and per-chunk extraction, wiring logprobs end-to-end on the unified path so `output_options` reaches `sampling_params` and chunks carry `log_probs` and `top_logprobs` keys ([#10149](https://github.com/ai-dynamo/dynamo/pull/10149)), and brought the custom logits-processor smoke path and shared serializable wire format to the unified vLLM and SGLang backends ([#10224](https://github.com/ai-dynamo/dynamo/pull/10224)).
* **Frontend nvext Switches and Relocation:** Added frontend master switches for the `nvext` and admin-API surfaces, letting operators close off non-OpenAI-spec surfaces on the frontend HTTP service without affecting inference, metrics, models, or health probes (both on by default) ([#10556](https://github.com/ai-dynamo/dynamo/pull/10556)), and moved typed `NvExt` handling out of the OpenAI module into the LLM common extension layer, keeping `dynamo-protocols` extension-agnostic and adding Anthropic Messages ingress support for `nvext.agent_context` ([#10784](https://github.com/ai-dynamo/dynamo/pull/10784)).
* **Dynamo Routing Header Standardization:** Added `x-dynamo-request-priority` and `x-dynamo-request-strict-priority` HTTP headers as alternatives to `nvext.agent_hints`, letting OpenAI and Anthropic clients set scheduling priority without modifying request bodies, behind `DYN_DISABLE_FRONTEND_NVEXT` ([#10871](https://github.com/ai-dynamo/dynamo/pull/10871)), then standardized every Dynamo-owned public routing header on the `x-dynamo-*` namespace while keeping the original unprefixed worker and DP-rank names as compatibility aliases across EPP, ext-proc, frontend, tests, and docs ([#10873](https://github.com/ai-dynamo/dynamo/pull/10873)).
* **HTTP Header Metadata Propagation:** Extracted HTTP request headers matching a configurable prefix into the context metadata map, so tenants and routing systems can inject key-value metadata at the HTTP boundary and flow it end-to-end across HTTP, streaming, and gRPC endpoints. Oversized or invalid header metadata now returns a clear "request headers too large" error ([#9726](https://github.com/ai-dynamo/dynamo/pull/9726)).
* **LoRA Controller and Routing:** Added a LoRA controller with HRW/MCF placement, filtered request routing, metrics, discovery, and configuration wiring across the vLLM and SGLang backends. Preserved session affinity when LoRA is disabled, rejected incompatible direct and advanced routes, and returned a resource-exhausted response when every worker in a selected replica set is saturated ([#8180](https://github.com/ai-dynamo/dynamo/pull/8180)).
* **Admission Control Flag Replacement:** Replaced the boolean `--no-admission-control` opt-out flag with an enum-valued `--admission-control {token-capacity,none}` flag defaulting to `none`, so operators must now explicitly opt in to busy-worker rejection. `DYN_NO_ADMISSION_CONTROL` gave way to `DYN_ADMISSION_CONTROL` ([#9694](https://github.com/ai-dynamo/dynamo/pull/9694)).
* **Configurable Overload Status Code:** Added the `DYN_HTTP_OVERLOAD_STATUS_CODE` environment variable to set the frontend's HTTP status code for overload rejections, with the default `529` unchanged. Deployments behind proxies or clients that only understand `503` can now select that code without a code change ([#11420](https://github.com/ai-dynamo/dynamo/pull/11420)).
* **L1 Tokenizer Prefix Cache:** Added an opt-in `CachedTokenizer` that records prefix tokenizations at special-token boundaries and re-encodes only the diverging suffix, enabled per-frontend via `DYN_TOKENIZER_CACHE=1` with a configurable budget and `/metrics` hit and miss counters ([#9742](https://github.com/ai-dynamo/dynamo/pull/9742)), then extended it with partial-hit handling for growing multi-turn conversations, a moka W-TinyLFU backend, single-pass Aho-Corasick boundary detection, and broadened tests across tokenizer families ([#10201](https://github.com/ai-dynamo/dynamo/pull/10201)).
* **Streaming Hot Path Optimization:** Optimized the OpenAI chat SSE streaming path by replacing the per-chunk flat\_map to Vec to stream::iter plumbing with a single async\_stream adapter that reuses event storage and serializes JSON directly ([#10433](https://github.com/ai-dynamo/dynamo/pull/10433)), applied the same rework to the Responses streaming adapter so it owns the converter and serializes events with borrowed Serialize wrappers ([#10498](https://github.com/ai-dynamo/dynamo/pull/10498)), and to the Anthropic Messages streaming adapter to drain a reusable event buffer from a single async stream while preserving error handling, disconnect monitoring, and keepalive behavior ([#10499](https://github.com/ai-dynamo/dynamo/pull/10499)).
* **Frontend Hot-Path Optimizations:** Applied three profile-guided optimizations to the frontend hot path: returning the shared `Arc` instead of cloning the cached prefix on a tokenizer L1 cache hit, caching per-model Prometheus metric handles in `ResponseMetricCollector` rather than resolving them per chunk and per token, and capping the `log` crate at compile-time `debug` so per-character `trace!` events from the HF tokenizers crate compile to no-ops in release. Stock-tokenizer-path throughput improved roughly 1.45x, from about 189 to about 275 requests per second ([#10273](https://github.com/ai-dynamo/dynamo/pull/10273)).
* **Self-Host Sibling Metadata Harvesting:** Harvested non-weight sibling files (preprocessor\_config.json, special\_tokens\_map.json, added\_tokens.json, tokenizer.model) into slug\_dir at the end of resolve\_metadata\_files to fix KV routing regressions for from\_pretrained consumers ([#9610](https://github.com/ai-dynamo/dynamo/pull/9610)), had the worker advertise non-typed siblings through a new ModelDeploymentCard.extra\_files field for custom directories without an HF source ([#9707](https://github.com/ai-dynamo/dynamo/pull/9707)), made the vLLM and SGLang native preprocessors consume the resolved slug\_dir instead of re-running fetch\_model ([#10037](https://github.com/ai-dynamo/dynamo/pull/10037)), and had the MetadataArtifactRegistry auto-clean entries on detach to stop leaks on LoRA detach or model reload ([#10351](https://github.com/ai-dynamo/dynamo/pull/10351)).
* **Disagg Processor Engine Support:** Routed the vLLM preprocessor through a new Rust RoutedEngine that wraps Client or KV router with PrefillRouter for disaggregated serving ([#9503](https://github.com/ai-dynamo/dynamo/pull/9503)), extended the same path to the SGLang processor and added structural\_tag for guided decoding on the Rust side ([#9577](https://github.com/ai-dynamo/dynamo/pull/9577)), and added request migration failover to the vLLM and SGLang processor paths ([#9617](https://github.com/ai-dynamo/dynamo/pull/9617)).
* **Disaggregated Topology Readiness:** Migrated the worker set key to include `worker_type`, removed the legacy `ModelType::Prefill` bit, registered encode workers, and added per-model topology gating in the frontend ([#9815](https://github.com/ai-dynamo/dynamo/pull/9815)), plus a read-only `GET /v1/models/{model}/ready` endpoint exposing structured per-namespace worker readiness for a model ([#10383](https://github.com/ai-dynamo/dynamo/pull/10383)).
* **Worker Type Topology Scaffolding:** Added a `WorkerType` enum with `Prefill`, `Decode`, `Encode`, and `Aggregated` roles, plus `worker_type` and `needs` fields on the model deployment card and live readiness methods on `Model`, a first step toward making the frontend aware of disaggregated serving topology before serving requests ([#8626](https://github.com/ai-dynamo/dynamo/pull/8626)).
* **Python-Rust PyO3 Bridge:** Exposed the runtime's bidirectional engine path to Python so an async generate worker can be registered and driven end-to-end via PyAsyncRequestStream and serve\_bidirectional\_endpoint ([#10135](https://github.com/ai-dynamo/dynamo/pull/10135)), propagated HTTP-status-carrying exceptions through the boundary so unsupported image formats return their intended status instead of HTTP 500 ([#10364](https://github.com/ai-dynamo/dynamo/pull/10364)), and removed the cyclic Python to env to Rust config side channels in the frontend by passing resolved config through the PyO3 contract as explicit Rust fields ([#10446](https://github.com/ai-dynamo/dynamo/pull/10446)).

### Agents

* **Agent Request Trace Capture:** Added agent traces to the v1/completions endpoint and refactored the internal streaming trace infrastructure ([#9125](https://github.com/ai-dynamo/dynamo/pull/9125)), plus optional finish\_reason\_metadata recording backend stop metadata, final chat and completions finish metadata, and tool-call metadata without storing arguments, surfaced in the Perfetto conversion ([#9817](https://github.com/ai-dynamo/dynamo/pull/9817)).
* **Header-Only Agent Trajectory Identity:** Added a deterministic program-close signal via `nvext.agent_context.trajectory_final`, with a symmetric resume INFO log and scheduler docs for `dynamo.thunderagent_router` ([#10172](https://github.com/ai-dynamo/dynamo/pull/10172)), and mapped trajectory identity into SGLang as a radix-cache session, normalizing an explicit final header into `kv_hints.evict_trajectory` ([#10214](https://github.com/ai-dynamo/dynamo/pull/10214)). Made `trajectory_id` the only required identity field and added the generic `x-dynamo-trajectory-id` HTTP fallback ([#10800](https://github.com/ai-dynamo/dynamo/pull/10800)), removed `nvext.agent_context` from public request bodies for a header-only contract parsed at the HTTP boundary ([#10808](https://github.com/ai-dynamo/dynamo/pull/10808)), and preserved the immediate Claude parent for nested subagents via `x-claude-code-parent-agent-id` ([#10942](https://github.com/ai-dynamo/dynamo/pull/10942)).

### Reinforcement Learning

* **RL Tokens-In-Tokens-Out Support:** Added the Rust `lib/llm` nvext Tokens-in-Tokens-Out protocol, preprocessor, and response plumbing, carrying `nvext.token_data`, `cache_salt`, `extra_fields`, token constraints, `prompt_logprobs`, detokenize, and stop token IDs through the request and response path, validating passthrough `cache_salt` and `stop_token_ids` while rejecting unsupported `truncate_prompt_tokens` ([#9649](https://github.com/ai-dynamo/dynamo/pull/9649)), plus a read-only `GET /v1/rl/workers` discovery endpoint in the new `dynamo-rl` crate gated behind `DYN_ENABLE_RL` and `DYN_RL_PORT` ([#9681](https://github.com/ai-dynamo/dynamo/pull/9681)).
* **vLLM RL Token-In/Token-Out Support:** Added vLLM worker-side support for nvext Tokens-in-Tokens-Out requests, forwarding RL/TITO flags through the CLI and runtime config, preserving cache salt on decode and disaggregated prefill prompts, and exposing completion token IDs, logprobs, and engine data parity fields ([#9651](https://github.com/ai-dynamo/dynamo/pull/9651)), registered worker RL admin routes under `/engine/<route>` with LoRA request validation ([#9680](https://github.com/ai-dynamo/dynamo/pull/9680)), and added NeMo-RL NCCL weight updates ([#11034](https://github.com/ai-dynamo/dynamo/pull/11034)).
* **SGLang RL Metadata Uploads:** Added an opt-in path to upload large reinforcement-learning serving metadata (routed expert captures, top-logprobs, tensors, NumPy arrays) as zstd-compressed msgpack objects through fsspec storage backends. Gated by SGLang RL mode and a per-request `nvext.metadata_upload.url` field, so callers can scope rollout metadata to any installed fsspec target ([#10034](https://github.com/ai-dynamo/dynamo/pull/10034)).

### Multimodal & Diffusion

* **Multimodal KV Routing Support:** Added MM-aware KV routing on the SGLang backend via per-image pad\_value substitution with auto-detected backend identity ([#9561](https://github.com/ai-dynamo/dynamo/pull/9561)), tightened post-merge tests to fail closed when routing degrades and fixed the Qwen2-VL, Qwen2.5-VL, and fastokens paths ([#9441](https://github.com/ai-dynamo/dynamo/pull/9441)), dropped Phi-3 from MM-aware routing since its sliding-window KV events are not admitted to the contiguous-prefix index ([#10441](https://github.com/ai-dynamo/dynamo/pull/10441)), and added the Qwen3.5/3.6 and Kimi K2 families with a tightened pre-merge gate ([#10999](https://github.com/ai-dynamo/dynamo/pull/10999)).
* **SGLang Multimodal Disaggregated Support:** Forwarded image and video inputs to both SGLang disaggregated prefill and decode workers so media is encoded and KV-aligned over NIXL instead of dropped and decoded as text ([#10643](https://github.com/ai-dynamo/dynamo/pull/10643)), normalized multimodal content shapes on the SGLang chat-processor path so chat templates bind media placeholders ([#10673](https://github.com/ai-dynamo/dynamo/pull/10673)), rejected multimodal requests when a PD worker lacks the multimodal-worker flag ([#10680](https://github.com/ai-dynamo/dynamo/pull/10680)), and unified multimodal E/P/D configuration flags across backends with deprecation warnings for the prior backend-specific flags ([#10690](https://github.com/ai-dynamo/dynamo/pull/10690)).
* **Encoder-Result Handoff Contract:** Added an engine-opaque, object-only `encoder_result` payload to `PreprocessedRequest`, `LLMEngineOutput`, and `BackendOutput`, with producer helpers in Rust and a new `dynamo.common.backend.multimodal` Python module, the foundation for multimodal Encode worker to Prefill peer handoff ([#10969](https://github.com/ai-dynamo/dynamo/pull/10969)).
* **Cache-Aware Embedding Device Routing:** Integrated EmbeddingCache awareness into the heterogeneous EPD weighted device routing so cache-hit requests are excluded from the CUDA-to-CPU ratio accounting and fall back to round-robin ([#8750](https://github.com/ai-dynamo/dynamo/pull/8750)), and added video embedding cache support to the SGLang multimodal encode worker, storing video-specific metadata and config-aware video cache keys while preserving image cache behavior ([#8298](https://github.com/ai-dynamo/dynamo/pull/8298)).
* **OpenAI Embeddings Wire Performance:** Added the OpenAI `/v1/embeddings` `dimensions` field with Matryoshka truncation in the SGLang handler ([#9722](https://github.com/ai-dynamo/dynamo/pull/9722)), submitted batched prompts concurrently via `asyncio.gather` so vLLM's scheduler coalesces them into a single forward pass ([#10117](https://github.com/ai-dynamo/dynamo/pull/10117)), and switched the internal worker-to-frontend wire format to base64-encoded bytes, decoded back to floats at the HTTP boundary, dropping latency at `--batch-size 15` from 215 ms to 124 ms on arm64 GB200 ([#10139](https://github.com/ai-dynamo/dynamo/pull/10139)).
* **Diffusion Unified Backend Onboarding:** Onboarded diffusion image and video generation onto the unified backend, so raw-media engines for TensorRT-LLM and SGLang share the same Worker lifecycle as token LLM engines: signal handling, graceful drain, 3-phase shutdown, and health checks ([#10371](https://github.com/ai-dynamo/dynamo/pull/10371)).

### Scheduling

#### KV Router

* **Branch-Sharded KV Indexer:** Added the `AsyncShardHandle` trait and generalized `BranchShardedIndexer` over in-process and future remote shard backends ([#9763](https://github.com/ai-dynamo/dynamo/pull/9763)); removed the scheduler hop from in-process anchored reads ([#10200](https://github.com/ai-dynamo/dynamo/pull/10200)); added O(1) anchor cleanup and worker-id lookup via per-worker indexes ([#10302](https://github.com/ai-dynamo/dynamo/pull/10302)); and borrowed shard-read suffix slices instead of per-lookup allocations ([#10495](https://github.com/ai-dynamo/dynamo/pull/10495)).
* **Standalone KV-Router Selection Service:** Added a runtime-independent slot-tracker microservice with HTTP worker registration, hash-based lifecycle accounting, and advisory load reads ([#10291](https://github.com/ai-dynamo/dynamo/pull/10291)), plus optional ZMQ replica sync for lifecycle events and dynamic peer management ([#10394](https://github.com/ai-dynamo/dynamo/pull/10394)). Added a runtime-free selection service with worker-catalog reconciliation, selection and reservation APIs, and scoring endpoints ([#10641](https://github.com/ai-dynamo/dynamo/pull/10641)), extended it with startup recovery from HTTP indexer peers, best-effort replica sync, and Mooncake-style token-overlap summaries ([#10745](https://github.com/ai-dynamo/dynamo/pull/10745)), and exposed the selection core as a feature-gated Python class for in-process use ([#10766](https://github.com/ai-dynamo/dynamo/pull/10766)).
* **Router Overlap Score Handling:** Added a KvRouter overlap-scores API and standalone router endpoint returning per-worker tiered KV overlap with optional shared-cache hit context ([#9538](https://github.com/ai-dynamo/dynamo/pull/9538)); refreshed overlap scores at dequeue time for requests whose scheduler queue wait exceeds a threshold so stale scores no longer misroute workers ([#9663](https://github.com/ai-dynamo/dynamo/pull/9663)); and added opt-in decay of device-local overlap credit under active prefill load, defaulting to off and configurable through Rust, Python, CLI, and environment ([#10574](https://github.com/ai-dynamo/dynamo/pull/10574)).
* **Topology-Aware Routing Constraints:** Added worker taints and request-side `RoutingConstraints` for KV router selection, enforcing `required_taints` and biasing traffic with `preferred_taints` ([#9558](https://github.com/ai-dynamo/dynamo/pull/9558)); added typed topology metadata and KV-transfer enforcement fields in `ModelRuntimeConfig` plus canonical topology-taint generation ([#9767](https://github.com/ai-dynamo/dynamo/pull/9767)); and propagated topology constraints to decode from the selected prefill worker, failing closed only for required KV-transfer topology policy when prefill-worker attribution is unavailable ([#9893](https://github.com/ai-dynamo/dynamo/pull/9893)).
* **Global Router SLA Retries:** Added priority-retry configuration for global-router pool failures, retrying prefill, decode, and aggregated requests on faster pools before streaming starts, with documented pool priorities, defaults, and unit coverage ([#9460](https://github.com/ai-dynamo/dynamo/pull/9460)), and moved per-request target TTFT and ITL fields into a schema-checked `router` sub-field under `nvext` ([#9845](https://github.com/ai-dynamo/dynamo/pull/9845)).
* **KV-Router Recovery Hardening:** Added a bounded local pending-live-event buffer for KV-router recovery and split the local-indexer recovery code into focused modules ([#9881](https://github.com/ai-dynamo/dynamo/pull/9881)); added batched replica-sync side effects, cancellation-aware push routing, and ghost-booking prevention ([#10331](https://github.com/ai-dynamo/dynamo/pull/10331)); and added per-rank cancellation tokens that abort in-flight recovery when a worker is removed ([#10616](https://github.com/ai-dynamo/dynamo/pull/10616)).
* **Lower-Tier Cache-Hit Scoring:** Fixed the KV-router lower-tier walk to seed `query_lower_tiers` from primary-only `MatchDetails`, then merge side-indexer scores into the device output, so side-only worker-block credit is no longer dropped from host and disk cache-hit math ([#9797](https://github.com/ai-dynamo/dynamo/pull/9797)); exposed `host_cache_hit_weight` and `disk_cache_hit_weight` through the `dynamo.frontend` and `dynamo.router` CLI and `DYN_ROUTER_*` env vars at existing defaults of 0.75 and 0.25 ([#10157](https://github.com/ai-dynamo/dynamo/pull/10157)); and routed vLLM native CPU-offload KV events to the HostPinned tier with lower-tier applies counted in `kv_cache_events_applied` ([#10368](https://github.com/ai-dynamo/dynamo/pull/10368)).
* **KV-Router Hot-Path Optimizations:** Reduced block-hash cloning in router-lookup and route-time recording paths ([#9960](https://github.com/ai-dynamo/dynamo/pull/9960)); replaced per-block approximate-prune heap entries with 100ms expiry buckets for a 13.7% throughput gain on the bs64 c384 multi-frontend AgentX benchmark ([#10521](https://github.com/ai-dynamo/dynamo/pull/10521)); and reused caller-owned child-traversal storage during CRTC lookup repair while skipping redundant same-node worker updates ([#10540](https://github.com/ai-dynamo/dynamo/pull/10540)). Changed compressed-edge remove bookkeeping to return only the newly uncovered hash range for a \~28x speedup in store/remove event processing ([#10676](https://github.com/ai-dynamo/dynamo/pull/10676)), and batched CRTC remove handling for hashes on the same compressed edge with a split-safe per-hash fallback ([#10858](https://github.com/ai-dynamo/dynamo/pull/10858)).
* **Active Sequence Load Tracking:** Optimized scheduler load projection by replacing the prompt registry's `DashMap` load table with a dense worker load table and updating `active_sequences_bench` to measure the production `project_worker_loads` path ([#10935](https://github.com/ai-dynamo/dynamo/pull/10935)); added incremental worker-overload tracking in `KvWorkerMonitor` via a dedicated `OverloadedWorkerTracker` that updates only the touched worker per `ActiveLoad` event ([#10645](https://github.com/ai-dynamo/dynamo/pull/10645)); and reduced hot-path locking by switching per-worker load entries to `SeqLock` and the worker load table to `IndexMap` ([#10967](https://github.com/ai-dynamo/dynamo/pull/10967)).
* **DP-Rank Session Affinity:** Promoted sticky-session affinity from worker to `(worker, dp_rank)` granularity so a session pinned to a multi-DP-rank engine stayed on the rank where its prefix was warm ([#9920](https://github.com/ai-dynamo/dynamo/pull/9920)); added header-based session identity via `X-Dynamo-Session-ID` with Claude Code, Codex, and OpenCode fallbacks, router-local atomic first-dispatch affinity with idle expiry and active-request leases, and an independent `--router-session-affinity-ttl-secs` setting ([#10875](https://github.com/ai-dynamo/dynamo/pull/10875)).
* **Unified Routing Log Output:** Standardized worker selection across all router modes into one structured info-level "Selected worker" log with consistent fields ([#10554](https://github.com/ai-dynamo/dynamo/pull/10554)), and reclassified prefill capacity rejections from error to warn so routine load-shed backpressure is no longer logged as a failure ([#10561](https://github.com/ai-dynamo/dynamo/pull/10561)).
* **Strict Priority Tiers in KV-Router:** Added `nvext.agent_hints.strict_priority` as a per-request pending-queue tier ordering higher values before lower, preserving FCFS, LCFS, and WSPT as secondary ordering within each tier and propagating the hint through push and picker routing, standalone requests, EPP, C JSON routing, and Python `KvRouter.best_worker` ([#10638](https://github.com/ai-dynamo/dynamo/pull/10638)), and fixed the `strict_priority` call sites in the selection service that broke the build on main ([#10689](https://github.com/ai-dynamo/dynamo/pull/10689)).
* **Tiered Router Queue Backpressure:** Added a configurable KV Router queue-depth backpressure option that caps per-worker queue depth by request cost (missing prefill tokens) and returns immediate rejections when caps are exceeded, shedding expensive low-cache-hit requests first under load. A new Prometheus counter tracks router backpressure by worker type and reason ([#8144](https://github.com/ai-dynamo/dynamo/pull/8144)).
* **Min-Cost Flow LoRA Placement:** Added a churn-minimizing LoRA-placement solver that assigns LoRA adapters to workers via a min-cost flow algorithm with top-M candidate generation, delta freezing, and a configurable cost function ([#8179](https://github.com/ai-dynamo/dynamo/pull/8179)).
* **Router Predict-On-Route Colocation:** Added `--router-predict-on-route` and `--router-predicted-ttl-secs` options so the KV Router co-locates burst-arrival sibling requests (parallel sampling, best-of-N, agent fan-out) on the worker that picked the first sibling, closing the window between the routing decision and the engine's first block-stored event. When KV events are enabled, the Router runs a secondary short-TTL approximate indexer and scores each worker with the per-worker max overlap of both trees ([#8276](https://github.com/ai-dynamo/dynamo/pull/8276)).
* **Experimental ThunderAgent Router:** Added the `thunderagent_router` service that wraps the native KvRouter with program-level pause/resume scheduling for agentic workloads, applying per-worker capacity accounting and request-boundary admission control. It is experimental and opt-in; requests lacking `nvext.agent_context.trajectory_id` pass through unchanged ([#9448](https://github.com/ai-dynamo/dynamo/pull/9448)).
* **KV-Aware Routing Unification:** Added KV event and metrics source descriptors to the unified backend abstraction so engines declare publishers through a shared interface, with the Rust Worker owning publisher construction, the metrics poll loop, and shutdown ordering. Migrated vLLM, SGLang, and TensorRT-LLM onto one component naming convention for KV-aware routing ([#9493](https://github.com/ai-dynamo/dynamo/pull/9493)).
* **Stable Worker Routing ID:** Added a `stable_routing_id` field on `ModelRuntimeConfig` that survives process restarts, sourced from the `DYN_STABLE_ROUTING_ID` or `HOSTNAME` environment variables and published through the worker's existing etcd discovery entry, giving caching layers a persistent worker identity so Kubernetes StatefulSet pod restarts no longer trigger cache rebalancing and cold-start misses ([#9665](https://github.com/ai-dynamo/dynamo/pull/9665)).
* **Workers Endpoint Filtering:** Enriched the GET /workers response with model name, tenant ID, and block size fields, added optional model\_name and tenant\_id query parameters for filtering, and fixed a block\_size=0 time-of-check-to-time-of-use race by skipping mid-deregistration workers ([#9983](https://github.com/ai-dynamo/dynamo/pull/9983)).
* **Optional Binary-Search Indexer Mode:** Added an optional binary-search algorithm to the KV Router's `PositionalIndexer::find_matches`, selectable via the `DYN_ROUTER_POSITIONAL_SEARCH_MODE` environment variable or API, cutting index probes for long query-block lengths while producing byte-identical results to the default strided mode ([#10181](https://github.com/ai-dynamo/dynamo/pull/10181)).
* **Standalone Router Timing Propagation:** Forwarded per-request timing from a standalone KV-router to the frontend on the data payload, so `prefill_time_ms`, `ttft_ms`, `prefill_wait_time_ms`, `kv_hit_rate`, and `queue_depth` reach the nvext `timing` field, Prometheus metrics, and agent traces in split deployments ([#10182](https://github.com/ai-dynamo/dynamo/pull/10182)).
* **Modeled Prefill Time Reads:** Added a Rust-internal slot-tracker read for modeled remaining prefill time in the KV Router, exposing per-worker projected prefill consumption in signed milliseconds for load visibility while keeping unmodeled workers on the fast error path ([#10190](https://github.com/ai-dynamo/dynamo/pull/10190)).
* **Router Load and Request APIs:** Added a potential-loads API, exposed active request counts to external APIs, and allowed markPrefill and markFree to execute without a context ID for uniform behavior, aiding cross-region balancing and detection of overloaded or queueing workers ([#10412](https://github.com/ai-dynamo/dynamo/pull/10412)).
* **KV Indexer Access Logging:** Added structured JSONL access logging to the standalone KV indexer via a `--access-log` option recording timestamp, trace-id, method, path, model, status code, and duration per request, plus `--trace-id-header` and `--access-log-local-time` flags and a `/reopen_logs` endpoint for logrotate integration, improving production observability and request tracing ([#10700](https://github.com/ai-dynamo/dynamo/pull/10700)).
* **Standalone Indexer Metrics Export:** Registered the core KvIndexerMetrics collectors in the standalone indexer's Prometheus registry, so event warnings and errors now export through `/metrics`, matching the native Dynamo path ([#10768](https://github.com/ai-dynamo/dynamo/pull/10768)).

#### Planner & Profiler

* **Configurable Prometheus Client Auth:** Made Planner's Prometheus client TLS verification configurable via `PROMETHEUS_SSL_VERIFY`, defaulting to off to preserve existing behavior ([#9510](https://github.com/ai-dynamo/dynamo/pull/9510)); added a custom CA-bundle path for private and internal roots ([#9511](https://github.com/ai-dynamo/dynamo/pull/9511)), a static bearer token ([#9512](https://github.com/ai-dynamo/dynamo/pull/9512)), and a bearer-token file re-read on every request for rotating credentials ([#9513](https://github.com/ai-dynamo/dynamo/pull/9513)); and fixed extra URL query parameters on every PromQL request for tenancy-enforcing front-ends ([#9557](https://github.com/ai-dynamo/dynamo/pull/9557)).
* **MTP Accept Length Scaling:** Added MTP accept-length scaling to the Planner, publishing backend spec-decode metadata in the ModelDeploymentCard and scraping speculative accept length from worker Prometheus metrics to discount decode ITL without altering raw OSL, KV, or context estimates, and rebuilding AIC perf models when capabilities change ([#10435](https://github.com/ai-dynamo/dynamo/pull/10435)); added replay support carrying accept length through mocker engine passes, PSM replay, orchestrator plugin metrics, and the plugin proto schema, and fixed PSM replay tick time to use the scheduled replay clock ([#10501](https://github.com/ai-dynamo/dynamo/pull/10501)).
* **Attention-DP Forward Pass Metrics:** Enabled TensorRT-LLM ForwardPassMetrics under Attention-DP by removing the Dynamo-side gate that disabled them when attention\_dp\_size exceeded one, so the Planner now receives one metrics stream per Attention-DP rank and can detect load imbalance ([#9059](https://github.com/ai-dynamo/dynamo/pull/9059)).
* **Planner Load Optimization Target:** Added an optimization\_target='load' option for the Planner with configurable prefill queue token and decode KV utilization thresholds, forcing reactive load scaling and disabling throughput scaling with a warning ([#9590](https://github.com/ai-dynamo/dynamo/pull/9590)).
* **Planner Gzip Diagnostics Logs:** Added a compressed JSONL diagnostics sidecar written next to each Planner HTML report by default, capturing per-tick snapshots; a `report_write_gzip_log` setting disables it, and writes stay best-effort so gzip failures warn instead of breaking the HTML report ([#9623](https://github.com/ai-dynamo/dynamo/pull/9623)).
* **Planner Recommended Replica Markers:** Added recommended prefill and decode replica markers to the Planner HTML diagnostics Replica Counts plot, and documented local Planner advisory mode as suggestion-only for evaluating configurations without scaling ([#9644](https://github.com/ai-dynamo/dynamo/pull/9644)).
* **SLA Target Metrics Exposure:** Added `sla_target_ttft_ms` and `sla_target_itl_ms` Gauges to the Planner's Prometheus metrics, publishing operator-configured SLA targets so Grafana dashboards can compare observed and estimated latency against the SLA boundary ([#10032](https://github.com/ai-dynamo/dynamo/pull/10032)).
* **Planner Plugin Framework Infrastructure:** Added an opt-in tick-engine path for the Planner that runs decisions through a PREDICT, PROPOSE, RECONCILE, CONSTRAIN pipeline driven by in-process or gRPC plugins, enabled per planner via `scheduling.use_orchestrator=true` while the legacy PlannerStateMachine path stays the default. This infrastructure-only change ships proto and Pydantic types, transport clients, a plugin registry and scheduler, merge algorithms, and static-config and gRPC self-register plugin registration, with builtin plugins to follow ([#10124](https://github.com/ai-dynamo/dynamo/pull/10124)).
* **Rust Engine Perf Shim:** Routed SLA Planner performance queries through a new PlannerEnginePerfModel adapter backed by the Rust engine perf model, with legacy Python regression fallback when the optional Rust shim is unavailable, and relaxed planner bootstrap requirements so missing pre-deployment data no longer blocks throughput mode ([#10229](https://github.com/ai-dynamo/dynamo/pull/10229)).

### Performance Modeling & Replay

#### Mocker & Simulation

* **KVBM Offload Replay Simulation:** Added mocker support for KVBM G3 offload simulation following the G1 to G2 to G3 tiering path with staged promotion, and a first-admission prefix-cache reused-ratio metric in replay reports ([#9337](https://github.com/ai-dynamo/dynamo/pull/9337)); modeled G4 object-storage offload as shared infinite-capacity storage with G2-to-G4 pipeline and bandwidth configs ([#9939](https://github.com/ai-dynamo/dynamo/pull/9939)); and extended the Rust-native offline-replay benchmark with aggregated and disaggregated topology selection, KVBM capacity and bandwidth options, worker-initialization error propagation, and benchmark documentation ([#11010](https://github.com/ai-dynamo/dynamo/pull/11010)).
* **AIC Latency Prediction Integration:** Added a feature-gated AIC forward-pass engine perf shim to the mocker, exposing `RustEnginePerfModel` through PyO3 ([#10150](https://github.com/ai-dynamo/dynamo/pull/10150)), and used AIC memory estimates to size KV cache blocks while preserving the legacy 16384 fallback when AIC is disabled ([#9598](https://github.com/ai-dynamo/dynamo/pull/9598)). Switched latency prediction to the AIC Rust crate as a GIL-free callback on the hot path ([#10615](https://github.com/ai-dynamo/dynamo/pull/10615)), forwarded AIC quantization-mode overrides so replay latency and capacity track quantized deployments ([#10914](https://github.com/ai-dynamo/dynamo/pull/10914)), scaled `num_gpu_blocks` by `attention_dp_size` for DP-attention KV provisioning ([#10964](https://github.com/ai-dynamo/dynamo/pull/10964)), and divided the scheduled batch by `attention_dp_size` for correct per-rank perf queries in offline replay ([#11002](https://github.com/ai-dynamo/dynamo/pull/11002)).
* **TensorRT-LLM Scheduler Simulation:** Added TensorRT-LLM scheduler simulation to the mocker via `engine_type=trtllm`, modeled as a scheduling-policy variant of the vLLM scheduler core with a GUARANTEED\_NO\_EVICT policy and AIC perf-model integration ([#10193](https://github.com/ai-dynamo/dynamo/pull/10193)), and terminal rejection of unschedulable requests whose footprint exceeds the KV pool, propagated through every replay driver via a new `rejected` flag on `OutputSignal` ([#10287](https://github.com/ai-dynamo/dynamo/pull/10287)).
* **Mocker Replay Performance Tuning:** Routed one-worker aggregated offline replay through `SingleRuntime` for vLLM, SGLang, and TRT-LLM across flat, workload, concurrency, and agentic entrypoints while retaining `AggRuntime` for multi-worker, KV-router, planner-driven, and disaggregated replay, and added cross-runtime parity coverage and an `--engine-type` option to `offline_replay_bench` ([#10629](https://github.com/ai-dynamo/dynamo/pull/10629)); and reduced scheduler and FPM publish overhead by skipping timerfd construction for consumed deadlines, reusing the publisher task's MessagePack buffer, and caching the event-plane subject ([#11025](https://github.com/ai-dynamo/dynamo/pull/11025)).
* **Eagle Speculative-Decoding Perf Model:** Added MTP/Eagle speculative-decoding support to the AIC perf model used by mocker and replay, exposing the `aic_nextn` draft-token count and `aic_nextn_accept_rates` per-position accept rates through the CLI, Python bindings, and mocker serde so AIC applies the speculative-decode decode speedup during latency modeling ([#10197](https://github.com/ai-dynamo/dynamo/pull/10197)).
* **MTP Burst Sampling:** Added speculative-decoding (Multi-Token Prediction) support to the mocker engines with conditional acceptance sampling and atomic burst reservations, exposed through new options `--aic-nextn`, `--aic-nextn-accept-rates`, and `--aic-mtp-seed` across live and replay paths ([#10436](https://github.com/ai-dynamo/dynamo/pull/10436)).
* **Atomic Source Hold Primitive:** Added a scheduler-owned source-hold registry with a stable HandoffId to the mocker, letting vLLM and SGLang retain completed request state and defer terminal cleanup so release and cancel clean up exactly once, even for early or duplicate commands ([#10831](https://github.com/ai-dynamo/dynamo/pull/10831)).
* **Disaggregated Handoff Lifecycle:** Replaced the mocker's synthetic second-request approximation of disaggregated serving with one explicit handoff lifecycle covering source hold, destination reservation, transfer, activation, release, completion, cancellation, and failure, shared across offline replay and live bootstrap. It preserves vLLM source-first and SGLang destination-first admission semantics and surfaced latent fixes in virtual-time visibility, offload wakeups, cancellation cleanup, worker retirement, and queue accounting ([#10915](https://github.com/ai-dynamo/dynamo/pull/10915)).
* **Unified KV-Cache Estimator:** Routed the mocker and replay `num_gpu_blocks` estimator through aiconfigurator's unified `sdk.memory.estimate_num_gpu_blocks` API instead of hand-rolled per-backend budget math, making aiconfigurator the single source of truth. SGLang block counts grew slightly more conservative by subtracting full non-KV memory; vLLM and TensorRT-LLM counts were unchanged ([#10686](https://github.com/ai-dynamo/dynamo/pull/10686)).

#### Trace Replay

* **Mooncake Trace Capture and Replay:** Added per-request Mooncake trace capture for Rust OpenAI chat and completion serving via the `DYN_REQUEST_TRACE` environment variable, with rolling sequence hashes, partial output length on cancellation, and configurable trace sinks ([#10381](https://github.com/ai-dynamo/dynamo/pull/10381)), introduced a mooncake-delta trace format that accumulates session input deltas into cumulative prompts before recomputing engine block hashes ([#9653](https://github.com/ai-dynamo/dynamo/pull/9653)), added an experimental agentic Mooncake replay path with dependency edges, branch markers, and tool wait timing ([#9728](https://github.com/ai-dynamo/dynamo/pull/9728)), and fixed mooncake-delta prompt reconstruction to include the previous prompt and generated output in each follow-up turn ([#11060](https://github.com/ai-dynamo/dynamo/pull/11060)).
* **Planner Replay Goodput Metrics:** Added SLA-satisfying goodput and provisioned GPU-hours to the planner-replay report, classifying each completed request against an optional SLA using aiperf-matched ITL, TTFT, and e2e bounds independent of the planner scaling SLA ([#10739](https://github.com/ai-dynamo/dynamo/pull/10739)); exposed the goodput SLA on the plain trace-replay binding so static non-autoscaling replays also report the goodput keys ([#10849](https://github.com/ai-dynamo/dynamo/pull/10849)); and supported concurrency-capped and synthetic workloads in the planner-in-the-loop path beyond arrival-timestamp traces ([#10888](https://github.com/ai-dynamo/dynamo/pull/10888)).
* **Offline Replay Event-Driven Path:** Unified offline replay onto one event-driven run loop where a planner tick is a first-class `PlannerTick` event, fixing an unbounded FPM-snapshot memory leak on the plain run path ([#10940](https://github.com/ai-dynamo/dynamo/pull/10940)), added a `--max-sim-time-minutes` flag that ends the benchmark at the specified simulated wall time across disagg and agg paths ([#9699](https://github.com/ai-dynamo/dynamo/pull/9699)), and changed `--replay-concurrency` to schedule multi-turn sessions depth-first as a session cap, admitting a new session only when an active one finishes ([#10516](https://github.com/ai-dynamo/dynamo/pull/10516)).
* **Replay Per-Request Metrics:** Added a `--report-jsonl` option to offline `dynamo.replay` that emits one JSON object per completed request with TTFT, ITL, ISL/OSL, cache hit rate, and worker index ([#9720](https://github.com/ai-dynamo/dynamo/pull/9720)); added native vLLM/SGLang-style Prometheus metrics for the live mocker covering request timing, cache hit rates, queue status, and preemption counts ([#10056](https://github.com/ai-dynamo/dynamo/pull/10056)); and trimmed online-replay diagnostics by demoting `replay_diag` breadcrumbs to debug level and removing test timing prints ([#9565](https://github.com/ai-dynamo/dynamo/pull/9565)).
* **Output Token Replay:** Added deterministic output-token replay for Dynamo mockers, where enriched Mooncake traces supply exact `output_token_ids` and live requests select a replay row via an `output_replay_id:<key>` annotation, retaining the random-token fallback when no replay plan is present ([#10877](https://github.com/ai-dynamo/dynamo/pull/10877)).
* **Claude Session Trace Export:** Added export of local Claude Code root and subagent sessions as canonical `dynamo.request.trace.v1` request and tool events, with explicit source-to-tool-to-child causality, making historical sessions replayable while an always-on verifier fails on fidelity drift ([#10971](https://github.com/ai-dynamo/dynamo/pull/10971)).

### Kubernetes Deployment

* **GMS Weight Loading Integration:** Integrated ModelExpress P2P weight transfer into the GMS loader and worker via a LoadStrategyChain for automatic weight-source detection ([#8218](https://github.com/ai-dynamo/dynamo/pull/8218)), and cut GMS load overhead by skipping torch import on the server startup path and bounding NIXL staging workers ([#9635](https://github.com/ai-dynamo/dynamo/pull/9635)). Auto-enabled the SGLang memory saver path for GMS startup ([#9647](https://github.com/ai-dynamo/dynamo/pull/9647)), used the DP-adjusted device for the early vLLM GMS connection ([#9840](https://github.com/ai-dynamo/dynamo/pull/9840)), refreshed vLLM KV cache wake state after GMS remaps VAs ([#10319](https://github.com/ai-dynamo/dynamo/pull/10319)), set a 512 MiB default scratch alias size for shadow-engine scratch mappings ([#10329](https://github.com/ai-dynamo/dynamo/pull/10329)), and dropped the implicit failover requirement from standalone inter-pod GMS in the Operator ([#10378](https://github.com/ai-dynamo/dynamo/pull/10378)).
* **GMS Sidecar Lifecycle Rework:** Coordinated GMS sidecar lifecycles through the lock state machine instead of kubelet probes and init-phase ordering, dropping the gms-server StartupProbe and fixing a saver Job-completion race and a too-short 30 second read-only reconnect timeout ([#9514](https://github.com/ai-dynamo/dynamo/pull/9514)). Reworked the checkpoint-client API around user-declared Kubernetes containers so the Operator no longer creates gms-loader or gms-saver containers ([#9641](https://github.com/ai-dynamo/dynamo/pull/9641)), and made the \_allocator\_ext C++ extension PEP 703 free-threading ready by replacing torn-pointer callback globals with a magic-statics singleton ([#9575](https://github.com/ai-dynamo/dynamo/pull/9575)).
* **Snapshot Restore Probing and Runtime:** Reworked snapshot checkpoint and restore handling to gate checkpoints on target-container readiness with an OCI runtime container-ID fallback ([#9534](https://github.com/ai-dynamo/dynamo/pull/9534)), tightened restore-target StartupProbe cadence so containers flip Ready as soon as CRIU restore completes ([#9627](https://github.com/ai-dynamo/dynamo/pull/9627)), and let restore pods start container-ID polling before Kubernetes marks the pod Running ([#9984](https://github.com/ai-dynamo/dynamo/pull/9984)). Refreshed restore-time runtime env before vLLM and SGLang create the DistributedRuntime and split the snapshot utils into focused modules ([#10376](https://github.com/ai-dynamo/dynamo/pull/10376)), added a CRIU UPDATE\_INETSK plugin that remaps IPv4 socket addresses across pod IPs and defaulted established TCP socket restore ([#10727](https://github.com/ai-dynamo/dynamo/pull/10727)), and applied fast 1s worker probes only once checkpointInfo is Ready, bounding restore startup probes to 30 minutes ([#10859](https://github.com/ai-dynamo/dynamo/pull/10859)).
* **DGD Auto-Checkpoint Lifecycle Management:** Scoped DynamoGraphDeployment automatic checkpoints to the owning DGD component worker generation instead of legacy identity-hash reuse, removing the need for `checkpoint.identity` in `mode: Auto` ([#10177](https://github.com/ai-dynamo/dynamo/pull/10177)). Added a `checkpoint.startupPolicy` and Pod CREATE mutating webhook that defaults workers to `Immediate` cold start while the checkpoint job runs in the background ([#10179](https://github.com/ai-dynamo/dynamo/pull/10179)), and a `checkpoint.deletionPolicy` with finalizer-based artifact cleanup that defaults to deleting DGD-managed checkpoint CRs and stored artifacts on DGD deletion ([#10183](https://github.com/ai-dynamo/dynamo/pull/10183)).
* **TensorRT-LLM Snapshot Support:** Added TensorRT-LLM support for Dynamo Snapshot restore, delaying `DistributedRuntime` creation until after restore and reloading the Kubernetes restore identity before endpoint registration, plus a Qwen3-0.6B single-GPU example ([#10432](https://github.com/ai-dynamo/dynamo/pull/10432)). Introduced the namespaced `PodSnapshot` and cluster-scoped `PodSnapshotContent` CRDs (`nvidia.com/v1alpha1`) with a `PodSnapshotReconciler` modeled on the CSI VolumeSnapshot pattern ([#10820](https://github.com/ai-dynamo/dynamo/pull/10820)).
* **Grove DGD Integration:** Preserved existing Grove PodCliqueSet replica fields during DGD reconciliation so live scaling stays driven through Grove scale subresources instead of PCS template updates ([#9773](https://github.com/ai-dynamo/dynamo/pull/9773)). Added a DynamoGraphDeployment `priorityClassName` field passed through to the Grove PodCliqueSet and rejected by the webhook unless the Grove pathway is active ([#10217](https://github.com/ai-dynamo/dynamo/pull/10217)), and a component-level `minAvailable` knob for Grove-backed DGDs controlling gang scheduling and termination ([#10532](https://github.com/ai-dynamo/dynamo/pull/10532)).
* **Worker Topology Label Injection:** Injected topology metadata into worker pod templates when `spec.experimental.kvTransferPolicy` is set, adding a Downward API volume and runtime topology env vars while leaving frontend pods and policy-less workers unchanged ([#9792](https://github.com/ai-dynamo/dynamo/pull/9792)). Added a `TopologyLabelReconciler` that watches annotated scheduled pods and copies the configured topology label from their node, marks worker DCDs with `nvidia.com/topology-label-key`, and adds node-read RBAC for cluster-wide and namespace-restricted installs ([#9879](https://github.com/ai-dynamo/dynamo/pull/9879)).
* **Volcano Scheduler Support:** Added Volcano scheduler support for Grove in the Operator ([#11348](https://github.com/ai-dynamo/dynamo/pull/11348)).
* **Rust ext\_proc EPP Server:** Added a Rust ext\_proc service for the Endpoint Picker (EPP) that Envoy calls to make routing decisions, replacing the llm-d Go implementation and cutting the call path from three language boundaries to one (Envoy C++ to Rust gRPC). The server calls Dynamo Router, tokenizes the request body for KV-aware routing, and locates workers via DistributedRuntime discovery plus Kubernetes pod watches ([#8783](https://github.com/ai-dynamo/dynamo/pull/8783)).
* **KV Transfer Policy API:** Added an experimental topology-aware KV transfer policy under `spec.experimental.kvTransferPolicy` on DynamoGraphDeployment, with `labelKey`, `domain`, `required` or `preferred` enforcement, and worker-side `DYN_KV_TRANSFER_*` env injection ([#9768](https://github.com/ai-dynamo/dynamo/pull/9768)). Added Grove `ClusterTopology` as an alternative topology source via `clusterTopologyName`, projecting topology levels as `nvidia.com/dynamo-topology.<domain>` pod labels ([#10041](https://github.com/ai-dynamo/dynamo/pull/10041)).
* **Custom Init Container Image:** Added a Helm value to configure the snapshot DaemonSet init container image, letting users override the default in their charts ([#8432](https://github.com/ai-dynamo/dynamo/pull/8432)).
* **Controller-Manager Pod Metadata:** Added `controllerManager.podLabels` and `controllerManager.podAnnotations` values to the Operator chart, letting users inject pod-level labels and annotations onto the controller-manager Deployment without forking the chart for observability integrations such as Datadog and Prometheus ([#9195](https://github.com/ai-dynamo/dynamo/pull/9195)).
* **Power Agent DaemonSet:** Added a standalone Power Agent DaemonSet for per-node GPU power-cap enforcement, packaged as a Helm chart with DaemonSet, ServiceAccount, and RBAC for cluster-scoped or namespace-restricted mode. Power caps apply through the `dynamo.nvidia.com/gpu-power-limit` pod annotation, with NVML clamping, multi-pod policy, SIGTERM restore, and fail-safe reconcile ([#9682](https://github.com/ai-dynamo/dynamo/pull/9682)).
* **GB10 GPU SKU Support:** Added support for the GB10 GPU SKU type, letting users select and deploy on GB10 hardware in deployment specs ([#9976](https://github.com/ai-dynamo/dynamo/pull/9976)).

### Fault Tolerance & Observability

* **Unified Backend Observability Surface:** Added end-to-end OpenTelemetry tracing across the unified backend, forwarding W3C traceparent to vLLM, TRT-LLM, and SGLang with cross-process trace linking for disaggregated serving ([#9543](https://github.com/ai-dynamo/dynamo/pull/9543)). Added Prometheus metric-name parity via a new `register_prometheus` engine hook and Rust `EngineMetrics` handle ([#9586](https://github.com/ai-dynamo/dynamo/pull/9586)), and health-check canary support through `LLMEngine.health_check_payload()` registered with the runtime `HealthCheckManager`, overridable by the `--health-check-payload` flag and `DYN_HEALTH_CHECK_PAYLOAD` env var ([#9642](https://github.com/ai-dynamo/dynamo/pull/9642)).
* **OTLP HTTP and Sampling:** Added OTLP HTTP/protobuf export for traces and logs with protocol and endpoint selection via `OTEL_EXPORTER_OTLP_PROTOCOL` and `OTEL_EXPORTER_OTLP_ENDPOINT`, plus optional trace sampling through `OTEL_TRACES_SAMPLE_RATIO` while keeping the default sampler ([#10576](https://github.com/ai-dynamo/dynamo/pull/10576)), fixed a startup crash when the `http/protobuf` transport selected no default HTTP client ([#10829](https://github.com/ai-dynamo/dynamo/pull/10829)), and preserved W3C `traceparent` sampling flags so inbound sampler decisions propagate to downstream workers ([#10980](https://github.com/ai-dynamo/dynamo/pull/10980)).
* **Embedding Workload Metrics Gating:** Added embedding-shaped Prometheus histograms for batch size and per-request input tokens on the SGLang embedding worker ([#9753](https://github.com/ai-dynamo/dynamo/pull/9753)), plus a `dynamo_embedding_latency_seconds` histogram for end-to-end `/v1/embeddings` latency in the Rust frontend ([#9758](https://github.com/ai-dynamo/dynamo/pull/9758)), and gated the chat-shaped collectors (KV gauges, prefill/decode counters, `LLMBackendMetrics`) so they no longer emit zeros on embedding workers in SGLang ([#9830](https://github.com/ai-dynamo/dynamo/pull/9830)) and vLLM ([#9886](https://github.com/ai-dynamo/dynamo/pull/9886)).
* **Request Trace Unification:** Removed the separate `DYN_AGENT_TRACE` path and made `DYN_REQUEST_TRACE` the single trace switch, emitting enriched `dynamo.request.trace.v1` rows with agent context, replay hashes, timing and token metrics, worker attribution, and finish-reason metadata for requests carrying `nvext.agent_context` ([#10701](https://github.com/ai-dynamo/dynamo/pull/10701)). The Perfetto converter now infers tool slices from `request.finish_reason_metadata.tool_calls` when explicit harness tool events are absent ([#10802](https://github.com/ai-dynamo/dynamo/pull/10802)).
* **Log Crate Trace Capping:** Capped the `log` crate's trace level at compile time via the `release_max_level_debug` feature so `log::trace!` compiles to a no-op in release builds, removing per-character trace spam from the HF tokenizers normalization path ([#10286](https://github.com/ai-dynamo/dynamo/pull/10286)), and removed the `DYNAMO_SKIP_PYTHON_LOG_INIT` env var from the replay and indexer launchers so logging state no longer leaks into child pytest processes ([#9724](https://github.com/ai-dynamo/dynamo/pull/9724)).
* **Chat Stream Metrics Annotations:** Emitted the `llm_metrics` annotation from the vLLM chat processor so per-chunk TTFT and ITL histograms record samples under `--dyn-chat-processor vllm` ([#10019](https://github.com/ai-dynamo/dynamo/pull/10019)), added a typed `LLMMetricAnnotation` fast path that attaches metrics directly to chat stream responses and skips the per-chunk JSON serialize/parse round trip while keeping the legacy path as a fallback ([#10512](https://github.com/ai-dynamo/dynamo/pull/10512)), and emitted token, TTFT, ITL, and cached-token annotations from the SGLang chat processor ([#10779](https://github.com/ai-dynamo/dynamo/pull/10779)).
* **Worker Request Admission Control:** Added worker-side defensive request admission, disabled by default, via `--engine-request-limit` and `--dynamo-request-queue-limit` flags that bound concurrent in-engine and Dynamo-queued requests with fast HTTP 503 shedding on overflow ([#10509](https://github.com/ai-dynamo/dynamo/pull/10509)). Added a `--no-admission-control` escape hatch that clears busy-worker rejection thresholds while keeping router queueing independent, and updated the default queue and prefill-fraction thresholds plus the 503 rejection hint ([#9547](https://github.com/ai-dynamo/dynamo/pull/9547)).
* **Planner SLA Dashboard Updates:** Added SLA target reference lines for TTFT and ITL to the planner Grafana dashboard's Observed Latency panel, matching each target to its observed counterpart's color and axis with a dashed style ([#10712](https://github.com/ai-dynamo/dynamo/pull/10712)), and removed the dead Correction Factors panels that rendered permanent No data once the correction-factor gauges gave way to online FPM regression ([#10824](https://github.com/ai-dynamo/dynamo/pull/10824)).
* **Configurable SGLang Trace Level:** Added a default SGLang trace level of 2 when `--enable-trace` is set, suppressing high-volume scheduler spans such as `decode_loop` while keeping per-request spans, and made the level overridable via `SGLANG_TRACE_LEVEL` ([#9327](https://github.com/ai-dynamo/dynamo/pull/9327)).
* **Standalone Resource Observability Dashboard:** Added a standalone optional WebSocket dashboard served at `/` alongside the `/metrics` endpoint, giving lightweight host-resource debugging without standing up Grafana or Docker Compose. If dashboard packages are missing, `/` shows the install command and `/metrics` still works ([#9780](https://github.com/ai-dynamo/dynamo/pull/9780)).
* **Per-Model Grafana Dashboard:** Added an engine-agnostic per-model Grafana dashboard with 24 panels across Overview, Frontend, KV Routing, and Workers rows, reading the standard `dynamo_frontend_*`, `dynamo_component_*`, and `dynamo_router_*` metrics filtered by a templated model selector. It preserves the existing UID so bookmarks survive ([#9811](https://github.com/ai-dynamo/dynamo/pull/9811)).
* **TensorRT-LLM Engine Health Monitor:** Added `TrtllmEngineMonitor`, which polls TensorRT-LLM's native `check_health` API and shuts down the engine, distributed runtime, and worker process on a fatal health state so Kubernetes can restart the pod ([#10266](https://github.com/ai-dynamo/dynamo/pull/10266)).
* **Registered Worker Metric:** Added a Router component gauge, `dynamo_component_router_worker_registered`, that tracks registered backend workers and updates as topology changes or workers are removed, labeled by router worker ID, data-parallel rank, and worker type for worker-level monitoring ([#10587](https://github.com/ai-dynamo/dynamo/pull/10587)).

### vLLM

* **Native vLLM Rust Backend:** Added a native vLLM backend built on the vLLM Rust crates that act as clients of the Python vLLM engine, replacing the older AsyncLLM wrapper and leaving the Dynamo frontend untouched ([#9206](https://github.com/ai-dynamo/dynamo/pull/9206)).
* **Unified vLLM LoRA Support:** Added dynamic LoRA adapter load, unload, and list operations to the unified vLLM engine, plus discovery publishing and per-request LoRA routing, closing the parity gap with the legacy vLLM path. Controls gate on `--enable-lora` and `DYN_LORA_ENABLED`, with a `/v1/loras` compatibility shim preserving the legacy HTTP surface ([#10347](https://github.com/ai-dynamo/dynamo/pull/10347)).
* **vLLM Worker Topology Registration:** Added worker\_type and peer-dependency declarations to each vLLM model registration call so aggregated, decode, and prefill workers declare their disaggregation role and dependencies. Re-exported WorkerType through dynamo.llm for backend imports ([#9395](https://github.com/ai-dynamo/dynamo/pull/9395)).
* **Mooncake PD Disaggregation Support:** Let vLLM workers running MooncakeConnector participate in prefill/decode disaggregation by branching the prefill handler to set push-based `kv_transfer_params` and synthesize the decode-side `disaggregated_params` payload. No Rust router changes were needed, since the router treats `disaggregated_params` as opaque ([#9414](https://github.com/ai-dynamo/dynamo/pull/9414)).
* **Aggregated Text-Embedding Worker:** Added an aggregated text-embedding worker shape to the vLLM backend, selected with a new `--embedding-worker` flag, so existing embedding deployments such as `Qwen3-Embedding-0.6B` can move onto Dynamo and vLLM without changing engine args. The worker serves the `/v1/embeddings` route and rejects incompatible combinations like prefill or decode disaggregation, multimodal flags, and benchmark mode at parse time ([#9713](https://github.com/ai-dynamo/dynamo/pull/9713)).
* **Aggregated LMCache MP Mode:** Added a launch script for vLLM aggregated serving that starts an LMCache server sidecar and runs the `LMCacheMPConnector`, replacing the in-process LMCache path on non-Kubernetes environments, and realigned the LMCache integration and observability docs to the multi-process sidecar architecture ([#9982](https://github.com/ai-dynamo/dynamo/pull/9982)).

### SGLang

* **SGLang Worker Topology Fields:** Added worker-type and dependency declarations at SGLang LLM registration sites, deriving Decode, Prefill, and Aggregated roles from the serving mode so each worker reports its topology role at registration. Embedding, diffusion, and multimodal-encode workers default to Aggregated, preserving current behavior ([#9397](https://github.com/ai-dynamo/dynamo/pull/9397)).

### TensorRT-LLM

* **Worker Type Registration:** Populated worker\_type and peer dependencies at the TensorRT-LLM `register_model` call site, deriving the disaggregation role (Prefill, Decode, or Aggregated) and required peers per branch of the configured disaggregation mode. Appended Encode to the dependency set when an encode endpoint is configured ([#9396](https://github.com/ai-dynamo/dynamo/pull/9396)).
* **Unified Backend Logits Processors:** Ported the logits-processor pipeline to the unified TensorRT-LLM backend and added a backend-agnostic `LogitsProcessorSpec` layer that expresses per-engine logits-processor activation as serializable spec entries and applies cross-backend policies for per-request runtime state and PREFILL skip in disaggregated serving ([#10080](https://github.com/ai-dynamo/dynamo/pull/10080)).

### Infrastructure Modernization

* **Call-Home Request-Plane Transport:** Implemented the request-stream half of the call-home TCP transport so an upstream server can stream frames to a downstream client over a dedicated socket, settling on a unidirectional-after-handshake design ([#9991](https://github.com/ai-dynamo/dynamo/pull/9991)). Wired end-to-end bidirectional dispatch through `PushRouter::generate(ManyIn<T>)` and `AddressedPushRouter::generate_bidirectional` with a new runtime-side ingress handler ([#9674](https://github.com/ai-dynamo/dynamo/pull/9674)), and added an opt-in `DYN_REQUEST_PLANE_CODEC=json|msgpack` setting carried in the request control message while keeping JSON as the default ([#10437](https://github.com/ai-dynamo/dynamo/pull/10437)).
* **Runtime Event Plane Selection:** Added an explicit `event_plane` parameter to the Python `DistributedRuntime` so callers can request ZMQ and avoid ambient NATS ([#10021](https://github.com/ai-dynamo/dynamo/pull/10021)), then defaulted ZMQ across all discovery backends and kept NATS as an opt-in via `DYN_EVENT_PLANE=nats` ([#10902](https://github.com/ai-dynamo/dynamo/pull/10902)).
* **Configurable ETCD Lease TTL:** Added the ETCD\_LEASE\_TTL environment variable to control the etcd lease time-to-live, previously hardcoded to 10 seconds, letting users tune lease duration at runtime ([#8076](https://github.com/ai-dynamo/dynamo/pull/8076)).
* **Context Metadata Propagation:** Added a string-keyed metadata field to `Context` that propagates end-to-end through the Dynamo pipeline, from the Python caller through the network into the backend engine, and survives cancellation, retries, and prefill handoffs. Python callers can attach metadata such as tenant or region to requests and read or mutate it on the backend ([#9662](https://github.com/ai-dynamo/dynamo/pull/9662)).

### Hardware

#### Intel XPU

* **XPU Multimodal Nixl Support:** Added Intel XPU aggregated multimodal examples, router launchers, and test profiles using the shared MultimodalModelProfile pattern ([#8112](https://github.com/ai-dynamo/dynamo/pull/8112)), with NIXL canonical VRAM/DRAM memory-type classification, Intel XPU device support for disaggregated E/PD, and XPU encode-worker kernel synchronization ([#9073](https://github.com/ai-dynamo/dynamo/pull/9073)), and aligned DYN\_SYSTEM\_PORT defaults plus ZE\_AFFINITY validation in the XPU launch scripts ([#10488](https://github.com/ai-dynamo/dynamo/pull/10488)).

### KV Block Manager

* **KVBM-Logical Backend Rework:** Restructured block lifecycle management into a unified store architecture, replacing the separate pool-based designs and simplifying block state transitions and registration flows under concurrent operations ([#8793](https://github.com/ai-dynamo/dynamo/pull/8793)). Added single-lock prefix matching that resolves an N-hash active-or-inactive prefix under one store-mutex acquisition, plus a Criterion benchmark scaffold covering the lru, multi\_lru, and lineage backends ([#9551](https://github.com/ai-dynamo/dynamo/pull/9551)).
* **Universal Hashing Consolidator:** Added a KV cache event consolidator that uses universal hashing to process KVBM v2 event streams and vLLM KV events, deduplicating across sources and publishing a unified stream ([#9480](https://github.com/ai-dynamo/dynamo/pull/9480)).
* **Configurable Block Reset Toggles:** Added pool-level and per-block controls to disable caching of inactive blocks in the KVBM block manager, keeping the reset-on-release preference across cache resurrections ([#9504](https://github.com/ai-dynamo/dynamo/pull/9504)).
* **Python Topology Config Reading:** Added a Python topology environment and file reader for worker startup, and wired topology plus KV-transfer-policy fields into vLLM, SGLang, and TensorRT-LLM runtime config publication through the `ModelRuntimeConfig` binding ([#9878](https://github.com/ai-dynamo/dynamo/pull/9878)).
* **ModelExpress Engine Integration:** Added ModelExpress engine-side weight loading for the vLLM and SGLang runtimes, installing the `modelexpress` client in both images by default and adding an `ignore_weights` option to `register_model()` so registration fetches tokenizer and config metadata without a full weight download. Deprecated the legacy Dynamo-owned vLLM `--model-express-url` wrapper in favor of the plugin-owned `--load-format modelexpress` path ([#10049](https://github.com/ai-dynamo/dynamo/pull/10049)).

### General

* **LoRA Load Estimation Pipeline:** Added a self-contained load-tracking pipeline for LoRA adapter allocation, with environment-variable configuration, an exponential-moving-average predictor, and a lock-free bucketed arrival-rate counter for per-LoRA request-rate estimation ([#8178](https://github.com/ai-dynamo/dynamo/pull/8178)).
* **Worker Topology in register\_model:** Extended `register_model` and `LocalModel::attach` with optional `worker_type` and `needs` parameters so backends can attach disaggregation role and peer dependencies to a worker's ModelDeploymentCard, rejecting non-canonical combinations at the binding. The card checksum now covers these fields, so a rolling update that changes only topology metadata is rejected as incompatible rather than joining a WorkerSet with a stale card ([#8700](https://github.com/ai-dynamo/dynamo/pull/8700)).
* **Engine Management Routes:** Added unified-backend engine-management plumbing exposed through Dynamo's `/engine/{route}` runtime endpoint, with per-backend callbacks for profiling, sleep/wake and memory release/resume, weight updates, and elastic scaling across vLLM, SGLang, and TensorRT-LLM ([#10094](https://github.com/ai-dynamo/dynamo/pull/10094)).
* **Hugging Face Hub LoRA Sources:** Added native `hf://<repo>[@revision]` support for dynamic LoRA loading, downloading adapters into the standard Hugging Face Hub cache and returning the immutable commit snapshot without a second copy under `DYN_LORA_PATH`. Revision-pinned, timeout-bounded snapshot downloads honor Hugging Face cache, token, endpoint, and offline semantics ([#11816](https://github.com/ai-dynamo/dynamo/pull/11816)).

***

## Recipes

* **Kimi K2 Agentic Recipes:** Replaced outdated Kimi K2.5 recipes with disaggregated TensorRT-LLM deployment using KV-aware routing and Eagle3 speculative decoding, aggregated round-robin variants, and ComputeDomain multi-node configuration tuned on the agentic code dataset ([#9621](https://github.com/ai-dynamo/dynamo/pull/9621)), and added Kimi K2.6 Turbo recipes with Kubernetes deployments for chat and agentic workloads on B200 and H200 GPUs plus a trace-replay benchmarking guide ([#10187](https://github.com/ai-dynamo/dynamo/pull/10187)).
* **Qwen3 vLLM Serving Recipes:** Added an aggregated serving recipe for Qwen3-0.6B with Gateway API Inference Extension (GAIE) integration, including an Istio DestinationRule required for the gateway proxy to reach the EPP service, plus a Qwen3.6-35B-A3B aggregated serving recipe with a standalone frontend ([#8355](https://github.com/ai-dynamo/dynamo/pull/8355)).
* **Qwen3-VL-32B-FP8 Recipe:** Added a production-ready deployment recipe for the Qwen3-VL-32B-Instruct-FP8 vision-language model with aggregated single-GPU and disaggregated cross-vendor modes, the latter using Intel XPU encode and NVIDIA GPU decode with NIXL embedding transfer over RDMA. The recipe includes model-cache jobs and QPS-sweep multimodal benchmarks ([#9252](https://github.com/ai-dynamo/dynamo/pull/9252)).
* **GLM-5-NVFP4 EFA Variant:** Added an EFA variant of the GLM-5-NVFP4 SGLang disaggregated recipe for GB200 clusters on AWS, routing KV transfer over AWS EFA RDMA via NIXL's LIBFABRIC backend instead of UCX for p6e-gb200.36xlarge deployments ([#9712](https://github.com/ai-dynamo/dynamo/pull/9712)).
* **Nemotron-3-Super Turbo Recipes:** Added NIM Turbo deployment recipes for NVIDIA-Nemotron-3-Super-120B-A12B using Dynamo and vLLM, covering NVFP4 on B200 and FP8 on H200 across chat and agentic serving modes, plus model-cache manifests and an AIPerf trace-replay performance job ([#10223](https://github.com/ai-dynamo/dynamo/pull/10223)).
* **Speculative Decoding Hardcoding Fix:** Removed a hardcoded speculative decoding variable from the Kimi K2.5 TensorRT-LLM recipes and corrected the README. ([#9637](https://github.com/ai-dynamo/dynamo/pull/9637)).
* **Model-Cache Memory Limits:** Added per-recipe memory and CPU resource requests and limits plus an absolute 16GB HF XET reconstruction buffer cap to all 15 model-download Jobs, preventing kubelet OOM eviction during downloads on Kubernetes ([#9884](https://github.com/ai-dynamo/dynamo/pull/9884)).
* **DeepSeek V4 SGLang Dependency:** Added an explicit `distro` install to the DeepSeek V4 SGLang B200 runtime image, fixing a `ModuleNotFoundError` that caused the worker to exit during startup when the OpenAI SDK imported the missing package ([#10085](https://github.com/ai-dynamo/dynamo/pull/10085)).
* **gpt-oss-120b Recipe Revert:** Reverted the gpt-oss-120b TensorRT-LLM aggregated recipe runtime image from 1.1.1 to 1.0.0, since the 1.1.1 runtime regressed the recipe on GB200 ([#10147](https://github.com/ai-dynamo/dynamo/pull/10147)).
* **Hugging Face Hub Pins:** Bumped the Hugging Face Hub CLI pin from 1.11.0 to 1.16.4 across 17 model-download recipes and two vLLM LoRA sync jobs, so helper jobs install the `click` dependency required by `hf download` ([#11273](https://github.com/ai-dynamo/dynamo/pull/11273)).

## Bug Fixes

### Frontend

* **Tool-Call & Reasoning Parser Hardening:** Aligned tool-call and reasoning parsers for GLM-4.7, DeepSeek V3/V3.2/V4, Kimi K2, Qwen, Gemma4, Mistral, MiniMax, and Nemotron with the vLLM/SGLang reference contracts, suppressing streaming markup leaks, recovering truncated and EOF-terminated tool calls, restoring chat-template fidelity, and routing structured output correctly ([#9355](https://github.com/ai-dynamo/dynamo/pull/9355), [#9524](https://github.com/ai-dynamo/dynamo/pull/9524), [#9594](https://github.com/ai-dynamo/dynamo/pull/9594), [#9462](https://github.com/ai-dynamo/dynamo/pull/9462), [#9411](https://github.com/ai-dynamo/dynamo/pull/9411), [#10833](https://github.com/ai-dynamo/dynamo/pull/10833)) and related fixes.
* **Worker-Set Readiness Routing:** Confined routing to complete worker sets and added 529/503 codes separating capacity overload from backend unavailability ([#10503](https://github.com/ai-dynamo/dynamo/pull/10503), [#10941](https://github.com/ai-dynamo/dynamo/pull/10941)).
* **KServe gRPC Inference Dispatch:** Fixed a KServe `model_ready`/`server_ready` race that reported ready before engines attached, and centralized dispatch to return clear model-not-found errors ([#9619](https://github.com/ai-dynamo/dynamo/pull/9619), [#10621](https://github.com/ai-dynamo/dynamo/pull/10621)).
* **EOS/BOS Token & Model Discovery:** Resolved `eos_token_id`/`bos_token_id` from tokenizer and special-tokens files when absent from model config ([#10672](https://github.com/ai-dynamo/dynamo/pull/10672)), and accepted the `deepseek_v32` model\_type spelling so V3.2 workers register ([#10332](https://github.com/ai-dynamo/dynamo/pull/10332)).
* **Sanitized Errors & API Compatibility:** Stopped HTTP errors from leaking paths, versions, and stack text (CWE-209) ([#10151](https://github.com/ai-dynamo/dynamo/pull/10151)), mapped backend `InvalidArgument` to HTTP 400 ([#11591](https://github.com/ai-dynamo/dynamo/pull/11591)), and relaxed deserialization for multimodal `EasyMessage` and Anthropic message-level system roles so agent clients like Claude Code pass validation ([#9470](https://github.com/ai-dynamo/dynamo/pull/9470), [#10108](https://github.com/ai-dynamo/dynamo/pull/10108)).
* **Request Draining & Context Length:** Drained inflight requests on shutdown so terminating frontend pods finish streaming instead of returning HTTP 500 ([#10862](https://github.com/ai-dynamo/dynamo/pull/10862)), and propagated the backend `context_length` as `max_model_len` so long-context requests reach the backend ([#10827](https://github.com/ai-dynamo/dynamo/pull/10827)).

### Agents

* **Coding Agent Header Handling:** Merged runs of leading system messages when converting `/v1/responses` to chat completions, fixing Codex CLI first-turn failures on templates that reject multiple system messages ([#10283](https://github.com/ai-dynamo/dynamo/pull/10283)), and normalized Codex, Claude Code, and OpenCode session headers into `nvext.agent_context`, mapping `x-claude-code-agent-id` to a child `trajectory_id` ([#10782](https://github.com/ai-dynamo/dynamo/pull/10782)).

### Reinforcement Learning

* **Routed-Experts Capture for Log-Prob Recompute:** Base64-encoded MoE routed-experts capture with a `start` offset so consumers align routing data to the completion ([#10529](https://github.com/ai-dynamo/dynamo/pull/10529)), and fixed a first-token crash from double re-encoding `routed_experts` on non-DSv4 MoE models under SGLang v0.5.11+ ([#9657](https://github.com/ai-dynamo/dynamo/pull/9657)).
* **Rollout Serving Robustness:** Rejected unsupported RL output fields `completion_token_ids` and `prompt_logprobs` on `/v1/responses` with a field-specific 501 ([#11265](https://github.com/ai-dynamo/dynamo/pull/11265)), added a configurable weight-update init watchdog `DYN_RL_INIT_WEIGHTS_TIMEOUT_S` ([#11692](https://github.com/ai-dynamo/dynamo/pull/11692)), and pinned `zstandard==0.23.0` in the SGLang image to restore the RL metadata upload path ([#11379](https://github.com/ai-dynamo/dynamo/pull/11379)).

### Multimodal & Diffusion

* **Diffusion & Omni Model Coverage:** Forced eager mode in the Wan2.2 video and image-to-video launchers to bypass a CUDA illegal memory access ([#9563](https://github.com/ai-dynamo/dynamo/pull/9563)), raised Qwen3-VL `max_seq_len` to 8192 restoring disaggregated multimodal coverage ([#9764](https://github.com/ai-dynamo/dynamo/pull/9764)), and emitted one image per diffusion output instead of capping at n=1 ([#9822](https://github.com/ai-dynamo/dynamo/pull/9822)).
* **Multimodal Request Safety & Limits:** Returned actionable 4xx for SSRF-blocked media URLs instead of 500 ([#11360](https://github.com/ai-dynamo/dynamo/pull/11360)), derived the `max_tokens` default from the image-expanded prompt length so requests no longer overshoot context ([#10757](https://github.com/ai-dynamo/dynamo/pull/10757)), and threaded `trust_remote_code` from engine settings (default `False`) across vLLM, SGLang, and TensorRT-LLM, closing CWE-829 ([#11366](https://github.com/ai-dynamo/dynamo/pull/11366)).
* **Multimodal Rendering & Tensor Path:** Derived CHW tensor image dimensions from `shape[-2:]` to keep SGLang prefill/decode on the exact-token path ([#11299](https://github.com/ai-dynamo/dynamo/pull/11299)), and flattened mixed text and image content in the native Rust renderer for verbatim-echo templates like NVIDIA-Nemotron-Parse ([#10380](https://github.com/ai-dynamo/dynamo/pull/10380)).

### Scheduling

* **Router Admission & Backpressure:** Gated prefill admission on overloaded-worker sets, excluded over-threshold workers across routing modes, randomized tie-breaks, and bounded event-subscriber channels to cap frontend memory ([#10631](https://github.com/ai-dynamo/dynamo/pull/10631), [#9688](https://github.com/ai-dynamo/dynamo/pull/9688), [#9516](https://github.com/ai-dynamo/dynamo/pull/9516)).
* **KV-Router Indexing & Attribution:** Fixed prefill-router hash-mode mismatches, invalidated cached hashes on LoRA/multimodal metadata, and reconciled instance state during worker churn ([#9857](https://github.com/ai-dynamo/dynamo/pull/9857), [#10500](https://github.com/ai-dynamo/dynamo/pull/10500), [#9631](https://github.com/ai-dynamo/dynamo/pull/9631)).
* **Planner Load & FPM Accounting:** Counted started requests for throughput load, classified decode-sweep rows in Forward-Pass Metrics, and guarded overlapping disagg scale operations ([#9526](https://github.com/ai-dynamo/dynamo/pull/9526), [#9528](https://github.com/ai-dynamo/dynamo/pull/9528), [#9774](https://github.com/ai-dynamo/dynamo/pull/9774)).

### KV Router

* **KV Event & Cancellation Correctness:** Stamped SGLang multinode KV events with routable leader worker IDs so remote DP ranks update Router trees, shared KV hashing via the dynamo-kv-hashing contract, attributed workers only after dispatch, and scoped cancellation to a child token ([#9505](https://github.com/ai-dynamo/dynamo/pull/9505), [#10095](https://github.com/ai-dynamo/dynamo/pull/10095), [#11324](https://github.com/ai-dynamo/dynamo/pull/11324), [#11455](https://github.com/ai-dynamo/dynamo/pull/11455)).

### Planner & Profiler

* **Planner Scaling & SLA Correctness:** Scaled down throughput-only SLA deployments after traffic stops, propagated user SLA targets into PlannerConfig, and kept Planner bound to the serving worker namespace through rollouts ([#11296](https://github.com/ai-dynamo/dynamo/pull/11296), [#9537](https://github.com/ai-dynamo/dynamo/pull/9537), [#11622](https://github.com/ai-dynamo/dynamo/pull/11622)).
* **Profiler Sweep & DGDR Fixes:** Preserved DGDR served-model identity across sweeps, made parallel-config labels injective, resolved local PVC model paths (no HF refetch), and normalized SGLang/vLLM/TRT-LLM CUDA-graph arguments ([#11196](https://github.com/ai-dynamo/dynamo/pull/11196), [#10086](https://github.com/ai-dynamo/dynamo/pull/10086), [#9877](https://github.com/ai-dynamo/dynamo/pull/9877)).
* **AIConfigurator Version Fixes:** Capped aiconfigurator below v0.10 to restore the v0.9 API the profiler targets and removed the deprecated profiler WebApp and its Gradio dependency ([#11732](https://github.com/ai-dynamo/dynamo/pull/11732), [#11361](https://github.com/ai-dynamo/dynamo/pull/11361)).

### Performance Modeling & Replay

* **Replay & Mocker Correctness:** Preserved zero-duration completions and single-rank KV capture in disagg replay, fixed cached chunked-prefill accounting, gated mocker admission on physically available KV, and hardened KVBM offload simulation to fail fast on invalid states ([#9771](https://github.com/ai-dynamo/dynamo/pull/9771), [#10919](https://github.com/ai-dynamo/dynamo/pull/10919), [#10439](https://github.com/ai-dynamo/dynamo/pull/10439), [#11012](https://github.com/ai-dynamo/dynamo/pull/11012)).

### Kubernetes Deployment

* **Inference Gateway & Service Mesh:** Switched the GAIE install and validation flow from kgateway to agentgateway ([#7607](https://github.com/ai-dynamo/dynamo/pull/7607)), fixed EPP DestinationRule generation for `tlsMode=MUTUAL` ([#10068](https://github.com/ai-dynamo/dynamo/pull/10068)), and added `serviceMesh.enabled` to disable Istio where the DestinationRule API is absent ([#11344](https://github.com/ai-dynamo/dynamo/pull/11344)).
* **GMS, DRA & Snapshot/CRIU Restore:** Gated GMS ResourceClaimTemplate sync on DRA `v1` and derived snapshot-restore CUDA order from `nvidia-smi` ([#9454](https://github.com/ai-dynamo/dynamo/pull/9454), [#11325](https://github.com/ai-dynamo/dynamo/pull/11325)) and related fixes.
* **DGD/DGDR/DCD Admission & CRD Versioning:** Ported DynamoGraphDeployment webhooks to `v1beta1` with legacy v1alpha1 conversion and moved CRDs into a `crd-apply` init container to fit the 1 MiB Secret limit ([#10934](https://github.com/ai-dynamo/dynamo/pull/10934), [#11689](https://github.com/ai-dynamo/dynamo/pull/11689)) and related fixes.
* **DGDR & Rollout Lifecycle:** Surfaced specific DGDR failure reasons, fixed a profiling race that stranded DGDRs in Deploying, and preserved availability during overlapping worker rollouts ([#8227](https://github.com/ai-dynamo/dynamo/pull/8227), [#11340](https://github.com/ai-dynamo/dynamo/pull/11340), [#10184](https://github.com/ai-dynamo/dynamo/pull/10184)) and related fixes.
* **Operator Config, Secrets, Volumes & Services:** Fixed spurious rollouts from non-deterministic pull-secret discovery, restored the LWS `<dcd-name>-0` service name for worker DNS, and gated vLLM MP-init to the multiprocessing backend ([#9826](https://github.com/ai-dynamo/dynamo/pull/9826), [#9612](https://github.com/ai-dynamo/dynamo/pull/9612)) and related fixes.

### Fault Tolerance & Observability

* **Frontend Metrics & Shutdown Hardening:** Normalized `model` metric label casing, collapsed unregistered names to a bounded `unknown_model` sentinel, added a configurable graceful-shutdown drain timeout, and migrated Grafana dashboards to the replacement request gauges ([#9775](https://github.com/ai-dynamo/dynamo/pull/9775), [#9836](https://github.com/ai-dynamo/dynamo/pull/9836), [#10705](https://github.com/ai-dynamo/dynamo/pull/10705), [#11690](https://github.com/ai-dynamo/dynamo/pull/11690)).

### vLLM

* **Config & Correctness Fixes:** Stopped overwriting the user `--runner` flag that broke embedding models, removed per-run torch-compile cache regeneration, and aligned the vLLM-Omni pin to fix an import-time crash that left workers unregistered ([#9710](https://github.com/ai-dynamo/dynamo/pull/9710), [#11374](https://github.com/ai-dynamo/dynamo/pull/11374)).
* **Embedding & Multimodal Fixes:** Fixed the text-embedding worker to return one pooled, L2-normalized vector instead of the full per-token matrix, and backported a deepstack fix that crashed EngineCore on disaggregated multimodal Qwen3-VL requests ([#10248](https://github.com/ai-dynamo/dynamo/pull/10248), [#9491](https://github.com/ai-dynamo/dynamo/pull/9491)).

### SGLang

* **Correctness & Routing Fixes:** Forwarded dropped OpenAI sampling controls that caused degenerate token repetition, and advertised the global data-parallel rank range so distributed routing targets the correct ranks ([#10310](https://github.com/ai-dynamo/dynamo/pull/10310), [#10196](https://github.com/ai-dynamo/dynamo/pull/10196)).
* **Stability & Backend Fixes:** Fixed an asyncio Future/Task leak in the cancellation monitor causing linear RSS growth and TTFT spikes under load, and resolved NIXL libraries from `nixl_cu13` on cuda13 images ([#10082](https://github.com/ai-dynamo/dynamo/pull/10082), [#10325](https://github.com/ai-dynamo/dynamo/pull/10325)).

### TensorRT-LLM

* **Crash & Compatibility Fixes:** Skipped unused NIXL connector creation in aggregated mode, and capped `mpmath<1.4` to prevent an import-time crash from a missing sympy symbol ([#9501](https://github.com/ai-dynamo/dynamo/pull/9501), [#11404](https://github.com/ai-dynamo/dynamo/pull/11404)).
* **Performance & Telemetry Fixes:** Enabled block reuse and CUDA graphs in the qwen2-vl-7b example to close a multimodal TTFT regression, and replaced multi-megabyte KV event dumps with compact summaries plus Prometheus telemetry ([#9942](https://github.com/ai-dynamo/dynamo/pull/9942), [#10298](https://github.com/ai-dynamo/dynamo/pull/10298)).

### Infrastructure Modernization

* **Request-Plane & Discovery Hardening:** Rejected oversized NATS payloads with a sanitized `InvalidArgument`, aborted the TCP writer when its reader fails to end a connection-monitor hang, and gave each publisher a fork-safe ID to stop discovery-key collisions from dropping subscribers ([#9769](https://github.com/ai-dynamo/dynamo/pull/9769), [#9716](https://github.com/ai-dynamo/dynamo/pull/9716), [#11311](https://github.com/ai-dynamo/dynamo/pull/11311)).

### Hardware

* **Intel XPU & AMD ROCm Import Compatibility:** Added a `MultiModalUUIDDict` import fallback for XPU builds on vLLM v0.16.0, and fixed ROCm import failures on Python 3.10 by deferring the `nixl_connect` import ([#8106](https://github.com/ai-dynamo/dynamo/pull/8106), [#9929](https://github.com/ai-dynamo/dynamo/pull/9929)).

### KV Block Manager

* **Leak & Lifecycle Fixes:** Added a prefill-drain hook to prevent a decode-side use-after-free during NIXL transfers ([#9937](https://github.com/ai-dynamo/dynamo/pull/9937)).
* **Compatibility & Accounting Fixes:** Defaulted NIXL install extras to `nixl[cu13]`, and normalized live KV metrics to per-data-parallel-rank units ([#10413](https://github.com/ai-dynamo/dynamo/pull/10413), [#9932](https://github.com/ai-dynamo/dynamo/pull/9932)).

### General

* **Model Loading & Storage Fixes:** Made `register_model`/`hub::from_hf` cache-first, and prepended the hidden NIXL wheel directory to `LD_LIBRARY_PATH` so the frontend loads `libnixl.so` ([#10102](https://github.com/ai-dynamo/dynamo/pull/10102), [#9911](https://github.com/ai-dynamo/dynamo/pull/9911)).
* **Concurrency & Store Consistency Fixes:** Centralized `get_lora_manager()` behind a thread-safe `OnceLock` to stop duplicate LoRAManager construction, and fixed FileStore to honor `revision = 0` as create-if-absent instead of overwriting existing files ([#10110](https://github.com/ai-dynamo/dynamo/pull/10110), [#10968](https://github.com/ai-dynamo/dynamo/pull/10968)).

***

## Documentation

* **Parser & Tool-Calling Docs:** Reorganized tool-call and reasoning parser docs into a single Tool Calling section with a top-level Parser Configuration reference page, split Dynamo-native from engine-fallback paths, and added a `logprobs`-based troubleshooting guide ([#10236](https://github.com/ai-dynamo/dynamo/pull/10236), [#9400](https://github.com/ai-dynamo/dynamo/pull/9400), [#9658](https://github.com/ai-dynamo/dynamo/pull/9658)) and related updates.
* **Recipes & Feature Benchmarks:** Added Recipes and Feature Benchmarks doc surfaces with model-family filters and deployable targets, a DynoSim simulation section, and Nemotron-3-Ultra NIM Turbo recipes ([#10628](https://github.com/ai-dynamo/dynamo/pull/10628), [#10156](https://github.com/ai-dynamo/dynamo/pull/10156), [#10303](https://github.com/ai-dynamo/dynamo/pull/10303)) and related updates.
* **Deployment & Configuration Docs:** Documented Standalone and Gateway (GAIE) deployment modes, consolidated EKS/GKE/AKS/ECS guides under `cloud-providers/`, and expanded DGDR, Planner, and router configuration and KV-transfer guides ([#10059](https://github.com/ai-dynamo/dynamo/pull/10059), [#10348](https://github.com/ai-dynamo/dynamo/pull/10348), [#9428](https://github.com/ai-dynamo/dynamo/pull/9428), [#10123](https://github.com/ai-dynamo/dynamo/pull/10123)) and related updates.

## Looking Ahead

The following is planned for the next release, v1.4.0 (targeted 2026-08-12). See the public [Dynamo roadmap](https://github.com/ai-dynamo/dynamo/issues/9178) for the full plan.

### Self-Tuning Planner & Live Autoscaling

Dynamo Planner moves from configured scaling to live, self-tuning scaling. Forward-Pass Metrics streamed from the vLLM, SGLang, and TensorRT-LLM engines feed an AIConfigurator performance model that the Planner continuously fine-tunes against real traffic, so scaling decisions track your actual workload rather than a static profile. This release line adds bring-your-own-trace warmup from production traffic replay, a self-tuning remote-prefill policy that shifts local-versus-remote prefill live from prefill-worker load, and a plugin interface for custom scaling logic such as cost-aware or spot-instance strategies.

### Fault Tolerance & Fast Recovery

Production resiliency hardens across the stack. GPU Memory Service targets sub-five-second recovery for failed workers through shadow failover backed by a warm compile cache, CRIU-based GPU process snapshots make multi-GPU checkpoint and restore practical, and in-flight requests migrate or cancel cleanly under SLA pressure instead of failing outright. That foundation extends toward WideEP fail-continue, where a single GPU failure lets the healthy ranks keep serving while a replacement rejoins at rank granularity.

### Voice & Multimodal Pipelining

Dynamo extends beyond text to orchestrate multi-model pipelines as a single request. Transparent pipelining chains stages such as speech-to-text, an LLM, and text-to-speech (STT → LLM → TTS) into one logical call, with Dynamo handling scheduling, colocation, streaming, and latency budgeting across stages — so real-time voice assistants and audio-to-audio translation no longer need custom orchestration outside the inference stack. Landing alongside are use-case-optimized Omni and vision-language recipes tuned for VLM multi-turn chat and streaming audio.

#### Open-source contributions and new contributors

Between v1.2.1 and v1.3.0 the project merged 930 PRs from 125 contributors. Thank you to the external community contributors in this release (organization identified via commit-author email domain or public GitHub profile, cross-referenced against the team roster):

* **Intel:** @dsocek, @joshuayao, @pallavijaini0525, @Spycsh, @tthakkal, @VincyZhang, @yuanwu2017.
* **DaoCloud:** @carlory, @flpanbin, @my-git9, @yankay.
* **Inferact:** @BugenZhao, @Dao007forever, @zhewenl.  **Gcore:** @kirillemilio.  **AMD:** @andyluo7.  **Epic Games:** @hatemfaheem.
* **Anyscale:** @jeffreywang88.  **CoreWeave:** @ritazh.  **Microsoft:** @Jont828.  **GMI Cloud:** @GavinZhu-GMI.  **NeuReality:** @VadimEisenberg.  **Ohio Supercomputer Center:** @treydock.  **SK Telecom:** @mcpark84.  **Solo.io:** @danehans.
* **University of Washington:** @Shaoting-Feng.  **Arizona State University:** @Change72.  **East China Normal University:** @Muqi1029.  **Rutgers University:** @aryanputta.  **AI Labs Taiwan:** @waynehacking8.
* **Independent:** @atomic, @bewestphal, @cmdy, @doujiang24, @esmeetu, @jellysnack, @RitwijParmar, @sungsooha, @xianlubird, @yuanchen8911.

**New Contributors** — welcome to the 23 first-time contributors in this release:

* @Dao007forever made their first contribution in [#9195](https://github.com/ai-dynamo/dynamo/pull/9195).
* @zhewenl made their first contribution in [#9414](https://github.com/ai-dynamo/dynamo/pull/9414).
* @kirillemilio made their first contribution in [#9824](https://github.com/ai-dynamo/dynamo/pull/9824).
* @Shaoting-Feng made their first contribution in [#9982](https://github.com/ai-dynamo/dynamo/pull/9982).
* @Harrilee made their first contribution in [#10076](https://github.com/ai-dynamo/dynamo/pull/10076).
* @andyluo7 made their first contribution in [#9929](https://github.com/ai-dynamo/dynamo/pull/9929).
* @Change72 made their first contribution in [#10157](https://github.com/ai-dynamo/dynamo/pull/10157).
* @mvillmow made their first contribution in [#9095](https://github.com/ai-dynamo/dynamo/pull/9095).
* @Muqi1029 made their first contribution in [#10203](https://github.com/ai-dynamo/dynamo/pull/10203).
* @hatemfaheem made their first contribution in [#9556](https://github.com/ai-dynamo/dynamo/pull/9556).
* @yuanchen8911 made their first contribution in [#8776](https://github.com/ai-dynamo/dynamo/pull/8776).
* @ritazh made their first contribution in [#8355](https://github.com/ai-dynamo/dynamo/pull/8355).
* @aryanputta made their first contribution in [#10281](https://github.com/ai-dynamo/dynamo/pull/10281).
* @kangclzjc made their first contribution in [#10124](https://github.com/ai-dynamo/dynamo/pull/10124).
* @RitwijParmar made their first contribution in [#10095](https://github.com/ai-dynamo/dynamo/pull/10095).
* @yuanwu2017 made their first contribution in [#7990](https://github.com/ai-dynamo/dynamo/pull/7990).
* @treydock made their first contribution in [#10597](https://github.com/ai-dynamo/dynamo/pull/10597).
* @VadimEisenberg made their first contribution in [#10595](https://github.com/ai-dynamo/dynamo/pull/10595).
* @BugenZhao made their first contribution in [#9206](https://github.com/ai-dynamo/dynamo/pull/9206).
* @waynehacking8 made their first contribution in [#10616](https://github.com/ai-dynamo/dynamo/pull/10616).
* @atomic made their first contribution in [#10827](https://github.com/ai-dynamo/dynamo/pull/10827).
* @yankay made their first contribution in [#10861](https://github.com/ai-dynamo/dynamo/pull/10861).
* @pallavijaini0525 made their first contribution in [#9252](https://github.com/ai-dynamo/dynamo/pull/9252).

If you would like to get involved, please see our [Contribution Guide](https://docs.nvidia.com/dynamo/dev/getting-started/contribution-guide).