> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Dynamo v1.2.0

Dynamo v1.2.0 is the 15th feature release of the open-source distributed inference platform. It expands **multimodal serving** (text-to-image on TensorRT-LLM, formalized audio/video streaming, and media-aware KV routing), scales the standalone KV indexer to **branch-sharded, anchor-aware** routing, and promotes the **DynamoGraphDeployment (DGD)** Kubernetes API to served `v1beta1`. It also productizes the **inter-pod GPU Memory Service**, extends **Dynamo Snapshot** to OpenShift and multi-namespace deployments, and adds **DeepSeek-V4 (Flash and Pro) recipes**, native on vLLM 0.20.1 and on SGLang via an upstream preview image.

Breaking changes and deprecations for this release are tracked on the [Deprecations](/dynamo/dev/reference/releases/deprecations#v120) ledger; known issues on the [Known Issues](/dynamo/dev/reference/releases/known-issues#v120) page. Key dependency pins live on [Compatibility](/dynamo/dev/reference/compatibility); shipped artifacts on [Release Artifacts](/dynamo/dev/reference/release-artifacts). Model early access builds (`vX.Y.Z-<model>-dev.N`) are tracked in [Model Early Access Builds](/dynamo/dev/reference/model-early-access-builds).

## Highlights

## Features & Improvements

### Scheduling

* **KV Router Operational Robustness:** Increased the JSON body-size limit to 8 MiB so the standalone `/query` endpoint can handle 1M-token payloads ([#8315](https://github.com/ai-dynamo/dynamo/pull/8315)), allowed unit block sizes in slot tracking with a bandwidth-flood warning ([#8395](https://github.com/ai-dynamo/dynamo/pull/8395)), added batch-level `start_position` support to flat KV events for absolute position restoration ([#8426](https://github.com/ai-dynamo/dynamo/pull/8426)), introduced a warning-only metrics path for duplicate store replays ([#8473](https://github.com/ai-dynamo/dynamo/pull/8473)), filtered non-main ZMQ KV event groups via `group_idx` to prevent mis-indexing of DeepSeek V4 auxiliary cache state ([#8669](https://github.com/ai-dynamo/dynamo/pull/8669)), added forward-compatible `kv_cache_spec_kind` parsing for hybrid vLLM cache metadata ([#8751](https://github.com/ai-dynamo/dynamo/pull/8751)), resolved a startup crash by polling discovery until a worker model card appears ([#8765](https://github.com/ai-dynamo/dynamo/pull/8765)), and fixed post-clear recovery replay so buffer queries skip events prior to the last clear barrier ([#9296](https://github.com/ai-dynamo/dynamo/pull/9296)).
* **Branch-Sharded KV Router Indexer:** Added `BranchShardedIndexer`, a sharding strategy that partitions the prefix space by conversation branch and routes queries to exactly one shard using FNV-hashed prefix blocks for linear read-throughput scaling ([#7859](https://github.com/ai-dynamo/dynamo/pull/7859)), with a write-triggered cleanup task that prunes stale compressed-tree children to reduce memory footprint ([#8127](https://github.com/ai-dynamo/dynamo/pull/8127)), a fix that registers intermediate FNV values at each depth to prevent false cache misses for short queries ([#8939](https://github.com/ai-dynamo/dynamo/pull/8939)), an opt-in anchor-aware mode that installs shared structural anchors at divergence points and lets compressed radix-tree reads walk anchored suffixes directly ([#9007](https://github.com/ai-dynamo/dynamo/pull/9007)), re-exported the anchor-backed implementation as the default `BranchShardedIndexer` path with corrected anchor bookkeeping on worker removal ([#9318](https://github.com/ai-dynamo/dynamo/pull/9318)), and renamed the internal type while preserving the public alias ([#9333](https://github.com/ai-dynamo/dynamo/pull/9333)).
* **Mooncake Multi-Tier KV Routing:** Added multi-tier KV cache indexing supporting device, host, and disk storage tiers with KV consolidation mode selection and external sequence hash tracking ([#8380](https://github.com/ai-dynamo/dynamo/pull/8380)), extended the remote indexer wire protocol to carry tiered match data end-to-end so host/disk continuation matching works correctly under `use_remote_indexer=true` ([#8632](https://github.com/ai-dynamo/dynamo/pull/8632)), aligned the standalone HTTP indexer with Mooncake RFC #1403 by unifying the tier-walk implementation and reshaping `/query`/`/query_by_hash` responses to expose per-instance tier breakdowns ([#8912](https://github.com/ai-dynamo/dynamo/pull/8912)), and updated the standalone-indexer documentation to reflect the new multi-tier response schema including `instances` map, cumulative tier semantics, and `additional_salt`/`cache_salt` fields ([#9247](https://github.com/ai-dynamo/dynamo/pull/9247)).
* **Global Planner Pool Arbiter:** Added aggregated routing mode to the global router with TTFT × ITL 2D grid pool selection and priority overrides from agent hints ([#8044](https://github.com/ai-dynamo/dynamo/pull/8044)), introduced a `--min-total-gpus` floor with paired intent-cache arbitration that searches for opposite-direction scale intents across managed DGDs to keep the cluster within bounds ([#8633](https://github.com/ai-dynamo/dynamo/pull/8633)), fixed the GlobalPlanner health endpoint never registering by awaiting serve endpoints concurrently instead of sequentially ([#8682](https://github.com/ai-dynamo/dynamo/pull/8682)), and moved per-request target TTFT/ITL fields into the `nvext.router` sub-object with schema validation at deserialization time ([#9845](https://github.com/ai-dynamo/dynamo/pull/9845)).
* **Concurrent Radix-Tree Router:** Defaulted approximate routing to the concurrent radix tree when router event threads exceed one, with support for recording precomputed routing-decision hashes and updated CLI help ([#9219](https://github.com/ai-dynamo/dynamo/pull/9219)), optimized child insertion by gating only the first child with a shape write lock and removed a redundant in-plan shape-version retry check while refactoring mooncake bench helpers for repeated runs with percentile summaries and improving prompt trie stale-lookup repair via BFS ([#9076](https://github.com/ai-dynamo/dynamo/pull/9076)), and implemented `block_count()` and `worker_count()` for the `SyncIndexer` so shard sizes report accurate values in benchmarks ([#8559](https://github.com/ai-dynamo/dynamo/pull/8559)).
* **LoRA Routing Foundation Layer:** Added foundational LoRA allocation primitives including the `LoraAllocator` trait with `compute_replica_set()`, slot-aware state tracking, and shared constants as the base layer for the multi-PR LoRA allocation system ([#8177](https://github.com/ai-dynamo/dynamo/pull/8177)), and fixed LoRA adapter registration in prefill workers to use `ModelType.Prefill` instead of `Chat | Completions`, preventing chat requests from being incorrectly routed to prefill workers that would stall waiting for KV transfers ([#8945](https://github.com/ai-dynamo/dynamo/pull/8945)).
* **KV Router Metrics and Reliability:** Added worker-side ZMQ KV relay counters for received, accepted, and filtered events, exposed normalizer filter reasons for observability, and consolidated KV publisher metrics into the existing KV router metrics module ([#9233](https://github.com/ai-dynamo/dynamo/pull/9233)), with a follow-up that deferred JetStream startup orphan cleanup by snapshotting candidates before a grace period to avoid treating newly started routers as orphans ([#9303](https://github.com/ai-dynamo/dynamo/pull/9303)).
* **Per-Worker Router Configuration:** Exposed an optional `router_config` field in the Model Deployment Card so individual worker sets can override the frontend's global router config, for different routing strategies (e.g., device-aware weighted) for different workers within the same deployment ([#8290](https://github.com/ai-dynamo/dynamo/pull/8290)).
* **Shared KV Cache Routing:** Added an interface for third-party shared KV caches to integrate into the Dynamo router, so routing decisions that account for both local device cache hits and shared cache state using a configurable `shared_cache_multiplier` (default 0.5) can scale shared cache matches relative to device-local matches ([#7536](https://github.com/ai-dynamo/dynamo/pull/7536)).
* **Approximate Pruning TTL-Only Mode:** Refactored KV Router approximate-mode pruning to use exclusively TTL-based eviction, removing `router_max_tree_size` and `router_prune_target_ratio` count-based configuration parameters while keeping normal KV event ingestion fire-and-forget ([#8893](https://github.com/ai-dynamo/dynamo/pull/8893)).
* **Prefill Load Scoring Refinement:** Replaced the hidden `DYN_ROUTER_LOAD_BLOCK_SIZE` scoring override with explicit block-space prefill scoring, added a new `prefill_load_scale` configuration parameter (`DYN_ROUTER_PREFILL_LOAD_SCALE` / `--router-prefill-load-scale`) plumbed through Rust, Python, and C bindings, and introduced a `token-dp-balance` frontend router mode that uses KV scheduler load balancing without prefix matching or KV event/indexer paths ([#9267](https://github.com/ai-dynamo/dynamo/pull/9267)).
* **Profiler and Rapid-Mode Hardening:** Fixed model path doubling when `modelCache` lacks `pvcModelPath` ([#8449](https://github.com/ai-dynamo/dynamo/pull/8449)), wired mocker-rapid to direct AIConfigurator (AIC) flags and removed dead profiler AIC interpolation code ([#8455](https://github.com/ai-dynamo/dynamo/pull/8455)), threaded the deployment's total GPU budget into AIC enumeration so candidates respect cluster size ([#8479](https://github.com/ai-dynamo/dynamo/pull/8479)), enforced `totalGpus` as a hard cap in rapid-mode DGD generation to prevent over-provisioning ([#8617](https://github.com/ai-dynamo/dynamo/pull/8617)), corrected `PickedParallelConfig.num_gpus` to reflect physical GPU count rather than overcounting attention DP ([#8610](https://github.com/ai-dynamo/dynamo/pull/8610)), added an explicit replay optimization objective knob defaulting to throughput ([#8518](https://github.com/ai-dynamo/dynamo/pull/8518)), resolved decode benchmarking failures by using actual deployment component names for log path lookup ([#8733](https://github.com/ai-dynamo/dynamo/pull/8733)), and hardened Pareto computation to skip NaN/inf/non-numeric points gracefully ([#9169](https://github.com/ai-dynamo/dynamo/pull/9169)).
* **Planner Advisory Mode Scaling Logic:** Added an `advisory` scaling mode so operators can observe planner decisions (replica recommendations, structured summary logs, Prometheus metrics, Plotly HTML reports) without applying changes, while the full pipeline runs identically to `active` mode ([#8244](https://github.com/ai-dynamo/dynamo/pull/8244)), with interactive HTML diagnostics and throughput-regression bootstrapping wired into offline planner-in-the-loop replay along with several scaling-logic correctness fixes surfaced by the new reports ([#8280](https://github.com/ai-dynamo/dynamo/pull/8280)), KV-cache hit-rate awareness threaded through load and throughput scaling paths so prefill compute is discounted on reuse-heavy workloads ([#8314](https://github.com/ai-dynamo/dynamo/pull/8314)), AIC interpolation moved in-process to the planner for rapid pre-deployment sweeping and MoE-DEP correctness fixes for models like Qwen3-235B-A22B-FP8 ([#8335](https://github.com/ai-dynamo/dynamo/pull/8335)), and scoped test stubs to prevent advisory-mode fixtures from leaking into other planner test modules ([#8418](https://github.com/ai-dynamo/dynamo/pull/8418)).
* **A30 GPU SKU Support:** Added the NVIDIA A30 to the DGDR `hardware.gpuSku` enum across the Go API, generated Python model, CRD, Helm CRD, and API docs, and taught GPU discovery to infer `a30` from A30 product labels ([#9176](https://github.com/ai-dynamo/dynamo/pull/9176)).
* **Discovery Concurrency Race Fixes:** Addressed multiple concurrency edge cases in the discovery service's `handle_put` lifecycle, including an abort-cleanup race where completed tasks could delete a newer task's handle, a poll timeout guard for `recover_concurrent_registration`, and a delete timeout to avoid blocking the watch loop indefinitely ([#8237](https://github.com/ai-dynamo/dynamo/pull/8237)), added the missing `Audios` variant to `is_model_type_list_empty` which previously caused spurious `ModelUpdate::Removed` emissions for audio models ([#8241](https://github.com/ai-dynamo/dynamo/pull/8241)), moved `ModelUpdate::Added` notification to after `add_worker_set` completes so HTTP endpoints are not exposed before a serving pipeline exists ([#8242](https://github.com/ai-dynamo/dynamo/pull/8242)), and replaced the 100ms polling loop in `recover_concurrent_registration` with a `tokio::sync::Notify` plus `enable()` pattern using an absolute deadline to prevent spurious wakeups from extending the total wait ([#8291](https://github.com/ai-dynamo/dynamo/pull/8291)).
* **Default TCP Worker Pool:** Raised the default TCP worker pool size to 10,000 to prevent unexpected performance drops when benchmarking at concurrency levels above 1,500 ([#9090](https://github.com/ai-dynamo/dynamo/pull/9090)).
* **Lock-Free TCP Pool Infrastructure:** Replaced the mutex-based TCP connection pool with a lock-free LRU design backed by `ArcSwap` + atomic round-robin + `SegQueue`, closing five correctness gaps including unbounded-queue OOM, RAII guard leaks that inflated the inflight counter, and a drop-before-first-poll scheduler leak ([#7806](https://github.com/ai-dynamo/dynamo/pull/7806)), replaced `panic!` calls with `warn + break` on TCP stream read errors (`ECONNRESET`) so connection resets no longer abort Tokio tasks under high concurrency ([#8254](https://github.com/ai-dynamo/dynamo/pull/8254)), and introduced per-stream buffer garbage collection with a configurable threshold to reclaim memory after large transfers while raising the frame-size ceiling for multi-modal embeddings ([#8420](https://github.com/ai-dynamo/dynamo/pull/8420)).

### Kubernetes Deployment

* **Snapshot Checkpoint-Restore Hardening:** Added DRA-aware GPU UUID resolution from allocated claims for the snapshot agent ([#8292](https://github.com/ai-dynamo/dynamo/pull/8292)), replaced the PID-1 SIGUSR1/SIGCONT signaling contract with file sentinels and fixed a shell-form entrypoint unwrap panic ([#8403](https://github.com/ai-dynamo/dynamo/pull/8403)), introduced a pluggable `Runtime` interface with containerd and CRI-O backends plus OpenShift chart enablement ([#8427](https://github.com/ai-dynamo/dynamo/pull/8427)), adopted an annotation-driven contract for multi-container checkpoint/restore targeting ([#8631](https://github.com/ai-dynamo/dynamo/pull/8631)), blocked the unsupported snapshot-plus-GPU-Memory-Service combination via admission validation ([#8689](https://github.com/ai-dynamo/dynamo/pull/8689)), enabled a single cross-namespace snapshot-agent DaemonSet using a new `podMount` PVC access mode ([#8740](https://github.com/ai-dynamo/dynamo/pull/8740)), reworked seccomp configuration to support OpenShift checkpoint flows with a structured `seccomp` config block ([#8902](https://github.com/ai-dynamo/dynamo/pull/8902)), and preserved the vLLM torch compile cache in rootfs diffs to prevent CRIU restore failures from missing Triton shared objects ([#9943](https://github.com/ai-dynamo/dynamo/pull/9943)).
* **Operator GPU Hardware Discovery:** Expanded hardware detection to cover Blackwell, Hopper, Ampere, Ada Lovelace, older NVIDIA, and AMD GPU families with interconnect and RDMA/SR-IOV profiling ([#7551](https://github.com/ai-dynamo/dynamo/pull/7551)), fixed a crash when `totalGpus` was missing by failing cleanly instead of proceeding with incomplete hardware info ([#8267](https://github.com/ai-dynamo/dynamo/pull/8267)), added inference fallbacks for unrecognized form factors such as defaulting to SXM when no PCIe variant exists ([#8507](https://github.com/ai-dynamo/dynamo/pull/8507)), hardened enrichment validation with nil guards and a `gpu.DiscoverGPUs` fallback when DCGM fails ([#8508](https://github.com/ai-dynamo/dynamo/pull/8508)), introduced a SKU-filtered GFD node-label discovery path for environments where DCGM pods are unavailable ([#8510](https://github.com/ai-dynamo/dynamo/pull/8510)), and documented the semantics and auto-discovery behavior of the new Interconnect and RDMA HardwareSpec fields ([#8300](https://github.com/ai-dynamo/dynamo/pull/8300)).
* **Operator v1beta1 API Migration:** Introduced `v1beta1` API types for DynamoGraphDeployment, DynamoComponentDeployment, and DynamoGraphDeploymentScalingAdapter with a cleaned-up surface replacing per-component fields with `podTemplate` and renaming `services` to `components` ([#8414](https://github.com/ai-dynamo/dynamo/pull/8414)), implemented lossless `v1alpha1 ↔ v1beta1` conversion and flipped `v1beta1` to served ([#8647](https://github.com/ai-dynamo/dynamo/pull/8647)), added round-trip fuzz coverage and fixed DGDR conversion mutability bugs ([#9164](https://github.com/ai-dynamo/dynamo/pull/9164)), migrated the DGSA controller to reconcile the v1beta1 shape ([#9194](https://github.com/ai-dynamo/dynamo/pull/9194)), migrated the DGD/DCD controllers to read v1beta1 objects while preserving upgrade/downgrade behavior ([#9235](https://github.com/ai-dynamo/dynamo/pull/9235)), and aligned DGDR conversion to the structural rules with sparse annotation preservation for downgrade compatibility ([#9262](https://github.com/ai-dynamo/dynamo/pull/9262)).
* **Istio Mesh Support Fixes:** Added automatic generation of Istio DestinationRule resources for EPP deployments in the Dynamo Operator, preventing double-TLS issues in Istio service meshes that previously required manual DestinationRule creation ([#8270](https://github.com/ai-dynamo/dynamo/pull/8270)).
* **Inter-Pod GMS Failover Support:** Added inter-pod GPU Memory Service failover for vLLM deployments, where each rank receives a dedicated GMS weight-server pod and one or more engine pods sharing GPUs via Kubernetes DRA ResourceClaims and rank-isolated hostPath volumes for CUDA IPC, with a failover cascade controller for fast group-wide cleanup on engine failure ([#7777](https://github.com/ai-dynamo/dynamo/pull/7777)). Introduced scratch-aliased KV cache initialization that preserves VA reservations across sleep/wake cycles so cudagraphs replay correctly during shadow-failover transitions ([#8686](https://github.com/ai-dynamo/dynamo/pull/8686)), fixed SGLang GMS startup by auto-enabling the memory saver path when `setup_gms()` is used ([#9647](https://github.com/ai-dynamo/dynamo/pull/9647)), and corrected the vLLM GMS worker to use the DP-adjusted CUDA device before connecting to GMS, preventing rank collisions under data parallelism ([#9840](https://github.com/ai-dynamo/dynamo/pull/9840)).
* **Helm Operator Configuration Hardening:** Added a temporary admission-level gate that rejects services combining GPU Memory Service with Snapshot unless explicitly opted in via the `dynamo-operator.featureGates.gmsSnapshot` Helm value ([#8829](https://github.com/ai-dynamo/dynamo/pull/8829)), consolidated the previously split NATS subchart flags into a single `global.nats.install` value with validation templates that fail on stale overrides ([#9232](https://github.com/ai-dynamo/dynamo/pull/9232)), and fixed remaining references to the removed flags in the Tiltfile and chart documentation ([#9281](https://github.com/ai-dynamo/dynamo/pull/9281)).

### Multimodal & Diffusion

* **Multimodal Media Fetcher Hardening:** Centralized URL validation into a shared `url_validator` module enforcing scheme, local-path, and redirect policies across audio, image, and video loaders ([#8282](https://github.com/ai-dynamo/dynamo/pull/8282)), wired the `DYN_MM_ALLOW_INTERNAL` env var through to the Rust `MediaFetcher` to honor localhost/on-prem semantics and unblock nightly tests ([#8535](https://github.com/ai-dynamo/dynamo/pull/8535)), hardened the Rust side against redirect bypass and DNS rebinding with RFC-based IP blocklists, cloud-metadata hostname checks, and a custom `BlocklistResolver` on the `reqwest::Client` ([#8569](https://github.com/ai-dynamo/dynamo/pull/8569)), introduced a dual-backend HTTP client supporting both httpx and aiohttp to improve throughput under high concurrency ([#8646](https://github.com/ai-dynamo/dynamo/pull/8646)), bounded total in-flight media fetches with a global semaphore and decoupled httpx pool/read timeouts via `DYN_MM_HTTP_*` env vars to eliminate `PoolTimeout` errors ([#8657](https://github.com/ai-dynamo/dynamo/pull/8657)), and added an opt-in byte-budgeted LRU cache to the Rust `MediaLoader` that skips network fetch, decode, and NIXL registration on cache hits ([#8863](https://github.com/ai-dynamo/dynamo/pull/8863)).
* **Multimodal Benchmarks Tooling Enhancements:** Added a sliding-window JSONL generation subcommand for exercising prefix-caching under interleaved multi-user image streams, refactoring argument parsing into composable parent parsers and subcommands ([#8201](https://github.com/ai-dynamo/dynamo/pull/8201)), fixed grouped single-turn sweeps to use `--conversation-num` so sessions run to completion without re-dispatch distortion ([#8458](https://github.com/ai-dynamo/dynamo/pull/8458)), and introduced multithreading with configurable processing delay to the local media server for production-realistic latency simulation ([#8822](https://github.com/ai-dynamo/dynamo/pull/8822)).
* **SGLang Diffusion Video Pipeline Support:** Added video input support for SGLang in the disaggregated prefill/decode path ([#8597](https://github.com/ai-dynamo/dynamo/pull/8597)), fixed a startup crash caused by a missing `enable_trace` field in the diffusion worker `ServerArgs` stub ([#8332](https://github.com/ai-dynamo/dynamo/pull/8332)), added the `accelerate` package to the SGLang runtime Docker image so diffusion pipelines can initialize successfully ([#8357](https://github.com/ai-dynamo/dynamo/pull/8357)), and preserved file permissions in the dev container venv setup to prevent the bundled `ffmpeg` binary from losing execute bits ([#8408](https://github.com/ai-dynamo/dynamo/pull/8408)).
* **Multimodal-Aware KV Routing Path:** Added a Rust-frontend multimodal routing path that extracts per-image hashes, dimensions, and token counts (via per-VLM-family math) before selecting a worker, for KV-cache-aware placement of multimodal workloads ([#9272](https://github.com/ai-dynamo/dynamo/pull/9272)), with a plumbing-only PR that wired the `lightseek-mm` cargo feature and exposed `LightseekMmCounter` and token-ID resolver helpers from the `llm-multimodal` crate ([#9352](https://github.com/ai-dynamo/dynamo/pull/9352)), and documented that Qwen3.5/Qwen3.6 models are not yet supported ([#9572](https://github.com/ai-dynamo/dynamo/pull/9572)).
* **Multimodal Routing Frontend Integration:** Moved multimodal-aware KV routing from the separate MM Router Worker process into the frontend's vLLM processor, eliminating an extra network hop and redundant image processing by transferring pre-processed `mm_kwargs` via shared memory (\~2ms) or NIXL RDMA, with automatic fallback to URL reprocessing for cross-node deployments ([#8065](https://github.com/ai-dynamo/dynamo/pull/8065)).
* **TRT-LLM Multimodal Image Support:** Added text-to-image generation capability to the TRT-LLM backend with a new `/v1/images/generations` API endpoint and Flux pipeline integration ([#8200](https://github.com/ai-dynamo/dynamo/pull/8200)), and hardened the multimodal processor by replacing `torch.load()` with safetensors loading and Rust frontend media decoding to eliminate arbitrary code execution risks ([#8295](https://github.com/ai-dynamo/dynamo/pull/8295)).
* **Model Family Lookup Resolver:** Introduced `ModelFamily` and `resolve_model_family` for vLLM multimodal encode-related handling in disaggregated prefill, replacing eight inconsistent per-callsite dispatch paths that failed when models were loaded from local paths not matching HF ids ([#8973](https://github.com/ai-dynamo/dynamo/pull/8973)).
* **NIXL Canonical Memtype XPU Support:** Refactored NIXL memory type classification to use canonical segment names (VRAM/DRAM) instead of device-specific strings for forward compatibility, added Intel XPU (Level-Zero) device support for disaggregated encode/prefill-decode transfers, and fixed XPU kernel synchronization to prevent race conditions before NIXL transfers ([#9073](https://github.com/ai-dynamo/dynamo/pull/9073)).
* **Formalized Video Streaming Protocol:** Introduced a dedicated `data_source` field in the audio protocol to disambiguate the overloaded `response_format` field, added an `output_format` container field to the video protocol, and extended the `/v1/videos` HTTP endpoint to support SSE streaming output in addition to batch JSON responses, for a consistent internal interface across all modalities ([#8491](https://github.com/ai-dynamo/dynamo/pull/8491)).
* **SGLang Frontend Decoding Support:** Added `--frontend-decoding` flag for the SGLang backend so aggregated multimodal requests are decoded by the Rust frontend and shipped via NIXL RDMA, removing the GIL-bound image-decode hop from the worker's P0 thread ([#9405](https://github.com/ai-dynamo/dynamo/pull/9405)).
* **Image UUID Benchmark Emission:** Added optional `--uuid` flag to the multimodal JSONL generator that emits deterministic `image_uuids` alongside `images`, so vLLM's cached-mm processor can short-circuit on cache hits across benchmark restarts and image reuse ([#8854](https://github.com/ai-dynamo/dynamo/pull/8854)).

### Frontend & Agents

* **Agent Context and Tool Relay:** Added passive `nvext.agent_context` parsing and a normalized agent trace bus with configurable `jsonl`/`stderr` sinks that emit chat-completion `request_end` records with Dynamo request metrics ([#8789](https://github.com/ai-dynamo/dynamo/pull/8789)), introduced a ZMQ-to-event-plane relay with a narrow `AgentToolEventRelay` Python binding that captures tool lifecycle events from external harnesses ([#8790](https://github.com/ai-dynamo/dynamo/pull/8790)), hardened the ingest topology so Dynamo owns the ZMQ `PULL` bind side, allowing multiple producers to connect without endpoint conflicts, and added multi-producer regression coverage ([#9105](https://github.com/ai-dynamo/dynamo/pull/9105)), and aligned agent context identifiers with ATIF terminology (`session_type_id`, `session_id`, `trajectory_id`, `parent_trajectory_id`) while splitting the Agents documentation into focused pages ([#9140](https://github.com/ai-dynamo/dynamo/pull/9140)).
* **Nvext Response Metadata Gating:** Fixed a regression where plain OpenAI-compatible requests leaked `nvext` response metadata (`worker_id`, `timing`) by default, introducing `NvExtResponseFieldSelection` to gate each response field independently behind `extra_fields` opt-in while preserving the `query_instance_id` exception ([#8252](https://github.com/ai-dynamo/dynamo/pull/8252)).
* **DeepSeek V4 Parser Support:** Added a DeepSeek V4 DSML tool-call parser and reasoning parser to the Rust parser crate, registering canonical names and compatibility aliases (`deepseek_v4`, `deepseek-v4`, `deepseekv4`) with unit coverage for tool parsing, streaming reasoning, and alias resolution ([#8665](https://github.com/ai-dynamo/dynamo/pull/8665)), hardened the V4 formatter so `thinking: false`, `enable_thinking: false`, `thinking_mode: "chat"`, `reasoning_effort="max"`, and per-request `drop_thinking` overrides all route end-to-end correctly ([#8670](https://github.com/ai-dynamo/dynamo/pull/8670)), fixed tool-continuation reasoning parsing where a prompt-injected `<think>` seed caused the closing tag to leak as normal content without regressing Kimi K2.5 behavior ([#8901](https://github.com/ai-dynamo/dynamo/pull/8901)), and deduplicated the DeepSeek V3.2 and V4 prompt encoders into a shared `deepseek_common` module to reduce code duplication ([#9322](https://github.com/ai-dynamo/dynamo/pull/9322)).
* **Self-Host MDC Metadata Pipeline:** Added worker-side metadata artifact hosting via `system_status_server`, where `LocalModel::attach()` registers local-disk MDC files in a process-local registry and rewrites paths to HTTP URLs served by the worker ([#8855](https://github.com/ai-dynamo/dynamo/pull/8855)), implemented a frontend verify-and-cache resolution pipeline that derives URIs per `CheckedFile`, fetches via scheme handlers, blake3-verifies, and atomically publishes to a content-addressed cache ([#9057](https://github.com/ai-dynamo/dynamo/pull/9057)), and fixed a regression where sibling files like `preprocessor_config.json` and `tokenizer.model` were missing from the slug directory by adding a harvest pass that symlinks non-weight siblings from resolved directories ([#9610](https://github.com/ai-dynamo/dynamo/pull/9610)).
* **OpenAI Multi-Choice N Support:** Added OpenAI-compatible `n` parameter plumbing to the Dynamo frontend and response contract so requests with `n > 1` return multiple choices, with vLLM backend integration ([#8744](https://github.com/ai-dynamo/dynamo/pull/8744)), SGLang backend support using its native multi-sequence generation ([#8745](https://github.com/ai-dynamo/dynamo/pull/8745)), and TensorRT-LLM backend support passing `n` through to its sampling params and keeping streamed choices separated ([#8746](https://github.com/ai-dynamo/dynamo/pull/8746)).
* **Cross-Impl Parser Parity Harness:** Added a pytest-driven parity harness that exercises Dynamo, vLLM, and SGLang parsers against shared YAML fixtures and diffs their outputs to surface cross-implementation divergences ([#9186](https://github.com/ai-dynamo/dynamo/pull/9186)), expanded coverage from 7 to all 19 registered Dynamo parser families with new fixtures and xfail-tracked divergences ([#9261](https://github.com/ai-dynamo/dynamo/pull/9261)), and documented the end-to-end workflow for finding, reproducing, fixing, and retiring divergences ([#9394](https://github.com/ai-dynamo/dynamo/pull/9394)).
* **Bidirectional Streaming WebSocket Frontend:** Added a `/v1/realtime` WebSocket endpoint that accepts `NvCreateChatCompletionRequest` JSON frames over a single connection and streams `NvCreateChatCompletionStreamResponse` chunks back, for realtime bidirectional communication as the first slice of the streaming-input feature ([#9079](https://github.com/ai-dynamo/dynamo/pull/9079)).

### TensorRT-LLM

* **TRT-LLM Metrics and KV Publisher:** Added ForwardPassMetrics publishing via a PyO3 FpmDirectPublisher for non-attention-DP TensorRT-LLM workers, so the Planner can treat them equivalently to vLLM workers for autoscaling and latency prediction ([#8356](https://github.com/ai-dynamo/dynamo/pull/8356)), fixed the `trtllm_kv_transfer_success_total` Prometheus counter that was never incremented due to a mutually exclusive gate between prefill mode and the decode-side timing check ([#8483](https://github.com/ai-dynamo/dynamo/pull/8483)), and eliminated a \~148 ms TTFT regression when `--publish-events-and-metrics` was enabled by moving the publisher off the request loop and batching KV-cache walks ([#8892](https://github.com/ai-dynamo/dynamo/pull/8892)).
* **TRT-LLM Canary Health Checks:** Set highest priority (1.0) on the canary health-check request to prevent false-negative timeouts when long-context inference requests starved the probe under load ([#8488](https://github.com/ai-dynamo/dynamo/pull/8488)), and fixed disaggregated decode workers staying permanently NotReady by injecting explicit `disaggregated_params` into the canary payload so the handler no longer rejects probes that lack prefill-peer context ([#8521](https://github.com/ai-dynamo/dynamo/pull/8521)).
* **Disable GC for TRT-LLM:** Added support for disabling Python garbage collection in the Dynamo TRT-LLM worker when `TRTLLM_SERVER_DISABLE_GC` or `DYN_TRTLLM_SERVER_DISABLE_GC` is set, preventing uvloop stalls at high concurrency and performance parity with trtllm-serve ([#9096](https://github.com/ai-dynamo/dynamo/pull/9096)).
* **TRT-LLM Wakeup RPC Caching:** Improved wakeup call handling to preserve original exceptions and cache the resolved RPC method name after first invocation, eliminating redundant lookups on subsequent calls ([#8255](https://github.com/ai-dynamo/dynamo/pull/8255)).

### Performance Modeling & Replay

* **Performance Modeling Offline Replay:** Added KV Block Manager (KVBM)-backed G1↔G2 offload simulation for the vLLM mocker in both online and offline replay modes ([#8184](https://github.com/ai-dynamo/dynamo/pull/8184)), simulated non-zero worker startup time in the offline discrete-event engine with `WorkerReady` events and `pending_startup` tracking ([#8231](https://github.com/ai-dynamo/dynamo/pull/8231)), hardened replay-router scaling invariants including single-worker queueing, 0→N recovery, and draining-state preservation ([#8236](https://github.com/ai-dynamo/dynamo/pull/8236)), introduced an agentic trace file format with loader and multi-turn smoke coverage ([#8627](https://github.com/ai-dynamo/dynamo/pull/8627)), published tiered `HostPinned` KV events for G2 offload with a new `StorageTier`-aware event sink ([#8961](https://github.com/ai-dynamo/dynamo/pull/8961)), and captured replay-friendly hashes on agent `request_end` records with a converter producing Mooncake JSONL for mocker replay ([#8998](https://github.com/ai-dynamo/dynamo/pull/8998)).
* **Claude Trace Exporter Tool:** Added a privacy-preserving Claude raw-trace exporter under `benchmarks/coding/claude`, supporting autodiscovery, compaction-aware parsing, structural sidecar output, and configurable tokenization for accurate output token computation ([#8096](https://github.com/ai-dynamo/dynamo/pull/8096)).

### Infrastructure Modernization

* **Context-Aware Event Plane Defaults:** Made the event-plane selection context-aware so local-only discovery backends (`file`/`mem`) default to ZMQ while distributed backends (`etcd`/`kubernetes`) default to NATS, eliminating unconditional NATS connections at startup ([#8398](https://github.com/ai-dynamo/dynamo/pull/8398)), removed the Python CLI's hard-coded `DYN_EVENT_PLANE=nats` override that still forced NATS connections and caused "Connection refused" failures in local workflows ([#8614](https://github.com/ai-dynamo/dynamo/pull/8614)), and removed hard-coded `NATS_SERVER` and `ETCD_ENDPOINTS` environment variables from profiling job configuration to respect the same infrastructure-aware defaults ([#9271](https://github.com/ai-dynamo/dynamo/pull/9271)).

### Fault Tolerance & Observability

* **Frontend Staged Request Gauges:** Added `dynamo_frontend_active_requests` and `dynamo_frontend_stage_requests` gauges with per-stage (preprocess, route, dispatch) and per-phase (prefill/decode/aggregated) inflight counts, backed by a `StageGuard` RAII type that increments on creation and decrements on drop ([#8162](https://github.com/ai-dynamo/dynamo/pull/8162)), and updated the metrics documentation with descriptions of the new gauges and deprecation notes for superseded ones ([#8459](https://github.com/ai-dynamo/dynamo/pull/8459)).
* **Dynamo Local Resource Monitor:** Added a lightweight, high-frequency (200 ms) per-process resource monitor that tracks VRAM, GPU, PCIe, CPU, disk, and network usage for Dynamo processes on a single host, with an accompanying Grafana dashboard integrated into the existing observability stack ([#9055](https://github.com/ai-dynamo/dynamo/pull/9055)).
* **Worker-Pool Saturation Metrics:** Added six Prometheus metrics (`dynamo_work_handler_queue_depth`, `queue_capacity`, `enqueue_rejected_total`, `permit_wait_seconds`, `pool_active_tasks`, `pool_capacity`) to the shared TCP server, so operators can detect queue buildup and permit starvation before workers are OOM-killed ([#8412](https://github.com/ai-dynamo/dynamo/pull/8412)).
* **Reusable Telemetry Bus Primitives:** Added a typed telemetry bus, stream completion helper, and async JSONL sink to the LLM library for shared event-recording infrastructure with configurable append mode, buffer sizing, and periodic flush while preserving the existing recorder JSONL format ([#8788](https://github.com/ai-dynamo/dynamo/pull/8788)).
* **Orphaned Pending Request Handling:** Implemented discovery-plane-driven cancellation of pending response-stream subjects when a worker is removed, preventing indefinite hangs where requests queued in a killed worker's bounded channel became permanently stuck; cancelled streams now return a migratable `Disconnected` error so the migration layer can retry on another worker ([#8182](https://github.com/ai-dynamo/dynamo/pull/8182)).

### vLLM

* **vLLM Multinode Elastic EP:** Added test infrastructure for validating vLLM's native elastic expert parallelism across multiple nodes using a warm-standby Ray topology, confirming all scale steps (dp=2→3→4→3→2→4→2) succeed on a 2-node AKS cluster ([#8183](https://github.com/ai-dynamo/dynamo/pull/8183)), and introduced operator support that routes `--enable-elastic-ep` deployments through a cross-node Ray cluster while fixing a concurrent `scale_elastic_ep` race condition that caused 300 s TCPStore timeouts on remote worker nodes during scale-up ([#8216](https://github.com/ai-dynamo/dynamo/pull/8216)).
* **vLLM Start Stop Profile:** Added start/stop profile endpoints (`engine/start_profile` and `engine/stop_profile`) for vLLM to maintain parity with the SGLang implementation, giving a unified profiling interface ([#8068](https://github.com/ai-dynamo/dynamo/pull/8068)).

### SGLang

* **SGLang Forward Pass Metrics:** Wired SGLang's ForwardPassMetrics ZMQ publisher into Dynamo's event plane via FpmEventRelay, injecting the endpoint instance ID as worker\_id and adding a cross-repo wire-format contract test ([#8154](https://github.com/ai-dynamo/dynamo/pull/8154)), and fixed the previously unpopulated `total_kv_blocks` runtime config by resolving rank-0 scheduler info through SGLang's canonical Engine scheduler path ([#8439](https://github.com/ai-dynamo/dynamo/pull/8439)).
* **SGLang Token ID Logprobs:** Added `return_tokens_as_token_ids` support, integer token-id stop arrays, and gated top-logprobs for SGLang to enable RL workflows while preserving OpenAI-compatible response shapes ([#8119](https://github.com/ai-dynamo/dynamo/pull/8119)).

### Unified Backend (Preview)

* **Unified Disaggregated Serving Abstraction:** Added a common backend framework that lets engines plug into Dynamo's disaggregated-serving path through a shared `LLMEngine` trait/ABC, eliminating the need for each backend to reinvent wire format, registration plumbing, or shutdown orchestration ([#9249](https://github.com/ai-dynamo/dynamo/pull/9249)).
* **Rust Backend Common Framework:** Added a shared Rust backend layer (`dynamo-backend-common`) with an `LLMEngine` trait and `Worker` lifecycle driver, for native Rust backend integrations that follow the same lifecycle and cancellation contract as existing Python backends, along with a `mocker` reference engine for end-to-end pipeline testing ([#8584](https://github.com/ai-dynamo/dynamo/pull/8584)).
* **Rust Worker PyO3 Backend:** Moved the unified backend Worker lifecycle from Python into the Rust `dynamo_backend_common` crate and exposed it to Python engines through a new `dynamo._core.backend.Worker` PyO3 binding, shrinking the Python Worker shim from 259 lines to \~115 while consolidating signal handling, discovery unregister, grace-period sleep, drain, cleanup, and 3-phase runtime shutdown entirely in Rust ([#9202](https://github.com/ai-dynamo/dynamo/pull/9202)).
* **Unified Worker Engine Args:** Enabled the `dynamo.trtllm` unified worker to honor `--extra-engine-args` (YAML) and `--override-engine-args` (JSON) flags, bringing it to parity with the existing worker by porting the YAML+JSON merge sequence and adding validation that parsed overrides are JSON objects ([#8886](https://github.com/ai-dynamo/dynamo/pull/8886)).

### KV Block Manager

* **KVBM-Logical Mocker Backend Replacement:** Replaced the mocker's manual vLLM block manager and evictor with a new `kvbm-logical::BlockManager<G1>` backend using a `Lineage` inactive pool, simplifying block lifecycle management and bridging KVBM's PositionalLineageHash to the router's SequenceHash via a new HashMap ([#8451](https://github.com/ai-dynamo/dynamo/pull/8451)).
* **PositionalLineageHash Ordering Support:** Added `Ord` and `PartialOrd` trait implementations to `PositionalLineageHash`, for deterministic sorting and comparison operations for KVBM token types ([#8687](https://github.com/ai-dynamo/dynamo/pull/8687)).

### TokenSpeed

* **Initial TokenSpeed Backend:** Added a TokenSpeed backend integration through the common LLMEngine path, introducing `python -m dynamo.tokenspeed` with argument parsing, request/sampling conversion, streaming token output, abort/cleanup handling, guided decoding support, and focused unit coverage ([#9212](https://github.com/ai-dynamo/dynamo/pull/9212)).

## Recipes

* **DeepSeek-V4 Recipe Consolidation:** Restructured the DeepSeek-V4 Flash and Pro recipes into a single `recipes/deepseek-v4/` subtree following the repo-wide convention, deduplicated Dockerfiles, and simplified the SGLang Dockerfile to consume the Dynamo donor image directly with a pinned base digest ([#8735](https://github.com/ai-dynamo/dynamo/pull/8735)), performed a post-base cleanup pass removing unnecessary system dependencies from the B200 SGLang image ([#8929](https://github.com/ai-dynamo/dynamo/pull/8929)), refreshed Python and Rust attribution files based on container scans of the runtime images ([#8948](https://github.com/ai-dynamo/dynamo/pull/8948)), bumped deploy manifests to dev.2 public image tags and consolidated apt purge directives ([#8971](https://github.com/ai-dynamo/dynamo/pull/8971)), refreshed the SGLang base image ([#9002](https://github.com/ai-dynamo/dynamo/pull/9002)), updated all recipe manifests to dev.3 images ([#9356](https://github.com/ai-dynamo/dynamo/pull/9356)), and aligned the docs support-matrix and release-artifacts pages to reflect the dev.3 experimental release ([#9358](https://github.com/ai-dynamo/dynamo/pull/9358)).
* **DeepSeek-V4 SGLang Serving Recipes:** Added SGLang serving recipe for DeepSeek-V4-Flash on B200 GPUs with TP4, MXFP4 MoE, and EAGLE speculative decoding via Dynamo frontend ([#8704](https://github.com/ai-dynamo/dynamo/pull/8704)), with Dockerfile PATH fixes for etcd resolution ([#8713](https://github.com/ai-dynamo/dynamo/pull/8713), [#8716](https://github.com/ai-dynamo/dynamo/pull/8716)), published both Flash and Pro SGLang recipes using public NGC images and updated documentation ([#8734](https://github.com/ai-dynamo/dynamo/pull/8734)), added aggregated and disaggregated recipes for DeepSeek-V4-Pro on GB200 with NIXL KV transfer over GKE RDMA ([#8960](https://github.com/ai-dynamo/dynamo/pull/8960)), and added a disaggregated prefill/decode recipe for DeepSeek-V4-Pro on B200 with InfiniBand RDMA including UCX configuration guidance ([#9278](https://github.com/ai-dynamo/dynamo/pull/9278)).
* **DeepSeek-V4 vLLM Serving Recipes:** Added aggregated vLLM recipes for DeepSeek-V4-Flash (4×B200, DP=4+EP) and DeepSeek-V4-Pro (8×B200, TP=8+EP) ([#8668](https://github.com/ai-dynamo/dynamo/pull/8668)), introduced a disaggregated prefill/decode recipe for V4-Pro on GB200 NVL72 using DRA ComputeDomain for cross-node MNNVL with 16 GPUs total ([#8811](https://github.com/ai-dynamo/dynamo/pull/8811)), extended GB200 aggregated recipes for both V4-Flash (TP=4+EP, FP4 path) and V4-Pro (TP=8+EP) ([#8876](https://github.com/ai-dynamo/dynamo/pull/8876)), and pinned `--no-enable-flashinfer-autotune` across all dsv4 vLLM recipes to prevent accuracy regressions ([#9268](https://github.com/ai-dynamo/dynamo/pull/9268)).
* **Qwen3.6-35B Benchmark Recipe:** Added a 3-way Kubernetes benchmark recipe for Qwen/Qwen3.6-35B-A3B-FP8 on a single H100 or GB200, comparing vanilla vLLM, Dynamo with frontend-decoding, and Dynamo with embedding cache across throughput and latency metrics ([#9392](https://github.com/ai-dynamo/dynamo/pull/9392)).
* **Qwen3-32B-FP8 vLLM Recipe:** Added a production-ready vLLM disaggregated single-node recipe for Qwen3-32B-FP8, including deploy and benchmark manifests with a topology of 2× prefill workers (TP=2) and 1× decode worker (TP=4) using NixlConnector KV transfer ([#7915](https://github.com/ai-dynamo/dynamo/pull/7915)).
* **Qwen3-235B Architecture-Specific Recipes:** Split TensorRT-LLM aggregated and disaggregated deployment recipes for Qwen3-235B-A22B-FP8 into separate Hopper and Blackwell subdirectories, eliminating the need for manual YAML editing when targeting different GPU architectures since TRT-LLM 1.3.x requires different `moe_config` settings on each ([#8470](https://github.com/ai-dynamo/dynamo/pull/8470)).
* **Allow Internal URLs Recipe Fix:** Added `DYN_MM_ALLOW_INTERNAL=1` environment variable to the Qwen3-VL-30B vLLM agg-embedding-cache recipe, preventing multimodal benchmark failures caused by the new default-deny URL validator rejecting COCO dataset image URLs ([#9309](https://github.com/ai-dynamo/dynamo/pull/9309)).
* **vLLM Recipe Flag Rename:** Renamed the deprecated `--disable-log-requests` argument to `--no-enable-log-requests` in recipe deployment YAMLs and benchmark documentation, preventing vLLM worker pods from crash-looping on startup with newer vLLM runtimes (0.19.x+) that removed the old flag ([#8693](https://github.com/ai-dynamo/dynamo/pull/8693)).
* **Qwen3 KVBM Recipe Addition:** Removed the Kimi K2.5 KVBM recipe (which required an unreleased container) and added a new Qwen3-32B KVBM recipe with single-GPU aggregated deployment using vLLM ([#8475](https://github.com/ai-dynamo/dynamo/pull/8475)).
* **Nemotron-3-Nano-Omni Recipe:** Added an experimental recipe for serving `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4` via vLLM in aggregated single-GPU mode, supporting text, image, video, and audio inputs with Kubernetes manifests for model caching and deployment ([#8799](https://github.com/ai-dynamo/dynamo/pull/8799)).
* **Kimi-K2.5 TokenSpeed Aggregated Recipe:** Added a complete deployment recipe for serving `nvidia/Kimi-K2.5-NVFP4` on the TokenSpeed engine under Dynamo's KV-aware aggregated frontend, including a multi-stage local-build Dockerfile, raw Kubernetes Deployments and Services (pending Operator support for TokenSpeed), and a full build/push/deploy walkthrough with TP=4, EP=4, NVFP4 weights, FP8 KV cache, and MLA/MoE plugin configuration ([#9231](https://github.com/ai-dynamo/dynamo/pull/9231)).
* **Empty Reasoning Content Fix:** Fixed the GPT-OSS reasoning parser producing empty `reasoning_content` when used with bounded KV cache by whitelisting special tokens needed by the gpt-oss and harmony parsers in the preprocessor, and resolved a Tokio runtime-drop panic during lazy initialization by moving init to a fresh OS thread ([#9050](https://github.com/ai-dynamo/dynamo/pull/9050)).
* **Pin Transformers for DeepSeek Perf:** Pinned `transformers==4.57.6` in DeepSeek-V3.2-FP4 performance job YAML files to prevent silent upgrades to transformers 5.x, which lacks native `deepseek_v32` model-type support and caused `TokenizerError: Failed to load tokenizer` failures during benchmarking ([#8690](https://github.com/ai-dynamo/dynamo/pull/8690)).
* **GLM-5 Disagg Frontend Flag:** Fixed the GLM-5 SGLang disaggregated deploy manifest to use the correct `--enforce-disagg` flag instead of the non-existent `--no-decode-fallback` flag, which would have caused a startup failure ([#8914](https://github.com/ai-dynamo/dynamo/pull/8914)).
* **Model Cache Resource Limits:** Added Kubernetes `resources.{requests,limits}.memory` blocks and an absolute `HF_XET_RECONSTRUCTION_DOWNLOAD_BUFFER_LIMIT=16GB` environment variable to all 15 model-download recipe Job pods, preventing OOM evictions caused by unbounded HF XET reconstruction buffers when `HF_XET_HIGH_PERFORMANCE=1` is set ([#9884](https://github.com/ai-dynamo/dynamo/pull/9884)).

## Bug Fixes

### Kubernetes Deployment

* **Operator DGD Reconciliation Lifecycle Fixes:** Fixed DGD reconciliation stalling at `Ready=False` by adding a PodCliqueScalingGroup (PCSG) watch so PCSG status changes trigger the controller ([#8328](https://github.com/ai-dynamo/dynamo/pull/8328)), with an analogous fix for the single-node PodClique readiness path that now reconciles on all relevant status fields ([#8423](https://github.com/ai-dynamo/dynamo/pull/8423)). Corrected planner ConfigMap ownership so profiling-generated ConfigMaps survive DGDR deletion and are adopted under the DGD lifecycle ([#8766](https://github.com/ai-dynamo/dynamo/pull/8766)), added profiler validation that surfaces DGD-plus-service name-length violations as terminal failures ([#8807](https://github.com/ai-dynamo/dynamo/pull/8807)), surfaced workload pod image-pull failures as Warning events on the DGDR instead of requiring `kubectl logs --previous` ([#8815](https://github.com/ai-dynamo/dynamo/pull/8815)), and rewrote the rolling-update replica calculation to use an availability-gated scale-down budget mirroring the Kubernetes Deployment controller, preventing maxUnavailable breaches during preemption scenarios ([#8823](https://github.com/ai-dynamo/dynamo/pull/8823)).
* **Planner Container Image Fixes:** Added `tail` and `env` utilities to the planner distroless image so container orchestration tooling and exec-based workflows function correctly ([#8603](https://github.com/ai-dynamo/dynamo/pull/8603)), introduced an inline `planner_test` build stage and unified all framework test image builds through inline `_test` stages routed via the shared `docker-remote-build` action with sccache and registry caching ([#8739](https://github.com/ai-dynamo/dynamo/pull/8739)), and included `aiperf` in the planner builder stage to fix `FileNotFoundError` crashes during thorough profiling benchmarks ([#8769](https://github.com/ai-dynamo/dynamo/pull/8769)).
* **Profiler Sidecar Retry Prevention:** Disabled automatic retries for profiling sidecar failures in the Operator, treating them as deterministic issues that require user intervention and now reporting explicit status and error details for clearer diagnostics ([#9166](https://github.com/ai-dynamo/dynamo/pull/9166)).
* **DGDR Optimization Type Validation:** Added `OptimizationType` as an enum-constrained string field to the DGDR SLASpec, so users can specify whether SLA-based profiling targets latency or throughput optimization with webhook validation enforcing valid values ([#8796](https://github.com/ai-dynamo/dynamo/pull/8796)).
* **Grove Resource Name Truncation:** Shortened TRT-LLM disaggregated worker service names from `TRTLLMPrefillWorker`/`TRTLLMDecodeWorker` to `prefill`/`decode` so that combined Grove resource names fit within the 45-character PodCliqueSet limit, fixing broken DGDR deployments with TRT-LLM disaggregation ([#8563](https://github.com/ai-dynamo/dynamo/pull/8563)).
* **Operator Least-Privilege RBAC Hardening:** Removed the cluster-admin-equivalent `*/*: *` wildcard rule from the Helm chart's manager ClusterRole and replaced it with explicit, audited least-privilege rules for every Kubernetes client operation in the operator codebase ([#8907](https://github.com/ai-dynamo/dynamo/pull/8907)), fixed an RBAC privilege-escalation rejection on fresh installs by adding the missing `pods/log: get` permission required for DGDR profiling job RoleBinding creation ([#9969](https://github.com/ai-dynamo/dynamo/pull/9969)), scoped the Docker secret indexer to the configured restricted namespace so that `mgr.GetAPIReader()` no longer bypasses namespace boundaries ([#9863](https://github.com/ai-dynamo/dynamo/pull/9863)), and added a projected service-account token volume for the profiling output-copier sidecar to keep it authenticated when `automountServiceAccountToken: false` is set without exposing the token to the main container ([#8771](https://github.com/ai-dynamo/dynamo/pull/8771)).
* **Doubled Model Path Fix:** Fixed a bug where `build_dgd_config()` produced doubled model paths (e.g. `/opt/model-cache/opt/model-cache`) when a DGDR specified `modelCache.pvcName` without `pvcModelPath`, by splitting the PVC branch so that an unset `pvcModelPath` now mounts the PVC as an HF cache directory while workers receive the HF model ID instead of the mount path ([#9177](https://github.com/ai-dynamo/dynamo/pull/9177)).
* **Operator v1beta1 Conversion Fixes:** Fixed v1alpha1 ↔ v1beta1 status conversion to preserve legacy scalar `componentName` values during round-trips ([#9184](https://github.com/ai-dynamo/dynamo/pull/9184)), relaxed v1beta1 component-name validation to accept uppercase letters historically allowed in v1alpha1 service keys ([#9192](https://github.com/ai-dynamo/dynamo/pull/9192)), preserved the legacy DGD worker hash across conversion to prevent unnecessary rolling updates of existing v1alpha1 deployments ([#9210](https://github.com/ai-dynamo/dynamo/pull/9210)), and restructured DGD/DCD conversion helpers with exported structural methods and documented conversion rules in `CONVERSION.md` ([#9257](https://github.com/ai-dynamo/dynamo/pull/9257)).
* **LWS Native Scaling Deployment Fixes:** Refactored the DynamoComponentDeployment controller to create a single LeaderWorkerSet object with `Spec.Replicas` set to the desired count instead of one LWS per replica, removing legacy indexed-resource logic and adding cleanup for old resources ([#5468](https://github.com/ai-dynamo/dynamo/pull/5468)), fixed multinode deployments (vLLM / SGLang / TRT-LLM) by emitting `$(LWS_LEADER_ADDRESS)` Kubernetes env-var expansion syntax so the kubelet correctly substitutes the leader hostname in direct-python container args ([#8369](https://github.com/ai-dynamo/dynamo/pull/8369)), and restored the LeaderWorkerSet resource name to the `<dcd-name>-0` form to avoid a service name collision between the operator-created ClusterIP service and the LWS-created headless service ([#9612](https://github.com/ai-dynamo/dynamo/pull/9612)).
* **Frontend Sidecar PVC Mounts:** Fixed the auto-generated frontend sidecar container in EPP/inference-gateway deployments to mirror the parent worker container's volume mounts, so PVC-backed model paths (tokenizer, config, chat-template) can resolve correctly instead of failing on file reads ([#8598](https://github.com/ai-dynamo/dynamo/pull/8598)).
* **vLLM Multiprocessing Init Gating:** Fixed vLLM multinode pod rendering so the `wait-for-leader-mp` init container is only injected when the rendered main container actually uses the `--distributed-executor-backend mp` flag, preventing worker pods on data-parallel or Ray-based paths from hanging indefinitely ([#9955](https://github.com/ai-dynamo/dynamo/pull/9955)).
* **Istio Sidecar Injection Exclusion:** Excluded Istio sidecar injection from kgateway-proxy pods by adding a GatewayParameters resource with `sidecar.istio.io/inject: "false"` in the install script, preventing HTTP 500 errors on ext\_proc gRPC connections when namespace-level Istio injection is active ([#9839](https://github.com/ai-dynamo/dynamo/pull/9839)).
* **DCGM Discovery Namespace-Scoped Mode:** Enabled DCGM GPU discovery for namespace-scoped operator deployments by removing the hard-coded short-circuit in `validateGPUHardwareInfo` and granting cluster-wide pod read access via the gpu-discovery ClusterRole, for exporter pod detection regardless of operator scope ([#8365](https://github.com/ai-dynamo/dynamo/pull/8365)).
* **Deduplicate Concurrent GPU Discovery:** Added singleflight coordination to DCGM GPU discovery so concurrent reconciles for the same GPU SKU share one in-flight scrape, eliminating redundant DCGM calls and reducing discovery latency under contention ([#8797](https://github.com/ai-dynamo/dynamo/pull/8797)).
* **Istio Sidecar EPP Exclusion:** Excluded Istio sidecar injection from the EPP pod in both the standalone Helm chart and operator-managed deployments, preventing double-TLS handshake failures that caused `cx_connect_fail` errors and HTTP 500 responses when secure serving was enabled ([#9935](https://github.com/ai-dynamo/dynamo/pull/9935)).
* **Persist Discovered Hardware Metadata:** Fixed persistence of auto-discovered DGDR hardware metadata, including interconnect, RDMA capability, GPU SKU, and VRAM, before profiling job creation, then requeued for a clean second reconcile pass to ensure accurate deployment profiling in Kubernetes ([#9890](https://github.com/ai-dynamo/dynamo/pull/9890)).
* **Bounded Retry Per-Engine Port:** Fixed unbounded cuMemCreate OOM retry by adding a default 60s timeout with elapsed and free/total memory logging, and resolved port collisions for failover engines sharing a pod network namespace by overriding the forward-pass metric port per engine ([#8919](https://github.com/ai-dynamo/dynamo/pull/8919)).
* **Operator Go Module Fix:** Fixed the `go.mod` file in the operator deployment to update core dependencies for system stability and compatibility ([#8804](https://github.com/ai-dynamo/dynamo/pull/8804)). Needs verification.
* **Forward Pass Metric Port:** Injected `DYN_FORWARDPASS_METRIC_PORT` environment variable into worker pods, for forward pass metrics collection and Planner load scaling ([#8817](https://github.com/ai-dynamo/dynamo/pull/8817)).
* **Conditional Container Name Injection:** Fixed the Operator to inject `CONTAINER_NAME=main` only when Kubernetes discovery mode is set to `container`, preventing unnecessary rolling updates for deployments using the default pod discovery mode ([#9366](https://github.com/ai-dynamo/dynamo/pull/9366)).
* **ImagePullSecrets Drift Prevention:** Fixed imagePullSecrets drift during operator startup, ensuring consistent secret configuration across reconciliation cycles ([#9841](https://github.com/ai-dynamo/dynamo/pull/9841)). Needs verification.
* **Prefill Worker Failover Initialization:** Fixed prefill worker missing the `FlockFailoverLock` initialization that decode workers already performed, which caused prefill standbys to register immediately alongside the primary instead of waiting for failover ([#8367](https://github.com/ai-dynamo/dynamo/pull/8367)).
* **Drop Recreate Override Restore-Target:** Removed the hardcoded `Recreate` deployment strategy on restore-target worker Deployments in checkpoint-enabled DynamoComponentDeployments, so the default `RollingUpdate` applies so scaling no longer evicts the serving cold-start replica before new restore-target replicas are ready ([#8434](https://github.com/ai-dynamo/dynamo/pull/8434)).
* **Embedded Pod Template Metadata:** Enabled `generateEmbeddedObjectMeta=true` for Operator CRD generation so embedded Kubernetes `ObjectMeta` fields preserve labels and annotations, fixing API server pruning that broke DGD-level annotation propagation and prevented failover engine containers from receiving required environment variables ([#9553](https://github.com/ai-dynamo/dynamo/pull/9553)).
* **Restore Succeeded Condition Write:** Fixed a missing `setSucceededCondition` call in the operator's `updatePhaseWithCondition` path that was accidentally dropped during a cherry-pick to `release/1.2.0`, which caused the aggregate `Succeeded` condition to never be written during phase transitions and broke the post-merge operator CI job ([#10008](https://github.com/ai-dynamo/dynamo/pull/10008)).
* **GMS ResourceClaim Name Normalization:** Fixed GMS ResourceClaimTemplate naming to route CamelCase component names through `NormalizeKubeResourceName`, ensuring generated names comply with DNS-1123 subdomain validation and preventing Grove reconciliation failures ([#9829](https://github.com/ai-dynamo/dynamo/pull/9829)).
* **Pod Discovery Retry Logic:** Fixed a race condition in `get_deployment_logs()` where pod label selectors returned zero results because labels were not yet propagated after deployment readiness, by adding retry logic (up to 12 attempts, 5 seconds apart) with clear warning messages if pods remain undiscovered ([#9171](https://github.com/ai-dynamo/dynamo/pull/9171)).
* **Fail Fast on CrashLoopBackOff:** Fixed deployment wait logic to immediately detect and handle unrecoverable pod states such as CrashLoopBackOff, triggering automatic cleanup to avoid stalled profiling runs ([#9215](https://github.com/ai-dynamo/dynamo/pull/9215)).

### Frontend & Agents

* **Tool-Call Streaming Parser Robustness:** Preserved logprobs through the tool-call jailing layer so streaming responses no longer return `logprobs: null` ([#8072](https://github.com/ai-dynamo/dynamo/pull/8072)), fixed guided-decoding handling and corrected `finish_reason` to `tool_calls` per the OpenAI spec along with model-specific parser patches ([#8442](https://github.com/ai-dynamo/dynamo/pull/8442)), reconstructed tool-call arguments that were split across multiple streaming deltas instead of returning empty strings ([#8582](https://github.com/ai-dynamo/dynamo/pull/8582)), bypassed the reasoning parser for Qwen3 `enable_thinking=False` so parallel tool calls are no longer silently dropped ([#8589](https://github.com/ai-dynamo/dynamo/pull/8589)), prevented raw internal protocol tokens from leaking into content when tool-call parsing is interrupted ([#8820](https://github.com/ai-dynamo/dynamo/pull/8820)), and recovered nine silent-drop scenarios across top model parsers when `max_tokens` or EOS truncates the closing fence or JSON arguments mid-value ([#8888](https://github.com/ai-dynamo/dynamo/pull/8888)).
* **XML Parser Whitespace Parity:** Fixed the generic XML parser to drop normal text after tool-call blocks while preserving prefix separator whitespace, aligning qwen3\_coder, nemotron\_nano, and minimax\_m2 parity fixtures with the upstream parser parity contract ([#9350](https://github.com/ai-dynamo/dynamo/pull/9350)).
* **Auto-Detect Force Reasoning Mode:** Added automatic detection of `force_reasoning` when a chat template's generation prompt ends with `<think>`, ensuring the reasoning parser starts in reasoning mode to correctly separate thinking content from normal output ([#8240](https://github.com/ai-dynamo/dynamo/pull/8240)).
* **Responses API Wire Compliance:** Fixed the `/v1/responses` input-chain handling to accept Codex and Agents SDK tool-call round-trip shapes (`function_call` → assistant → `function_call_output`) that previously caused deserialization failures ([#8275](https://github.com/ai-dynamo/dynamo/pull/8275)), aligned `NvResponse` serialization with the OpenResponses spec by emitting `null` for nullable-required fields, injecting missing top-level parameters, defaulting image-input `detail` to `auto`, and adding compliance CI ([#8283](https://github.com/ai-dynamo/dynamo/pull/8283)), and removed the hardcoded 4 096-token `max_output_tokens` default that silently truncated reasoning-model outputs, deferring instead to the engine's own limit ([#9181](https://github.com/ai-dynamo/dynamo/pull/9181)).
* **Structured SSE Error Propagation:** Replaced bare or unspecified SSE error comments with properly structured `data: {"error":{...}}` frames followed by `data: [DONE]` on mid-stream faults, ensuring OpenAI-style clients receive actionable error messages instead of silent hangs ([#8430](https://github.com/ai-dynamo/dynamo/pull/8430)), and propagated actual backend error messages (e.g. multimodal image load failures) through streaming responses that previously surfaced only as "unspecified error" or premature disconnects ([#8674](https://github.com/ai-dynamo/dynamo/pull/8674)).
* **Kimi K2 Parser Correctness:** Fixed Kimi K2 tool-call parsing to recover partial tool calls when the model hits max\_tokens before emitting the section\_end marker, instead of silently discarding them ([#8208](https://github.com/ai-dynamo/dynamo/pull/8208)), back-ported upstream SGLang fixes to widen the tool-name regex to support hyphenated names and added a force-exit from reasoning mode when tool\_start\_token appears before `</think>` ([#8532](https://github.com/ai-dynamo/dynamo/pull/8532)), and ensured the preprocessor preserves special tokens required by the kimi\_k2 and kimi\_k25 parsers so markers like `<|tool_calls_section_begin|>` and `</think>` are not stripped during decoding ([#9227](https://github.com/ai-dynamo/dynamo/pull/9227)).
* **SGLang Tool Calling Update:** Updated the SGLang frontend processor's tool-calling functionalities to match the latest SGLang implementation, including guided decoding integration for required or named tool choices, streaming result buffering with fallback non-stream parsing, trust-remote-code passthrough, and special token preservation for tool-calling requests ([#8269](https://github.com/ai-dynamo/dynamo/pull/8269)).
* **Nemotron V3 Parser Parity:** Added `nemotron_v3` as a reasoning parser alias and ported vLLM Nemotron v3 reasoning parser test coverage into Dynamo's Rust implementation, aligning disabled-thinking behavior and streaming/non-streaming extraction paths with the vLLM contract ([#9058](https://github.com/ai-dynamo/dynamo/pull/9058)).
* **Chat Message Content Serialization:** Fixed non-streaming `/v1/chat/completions` responses to always serialize the `content` key as `null` when absent, instead of omitting it entirely, matching OpenAI's wire format and preventing client-side `KeyError` failures for reasoning-only or tool-call-only responses ([#8372](https://github.com/ai-dynamo/dynamo/pull/8372)).
* **KServe gRPC Readiness Race Fix:** Fixed a race condition in KServe gRPC `model_ready`/`server_ready` endpoints where readiness was reported as soon as a `ModelDeploymentCard` was registered, before the corresponding `WorkerSet` and engines were attached, causing clients to fail with "Connection closed unexpectedly" on immediate inference requests ([#9619](https://github.com/ai-dynamo/dynamo/pull/9619)).
* **Top Logprobs Token Detokenization:** Fixed missing `token` and `bytes` fields in `top_logprobs` responses from the SGLang backend by detokenizing `token_id` values in the Dynamo backend layer when the upstream engine returns `None` for decoded tokens, ensuring correct OpenAI-compatible logprobs output regardless of backend ([#8911](https://github.com/ai-dynamo/dynamo/pull/8911)).
* **Qwen 3.5 Tool Calling:** Fixed assistant tool-call argument handling for Qwen 3.5 models, which require special rendering when tool-calling messages contain a list of dicts, and improved error handling in the chat processor to surface similar issues faster ([#8792](https://github.com/ai-dynamo/dynamo/pull/8792)).
* **Default Skip Special Tokens:** Fixed the Rust backend's streaming detokenizer to default `skip_special_tokens` to `true` when the OpenAI request omits the field, aligning with upstream engines (vLLM, SGLang, TRT-LLM) and preventing special-token text such as `<｜begin▁of▁sentence｜>` from leaking into `content` or `reasoning_content` ([#8780](https://github.com/ai-dynamo/dynamo/pull/8780)).

### Scheduling

* **Planner Scheduling Operational Fixes:** Backfilled `max_num_batched_tokens` from discovery model cards when the VirtualConnector left it unset, unblocking load-based and throughput-based scaling in aggregated and prefill modes ([#8042](https://github.com/ai-dynamo/dynamo/pull/8042)), normalized model-name case comparisons in KubernetesConnector to prevent `CrashLoopBackOff` with mixed-case model identifiers ([#8384](https://github.com/ai-dynamo/dynamo/pull/8384)), corrected the profiler Job and example manifests to reference the standalone `dynamo-planner` image after runtime dependencies were split out ([#8407](https://github.com/ai-dynamo/dynamo/pull/8407)), tracked pending scaling targets in `GlobalPlannerConnector` so in-flight rollouts no longer trigger stacked scaling decisions ([#8422](https://github.com/ai-dynamo/dynamo/pull/8422)), exposed `get_worker_info` and fixed readiness signaling in multi-DGD GlobalPlanner topologies ([#8482](https://github.com/ai-dynamo/dynamo/pull/8482)), matched MDC `component` fields against the backend default instead of the PascalCase DGD service key to restore context-length resolution ([#8489](https://github.com/ai-dynamo/dynamo/pull/8489)), moved inventory and GPU-hour gauge publication out of the throughput-only tick path so Prometheus metrics are emitted in all planner modes ([#8575](https://github.com/ai-dynamo/dynamo/pull/8575)), delegated `wait_for_deployment_ready` to the pool-local KubernetesConnector so pool Planners no longer skip worker discovery ([#8694](https://github.com/ai-dynamo/dynamo/pull/8694)), and made SLA-mode scale-down decisions consolidation-aware by re-predicting per-survivor load to eliminate 2↔1 oscillation under steady traffic ([#9294](https://github.com/ai-dynamo/dynamo/pull/9294)).
* **Latency Ratio Scaling Floor:** Used the latency ratio (predicted TTFT divided by target) as a minimum replica floor in the Planner, so that latency-violation signals now drive scale-up decisions instead of being logged but ignored ([#8861](https://github.com/ai-dynamo/dynamo/pull/8861)).
* **Scale Status Budget Rejection:** Added a `ScaleStatus.REJECTED` status for GPU budget exceeded responses from the GlobalPlanner, so over-budget rejections now produce a warning log and continued operation rather than raising a RuntimeError ([#8774](https://github.com/ai-dynamo/dynamo/pull/8774)).
* **SGLang Prefill CUDA Graph:** Fixed SGLang prefill argument normalization to emit `--cuda-graph-bs 1` when prefill uses `--max-running-requests 1`, avoiding duplicated explicit CUDA graph settings in the rapid-path disaggregated generation path ([#9852](https://github.com/ai-dynamo/dynamo/pull/9852)).
* **Planner Config Serialization Fix:** Excluded environment-dynamic fields (namespace and Prometheus endpoint) from `planner_config.json` serialization so they are read from environment variables at Planner startup, and injected `PROMETHEUS_ENDPOINT` into profiler environment from Operator config ([#8805](https://github.com/ai-dynamo/dynamo/pull/8805)).
* **Prefill Avg ISL Decay:** Decayed the prefill regression average ISL tracker with idle Forward Pass Metrics (FPM) samples so stale long-prompt traffic no longer permanently blocks SLA-driven scale-down ([#9759](https://github.com/ai-dynamo/dynamo/pull/9759)).
* **Remove Double Tokenization EPP:** Removed redundant double tokenization in the EPP integration by upgrading the Gateway API Inference Extension (GAIE) dependency from v1.2.1 to v1.5.0-rc.2 and using upstream support for pre-computed token ID injection, so tokens are now computed once in GAIE and read directly in the frontend ([#8093](https://github.com/ai-dynamo/dynamo/pull/8093)).
* **Enable Priority Hints Routing:** Enabled priority hints in the Go Endpoint Picker (EPP) so that requests can forward priority information through the routing system to influence scheduling decisions ([#9353](https://github.com/ai-dynamo/dynamo/pull/9353)).
* **TRT-LLM Override Merge Fix:** Fixed a crash where thorough-mode profiler generated both `--override-engine-args` and `--trtllm.*` flags simultaneously, which TRT-LLM rejects as mutually exclusive, by introducing `_merge_overrides_into_args()` to detect and merge profiler overrides into an existing JSON blob instead of appending conflicting flags ([#9107](https://github.com/ai-dynamo/dynamo/pull/9107)).
* **Persist Endpoint After Handshake:** Fixed a routing-chain stall in the disaggregated frontend where transient decode-pod restarts caused permanent inference outages by persisting the prefill endpoint in the activator map after handshake completion, ensuring decode rebuilds can reactivate the `PrefillRouter` without waiting indefinitely; also resolved a stale `DecodeWaiting` entry left when decode registers before prefill and is removed before prefill arrives ([#8965](https://github.com/ai-dynamo/dynamo/pull/8965)).
* **Router Rejection Threshold Guard:** Guarded rejection logic with `None` when CLI busy-threshold flags (`--active-decode-blocks-threshold`, `--active-prefill-tokens-threshold`, `--active-prefill-tokens-threshold-frac`) are unset, preventing silent fallback values from triggering spurious `503 "All workers are busy"` rejections ([#8333](https://github.com/ai-dynamo/dynamo/pull/8333)).
* **Prefill DP Rank Balancing:** Fixed load imbalance across data-parallel ranks in prefill engines by generating `bootstrap_room` based on the prefill DP rank instead of a random value, ensuring decode engines correctly route to the intended rank ([#9080](https://github.com/ai-dynamo/dynamo/pull/9080)).
* **Cancel Replay Router Tasks:** Fixed the online replay router's LocalScheduler to cancel background tasks during shutdown, preventing replay-scoped tasks from outliving the router in tests and short-lived runtimes ([#8429](https://github.com/ai-dynamo/dynamo/pull/8429)).
* **Orphan Cleanup Race Guard:** Fixed a race condition in the KV Router where simultaneous startup caused `cleanup_orphaned_consumers` to see a transiently-empty active-instance set from discovery and incorrectly delete every peer's NATS consumer ([#9132](https://github.com/ai-dynamo/dynamo/pull/9132)).
* **Prefill Router Hash Mode:** Disabled EAGLE hash mode for the disaggregated prefill router so it uses prefill-side KV hash semantics instead of inheriting decode-only EAGLE settings, preventing prefill KV events from being indexed under one hash mode and queried under another ([#9871](https://github.com/ai-dynamo/dynamo/pull/9871)).
* **FPM ZMQ Port Offset Fix:** Fixed data-parallel ZMQ port assignment in vLLM instrumented scheduler by reading `data_parallel_index` instead of `data_parallel_rank`, which vLLM resets to 0 for dense models in external DP mode, preventing `Address already in use` errors when starting multiple DP children ([#8696](https://github.com/ai-dynamo/dynamo/pull/8696)).

### Multimodal & Diffusion

* **Disaggregated Omni Serving Pipeline Fixes:** Fixed multiple bugs preventing disaggregated Qwen2.5-Omni multi-worker serving, including connector routing gated incorrectly by the `final_output` flag and missing chat template handling so the full thinker→talker→code2wav pipeline completes correctly ([#8301](https://github.com/ai-dynamo/dynamo/pull/8301)), preserved caller-provided runtime device mappings for tensor-parallel workers instead of forcibly overwriting them ([#9034](https://github.com/ai-dynamo/dynamo/pull/9034)), added `--enforce-eager` to Wan2.2 video launchers to bypass a CUDA illegal memory access caused by `torch.compile` graph breaks in `WanSelfAttention` ([#9563](https://github.com/ai-dynamo/dynamo/pull/9563)), and removed the obsolete dummy `tokenizer.json` placeholder for TTS models that caused a race condition between cleanup and the frontend watcher ([#9954](https://github.com/ai-dynamo/dynamo/pull/9954)).
* **LLaVA E/P/D Deployment Fix:** Fixed LLaVA Encode/Prefill/Decode (E/P/D) deployment and added LLaVA test coverage for multimodal model profile configurations across multiple topology variants ([#8330](https://github.com/ai-dynamo/dynamo/pull/8330)).
* **Prompt Embeds Loading Fix:** Used vLLM to load prompt embeddings instead of custom logic, improving safety with stricter type and dimension validation checks ([#8228](https://github.com/ai-dynamo/dynamo/pull/8228)).
* **Multimodal Image Hash Collision:** Fixed `compute_mm_uuids_from_images` to include image geometry (width, height) in the blake3 hash preimage, preventing UUID collisions between RGB images with different dimensions but equal pixel counts ([#8341](https://github.com/ai-dynamo/dynamo/pull/8341)).
* **Encoder Model Memory Cap:** Capped `gpu_memory_utilization` to 0.2 for the nested vLLM engine in `load_vision_model()`, preventing a free-memory precheck failure on GPUs with \~24 GiB that caused encoder-only multimodal disaggregated tests to crash on CI ([#8466](https://github.com/ai-dynamo/dynamo/pull/8466)).
* **Qwen3-VL Disagg Multimodal Fix:** Backported vLLM PR #40932 to remove an invalid deepstack boundary check, unblocking Qwen3-VL disaggregated multimodal inference ([#9522](https://github.com/ai-dynamo/dynamo/pull/9522)).
* **SGLang Multimodal Encoder Mode:** Enabled `encoder_only` mode for SGLang multimodal encode workers, preventing the full model (including LLM weights) from loading and avoiding out-of-memory errors on GPUs with limited memory ([#9292](https://github.com/ai-dynamo/dynamo/pull/9292)).

### Fault Tolerance & Observability

* **Canary Health Check Reliability:** Made the canary health check the sole authority on endpoint readiness when enabled, preventing crash loops caused by eager `Ready` signals racing against canary verification ([#8165](https://github.com/ai-dynamo/dynamo/pull/8165)), replaced the fragile discovery/routing-based canary path with a direct in-process call via `LocalEndpointRegistry` to eliminate stale-pod and instance-id-mismatch failures in Kubernetes ([#8294](https://github.com/ai-dynamo/dynamo/pull/8294)), and reset the canary timer on request arrival and per streaming chunk to prevent false-negative health check triggers during long-context prefills and extended streaming responses ([#8467](https://github.com/ai-dynamo/dynamo/pull/8467)).
* **Preserve Original Model Casing:** Removed `.to_lowercase()` calls on the `model` label in `InflightGuard`, `ResponseMetricCollector`, and `HttpQueueGuard` so that all `dynamo_frontend_*` metric families emit the same casing as the originally-registered model card, ensuring dashboard filters match consistently across metric families ([#9953](https://github.com/ai-dynamo/dynamo/pull/9953)).
* **KV Publisher Metrics Registration:** Fixed the `kv_publisher` dropped-events counter so it is properly registered and exposed on `/metrics`, by removing a redundant `worker_id` variable label that collided with the runtime's auto-injected constant label of the same name ([#8660](https://github.com/ai-dynamo/dynamo/pull/8660)).

### SGLang

* **SGLang Disagg Prefill Health Probes:** Honored the `_HEALTH_CHECK` marker in the SGLang prefill handler so the canary probe detects a hung scheduler instead of returning a false-positive 200, and consolidated `HEALTH_CHECK_KEY` into the shared `dynamo.health_check` module for wire-format consistency across backends ([#8611](https://github.com/ai-dynamo/dynamo/pull/8611)), with a follow-up that aligned the injected `--cuda-graph-bs` value to the effective data-parallel size when `--max-running-requests 1`, preventing a CUDA-graph capture assertion that crashed the prefill worker under DP attention ([#9962](https://github.com/ai-dynamo/dynamo/pull/9962)).
* **SGLang Stop Token Forwarding:** Fixed the SGLang decode handler to pass `stop_token_ids_hidden` from the Rust frontend to the engine's sampling parameters, for proactive EOS detection and avoiding wasted compute on tokens generated past EOS when `skip_tokenizer_init=True` ([#8084](https://github.com/ai-dynamo/dynamo/pull/8084)).
* **Piecewise CUDA Graph Disabled:** Disabled piecewise CUDA graph in SGLang launch scripts (`agg.sh`, `agg_router.sh`, `disagg.sh`, `disagg_router.sh`) to prevent worker crashes caused by `CUBLAS_STATUS_EXECUTION_FAILED` during warmup when `--context-length` is smaller than the default piecewise bucket size ([#8609](https://github.com/ai-dynamo/dynamo/pull/8609)).
* **Guided Decoding Aggregated Serving:** Removed the `--skip-tokenizer-init` flag from the SGLang aggregated-serving launch script so the grammar backend initializes correctly, for `response_format: json_schema` and other constrained-decoding features in aggregated mode ([#8843](https://github.com/ai-dynamo/dynamo/pull/8843)).
* **Preserve Max New Tokens:** Fixed SGLang decode handler to preserve `max_new_tokens=None` in sampling params, preventing a silent 128-token output cap when clients omit `max_tokens` from chat completion requests, so generation now correctly continues until EOS ([#8743](https://github.com/ai-dynamo/dynamo/pull/8743)).
* **NVTX Profiling Dependency Fix:** Fixed the SGLang runtime image to ship the `nvtx` Python package and expose the `nsys` binary on PATH, resolving `ModuleNotFoundError` crashes and `command not found` errors when using `DYN_NVTX=1` profiling ([#8629](https://github.com/ai-dynamo/dynamo/pull/8629)).
* **Gated Routed Experts Kwarg:** Gated the `return_routed_experts` keyword argument behind an explicit `enable_return_routed_experts` opt-in flag and resolved it once at handler initialization rather than per-request, ensuring DeepSeek-V4 compatibility with older SGLang builds that lack the kwarg in their `async_generate` signature ([#8798](https://github.com/ai-dynamo/dynamo/pull/8798)).
* **SGLang Mooncake Dependency Fix:** Fixed a missing `libjsoncpp25` runtime dependency in the SGLang runtime Docker image that caused `ImportError` when importing the bundled Mooncake transfer engine ([#8645](https://github.com/ai-dynamo/dynamo/pull/8645)).

### vLLM

* **Stream Interval Flag Propagation:** Propagated vLLM's `--stream-interval` flag to the Dynamo frontend via `ModelRuntimeConfig.set_engine_specific()`, so the flag now takes effect in disaggregated serving mode instead of being silently ignored with a hardcoded default of 20 ([#8101](https://github.com/ai-dynamo/dynamo/pull/8101)).
* **Prefill Cancellation Log Race:** Fixed an asyncio race condition in `_monitor_abort` where `CancelledError` could interrupt `engine_client.abort()` mid-call, leaving the engine with a dangling request and preventing the "Aborted Request ID" completion log from appearing. The fix shields the abort coroutine from cancellation using `asyncio.shield` on the no-guard path and `asyncio.create_task` on the disaggregated-decode guard path, ensuring abort always runs to completion ([#8768](https://github.com/ai-dynamo/dynamo/pull/8768)).
* **Disagg Decode Benchmark Unblocked:** Fixed two independent bugs in `InstrumentedScheduler` decode benchmark mode: attached `kv_connector_metadata` to synthetic `SchedulerOutput`s (preventing `AssertionError` on workers with KV connectors in disaggregated deployments) and padded fake-decode prompts by one token to avoid the async sampler's `-1` placeholder causing out-of-vocabulary embedding lookups at batch sizes greater than one ([#9360](https://github.com/ai-dynamo/dynamo/pull/9360)).
* **Prompt Embeds Feature Guard:** Added validation to reject requests containing prompt embeddings when the feature is not globally enabled, preventing unexpected behavior in deployments that do not use this capability ([#8248](https://github.com/ai-dynamo/dynamo/pull/8248)).
* **Disagg Decode Queue Classification:** Fixed `InstrumentedScheduler._compute_queued` to iterate `self.skipped_waiting` alongside `self.waiting`, correctly classifying `WAITING_FOR_REMOTE_KVS` requests as queued decode work so the planner accurately observes decode-engine pressure in disaggregated serving ([#8471](https://github.com/ai-dynamo/dynamo/pull/8471)).
* **Deferred vLLM Request Cancellation:** Delayed vLLM request cancellation on the decode side until the engine produces its first token, preventing EngineCore crashes caused by aborting a request while NIXL KV transfer is still in flight during disaggregated serving ([#8624](https://github.com/ai-dynamo/dynamo/pull/8624)).
* **Max Thinking Tokens Mapping:** Fixed silent dropping of `max_thinking_tokens` at the vLLM worker by mapping it to vLLM 0.20+'s renamed `thinking_token_budget` field on `SamplingParams`, ensuring the thinking-budget logits processor correctly enforces the configured limit ([#9571](https://github.com/ai-dynamo/dynamo/pull/9571)).
* **Ray Dependency in vLLM Runtime:** Fixed missing Ray binary in the vLLM runtime container image by pinning Ray in `requirements.vllm.txt` and the `[vllm]` pyproject extras, preventing crash on pods using the Ray distributed executor backend after vLLM 0.19.1 removed Ray from its dependency graph ([#8515](https://github.com/ai-dynamo/dynamo/pull/8515)).

### TensorRT-LLM

* **NVRTC JIT Include Discovery:** Fixed TRT-LLM NVRTC JIT compilation failures on Blackwell (sm\_100a) by installing `pip` into the runtime venv, so `pip show tensorrt_llm` can resolve the correct include path for kernel sources ([#8296](https://github.com/ai-dynamo/dynamo/pull/8296)).
* **Event Buffer Size Preserved:** Fixed the TensorRT-LLM worker to no longer override a user-provided `event_buffer_max_size` with the default value of 1024, ensuring custom engine configurations are respected and event publishing rates are not unintentionally affected ([#9284](https://github.com/ai-dynamo/dynamo/pull/9284)).
* **NIXL Initialization Mode Fix:** Skipped NIXL connector creation in aggregated mode where it is unused, and wrapped disaggregated-mode initialization in error handling so missing IB/RDMA hardware surfaces as a warning at init time instead of crashing on first request ([#9501](https://github.com/ai-dynamo/dynamo/pull/9501)).
* **TRT-LLM MDC Registration Fix:** Fixed TRT-LLM MDC registration so that `max_seq_len`, `max_batch_size`, and `max_num_tokens` supplied via `--extra-engine-args` or `--override-engine-args` are propagated back to `config` before handler and MDC registration, preventing MDC from advertising the model-native context length instead of the engine's actual limits ([#9130](https://github.com/ai-dynamo/dynamo/pull/9130)).
* **Router E2E Test Fix:** Fixed the TRT-LLM disaggregated router end-to-end test that was failing due to a missing `cache_transceiver_config` setting, causing workers to assert on the first request and re-enabled the test in nightly CI ([#8954](https://github.com/ai-dynamo/dynamo/pull/8954)).
* **Diffusion Multi-Image Response Fix:** Removed the restriction that truncated multi-image diffusion responses to a single entry and stopped rejecting `n > 1` requests; the engine now emits a warning when the pipeline's actual batch differs from the requested `num_images_per_prompt`, with a `1 ≤ n ≤ 10` validation guard on the handler side ([#9853](https://github.com/ai-dynamo/dynamo/pull/9853)).

### KV Block Manager

* **Single-Block Cache Match Speedup:** Optimized the KVBM prefix-matching path so that single-block lookups probe the first hash before allocating the full result vector, reducing overhead in the common one-block case ([#9196](https://github.com/ai-dynamo/dynamo/pull/9196)).

* **KVBM Consolidator E2E Tests:** Fixed consolidator end-to-end tests by adding the `--kv-events-config '{"enable_kv_cache_events": true}'` flag and setting `DYN_KVBM_DISK_DISABLE_O_DIRECT=true` in worker fixtures, preventing silent consolidator attachment failures and hangs on kernels lacking O\_DIRECT support ([#8464](https://github.com/ai-dynamo/dynamo/pull/8464)).

* **KVBM Connector Import Update:** Updated KVBM's kv\_cache\_connector library imports to align with the module reorganization introduced in TensorRT-LLM 1.3.0rc14, dropping compatibility with earlier TensorRT-LLM versions ([#9622](https://github.com/ai-dynamo/dynamo/pull/9622)).

* **FlexKV Connector PdConnector Support:** Fixed a `TypeError` in `PdConnector` that prevented `FlexKVConnectorV1` from being used as the first connector when launching disaggregated FlexKV serving, by adding it to the allowed first connector types alongside `DynamoConnector` and `LMCacheConnectorV1` ([#8787](https://github.com/ai-dynamo/dynamo/pull/8787)).

* **Main Attention KV Block Size:** Fixed KV event block size resolution for vLLM to use the main-attention KV cache group metadata, ensuring hybrid Nemotron-style events align with the router/indexer block size instead of falling back to `cache_config.block_size` ([#9228](https://github.com/ai-dynamo/dynamo/pull/9228)).

* **Replay Wall Time Accuracy:** Excluded report bookkeeping from replay wall-time measurement so the metric reflects actual replay execution rather than post-processing overhead ([#9190](https://github.com/ai-dynamo/dynamo/pull/9190)).

* **NIXL SDK Dev Stage:** Fixed the sglang dev container image by copying the NIXL C++ SDK (`libnixl_common`, UCX, libfabric, `gdrapi.h`) into the dev stage, resolving `rust-lld: error: unable to find library -lnixl_common` failures during source compilation ([#9216](https://github.com/ai-dynamo/dynamo/pull/9216)).

* **GAIE kgateway URL Update:** Fixed the GAIE kgateway URL to reflect the changed endpoint, ensuring correct gateway connectivity ([#9331](https://github.com/ai-dynamo/dynamo/pull/9331)).

* **Sync vLLM Version on Ref:** Fixed the `install_vllm.sh` script to derive `VLLM_VER` from `VLLM_REF` when `--vllm-ref` is passed, ensuring the correct vLLM version is installed instead of silently falling back to the default ([#8257](https://github.com/ai-dynamo/dynamo/pull/8257)).

* **Loosened Ray Version Pin:** Relaxed the `ray==2.55.0` dependency to `ray>=2.55.0` in the `dynamo[vllm]` extra and container requirements, so downstream consumers can use any compatible Ray version without version conflicts ([#9439](https://github.com/ai-dynamo/dynamo/pull/9439)).

* **Auditwheel Hidden Directory Copy:** Fixed the container build's dist-packages merge step to include dotfile directories (e.g., `.nixl_cu13.mesonpy.libs/`) by changing the glob from `*` to `.`, ensuring shared libraries like `libnixl.so` and `libserdes.so` are correctly copied into the SGLang venv and `import nixl` no longer fails ([#8966](https://github.com/ai-dynamo/dynamo/pull/8966)).

* **CuPy CUDA Version Matching:** Fixed runtime Dockerfiles for vLLM and TRT-LLM to dynamically select the CuPy package variant matching the build target CUDA version, replacing a hardcoded `cupy-cuda13x` that caused `ImportError: libcudart.so.12` failures on CUDA 12 builds ([#9379](https://github.com/ai-dynamo/dynamo/pull/9379)).

* **XPU Container Build Fix:** Fixed the vLLM runtime Dockerfile for XPU builds by guarding cupy-cuda installation behind a CUDA device check and removing leftover `nixl-cu*` packages that caused import failures in `dynamo.nixl_connect` ([#9419](https://github.com/ai-dynamo/dynamo/pull/9419)).

* **Worker Cleanup Shutdown Ordering:** Fixed shutdown ordering so engine cleanup (releasing GPU memory, tearing down PyTorch process groups) runs before the Rust runtime is shut down, and made cleanup idempotent so the signal-handler path and the Worker.run() finally path cannot execute it twice ([#8857](https://github.com/ai-dynamo/dynamo/pull/8857)).

* **Triton Backend Directory Default:** Fixed the Triton worker container image to set `BACKEND_DIR=/opt/tritonserver/backends` and updated the launch script to honor this environment variable, resolving a startup failure when using the container quick-start path instead of a local source build ([#8697](https://github.com/ai-dynamo/dynamo/pull/8697)).

* **Broken Readme References Removed:** Removed invalid `readme` declarations from `dynamo-runtime`, `dynamo-llm`, and `dynamo-protocols` Cargo manifests that referenced nonexistent files, unblocking `cargo package` and `cargo publish` for those crates ([#9809](https://github.com/ai-dynamo/dynamo/pull/9809)).

* **Cargo Lock Workspace Consistency:** Regenerated `Cargo.lock` to resolve internal version mismatches for `clap` and `rand` dependencies introduced by new workspace members, restoring `cargo metadata --locked` compatibility on `main` ([#8785](https://github.com/ai-dynamo/dynamo/pull/8785)).

* **Child Exit Code Propagation:** Fixed `wait_any_exit` in launch utilities to correctly propagate a failing child's exit code instead of silently overwriting it with `0` due to the EXIT trap's `kill 0` looping SIGTERM back to the script ([#8883](https://github.com/ai-dynamo/dynamo/pull/8883)).

## Documentation

### New Content

* **Tool Calling Docs Reorganization:** Reorganized tool calling and reasoning documentation into dedicated top-level User Guides sections (`docs/tool-calling/`, `docs/reasoning/`), splitting content into Dynamo-native and engine-fallback subpages so each topic is self-contained and easier to discover ([#9400](https://github.com/ai-dynamo/dynamo/pull/9400)).
* **Chat Processor Options Page:** Added a new `chat-processor-options.md` overview page and expanded the tool-calling and reasoning parser tables with per-model detail, upstream name divergences (vLLM/SGLang), and cross-links between documentation pages ([#8497](https://github.com/ai-dynamo/dynamo/pull/8497)).
* **Fastokens Tokenizer User Guide:** Added a user-facing guide for the Fastokens tokenizer backend, covering integration details, enablement criteria, CLI/env-var configuration, model compatibility, verification steps, and troubleshooting, so operators can decide whether and how to enable `fastokens` on the Dynamo frontend ([#9430](https://github.com/ai-dynamo/dynamo/pull/9430)).
* **Agentic Harnesses Blog Post:** Added documentation covering prompt stability, reasoning fidelity, and streaming-dispatch optimizations at the harness boundary, detailing how Dynamo preserves KV prefix reuse, maintains interleaved reasoning/tool-call order, and emits explicit `tool_call_dispatch` SSE events for Claude Code, OpenClaw, and Codex integrations ([#7561](https://github.com/ai-dynamo/dynamo/pull/7561)).
* **Sharded Indexer Benchmarking Docs:** Added benchmarking documentation for the sharded indexer, including trace dataset download instructions, benchmark execution modes, output metrics reference, and CLI flags ([#8695](https://github.com/ai-dynamo/dynamo/pull/8695)).
* **Anchor-Aware BSI Benchmark Docs:** Added anchor-aware BSI benchmark results and commands to the KV router indexer documentation, covering steady-state, sweep, and repeated-overload runs with observed tradeoffs such as stronger routing structure, higher routing cost, and hot-branch collapse on dominant-prefix workloads ([#9275](https://github.com/ai-dynamo/dynamo/pull/9275)).
* **Kubernetes Documentation Restructuring:** Refactored the monolithic AKS guide into focused sub-pages (RDMA/InfiniBand, storage, Spot VMs) and rewrote the main setup flow as a four-step CLI guide, including a full RDMA setup walkthrough with known-issue documentation, and removed deprecated `dynamoNamespace` references now auto-computed by the Operator ([#8777](https://github.com/ai-dynamo/dynamo/pull/8777), [#9101](https://github.com/ai-dynamo/dynamo/pull/9101)).
* **EFA Setup Guide:** Added a guide for using Dynamo with the AWS Elastic Fabric Adapter (EFA) on EKS, covering prerequisites, device-plugin installation, kernel-module verification, LIBFABRIC backend configuration, environment variables, pod security settings, and troubleshooting ([#9521](https://github.com/ai-dynamo/dynamo/pull/9521)).
* **Disaggregated Communication Guide:** Updated the disaggregated communication guide with AWS EFA benchmark results, including 9.6 GB/s KV transfer bandwidth and 31% lower inter-token latency measured on p5.48xlarge instances, along with expanded EFA configuration details, a kernel compatibility matrix, and EAGAIN troubleshooting guidance ([#7764](https://github.com/ai-dynamo/dynamo/pull/7764)).
* **GPU Memory Service Page:** Added a dedicated GPU Memory Service Kubernetes documentation page clarifying its experimental status, current limitations with Snapshot, and decision guidance for users choosing between GMS, Snapshot, and failover ([#9119](https://github.com/ai-dynamo/dynamo/pull/9119)).
* **ModelExpress with ModelStreamer:** Expanded the Kubernetes model-caching guide with customer-facing ModelExpress peer-to-peer instructions, showing how ModelStreamer is selected via `MX_MODEL_URI` for S3, GCS, Azure Blob Storage, and local/PVC safetensors paths ([#9417](https://github.com/ai-dynamo/dynamo/pull/9417)).
* **Planner Replay Benchmarking Guide:** Added a guide for benchmarking the Dynamo Planner in replay mode, covering configuration knobs, aggregated vs disaggregated invocation, output artifacts, remote HTML viewing workflows, parallel sweep tips, and a case study showing an engine-startup-time sweep that reveals an SLA cliff near 100 to 120 s startup ([#8641](https://github.com/ai-dynamo/dynamo/pull/8641)).
* **Unified Backend Guides:** Added Rust and Python unified backend guides, with follow-up fixes correcting a test link, a `CARGO_BIN_NAME` compile error in integration tests, and a hang in the channel-less `generate` template when delay is zero ([#9581](https://github.com/ai-dynamo/dynamo/pull/9581)).
* **Observability Metrics Docs:** Documented the full set of `dynamo_component_*` metric labels, component values, endpoints, and error types to remove ambiguity around Dynamo-emitted versus Kubernetes-injected labels, corrected the `DYN_SYSTEM_PORT` default in the health-checks table, and explained Prometheus label-warmup behavior so users understand why labeled metrics only appear after the first matching request ([#8478](https://github.com/ai-dynamo/dynamo/pull/8478), [#8545](https://github.com/ai-dynamo/dynamo/pull/8545), [#8834](https://github.com/ai-dynamo/dynamo/pull/8834)).

## Looking Ahead

The following is planned for the next release, v1.3.0. See the public [Dynamo roadmap](https://github.com/ai-dynamo/dynamo/issues/9178) for the full plan.

### DeepSeek-V4 for Agentic Coding

Performant DeepSeek-V4 recipes across vLLM and SGLang on Hopper and Blackwell, tuned for agentic-coding workloads with KV-cache reuse and disaggregated serving.

### Pluggable, Topology-Aware Planner

A re-architected Planner with a plugin framework for custom scaling logic, extending toward topology-aware placement that accounts for interconnect and node layout across heterogeneous worker pools.

### CRIU GPU Snapshots

Dynamo Snapshot's CRIU-based GPU process checkpoint and restore matures toward faster, model-reload-free recovery at scale, paired with operator-driven checkpoint lifecycle management.

### Realtime Streaming I/O

A bidirectional `/v1/realtime` request plane lays the foundation for streaming input and output, including voice and multimodal streaming.

## Patch releases

### v1.2.1 — Jun 13, 2026

#### Summary

Dynamo v1.2.1 is a patch release on top of v1.2.0, focused on **ModelExpress 0.4.0 engine-side model loading** (including object-storage model sources), **AMD ROCm / Python 3.10 import compatibility**, **EFA container build fixes**, and backend correctness fixes for **SGLang** and the **gpt-oss-120b recipe**.

**Base Branch**: `release/1.2.1`

#### Features & Improvements

* **ModelExpress 0.4.0 Integration:** Added engine-side ModelExpress model loading to the vLLM and SGLang runtimes (#10578). The runtime images now ship `modelexpress==0.4.0` by default (installed with `--no-deps` so the upstream engine dependency stacks are untouched), the new vLLM path is owned by the ModelExpress vLLM plugin, and the legacy Dynamo-owned `--model-express-url` / `MODEL_EXPRESS_URL` wrapper is retained only as deprecated compatibility parsing.

#### Bug Fixes

* **ModelExpress Object-Storage Loading:** Fixed a model-load stall on the vLLM and SGLang ModelExpress / RunAI Model Streamer object-storage path (`--model s3://…`, also `gs://` / `az://`) (#10674). Avoided constructing the vLLM `ModelConfig` twice (which ran duplicate `__post_init__()` side effects during engine startup) and made `register_model()` use the engine's pulled local directory rather than re-resolving the object-storage URI.
* **SGLang Routed-Experts Encoding:** Fixed a crash on the first decoded token when `--enable-return-routed-experts` is used with non-DeepSeek-V4 MoE models on SGLang 0.5.11+ (#10543). SGLang v0.5.11 moved the base64 encoding of `routed_experts` upstream into `tokenizer_manager`, so the Dynamo decode handler no longer re-encodes the already-encoded string; `nvext.routed_experts` is emitted as a base64 UTF-8 string at both emit sites.
* **ROCm / Python 3.10 Import Compatibility:** Fixed two import-time failures that blocked `import dynamo.*` on AMD ROCm / Python 3.10 hosts (#10545). `nixl_connect` now defers the CUDA-only NIXL `ImportError` until first use so the router, planner, and frontend import cleanly on hosts without a NIXL wheel, and `dynamo.common.configuration` uses `typing_extensions.Self` instead of `typing.Self` for Python 3.10. CUDA behavior is unchanged.
* **gpt-oss-120b Recipe Revert:** Reverted the gpt-oss-120b TensorRT-LLM aggregated recipe to runtime image `1.0.0` to restore the prior recipe baseline (#10549).

#### Build, CI and Test

* **EFA Container Build Fixes:** Backported two EFA container build fixes to the release branch (#10425): bumped `nixl_gdrcopy_ref` to v2.5.2 for Linux kernel ≥6.15 compatibility, and built and overlaid libfabric v2.5.1 (the first release with the CUDA dmabuf fix for GB200 EFA) onto the EFA installer's stock binary. Together these restore the Dynamo EFA RDMA container build.

#### Open-source contributions and new contributors

Between v1.1.1 and v1.2.0, the project merged 603 PRs from 82 authors. Thank you to the external community contributors in this release (organization confirmed via commit-author email, cross-referenced against the team roster):

* **Intel:** @dsocek, @sywangyi, @tthakkal, @sandeep-maddipatla, @VincyZhang, @ZhengHongming888.
* **Microsoft:** @Jont828, @ashnamehrotra, @devivasudevan, @avinashpenmetsa.
* **Baseten:** @the-david-oy, @michaelfeil.
* **vLLM:** @ywang96 (Roger Wang).  **SGLang:** @ch-wan (Cheng Wan).
* **Roblox:** @navmarri14.
* **Independent:** @Ayobami-00, @Kaonael.

If you would like to get involved, please see our [Contribution Guide](https://docs.nvidia.com/dynamo/dev/getting-started/contribution-guide).