Dynamo v1.2.0

Release notes

Dynamo v1.2.0GA releaseJun 2, 2026

GitHub ↗Artifacts

603PRs merged

82Contributors

5Breaking changes11Known issues

Dynamo v1.2.0 is the 15th feature release of the open-source distributed inference platform. It expands multimodal serving (text-to-image on TensorRT-LLM, formalized audio/video streaming, and media-aware KV routing), scales the standalone KV indexer to branch-sharded, anchor-aware routing, and promotes the DynamoGraphDeployment (DGD) Kubernetes API to served v1beta1. It also productizes the inter-pod GPU Memory Service, extends Dynamo Snapshot to OpenShift and multi-namespace deployments, and adds DeepSeek-V4 (Flash and Pro) recipes, native on vLLM 0.20.1 and on SGLang via an upstream preview image.

Highlights

Features & Improvements

Scheduling

• KV Router Operational Robustness: Increased the JSON body-size limit to 8 MiB so the standalone /query endpoint can handle 1M-token payloads (#8315), allowed unit block sizes in slot tracking with a bandwidth-flood warning (#8395), added batch-level start_position support to flat KV events for absolute position restoration (#8426), introduced a warning-only metrics path for duplicate store replays (#8473), filtered non-main ZMQ KV event groups via group_idx to prevent mis-indexing of DeepSeek V4 auxiliary cache state (#8669), added forward-compatible kv_cache_spec_kind parsing for hybrid vLLM cache metadata (#8751), resolved a startup crash by polling discovery until a worker model card appears (#8765), and fixed post-clear recovery replay so buffer queries skip events prior to the last clear barrier (#9296).
• Branch-Sharded KV Router Indexer: Added BranchShardedIndexer, a sharding strategy that partitions the prefix space by conversation branch and routes queries to exactly one shard using FNV-hashed prefix blocks for linear read-throughput scaling (#7859), with a write-triggered cleanup task that prunes stale compressed-tree children to reduce memory footprint (#8127), a fix that registers intermediate FNV values at each depth to prevent false cache misses for short queries (#8939), an opt-in anchor-aware mode that installs shared structural anchors at divergence points and lets compressed radix-tree reads walk anchored suffixes directly (#9007), re-exported the anchor-backed implementation as the default BranchShardedIndexer path with corrected anchor bookkeeping on worker removal (#9318), and renamed the internal type while preserving the public alias (#9333).
• Mooncake Multi-Tier KV Routing: Added multi-tier KV cache indexing supporting device, host, and disk storage tiers with KV consolidation mode selection and external sequence hash tracking (#8380), extended the remote indexer wire protocol to carry tiered match data end-to-end so host/disk continuation matching works correctly under use_remote_indexer=true (#8632), aligned the standalone HTTP indexer with Mooncake RFC #1403 by unifying the tier-walk implementation and reshaping /query//query_by_hash responses to expose per-instance tier breakdowns (#8912), and updated the standalone-indexer documentation to reflect the new multi-tier response schema including instances map, cumulative tier semantics, and additional_salt/cache_salt fields (#9247).
• Global Planner Pool Arbiter: Added aggregated routing mode to the global router with TTFT × ITL 2D grid pool selection and priority overrides from agent hints (#8044), introduced a --min-total-gpus floor with paired intent-cache arbitration that searches for opposite-direction scale intents across managed DGDs to keep the cluster within bounds (#8633), fixed the GlobalPlanner health endpoint never registering by awaiting serve endpoints concurrently instead of sequentially (#8682), and moved per-request target TTFT/ITL fields into the nvext.router sub-object with schema validation at deserialization time (#9845).
• Concurrent Radix-Tree Router: Defaulted approximate routing to the concurrent radix tree when router event threads exceed one, with support for recording precomputed routing-decision hashes and updated CLI help (#9219), optimized child insertion by gating only the first child with a shape write lock and removed a redundant in-plan shape-version retry check while refactoring mooncake bench helpers for repeated runs with percentile summaries and improving prompt trie stale-lookup repair via BFS (#9076), and implemented block_count() and worker_count() for the SyncIndexer so shard sizes report accurate values in benchmarks (#8559).
• LoRA Routing Foundation Layer: Added foundational LoRA allocation primitives including the LoraAllocator trait with compute_replica_set(), slot-aware state tracking, and shared constants as the base layer for the multi-PR LoRA allocation system (#8177), and fixed LoRA adapter registration in prefill workers to use ModelType.Prefill instead of Chat | Completions, preventing chat requests from being incorrectly routed to prefill workers that would stall waiting for KV transfers (#8945).
• KV Router Metrics and Reliability: Added worker-side ZMQ KV relay counters for received, accepted, and filtered events, exposed normalizer filter reasons for observability, and consolidated KV publisher metrics into the existing KV router metrics module (#9233), with a follow-up that deferred JetStream startup orphan cleanup by snapshotting candidates before a grace period to avoid treating newly started routers as orphans (#9303).
• Per-Worker Router Configuration: Exposed an optional router_config field in the Model Deployment Card so individual worker sets can override the frontend’s global router config, for different routing strategies (e.g., device-aware weighted) for different workers within the same deployment (#8290).
• Shared KV Cache Routing: Added an interface for third-party shared KV caches to integrate into the Dynamo router, so routing decisions that account for both local device cache hits and shared cache state using a configurable shared_cache_multiplier (default 0.5) can scale shared cache matches relative to device-local matches (#7536).
• Approximate Pruning TTL-Only Mode: Refactored KV Router approximate-mode pruning to use exclusively TTL-based eviction, removing router_max_tree_size and router_prune_target_ratio count-based configuration parameters while keeping normal KV event ingestion fire-and-forget (#8893).
• Prefill Load Scoring Refinement: Replaced the hidden DYN_ROUTER_LOAD_BLOCK_SIZE scoring override with explicit block-space prefill scoring, added a new prefill_load_scale configuration parameter (DYN_ROUTER_PREFILL_LOAD_SCALE / --router-prefill-load-scale) plumbed through Rust, Python, and C bindings, and introduced a token-dp-balance frontend router mode that uses KV scheduler load balancing without prefix matching or KV event/indexer paths (#9267).
• Profiler and Rapid-Mode Hardening: Fixed model path doubling when modelCache lacks pvcModelPath (#8449), wired mocker-rapid to direct AIConfigurator (AIC) flags and removed dead profiler AIC interpolation code (#8455), threaded the deployment’s total GPU budget into AIC enumeration so candidates respect cluster size (#8479), enforced totalGpus as a hard cap in rapid-mode DGD generation to prevent over-provisioning (#8617), corrected PickedParallelConfig.num_gpus to reflect physical GPU count rather than overcounting attention DP (#8610), added an explicit replay optimization objective knob defaulting to throughput (#8518), resolved decode benchmarking failures by using actual deployment component names for log path lookup (#8733), and hardened Pareto computation to skip NaN/inf/non-numeric points gracefully (#9169).
• Planner Advisory Mode Scaling Logic: Added an advisory scaling mode so operators can observe planner decisions (replica recommendations, structured summary logs, Prometheus metrics, Plotly HTML reports) without applying changes, while the full pipeline runs identically to active mode (#8244), with interactive HTML diagnostics and throughput-regression bootstrapping wired into offline planner-in-the-loop replay along with several scaling-logic correctness fixes surfaced by the new reports (#8280), KV-cache hit-rate awareness threaded through load and throughput scaling paths so prefill compute is discounted on reuse-heavy workloads (#8314), AIC interpolation moved in-process to the planner for rapid pre-deployment sweeping and MoE-DEP correctness fixes for models like Qwen3-235B-A22B-FP8 (#8335), and scoped test stubs to prevent advisory-mode fixtures from leaking into other planner test modules (#8418).
• A30 GPU SKU Support: Added the NVIDIA A30 to the DGDR hardware.gpuSku enum across the Go API, generated Python model, CRD, Helm CRD, and API docs, and taught GPU discovery to infer a30 from A30 product labels (#9176).
• Discovery Concurrency Race Fixes: Addressed multiple concurrency edge cases in the discovery service’s handle_put lifecycle, including an abort-cleanup race where completed tasks could delete a newer task’s handle, a poll timeout guard for recover_concurrent_registration, and a delete timeout to avoid blocking the watch loop indefinitely (#8237), added the missing Audios variant to is_model_type_list_empty which previously caused spurious ModelUpdate::Removed emissions for audio models (#8241), moved ModelUpdate::Added notification to after add_worker_set completes so HTTP endpoints are not exposed before a serving pipeline exists (#8242), and replaced the 100ms polling loop in recover_concurrent_registration with a tokio::sync::Notify plus enable() pattern using an absolute deadline to prevent spurious wakeups from extending the total wait (#8291).
• Default TCP Worker Pool: Raised the default TCP worker pool size to 10,000 to prevent unexpected performance drops when benchmarking at concurrency levels above 1,500 (#9090).
• Lock-Free TCP Pool Infrastructure: Replaced the mutex-based TCP connection pool with a lock-free LRU design backed by ArcSwap + atomic round-robin + SegQueue, closing five correctness gaps including unbounded-queue OOM, RAII guard leaks that inflated the inflight counter, and a drop-before-first-poll scheduler leak (#7806), replaced panic! calls with warn + break on TCP stream read errors (ECONNRESET) so connection resets no longer abort Tokio tasks under high concurrency (#8254), and introduced per-stream buffer garbage collection with a configurable threshold to reclaim memory after large transfers while raising the frame-size ceiling for multi-modal embeddings (#8420).

Kubernetes Deployment

• Snapshot Checkpoint-Restore Hardening: Added DRA-aware GPU UUID resolution from allocated claims for the snapshot agent (#8292), replaced the PID-1 SIGUSR1/SIGCONT signaling contract with file sentinels and fixed a shell-form entrypoint unwrap panic (#8403), introduced a pluggable Runtime interface with containerd and CRI-O backends plus OpenShift chart enablement (#8427), adopted an annotation-driven contract for multi-container checkpoint/restore targeting (#8631), blocked the unsupported snapshot-plus-GPU-Memory-Service combination via admission validation (#8689), enabled a single cross-namespace snapshot-agent DaemonSet using a new podMount PVC access mode (#8740), reworked seccomp configuration to support OpenShift checkpoint flows with a structured seccomp config block (#8902), and preserved the vLLM torch compile cache in rootfs diffs to prevent CRIU restore failures from missing Triton shared objects (#9943).
• Operator GPU Hardware Discovery: Expanded hardware detection to cover Blackwell, Hopper, Ampere, Ada Lovelace, older NVIDIA, and AMD GPU families with interconnect and RDMA/SR-IOV profiling (#7551), fixed a crash when totalGpus was missing by failing cleanly instead of proceeding with incomplete hardware info (#8267), added inference fallbacks for unrecognized form factors such as defaulting to SXM when no PCIe variant exists (#8507), hardened enrichment validation with nil guards and a gpu.DiscoverGPUs fallback when DCGM fails (#8508), introduced a SKU-filtered GFD node-label discovery path for environments where DCGM pods are unavailable (#8510), and documented the semantics and auto-discovery behavior of the new Interconnect and RDMA HardwareSpec fields (#8300).
• Operator v1beta1 API Migration: Introduced v1beta1 API types for DynamoGraphDeployment, DynamoComponentDeployment, and DynamoGraphDeploymentScalingAdapter with a cleaned-up surface replacing per-component fields with podTemplate and renaming services to components (#8414), implemented lossless v1alpha1 ↔ v1beta1 conversion and flipped v1beta1 to served (#8647), added round-trip fuzz coverage and fixed DGDR conversion mutability bugs (#9164), migrated the DGSA controller to reconcile the v1beta1 shape (#9194), migrated the DGD/DCD controllers to read v1beta1 objects while preserving upgrade/downgrade behavior (#9235), and aligned DGDR conversion to the structural rules with sparse annotation preservation for downgrade compatibility (#9262).
• Istio Mesh Support Fixes: Added automatic generation of Istio DestinationRule resources for EPP deployments in the Dynamo Operator, preventing double-TLS issues in Istio service meshes that previously required manual DestinationRule creation (#8270).
• Inter-Pod GMS Failover Support: Added inter-pod GPU Memory Service failover for vLLM deployments, where each rank receives a dedicated GMS weight-server pod and one or more engine pods sharing GPUs via Kubernetes DRA ResourceClaims and rank-isolated hostPath volumes for CUDA IPC, with a failover cascade controller for fast group-wide cleanup on engine failure (#7777). Introduced scratch-aliased KV cache initialization that preserves VA reservations across sleep/wake cycles so cudagraphs replay correctly during shadow-failover transitions (#8686), fixed SGLang GMS startup by auto-enabling the memory saver path when setup_gms() is used (#9647), and corrected the vLLM GMS worker to use the DP-adjusted CUDA device before connecting to GMS, preventing rank collisions under data parallelism (#9840).
• Helm Operator Configuration Hardening: Added a temporary admission-level gate that rejects services combining GPU Memory Service with Snapshot unless explicitly opted in via the dynamo-operator.featureGates.gmsSnapshot Helm value (#8829), consolidated the previously split NATS subchart flags into a single global.nats.install value with validation templates that fail on stale overrides (#9232), and fixed remaining references to the removed flags in the Tiltfile and chart documentation (#9281).

Multimodal & Diffusion

• Multimodal Media Fetcher Hardening: Centralized URL validation into a shared url_validator module enforcing scheme, local-path, and redirect policies across audio, image, and video loaders (#8282), wired the DYN_MM_ALLOW_INTERNAL env var through to the Rust MediaFetcher to honor localhost/on-prem semantics and unblock nightly tests (#8535), hardened the Rust side against redirect bypass and DNS rebinding with RFC-based IP blocklists, cloud-metadata hostname checks, and a custom BlocklistResolver on the reqwest::Client (#8569), introduced a dual-backend HTTP client supporting both httpx and aiohttp to improve throughput under high concurrency (#8646), bounded total in-flight media fetches with a global semaphore and decoupled httpx pool/read timeouts via DYN_MM_HTTP_* env vars to eliminate PoolTimeout errors (#8657), and added an opt-in byte-budgeted LRU cache to the Rust MediaLoader that skips network fetch, decode, and NIXL registration on cache hits (#8863).
• Multimodal Benchmarks Tooling Enhancements: Added a sliding-window JSONL generation subcommand for exercising prefix-caching under interleaved multi-user image streams, refactoring argument parsing into composable parent parsers and subcommands (#8201), fixed grouped single-turn sweeps to use --conversation-num so sessions run to completion without re-dispatch distortion (#8458), and introduced multithreading with configurable processing delay to the local media server for production-realistic latency simulation (#8822).
• SGLang Diffusion Video Pipeline Support: Added video input support for SGLang in the disaggregated prefill/decode path (#8597), fixed a startup crash caused by a missing enable_trace field in the diffusion worker ServerArgs stub (#8332), added the accelerate package to the SGLang runtime Docker image so diffusion pipelines can initialize successfully (#8357), and preserved file permissions in the dev container venv setup to prevent the bundled ffmpeg binary from losing execute bits (#8408).
• Multimodal-Aware KV Routing Path: Added a Rust-frontend multimodal routing path that extracts per-image hashes, dimensions, and token counts (via per-VLM-family math) before selecting a worker, for KV-cache-aware placement of multimodal workloads (#9272), with a plumbing-only PR that wired the lightseek-mm cargo feature and exposed LightseekMmCounter and token-ID resolver helpers from the llm-multimodal crate (#9352), and documented that Qwen3.5/Qwen3.6 models are not yet supported (#9572).
• Multimodal Routing Frontend Integration: Moved multimodal-aware KV routing from the separate MM Router Worker process into the frontend’s vLLM processor, eliminating an extra network hop and redundant image processing by transferring pre-processed mm_kwargs via shared memory (~2ms) or NIXL RDMA, with automatic fallback to URL reprocessing for cross-node deployments (#8065).
• TRT-LLM Multimodal Image Support: Added text-to-image generation capability to the TRT-LLM backend with a new /v1/images/generations API endpoint and Flux pipeline integration (#8200), and hardened the multimodal processor by replacing torch.load() with safetensors loading and Rust frontend media decoding to eliminate arbitrary code execution risks (#8295).
• Model Family Lookup Resolver: Introduced ModelFamily and resolve_model_family for vLLM multimodal encode-related handling in disaggregated prefill, replacing eight inconsistent per-callsite dispatch paths that failed when models were loaded from local paths not matching HF ids (#8973).
• NIXL Canonical Memtype XPU Support: Refactored NIXL memory type classification to use canonical segment names (VRAM/DRAM) instead of device-specific strings for forward compatibility, added Intel XPU (Level-Zero) device support for disaggregated encode/prefill-decode transfers, and fixed XPU kernel synchronization to prevent race conditions before NIXL transfers (#9073).
• Formalized Video Streaming Protocol: Introduced a dedicated data_source field in the audio protocol to disambiguate the overloaded response_format field, added an output_format container field to the video protocol, and extended the /v1/videos HTTP endpoint to support SSE streaming output in addition to batch JSON responses, for a consistent internal interface across all modalities (#8491).
• SGLang Frontend Decoding Support: Added --frontend-decoding flag for the SGLang backend so aggregated multimodal requests are decoded by the Rust frontend and shipped via NIXL RDMA, removing the GIL-bound image-decode hop from the worker’s P0 thread (#9405).
• Image UUID Benchmark Emission: Added optional --uuid flag to the multimodal JSONL generator that emits deterministic image_uuids alongside images, so vLLM’s cached-mm processor can short-circuit on cache hits across benchmark restarts and image reuse (#8854).

Frontend & Agents

• Agent Context and Tool Relay: Added passive nvext.agent_context parsing and a normalized agent trace bus with configurable jsonl/stderr sinks that emit chat-completion request_end records with Dynamo request metrics (#8789), introduced a ZMQ-to-event-plane relay with a narrow AgentToolEventRelay Python binding that captures tool lifecycle events from external harnesses (#8790), hardened the ingest topology so Dynamo owns the ZMQ PULL bind side, allowing multiple producers to connect without endpoint conflicts, and added multi-producer regression coverage (#9105), and aligned agent context identifiers with ATIF terminology (session_type_id, session_id, trajectory_id, parent_trajectory_id) while splitting the Agents documentation into focused pages (#9140).
• Nvext Response Metadata Gating: Fixed a regression where plain OpenAI-compatible requests leaked nvext response metadata (worker_id, timing) by default, introducing NvExtResponseFieldSelection to gate each response field independently behind extra_fields opt-in while preserving the query_instance_id exception (#8252).
• DeepSeek V4 Parser Support: Added a DeepSeek V4 DSML tool-call parser and reasoning parser to the Rust parser crate, registering canonical names and compatibility aliases (deepseek_v4, deepseek-v4, deepseekv4) with unit coverage for tool parsing, streaming reasoning, and alias resolution (#8665), hardened the V4 formatter so thinking: false, enable_thinking: false, thinking_mode: "chat", reasoning_effort="max", and per-request drop_thinking overrides all route end-to-end correctly (#8670), fixed tool-continuation reasoning parsing where a prompt-injected <think> seed caused the closing tag to leak as normal content without regressing Kimi K2.5 behavior (#8901), and deduplicated the DeepSeek V3.2 and V4 prompt encoders into a shared deepseek_common module to reduce code duplication (#9322).
• Self-Host MDC Metadata Pipeline: Added worker-side metadata artifact hosting via system_status_server, where LocalModel::attach() registers local-disk MDC files in a process-local registry and rewrites paths to HTTP URLs served by the worker (#8855), implemented a frontend verify-and-cache resolution pipeline that derives URIs per CheckedFile, fetches via scheme handlers, blake3-verifies, and atomically publishes to a content-addressed cache (#9057), and fixed a regression where sibling files like preprocessor_config.json and tokenizer.model were missing from the slug directory by adding a harvest pass that symlinks non-weight siblings from resolved directories (#9610).
• OpenAI Multi-Choice N Support: Added OpenAI-compatible n parameter plumbing to the Dynamo frontend and response contract so requests with n > 1 return multiple choices, with vLLM backend integration (#8744), SGLang backend support using its native multi-sequence generation (#8745), and TensorRT-LLM backend support passing n through to its sampling params and keeping streamed choices separated (#8746).
• Cross-Impl Parser Parity Harness: Added a pytest-driven parity harness that exercises Dynamo, vLLM, and SGLang parsers against shared YAML fixtures and diffs their outputs to surface cross-implementation divergences (#9186), expanded coverage from 7 to all 19 registered Dynamo parser families with new fixtures and xfail-tracked divergences (#9261), and documented the end-to-end workflow for finding, reproducing, fixing, and retiring divergences (#9394).
• Bidirectional Streaming WebSocket Frontend: Added a /v1/realtime WebSocket endpoint that accepts NvCreateChatCompletionRequest JSON frames over a single connection and streams NvCreateChatCompletionStreamResponse chunks back, for realtime bidirectional communication as the first slice of the streaming-input feature (#9079).

TensorRT-LLM

• TRT-LLM Metrics and KV Publisher: Added ForwardPassMetrics publishing via a PyO3 FpmDirectPublisher for non-attention-DP TensorRT-LLM workers, so the Planner can treat them equivalently to vLLM workers for autoscaling and latency prediction (#8356), fixed the trtllm_kv_transfer_success_total Prometheus counter that was never incremented due to a mutually exclusive gate between prefill mode and the decode-side timing check (#8483), and eliminated a ~148 ms TTFT regression when --publish-events-and-metrics was enabled by moving the publisher off the request loop and batching KV-cache walks (#8892).
• TRT-LLM Canary Health Checks: Set highest priority (1.0) on the canary health-check request to prevent false-negative timeouts when long-context inference requests starved the probe under load (#8488), and fixed disaggregated decode workers staying permanently NotReady by injecting explicit disaggregated_params into the canary payload so the handler no longer rejects probes that lack prefill-peer context (#8521).
• Disable GC for TRT-LLM: Added support for disabling Python garbage collection in the Dynamo TRT-LLM worker when TRTLLM_SERVER_DISABLE_GC or DYN_TRTLLM_SERVER_DISABLE_GC is set, preventing uvloop stalls at high concurrency and performance parity with trtllm-serve (#9096).
• TRT-LLM Wakeup RPC Caching: Improved wakeup call handling to preserve original exceptions and cache the resolved RPC method name after first invocation, eliminating redundant lookups on subsequent calls (#8255).

Performance Modeling & Replay

• Performance Modeling Offline Replay: Added KV Block Manager (KVBM)-backed G1↔G2 offload simulation for the vLLM mocker in both online and offline replay modes (#8184), simulated non-zero worker startup time in the offline discrete-event engine with WorkerReady events and pending_startup tracking (#8231), hardened replay-router scaling invariants including single-worker queueing, 0→N recovery, and draining-state preservation (#8236), introduced an agentic trace file format with loader and multi-turn smoke coverage (#8627), published tiered HostPinned KV events for G2 offload with a new StorageTier-aware event sink (#8961), and captured replay-friendly hashes on agent request_end records with a converter producing Mooncake JSONL for mocker replay (#8998).
• Claude Trace Exporter Tool: Added a privacy-preserving Claude raw-trace exporter under benchmarks/coding/claude, supporting autodiscovery, compaction-aware parsing, structural sidecar output, and configurable tokenization for accurate output token computation (#8096).

Infrastructure Modernization

• Context-Aware Event Plane Defaults: Made the event-plane selection context-aware so local-only discovery backends (file/mem) default to ZMQ while distributed backends (etcd/kubernetes) default to NATS, eliminating unconditional NATS connections at startup (#8398), removed the Python CLI’s hard-coded DYN_EVENT_PLANE=nats override that still forced NATS connections and caused “Connection refused” failures in local workflows (#8614), and removed hard-coded NATS_SERVER and ETCD_ENDPOINTS environment variables from profiling job configuration to respect the same infrastructure-aware defaults (#9271).

Fault Tolerance & Observability

• Frontend Staged Request Gauges: Added dynamo_frontend_active_requests and dynamo_frontend_stage_requests gauges with per-stage (preprocess, route, dispatch) and per-phase (prefill/decode/aggregated) inflight counts, backed by a StageGuard RAII type that increments on creation and decrements on drop (#8162), and updated the metrics documentation with descriptions of the new gauges and deprecation notes for superseded ones (#8459).
• Dynamo Local Resource Monitor: Added a lightweight, high-frequency (200 ms) per-process resource monitor that tracks VRAM, GPU, PCIe, CPU, disk, and network usage for Dynamo processes on a single host, with an accompanying Grafana dashboard integrated into the existing observability stack (#9055).
• Worker-Pool Saturation Metrics: Added six Prometheus metrics (dynamo_work_handler_queue_depth, queue_capacity, enqueue_rejected_total, permit_wait_seconds, pool_active_tasks, pool_capacity) to the shared TCP server, so operators can detect queue buildup and permit starvation before workers are OOM-killed (#8412).
• Reusable Telemetry Bus Primitives: Added a typed telemetry bus, stream completion helper, and async JSONL sink to the LLM library for shared event-recording infrastructure with configurable append mode, buffer sizing, and periodic flush while preserving the existing recorder JSONL format (#8788).
• Orphaned Pending Request Handling: Implemented discovery-plane-driven cancellation of pending response-stream subjects when a worker is removed, preventing indefinite hangs where requests queued in a killed worker’s bounded channel became permanently stuck; cancelled streams now return a migratable Disconnected error so the migration layer can retry on another worker (#8182).

vLLM

• vLLM Multinode Elastic EP: Added test infrastructure for validating vLLM’s native elastic expert parallelism across multiple nodes using a warm-standby Ray topology, confirming all scale steps (dp=2→3→4→3→2→4→2) succeed on a 2-node AKS cluster (#8183), and introduced operator support that routes --enable-elastic-ep deployments through a cross-node Ray cluster while fixing a concurrent scale_elastic_ep race condition that caused 300 s TCPStore timeouts on remote worker nodes during scale-up (#8216).
• vLLM Start Stop Profile: Added start/stop profile endpoints (engine/start_profile and engine/stop_profile) for vLLM to maintain parity with the SGLang implementation, giving a unified profiling interface (#8068).

SGLang

• SGLang Forward Pass Metrics: Wired SGLang’s ForwardPassMetrics ZMQ publisher into Dynamo’s event plane via FpmEventRelay, injecting the endpoint instance ID as worker_id and adding a cross-repo wire-format contract test (#8154), and fixed the previously unpopulated total_kv_blocks runtime config by resolving rank-0 scheduler info through SGLang’s canonical Engine scheduler path (#8439).
• SGLang Token ID Logprobs: Added return_tokens_as_token_ids support, integer token-id stop arrays, and gated top-logprobs for SGLang to enable RL workflows while preserving OpenAI-compatible response shapes (#8119).

Unified Backend (Preview)

• Unified Disaggregated Serving Abstraction: Added a common backend framework that lets engines plug into Dynamo’s disaggregated-serving path through a shared LLMEngine trait/ABC, eliminating the need for each backend to reinvent wire format, registration plumbing, or shutdown orchestration (#9249).
• Rust Backend Common Framework: Added a shared Rust backend layer (dynamo-backend-common) with an LLMEngine trait and Worker lifecycle driver, for native Rust backend integrations that follow the same lifecycle and cancellation contract as existing Python backends, along with a mocker reference engine for end-to-end pipeline testing (#8584).
• Rust Worker PyO3 Backend: Moved the unified backend Worker lifecycle from Python into the Rust dynamo_backend_common crate and exposed it to Python engines through a new dynamo._core.backend.Worker PyO3 binding, shrinking the Python Worker shim from 259 lines to ~115 while consolidating signal handling, discovery unregister, grace-period sleep, drain, cleanup, and 3-phase runtime shutdown entirely in Rust (#9202).
• Unified Worker Engine Args: Enabled the dynamo.trtllm unified worker to honor --extra-engine-args (YAML) and --override-engine-args (JSON) flags, bringing it to parity with the existing worker by porting the YAML+JSON merge sequence and adding validation that parsed overrides are JSON objects (#8886).

KV Block Manager

• KVBM-Logical Mocker Backend Replacement: Replaced the mocker’s manual vLLM block manager and evictor with a new kvbm-logical::BlockManager<G1> backend using a Lineage inactive pool, simplifying block lifecycle management and bridging KVBM’s PositionalLineageHash to the router’s SequenceHash via a new HashMap (#8451).
• PositionalLineageHash Ordering Support: Added Ord and PartialOrd trait implementations to PositionalLineageHash, for deterministic sorting and comparison operations for KVBM token types (#8687).

TokenSpeed

• Initial TokenSpeed Backend: Added a TokenSpeed backend integration through the common LLMEngine path, introducing python -m dynamo.tokenspeed with argument parsing, request/sampling conversion, streaming token output, abort/cleanup handling, guided decoding support, and focused unit coverage (#9212).

Recipes

• DeepSeek-V4 Recipe Consolidation: Restructured the DeepSeek-V4 Flash and Pro recipes into a single recipes/deepseek-v4/ subtree following the repo-wide convention, deduplicated Dockerfiles, and simplified the SGLang Dockerfile to consume the Dynamo donor image directly with a pinned base digest (#8735), performed a post-base cleanup pass removing unnecessary system dependencies from the B200 SGLang image (#8929), refreshed Python and Rust attribution files based on container scans of the runtime images (#8948), bumped deploy manifests to dev.2 public image tags and consolidated apt purge directives (#8971), refreshed the SGLang base image (#9002), updated all recipe manifests to dev.3 images (#9356), and aligned the docs support-matrix and release-artifacts pages to reflect the dev.3 experimental release (#9358).
• DeepSeek-V4 SGLang Serving Recipes: Added SGLang serving recipe for DeepSeek-V4-Flash on B200 GPUs with TP4, MXFP4 MoE, and EAGLE speculative decoding via Dynamo frontend (#8704), with Dockerfile PATH fixes for etcd resolution (#8713, #8716), published both Flash and Pro SGLang recipes using public NGC images and updated documentation (#8734), added aggregated and disaggregated recipes for DeepSeek-V4-Pro on GB200 with NIXL KV transfer over GKE RDMA (#8960), and added a disaggregated prefill/decode recipe for DeepSeek-V4-Pro on B200 with InfiniBand RDMA including UCX configuration guidance (#9278).
• DeepSeek-V4 vLLM Serving Recipes: Added aggregated vLLM recipes for DeepSeek-V4-Flash (4×B200, DP=4+EP) and DeepSeek-V4-Pro (8×B200, TP=8+EP) (#8668), introduced a disaggregated prefill/decode recipe for V4-Pro on GB200 NVL72 using DRA ComputeDomain for cross-node MNNVL with 16 GPUs total (#8811), extended GB200 aggregated recipes for both V4-Flash (TP=4+EP, FP4 path) and V4-Pro (TP=8+EP) (#8876), and pinned --no-enable-flashinfer-autotune across all dsv4 vLLM recipes to prevent accuracy regressions (#9268).
• Qwen3.6-35B Benchmark Recipe: Added a 3-way Kubernetes benchmark recipe for Qwen/Qwen3.6-35B-A3B-FP8 on a single H100 or GB200, comparing vanilla vLLM, Dynamo with frontend-decoding, and Dynamo with embedding cache across throughput and latency metrics (#9392).
• Qwen3-32B-FP8 vLLM Recipe: Added a production-ready vLLM disaggregated single-node recipe for Qwen3-32B-FP8, including deploy and benchmark manifests with a topology of 2× prefill workers (TP=2) and 1× decode worker (TP=4) using NixlConnector KV transfer (#7915).
• Qwen3-235B Architecture-Specific Recipes: Split TensorRT-LLM aggregated and disaggregated deployment recipes for Qwen3-235B-A22B-FP8 into separate Hopper and Blackwell subdirectories, eliminating the need for manual YAML editing when targeting different GPU architectures since TRT-LLM 1.3.x requires different moe_config settings on each (#8470).
• Allow Internal URLs Recipe Fix: Added DYN_MM_ALLOW_INTERNAL=1 environment variable to the Qwen3-VL-30B vLLM agg-embedding-cache recipe, preventing multimodal benchmark failures caused by the new default-deny URL validator rejecting COCO dataset image URLs (#9309).
• vLLM Recipe Flag Rename: Renamed the deprecated --disable-log-requests argument to --no-enable-log-requests in recipe deployment YAMLs and benchmark documentation, preventing vLLM worker pods from crash-looping on startup with newer vLLM runtimes (0.19.x+) that removed the old flag (#8693).
• Qwen3 KVBM Recipe Addition: Removed the Kimi K2.5 KVBM recipe (which required an unreleased container) and added a new Qwen3-32B KVBM recipe with single-GPU aggregated deployment using vLLM (#8475).
• Nemotron-3-Nano-Omni Recipe: Added an experimental recipe for serving nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 via vLLM in aggregated single-GPU mode, supporting text, image, video, and audio inputs with Kubernetes manifests for model caching and deployment (#8799).
• Kimi-K2.5 TokenSpeed Aggregated Recipe: Added a complete deployment recipe for serving nvidia/Kimi-K2.5-NVFP4 on the TokenSpeed engine under Dynamo’s KV-aware aggregated frontend, including a multi-stage local-build Dockerfile, raw Kubernetes Deployments and Services (pending Operator support for TokenSpeed), and a full build/push/deploy walkthrough with TP=4, EP=4, NVFP4 weights, FP8 KV cache, and MLA/MoE plugin configuration (#9231).
• Empty Reasoning Content Fix: Fixed the GPT-OSS reasoning parser producing empty reasoning_content when used with bounded KV cache by whitelisting special tokens needed by the gpt-oss and harmony parsers in the preprocessor, and resolved a Tokio runtime-drop panic during lazy initialization by moving init to a fresh OS thread (#9050).
• Pin Transformers for DeepSeek Perf: Pinned transformers==4.57.6 in DeepSeek-V3.2-FP4 performance job YAML files to prevent silent upgrades to transformers 5.x, which lacks native deepseek_v32 model-type support and caused TokenizerError: Failed to load tokenizer failures during benchmarking (#8690).
• GLM-5 Disagg Frontend Flag: Fixed the GLM-5 SGLang disaggregated deploy manifest to use the correct --enforce-disagg flag instead of the non-existent --no-decode-fallback flag, which would have caused a startup failure (#8914).
• Model Cache Resource Limits: Added Kubernetes resources.{requests,limits}.memory blocks and an absolute HF_XET_RECONSTRUCTION_DOWNLOAD_BUFFER_LIMIT=16GB environment variable to all 15 model-download recipe Job pods, preventing OOM evictions caused by unbounded HF XET reconstruction buffers when HF_XET_HIGH_PERFORMANCE=1 is set (#9884).

Bug Fixes

Kubernetes Deployment

• Operator DGD Reconciliation Lifecycle Fixes: Fixed DGD reconciliation stalling at Ready=False by adding a PodCliqueScalingGroup (PCSG) watch so PCSG status changes trigger the controller (#8328), with an analogous fix for the single-node PodClique readiness path that now reconciles on all relevant status fields (#8423). Corrected planner ConfigMap ownership so profiling-generated ConfigMaps survive DGDR deletion and are adopted under the DGD lifecycle (#8766), added profiler validation that surfaces DGD-plus-service name-length violations as terminal failures (#8807), surfaced workload pod image-pull failures as Warning events on the DGDR instead of requiring kubectl logs --previous (#8815), and rewrote the rolling-update replica calculation to use an availability-gated scale-down budget mirroring the Kubernetes Deployment controller, preventing maxUnavailable breaches during preemption scenarios (#8823).
• Planner Container Image Fixes: Added tail and env utilities to the planner distroless image so container orchestration tooling and exec-based workflows function correctly (#8603), introduced an inline planner_test build stage and unified all framework test image builds through inline _test stages routed via the shared docker-remote-build action with sccache and registry caching (#8739), and included aiperf in the planner builder stage to fix FileNotFoundError crashes during thorough profiling benchmarks (#8769).
• Profiler Sidecar Retry Prevention: Disabled automatic retries for profiling sidecar failures in the Operator, treating them as deterministic issues that require user intervention and now reporting explicit status and error details for clearer diagnostics (#9166).
• DGDR Optimization Type Validation: Added OptimizationType as an enum-constrained string field to the DGDR SLASpec, so users can specify whether SLA-based profiling targets latency or throughput optimization with webhook validation enforcing valid values (#8796).
• Grove Resource Name Truncation: Shortened TRT-LLM disaggregated worker service names from TRTLLMPrefillWorker/TRTLLMDecodeWorker to prefill/decode so that combined Grove resource names fit within the 45-character PodCliqueSet limit, fixing broken DGDR deployments with TRT-LLM disaggregation (#8563).
• Operator Least-Privilege RBAC Hardening: Removed the cluster-admin-equivalent */*: * wildcard rule from the Helm chart’s manager ClusterRole and replaced it with explicit, audited least-privilege rules for every Kubernetes client operation in the operator codebase (#8907), fixed an RBAC privilege-escalation rejection on fresh installs by adding the missing pods/log: get permission required for DGDR profiling job RoleBinding creation (#9969), scoped the Docker secret indexer to the configured restricted namespace so that mgr.GetAPIReader() no longer bypasses namespace boundaries (#9863), and added a projected service-account token volume for the profiling output-copier sidecar to keep it authenticated when automountServiceAccountToken: false is set without exposing the token to the main container (#8771).
• Doubled Model Path Fix: Fixed a bug where build_dgd_config() produced doubled model paths (e.g. /opt/model-cache/opt/model-cache) when a DGDR specified modelCache.pvcName without pvcModelPath, by splitting the PVC branch so that an unset pvcModelPath now mounts the PVC as an HF cache directory while workers receive the HF model ID instead of the mount path (#9177).
• Operator v1beta1 Conversion Fixes: Fixed v1alpha1 ↔ v1beta1 status conversion to preserve legacy scalar componentName values during round-trips (#9184), relaxed v1beta1 component-name validation to accept uppercase letters historically allowed in v1alpha1 service keys (#9192), preserved the legacy DGD worker hash across conversion to prevent unnecessary rolling updates of existing v1alpha1 deployments (#9210), and restructured DGD/DCD conversion helpers with exported structural methods and documented conversion rules in CONVERSION.md (#9257).
• LWS Native Scaling Deployment Fixes: Refactored the DynamoComponentDeployment controller to create a single LeaderWorkerSet object with Spec.Replicas set to the desired count instead of one LWS per replica, removing legacy indexed-resource logic and adding cleanup for old resources (#5468), fixed multinode deployments (vLLM / SGLang / TRT-LLM) by emitting $(LWS_LEADER_ADDRESS) Kubernetes env-var expansion syntax so the kubelet correctly substitutes the leader hostname in direct-python container args (#8369), and restored the LeaderWorkerSet resource name to the <dcd-name>-0 form to avoid a service name collision between the operator-created ClusterIP service and the LWS-created headless service (#9612).
• Frontend Sidecar PVC Mounts: Fixed the auto-generated frontend sidecar container in EPP/inference-gateway deployments to mirror the parent worker container’s volume mounts, so PVC-backed model paths (tokenizer, config, chat-template) can resolve correctly instead of failing on file reads (#8598).
• vLLM Multiprocessing Init Gating: Fixed vLLM multinode pod rendering so the wait-for-leader-mp init container is only injected when the rendered main container actually uses the --distributed-executor-backend mp flag, preventing worker pods on data-parallel or Ray-based paths from hanging indefinitely (#9955).
• Istio Sidecar Injection Exclusion: Excluded Istio sidecar injection from kgateway-proxy pods by adding a GatewayParameters resource with sidecar.istio.io/inject: "false" in the install script, preventing HTTP 500 errors on ext_proc gRPC connections when namespace-level Istio injection is active (#9839).
• DCGM Discovery Namespace-Scoped Mode: Enabled DCGM GPU discovery for namespace-scoped operator deployments by removing the hard-coded short-circuit in validateGPUHardwareInfo and granting cluster-wide pod read access via the gpu-discovery ClusterRole, for exporter pod detection regardless of operator scope (#8365).
• Deduplicate Concurrent GPU Discovery: Added singleflight coordination to DCGM GPU discovery so concurrent reconciles for the same GPU SKU share one in-flight scrape, eliminating redundant DCGM calls and reducing discovery latency under contention (#8797).
• Istio Sidecar EPP Exclusion: Excluded Istio sidecar injection from the EPP pod in both the standalone Helm chart and operator-managed deployments, preventing double-TLS handshake failures that caused cx_connect_fail errors and HTTP 500 responses when secure serving was enabled (#9935).
• Persist Discovered Hardware Metadata: Fixed persistence of auto-discovered DGDR hardware metadata, including interconnect, RDMA capability, GPU SKU, and VRAM, before profiling job creation, then requeued for a clean second reconcile pass to ensure accurate deployment profiling in Kubernetes (#9890).
• Bounded Retry Per-Engine Port: Fixed unbounded cuMemCreate OOM retry by adding a default 60s timeout with elapsed and free/total memory logging, and resolved port collisions for failover engines sharing a pod network namespace by overriding the forward-pass metric port per engine (#8919).
• Operator Go Module Fix: Fixed the go.mod file in the operator deployment to update core dependencies for system stability and compatibility (#8804). Needs verification.
• Forward Pass Metric Port: Injected DYN_FORWARDPASS_METRIC_PORT environment variable into worker pods, for forward pass metrics collection and Planner load scaling (#8817).
• Conditional Container Name Injection: Fixed the Operator to inject CONTAINER_NAME=main only when Kubernetes discovery mode is set to container, preventing unnecessary rolling updates for deployments using the default pod discovery mode (#9366).
• ImagePullSecrets Drift Prevention: Fixed imagePullSecrets drift during operator startup, ensuring consistent secret configuration across reconciliation cycles (#9841). Needs verification.
• Prefill Worker Failover Initialization: Fixed prefill worker missing the FlockFailoverLock initialization that decode workers already performed, which caused prefill standbys to register immediately alongside the primary instead of waiting for failover (#8367).
• Drop Recreate Override Restore-Target: Removed the hardcoded Recreate deployment strategy on restore-target worker Deployments in checkpoint-enabled DynamoComponentDeployments, so the default RollingUpdate applies so scaling no longer evicts the serving cold-start replica before new restore-target replicas are ready (#8434).
• Embedded Pod Template Metadata: Enabled generateEmbeddedObjectMeta=true for Operator CRD generation so embedded Kubernetes ObjectMeta fields preserve labels and annotations, fixing API server pruning that broke DGD-level annotation propagation and prevented failover engine containers from receiving required environment variables (#9553).
• Restore Succeeded Condition Write: Fixed a missing setSucceededCondition call in the operator’s updatePhaseWithCondition path that was accidentally dropped during a cherry-pick to release/1.2.0, which caused the aggregate Succeeded condition to never be written during phase transitions and broke the post-merge operator CI job (#10008).
• GMS ResourceClaim Name Normalization: Fixed GMS ResourceClaimTemplate naming to route CamelCase component names through NormalizeKubeResourceName, ensuring generated names comply with DNS-1123 subdomain validation and preventing Grove reconciliation failures (#9829).
• Pod Discovery Retry Logic: Fixed a race condition in get_deployment_logs() where pod label selectors returned zero results because labels were not yet propagated after deployment readiness, by adding retry logic (up to 12 attempts, 5 seconds apart) with clear warning messages if pods remain undiscovered (#9171).
• Fail Fast on CrashLoopBackOff: Fixed deployment wait logic to immediately detect and handle unrecoverable pod states such as CrashLoopBackOff, triggering automatic cleanup to avoid stalled profiling runs (#9215).

Frontend & Agents

• Tool-Call Streaming Parser Robustness: Preserved logprobs through the tool-call jailing layer so streaming responses no longer return logprobs: null (#8072), fixed guided-decoding handling and corrected finish_reason to tool_calls per the OpenAI spec along with model-specific parser patches (#8442), reconstructed tool-call arguments that were split across multiple streaming deltas instead of returning empty strings (#8582), bypassed the reasoning parser for Qwen3 enable_thinking=False so parallel tool calls are no longer silently dropped (#8589), prevented raw internal protocol tokens from leaking into content when tool-call parsing is interrupted (#8820), and recovered nine silent-drop scenarios across top model parsers when max_tokens or EOS truncates the closing fence or JSON arguments mid-value (#8888).
• XML Parser Whitespace Parity: Fixed the generic XML parser to drop normal text after tool-call blocks while preserving prefix separator whitespace, aligning qwen3_coder, nemotron_nano, and minimax_m2 parity fixtures with the upstream parser parity contract (#9350).
• Auto-Detect Force Reasoning Mode: Added automatic detection of force_reasoning when a chat template’s generation prompt ends with <think>, ensuring the reasoning parser starts in reasoning mode to correctly separate thinking content from normal output (#8240).
• Responses API Wire Compliance: Fixed the /v1/responses input-chain handling to accept Codex and Agents SDK tool-call round-trip shapes (function_call → assistant → function_call_output) that previously caused deserialization failures (#8275), aligned NvResponse serialization with the OpenResponses spec by emitting null for nullable-required fields, injecting missing top-level parameters, defaulting image-input detail to auto, and adding compliance CI (#8283), and removed the hardcoded 4 096-token max_output_tokens default that silently truncated reasoning-model outputs, deferring instead to the engine’s own limit (#9181).
• Structured SSE Error Propagation: Replaced bare or unspecified SSE error comments with properly structured data: {"error":{...}} frames followed by data: [DONE] on mid-stream faults, ensuring OpenAI-style clients receive actionable error messages instead of silent hangs (#8430), and propagated actual backend error messages (e.g. multimodal image load failures) through streaming responses that previously surfaced only as “unspecified error” or premature disconnects (#8674).
• Kimi K2 Parser Correctness: Fixed Kimi K2 tool-call parsing to recover partial tool calls when the model hits max_tokens before emitting the section_end marker, instead of silently discarding them (#8208), back-ported upstream SGLang fixes to widen the tool-name regex to support hyphenated names and added a force-exit from reasoning mode when tool_start_token appears before </think> (#8532), and ensured the preprocessor preserves special tokens required by the kimi_k2 and kimi_k25 parsers so markers like <|tool_calls_section_begin|> and </think> are not stripped during decoding (#9227).
• SGLang Tool Calling Update: Updated the SGLang frontend processor’s tool-calling functionalities to match the latest SGLang implementation, including guided decoding integration for required or named tool choices, streaming result buffering with fallback non-stream parsing, trust-remote-code passthrough, and special token preservation for tool-calling requests (#8269).
• Nemotron V3 Parser Parity: Added nemotron_v3 as a reasoning parser alias and ported vLLM Nemotron v3 reasoning parser test coverage into Dynamo’s Rust implementation, aligning disabled-thinking behavior and streaming/non-streaming extraction paths with the vLLM contract (#9058).
• Chat Message Content Serialization: Fixed non-streaming /v1/chat/completions responses to always serialize the content key as null when absent, instead of omitting it entirely, matching OpenAI’s wire format and preventing client-side KeyError failures for reasoning-only or tool-call-only responses (#8372).
• KServe gRPC Readiness Race Fix: Fixed a race condition in KServe gRPC model_ready/server_ready endpoints where readiness was reported as soon as a ModelDeploymentCard was registered, before the corresponding WorkerSet and engines were attached, causing clients to fail with “Connection closed unexpectedly” on immediate inference requests (#9619).
• Top Logprobs Token Detokenization: Fixed missing token and bytes fields in top_logprobs responses from the SGLang backend by detokenizing token_id values in the Dynamo backend layer when the upstream engine returns None for decoded tokens, ensuring correct OpenAI-compatible logprobs output regardless of backend (#8911).
• Qwen 3.5 Tool Calling: Fixed assistant tool-call argument handling for Qwen 3.5 models, which require special rendering when tool-calling messages contain a list of dicts, and improved error handling in the chat processor to surface similar issues faster (#8792).
• Default Skip Special Tokens: Fixed the Rust backend’s streaming detokenizer to default skip_special_tokens to true when the OpenAI request omits the field, aligning with upstream engines (vLLM, SGLang, TRT-LLM) and preventing special-token text such as <｜begin▁of▁sentence｜> from leaking into content or reasoning_content (#8780).

Scheduling

• Planner Scheduling Operational Fixes: Backfilled max_num_batched_tokens from discovery model cards when the VirtualConnector left it unset, unblocking load-based and throughput-based scaling in aggregated and prefill modes (#8042), normalized model-name case comparisons in KubernetesConnector to prevent CrashLoopBackOff with mixed-case model identifiers (#8384), corrected the profiler Job and example manifests to reference the standalone dynamo-planner image after runtime dependencies were split out (#8407), tracked pending scaling targets in GlobalPlannerConnector so in-flight rollouts no longer trigger stacked scaling decisions (#8422), exposed get_worker_info and fixed readiness signaling in multi-DGD GlobalPlanner topologies (#8482), matched MDC component fields against the backend default instead of the PascalCase DGD service key to restore context-length resolution (#8489), moved inventory and GPU-hour gauge publication out of the throughput-only tick path so Prometheus metrics are emitted in all planner modes (#8575), delegated wait_for_deployment_ready to the pool-local KubernetesConnector so pool Planners no longer skip worker discovery (#8694), and made SLA-mode scale-down decisions consolidation-aware by re-predicting per-survivor load to eliminate 2↔1 oscillation under steady traffic (#9294).
• Latency Ratio Scaling Floor: Used the latency ratio (predicted TTFT divided by target) as a minimum replica floor in the Planner, so that latency-violation signals now drive scale-up decisions instead of being logged but ignored (#8861).
• Scale Status Budget Rejection: Added a ScaleStatus.REJECTED status for GPU budget exceeded responses from the GlobalPlanner, so over-budget rejections now produce a warning log and continued operation rather than raising a RuntimeError (#8774).
• SGLang Prefill CUDA Graph: Fixed SGLang prefill argument normalization to emit --cuda-graph-bs 1 when prefill uses --max-running-requests 1, avoiding duplicated explicit CUDA graph settings in the rapid-path disaggregated generation path (#9852).
• Planner Config Serialization Fix: Excluded environment-dynamic fields (namespace and Prometheus endpoint) from planner_config.json serialization so they are read from environment variables at Planner startup, and injected PROMETHEUS_ENDPOINT into profiler environment from Operator config (#8805).
• Prefill Avg ISL Decay: Decayed the prefill regression average ISL tracker with idle Forward Pass Metrics (FPM) samples so stale long-prompt traffic no longer permanently blocks SLA-driven scale-down (#9759).
• Remove Double Tokenization EPP: Removed redundant double tokenization in the EPP integration by upgrading the Gateway API Inference Extension (GAIE) dependency from v1.2.1 to v1.5.0-rc.2 and using upstream support for pre-computed token ID injection, so tokens are now computed once in GAIE and read directly in the frontend (#8093).
• Enable Priority Hints Routing: Enabled priority hints in the Go Endpoint Picker (EPP) so that requests can forward priority information through the routing system to influence scheduling decisions (#9353).
• TRT-LLM Override Merge Fix: Fixed a crash where thorough-mode profiler generated both --override-engine-args and --trtllm.* flags simultaneously, which TRT-LLM rejects as mutually exclusive, by introducing _merge_overrides_into_args() to detect and merge profiler overrides into an existing JSON blob instead of appending conflicting flags (#9107).
• Persist Endpoint After Handshake: Fixed a routing-chain stall in the disaggregated frontend where transient decode-pod restarts caused permanent inference outages by persisting the prefill endpoint in the activator map after handshake completion, ensuring decode rebuilds can reactivate the PrefillRouter without waiting indefinitely; also resolved a stale DecodeWaiting entry left when decode registers before prefill and is removed before prefill arrives (#8965).
• Router Rejection Threshold Guard: Guarded rejection logic with None when CLI busy-threshold flags (--active-decode-blocks-threshold, --active-prefill-tokens-threshold, --active-prefill-tokens-threshold-frac) are unset, preventing silent fallback values from triggering spurious 503 "All workers are busy" rejections (#8333).
• Prefill DP Rank Balancing: Fixed load imbalance across data-parallel ranks in prefill engines by generating bootstrap_room based on the prefill DP rank instead of a random value, ensuring decode engines correctly route to the intended rank (#9080).
• Cancel Replay Router Tasks: Fixed the online replay router’s LocalScheduler to cancel background tasks during shutdown, preventing replay-scoped tasks from outliving the router in tests and short-lived runtimes (#8429).
• Orphan Cleanup Race Guard: Fixed a race condition in the KV Router where simultaneous startup caused cleanup_orphaned_consumers to see a transiently-empty active-instance set from discovery and incorrectly delete every peer’s NATS consumer (#9132).
• Prefill Router Hash Mode: Disabled EAGLE hash mode for the disaggregated prefill router so it uses prefill-side KV hash semantics instead of inheriting decode-only EAGLE settings, preventing prefill KV events from being indexed under one hash mode and queried under another (#9871).
• FPM ZMQ Port Offset Fix: Fixed data-parallel ZMQ port assignment in vLLM instrumented scheduler by reading data_parallel_index instead of data_parallel_rank, which vLLM resets to 0 for dense models in external DP mode, preventing Address already in use errors when starting multiple DP children (#8696).

Multimodal & Diffusion

• Disaggregated Omni Serving Pipeline Fixes: Fixed multiple bugs preventing disaggregated Qwen2.5-Omni multi-worker serving, including connector routing gated incorrectly by the final_output flag and missing chat template handling so the full thinker→talker→code2wav pipeline completes correctly (#8301), preserved caller-provided runtime device mappings for tensor-parallel workers instead of forcibly overwriting them (#9034), added --enforce-eager to Wan2.2 video launchers to bypass a CUDA illegal memory access caused by torch.compile graph breaks in WanSelfAttention (#9563), and removed the obsolete dummy tokenizer.json placeholder for TTS models that caused a race condition between cleanup and the frontend watcher (#9954).
• LLaVA E/P/D Deployment Fix: Fixed LLaVA Encode/Prefill/Decode (E/P/D) deployment and added LLaVA test coverage for multimodal model profile configurations across multiple topology variants (#8330).
• Prompt Embeds Loading Fix: Used vLLM to load prompt embeddings instead of custom logic, improving safety with stricter type and dimension validation checks (#8228).
• Multimodal Image Hash Collision: Fixed compute_mm_uuids_from_images to include image geometry (width, height) in the blake3 hash preimage, preventing UUID collisions between RGB images with different dimensions but equal pixel counts (#8341).
• Encoder Model Memory Cap: Capped gpu_memory_utilization to 0.2 for the nested vLLM engine in load_vision_model(), preventing a free-memory precheck failure on GPUs with ~24 GiB that caused encoder-only multimodal disaggregated tests to crash on CI (#8466).
• Qwen3-VL Disagg Multimodal Fix: Backported vLLM PR #40932 to remove an invalid deepstack boundary check, unblocking Qwen3-VL disaggregated multimodal inference (#9522).
• SGLang Multimodal Encoder Mode: Enabled encoder_only mode for SGLang multimodal encode workers, preventing the full model (including LLM weights) from loading and avoiding out-of-memory errors on GPUs with limited memory (#9292).

Fault Tolerance & Observability

• Canary Health Check Reliability: Made the canary health check the sole authority on endpoint readiness when enabled, preventing crash loops caused by eager Ready signals racing against canary verification (#8165), replaced the fragile discovery/routing-based canary path with a direct in-process call via LocalEndpointRegistry to eliminate stale-pod and instance-id-mismatch failures in Kubernetes (#8294), and reset the canary timer on request arrival and per streaming chunk to prevent false-negative health check triggers during long-context prefills and extended streaming responses (#8467).
• Preserve Original Model Casing: Removed .to_lowercase() calls on the model label in InflightGuard, ResponseMetricCollector, and HttpQueueGuard so that all dynamo_frontend_* metric families emit the same casing as the originally-registered model card, ensuring dashboard filters match consistently across metric families (#9953).
• KV Publisher Metrics Registration: Fixed the kv_publisher dropped-events counter so it is properly registered and exposed on /metrics, by removing a redundant worker_id variable label that collided with the runtime’s auto-injected constant label of the same name (#8660).

SGLang

• SGLang Disagg Prefill Health Probes: Honored the _HEALTH_CHECK marker in the SGLang prefill handler so the canary probe detects a hung scheduler instead of returning a false-positive 200, and consolidated HEALTH_CHECK_KEY into the shared dynamo.health_check module for wire-format consistency across backends (#8611), with a follow-up that aligned the injected --cuda-graph-bs value to the effective data-parallel size when --max-running-requests 1, preventing a CUDA-graph capture assertion that crashed the prefill worker under DP attention (#9962).
• SGLang Stop Token Forwarding: Fixed the SGLang decode handler to pass stop_token_ids_hidden from the Rust frontend to the engine’s sampling parameters, for proactive EOS detection and avoiding wasted compute on tokens generated past EOS when skip_tokenizer_init=True (#8084).
• Piecewise CUDA Graph Disabled: Disabled piecewise CUDA graph in SGLang launch scripts (agg.sh, agg_router.sh, disagg.sh, disagg_router.sh) to prevent worker crashes caused by CUBLAS_STATUS_EXECUTION_FAILED during warmup when --context-length is smaller than the default piecewise bucket size (#8609).
• Guided Decoding Aggregated Serving: Removed the --skip-tokenizer-init flag from the SGLang aggregated-serving launch script so the grammar backend initializes correctly, for response_format: json_schema and other constrained-decoding features in aggregated mode (#8843).
• Preserve Max New Tokens: Fixed SGLang decode handler to preserve max_new_tokens=None in sampling params, preventing a silent 128-token output cap when clients omit max_tokens from chat completion requests, so generation now correctly continues until EOS (#8743).
• NVTX Profiling Dependency Fix: Fixed the SGLang runtime image to ship the nvtx Python package and expose the nsys binary on PATH, resolving ModuleNotFoundError crashes and command not found errors when using DYN_NVTX=1 profiling (#8629).
• Gated Routed Experts Kwarg: Gated the return_routed_experts keyword argument behind an explicit enable_return_routed_experts opt-in flag and resolved it once at handler initialization rather than per-request, ensuring DeepSeek-V4 compatibility with older SGLang builds that lack the kwarg in their async_generate signature (#8798).
• SGLang Mooncake Dependency Fix: Fixed a missing libjsoncpp25 runtime dependency in the SGLang runtime Docker image that caused ImportError when importing the bundled Mooncake transfer engine (#8645).

vLLM

• Stream Interval Flag Propagation: Propagated vLLM’s --stream-interval flag to the Dynamo frontend via ModelRuntimeConfig.set_engine_specific(), so the flag now takes effect in disaggregated serving mode instead of being silently ignored with a hardcoded default of 20 (#8101).
• Prefill Cancellation Log Race: Fixed an asyncio race condition in _monitor_abort where CancelledError could interrupt engine_client.abort() mid-call, leaving the engine with a dangling request and preventing the “Aborted Request ID” completion log from appearing. The fix shields the abort coroutine from cancellation using asyncio.shield on the no-guard path and asyncio.create_task on the disaggregated-decode guard path, ensuring abort always runs to completion (#8768).
• Disagg Decode Benchmark Unblocked: Fixed two independent bugs in InstrumentedScheduler decode benchmark mode: attached kv_connector_metadata to synthetic SchedulerOutputs (preventing AssertionError on workers with KV connectors in disaggregated deployments) and padded fake-decode prompts by one token to avoid the async sampler’s -1 placeholder causing out-of-vocabulary embedding lookups at batch sizes greater than one (#9360).
• Prompt Embeds Feature Guard: Added validation to reject requests containing prompt embeddings when the feature is not globally enabled, preventing unexpected behavior in deployments that do not use this capability (#8248).
• Disagg Decode Queue Classification: Fixed InstrumentedScheduler._compute_queued to iterate self.skipped_waiting alongside self.waiting, correctly classifying WAITING_FOR_REMOTE_KVS requests as queued decode work so the planner accurately observes decode-engine pressure in disaggregated serving (#8471).
• Deferred vLLM Request Cancellation: Delayed vLLM request cancellation on the decode side until the engine produces its first token, preventing EngineCore crashes caused by aborting a request while NIXL KV transfer is still in flight during disaggregated serving (#8624).
• Max Thinking Tokens Mapping: Fixed silent dropping of max_thinking_tokens at the vLLM worker by mapping it to vLLM 0.20+‘s renamed thinking_token_budget field on SamplingParams, ensuring the thinking-budget logits processor correctly enforces the configured limit (#9571).
• Ray Dependency in vLLM Runtime: Fixed missing Ray binary in the vLLM runtime container image by pinning Ray in requirements.vllm.txt and the [vllm] pyproject extras, preventing crash on pods using the Ray distributed executor backend after vLLM 0.19.1 removed Ray from its dependency graph (#8515).

TensorRT-LLM

• NVRTC JIT Include Discovery: Fixed TRT-LLM NVRTC JIT compilation failures on Blackwell (sm_100a) by installing pip into the runtime venv, so pip show tensorrt_llm can resolve the correct include path for kernel sources (#8296).
• Event Buffer Size Preserved: Fixed the TensorRT-LLM worker to no longer override a user-provided event_buffer_max_size with the default value of 1024, ensuring custom engine configurations are respected and event publishing rates are not unintentionally affected (#9284).
• NIXL Initialization Mode Fix: Skipped NIXL connector creation in aggregated mode where it is unused, and wrapped disaggregated-mode initialization in error handling so missing IB/RDMA hardware surfaces as a warning at init time instead of crashing on first request (#9501).
• TRT-LLM MDC Registration Fix: Fixed TRT-LLM MDC registration so that max_seq_len, max_batch_size, and max_num_tokens supplied via --extra-engine-args or --override-engine-args are propagated back to config before handler and MDC registration, preventing MDC from advertising the model-native context length instead of the engine’s actual limits (#9130).
• Router E2E Test Fix: Fixed the TRT-LLM disaggregated router end-to-end test that was failing due to a missing cache_transceiver_config setting, causing workers to assert on the first request and re-enabled the test in nightly CI (#8954).
• Diffusion Multi-Image Response Fix: Removed the restriction that truncated multi-image diffusion responses to a single entry and stopped rejecting n > 1 requests; the engine now emits a warning when the pipeline’s actual batch differs from the requested num_images_per_prompt, with a 1 ≤ n ≤ 10 validation guard on the handler side (#9853).

KV Block Manager

• Single-Block Cache Match Speedup: Optimized the KVBM prefix-matching path so that single-block lookups probe the first hash before allocating the full result vector, reducing overhead in the common one-block case (#9196).

• KVBM Consolidator E2E Tests: Fixed consolidator end-to-end tests by adding the --kv-events-config '{"enable_kv_cache_events": true}' flag and setting DYN_KVBM_DISK_DISABLE_O_DIRECT=true in worker fixtures, preventing silent consolidator attachment failures and hangs on kernels lacking O_DIRECT support (#8464).

• KVBM Connector Import Update: Updated KVBM’s kv_cache_connector library imports to align with the module reorganization introduced in TensorRT-LLM 1.3.0rc14, dropping compatibility with earlier TensorRT-LLM versions (#9622).

• FlexKV Connector PdConnector Support: Fixed a TypeError in PdConnector that prevented FlexKVConnectorV1 from being used as the first connector when launching disaggregated FlexKV serving, by adding it to the allowed first connector types alongside DynamoConnector and LMCacheConnectorV1 (#8787).

• Main Attention KV Block Size: Fixed KV event block size resolution for vLLM to use the main-attention KV cache group metadata, ensuring hybrid Nemotron-style events align with the router/indexer block size instead of falling back to cache_config.block_size (#9228).

• Replay Wall Time Accuracy: Excluded report bookkeeping from replay wall-time measurement so the metric reflects actual replay execution rather than post-processing overhead (#9190).

• NIXL SDK Dev Stage: Fixed the sglang dev container image by copying the NIXL C++ SDK (libnixl_common, UCX, libfabric, gdrapi.h) into the dev stage, resolving rust-lld: error: unable to find library -lnixl_common failures during source compilation (#9216).

• GAIE kgateway URL Update: Fixed the GAIE kgateway URL to reflect the changed endpoint, ensuring correct gateway connectivity (#9331).

• Sync vLLM Version on Ref: Fixed the install_vllm.sh script to derive VLLM_VER from VLLM_REF when --vllm-ref is passed, ensuring the correct vLLM version is installed instead of silently falling back to the default (#8257).

• Loosened Ray Version Pin: Relaxed the ray==2.55.0 dependency to ray>=2.55.0 in the dynamo[vllm] extra and container requirements, so downstream consumers can use any compatible Ray version without version conflicts (#9439).

• Auditwheel Hidden Directory Copy: Fixed the container build’s dist-packages merge step to include dotfile directories (e.g., .nixl_cu13.mesonpy.libs/) by changing the glob from * to ., ensuring shared libraries like libnixl.so and libserdes.so are correctly copied into the SGLang venv and import nixl no longer fails (#8966).

• CuPy CUDA Version Matching: Fixed runtime Dockerfiles for vLLM and TRT-LLM to dynamically select the CuPy package variant matching the build target CUDA version, replacing a hardcoded cupy-cuda13x that caused ImportError: libcudart.so.12 failures on CUDA 12 builds (#9379).

• XPU Container Build Fix: Fixed the vLLM runtime Dockerfile for XPU builds by guarding cupy-cuda installation behind a CUDA device check and removing leftover nixl-cu* packages that caused import failures in dynamo.nixl_connect (#9419).

• Worker Cleanup Shutdown Ordering: Fixed shutdown ordering so engine cleanup (releasing GPU memory, tearing down PyTorch process groups) runs before the Rust runtime is shut down, and made cleanup idempotent so the signal-handler path and the Worker.run() finally path cannot execute it twice (#8857).

• Triton Backend Directory Default: Fixed the Triton worker container image to set BACKEND_DIR=/opt/tritonserver/backends and updated the launch script to honor this environment variable, resolving a startup failure when using the container quick-start path instead of a local source build (#8697).

• Broken Readme References Removed: Removed invalid readme declarations from dynamo-runtime, dynamo-llm, and dynamo-protocols Cargo manifests that referenced nonexistent files, unblocking cargo package and cargo publish for those crates (#9809).

• Cargo Lock Workspace Consistency: Regenerated Cargo.lock to resolve internal version mismatches for clap and rand dependencies introduced by new workspace members, restoring cargo metadata --locked compatibility on main (#8785).

• Child Exit Code Propagation: Fixed wait_any_exit in launch utilities to correctly propagate a failing child’s exit code instead of silently overwriting it with 0 due to the EXIT trap’s kill 0 looping SIGTERM back to the script (#8883).

Documentation

New Content

• Tool Calling Docs Reorganization: Reorganized tool calling and reasoning documentation into dedicated top-level User Guides sections (docs/tool-calling/, docs/reasoning/), splitting content into Dynamo-native and engine-fallback subpages so each topic is self-contained and easier to discover (#9400).
• Chat Processor Options Page: Added a new chat-processor-options.md overview page and expanded the tool-calling and reasoning parser tables with per-model detail, upstream name divergences (vLLM/SGLang), and cross-links between documentation pages (#8497).
• Fastokens Tokenizer User Guide: Added a user-facing guide for the Fastokens tokenizer backend, covering integration details, enablement criteria, CLI/env-var configuration, model compatibility, verification steps, and troubleshooting, so operators can decide whether and how to enable fastokens on the Dynamo frontend (#9430).
• Agentic Harnesses Blog Post: Added documentation covering prompt stability, reasoning fidelity, and streaming-dispatch optimizations at the harness boundary, detailing how Dynamo preserves KV prefix reuse, maintains interleaved reasoning/tool-call order, and emits explicit tool_call_dispatch SSE events for Claude Code, OpenClaw, and Codex integrations (#7561).
• Sharded Indexer Benchmarking Docs: Added benchmarking documentation for the sharded indexer, including trace dataset download instructions, benchmark execution modes, output metrics reference, and CLI flags (#8695).
• Anchor-Aware BSI Benchmark Docs: Added anchor-aware BSI benchmark results and commands to the KV router indexer documentation, covering steady-state, sweep, and repeated-overload runs with observed tradeoffs such as stronger routing structure, higher routing cost, and hot-branch collapse on dominant-prefix workloads (#9275).
• Kubernetes Documentation Restructuring: Refactored the monolithic AKS guide into focused sub-pages (RDMA/InfiniBand, storage, Spot VMs) and rewrote the main setup flow as a four-step CLI guide, including a full RDMA setup walkthrough with known-issue documentation, and removed deprecated dynamoNamespace references now auto-computed by the Operator (#8777, #9101).
• EFA Setup Guide: Added a guide for using Dynamo with the AWS Elastic Fabric Adapter (EFA) on EKS, covering prerequisites, device-plugin installation, kernel-module verification, LIBFABRIC backend configuration, environment variables, pod security settings, and troubleshooting (#9521).
• Disaggregated Communication Guide: Updated the disaggregated communication guide with AWS EFA benchmark results, including 9.6 GB/s KV transfer bandwidth and 31% lower inter-token latency measured on p5.48xlarge instances, along with expanded EFA configuration details, a kernel compatibility matrix, and EAGAIN troubleshooting guidance (#7764).
• GPU Memory Service Page: Added a dedicated GPU Memory Service Kubernetes documentation page clarifying its experimental status, current limitations with Snapshot, and decision guidance for users choosing between GMS, Snapshot, and failover (#9119).
• ModelExpress with ModelStreamer: Expanded the Kubernetes model-caching guide with customer-facing ModelExpress peer-to-peer instructions, showing how ModelStreamer is selected via MX_MODEL_URI for S3, GCS, Azure Blob Storage, and local/PVC safetensors paths (#9417).
• Planner Replay Benchmarking Guide: Added a guide for benchmarking the Dynamo Planner in replay mode, covering configuration knobs, aggregated vs disaggregated invocation, output artifacts, remote HTML viewing workflows, parallel sweep tips, and a case study showing an engine-startup-time sweep that reveals an SLA cliff near 100 to 120 s startup (#8641).
• Unified Backend Guides: Added Rust and Python unified backend guides, with follow-up fixes correcting a test link, a CARGO_BIN_NAME compile error in integration tests, and a hang in the channel-less generate template when delay is zero (#9581).
• Observability Metrics Docs: Documented the full set of dynamo_component_* metric labels, component values, endpoints, and error types to remove ambiguity around Dynamo-emitted versus Kubernetes-injected labels, corrected the DYN_SYSTEM_PORT default in the health-checks table, and explained Prometheus label-warmup behavior so users understand why labeled metrics only appear after the first matching request (#8478, #8545, #8834).

Looking Ahead

The following is planned for the next release, v1.3.0. See the public Dynamo roadmap for the full plan.

DeepSeek-V4 for Agentic Coding

Performant DeepSeek-V4 recipes across vLLM and SGLang on Hopper and Blackwell, tuned for agentic-coding workloads with KV-cache reuse and disaggregated serving.

Pluggable, Topology-Aware Planner

A re-architected Planner with a plugin framework for custom scaling logic, extending toward topology-aware placement that accounts for interconnect and node layout across heterogeneous worker pools.

CRIU GPU Snapshots

Dynamo Snapshot’s CRIU-based GPU process checkpoint and restore matures toward faster, model-reload-free recovery at scale, paired with operator-driven checkpoint lifecycle management.

Realtime Streaming I/O

A bidirectional /v1/realtime request plane lays the foundation for streaming input and output, including voice and multimodal streaming.

Patch releases

v1.2.1 — Jun 13, 2026

Summary

Dynamo v1.2.1 is a patch release on top of v1.2.0, focused on ModelExpress 0.4.0 engine-side model loading (including object-storage model sources), AMD ROCm / Python 3.10 import compatibility, EFA container build fixes, and backend correctness fixes for SGLang and the gpt-oss-120b recipe.

Base Branch: release/1.2.1

Features & Improvements

• ModelExpress 0.4.0 Integration: Added engine-side ModelExpress model loading to the vLLM and SGLang runtimes (#10578). The runtime images now ship modelexpress==0.4.0 by default (installed with --no-deps so the upstream engine dependency stacks are untouched), the new vLLM path is owned by the ModelExpress vLLM plugin, and the legacy Dynamo-owned --model-express-url / MODEL_EXPRESS_URL wrapper is retained only as deprecated compatibility parsing.

Bug Fixes

• ModelExpress Object-Storage Loading: Fixed a model-load stall on the vLLM and SGLang ModelExpress / RunAI Model Streamer object-storage path (--model s3://…, also gs:// / az://) (#10674). Avoided constructing the vLLM ModelConfig twice (which ran duplicate __post_init__() side effects during engine startup) and made register_model() use the engine’s pulled local directory rather than re-resolving the object-storage URI.
• SGLang Routed-Experts Encoding: Fixed a crash on the first decoded token when --enable-return-routed-experts is used with non-DeepSeek-V4 MoE models on SGLang 0.5.11+ (#10543). SGLang v0.5.11 moved the base64 encoding of routed_experts upstream into tokenizer_manager, so the Dynamo decode handler no longer re-encodes the already-encoded string; nvext.routed_experts is emitted as a base64 UTF-8 string at both emit sites.
• ROCm / Python 3.10 Import Compatibility: Fixed two import-time failures that blocked import dynamo.* on AMD ROCm / Python 3.10 hosts (#10545). nixl_connect now defers the CUDA-only NIXL ImportError until first use so the router, planner, and frontend import cleanly on hosts without a NIXL wheel, and dynamo.common.configuration uses typing_extensions.Self instead of typing.Self for Python 3.10. CUDA behavior is unchanged.
• gpt-oss-120b Recipe Revert: Reverted the gpt-oss-120b TensorRT-LLM aggregated recipe to runtime image 1.0.0 to restore the prior recipe baseline (#10549).

Build, CI and Test

• EFA Container Build Fixes: Backported two EFA container build fixes to the release branch (#10425): bumped nixl_gdrcopy_ref to v2.5.2 for Linux kernel ≥6.15 compatibility, and built and overlaid libfabric v2.5.1 (the first release with the CUDA dmabuf fix for GB200 EFA) onto the EFA installer’s stock binary. Together these restore the Dynamo EFA RDMA container build.