Dynamo v1.5.0

Dynamo v1.5.0 is the 18th feature release of the open-source distributed inference platform, spanning 658 merged PRs from 123 contributors. It opens Dynamo Router worker selection to custom scoring and picking policies and extends KV indexing to Mooncake and disk tiers. KVBM is deprecated, with removal targeted for v1.6.0. Dynamo Frontend streams guided tool calls by default and rejects over-context requests before the first token, and multimodal encode-prefill-decode deployments can decode images once at the Frontend. On Kubernetes, nvidia.com/v1beta1 becomes the CRD storage version, the Rust Endpoint Picker replaces the Go EPP, and Dynamo Snapshot moves to a standalone operator. TLS and mutual TLS now cover the TCP and NATS transports. AIConfigurator becomes AISimulate and ships as its own package, and the engines move to SGLang v0.5.18, TensorRT-LLM v1.3.0rc25, and vLLM v0.28.0.

Get v1.5.0

Pull, deploy, and install with every artifact pinned to the v1.5.0 release set.

docker pull nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.0
docker pull nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.5.0
docker pull nvcr.io/nvidia/ai-dynamo/kubernetes-operator:1.5.0
helm install dynamo-platform https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-1.5.0.tgz
uv pip install "ai-dynamo[sglang]==1.5.0"

docker pull nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:1.5.0
docker pull nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.5.0
docker pull nvcr.io/nvidia/ai-dynamo/kubernetes-operator:1.5.0
helm install dynamo-platform https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-1.5.0.tgz
# TensorRT-LLM ships via the NGC container

docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.0
docker pull nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.5.0
docker pull nvcr.io/nvidia/ai-dynamo/kubernetes-operator:1.5.0
helm install dynamo-platform https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-1.5.0.tgz
uv pip install "ai-dynamo[vllm]==1.5.0"

Highlights

Features & Improvements

Frontend & Agents

• Native Kimi K3 Processing: Added native Kimi K3 rendering, parsing, and tokenization to the frontend, covering thinking, XTML tool calls, and response wrappers. Fixed the SGLang chat processor to split K3 reasoning content and strip the terminal <|end_of_msg|> token (#12396, #12395).
• Pluggable Tokenizer Backend Selection: Added the Baseten tokenizer as a selectable frontend backend via --tokenizer basetenkenizer or DYN_TOKENIZER=basetenkenizer, and made the Hugging Face fallback policy configurable so a failed load no longer silently serves Hugging Face instead (#12376, #12923).

OpenAI API & Validation

• Assistant Message Continuation Support: Added support for add_generation_prompt and continue_final_message in the chat preprocessor, which previously ignored both. Continuation requires add_generation_prompt set explicitly to false, since it defaults to true and the two flags together are rejected; the final message may be from any role, and prompts truncate to match Hugging Face apply_chat_template (#13841).
• Responses API Correctness Fixes: Fixed streamed backend error propagation, tool-call and reasoning ordering, and parallel_tool_calls handling, added chat_template_args support, and added POST /v1/responses/input_tokens for estimated input token counts (#12442, #12462, #13367, #13624, #13768).
• Exact Multimodal Generate Routing: Added routing-only token projection for /inference/v1/generate so image identity feeds KV cache-aware routing without changing execution tokens, and made engine, block size, and LoRA selection atomic per WorkerSet. Malformed placeholders fall back to token-only routing (#11588).
• Opt-In SSE Keep-Alive: Added the DYN_HTTP_SSE_KEEP_ALIVE_INTERVAL_MS environment variable to emit spec-compliant Server-Sent Events (SSE) comment frames during idle streams. Unset, zero, or invalid values keep keep-alive disabled, so the default behavior is unchanged for clients that do not ignore comment frames (#12453).

Guided Decoding & Tool Calling

• V2 Unified Tool-Call Parser: Streamed guided tool calls by default, routed Qwen3 through the v2 unified parser so reasoning stays after tool calls, and served Muse-Glimmer-30B default-on through the same parser, adding --dyn-tool-call-parser muse_glimmer (#12576, #13340).

Reasoning & Thinking Controls

• Deployment-Level Thinking Controls: Added a deployment-level default thinking mode applied at chat-template rendering, with request-level overrides still taking priority, plus fixes for MiniMax M3’s thinking_mode default and Gemma 4 reasoning parsing gated on explicit enable_thinking (#11047, #11739, #12687, #13061).
• Configurable Reasoning Output Field: Added the DYN_REASONING_FIELD_NAME environment variable to select whether chat completion responses carry reasoning content in the reasoning or reasoning_content field, defaulting to reasoning_content. Routed both streaming and non-streaming emit sites through the new selection so reasoning-only chunks are no longer dropped (#11464).

Tokenizers & Model Processing

• Single-Flush Detokenization Metrics: Changed the frontend to publish detokenization Prometheus counters once per stream instead of per chunk, removing cumulative overcounting while still recording partial metrics for cancellations, errors, timeouts, and early drops (#12999).

Agents & Request Classification

• Agent Request Trigger Classification: Added an AgentContext.input_trigger field classifying incoming agent requests as user_message, tool_result, or other across OpenAI and Anthropic APIs, extended to Responses tool-output variants, and carried Codex compaction and thread-lineage metadata into agent context (#12787, #13819, #13012, #12331).

Scheduling

• Two-Tier Cost Worker Selection Policy: Added dynamo-two-tier-cost-fn as a built-in worker-selection policy that ranks workers on active-request load first and device-KV prefix overlap second, and enabled custom-policy by default so a stock build selects it from YAML with no rebuild (#14498).
• Custom Worker-Selection Policy Catalog: Added native WorkerScorer and WorkerPicker policies through the built-in selection path, with a worker_selection YAML catalog for the KV frontend and selection services. Routed built-in strategies through one WorkerSelector::select_worker entry point (#12526, #12536, #13513, #12891, #12585).
• Pluggable Worker Selection Filters: Added composable WorkerFilter policies to custom worker selection, running eligibility first and filters before scorers and the picker, returning HTTP 503 when filters reject every eligible worker, plus a min_effective_overlap_blocks cache-affinity example (#12897, #12896).
• Dynamic Worker Taint Updates: Added an update_model_taints Python API and POST /engine/update/model_taints route across SGLang, TensorRT-LLM, and vLLM worker modes, so base-model routing metadata can change at runtime without restarting workers, waking pending scheduler requests (#12620).
• Tiered KV Event Ingestion: Added an event-driven Mooncake shared-cache index for SGLang HiCache and routed vLLM native-storage KV events to Dynamo Router’s Disk tier with a fail-closed locality policy. Hardened event conversion to fail closed on unrecognized KV event media (#11239, #11571, #12456).
• Remote KV Reuse Hints: Added an experimental router-generated router_hint for backend requests when Dynamo Router finds a better non-target source prefix, a signal that remote KV fetchers such as the early-stage KV Cache Runner (KVCR) can use to pull KV blocks from a peer. Carried router-hint and worker-role metadata through vLLM attachment intents, and preserved KV source-health warnings across endpoint rebinds (#11695, #13725, #12959).
• Conditional Disaggregation Bypass Orchestration: Added Router orchestration of the conditional disaggregation bypass path (#11725), with the bypass behavior exposed through Router configuration (#11723).
• Planner Minimum Endpoint Floors: Added runtime-configurable prefill_min_endpoint, decode_min_endpoint, and min_endpoint settings with a loopback-only GET/PATCH /v1/min-endpoints API, enforced across scaling and budgets, and fixed SLA-driven scaling that left deployments below the configured minimum during a metrics outage (#13038, #14575).
• Power-Aware Planner Scaling: Added Kubernetes-only power-aware scaling where per-GPU caps set through dynamo.nvidia.com/gpu-power-limit annotations are enforced by the Power Agent, and the Planner clamps scaling proposals against total_gpu_power_limit without needing Pod write access (#12012).

KV Event Ingestion & Tiering

• Capacity-Bounded Approximate LRU: Added an experimental, opt-in capacity-bounded LRU eviction policy for Dynamo Router’s local approximate primary indexer, configurable through the CLI, environment variables, and the Python and Rust APIs. Each request lease is released on completion, abort, cancellation, or failure (#13701).
• Multimodal KV Event Normalization: Normalized vLLM image and video object hashes to frontend-compatible pad values on the worker side, including mixed boundary blocks. Propagated an optional video_token_id through the KV-event publisher paths, falling back to native vLLM behavior when mapping is ambiguous (#13910).
• Parallelized Brokered ZMQ Ingress: Reworked the KV Router broker subscriber to poll broker SUB sockets and dispatch batches to bounded per-publisher lanes instead of a lossy shared broadcast channel. A slow publisher no longer blocks the shared reader (#14167).

Multi-DC Relay & Remote KV Hints

• KV DC Relay Contract Alignment: Indexed canonical Dynamo sequence hashes in the DC exact-state cuckoo filter instead of engine-specific hashes, and aligned the VllmResidencyV1 raw-event contract on ownership. Added a transport-neutral RelayPublicationSource interface decoupling publication drivers from gRPC and protobuf (#13471, #13510, #14041).
• Relay Serving Topology Derivation: Added per-namespace model readiness, pool query semantics, and per-worker load observations to the KV DC Relay, derived from deployment cards and the ActiveLoad stream. All facts are consumable in-process through watch and snapshot handles on KvDcRelay (#13539).

Router Configuration & State

• Worker-Advertised Router Configuration: Exposed router flags on workers so a worker set declares its own routing strategy in its model deployment card, reachable from SGLang, TensorRT-LLM, and vLLM. Applied it to the prefill hop, which previously inherited the decode set’s mode (#13192, #13197).
• Hard and Soft Session Affinity: Added a --router-session-affinity-mode hard|soft flag and DYN_ROUTER_SESSION_AFFINITY_MODE variable, with hard as the default. In soft mode the stored session target passes through selection as an affinity target custom policies can override (#13907).

Planner & Profiler

• Profiler DGD v1beta1 Migration: Moved profiler-generated DynamoGraphDeployment resources to the native nvidia.com/v1beta1 spec.components and podTemplate schema, with packaged v1beta1 blueprints for SGLang, TensorRT-LLM, vLLM, and Mocker so profiler behavior no longer depends on public example manifests. Explicitly versioned v1alpha1 DGD overrides remain supported while generated deployments emit v1beta1 (#12506).

Kubernetes Deployment

• llm-d Batch Gateway Integration: Added an experimental llm-d Batch Gateway example for offline inference covering the OpenAI Batch API file and job lifecycle. Gated asynchronous dispatch on frontend model readiness, failing closed with a zero budget when the metric is absent (#12784, #13978).
• Runtime Version Override Field: Added runtimeVersionOverride to DGD, DCD, and DGDR specs as an explicit runtime-compatibility declaration when an image tag lacks one. Included the resolved version in the v2 worker hash so an override matches its equivalent image tag (#10494, #12633).
• Runtime Feature Gates: Added per-feature runtime gates in the Operator whose rendered defaults depend only on the resolved Dynamo runtime compatibility version, including a CanaryHealthChecks gate introduced by Dynamo runtime v1.4.0 and normalized runtime-version comparison. Rendering stays stable across operator-only upgrades, and explicit user configuration remains authoritative (#12421).
• Standalone Snapshot CRD Migration: Migrated checkpoint capture and restore to standalone Snapshot resources, replacing DGD-managed DynamoCheckpoint, moved the capture target to a first-class PodReference.Containers field, and switched restore-compatibility identity to a SHA-256 fingerprint (#14076, #12068, #14724, #13448, #13452).
• GMS V1 Checkpoint Lifecycle: Added an experimental, isolated GPU memory service V1 process-restore lifecycle preserving weight backing across suspend and resume, selected with --use-v1. Added exact-ID weight save and load plus a shared checkpoint lifecycle across weights and KV-cache servers (#12011, #12392, #12989, #14797).
• GMS V1 Engine Integration: Added an opt-in SGLang plugin routing weight and KV-cache allocation through the shared GMS V1 Torch MemPool client, reused for vLLM sleep mode. Fixed SGLang memory saver so GMS workers no longer exit at startup on SGLang v0.5.18 (#13218, #14360).
• LMCache Aggregated Serving Example: Added a DynamoGraphDeployment example and integration guide for running aggregated vLLM serving with LMCache MP as the KV cache backend, sharing tensors with the worker over cross-Pod CUDA IPC (#10185).

Inference Gateway (EPP)

• Reusable EPP Bootstrap Runner: Moved the Rust endpoint picker (EPP) process bootstrap into the dynamo-ext-proc library as reusable run() and run_with_selection_service() functions. Custom linked selection policies can now supply a prebuilt SelectionService without duplicating Dynamo’s TLS, health, metrics, discovery, readiness, and server setup (#12720).
• EPP Graceful Shutdown Handling: Added graceful shutdown coordination to the inference gateway ext-proc server. It marks its gRPC health service NOT_SERVING on SIGTERM or SIGINT and waits a configurable propagation window before stopping new connections, keeping requests off terminating replicas (#13537).

Operator & CRDs

• v1beta1 CRD Storage Migration: Promoted DCD, DGD, DGDR, and DGDR-scaling-adapter storage to v1beta1 while still serving v1alpha1, then fixed conversion fidelity for the metadata envelope, empty container fields, and preserved service annotations ahead of spec defaults (#11904, #12432, #14338, #14855).
• Durable DGD Workload Provider Selection: Made workload-provider selection durable by materializing nvidia.com/workload-provider before reconciliation and treating it as authoritative, and added provider-native providerOverride fields at the DGD, component, and multinode levels, covering Grove topology configuration (#12942, #13273).
• Explicit Multinode Component Roles: Added a typed roles list to DGD and DCD components so multinode leader and worker roles can be declared explicitly and provider overrides attach to the role they configure. Existing implicit multinode manifests remain valid (#14419).
• Performance Optimization Skill Pack: Added repository-native agent skills, roles, guides, and rules so an AI coding agent captures a baseline and workload contract. It deploys and smoke-tests it, benchmarks with AIPerf, and proposes one evidence-backed change at a time (#13557, #13625).

DGDR & Profiling Jobs

• DGDR Profiling Job Overrides: Extended applyProfilingJobOverrides to merge profiling job container overrides by name, so the output-copier sidecar image can be configured without new CRD fields. Fixed a conversion binding bug and added a kubectl check to prevent stuck profiling jobs (#11392, #13153, #14589).
• DGDR KV Routing Support: Exposed features.kvRouter.enabled in the v1beta1 DGDR API, which sets DYN_ROUTER_MODE=kv on the generated frontend container so users can request KV-cache-aware routing without a low-level DGD override, preserved across v1alpha1 conversion (#13079).
• Append Args in DGDR Overrides: Added a $patch: args: append directive to v1beta1 DynamoGraphDeploymentRequest (DGDR) overrides, so container arguments can be added to a profiler-generated blueprint without restating the generated defaults (#13091).

Elastic Expert Parallelism

• Elastic EP Leader Wiring: Started a Ray head on single-pod elastic EP components in Dynamo Operator, so --enable-elastic-ep no longer requires the component to be declared multinode. Added a headless leader Service for follower pods to join (#12943, #13178).

Dynamo Snapshot

• Shared CUDA CustomStorage Snapshot Layer: Added a CUDA CustomStorage operation wrapper shared by checkpoint and restore, bounded helper cancellation through process-group isolation, and persisted and staged the CUDA launch-job file through the source container’s resolved root filesystem (#12488, #12936, #12961, #13252).
• Namespace Bind-Mount Restore Helper: Added a C helper and Go mount package for cross-namespace bind mounting, working around Go’s inability to call setns(CLONE_NEWNS) from a multithreaded runtime. Replaced static path config with on-demand binary injection into the restore container (#12542, #12626).
• Snapshot Shadow Engine Promotion: Added a promotion step to the snapshot restore path so restored shadow engines stay paused until a single winner is elected. The winner resumes and registers with the frontend, preventing several engines from claiming the same GPU (#12869).

GPU Memory Service

• GMS Snapshot Pod Replacement: Added an Operator controller that replaces an owned Snapshot restore-target Pod after its native gms-server sidecar restarts, using UID checks to avoid deleting the wrong Pod. Excluded restore targets from the Grove InterPod failover cascade (#11286).
• GMS V1 Allocation Internals: Reworked GMS V1 create_mapping to carve first-fit regions from large client slabs instead of one allocation per Torch segment, and added allocation backpressure. Renamed V1 sockets by device ordinal for stability and supervised one process per device (#13516, #13262, #13156, #13301).
• GMS KV Layout Commit: Added commit_layout() to the GPU Memory Service, freezing an allocation’s shape while the writer keeps writing, so a standby engine can adopt the same physical KV cache pages instead of allocating an empty pool. Disabled by default behind DYN_GMS_PERSIST_KV (#12648).

Security & Transport

• Operator TLS Auto-Injection: Added 10 TLS certificate path fields to Dynamo Operator’s InfrastructureConfiguration, so TLS and mTLS can be configured once. They auto-inject as NATS and TCP TLS environment variables into every DGD pod, and Dynamo Operator validates the dependency chain at startup (#13689).

Health & Rollouts

• Configurable Monitor Labels and Paths: Added dynamo.metrics.podMonitors.labels, a configurable operator metrics ServiceMonitor label, and dynamo.metrics.podMonitors.frontendPath for the frontend PodMonitor and worker http endpoint, plus Helm unit tests covering the defaults and overrides (#14083, #12716, #13014).
• Forced Grove Scaling Group: Added an opt-in experimental.grove.forceScalingGroup boolean in v1beta1 that renders a single-node component as a Grove PodCliqueScalingGroup instead of a standalone PodClique, so each replica becomes independently gang-scheduled and scales as a group (#11772, #14341).

Multimodal & Diffusion

• Colocated EPD Layout and Benchmarks: Added a reproducible multimodal benchmark sweep comparing one-GPU Aggregated and colocated EPD serving for vLLM and SGLang, and corrected the vLLM disagg_multimodal_e_pd.sh GPU layout so it no longer requires a third GPU (#13466, #13111).
• Frontend Image Decoding for EPD: Added opt-in frontend image decoding for multimodal encode-prefill-decode deployments on SGLang and vLLM. Dynamo Frontend decodes images once and hands RGB pixels plus a content hash to the encode worker over a single NIXL transfer, so pixels never relay through the prefill worker (#11880, #12004). Backend decoding stays the default; frontend-decoded video is not yet supported.
• Multimodal Sidecar Input Support: Added conversion of Dynamo multimodal inputs into vLLM MediaItem messages, mapping mm_hashes to reuse vLLM KV-event keys. Extended the path to forward image, video, and audio URLs as vLLM media modalities to both prefill and decode (#12214, #13717).
• vLLM Custom Encoder Adapter: Added a model-aware Qwen3-VL custom encoder adapter preserving pre-merge grids and DeepStack features, with a Qwen3.5 native external-multimodal example. Replaced finish_reason as an error channel so custom-encoder failures reach callers as a typed HTTP error (#11850, #13658, #13128).
• vLLM-Omni Realtime Audio Streaming: Added realtime session support so streaming ASR models are served through the OpenAI-compatible /v1/realtime WebSocket API. Added incremental audio delivery for POST /v1/audio/speech on vLLM-Omni TTS workers, streaming PCM chunks after one open-ended WAV header (#11830, #12100).
• Dynamic LoRA for Omni: Added dynamic LoRA lifecycle support to aggregated vLLM Omni workers, including load, unload, and list endpoints, request-time adapter resolution for image and video generation, and registration of loaded adapters as distinct frontend-visible models (#11551).

Frontend Media Decoding

• Frontend Media Decoding Upgrades: Added a default-on libjpeg-turbo JPEG decoder (when libturbojpeg is present) and an optimized video-rs FFmpeg path decoding frames in native pixel format. Reworked media_io_kwargs to carry raw JSON for worker-side decoding and kept unknown request fields as an extra_body passthrough (#11157, #11364, #13826, #13817).

Multimodal Routing & Sidecar Inputs

• Multimodal KV Routing Coverage: Added Kimi K3 support to multimodal KV routing, matching routing-side image expansion to the backend prompt shape while retaining existing Kimi behavior. Disabled exact routing when K3 control tokens cannot resolve, and added Qwen video-aware routing (#13371, #14239).

General

• FastVideo Typed API Migration: Updated the FastVideo diffusers example worker to FastVideo’s typed fastvideo.api interface, replacing the legacy generate_video() call with structured config objects and CLI flags. Added max video dimension validation and cancellation cleanup for staged MP4 files (#9591).
• Per-Request Encode Buffer Reuse: Reused a single 8 KiB encode buffer across all frames of a push-egress request and read the request-plane codec setting once per request instead of per frame. This cut the encode step for a single-token frame from 312 ns to 232 ns (#13983).

Engines

• Heterogeneous XPU CPU Disaggregation: Added example launch scripts for disaggregated vLLM serving that run prefill on XPU workers and decode on CPU workers, including a router variant with GPU-direct KV transfer. Users can now reproduce heterogeneous prefill-decode deployments (#7977).
• Neuron NIXL Connector Registration: Registered NeuronNixlConnector from vllm-neuron in the vLLM KV connector protocol registry, so disaggregated prefill/decode serving on AWS Trainium over NIXL routes through Dynamo without additional code changes (#13771).
• Elastic Expert-Parallelism Scale-Up: Added runtime elastic Expert-Parallelism scale-up control to the SGLang worker, exposing /engine/control/scale_elastic_ep and /engine/control/is_scaling_elastic_ep routes that integrate a joining group’s GPUs while the primary keeps serving, requiring SGLang v0.5.16 or newer (#12575).
• Conversation Affinity Rank Owner: Added the --conversation-affinity-dp-rank-source {engine,dynamo} flag and matching environment variable for the TensorRT-LLM backend, so Dynamo can own the initial attention data-parallel placement of a conversation while TensorRT-LLM preserves the resulting affinity (#12225).

vLLM

• FPM Self-Benchmark Fidelity: Seeded decode self-benchmark points from KV produced by real prefill chains and salted every prompt with a unique cache_salt, closing the gap between benchmarked and production decode latency on MoE models and providing the scheduler interface the AISimulate v0.12.0 collector requires (#14029).
• vLLM Self-Benchmark Capacity Sync: Synchronized self-benchmark capacity envelopes across attention-DP ranks so all ranks generate identical grids instead of deadlocking. Added variable-length prefill batch sampling behind an off-by-default flag and rejected --benchmark-mode with data-parallel size above one for dense models (#12021, #12627, #14292).
• vLLM Embedding Worker Pool: Added a multi-process Dynamo endpoint pool sharing one EngineCore per GPU and aligned embedding tokenization with vLLM pooling defaults. Fixed the pool dying at startup with multiple embedding-worker processes and preserved prompt embedding failures as typed errors (#12157, #14449, #13381).
• Elastic EP Capacity Reporting: Added a read-only control/ep_capacity endpoint reporting how a vLLM worker is parallelized and how many GPUs are idle. Derived the elastic EP scale floor from tensor parallel size instead of a hardcoded minimum of two data-parallel ranks (#13179, #12937).

TensorRT-LLM

• Response Streaming Path Performance: Inverted the TensorRT-LLM decode worker egress path so the Python handler pushes responses into a Rust ResponseSender backed by a bounded tokio channel, keeping tokio threads off Python. Stripped num_postprocess_workers from merged engine args before initialization (#12845, #12592).
• TensorRT-LLM Engine Metrics Accuracy: Reported the exact per-request cached_tokens value from the engine result instead of reconstructing it from block-level metrics, always emitted prompt_tokens_details, and gated SamplingParams.return_perf_metrics on --publish-kv-events instead of hardcoding it (#13567, #13922).
• TensorRT-LLM Prefill Completion Publishing: Added worker-side first-token completion publication in TensorRT-LLM using the shared endpoint-scoped source, binding the source per request and notifying once on the first substantive token output. When the source is unavailable, the existing Router response fallback is preserved (#13672).

Experimental

Dynamo Sidecar, the Triton backend, and the SGLang RL rollout contract are experimental in v1.5.0. Their interfaces, defaults, and image layout may change without a deprecation cycle. Dynamo Sidecar runs beside a stock inference engine over its native gRPC API and does not yet match every feature of the in-process backends.

• SGLang Sidecar KV-Routing Examples: Added experimental aggregated and disaggregated SGLang sidecar deployment examples with exact KV-aware routing, pinning SGLang v0.5.18 as the first version whose gRPC GetServerInfo reports the structured KV-event descriptor the sidecar requires (#14127). Inference through this path returns 500 on the bundled SGLang (see Known Issues).
• vLLM Python Sidecar Launchers: Added a wheel-installed dynamo.vllm.sidecar Python launcher forwarding CLI arguments through the PyO3 backend, plus launch examples and removal of unused OTEL labels. Fixed vLLM-Omni plugin auto-loading that broke every sidecar inference request (#13923, #13735, #13864, #14731).
• Engine-Native SGLang Generate Endpoint: Added an opt-in SGLang-native POST/PUT /generate endpoint for streaming token-in/token-out inference, replaced the call_tokenizer_manager passthrough with allowlisted /engine/* routes, and advertised native generate capability when GetServerInfo reports the co-located HTTP port (#11640, #12246, #13659).
• TensorRT-LLM Python Sidecar Launcher: Added a wheel-installed dynamo.trtllm.sidecar Python launcher forwarding CLI arguments through the PyO3 backend. Installed the smg-grpc-proto package before starting the gRPC engine in both the launch script and container, since gRPC is the sidecar’s only transport (#13924, #14286).
• Dynamo Triton Runtime Backend: Added a dynamo.triton backend that serves NVIDIA Triton models through Dynamo, launched with python -m dynamo.triton, with model discovery, configurable server settings, health checks, logging, metrics, and an example identity model (#13774). The backend is experimental and ships as source with its own container recipe. Render it with container/render.py --framework=triton --target=runtime and build on an nvcr.io/nvidia/tritonserver release image, selected with RUNTIME_IMAGE_TAG. No prebuilt Triton runtime image is published for v1.5.0.
• SGLang RL Rollout Contract: Added an experimental, opt-in reinforcement learning (RL) rollout contract for SGLang workers: nvext.detailed_finish_reason reports Dynamo’s canonical finish reason such as cancelled on a pause abort, nvext.extra_fields carries prompt_token_ids and sampled logprobs, a --publish-sglang-generate flag advertises the native /generate endpoint, the engine’s native pause and memory controls are exposed under /engine/* and drive Dynamo discovery, and worker output_ids keep matched stop tokens so training sees exactly what the engine sampled. A Slime example runs two SGLang engines with the in-container sidecar through this contract. Everything in this set is experimental in v1.5.0 and tested only on the example paths (#13640, #14525, #13951, #14317, #12562, #12856).
• Split gRPC Control Discovery: Moved the vLLM sidecar and mocker to vLLM’s split native Inference and Control gRPC services, discovering model identity, capacity, parallelism, KV-event sources, and readiness through Control. Removed the required model-path configuration while preserving existing launch, deployment, and documentation surfaces (#12734).
• Deterministic vLLM DP Routing: Added forwarding of Dynamo’s selected data-parallel rank as per-request x-data-parallel-rank gRPC metadata in the vLLM sidecar. Required a unique ZMQ KV-event source per rank when KV routing is enabled, and preserved cache-salt propagation for consistent block hashes (#12735).
• SGLang Sidecar KV Routing: Added event-driven KV routing for the native SGLang sidecar by consuming the KV-event publisher descriptor from GetServerInfo.json_info. Validated publisher type and topology before worker registration, and preserved native Clap exit behavior for the wheel-installed sidecar launcher (#13579, #13929).

Reinforcement Learning

• vLLM RL Control Plane: Routed Dynamo vLLM sidecar reinforcement-learning controls and model updates through vLLM’s native vllm.Control gRPC API, and added capability-gated pause and resume, sleep and wake, and weight-transfer routes. Published RL worker metadata with world size computed from parallelism dimensions (#13066, #13261, #13607, #14840).
• RL Route Discovery Endpoint: Added an opt-in RL request-plane endpoint behind --enable-rl or DYN_ENABLE_RL. It reports a worker’s engine-route names and system-server URL for the SGLang, TensorRT-LLM, and vLLM sidecars plus the Rust mocker, and ships as a versioned discovery response (#13260, #13606).

Fault Tolerance & Observability

• Request-Trace S3 Sink Durability: Added a native S3 RequestTraceSink that batches request-trace records as gzipped JSONL behind an opt-in cargo feature. Hardened shutdown to await the final flush and drain outstanding records, and corrected a shutdown drop summary that previously undercounted (#11806, #13298, #14642).
• Model and Worker Readiness Signals: Added the frontend-authoritative dynamo_frontend_model_ready{model} Prometheus metric reporting whether Dynamo Frontend can route requests through a complete serving topology. Fixed Rust backends that left the health route reporting NotReady after the worker began serving traffic (#13976, #14458).

Request Migration & Tracing

• Request Migration Reliability Hardening: Added a WorkerOverloaded error class and classified TCP connection failures so migration retries another worker. Hardened retry bookkeeping against overflow, exposed migration recovery latency via a histogram, and added a span recording migration reasons per worker attempt (#11647, #12927, #13600, #12930, #11133).
• OTLP Export Configuration Hardening: Decoupled the stderr console log format from OpenTelemetry export by adding DYN_LOGGING_CONSOLE_FORMAT, keeping the legacy DYN_LOGGING_JSONL setting as a fallback. Fixed a panic when logging::init() ran from a synchronous entrypoint with OTEL export enabled (#12424, #11143).

Metrics & Readiness

• ThunderAgent Router Observability: Added ThunderAgent Router observability through INFO and DEBUG logs for program lifecycle and route decisions, plus new status and metrics runtime endpoints. Added opt-in route proof so operators can confirm traffic passed through ThunderAgent and inspect live program and worker state (#12196).
• Model Alias Metric Labeling: Resolved aliased model names to the primary served model name in frontend metrics, so dynamo_frontend_model_ready emits a single readiness series per deployment. Logged each alias claim as it registers to show which worker roles claimed which name (#14364, #14365).
• Generate Route Frontend Metrics: Added frontend metrics for the token-native /inference/v1/generate route, observing raw engine deltas and split-router metadata, and starting request, queue, and latency lifecycles before frontend processing. Canonicalized model aliases before engine selection so metric labels stay bounded and consistent (#11385).
• EPP Cached Token Metrics: Added parsing of OpenAI-style usage fields including usage.prompt_tokens_details.cached_tokens in the inference gateway endpoint picker, exported as dynamo_epp_cached_tokens Prometheus metrics, and validated that Envoy sets response_body_mode to FULL_DUPLEX_STREAMED (#11868).
• Embedding Cache Metrics: Added Prometheus metrics to the vLLM multimodal embedding cache connector, emitting the dynamo_component_embedding_cache_{hits,misses,evictions}_total, _utilization, _current_bytes, and _entries series. Cache hit rate, occupancy, and eviction pressure are now visible in production dashboards under one metric family whichever cache implementation serves the model (#11969).
• Estimated Reasoning Token Reporting: Added an estimate of reasoning-token usage in the shared Chat Completions stream, populating completion_tokens_details.reasoning_tokens for /v1/chat/completions and /v1/responses in both streaming and non-streaming modes, with positive backend-provided counts still treated as authoritative (#12181).
• Engine Request ID Mapping: Added an INFO log line in the TensorRT-LLM handler that records the Dynamo request UUID alongside the TensorRT-LLM executor client ID and the disaggregated request ID immediately after submit. Engine-internal timing records can now be joined offline to Dynamo request traces (#13970).

Performance Modeling & Replay

• Unified AISimulate CLI Integration: Registered the Dynamo runner factory and Router and Planner configuration adapters for aisimulate predict and aisimulate recommend, removed the legacy python -m dynamo.replay CLI, and consolidated dependencies into the published aisimulate wheel (#13665).
• Mooncake Request Trace Converter: Restored the request_trace_to_mooncake binary in dynamo-bench behind an opt-in feature, so callers can export Dynamo request traces to reusable Mooncake JSONL files, published through a same-directory temporary file renamed after successful conversion (#12504).

Mocker

• Offline Replay Hot-Path Optimization: Reduced offline-replay coordinator, cache-index, and completion-event overhead through O(1) worker tracking, an ordered evictable-leaf index, and removed decode accept-length rescans, with public APIs, configuration, and KV-event formats left unchanged (#12341, #12772).
• Mocker KV Accounting Rework: Moved per-request KV ownership into move-only backend leases, and switched vLLM and SGLang decode timing to summed logical KV tokens instead of distinct resident blocks. Published buffered KV events at pass completion instead of pass start (#12329, #12583, #12653).
• Typed LiveEngine Handoff Control: Added a typed LiveEngine boundary for the Mocker scheduler with configured KV-event and FPM publishers, request registration, and idempotent graceful shutdown. Aligned live disaggregated prefill with offline replay by using the first replay-plan token (#12380, #13635).
• Mocker Cached-Token Usage Reporting: Added cached prefix token counts, taken when the request is accepted, to vLLM-mode mocker output as completion_usage.prompt_tokens_details.cached_tokens. Multi-worker simulations can compare the Router’s radix overlap estimate against tokens the scheduler actually reused (#12711).

Infrastructure Modernization

• TLS and mTLS Transport Encryption: Added opt-in TLS encryption to the TCP response and request dispatch planes, and covered the NATS transport with matching environment variables and tls:// URL support. Layered opt-in mutual TLS with client certificate verification on top (#10921, #12533, #13096, #13528).

Recipes

• Qwen3.8-2.4T-A95B Recipe Support: Added the Qwen3.8-2.4T-A95B recipe to the catalog with SGLang and vLLM deployment options for GB200 and GB300, covering chat, agentic, and aggregated configurations (#13105). Corrected the model ID to Qwen/Qwen3.8-2.4T-A95B-FP8 across recipe manifests, model download commands, and docs to match the validated checkpoint (#13265).
• Kustomize Recipe Variant Rendering: Added a Kustomize workflow for provider and fabric recipe variants that keeps fully materialized deploy-*.yaml manifests checked in and reviewable. Converted the Qwen3-32B vLLM cloud-provider variants and the Qwen3-235B-A22B-FP8 TensorRT-LLM Blackwell disaggregated variants (#10989). Value-aware Component templates are selected by source and an overlay-relative path. They render through strict sandboxed Jinja against a resource index from the built base and materialize as normal local Components, so users need neither Python nor Jinja. Replaced the fixed aws-efa-p8d8, aws-efa-p8d16, and aws-efa-p16d16 Components with AWS instance-type selections, where P5 adds its instance-affinity patch and P6 uses the family defaults (#13125).
• Qwen3.5-122B FP8 H200 Recipes: Added H200 deployment recipes for Qwen/Qwen3.5-122B-A10B-FP8 on vLLM. Coverage spans an aggregated topology (TP2 with MTP, KV-aware routing, two replicas) and a disaggregated 1P2D topology using NIXL KV transfer over InfiniBand, plus model cache and benchmarking manifests (#12088).
• Nemotron-3.5-Lightning Deployment Recipes: Added recipes/nemotron-3.5-lightning with vLLM aggregated and disaggregated profiles, TensorRT-LLM aggregated profiles, and model-cache manifests. Added performance traces for B200, GB200, H100, and H200 GPUs, plus documentation and catalog entries so the recipe appears in the recipes overview (#13021).
• Nemotron-3-Ultra Refresh Recipes: Added 12 Nemotron-3-Ultra Refresh deployment profiles for B200, GB200, and H200 at 256K and 1M context lengths, covering aggregated two-worker and disaggregated topologies, and replaced the legacy Day-0 profiles. The profiles pin the public Dynamo v1.4.0 vLLM runtime image and use KV-aware routing, prefix caching, FP8 KV cache, and hybrid KV cache management (#13226).
• GLM-5.3-Flash vLLM Recipes: Added aggregated and disaggregated 1P1D vLLM deployment recipes for GLM-5.3-Flash, a GLA/KDA hybrid attention model, on GB200 NVL72, covering text and image requests, and added the missing Qwen3.8-2.4T-A95B catalog card (#13899).
• Nemotron-3-Super Dev Image Revert: Reverted the Nemotron-3-Super vLLM recipes from the generic vllm-runtime:1.3.0 release image back to the dedicated vllm-runtime:1.3.0-nemotron-super-dev.1 build, which carries the model-specific vLLM 0.21.0 patches the recipe was qualified against. Marked the recipe as Dev-only in the Model Early Access Builds reference (#12520).
• DeepSeek-V4 Checkpoint Download Parameterization: Restored the MODEL_NAME environment variable in the shared DeepSeek-V4 Pro and Flash model-download Jobs, defaulting to the public FP8 checkpoint that most variants request, and documented the NVFP4 override for the agentic B200 variants. Because each variant mounts the model-cache PVC read-only with offline Hugging Face settings, a checkpoint the Job did not download was not fetchable at runtime and the worker never started (#13034).
• Obsolete Router Flag Removal: Removed the --router-reset-states flag from the DeepSeek V4 Flash and Pro vLLM recipes for B200 and H200, and from the load-only routing documentation, so deployments no longer pass an obsolete option (#13116).
• SGLang GB200 Frontend Args: Fixed the Qwen3.8-2.4T-A95B-FP8 GB200 SGLang deployment manifests by removing --model-path and --model-name from the frontend and replacing --dyn-tool-call-parser with --tool-call-parser, which previously caused the frontend to exit at startup (#13324).
• Nemotron 3.5 Lightning Recipe Refresh: Updated the Nemotron 3.5 Lightning recipes to use BF16 weights on B200 and GB200. BF16 measured +80% over NVFP4 with Marlin dense and MoE backends and +12% over NVFP4 with CuTeDSL. Refreshed performance data, corrected the documented context length, added AWS EFA setup instructions, and removed redundant recipes and variants that did not meet the benchmark SLAs (#13370).
• DeepSeek-V3.2 and Llama-3-70B Recipe Removal: Removed the deepseek-v32-fp4 and llama-3-70b recipe directories along with their documentation pages, catalog entries, navigation, and dependent feature-benchmark pages for WideEP routing and Llama topology. Refreshed READMEs, codeowners, and remaining references that pointed at the removed paths (#13663).

Bug Fixes

Frontend & Agents

OpenAI API & Validation

• Client Error Status Mapping: Returned proper HTTP 400 errors, instead of streamed 200s or plain-text failures, for invalid requests, template failures, out-of-range parameters, and unsupported multimodal content across /v1/chat/completions, /v1/responses, and /v1/messages (#12155, #12036, #12404, #13006, #14056, #13102).
• Backend Error Status Propagation: Preserved backend 503 and request-rejection status codes instead of masking them as generic 500s, and fixed non-streaming completions so typed errors like Backend(InvalidArgument) keep their original HTTP 400 status and message (#12412, #12966, #12706).
• Single Role Emission in Streams: Fixed chat streams emitting delta.role on every chunk instead of only the first, in both the Rust delta generator and the SGLang streaming postprocessor. Preserved the initial role-only chunk when generation finishes with no content (#12639, #12741).
• Explicit Zero Temperature Preservation: Fixed chat-completions request-template handling so an explicitly supplied temperature: 0 is preserved instead of being overwritten by the configured template temperature, keeping deterministic greedy requests from becoming sampled requests (#12640).
• Per-Choice Tool Dispatch Dedup: Fixed streaming tool dispatch dropping tool_call_dispatch events when n > 1 produced the same tool-call ID across completions, by keying the dedup guard on choice index plus ID. Repeated IDs within one choice still collapse to one event (#12680).
• Split UTF-8 Character Preservation: Fixed multi-token UTF-8 characters showing up as replacement characters (U+FFFD) in the SGLang frontend’s incremental decoding, by tracking committed decode context separately from pending token IDs and committing only at clean boundaries (#12688).
• Converted Protocol Request Validation: Added validation for Responses and Anthropic requests after conversion to the shared Chat format, rejecting empty message lists, out-of-range sampling parameters, and malformed content blocks with 400 before the request reaches the backend (#12809).
• Runtime Shutdown on Startup Failure: Fixed the HTTP and gRPC entrypoints to cancel the distributed runtime before propagating a server startup or bind error, for both discovery-backed and in-process engines. A failed bind no longer leaves the runtime running (#13415).
• Cancelled Request Intake Rollback: Fixed router state corruption when a request was cancelled while being accepted, which left stale capacity or wait state or overwrote a later request for the same session. Request intake is now serialized per session with guarded rollback (#13553).

Guided Decoding & Tool Calling

• Reasoning Parser Lifecycle Fixes: Fixed reasoning-parser lifecycle bugs that leaked <think> text into visible content or dropped it when it should have shown. Covered Nemotron force-nonempty-content requests, include_reasoning=false responses, interleaved n>1 streams, and text-grammar parsers missing their declared sampling params (#11251, #13487, #11563, #12541).
• Tool Call Guided Decoding Constraints: Fixed tool-call guided decoding constraints for vLLM and SGLang across named, forced, and zero-argument tool choices, schemas with reference cycles, double-constrained vLLM reasoning, and Kimi K2 calls wrongly constrained as generic JSON (#12684, #12878, #12795, #13235, #13785, #12638).
• NvExt Metadata Preservation Fixes: Fixed NvExt metadata loss across the frontend, EPP, and legacy streaming tool-call guard, the buffer that holds back partial tool-call markup. The loss dropped cache_salt, x-tenant-id, nvext, engine_data, token IDs, timing, and routed-expert data from responses. That metadata now survives intact (#13157, #13730).
• Tool Call Argument Normalization: Fixed tool call arguments arriving as JSON strings instead of objects, which produced empty or garbled tool calls in multi-turn conversations for templates like GLM-5.2. Malformed values are now replaced with an empty object and logged (#12332).
• Truncated Tool Call Recovery: Fixed dropped GLM <tool_call> XML blocks when max_tokens truncates a response. The partial block now returns as raw content in both the streaming and non-streaming chat completion paths, so clients no longer receive an incomplete turn (#12333).
• Unconsumed Control Marker Stripping: Fixed unconsumed <|...|> control markers leaking into user-visible content when a reasoning or tool parser was active, previously forcing clients to choose between empty reasoning_content and visible markers. Stripping now applies uniformly to streaming deltas and plain-text responses alike (#12510).
• vLLM JSON Tool-Call Fallback: Fixed forced vLLM tool choices using the JSON-schema fallback so generated JSON returns as proper tool_calls instead of assistant content, across required and named choices, streaming, non-streaming, and parallel tool calls (#12908).
• Request-Scoped Tool Call Parsing: Fixed false-positive tool-call parsing by checking each request’s tools and tool_choice fields across Chat Completions, Responses, and Anthropic Messages, disabling ordinary parsers when no tools are declared or tool_choice is none (#13359).
• Tool-Call Parser Name Validation: Added validation of tool_call_parser values against Dynamo’s v1 registry and unified parser families during model registration. Returned an error that names the unsupported parser and lists the available ones instead of failing silently during response aggregation (#13828).

Tokenizers & Model Processing

• Serialized Tokenizer Padding Disabled: Fixed models that ship fixed tokenizer padding, like nvidia/NVIDIA-Nemotron-Parse-2.0, expanding short prompts into padded token IDs and failing context-length validation. Cleared serialized Hugging Face padding when loading tokenizer.json, matching how Dynamo already clears serialized truncation (#12656).

Logprobs & Usage Reporting

• Chat Logprobs Reporting Fixes: Fixed OpenAI-compatible chat logprobs so SGLang responses return populated logprobs.content instead of null, preserved explicit top_logprobs=0, and fixed chosen-token and zero top_logprobs reporting (#12820, #12444, #12650, #13209, #14621).
• vLLM Chat-Processor Path Fixes: Fixed the vLLM Python chat-processor path, used with --dyn-chat-processor vllm, so it reports completion_tokens_details.reasoning_tokens instead of omitting it. Worker-side sampling and grammar rejections now surface their original status and message instead of a generic HTTP 500 (#13947, #13980).

Anthropic Messages API

• Anthropic Messages Response Fidelity: Brought /v1/messages streaming to usage parity with /v1/chat/completions, emitting token usage on every chunk including tokens before a client aborts, reporting cache_creation_input_tokens as 0 instead of omitting it, with native toolu_ tool-call IDs (#12473, #12472, #12614).
• Typed Tool-Call Guard Error Preservation: Fixed the v1 streaming tool-call guard collapsing typed backend errors like Backend(InvalidArgument) to Unknown by chaining terminal errors instead of reconstructing them. Error messages on /v1/chat/completions, /v1/responses, and /v1/messages no longer carry an internal prefix (#13930).

Agents & Request Classification

• SGLang Priority Hint Routing: Fixed the SGLang preprocessing path treating all requests as default-priority traffic by projecting nvext.agent_hints into internal routing hints for priority, strict priority, expected output tokens, and latency sensitivity. Priority-aware Router policies now see SGLang requests correctly (#12628).

LoRA

• LoRA Adapter Unload Handling: Fixed unloading one LoRA adapter making /v1/chat/completions and /v1/responses return 404 for its base model and unrelated models, by deriving endpoint retraction from the live model catalog instead of the removal event alone (#14216, #14548).
• Shared S3 LoRA Credentials: Fixed S3 LoRA credential resolution to use the AWS SDK default provider chain, including shared profiles. Honored AWS_ENDPOINT_URL through the object-store environment builder, so custom S3 endpoints and profile-based authentication now work for LoRA adapter downloads (#13844).

Scheduling

Worker Selection Policies

• Unknown Worker Target Rejection: Returned HTTP 400 naming the field and the rejected id when an explicit nvext worker target is not in the endpoint’s discovery snapshot, across built-in and KV routing, prefill, and route previews (#14858).
• Full Request Body Forwarding: Fixed the Endpoint Picker’s KV-router scorer forwarding a reconstructed {role, content} view instead of the original request body. The reconstruction dropped tool_calls, tool_call_id, name, and tools, and caused multi-turn tool-calling requests to fail with 503 “no healthy upstream” (#11991).
• Post-Prefill Decode Dispatch Fix: Fixed disaggregated serving leaving decode-bound KV blocks pinned until expiry when the client disconnected after remote prefill staged them, instead of dispatching the decode request anyway. Cancellation now keys off staged KV instead of request phase (#13855).

Router Configuration & State

• Local Output Block Accounting: Fixed the KV Router so one frontend replica’s local output-block mutations no longer overwrite shared worker load on the shared load channel with its own state. Prometheus load observation and normal add, prefill, and free lifecycle publishing are unchanged (#12514).
• Stale Worker Pin Recovery: Fixed ThunderAgent session handling so a program pinned to a replaced backend worker is reassigned to a live worker when the request is accepted. Previously it returned HTTP 500 until the session ended or Dynamo Router restarted (#12783).
• WorkerSet Monitor Lifecycle Scoping: Fixed the frontend KV worker monitor task outliving its WorkerSet. Repeated discovery churn, like LoRA load and unload cycles, had accumulated NATS subscriptions, etcd watches, and per-worker state that grew file descriptors and memory (#13077).
• Bounded Worker Load Publishing: Fixed Dynamo Router worker-load publishing growing unbounded with request rate by routing snapshots through a single publisher task that keeps only the newest snapshot per worker and data-parallel rank. This bounds pending state by worker count instead. Wire formats are unchanged (#13435).
• Router Weight Download Skip: Fixed GlobalRouter model registration to pass ignore_weights=True in both disaggregated and aggregated modes, so a routing-only process no longer downloads inference weights and delays model discovery (#13916).
• Disaggregated Prefill Error Propagation: Fixed invalid-argument errors from the prefill worker flattening into opaque 500 responses in disaggregated deployments, by folding the worker’s error text into the prefill router’s message. Bad sampling parameters or invalid schemas now return 400 (#14063).

Planner & Profiler

• Actionable Model Path Trust Errors: Reported the original filesystem cause with ownership guidance, and the option to set modelCache.pvcModelPath to the real snapshot path, when a local model path cannot be inspected during DGD materialization, instead of an unhandled error or a silent fallback to a Hugging Face id (#14860).
• RAPID Profiling Mode Fixes: Fixed RAPID profiling for mocker-only DGDRs, which ignored declared spec.workload.isl/osl and mis-sized every worker at --max-model-len 6500. Selected AIC or NPZ profile data via spec.searchStrategy and passed workload sizes through on the fallback path (#12573, #14205).
• Planner KV Hit Rate Namespace: Fixed the Planner’s get_avg_kv_hit_rate querying the wrong namespace on operator-managed deployments, where router metrics carry a suffixed namespace the old filter never matched, sizing as if every request missed the prefix cache. It now falls back correctly (#13480).
• Worker Prefill Completion Events: Added MarkPrefillCompleted publication from LLM workers on first token or first output chunk. Made Dynamo Router consume these marks even with replica sync disabled, so duplicate and stale events become silent no-ops instead of triggering repeated wakes (#13655).
• Planner Plotext Version Pin: Pinned plotext below version 6 in the Planner requirements, since plotext 6 no longer exports the top-level plot_size function used by AISimulate (formerly aiconfigurator), which caused every profiler path to fail in image builds (#13716).
• Profiler DP and Decode Fixes: Fixed SGLang prefill workers starving attention ranks by clamping --max-running-requests to the data-parallel size, and rejected invalid SGLang data-parallel and request-limit values during profiler config generation. Stripped conflicting duplicate vLLM --disaggregation-mode arguments too (#14351).
• THOROUGH Profiling Manifest Compatibility: Fixed a KeyError: 'services' crash that stopped THOROUGH profiling from deploying its first candidate. DynamoDeploymentClient now reads component names from either the v1beta1 spec.components list or v1alpha1 spec.services, and derives the API version from the manifest (#14591).

General

• Router Worker Namespace Suffix: Fixed standalone Router worker-endpoint resolution using the wrong namespace by resolving targets through get_worker_namespace(), including DYN_NAMESPACE_WORKER_SUFFIX when Dynamo Operator injects it. Dynamo Router still serves its own endpoints from the base DYN_NAMESPACE, fixing mismatches in Kubernetes deployments (#12160).
• KvRouter Initialization GIL Release: Fixed the Python binding for KvRouter.__new__ to release the Global Interpreter Lock (GIL) while waiting for min_initial_workers to register. Callers can now enforce their own timeouts instead of hanging indefinitely when a worker dies before registering (#12762).

Kubernetes Deployment

Inference Gateway (EPP)

• Explicit EPP Replica Sync Port: Added DYN_EPP_REPLICA_SYNC_PORT, defaulting to 9092, so Endpoint Picker replicas bind and dial a fixed peer-to-peer sync port instead of resolving it from Service and EndpointSlice ports. Incomplete EndpointSlices no longer block EPP startup (#13811).

Operator & CRDs

• vLLM Multinode DRA GPU Counts: Fixed per-node GPU count resolution for components using ResourceClaim/ResourceClaimTemplate objects with no scalar GPU resource, which broke multinode vLLM launch flags and TensorRT-LLM mpirun world size under pure Dynamic Resource Allocation (#12166).
• Grove Worker Namespace Suffixing: Fixed concurrent Grove worker generations publishing under the same runtime namespace during a rollout, which made discovery reject the new MDC checksum as incompatible. Rendered the worker hash into Grove templates so each generation gets its own namespace (#12769).
• Immutable Component Type Fields: Made the v1alpha1 componentType and v1beta1 type fields immutable once set using CRD CEL transition rules, applied to standalone DynamoComponentDeployments and DynamoGraphDeployment components. Updates that alter an existing component type are now rejected, while legacy objects without a type can still initialize it once (#13763).
• Worker Hash Projection Gating: Fixed Dynamo Operator completing rolling updates prematurely and writing unreliable annotations, by gating worker hash writes on informer cache confirmation. Required old DynamoComponentDeployments to drain to zero pods before deletion, and scoped DCD discovery accordingly (#14249).
• KAI Staleness Duration Validation: Fixed the platform Helm chart accepting an unset default-staleness-grace-period unit that passed validation but failed at KAI Scheduler startup. Required the Go duration value -1s when Grove and KAI Scheduler are installed together (#14722).

DGDR & Profiling Jobs

• DGDR Model Cache Paths: Fixed DynamoGraphDeploymentRequest profiling double-prefixing a pvcModelPath already under pvcMountPath, producing duplicated paths like /opt/models/opt/models/... that pointed workloads at nonexistent checkpoints. Relative and legacy leading-slash paths continue to resolve as before (#12493).

Dynamo Snapshot

• Snapshot Restore Socket Deconfliction: Added per-restore remapping of clone-conflicting sockets so one CRIU checkpoint restores repeatedly into sibling engine containers sharing a Pod network namespace, rebinding the CUDA UVM listener and TCP pair. Preserved the restore-complete sentinel for the kubelet probe (#12887, #12704).
• vLLM Communicator Checkpoint Hooks: Wrapped the vLLM snapshot sleep/wake sequence with checkpoint_prepare()/checkpoint_restore() hooks so FlashInfer and NCCL communicators tear down before a CRIU dump and rebuild after restore. Hook failures now propagate so the process dies instead of continuing with half-rebuilt communicators (#12226).
• NCCL Registered Windows Disabled: Forced TLLM_NCCL_SYMMETRIC_ZERO_COPY=0 before TensorRT-LLM Snapshot engine initialization and logged a warning when an existing setting was overridden, removing a known checkpoint-unsafe resource from snapshot capture (#12570).

Health & Rollouts

• Component-Scoped Topology Spread Validation: Relaxed DynamoGraphDeployment webhook validation to accept a deployment topology without packDomain when at least one component defines its own topology constraint, while still rejecting configurations that set neither (#12448).

Multimodal & Diffusion

• EPD Encode Worker Cache Bounds: Fixed the vLLM encode worker’s image embedding cache growing without limit, by routing it through the capacity-bounded LRU MultimodalEmbeddingCacheManager already used by prefill and decode workers, and removed a hardcoded cache_size=8 limit (#14452, #14444).
• Multimodal EPD Decode Handoff: Fixed the vLLM E/P/D decode handoff dropping mixed image and video media by reloading remaining video and audio when an image placeholder is present. Raised the multimodal NIXL KV lease from vLLM’s 30-second default to 100 seconds (#14522, #14502).
• NVDEC Bitstream Filter Support: Enabled the H.264 and HEVC MP4-to-Annex-B bitstream filters in the in-tree FFmpeg build while keeping software H.264/HEVC encoders, decoders, and parsers disabled. Restored PyNvVideoCodec NVDEC construction after the media-ffmpeg binding loads the in-tree FFmpeg (#13936).
• Kimi K3 Media Pads: Fixed Kimi K3 multimodal handling in the vLLM adapter by expanding each frontend media pad into the checkpoint’s image-placeholder token sequence at the worker. Pad and media mismatches and invalid K3 metadata are now rejected instead of silently misaligning (#12394).
• Multimodal Embedding Leak Fix: Fixed the SGLang worker handler to release received multimodal embedding allocations when image or video item construction fails, so validation errors and cancellations no longer leak receiver allocations (#13587).
• Dynamic Diffusion Rendezvous Port: Fixed SGLang diffusion workers to pass the allocated NCCL port through the diffusion config as the DiffGenerator master port for image and video workers. It falls back to the SGLang default when no port is provided, which avoids port collisions during concurrent deployments (#13568).
• Multimodal NIXL Buffer Cleanup: Fixed MmKwargsNixlSender.cleanup() hanging forever on cancelled, rejected, or never-read transfers, which kept buffers alive and grew frontend memory until OOMKilled. Cleanup now bounds the wait with a timeout (DYN_MM_NIXL_CLEANUP_TIMEOUT_S, default 60s) (#12759).

Engines

vLLM

• vLLM KV-Router gRPC Flag: Fixed the vLLM KV-router sidecar launch scripts to pass --grpc-endpoint instead of the removed --vllm-endpoint flag, so aggregated and disaggregated KV-router deployments start with the unified sidecar gRPC endpoint configuration (#13894).
• Codec-Free OpenCV for Image Inputs: Rebuilt the vLLM CUDA image’s OpenCV without software video backends so still-image preprocessing keeps cv2.resize while the image ships no software video codecs, and kept install_media_decoders vllm as the explicit step that swaps in the full wheel and PyAV (#14361).
• XPU Wheel and Omni Pin: Tagged xpu wheels manylinux_2_39 through auditwheel, pinned vLLM-Omni to v0.27.0rc1 on the xpu image to match its vLLM v0.27.1, and cleared an inherited metric-port variable in the XPU serve tests so the vllm-xpu build and tests pass (#14034).
• Decode Handoff Cancellation Preservation: Fixed vLLM sidecar cancellation dropping decode requests when client cancellation won the initial race. Kept the decode stream alive until the first token or terminal transfer signal, so the disaggregated NIXL handoff completes (#12736).
• Decode Worker Engine Shutdown: Fixed vLLM decode workers hanging when startup failed after the engine started, by tracking partially constructed resources and shutting down the handler and EngineCore on every worker exit instead of leaving EngineCore alive (#12829).
• vLLM Native Generate Capability: Added publication of the vllm_inference_v1_generate runtime capability flag from token-input vLLM prefill, decode, and aggregated workers, so requests to /inference/v1/generate reach a matching worker instead of failing with a model-not-found error. Text-input, embedding, and incompatible role or type combinations do not advertise the capability (#12881).
• vLLM Abort Monitor Cleanup: Fixed abort-monitor tasks being retained after normal request completion in vLLM handlers, which caused unbounded memory growth (6.05 GB RSS at 1.9 million requests) and longer GIL waits. Fixed this by tracking, cancelling, and draining child wait tasks (#13637).
• Accelerator-Free Omni Stage Router: Fixed the vLLM omni stage router crashing on hosts with no accelerator with RuntimeError: Failed to infer device type, by detecting the router role before parser construction so it registers only the options it reads (#14359).
• LoRA Flag Exclusivity Enforcement: Fixed vLLM worker argument parsing so --enable-lora combined with --realtime or --classify-worker is rejected instead of accepted, by assigning engine arguments before validation runs. Previously both invalid combinations passed through to worker startup (#14446).
• vLLM Runtime CRC32C Dependency: Added the google-crc32c package to the vLLM runtime container image, which the ModelExpress v0.5.0 vLLM plugin imports at every vLLM startup, and added a build-time guard that verifies the ModelExpress loader registers. This fixes startup failures on XPU base images that lack the package (#12664).
• vLLM-RS Binary on PATH: Fixed the vLLM sidecar failing to start with a command-not-found error by symlinking the vllm-rs binary shipped inside the installed vllm Python package onto PATH. This keeps it at the same revision as the Python package (#14362).

SGLang

• Mooncake Backend Preflight: Checked that the mooncake ProcessGroup extension imports and registers before any model I/O, so a worker started with --elastic-ep-backend mooncake in an image that cannot use it fails at argument parsing with the flag named instead of crashing after model load (#14461).
• Pre-Tokenized SGLang Input Handling: Added model-aware validation for pre-tokenized nvext.token_data requests, resolving the max accepted token ID from model or tokenizer vocabulary bounds and returning out-of-range IDs as a client error. Routed pre-tokenized embedding inputs through EmbeddingReqInput (#12798, #13594).
• Conditional Disaggregation Error Code: Changed the SGLang backend rejection of the x-bypass-remote-prefill annotation from a generic 500 internal server error to an HTTP 400 response. Clients now receive the explanatory message directing them to vLLM or TensorRT-LLM for conditional disaggregation (#12578).
• DCP-Aware KV Block Size: Fixed the router-facing SGLang KV block size using raw page_size instead of page_size * dcp_size for decode context parallelism, which made the event converter reject DCP blocks and left KV-aware routing blind (#12765).
• Temporary Disaggregated Config Cleanup: Fixed cleanup of the temporary SGLang YAML file generated for a selected --disagg-config-key section so it is removed even when configuration merging, topology validation, or argument parsing fails. This prevents dynamo_config_*.yaml files from accumulating in the system temporary directory after failed worker launches (#12913).
• Incremental gRPC Stream Contract: Switched the SGLang gRPC sidecar to SGLang’s incremental streaming contract, consuming native token IDs and logprob metadata as per-chunk deltas instead of cumulative streams, and enabled --incremental-streaming-output across launch scripts and manifests (#13739).
• SGLang Min Tokens Mapping: Fixed the SGLang backend to map min_tokens to SGLang’s min_new_tokens in both the decode handler and the multimodal worker handler. End-of-sequence tokens are now suppressed until the minimum token floor is met, consistent with the TensorRT-LLM, TokenSpeed, and vLLM backends (#13865).
• SGLang Resolved Args Handling: Fixed SGLang workers reading raw ServerArgs after engine initialization, which left values like --page-size unset and crashed the KV event publisher with a TypeError on a None block size. Dynamo now reads the resolved configuration view instead (#13905).
• SGLang Config Parser Compatibility: Fixed the SGLang integration importing ConfigArgumentMerger from a path that broke on current nightlies, by preferring the relocated sglang.srt.utils module through a compatibility shim. Dynamo now works with both pinned SGLang releases and newer layouts (#14054).
• Per-Rank NIXL Telemetry Ports: Fixed SGLang co-located ranks crashing on a shared NIXL exporter port by treating NIXL_TELEMETRY_PROMETHEUS_PORT as the base of a per-rank range with bounds checks against overflow. Disaggregated SGLang deployments now reach Ready with KV-transfer counters intact (#14284).

TensorRT-LLM

• Zero Prompt Logprobs Preservation: Fixed the TensorRT-LLM request translation path so an explicit prompt_logprobs=0 is forwarded to the engine instead of being treated as omitted. Callers requesting log probabilities for selected prompt tokens only now receive them (#12689).
• Prefill Stop Condition Handling: Fixed the TensorRT-LLM handler overriding only max_tokens during prefill instead of applying full stop conditions, so a request with ignore_eos=true whose first token was EOS terminated immediately with HTTP 200 and no content. Stop conditions now pass through unchanged (#13442).
• Cancellation Listener Cleanup Fix: Fixed a leak in the TensorRT-LLM backend where per-request client cancellation and service shutdown listeners were not stopped after a request completed normally. The leak accumulated tasks and memory under sustained traffic and ended in worker out-of-memory failures (#13901).
• Context-Length Precedence Fix: Fixed the TensorRT-LLM sidecar discarding a configured --context-length (or TRTLLM_CONTEXT_LENGTH) whenever GetModelInfo reported a positive value, so the operator-supplied value now wins with a warning naming both numbers when they disagree (#14450).
• Environment-Independent Triton JIT: Fixed a cuda.h: No such file or directory failure in Triton’s cuda_utils on SSH-launched multinode TensorRT-LLM worker ranks. Restored Triton’s default lookup layout with symlinks to the CUDA toolkit’s cuda.h, ptxas, cuobjdump, and nvdisasm, so JIT compilation no longer depends on image environment variables (#14865).

Reinforcement Learning

• Worker Discovery Visibility Fixes: Fixed materialization_fingerprint hashing fields the frontend overrides, which split a WorkerSet into cohorts and hid the model from /v1/models during rolling upgrades, and reworked RL worker discovery to resolve a NamespaceFilter on Kubernetes (#14451, #14455).

Fault Tolerance & Observability

• Worker Shutdown Request Migration: Fixed requests interrupted by a worker shutdown terminating as clean cancellations instead of migrating, by briefly holding the terminal cancellation frame so the shutdown failure reaches the migration layer. Client-started cancellations and ordinary response streams keep their existing behavior (#13952).
• Worker Sidecar Metrics Scraping: Added an http port endpoint to the dynamo-worker PodMonitor so dynamo_frontend_* metrics from the frontend sidecar on Gateway API and EPP worker pods reach Prometheus, restoring the cached-token Grafana panel (#11981).
• HTTP Graceful Shutdown Guard: Fixed HttpService::run() to register and hold a graceful-shutdown guard for the lifetime of the service. The runtime no longer cancels discovery watches while in-flight requests and long-running streams are still draining after SIGTERM (#12031).
• Raced Final Marker Errors: Fixed false-positive worker errors when the trailing complete_final marker lost a race with frontend teardown on successful streaming requests, which incremented dynamo_component_errors_total{error_type="publish_final"}. Those sends now log at DEBUG, while genuine dropped connections detected through kill() are still counted (#12374).
• Prometheus Name Generator Sync: Fixed the Prometheus name generator and its Python and Rust sync so metric names dropped by the codegen gap (transport, frontend_service, work_handler, and tokenizer cache names) are restored. Made the generator fail on a name collision instead of silently dropping a constant (#12745, #13283).
• Bounded TCP Accept Backoff: Added exponential backoff and rate-limited logging for TCP response server accept failures from file-descriptor or kernel memory exhaustion. This replaced a busy retry loop that spun at microsecond cadence and repeated the same warning every iteration (#13146).
• Administrative Route Lifecycle Hardening: Gated Backend SDK engine control, engine update, and model-taint routes on the worker serving lifecycle and serialized administrative mutations with discovery transitions, so requests no longer race worker startup, discovery registration, or shutdown (#13259).
• Typed Backend Refusal Errors: Fixed unservable requests, like multimodal input sent to a backend without multimodal support, returning a generic HTTP 500 instead of a useful error. Worker refusals now carry a typed error, and SGLang and TensorRT-LLM guards raise InvalidArgument (#14467).

Performance Modeling & Replay

• Public Mocker Worker CLI: Restored the python3 -m dynamo.mocker live worker CLI and migrated in-repo launchers, manifests, and documentation to it, while retaining python3 -m dynamo.mocker._worker as a compatibility shim for existing 1.5 configurations (#14230).

General

• Shared-Source Served Aliases: Allowed non-LoRA workers with different served names to register on the same endpoint when they advertise the same source path, with each served name selecting its own worker group, while base and LoRA name collisions stay rejected (#14857).
• CUDA and uv Dependency Pinning: Pinned cupy to the cuda13x distribution in the sglang extra and enforced the CUDA major version, removing the last CUDA 12 reference, and declared mutually exclusive backend extras through tool.uv.conflicts (#13650, #12933).
• Discovery Instance State Consistency: Reworked Kubernetes discovery to emit Added as an upsert for same-ID endpoint updates, propagating metadata changes instead of detecting only added or removed IDs. Restored first-wins model deployment card acceptance so an incompatible card no longer withdraws the set (#12574, #14632).
• Configured Dynamo Runtime Propagation: Handed the configured Dynamo runtime to the PyO3 async bridge so DYN_RUNTIME_NUM_WORKER_THREADS and DYN_RUNTIME_MAX_BLOCKING_THREADS apply to the frontend-serving runtime instead of a second Tokio runtime, and pinned mocker startup to the owning executor (#13849, #13352).
• Bundled Runtime Dependency Refresh: Refreshed third-party packages the SGLang and vLLM runtime images inherit from their base images, raising the bundled mooncake transfer engine and flooring cryptography for vLLM to pick up an updated etcd stack (#14082).

New Release Artifacts

• Unified Sidecar Image (Experimental): Shipped one CPU-only, multi-arch dynamo-sidecar image carrying the SGLang, TensorRT-LLM, and vLLM sidecar executables, replacing the per-engine images. Select the engine with the container command (dynamo-sglang-sidecar, dynamo-trtllm-sidecar, dynamo-vllm-sidecar); the engine stays in its own GPU container (#13781, #13917).
• Rust EPP Frontend Image: Shipped the Rust Endpoint Picker binary inside the Dynamo Frontend image at /epp, replacing the Go EPP. A standalone dynamo-epp image is planned for a later release (#11355).
• Nightly Helm Charts: Published nightly Dynamo Helm charts through both ChartMuseum and OCI so nightly deployments no longer need a local chart build (#12962).
• Standalone Snapshot Operator: Moved Dynamo Snapshot to its own repository and chart, shipped as Snapshot v0.1.0 at oci://ghcr.io/ai-dynamo/snapshot/snapshot:0.1.0. The PodSnapshot and PodSnapshotContent CRDs and their reconciler now come from that operator. The Dynamo platform chart can pull it in as a subchart with global.snapshot.install=true for evaluation; production deployments should install it separately (#13177).
• AISimulate v0.12.0: Consumed AISimulate as a standalone release with one unified CLI, replacing the separate aiconfigurator wheels (#13478).

Documentation

• Release and Version Page Updates: Promoted the docs current-release surfaces through v1.4.0, v1.4.1, and v1.4.2, and refreshed the README community events table, Slack badge, and Dynamo Enterprise support section. Corrected nightly release artifact listings and hardened docs CI with a lint gate, verified-commit checks, test ownership, and a preview-build flake fix (#13269, #12752, #13678, #13999, #13310, #13344, #12469, #12498, #13720, #12737, #13843, #13854, #12528, #12763, #13882, #12388).
• Generated API Reference Docs: Added deterministic, CI-gated API reference generation for Python, Rust crates, and Kubernetes CRDs as native Fern MDX. Moved generation to publish time so source PRs no longer carry machine-generated content, and fixed stale CRD reference links and a reference-checker crash (#12110, #13556, #13035, #12985, #13320, #12815, #12342).
• Support Matrix and Deploy Guides: Replaced the CUDA driver picker with a generated support matrix reporting containers-versus-wheels-only scope, and defined the N-2 worker/frontend compatibility window. Aligned the Kubernetes build selector with the CLI guide, and added Vanilla vLLM GAIE, MatrixHub, and deploy contributing guides (#12215, #12529, #12479, #13668, #12800, #12894, #13886, #10957, #11197, #11700, #12729).
• Router and KV Routing Docs: Refreshed Dynamo Router cost, scheduling, event-recovery, topology-aware KV transfer, and standalone-mode documentation to match the current implementation. Added a runnable custom worker-selection policy walkthrough, corrected the Selection API environment variable, and reorganized the Router docs into six task-oriented sections (#12635, #12634, #12431, #12992, #13039, #13915, #13015).
• Docs Navigation and Fern Styling: Restructured guides around reader intent with dedicated Kubernetes, Local, and backend areas, and fixed Fern home page and navigation styling defects. Repointed the Fern CLI and dead SGLang and Helm chart links, and added tab-aligned authoring templates (#12993, #12373, #12410, #12436, #13942, #13973, #12539, #12615, #12861, #13115, #12817, #12823, #13138, #12668, #12426).
• RL and Agent Skill Docs: Added a reinforcement learning integration guide covering rollout inference and the SGLang-compatible /generate API. Restructured the guides into a docs-only set spanning trainers, KV-aware routing, and simulation, and corrected the shipped agent optimization skill pack (#11381, #13127, #13912, #13608, #13639, #13067).
• Operator, CRD, and EPP Docs: Clarified scalingAdapter and worker-hash rollout semantics per API version, and refreshed CRD YAML and image tags to 1.4.0. Documented the webhook failure-policy default, and announced the Go Endpoint Picker deprecation with Rust EPP migration guidance (#12986, #13270, #12744, #12563, #13243, #13244, #13401).
• Backend Flag and Config Corrections: Removed retired vLLM prefill and decode worker flags in favor of --disaggregation-mode, and replaced stale benchmark-granularity flags with the six sampling flags the parser exposes. Fixed the SGLang MAX_TOTAL_TOKENS description, and clarified SGLang LoRA support status (#12568, #12581, #13036, #13271, #14057, #13822).
• Recipe and Multimodal Serving Docs: Added catalog-owned image provenance metadata for six recipes, published Qwen3.5-122B-A10B and GLM-5.3 recipe pages, and added a multimodal serving guide covering parallel image decoding and custom vision encoders across backends (#13232, #13469, #12931, #13995, #12417, #13651, #13673).
• Planner, Profiler, and Observability Docs: Synchronized Planner, Profiler, DGDR, and SPICA documentation with the current implementation, and documented profiler component names and the unified AISimulate CLI. Corrected the request-trace S3 sink behavior, and cleaned up 151 rustdoc warnings and the external and research publications pages (#12408, #13446, #14148, #14623, #13223, #12945).
• verl Guide Version Bounds: Bounded the verl reinforcement-learning guide to vLLM v0.26.0, added flash-attn as an explicit prerequisite, and passed the three _per_gpu micro-batch overrides the pinned verl requires, so readers no longer hit import errors on the v1.5.0 vLLM pin (#14571).

Dynamo Enterprise

A curated set of v1.5.0 release artifacts is also published under the -enterprise suffix in the Dynamo Enterprise collection on NGC. These artifacts are eligible for NVIDIA Enterprise Support under an active NVIDIA AI Enterprise subscription and carry no functional or binary differences from their open-source counterparts. See Dynamo Enterprise for scope, supported artifacts, and coverage terms.

Looking Ahead

Planner Runtime Reconfiguration

Planner reconfigures GPU budgets at runtime through a /v1/min-endpoints API and caps how many replicas change in one throughput-scaling step, so autoscaling can be steered without a redeploy.

Kubernetes Roles & Standalone Endpoint Picker

Component roles gain a gated podTemplate API and explicit per-role replica counts. The Rust Endpoint Picker, now the default inside the Frontend image, continues toward a standalone image, adding an SGLang tokenizer provider and routing runtime requests under the requested policy class.

Frontend Transport Security

The transport encryption in this release extends to the Frontend HTTP server with mutual TLS.