> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# v1.3.0

## Docs

- [Quickstart](https://docs.nvidia.com/dynamo/v1.3.0/getting-started/quickstart.md): Get a Dynamo OpenAI-compatible endpoint running in a container in about 5 minutes.
- [Introduction to Dynamo](https://docs.nvidia.com/dynamo/v1.3.0/getting-started/introduction.md): Overview of Dynamo's design principles, performance techniques, and production features for distributed LLM inference.
- [Local Installation](https://docs.nvidia.com/dynamo/v1.3.0/getting-started/local-installation.md): Install and run Dynamo on a local machine or VM with containers or PyPI
- [Building from Source](https://docs.nvidia.com/dynamo/v1.3.0/getting-started/building-from-source.md): Build Dynamo from source for development and contributions
- [Kubernetes Deployment](https://docs.nvidia.com/dynamo/v1.3.0/getting-started/kubernetes-deployment.md): Use Dynamo's Kubernetes-native path when you are ready to deploy on a GPU cluster.
- [Contribution Guide](https://docs.nvidia.com/dynamo/v1.3.0/getting-started/contribution-guide.md): How to contribute to Dynamo
- [Support Matrix](https://docs.nvidia.com/dynamo/v1.3.0/resources/support-matrix.md): Hardware, software, and build compatibility for Dynamo
- [Feature Matrix](https://docs.nvidia.com/dynamo/v1.3.0/resources/feature-matrix.md): Compatibility matrix for Dynamo features across the SGLang, TensorRT-LLM, and vLLM backends.
- [Release Artifacts](https://docs.nvidia.com/dynamo/v1.3.0/resources/release-artifacts.md): Container images, Python wheels, Helm charts, Rust crates, and release history
- [Model Early Access Builds](https://docs.nvidia.com/dynamo/v1.3.0/resources/model-early-access-builds.md): Per-model early access container builds shipped ahead of stable releases
- [Examples](https://docs.nvidia.com/dynamo/v1.3.0/resources/examples.md): Reference deployments for the SGLang, TensorRT-LLM, and vLLM backends, plus a Hello World walkthrough.
- [Glossary](https://docs.nvidia.com/dynamo/v1.3.0/resources/glossary.md): Definitions of core Dynamo terms covering disaggregated serving, KV cache, components, and the distributed runtime.
- [Feature Benchmarks](https://docs.nvidia.com/dynamo/v1.3.0/benchmarks.md): Start with the feature or performance claim you want to inspect.
- [Dynamo Blog](https://docs.nvidia.com/dynamo/v1.3.0/digest.md): Engineering deep dives, architecture notes, and project updates from the NVIDIA Dynamo team.
- [NVIDIA Dynamo Snapshot: Fast Startup for Inference Workloads on Kubernetes](https://docs.nvidia.com/dynamo/v1.3.0/digest/dynamo-snapshot-fast-startup.md): NVIDIA Dynamo Snapshot combines CUDA and host checkpointing to restore warm inference workers quickly on Kubernetes.
- [DynoSim: Simulating the Pareto Frontier](https://docs.nvidia.com/dynamo/v1.3.0/digest/dynosim-pareto-frontier.md): DynoSim is a workload-driven discrete-event simulation of the NVIDIA Dynamo serving stack for mapping Pareto frontiers before real-cluster validation.
- [Dynamo Day 0 support for TokenSpeed](https://docs.nvidia.com/dynamo/v1.3.0/digest/tokenspeed-day-0.md): Dynamo adds day-0 TokenSpeed support with the Dynamo frontend for Kimi K2.5.
- [Streaming Tokens and Tools: Multi-Turn Agentic Harness Support in Dynamo](https://docs.nvidia.com/dynamo/v1.3.0/digest/agentic-harnesses.md): Streaming Tokens and Tools: Multi-Turn Agentic Harness Support in Dynamo
- [Full-Stack Optimizations for Agentic Inference with Dynamo](https://docs.nvidia.com/dynamo/v1.3.0/digest/agentic-inference.md): How Dynamo optimizes for agentic workloads at three layers: frontend API, router, and KV cache management.
- [Flash Indexer: A Story of Inter-Galactic KV Routing](https://docs.nvidia.com/dynamo/v1.3.0/digest/flash-indexer.md): Dynamo's Flash Indexer tracks every cached KV block across all inference workers at 170M ops/s. Six iterations of data structure design got it there.
- [Kubernetes Quickstart](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/start-here/kubernetes-quickstart.md): Quickstart for deploying a model on Kubernetes with Helm and the Dynamo operator.
- [Installation Guide](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/start-here/installation-guide.md): Installs accelerator resource components and the Dynamo Platform with optional Grove, RDMA, and Prometheus add-ons.
- [Dynamo Operator](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/start-here/dynamo-operator.md): Reference for the Dynamo Kubernetes operator covering its controllers, deployment modes, and reconciliation workflow.
- [Minikube Setup](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/start-here/minikube-setup.md): Sets up a local Minikube cluster with GPU support and Istio for running the Dynamo Kubernetes Platform.
- [Deployment Overview](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/deploy-models/model-deployment-guide.md): Understand DynamoGraphDeployments, DynamoComponentDeployments, DGDR, and recipes
- [Managing Models with DynamoModel](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/deploy-models/managing-models-with-dynamo-model.md): Deploys LoRA adapters and tracks model endpoints with the DynamoModel custom resource alongside DGDs.
- [DGDR Reference](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/deploy-models/dgdr-reference.md): Field reference for DynamoGraphDeploymentRequest, the deploy-by-intent generator that profiles and produces a DGD.
- [DGDR Examples](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/deploy-models/dgdr-examples.md): Practical DynamoGraphDeploymentRequest examples covering AIC estimates, online profiling, and SLA-driven generation.
- [Gateway API Inference Extension (GAIE)](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/request-routing/gateway-api-inference-extension/overview.md): Expose DynamoGraphDeployments through Kubernetes Gateway API and Dynamo EPP routing.
- [GAIE Quickstart](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/request-routing/gateway-api-inference-extension/quickstart.md): Deploy a DynamoGraphDeployment behind Gateway API Inference Extension.
- [GAIE Reference](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/request-routing/gateway-api-inference-extension/reference.md): Runtime contracts, routing controls, and mesh integration for Dynamo with GAIE.
- [Model Caching](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/model-loading/model-caching.md): Download models once and share across all pods in a Kubernetes cluster
- [ModelExpress](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/model-loading/model-express.md): Speed up model weight distribution across Kubernetes workers
- [Autoscaling](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/operate/autoscaling.md): Scales DGD services with the DynamoGraphDeploymentScalingAdapter using KEDA, Kubernetes HPA, or the Dynamo Planner.
- [Rolling Updates](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/operate/rolling-update.md): Updates DGD worker images, resources, and arguments with managed rolling updates across Deployment, Grove, and LWS backends.
- [Disaggregated Inference Communication Guide](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/operate/disagg-communication.md): Best practices for prefill/decode worker communication on Kubernetes
- [Topology-Aware KV Transfer](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/operate/topology-aware-kv-transfer.md): Keep disaggregated prefill and decode KV-cache transfers within a selected topology domain
- [Metrics](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/operate/observability/metrics.md): Collects and visualizes Dynamo component metrics on Kubernetes with kube-prometheus-stack PodMonitors and Grafana.
- [Logging](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/operate/observability/logging.md): Aggregates Dynamo logs on Kubernetes with Grafana Loki and Alloy for cluster-wide log querying.
- [Operator Metrics](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/operate/observability/operator-metrics.md): Monitors Dynamo operator health with Prometheus metrics for controller reconciliation, webhooks, and managed resources.
- [Multinode Deployments](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/scale/multinode-deployments.md): Scales Dynamo inference across multiple GPU nodes with Grove and KAI-Scheduler for large-model tensor parallelism.
- [Grove](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/scale/grove.md): Kubernetes API for orchestrating disaggregated inference with gang scheduling, multi-level autoscaling, and topology-aware placement.
- [Topology Aware Scheduling](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/scale/topology-aware-scheduling.md): Packs prefill, decode, and routing pods within the same rack or zone to reduce inter-node latency.
- [Service Discovery](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/advanced-platform/service-discovery.md): Lets Dynamo frontends, workers, and planners locate each other on Kubernetes via native CRDs and EndpointSlices, or via etcd.
- [Webhooks](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/advanced-platform/webhooks.md): Dynamo operator admission webhooks that validate and default Dynamo custom resources before persistence.
- [Snapshotting GPU Workers](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/advanced-platform/snapshot.md): Checkpoints initialized GPU workers with CRIU and cuda-checkpoint so later pods warm-start in seconds instead of minutes.
- [Shadow Engine Failover](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/advanced-platform/shadow-engine-failover.md): Keeps a standby engine attached to GPU-resident model weights so process failures recover without reloading the model.
- [Developing the Operator with Tilt](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/advanced-platform/developing-with-tilt.md): Fast, live-reload development loop for the Dynamo Kubernetes operator
- [Amazon Elastic Kubernetes Service (EKS)](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/cloud-provider-guides/aws/eks-setup.md): Deploys the Dynamo platform on Amazon EKS, including cluster creation, GPU node groups, and add-ons.
- [EFA (RDMA over AWS Fabric) on EKS](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/cloud-provider-guides/aws/efa-rdma-over-aws-fabric.md): Enables GPU-Direct RDMA over AWS Elastic Fabric Adapter so disaggregated workers transfer KV cache across EKS nodes without TCP.
- [Amazon EFS Setup for EKS](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/cloud-provider-guides/aws/efs.md): Creates an Amazon EFS file system so Dynamo workers on EKS share model weights and compilation cache across nodes.
- [Amazon Elastic Container Service (ECS)](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/cloud-provider-guides/aws/ecs.md): Runs Dynamo on Amazon ECS with EC2 GPU clusters for vLLM workers and Fargate tasks for etcd and NATS.
- [Azure Kubernetes Service (AKS)](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/cloud-provider-guides/azure/aks-setup.md)
- [RDMA / InfiniBand on AKS](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/cloud-provider-guides/azure/rdma-infini-band.md)
- [Storage for Model Caching on AKS](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/cloud-provider-guides/azure/aks-storage.md)
- [Azure Lustre CSI Driver for AKS](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/cloud-provider-guides/azure/azure-lustre-csi-driver.md)
- [AKS Spot VMs](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/cloud-provider-guides/azure/spot-v-ms.md)
- [Google Kubernetes Engine (GKE)](https://docs.nvidia.com/dynamo/v1.3.0/kubernetes-deployment/cloud-provider-guides/gcp/gke-setup.md)
- [Feature Guides](https://docs.nvidia.com/dynamo/v1.3.0/user-guides.md): Start with Dynamo's core serving optimizations, then branch into operations and model capabilities.
- [Router Guide](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/kv-cache-aware-routing.md): Deployment modes, quick start, and page map for Dynamo routing docs
- [Disaggregated Serving](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/disaggregated-serving.md): Split prefill and decode into independently scalable worker pools
- [Sizing with AIConfigurator](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/disaggregated-serving/sizing-with-ai-configurator.md): Compare aggregated and disaggregated layouts before deployment
- [KVBM Guide](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/kv-cache-offloading.md): Enable KV offloading using KV Block Manager (KVBM) for Dynamo deployments
- [Dynamo Benchmarking](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/benchmarking.md): Benchmark and compare performance across Dynamo deployment configurations
- [Tool Calling & Reasoning Parsing](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/parsing.md): Parse tool calls and reasoning out of model output into OpenAI-compatible tool_calls and reasoning_content
- [Tool Call Parsing (Dynamo)](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/parsing/tool-call-parsing-dynamo.md): Connect Dynamo to external tools and services using Dynamo's built-in tool call parsers
- [Reasoning Parsing (Dynamo)](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/parsing/reasoning-parsing-dynamo.md): Configure Dynamo's built-in reasoning parsers for models that emit thinking content
- [Parser Engine Fallback](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/parsing/parser-engine-fallback.md): Use upstream vLLM or SGLang tool-call and reasoning parsers when Dynamo does not ship one
- [Parser Configuration](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/parsing/parser-configuration.md): How --dyn-chat-processor, --dyn-tool-call-parser, and --dyn-reasoning-parser fit together
- [Tool Calling Probe Snapshot for Dynamo 1.2](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/parsing/tool-calling-probe-snapshot-for-dynamo-1-2.md): Static release snapshot of tool-calling probe results across supported model families
- [Troubleshooting Tool Calls](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/parsing/troubleshooting-tool-calls.md): Capture raw model output with logprobs so issues can be localized
- [Fault Tolerance](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/fault-tolerance.md): Handle failures gracefully with request migration, cancellation, and graceful shutdown
- [Request Migration](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/fault-tolerance/request-migration.md): Preserves token state and reissues in-flight requests on healthy workers when their original worker fails.
- [Request Cancellation](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/fault-tolerance/request-cancellation.md): Cancels in-flight requests across frontend and workers when clients disconnect, freeing GPU resources end to end.
- [Request Rejection](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/fault-tolerance/request-rejection.md): Configure independent worker-load thresholds that shed traffic with HTTP 529
- [Graceful Shutdown](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/fault-tolerance/graceful-shutdown.md): Coordinates SIGTERM and SIGINT handling so endpoints drain, in-flight requests finish, and pods restart cleanly.
- [Testing](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/fault-tolerance/testing.md): Pytest framework for validating Dynamo's cancellation, migration, etcd HA, and hardware fault-injection paths under induced failures.
- [Observability (Local)](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/observability-local.md): Monitor Dynamo deployments with metrics, logging, and tracing
- [Prometheus + Grafana Setup](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/observability-local/prometheus-grafana-setup.md): Single-machine setup for collecting Dynamo metrics in Prometheus and visualizing them in Grafana for local development and demos.
- [Metrics](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/observability-local/metrics.md): Reference for the Prometheus metrics Dynamo's runtime, frontend, and workers expose on the local system metrics endpoint.
- [Metrics Developer Guide](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/observability-local/metrics-developer-guide.md): Reference for creating counters, gauges, and histograms with Dynamo's metrics API, surfaced on the local /metrics endpoint.
- [Health Checks](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/observability-local/health-checks.md): HTTP health and liveness endpoints Dynamo components expose locally for wiring startup, liveness, and readiness probes.
- [Tracing](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/observability-local/tracing.md): OpenTelemetry distributed tracing for Dynamo running locally via Docker Compose, exporting spans to Tempo and visualizing them in Grafana.
- [Forward Pass Metrics Tracing](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/observability-local/forward-pass-metrics-tracing.md): Persist backend forward pass metrics to rotating gzip JSONL files
- [Logging](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/observability-local/logging.md): Structured text and JSONL logging for local Dynamo processes, with optional OTLP export to Loki via the OpenTelemetry Collector.
- [DynoSim](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/dynosim.md): Simulate Dynamo deployment choices before spending GPU time
- [Live Simulation with Mocker](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/dynosim/mocker.md): Mocker simulates production engine scheduling, KV cache, and timing so you can exercise Dynamo's frontend, router, and planner without GPUs.
- [DynoSim Runs](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/dynosim/runs.md): Run one trace or synthetic workload through a simulated Dynamo configuration
- [DynoSim Sweeps](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/dynosim/sweeps.md): Search simulated deployment candidates across topology, router, and SLA constraints
- [Planner DynoSim Benchmarking](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/dynosim/planner-benchmarking.md): Drive the planner in the simulation loop against a saved trace to evaluate SLA behavior and scaling decisions
- [Agents](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/agents.md): Agent-aware serving features in Dynamo
- [Session IDs](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/agents/session-i-ds.md): Identify agent sessions from supported coding agents and custom clients
- [Agent Tracing](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/agents/agent-tracing.md): Export Dynamo request traces, tool-call metadata, and Perfetto timelines
- [Agent Trace Replay](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/agents/agent-simulation.md): Capture an agent workload once, then replay it on real or simulated infrastructure
- [Agent Hints](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/agents/agent-hints.md): Per-request serving hints for agentic workloads
- [Agent Harnesses](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/agents/agent-harnesses.md): Point coding-agent CLIs at a Dynamo deployment
- [ThunderAgent Program Scheduler](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/agents/thunder-agent-program-scheduler.md): Program-level scheduling with tool-boundary pause/resume on top of KV-aware routing
- [LoRA Adapters](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/lo-ra-adapters.md): Serve fine-tuned LoRA adapters with dynamic loading and routing in Dynamo
- [Multimodal Model Serving](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/multimodal.md): Deploy multimodal models with image, video, and audio support in Dynamo
- [Embedding Cache](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/multimodal/embedding-cache.md): Cache vision encoder embeddings to skip re-encoding repeated multimodal content
- [Encoder Disaggregation](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/multimodal/encoder-disaggregation.md): Separate vision encoding into a dedicated worker for independent scaling
- [Custom Vision Encoders](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/multimodal/custom-vision-encoders.md): Run a bespoke vision tower in an aggregated Dynamo vLLM worker
- [Multimodal KV Routing](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/multimodal/multimodal-kv-routing.md): Route multimodal requests to workers with the best KV cache overlap
- [SGLang Multimodal](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/multimodal/sg-lang-multimodal.md)
- [TensorRT-LLM Multimodal](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/multimodal/tensor-rt-llm-multimodal.md)
- [vLLM Multimodal](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/multimodal/v-llm-multimodal.md)
- [Diffusion](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/diffusion.md): Deploy diffusion models for text-to-image, text-to-video, and more in Dynamo
- [FastVideo](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/diffusion/fastvideo.md): Deploys FastVideo text-to-video generation on Dynamo through a custom worker that serves the /v1/videos endpoint.
- [Fastokens Tokenizer](https://docs.nvidia.com/dynamo/v1.3.0/user-guides/fastokens-tokenizer.md): Reduce frontend tokenization latency for long-context BPE models
- [SGLang](https://docs.nvidia.com/dynamo/v1.3.0/backends/sg-lang.md): SGLang engines run in Dynamo's distributed runtime with disaggregated serving, KV-aware routing, and request cancellation.
- [Reference Guide](https://docs.nvidia.com/dynamo/v1.3.0/backends/sg-lang/reference-guide.md): Architecture, configuration, and operational details for the SGLang backend
- [Examples](https://docs.nvidia.com/dynamo/v1.3.0/backends/sg-lang/examples.md): Launch scripts cover aggregated, disaggregated, KV-routed, multimodal, and diffusion deployments on the SGLang backend.
- [Disaggregation](https://docs.nvidia.com/dynamo/v1.3.0/backends/sg-lang/disaggregation.md): SGLang runs prefill and decode as separate workers in Dynamo, coordinated by a bootstrap handshake with RDMA-based KV cache transfer.
- [Diffusion](https://docs.nvidia.com/dynamo/v1.3.0/backends/sg-lang/diffusion.md): SGLang supports LLM diffusion, image diffusion, and text-to-video generation through dedicated worker flags.
- [SGLang Chat Processor](https://docs.nvidia.com/dynamo/v1.3.0/backends/sg-lang/frontend-processor-fallback.md): SGLang-native preprocessing and postprocessing for chat completions
- [Logits Processing](https://docs.nvidia.com/dynamo/v1.3.0/backends/sg-lang/logits-processing.md): Custom logits processors plug into SGLang's per-step callback through Dynamo's backend-agnostic spec adapter.
- [Observability](https://docs.nvidia.com/dynamo/v1.3.0/backends/sg-lang/observability.md): Prometheus metrics, forward pass telemetry, distributed tracing, and Grafana dashboards for SGLang workers in Dynamo.
- [SGLang for Agentic Workloads](https://docs.nvidia.com/dynamo/v1.3.0/backends/sg-lang/agentic-workloads.md): Priority scheduling and KV cache tuning for agentic serving
- [TensorRT-LLM](https://docs.nvidia.com/dynamo/v1.3.0/backends/tensor-rt-llm.md): TensorRT-LLM engines run in Dynamo's distributed runtime with disaggregated serving, KV-aware routing, and multinode support.
- [Reference Guide](https://docs.nvidia.com/dynamo/v1.3.0/backends/tensor-rt-llm/reference-guide.md): Features, configuration, and operational details for the TensorRT-LLM backend
- [Examples](https://docs.nvidia.com/dynamo/v1.3.0/backends/tensor-rt-llm/examples.md): Launch scripts cover aggregated, disaggregated, KV-routed, and multinode deployment patterns for the TensorRT-LLM backend.
- [Prometheus](https://docs.nvidia.com/dynamo/v1.3.0/backends/tensor-rt-llm/observability.md): TensorRT-LLM exposes request, latency, and KV cache transfer metrics through Dynamo's Prometheus /metrics endpoint.
- [Video Diffusion Support (Experimental)](https://docs.nvidia.com/dynamo/v1.3.0/backends/tensor-rt-llm/diffusion-experimental.md): Experimental text-to-video and text-to-image diffusion in TensorRT-LLM using the visual_gen module and Diffusers pipelines.
- [Known Issues and Mitigations](https://docs.nvidia.com/dynamo/v1.3.0/backends/tensor-rt-llm/known-issues-and-mitigations.md): Known issues for TensorRT-LLM in Dynamo, including disaggregated KV cache exhaustion and driver mismatch errors, with mitigations.
- [vLLM](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm.md): vLLM engines run in Dynamo's distributed runtime with disaggregated serving, NIXL KV transfer, and KV-aware routing.
- [Reference Guide](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm/reference-guide.md): Configuration, arguments, and operational details for the vLLM backend
- [vLLM Chat Processor](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm/frontend-processor-fallback.md): vLLM-native preprocessing and postprocessing for chat completions
- [Logits Processing](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm/logits-processing.md): Custom logits processors plug into vLLM through Dynamo's engine-loaded adapter and per-request SamplingParams.extra_args.
- [Examples](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm/examples.md): Launch scripts cover aggregated, disaggregated, KV-routed, and expert-parallel deployment patterns for the vLLM backend.
- [KV Cache Offloading](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm/kv-cache-offloading.md): CPU and disk offloading integrations for vLLM in Dynamo
- [Prometheus](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm/observability.md): vLLM exposes request, latency, scheduler, and disaggregation metrics through Dynamo's Prometheus /metrics endpoint.
- [vLLM-Omni](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm/v-llm-omni.md): vLLM-Omni serves text-to-image, text-to-video, image-to-video, and text-to-speech models through OpenAI-compatible endpoints.
- [Custom Backend Overview](https://docs.nvidia.com/dynamo/v1.3.0/backends/custom-backend/custom-backend-overview.md): Choose the right path for bringing your own engine to Dynamo
- [Writing Unified Backends](https://docs.nvidia.com/dynamo/v1.3.0/backends/custom-backend/writing-unified-backends.md): Choose Python or Rust for Dynamo's shared backend contract
- [Writing Python Workers in Dynamo](https://docs.nvidia.com/dynamo/v1.3.0/backends/custom-backend/python-workers-lower-level.md): Create custom Python workers and engines for Dynamo
- [Runtime Containers](https://docs.nvidia.com/dynamo/v1.3.0/backends/custom-backend/runtime-containers.md): Build Dynamo runtime images for built-in or custom backends
- [Frontend](https://docs.nvidia.com/dynamo/v1.3.0/components/frontend.md): API gateway that exposes OpenAI HTTP and KServe gRPC endpoints and handles request preprocessing, routing, and response formatting.
- [Frontend Guide](https://docs.nvidia.com/dynamo/v1.3.0/components/frontend/frontend-guide.md): Configures the KServe gRPC frontend, including supported endpoints, gRPC tuning, and registering OpenAI or tensor-based backends.
- [Frontend Configuration Reference](https://docs.nvidia.com/dynamo/v1.3.0/components/frontend/configuration-reference.md): Complete reference for all frontend CLI arguments, environment variables, and HTTP endpoints
- [Python Route Extensions](https://docs.nvidia.com/dynamo/v1.3.0/components/frontend/python-route-extensions.md)
- [Tokenizer](https://docs.nvidia.com/dynamo/v1.3.0/components/frontend/tokenizer.md): Selects between the default HuggingFace and fastokens tokenizer backends for BPE models served through the Dynamo Frontend.
- [Router](https://docs.nvidia.com/dynamo/v1.3.0/components/router.md): KV cache-aware router that picks workers by combined prefill and decode cost to maximize throughput and minimize latency.
- [Routing Concepts](https://docs.nvidia.com/dynamo/v1.3.0/components/router/routing-concepts.md): Cost model, worker selection, and routing primitives for the Dynamo router
- [Multi-DC KV Routing and the DC Relay](https://docs.nvidia.com/dynamo/v1.3.0/components/router/multi-dc-kv-routing-and-the-dc-relay.md): Aggregate exact KV ownership in each DC and publish compact state to a global consumer
- [Configuration and Tuning](https://docs.nvidia.com/dynamo/v1.3.0/components/router/configuration-and-tuning.md): Router behavior, event transport, load tracking, and tuning guidance
- [Deficit Round Robin Queue Scheduling](https://docs.nvidia.com/dynamo/v1.3.0/components/router/deficit-round-robin-queue-scheduling.md): Weighted arbitration across router policy classes
- [Priority Scheduling](https://docs.nvidia.com/dynamo/v1.3.0/components/router/priority-scheduling.md): Request priority across the Dynamo router and backend engines
- [Disaggregated Serving](https://docs.nvidia.com/dynamo/v1.3.0/components/router/disaggregated-serving.md): Prefill and decode routing with the Dynamo router
- [Topology-Aware KV Transfer](https://docs.nvidia.com/dynamo/v1.3.0/components/router/topology-aware-kv-transfer.md): Runtime metadata and decode routing semantics for topology-aware prefill/decode handoff
- [Router Operations](https://docs.nvidia.com/dynamo/v1.3.0/components/router/router-operations.md): Replica topology, state management, recovery, and remote indexers
- [Router Examples](https://docs.nvidia.com/dynamo/v1.3.0/components/router/router-examples.md): Worked examples for the KvRouter Python API, Kubernetes deployments, and custom routing patterns.
- [Standalone KV Indexer](https://docs.nvidia.com/dynamo/v1.3.0/components/router/standalone-indexer.md): Run the KV cache indexer as an independent HTTP service for querying block state
- [Standalone Selection Service](https://docs.nvidia.com/dynamo/v1.3.0/components/router/standalone-selection-service.md): Select workers and account for reservations without forwarding inference requests
- [Standalone Slot Tracker](https://docs.nvidia.com/dynamo/v1.3.0/components/router/standalone-slot-tracker.md): Run active-request load accounting as an independent HTTP service
- [KV Event Replay — Dynamo vs vLLM](https://docs.nvidia.com/dynamo/v1.3.0/components/router/kv-event-replay-dynamo-vs-v-llm.md): How the two systems handle gap detection, replay, and recovery for KV cache events
- [Planner](https://docs.nvidia.com/dynamo/v1.3.0/components/planner.md): Autoscaler that adjusts prefill and decode replicas using engine performance models and traffic prediction to meet TTFT and ITL SLAs.
- [Planner Guide](https://docs.nvidia.com/dynamo/v1.3.0/components/planner/planner-guide.md): Configures Planner optimization targets, scaling modes, and PlannerConfig fields for production deployments.
- [Global Planner Deployment Guide](https://docs.nvidia.com/dynamo/v1.3.0/components/planner/global-planner-guide.md): Deploys GlobalPlanner as the centralized scaling layer for multi-DGD policy enforcement and single-endpoint multi-pool deployments.
- [Planner Examples](https://docs.nvidia.com/dynamo/v1.3.0/components/planner/planner-examples.md): Examples for custom load predictors and the VirtualConnector for non-Kubernetes scaling environments.
- [Profiler](https://docs.nvidia.com/dynamo/v1.3.0/components/profiler.md): Automated performance analysis that finds optimal prefill and decode parallelism and generates SLA-driven deployment configurations.
- [Profiler Guide](https://docs.nvidia.com/dynamo/v1.3.0/components/profiler/profiler-guide.md): Runs the AI Configurator-driven profiling pipeline that turns a model, SLA targets, and backend into a ready-to-deploy DGD.
- [Profiler Examples](https://docs.nvidia.com/dynamo/v1.3.0/components/profiler/profiler-examples.md): Complete DGDR profiling examples for dense and MoE models across rapid and thorough search strategies.
- [KVBM](https://docs.nvidia.com/dynamo/v1.3.0/components/kvbm.md): Unified memory layer and write-through KV cache spanning GPU, host, SSD, and remote storage via NIXL for TensorRT-LLM and vLLM.
- [Using HiCache](https://docs.nvidia.com/dynamo/v1.3.0/integrations/kv-cache-integrations/hi-cache.md): Hierarchical KV caching with tier-aware router integration
- [LMCache](https://docs.nvidia.com/dynamo/v1.3.0/integrations/kv-cache-integrations/lm-cache.md): Integrate LMCache with Dynamo's vLLM backend for prefill-once, reuse-everywhere KV cache across requests.
- [FlexKV](https://docs.nvidia.com/dynamo/v1.3.0/integrations/kv-cache-integrations/flex-kv.md): Integrate FlexKV with Dynamo's vLLM backend for multi-level KV cache offloading across CPU, SSD, and cloud storage.
- [KV Events for Custom Engines](https://docs.nvidia.com/dynamo/v1.3.0/integrations/kv-cache-integrations/kv-events-for-custom-engines.md)
- [LWS](https://docs.nvidia.com/dynamo/v1.3.0/integrations/kubernetes-integrations/lws.md): LeaderWorkerSet integration for multinode Dynamo deployments
- [Overall Architecture](https://docs.nvidia.com/dynamo/v1.3.0/design-docs/overall-architecture.md): Architecture and components of the Dynamo inference runtime
- [Architecture Flow](https://docs.nvidia.com/dynamo/v1.3.0/design-docs/architecture-flow.md)
- [Disaggregated Serving](https://docs.nvidia.com/dynamo/v1.3.0/design-docs/disaggregated-serving.md)
- [Distributed Runtime](https://docs.nvidia.com/dynamo/v1.3.0/design-docs/distributed-runtime.md)
- [Discovery Plane](https://docs.nvidia.com/dynamo/v1.3.0/design-docs/communication-planes/discovery-plane.md)
- [Request Plane](https://docs.nvidia.com/dynamo/v1.3.0/design-docs/communication-planes/request-plane.md)
- [Event Plane](https://docs.nvidia.com/dynamo/v1.3.0/design-docs/communication-planes/event-plane.md)
- [Router Design](https://docs.nvidia.com/dynamo/v1.3.0/design-docs/component-design/router-design.md)
- [KVBM Design](https://docs.nvidia.com/dynamo/v1.3.0/design-docs/component-design/kvbm-design.md)
- [Planner Design](https://docs.nvidia.com/dynamo/v1.3.0/design-docs/component-design/planner-design.md)
- [Documentation Style Guide](https://docs.nvidia.com/dynamo/v1.3.0/documentation/style-guide.md): How to write and structure Dynamo docs, examples, and recipes
- [How to Build and Publish Dynamo Docs](https://docs.nvidia.com/dynamo/v1.3.0/documentation/building-and-publishing.md): Branch model, sync workflow, and publishing pipeline for the Fern-powered Dynamo documentation site.
- [Recipes](https://docs.nvidia.com/dynamo/v1.3.0/recipes/browse.md): Pick a model, choose the hardware/runtime target, and deploy the matching Dynamo recipe.
- [GLM-5.2](https://docs.nvidia.com/dynamo/v1.3.0/recipes/glm-5-2.md): Serve GLM-5.2 with Dynamo and SGLang on B200 or H200 for long-context agentic workloads.
- [Inkling NVFP4](https://docs.nvidia.com/dynamo/v1.3.0/recipes/inkling.md): Day-0 SGLang recipe for Thinking Machines' Inkling on 8x B200.
- [Kimi-K2.6](https://docs.nvidia.com/dynamo/v1.3.0/recipes/kimi-k2-6.md): Serve moonshotai/Kimi-K2.6 with Dynamo and vLLM, tuned per GPU and workload.
- [Nemotron-3-Ultra](https://docs.nvidia.com/dynamo/v1.3.0/recipes/nemotron-3-ultra.md): Serve nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 with Dynamo and vLLM, tuned per GPU and workload.
- [Nemotron-3-Super](https://docs.nvidia.com/dynamo/v1.3.0/recipes/nemotron-3-super.md): Serve NVIDIA-Nemotron-3-Super-120B-A12B with Dynamo and vLLM, tuned per GPU and workload.
- [GLM-5 NVFP4](https://docs.nvidia.com/dynamo/v1.3.0/recipes/glm-5-nvfp4.md): Serve GLM-5 NVFP4 with Dynamo and SGLang, disaggregated across five GB200 nodes.
- [DeepSeek-V4-Pro](https://docs.nvidia.com/dynamo/v1.3.0/recipes/deepseek-v4-pro.md): Serve DeepSeek-V4-Pro with Dynamo and vLLM — aggregated or disaggregated, on B200 or H200, tuned for the agentic workload.
- [DeepSeek-V4-Flash](https://docs.nvidia.com/dynamo/v1.3.0/recipes/deepseek-v4-flash.md): Serve DeepSeek-V4-Flash with Dynamo and vLLM — aggregated or disaggregated, on B200 or H200, tuned for the agentic workload.
- [DeepSeek V3.2 NVFP4](https://docs.nvidia.com/dynamo/v1.3.0/recipes/deepseek-v3-2-nvfp4.md): Serve nvidia/DeepSeek-V3.2-NVFP4 with Dynamo and TensorRT-LLM, disaggregated with KV routing and WideEP on GB200 NVL72.
- [GPT-OSS-120B](https://docs.nvidia.com/dynamo/v1.3.0/recipes/gpt-oss-120b.md): Serve openai/gpt-oss-120b with Dynamo on vLLM (B200/H200) or TensorRT-LLM (GB200), aggregated or disaggregated.
- [Qwen3-32B](https://docs.nvidia.com/dynamo/v1.3.0/recipes/qwen3-32b.md): Serve Qwen3-32B with Dynamo and vLLM — disaggregated serving with KV-aware routing for multi-turn conversational traffic.
- [Qwen3-235B-A22B FP8](https://docs.nvidia.com/dynamo/v1.3.0/recipes/qwen3-235b-a22b-fp8.md): Serve Qwen/Qwen3-235B-A22B-FP8 with Dynamo and TensorRT-LLM on 16 Hopper or Blackwell GPUs.
- [Qwen3-32B FP8](https://docs.nvidia.com/dynamo/v1.3.0/recipes/qwen3-32b-fp8.md): Serve Qwen/Qwen3-32B-FP8 with Dynamo on TensorRT-LLM or vLLM, aggregated or disaggregated.
- [Llama-3.3-70B FP8](https://docs.nvidia.com/dynamo/v1.3.0/recipes/llama-3-3-70b.md): Serve Llama-3.3-70B FP8 with Dynamo and vLLM, sized from one node to two.