> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Knowledge Base

## Docs

- [Overall Architecture](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/overview.md): Architecture and components of the Dynamo inference runtime
- [Architecture Flow](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/concepts/system-architecture/architecture-flow.md)
- [Distributed Runtime](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/concepts/system-architecture/distributed-runtime.md)
- [Sidecar Backends](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/concepts/system-architecture/sidecar-backends-experimental.md): Run Dynamo beside a stock inference engine through its native gRPC API.
- [Disaggregated Serving](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/concepts/system-architecture/disaggregated-serving.md)
- [Discovery Plane](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/concepts/communication-planes/discovery-plane.md)
- [Request Plane](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/concepts/communication-planes/request-plane.md)
- [Event Plane](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/concepts/communication-planes/event-plane.md)
- [Observability Architecture](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/concepts/observability-architecture.md): Signal transport, request correlation, and active worker health checks
- [DynoSim Architecture](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/concepts/simulation/dyno-sim-architecture.md): How the replay harness composes simulated engines, routing, and Planner behavior
- [Request Migration Architecture](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/concepts/fault-tolerance/request-migration-architecture.md): Preserve token state and reissue in-flight requests on healthy workers after a worker failure.
- [Request Cancellation Architecture](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/concepts/fault-tolerance/request-cancellation-architecture.md): Cancel in-flight work across the Frontend and workers when a client disconnects.
- [Request Rejection Architecture](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/concepts/fault-tolerance/request-rejection-architecture.md): Worker-load event processing, busy-state aggregation, overload errors, and hard worker admission limits.
- [Graceful Shutdown Architecture](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/concepts/fault-tolerance/graceful-shutdown-architecture.md)
- [Fault Tolerance Testing](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/concepts/fault-tolerance/fault-tolerance-testing.md): Pytest infrastructure for cancellation, migration, etcd high availability, and hardware fault injection.
- [Frontend](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/frontend/overview.md): API gateway that exposes OpenAI HTTP and KServe gRPC endpoints and handles request preprocessing, routing, and response formatting.
- [Frontend Guide](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/frontend/frontend-guide.md): Configures the KServe gRPC frontend, including supported endpoints, gRPC tuning, and registering OpenAI or tensor-based backends.
- [Python Route Extensions](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/frontend/python-route-extensions.md): Add trusted HTTP routes to the Dynamo Frontend from an installed Python package or importable module.
- [Tokenizer](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/frontend/tokenizer.md): Selects between the default HuggingFace and fastokens tokenizer backends for BPE models served through the Dynamo Frontend.
- [Router](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/overview.md): KV cache-aware router that picks workers by combined prefill and decode cost to maximize throughput and minimize latency.
- [Router Guide](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/router-guide.md): Deployment modes, quick start, and page map for Dynamo routing docs
- [Routing Concepts](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/routing-concepts.md): Cost model, worker selection, and routing primitives for the Dynamo router
- [Router Filtering](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/router-filtering.md): Candidate eligibility, DP-rank filtering, and queue backpressure boundaries
- [Configuration and Tuning](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/configuration-and-tuning.md): Router behavior, event transport, load tracking, and tuning guidance
- [Deficit Round Robin Queue Scheduling](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/deficit-round-robin.md): Weighted arbitration across router policy classes
- [Disaggregated Serving](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/disaggregated-serving.md): Prefill and decode routing with the Dynamo router
- [Offloading Support Matrix](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/offloading-support-matrix.md): Per-framework support and setup for KV routing with tiered KV cache offloading
- [Topology-Aware KV Transfer](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/topology-aware-kv-transfer.md): Runtime metadata and decode routing semantics for topology-aware prefill/decode handoff
- [Multi-DC KV Routing and the DC Relay](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/multi-dc-kv-routing.md): Aggregate exact KV ownership in each DC and publish compact state to a global consumer
- [Router Operations](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/router-operations.md): Replica topology, state management, recovery, and remote indexers
- [Router Testing](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/router-testing.md): Test layers for router changes
- [Router Examples](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/router-examples.md): Worked examples for the KvRouter Python API, Kubernetes deployments, and custom routing patterns.
- [Global Router Configuration](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/global-router-configuration.md): Field reference for the CLI flags and JSON config file consumed by the Global Router service.
- [Standalone KV Indexer](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/standalone-indexer.md): Run the KV cache indexer as an independent HTTP service for querying block state
- [Standalone Slot Tracker](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/standalone-slot-tracker.md): Run active-request load accounting as an independent HTTP service
- [Standalone Selection Service](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/standalone-selection.md): Select workers and account for reservations without forwarding inference requests
- [KV Event Replay — Dynamo vs vLLM](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/kv-event-replay-comparison.md): How the two systems handle gap detection, replay, and recovery for KV cache events
- [Router Design](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/router/router-design.md)
- [Planner](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/planner/overview.md): Autoscaler that adjusts prefill and decode replicas using engine performance models and traffic prediction to meet TTFT and ITL SLAs.
- [Planner Guide](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/planner/planner-guide.md): Configures Planner optimization targets, scaling modes, and PlannerConfig fields for production deployments.
- [Global Planner Deployment Guide](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/planner/global-planner-guide.md): Deploys GlobalPlanner as the centralized scaling layer for multi-DGD policy enforcement and single-endpoint multi-pool deployments.
- [Planner Examples](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/planner/planner-examples.md): Examples for custom load predictors and the VirtualConnector for non-Kubernetes scaling environments.
- [Planner Design](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/planner/planner-design.md)
- [Profiler](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/profiler/overview.md): Automated performance analysis that finds optimal prefill and decode parallelism and generates SLA-driven deployment configurations.
- [Profiler Guide](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/profiler/profiler-guide.md): Runs the AI Configurator-driven profiling pipeline that turns a model, SLA targets, and backend into a ready-to-deploy DGD.
- [Profiler Examples](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/profiler/profiler-examples.md): Complete DGDR profiling examples for dense and MoE models across rapid and thorough search strategies.
- [AI Simulate (Experimental)](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/ai-simulate/overview.md): Experimental simulation and configuration-search tools for Dynamo deployments
- [Spica](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/ai-simulate/spica/overview.md): Experimental replay-backed configuration search for Dynamo deployments
- [Spica Search Flow](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/ai-simulate/spica/search-flow.md): How Spica turns a deployment-tuning problem into a replay-backed search
- [Spica Optimization Goals](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/ai-simulate/spica/optimization-goals.md): Objective metrics, SLA constraints, and Pareto-front scoring
- [Spica Traffic](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/ai-simulate/spica/traffic.md): Trace, request-rate, concurrency, and KV-load workload definitions
- [Spica Search Space](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/ai-simulate/spica/search-space.md): Searchable deployment, engine, router, Planner, and parallelism settings
- [Spica Unrolled Samples](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/ai-simulate/spica/unrolled-samples.md): How Spica turns a search trial into a concrete deployment configuration
- [GLM-5-FP8 Disaggregated Pareto Sweep](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/ai-simulate/spica/glm-5-fp-8-pareto-sweep.md): Experimental Spica search for a GLM-5-FP8 Pareto frontier on B200 GPUs
- [Planner Goodput-per-GPU Sweep](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/ai-simulate/spica/planner-goodput-per-gpu-sweep.md): Experimental Spica search over Planner policies for MiniMax-M2.5 on B200 GPUs
- [Router End-to-End Latency Sweep](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/ai-simulate/spica/router-end-to-end-latency-sweep.md): Experimental Spica search over KV router settings for MiniMax-M2.5 on B200 GPUs
- [KVBM](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/kvbm/overview.md): Unified memory layer and write-through KV cache spanning GPU, host, SSD, and remote storage via NIXL for TensorRT-LLM and vLLM.
- [KVBM Guide](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/kvbm/kvbm-guide.md): Enable KV offloading using KV Block Manager (KVBM) for Dynamo deployments
- [KVBM Design](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/kvbm/kvbm-design.md)
- [SGLang](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/sg-lang/overview.md): SGLang engines run in Dynamo's distributed runtime with disaggregated serving, KV-aware routing, and request cancellation.
- [SGLang Sidecar](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/sg-lang/sidecar.md): Run Dynamo beside a stock SGLang engine through native gRPC.
- [Reference Guide](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/sg-lang/reference-guide.md): Architecture, configuration, and operational details for the SGLang backend
- [SGLang for Agentic Workloads](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/sg-lang/agents-on-sg-lang.md): Priority scheduling and session control for multi-turn agentic serving
- [SGLang Chat Processor](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/sg-lang/chat-processor.md): SGLang-native preprocessing and postprocessing for chat completions
- [Disaggregation](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/sg-lang/disaggregation.md): SGLang runs prefill and decode as separate workers in Dynamo, coordinated by a bootstrap handshake with RDMA-based KV cache transfer.
- [Using HiCache](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/sg-lang/hi-cache.md): Hierarchical KV caching with tier-aware router integration
- [Observability](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/sg-lang/observability.md): Prometheus metrics, forward pass telemetry, distributed tracing, and Grafana dashboards for SGLang workers in Dynamo.
- [SGLang Multimodal](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/sg-lang/sglang-multimodal.md)
- [TensorRT-LLM](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/tensor-rt-llm/overview.md): TensorRT-LLM engines run in Dynamo's distributed runtime with disaggregated serving, KV-aware routing, and multinode support.
- [TensorRT-LLM Sidecar](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/tensor-rt-llm/sidecar.md): Run Dynamo beside a stock TensorRT-LLM engine through native gRPC.
- [Reference Guide](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/tensor-rt-llm/reference-guide.md): Features, configuration, and operational details for the TensorRT-LLM backend
- [TensorRT-LLM Observability](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/tensor-rt-llm/observability.md): TensorRT-LLM exposes request, latency, and KV cache transfer metrics through Dynamo's Prometheus /metrics endpoint.
- [Known Issues and Mitigations](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/tensor-rt-llm/known-issues.md): Known issues for TensorRT-LLM in Dynamo, including disaggregated KV cache exhaustion and driver mismatch errors, with mitigations.
- [TensorRT-LLM Multimodal](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/tensor-rt-llm/tensorrt-llm-multimodal.md)
- [vLLM](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/v-llm/overview.md)
- [vLLM Sidecar](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/v-llm/sidecar.md): Run Dynamo beside a stock vLLM engine through native gRPC.
- [Reference Guide](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/v-llm/reference-guide.md): Configuration, arguments, and operational details for the vLLM backend
- [vLLM Chat Processor](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/v-llm/chat-processor.md): vLLM-native preprocessing and postprocessing for chat completions
- [KV Cache Offloading](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/v-llm/kv-cache-offloading.md): CPU and disk offloading integrations for vLLM in Dynamo
- [Native KV Offloading](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/v-llm/native-kv-offloading.md): vLLM's OffloadingConnector with KV-aware routing in Dynamo
- [vLLM Observability](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/v-llm/observability.md): vLLM exposes request, latency, scheduler, and disaggregation metrics through Dynamo's Prometheus /metrics endpoint.
- [vLLM Multimodal](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/v-llm/vllm-multimodal.md)
- [vLLM-Omni Disaggregated Multi-Stage Serving](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/v-llm/v-llm-omni-disaggregated-serving.md): Architecture of multi-stage vLLM-Omni pipelines (e.g. AR + Diffusion), with each stage on its own GPU for independent scaling and isolation.
- [Mocker Engine Architecture](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/modular-components/backends/mocker/mocker-engine-architecture.md)
- [Model Caching](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/model-loading/model-caching.md): Download models once and share across all pods in a Kubernetes cluster
- [ModelExpress](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/model-loading/model-express.md): Speed up model weight distribution across Kubernetes workers
- [Dynamo Operator](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/kubernetes-operator/dynamo-operator.md): Reference for the Dynamo Kubernetes operator covering its controllers, deployment modes, and reconciliation workflow.
- [Service Discovery](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/kubernetes-operator/service-discovery.md): Lets Dynamo frontends, workers, and planners locate each other on Kubernetes via native CRDs and EndpointSlices, or via etcd.
- [Webhooks](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/kubernetes-operator/webhooks.md): Dynamo operator admission webhooks that validate and default Dynamo custom resources before persistence.
- [Managing Models with DynamoModel](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/kubernetes-operator/managing-models-dynamo-model.md): Deploys LoRA adapters and tracks model endpoints with the DynamoModel custom resource alongside DGDs.
- [Autoscaling](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/kubernetes-operator/autoscaling.md): Scales DGD services with the DynamoGraphDeploymentScalingAdapter using KEDA, Kubernetes HPA, or the Dynamo Planner.
- [Rolling Updates](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/kubernetes-operator/rolling-update.md): Updates DGD worker images, resources, and arguments with managed rolling updates across Deployment, Grove, and LWS backends.
- [Snapshotting GPU Workers](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/kubernetes-operator/snapshot.md): Checkpoints initialized GPU workers with CRIU and cuda-checkpoint so later pods warm-start in seconds instead of minutes.
- [Shadow Engine Failover](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/kubernetes-operator/shadow-engine-failover.md): Keeps a standby engine attached to GPU-resident model weights so process failures recover without reloading the model.
- [Disaggregated Inference Communication Guide](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/kubernetes-operator/disagg-communication.md): Best practices for prefill/decode worker communication on Kubernetes
- [Gateway API Routing Architecture](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/kubernetes-operator/gateway-api-routing.md): How Gateway API, the Dynamo Endpoint Picker Plugin, and worker Frontend sidecars divide request-routing responsibilities.
- [Multinode Deployments](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/multinode/multinode-deployments.md): Scales Dynamo inference across multiple GPU nodes with Grove and KAI-Scheduler for large-model tensor parallelism.
- [Grove](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/multinode/grove.md): Kubernetes API for orchestrating disaggregated inference with gang scheduling, multi-level autoscaling, and topology-aware placement.
- [Topology Aware Scheduling](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/multinode/topology-aware-scheduling.md): Packs prefill, decode, and routing pods within the same rack or zone to reduce inter-node latency.
- [Topology-Aware KV Transfer](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/multinode/topology-aware-kv-transfer.md): Keep disaggregated prefill and decode KV-cache transfers within a selected topology domain
- [LWS](https://docs.nvidia.com/dynamo/v1.4.0/knowledge-base/kubernetes/multinode/lws.md): LeaderWorkerSet integration for multinode Dynamo deployments