> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Knowledge Base

## Docs

- [Overall Architecture](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/overview.md): Architecture and components of the Dynamo inference runtime
- [Architecture](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/architecture.md): Request flow, distributed runtime, service discovery, and communication planes
- [Disaggregated Serving](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/system-architecture/disaggregated-serving.md)
- [KV-Aware Routing](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/system-architecture/kv-aware-routing.md): Select workers using reusable KV cache state and projected active load.
- [Observability Architecture](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/observability-architecture.md): Signal transport, request correlation, and active worker health checks
- [DynoSim](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/overview.md): Simulate Dynamo deployment choices before spending GPU time
- [DynoSim Architecture](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/dyno-sim-architecture.md): How the replay harness composes simulated engines, routing, and Planner behavior
- [DynoSim Simulation Model](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/simulation-model.md): Engine behavior, timing sources, KV movement, and fidelity boundaries
- [Run a DynoSim Simulation](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/simulation-runs.md): Predict a synthetic workload or saved trace against one simulated configuration
- [Sweep DynoSim Configurations](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/simulation-sweeps.md): Recommend simulated topology, worker, and router choices before using GPU time
- [Simulate a Local Deployment with Mocker](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/local-mocker.md): Run a GPU-free frontend and simulated worker from the command line
- [Simulate a Kubernetes Deployment with Mocker](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/kubernetes-mocker.md): Exercise the Dynamo frontend, router, and worker lifecycle without GPUs
- [Benchmark Planner Decisions with DynoSim](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/planner-replay.md): Compare topology and scaling decisions against a saved trace
- [AISimulate (Experimental)](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/ai-simulate/overview.md): Backend-neutral simulation and configuration-search tools
- [Sweeper](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/ai-simulate/sweeper/overview.md): Experimental backend-neutral configuration search
- [Sweeper Quickstart](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/ai-simulate/sweeper/quickstart.md): Run a backend-neutral sweep with an injected replay runtime
- [Sweeper Tutorial](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/ai-simulate/sweeper/tutorial.md): Configure, execute, and inspect a replay-backed search
- [Sweeper Architecture](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/ai-simulate/sweeper/architecture.md): Backend-neutral search, provider materialization, and replay execution
- [Sweeper Configuration](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/ai-simulate/sweeper/configuration.md): Core fields and optional adapter-owned search spaces
- [Sweeper Optimization Goals](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/ai-simulate/sweeper/optimization-goals.md): Objective metrics, SLA constraints, and Pareto-front scoring
- [Sweeper Traffic](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/ai-simulate/sweeper/traffic.md): Trace, request-rate, concurrency, and KV-load workload definitions
- [Sweeper Results](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/ai-simulate/sweeper/results.md): Replay specifications, ranked candidates, and Pareto fronts
- [Sweep Configuration Providers](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/ai-simulate/sweeper/sweep-configuration-providers.md): Extend Sweeper without adding application dependencies to its core
- [Dynamo Sweeper Integration](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/ai-simulate/sweeper/dynamo-integration.md): Install and compose Planner, Router, and Dynamo Replay with AISimulate
- [GLM-5-FP8 Disaggregated Pareto Sweep](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/ai-simulate/sweeper/glm-5-fp-8-pareto-sweep.md): Experimental Sweeper search for a GLM-5-FP8 Pareto frontier on B200 GPUs
- [Planner Goodput-per-GPU Sweep](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/ai-simulate/sweeper/planner-goodput-per-gpu-sweep.md): Experimental Sweeper search over Planner policies for MiniMax-M2.5 on B200 GPUs
- [Router End-to-End Latency Sweep](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/simulation/ai-simulate/sweeper/router-end-to-end-latency-sweep.md): Experimental Sweeper search over KV router settings for MiniMax-M2.5 on B200 GPUs
- [Request Migration Architecture](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/fault-tolerance/request-migration-architecture.md): Preserve token state and reissue in-flight requests on healthy workers after a worker failure.
- [Request Cancellation Architecture](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/fault-tolerance/request-cancellation-architecture.md): Cancel in-flight work across the Frontend and workers when a client disconnects.
- [Request Rejection Architecture](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/fault-tolerance/request-rejection-architecture.md): Worker-load event processing, busy-state aggregation, overload errors, and hard worker admission limits.
- [Graceful Shutdown Architecture](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/fault-tolerance/graceful-shutdown-architecture.md)
- [Fault Tolerance Testing](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/fault-tolerance/fault-tolerance-testing.md): Pytest infrastructure for cancellation, migration, etcd high availability, and hardware fault injection.
- [Sidecar Backends](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/concepts/system-architecture/sidecar-backends.md): Run Dynamo beside a stock inference engine through its native gRPC API.
- [Frontend](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/frontend/overview.md): API gateway that exposes OpenAI HTTP and KServe gRPC endpoints and handles request preprocessing, routing, and response formatting.
- [KServe gRPC Frontend](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/frontend/kserve-grpc-frontend.md): Configure KServe endpoints, backend registration, message conversion, and HTTP/2 flow control
- [Python Route Extensions](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/frontend/python-route-extensions.md): Add trusted HTTP routes to the Dynamo Frontend from an installed Python package or importable module.
- [Tokenizer](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/frontend/tokenizer.md): Selects the HuggingFace, fastokens, or Baseten tokenizer backend for BPE models served through the Dynamo Frontend.
- [Router](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/overview.md): Choose a request path, understand worker selection, and operate KV-aware routing.
- [Router Guide](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/router-guide.md): Choose where routing runs and how worker-set configuration takes effect.
- [Routing Concepts](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/routing-concepts.md): Cost model, worker selection, and routing primitives for the Dynamo router
- [Router Filtering](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/router-filtering.md): Candidate eligibility, DP-rank filtering, and queue backpressure boundaries
- [Deficit Round Robin Queue Scheduling](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/deficit-round-robin.md): Weighted arbitration across router policy classes
- [Configuration and Tuning](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/configuration-and-tuning.md): Router behavior, event transport, load tracking, and tuning guidance
- [Router Operations](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/router-operations.md): Replica topology, state management, recovery, and remote indexers
- [KV Event Replay — Dynamo vs vLLM](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/kv-event-replay-comparison.md): How the two systems handle gap detection, replay, and recovery for KV cache events
- [Global Router Configuration](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/global-router-configuration.md): Field reference for the CLI flags and JSON config file consumed by the Global Router service.
- [Disaggregated Serving](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/disaggregated-serving.md): Prefill and decode routing with the Dynamo router
- [Topology-Aware KV Transfer](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/topology-aware-kv-transfer.md): Runtime metadata and decode routing semantics for topology-aware prefill/decode handoff
- [Offloading Support Matrix](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/offloading-support-matrix.md): Per-framework support and setup for KV routing with tiered KV cache offloading
- [Multi-DC KV Routing and the DC Relay](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/multi-dc-kv-routing.md): Aggregate exact KV ownership in each DC and publish compact state to a global consumer
- [Standalone KV Indexer](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/standalone-indexer.md): Run the KV cache indexer as an independent HTTP service for querying block state
- [Standalone Slot Tracker](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/standalone-slot-tracker.md): Run active-request load accounting as an independent HTTP service
- [Standalone Selection Service](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/standalone-selection.md): Select workers and account for reservations without forwarding inference requests
- [Write Custom Routing Strategies](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/write-custom-routing-strategies.md): Build Rust filters, scorers, and pickers for the Dynamo frontend or EPP
- [Router Examples](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/router-examples.md): Worked examples for the KvRouter Python API, Kubernetes deployments, and custom routing patterns.
- [Router Testing](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/router-testing.md): Test layers for router changes
- [Router Design](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/router/router-design.md)
- [Planner](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/planner/overview.md): Autoscaler that adjusts prefill and decode replicas using engine performance models and traffic prediction to meet TTFT and ITL SLAs.
- [Planner Guide](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/planner/planner-guide.md): Configures Planner optimization targets, scaling modes, and PlannerConfig fields for production deployments.
- [Global Planner Deployment Guide](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/planner/global-planner-guide.md): Deploys GlobalPlanner as the centralized scaling layer for multi-DGD policy enforcement and single-endpoint multi-pool deployments.
- [Planner Examples](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/planner/planner-examples.md): Examples for custom load predictors and the VirtualConnector for non-Kubernetes scaling environments.
- [Planner Design](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/planner/planner-design.md)
- [Profiler](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/profiler/overview.md): Automated performance analysis that finds optimal prefill and decode parallelism and generates SLA-driven deployment configurations.
- [Profiler Guide](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/profiler/profiler-guide.md): Runs the AIConfigurator-driven profiling pipeline that turns a model, SLA targets, and backend into a ready-to-deploy DGD.
- [Profiler Examples](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/profiler/profiler-examples.md): Complete DGDR profiling examples for dense and MoE models across rapid and thorough search strategies.
- [SGLang](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/sg-lang/overview.md): SGLang engines run in Dynamo's distributed runtime with disaggregated serving, KV-aware routing, and request cancellation.
- [SGLang Sidecar](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/sg-lang/sidecar.md): Run Dynamo beside a stock SGLang engine through native gRPC.
- [Reference Guide](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/sg-lang/reference-guide.md): Architecture, configuration, and operational details for the SGLang backend
- [SGLang for Agentic Workloads](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/sg-lang/agents-on-sg-lang.md): Priority scheduling and session control for multi-turn agentic serving
- [SGLang Chat Processor](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/sg-lang/chat-processor.md): SGLang-native preprocessing and postprocessing for chat completions
- [Disaggregation](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/sg-lang/disaggregation.md): SGLang runs prefill and decode as separate workers in Dynamo, coordinated by a bootstrap handshake with RDMA-based KV cache transfer.
- [Using HiCache](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/sg-lang/hi-cache.md): Hierarchical KV caching with tier-aware router integration
- [Observability](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/sg-lang/observability.md): Prometheus metrics, forward pass telemetry, distributed tracing, and Grafana dashboards for SGLang workers in Dynamo.
- [SGLang Multimodal](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/sg-lang/sglang-multimodal.md)
- [TensorRT-LLM](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/tensor-rt-llm/overview.md): TensorRT-LLM engines run in Dynamo's distributed runtime with disaggregated serving, KV-aware routing, and multinode support.
- [TensorRT-LLM Sidecar](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/tensor-rt-llm/sidecar.md): Run Dynamo beside a stock TensorRT-LLM engine through native gRPC.
- [Reference Guide](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/tensor-rt-llm/reference-guide.md): Features, configuration, and operational details for the TensorRT-LLM backend
- [TensorRT-LLM Observability](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/tensor-rt-llm/observability.md): TensorRT-LLM exposes request, latency, and KV cache transfer metrics through Dynamo's Prometheus /metrics endpoint.
- [Known Issues and Mitigations](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/tensor-rt-llm/known-issues.md): Known issues for TensorRT-LLM in Dynamo, including disaggregated KV cache exhaustion and driver mismatch errors, with mitigations.
- [TensorRT-LLM Multimodal](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/tensor-rt-llm/tensorrt-llm-multimodal.md)
- [Triton](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/triton/overview.md)
- [vLLM](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/v-llm/overview.md)
- [vLLM Sidecar](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/v-llm/sidecar.md): Run Dynamo beside a stock vLLM engine through native gRPC.
- [Reference Guide](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/v-llm/reference-guide.md): Configuration, arguments, and operational details for the vLLM backend
- [vLLM Chat Processor](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/v-llm/chat-processor.md): vLLM-native preprocessing and postprocessing for chat completions
- [KV Cache Offloading](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/v-llm/kv-cache-offloading.md): CPU and disk offloading integrations for vLLM in Dynamo
- [Native KV Offloading](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/v-llm/native-kv-offloading.md): vLLM's OffloadingConnector with KV-aware routing in Dynamo
- [vLLM Observability](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/v-llm/observability.md): vLLM exposes request, latency, scheduler, and disaggregation metrics through Dynamo's Prometheus /metrics endpoint.
- [vLLM Multimodal](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/v-llm/vllm-multimodal.md)
- [vLLM-Omni Disaggregated Multi-Stage Serving](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/v-llm/v-llm-omni-disaggregated-serving.md): Architecture of multi-stage vLLM-Omni pipelines (e.g. AR + Diffusion), with each stage on its own GPU for independent scaling and isolation.
- [Mocker Engine Architecture](https://docs.nvidia.com/dynamo/v1.5.1/knowledge-base/modular-components/backends/mocker/mocker-engine-architecture.md)