> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # vLLM Dynamo vLLM integrates [vLLM](https://github.com/vllm-project/vllm) engines into Dynamo's distributed runtime, enabling disaggregated serving, KV-aware routing, and request cancellation while maintaining full compatibility with vLLM's native engine arguments. Dynamo leverages vLLM's native KV cache events, NIXL-based transfer mechanisms, and metric reporting to enable KV-aware routing and P/D disaggregation. ## Installation ### Install Latest Release We recommend using [uv](https://github.com/astral-sh/uv) to install: ```bash uv venv --python 3.12 --seed uv pip install "ai-dynamo[vllm]" ``` This installs Dynamo with the compatible vLLM version. --- ### Container We have public images available on [NGC Catalog](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-dynamo/collections/ai-dynamo/artifacts): ```bash docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime: ./container/run.sh -it --framework VLLM --image nvcr.io/nvidia/ai-dynamo/vllm-runtime: ``` ```bash python container/render.py --framework vllm --output-short-filename docker build -f container/rendered.Dockerfile -t dynamo:latest-vllm . ``` ```bash ./container/run.sh -it --framework VLLM [--mount-workspace] ``` ### Development Setup For development, use the [devcontainer](https://github.com/ai-dynamo/dynamo/tree/v1.3.0/.devcontainer) which has all dependencies pre-installed. ## Feature Support Matrix | Feature | Status | Notes | |---------|--------|-------| | [**Disaggregated Serving**](/dynamo/v1.3.0/design-docs/disaggregated-serving) | ✅ | Prefill/decode separation with NIXL KV transfer | | [**KV-Aware Routing**](/dynamo/v1.3.0/components/router) | ✅ | Requires explicit KV event publishing on workers for event-driven cache state | | [**SLA-Based Planner**](/dynamo/v1.3.0/components/planner/planner-guide) | ✅ | | | [**KVBM**](/dynamo/v1.3.0/components/kvbm) | ✅ | | | [**LMCache**](/dynamo/v1.3.0/integrations/kv-cache-integrations/lm-cache) | ✅ | CUDA 12.9 and arm64/aarch64 containers may require building LMCache from source | | [**FlexKV**](/dynamo/v1.3.0/integrations/kv-cache-integrations/flex-kv) | ✅ | | | [**Multimodal Support**](/dynamo/v1.3.0/user-guides/multimodal/v-llm-multimodal) | ✅ | Aggregated and P/D image/video serving on legacy and unified Python backends; separate Encode workers use the legacy path | | [**Observability**](/dynamo/v1.3.0/backends/v-llm/observability) | ✅ | Metrics and monitoring | | **WideEP** | ✅ | Support for DeepEP | | **DP Rank Routing** | ✅ | [Hybrid load balancing](https://docs.vllm.ai/en/stable/serving/data_parallel_deployment/?h=external+dp#hybrid-load-balancing) via external DP rank control | | [**LoRA**](https://github.com/ai-dynamo/dynamo/tree/v1.3.0/examples/backends/vllm/launch/lora/README.md) | ✅ | Dynamic loading/unloading from S3-compatible storage | | **GB200 Support** | ✅ | Container functional on main | ## KV Routing Requirements Starting the frontend with `--router-mode kv` does not enable KV event publishing on vLLM workers. For event-driven cache-aware routing, enable publishing on every aggregated or prefill worker whose cache state the router should track: ```bash python -m dynamo.vllm \ --model Qwen/Qwen3-0.6B \ --enable-prefix-caching \ --kv-events-config '{"enable_kv_cache_events":true,"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' ``` `endpoint` is the base ZMQ port, and vLLM offsets it by data-parallel rank. For worker processes that share a host or network namespace, reserve one port per rank and choose base ports whose resulting ranges do not overlap. If workers will not publish events, start the frontend with `--no-router-kv-events` for approximate cache prediction or `--load-aware` for load-only routing. ## Quick Start Start infrastructure services for local development: ```bash docker compose -f dev/docker-compose.yml up -d ``` Launch an aggregated serving deployment: ```bash cd $DYNAMO_HOME/examples/backends/vllm bash launch/agg.sh ``` > **Running launch scripts standalone.** The `launch/*.sh` scripts expect etcd and NATS to be reachable on localhost. Bring them up first (run from the repo root, or use the absolute path shown): > > ```bash > docker compose -f "$DYNAMO_HOME/dev/docker-compose.yml" up -d > ``` > > Then run the launch script. Without these, workers register but the frontend cannot discover them and requests hang. ### Rust Backend Preview The Python vLLM backend remains the recommended entry point for production deployments and examples. The Rust backend is a development preview for validating the Rust `LLMEngine` integration with vLLM's engine-core client. Use it when working on the Rust backend contract, cancellation, metrics, or P/D wiring; use `python -m dynamo.vllm` or `python -m dynamo.vllm.unified_main` for the most complete vLLM feature coverage. The Rust backend depends on vLLM's engine-core crates, which are not yet published to crates.io and are pulled as git dependencies. They are gated behind the off-by-default `vllm_rs` cargo feature, so the default workspace build does not require the git sources and the crate is excluded from the published Dynamo crates. You must pass `--features vllm_rs` to build or run it. To run the Rust backend locally, start the same infrastructure services and frontend, then launch the Rust worker in another terminal: ```bash docker compose -f dev/docker-compose.yml up -d python -m dynamo.frontend --http-port 8000 ``` ```bash DYN_SYSTEM_PORT=8081 cargo run -p dynamo-vllm-rs-backend --features vllm_rs -- Qwen/Qwen3-0.6B -- \ --enforce-eager \ --max-model-len 4096 ``` The Rust worker starts a managed vLLM engine-core process and registers with the Dynamo frontend using the same discovery path as the Python unified backend. The Rust backend is expected to become the default only after it reaches feature and operational parity with the Python vLLM backend. ## Next Steps - **[Reference Guide](/dynamo/v1.3.0/backends/v-llm/reference-guide)**: Configuration, arguments, and operational details - **[Examples](/dynamo/v1.3.0/backends/v-llm/examples)**: All deployment patterns with launch scripts - **[KV Cache Offloading](/dynamo/v1.3.0/backends/v-llm/kv-cache-offloading)**: KVBM, LMCache, and FlexKV integrations - **[Observability](/dynamo/v1.3.0/backends/v-llm/observability)**: Metrics and monitoring - **[vLLM-Omni](/dynamo/v1.3.0/backends/v-llm/v-llm-omni)**: Multimodal model serving - **[Kubernetes Deployment](https://github.com/ai-dynamo/dynamo/tree/v1.3.0/examples/backends/vllm/deploy/README.md)**: Kubernetes deployment guide - **[vLLM Documentation](https://docs.vllm.ai/en/stable/)**: Upstream vLLM serve arguments > vLLM engines run in Dynamo's distributed runtime with disaggregated serving, NIXL KV transfer, and KV-aware routing. ## Docs - [Reference Guide](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm/reference-guide.md): Configuration, arguments, and operational details for the vLLM backend - [vLLM Chat Processor](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm/frontend-processor-fallback.md): vLLM-native preprocessing and postprocessing for chat completions - [Logits Processing](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm/logits-processing.md): Custom logits processors plug into vLLM through Dynamo's engine-loaded adapter and per-request SamplingParams.extra_args. - [Examples](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm/examples.md): Launch scripts cover aggregated, disaggregated, KV-routed, and expert-parallel deployment patterns for the vLLM backend. - [KV Cache Offloading](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm/kv-cache-offloading.md): CPU and disk offloading integrations for vLLM in Dynamo - [Prometheus](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm/observability.md): vLLM exposes request, latency, scheduler, and disaggregation metrics through Dynamo's Prometheus /metrics endpoint. - [vLLM-Omni](https://docs.nvidia.com/dynamo/v1.3.0/backends/v-llm/v-llm-omni.md): vLLM-Omni serves text-to-image, text-to-video, image-to-video, and text-to-speech models through OpenAI-compatible endpoints.