Local Installation | NVIDIA Dynamo Documentation

This guide walks through installing Dynamo on a local machine or VM with one or more GPUs.

For production multi-node clusters, see the Kubernetes Deployment Guide. To build from source for development, see Building from Source.

System Requirements

Requirement	Supported
GPU	NVIDIA Ampere, Ada Lovelace, Hopper, Blackwell
OS	Ubuntu 22.04, Ubuntu 24.04
Architecture	x86_64, ARM64 (ARM64 requires Ubuntu 24.04)
CUDA	12.9+ or 13.0+ (B300/GB300 require CUDA 13)
Python	3.10, 3.12
Driver	575.51.03+ (CUDA 12) or 580.00.03+ (CUDA 13)

TensorRT-LLM does not support Python 3.11.

For the full compatibility matrix including backend framework versions, see the Support Matrix.

Prerequisites

Before installing Dynamo, make sure the host has the following. The GPU driver is the only piece that must live on the host in every case — the prebuilt container bundles CUDA and all framework dependencies, so the CUDA toolkit is only needed for the Python virtual environment path.

Prerequisite	Container path	Python venv path	Notes
NVIDIA GPU driver	✅ Required	✅ Required	The kernel driver cannot be containerized. Use the version from System Requirements above and verify with `nvidia-smi`.
Docker Engine	✅ Required	—	Runs the runtime image and the NATS/etcd Compose stack.
NVIDIA Container Toolkit	✅ Required	—	Makes `--gpus all` work. Without it, the container starts but cannot see the GPU.
CUDA toolkit (`nvcc`, headers)	Bundled in container	✅ Required	Only the host-install path needs it; the container already includes CUDA.
`HF_TOKEN`	For gated models	For gated models	Export it before launch and accept the model license on the Hugging Face model page: `export HF_TOKEN=hf_...`

Verify the driver is installed and the GPU is visible:

$ nvidia-smi

This covers single-machine setups only. Multi-node disaggregated serving additionally needs an RDMA-capable interconnect (NIXL/UCX) — out of scope for this guide. For clusters, the Kubernetes Deployment Guide covers the GPU Operator and, where applicable, the Network Operator instead of these host-level installs.

Install Dynamo

Clone the Dynamo repository

The repository provides the infrastructure Compose file, launch scripts, and engine configs referenced throughout this guide:

$ git clone https://github.com/ai-dynamo/dynamo.git
$ cd dynamo

Start infrastructure services (NATS + etcd)

Dynamo components discover each other through etcd and exchange events through NATS. Bring both up in the background with Docker Compose:

$ docker compose -f dev/docker-compose.yml up -d

Leave these running for the rest of the guide.

NATS and etcd are not strictly required for a single-machine setup — you can pass --discovery-backend file to the frontend and worker to skip them entirely (discovery falls back to the local filesystem and events to ZMQ). They are recommended because they match the multi-node, production, and Kubernetes deployment path. See Discovery and events on a single machine below for the trade-offs.

Install for your backend

You can run Dynamo two ways. Choose one:

Prebuilt container (recommended) — all dependencies are pre-installed and the repository contents are bundled at /workspace. Because the frontend and worker run inside this container, you’ll need to docker exec into it to run subsequent commands.
Python virtual environment (PyPI) — install the ai-dynamo wheel for your backend directly on the host.

Pick a backend, release channel, backend version, and install form. The default selection starts the latest stable SGLang container.

Choose your local build

Unavailable combinations remain visible to show the current support boundary.

Backend

Dynamo build

SGLang version

Install form

Latest stable release that supports this version

StableDynamo 1.3.0

NVIDIA GPU · SGLang 0.5.14

docker run --gpus all --network host --ipc host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0

The container command drops you into a shell with Dynamo and its backend dependencies installed. --network host lets the container reach NATS and etcd on the host.

Before using a wheel command, create and activate a Python virtual environment:

$ # Install uv (recommended Python package manager)
$ curl -LsSf https://astral.sh/uv/install.sh | sh
$ 
$ # Create and activate the virtual environment
$ uv venv venv
$ source venv/bin/activate
$ uv pip install pip

Install the host packages required by your backend before running the selected wheel command:

SGLang

TensorRT-LLM

vLLM

$ sudo apt install python3-dev

For CUDA 13 (B300/GB300), the container is recommended. See the SGLang install docs for details.

See Release Artifacts for the complete artifact inventory.

Verify installation

If you used the prebuilt container, run this command inside the container. If you started it in another shell, docker exec into it first:

$ docker exec -it <container_id> bash

Verify the CLI is installed and callable:

$ python3 -m dynamo.frontend --help

You can also run additional system checks from the repository root:

$ python3 dev/sanity_check.py

Discovery and events on a single machine

Dynamo is a distributed runtime: the frontend (the OpenAI-compatible HTTP server) and the worker (the engine running your model) are separate processes that have to find each other. Two planes make that happen:

Discovery plane — how the frontend learns which workers exist. Workers register their endpoints on startup; the frontend watches for them and routes requests accordingly.
Event plane — how components exchange KV-cache events and worker metrics, which power features like KV-aware routing and the planner.

On a single machine you can run either of two configurations:

	`--discovery-backend file`	etcd + NATS (this guide)
Extra services	None	etcd + NATS (via Docker Compose)
Discovery	Local filesystem	etcd
Events	ZMQ (in-process)	NATS
Scope	Single machine only	Single or multi-node
Failure cleanup	None	Lease-based auto-cleanup (default 10s TTL) — a crashed worker’s endpoints are removed automatically and the frontend reroutes
KV events / routing / planner	Not available	Available
Matches production / Kubernetes	No	Yes

The file mode is the lightest way to get a model answering with no prerequisites — ideal for a quick local smoke test. This guide uses etcd + NATS instead because it is the same discovery path Dynamo uses for multi-node and Kubernetes deployments, so what you learn here carries forward to production.

To use the zero-dependency mode instead, skip the infrastructure step above and pass --discovery-backend file to both the frontend and each worker.

This applies to local, single-machine setups. On Kubernetes, the Dynamo operator manages discovery natively (via custom resources and EndpointSlices) and sets DYN_DISCOVERY_BACKEND=kubernetes automatically — neither file mode nor a standalone etcd is used. See the Discovery Plane design doc for the full picture.

Troubleshooting

CUDA/driver version mismatch

Run nvidia-smi to check your driver version. Dynamo requires driver 575.51.03+ for CUDA 12 or 580.00.03+ for CUDA 13. B300/GB300 GPUs require CUDA 13. See the Support Matrix for full requirements.

Python 3.11 with TensorRT-LLM

TensorRT-LLM does not support Python 3.11. If you see installation failures with TensorRT-LLM, check your Python version with python3 --version. Use Python 3.10 or 3.12 instead.

Container runs but GPU not detected

Ensure you passed --gpus all to docker run. Without this flag, the container won’t have access to GPUs:

$ # Correct
$ docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.2.1
$ 
$ # Wrong -- no GPU access
$ docker run --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.2.1

Next Steps

Backend Guides — Backend-specific configuration and features
Disaggregated Serving — Scale prefill and decode independently
KV Cache Aware Routing — Smart request routing
Kubernetes Deployment — Production multi-node deployments