> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# Local Installation

> Install Dynamo on a local machine or VM with containers or PyPI

This guide walks through installing Dynamo on a local machine or VM with one or more GPUs.

For production multi-node clusters, see the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart). To build from source for development, see [Building from Source](/dynamo/dev/advanced-customizations/building-from-source).

## System Requirements

| Requirement      | Supported                                      |
| ---------------- | ---------------------------------------------- |
| **GPU**          | NVIDIA Ampere, Ada Lovelace, Hopper, Blackwell |
| **OS**           | Ubuntu 22.04, Ubuntu 24.04                     |
| **Architecture** | x86\_64, ARM64 (ARM64 requires Ubuntu 24.04)   |
| **CUDA**         | 12.9+ or 13.0+ (B300/GB300 require CUDA 13)    |
| **Python**       | 3.10, 3.12                                     |
| **Driver**       | 575.51.03+ (CUDA 12) or 580.00.03+ (CUDA 13)   |

TensorRT-LLM does not support Python 3.11.

For the full compatibility matrix including backend framework versions, see the [Support Matrix](/dynamo/dev/reference/compatibility).

## Prerequisites

Before installing Dynamo, make sure the host has the following. The GPU driver is the only piece that must live on the host in every case — the prebuilt container bundles CUDA and all framework dependencies, so the CUDA toolkit is only needed for the Python virtual environment path.

| Prerequisite                                                                                                                | Container path       | Python venv path | Notes                                                                                                                   |
| --------------------------------------------------------------------------------------------------------------------------- | -------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------- |
| **NVIDIA GPU driver**                                                                                                       | ✅ Required           | ✅ Required       | The kernel driver cannot be containerized. Use the version from System Requirements above and verify with `nvidia-smi`. |
| **Docker Engine**                                                                                                           | ✅ Required           | —                | Runs the runtime image and the NATS/etcd Compose stack.                                                                 |
| **[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)** | ✅ Required           | —                | Makes `--gpus all` work. Without it, the container starts but cannot see the GPU.                                       |
| **CUDA toolkit** (`nvcc`, headers)                                                                                          | Bundled in container | ✅ Required       | Only the host-install path needs it; the container already includes CUDA.                                               |
| **`HF_TOKEN`**                                                                                                              | For gated models     | For gated models | Export it before launch and accept the model license on the Hugging Face model page: `export HF_TOKEN=hf_...`           |

Verify the driver is installed and the GPU is visible:

```bash
nvidia-smi
```

This covers single-machine setups only. Multi-node disaggregated serving additionally needs an RDMA-capable interconnect (NIXL/UCX) — out of scope for this guide. For clusters, the [Kubernetes Deployment Guide](/dynamo/dev/kubernetes/getting-started/quickstart) covers the GPU Operator and, where applicable, the Network Operator instead of these host-level installs.

## Install Dynamo

#### Clone the Dynamo repository

The repository provides the infrastructure Compose file, launch scripts, and engine configs referenced throughout this guide:

```bash
git clone https://github.com/ai-dynamo/dynamo.git
cd dynamo
```

#### Start infrastructure services (NATS + etcd)

Dynamo components discover each other through etcd and exchange events through NATS. Bring both up in the background with Docker Compose:

```bash
docker compose -f dev/docker-compose.yml up -d
```

Leave these running for the rest of the guide.

NATS and etcd are **not strictly required** for a single-machine setup — you can pass `--discovery-backend file` to the frontend and worker to skip them entirely (discovery falls back to the local filesystem and events to ZMQ). They are **recommended** because they match the multi-node, production, and Kubernetes deployment path. See [Discovery and events on a single machine](#discovery-and-events-on-a-single-machine) below for the trade-offs.

#### Install for your backend

You can run Dynamo two ways. Choose one:

* **Prebuilt container (recommended)** — all dependencies are pre-installed and the repository contents are bundled at `/workspace`. Because the frontend and worker run *inside* this container, you'll need to `docker exec` into it to run subsequent commands.
* **Python virtual environment (PyPI)** — install the `ai-dynamo` wheel for your backend directly on the host.

Pick a backend, release channel, backend version, and install form. The default selection starts the latest stable SGLang container.

The container command drops you into a shell with Dynamo and its backend dependencies installed. `--network host` lets the container reach NATS and etcd on the host.

Before using a wheel command, create and activate a Python virtual environment:

```bash
# Install uv (recommended Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create and activate the virtual environment
uv venv venv
source venv/bin/activate
uv pip install pip
```

Install the host packages required by your backend before running the selected wheel command:

#### SGLang

```bash
sudo apt install python3-dev
```

For CUDA 13 (B300/GB300), the container is recommended. See the [SGLang install docs](https://docs.sglang.io/get_started/install.html) for details.

#### TensorRT-LLM

TensorRT-LLM is available through the prebuilt container. The selector does not offer a wheel command for this backend. See the [TensorRT-LLM backend guide](/dynamo/dev/knowledge-base/modular-components/backends/tensor-rt-llm/overview) for details.

#### vLLM

```bash
sudo apt install python3-dev libxcb1
```

See [Release Artifacts](/dynamo/dev/reference/release-artifacts#container-images) for the complete artifact inventory.

#### Verify installation

If you used the **prebuilt container**, run this command *inside* the container. If you started it in another shell, `docker exec` into it first:

```bash
docker exec -it <container_id> bash
```

Verify the CLI is installed and callable:

```bash
python3 -m dynamo.frontend --help
```

You can also run additional system checks from the repository root:

```bash
python3 dev/sanity_check.py
```

## Discovery and events on a single machine

Dynamo is a distributed runtime: the **frontend** (the OpenAI-compatible HTTP server) and the **worker** (the engine running your model) are separate processes that have to find each other. Two planes make that happen:

* **Discovery plane** — how the frontend learns which workers exist. Workers register their endpoints on startup; the frontend watches for them and routes requests accordingly.
* **Event plane** — how components exchange KV-cache events and worker metrics, which power features like KV-aware routing and the planner.

On a single machine you can run either of two configurations:

|                                     | `--discovery-backend file` | etcd + NATS (this guide)                                                                                                      |
| ----------------------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| **Extra services**                  | None                       | etcd + NATS (via Docker Compose)                                                                                              |
| **Discovery**                       | Local filesystem           | etcd                                                                                                                          |
| **Events**                          | ZMQ (in-process)           | NATS                                                                                                                          |
| **Scope**                           | Single machine only        | Single **or** multi-node                                                                                                      |
| **Failure cleanup**                 | None                       | Lease-based auto-cleanup (default 10s TTL) — a crashed worker's endpoints are removed automatically and the frontend reroutes |
| **KV events / routing / planner**   | Not available              | Available                                                                                                                     |
| **Matches production / Kubernetes** | No                         | Yes                                                                                                                           |

The `file` mode is the lightest way to get a model answering with no prerequisites — ideal for a quick local smoke test. This guide uses **etcd + NATS** instead because it is the same discovery path Dynamo uses for multi-node and Kubernetes deployments, so what you learn here carries forward to production.

To use the zero-dependency mode instead, skip the infrastructure step above and pass `--discovery-backend file` to both the frontend and each worker.

This applies to **local, single-machine** setups. On Kubernetes, the Dynamo operator manages discovery natively (via custom resources and EndpointSlices) and sets `DYN_DISCOVERY_BACKEND=kubernetes` automatically — neither `file` mode nor a standalone etcd is used. See the [Discovery Plane](/dynamo/dev/knowledge-base/design-documents/communication-planes/discovery-plane) design doc for the full picture.

## Troubleshooting

**CUDA/driver version mismatch**

Run `nvidia-smi` to check your driver version. Dynamo requires driver 575.51.03+ for CUDA 12 or 580.00.03+ for CUDA 13. B300/GB300 GPUs require CUDA 13. See the [Support Matrix](/dynamo/dev/reference/compatibility) for full requirements.

**Python 3.11 with TensorRT-LLM**

TensorRT-LLM does not support Python 3.11. If you see installation failures with TensorRT-LLM, check your Python version with `python3 --version`. Use Python 3.10 or 3.12 instead.

**Container runs but GPU not detected**

Ensure you passed `--gpus all` to `docker run`. Without this flag, the container won't have access to GPUs:

```bash
# Correct
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.2.1

# Wrong -- no GPU access
docker run --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.2.1
```

## Next Steps

* [Backend Guides](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/overview) -- Backend-specific configuration and features
* [Disaggregated Serving](/dynamo/dev/kubernetes/model-deployment/disaggregated-serving) -- Scale prefill and decode independently
* [KV Cache Aware Routing](/dynamo/dev/knowledge-base/modular-components/router/router-guide) -- Smart request routing
* [Kubernetes Deployment](/dynamo/dev/kubernetes/getting-started/quickstart) -- Production multi-node deployments