> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.

# TensorRT-LLM

## Use the Latest Release

We recommend using the [latest stable release](https://github.com/ai-dynamo/dynamo/releases/latest) of Dynamo to avoid breaking changes.

---

Dynamo TensorRT-LLM integrates [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) engines into Dynamo's distributed runtime, enabling disaggregated serving, KV-aware routing, multi-node deployments, and request cancellation. It supports LLM inference, multimodal models, video diffusion, and advanced features like speculative decoding and attention data parallelism.

## Feature Support Matrix

### Core Dynamo Features

| Feature | TensorRT-LLM | Notes |
|---------|--------------|-------|
| [**Disaggregated Serving**](/dynamo/v1.3.0/design-docs/disaggregated-serving) | ✅ |  |
| [**Conditional Disaggregation**](/dynamo/v1.3.0/design-docs/disaggregated-serving) | 🚧 | Not supported yet |
| [**KV-Aware Routing**](/dynamo/v1.3.0/components/router) | ✅ |  |
| [**SLA-Based Planner**](/dynamo/v1.3.0/components/planner/planner-guide) | ✅ |  |
| [**Load Based Planner**](/dynamo/v1.3.0/components/planner) | 🚧 | Planned |
| [**KVBM**](/dynamo/v1.3.0/components/kvbm) | ✅ | |

### Large Scale P/D and WideEP Features

| Feature            | TensorRT-LLM | Notes                                                           |
|--------------------|--------------|-----------------------------------------------------------------|
| **WideEP**         | ✅           |                                                                 |
| **DP Rank Routing**| ✅           |                                                                 |
| **GB200 Support**  | ✅           |                                                                 |

## Prerequisites

- **`yq`** for in-place YAML edits. Install with `wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq && chmod +x /usr/local/bin/yq` or `pip install yq` (the latter is a different tool with the same name but similar syntax). If neither is available, a `sed` fallback is shown inline where `yq` is used.

## Container / driver matrix

| Container tag | Backend version | CUDA | Min NVIDIA driver |
|---|---|---|---|
| `tensorrtllm-runtime:1.0.2` | TRT-LLM `v1.3.0rc5.post1` | `v13.1` | `580+` |
| `vllm-runtime:1.0.2` | vLLM `v0.16.0` | `v12.9` | `575+` |
| `vllm-runtime:1.0.2-cuda13` | vLLM `v0.16.0` | `v13.0` | `580+` |
| `sglang-runtime:1.0.2` | SGLang `v0.5.9` | `v12.9` | `575+` |
| `sglang-runtime:1.0.2-cuda13` | SGLang `v0.5.9` | `v13.0` | `580+` |

Source of truth: [`docs/reference/support-matrix.md`](/dynamo/v1.3.0/resources/support-matrix#cuda-and-driver-requirements) and [`docs/reference/release-artifacts.md`](/dynamo/v1.3.0/resources/release-artifacts). If those differ from the values above, the source-of-truth files win.

## Quick Start

**Step 1 (host terminal):** Start infrastructure services:

```bash
docker compose -f dev/docker-compose.yml up -d
```

**Step 2 (host terminal):** Pull and run the prebuilt container:

```bash
DYNAMO_VERSION=1.0.2
docker pull nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:$DYNAMO_VERSION
docker run --gpus all -it --network host --ipc host \
  nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:$DYNAMO_VERSION
```

<Note>
The `DYNAMO_VERSION` variable above can be set to any specific available version of the container.
To find the available `tensorrtllm-runtime` versions for Dynamo, visit the [NVIDIA NGC Catalog for Dynamo TensorRT-LLM Runtime](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-dynamo/containers/tensorrtllm-runtime).
</Note>

**Step 3 (inside the container):** Launch an aggregated serving deployment (uses `Qwen/Qwen3-0.6B` by default):

```bash
cd $DYNAMO_HOME/examples/backends/trtllm
./launch/agg.sh
```

The launch script will automatically download the model and start the TensorRT-LLM engine. You can override the model by setting `MODEL_PATH` and `SERVED_MODEL_NAME` environment variables before running the script.

**Step 4 (host terminal):** Verify the deployment:

```bash
curl localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [{"role": "user", "content": "Explain why Roger Federer is considered one of the greatest tennis players of all time"}],
    "stream": true,
    "max_tokens": 30
  }'
```

## Deploy

Deploy TensorRT-LLM with Dynamo on Kubernetes using a `DynamoGraphDeployment`. Before `kubectl apply`, substitute the container image tag in the deployment YAML. The `sed` fallback is shown inline for environments without `yq`:

```bash
# yq
yq -i '(.spec.services[].extraPodSpec.mainContainer.image) |= sub(":1\.0\.2", ":<your-tag>")' deploy.yaml
# sed fallback
sed -i.bak 's|:1\.0\.2|:<your-tag>|g' deploy.yaml
```

For full Kubernetes deployment instructions, see the [TensorRT-LLM Kubernetes Deployment Guide](https://github.com/ai-dynamo/dynamo/tree/v1.3.0/examples/backends/trtllm/deploy/README.md).

## Next Steps

- **[Reference Guide](/dynamo/v1.3.0/backends/tensor-rt-llm/reference-guide)**: Features, configuration, and operational details
- **[Examples](/dynamo/v1.3.0/backends/tensor-rt-llm/examples)**: All deployment patterns with launch scripts
- **[KV Cache Transfer](/dynamo/v1.3.0/additional-resources/tensor-rt-llm-details/kv-cache-transfer)**: KV cache transfer methods for disaggregated serving
- **[Observability](/dynamo/v1.3.0/backends/tensor-rt-llm/observability)**: Metrics and monitoring
- **[Multinode Examples](/dynamo/v1.3.0/additional-resources/tensor-rt-llm-details/multinode-examples)**: Multi-node deployment with SLURM
- **[Deploying TensorRT-LLM with Dynamo on Kubernetes](https://github.com/ai-dynamo/dynamo/tree/v1.3.0/examples/backends/trtllm/deploy/README.md)**: Kubernetes deployment guide

> TensorRT-LLM engines run in Dynamo's distributed runtime with disaggregated serving, KV-aware routing, and multinode support.

## Docs

- [Reference Guide](https://docs.nvidia.com/dynamo/v1.3.0/backends/tensor-rt-llm/reference-guide.md): Features, configuration, and operational details for the TensorRT-LLM backend
- [Examples](https://docs.nvidia.com/dynamo/v1.3.0/backends/tensor-rt-llm/examples.md): Launch scripts cover aggregated, disaggregated, KV-routed, and multinode deployment patterns for the TensorRT-LLM backend.
- [Prometheus](https://docs.nvidia.com/dynamo/v1.3.0/backends/tensor-rt-llm/observability.md): TensorRT-LLM exposes request, latency, and KV cache transfer metrics through Dynamo's Prometheus /metrics endpoint.
- [Video Diffusion Support (Experimental)](https://docs.nvidia.com/dynamo/v1.3.0/backends/tensor-rt-llm/diffusion-experimental.md): Experimental text-to-video and text-to-image diffusion in TensorRT-LLM using the visual_gen module and Diffusers pipelines.
- [Known Issues and Mitigations](https://docs.nvidia.com/dynamo/v1.3.0/backends/tensor-rt-llm/known-issues-and-mitigations.md): Known issues for TensorRT-LLM in Dynamo, including disaggregated KV cache exhaustion and driver mismatch errors, with mitigations.