Building a Custom TensorRT-LLM Container

This page is for version v1.1.1. For other versions, use one of these documentation indexes: - Latest (v1.5.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt - dev: https://docs.nvidia.com/dynamo/dev/llms.txt - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt

# TensorRT-LLM uses git-lfs, which needs to be installed in advance.
apt-get update && apt-get -y install git git-lfs
# On an x86 machine:
python container/render.py --framework=trtllm --target=runtime --output-short-filename --cuda-version=13.1
docker build -t dynamo:trtllm-latest -f container/rendered.Dockerfile .
# On an ARM machine:
python container/render.py --framework=trtllm --target=runtime --platform=arm64 --output-short-filename --cuda-version=13.1
docker build -t dynamo:trtllm-latest -f container/rendered.Dockerfile .

./container/run.sh --framework trtllm -it