TensorRT-LLM Deployment Templates | NVIDIA Dynamo Documentation

Copy-paste DynamoGraphDeployment (nvidia.com/v1beta1) manifests for the TensorRT-LLM backend, grouped by topology. Each manifest is embedded from examples/backends/trtllm/deploy/ — open an entry, use the copy button, then set your image tag and hf-token-secret before applying.

Apply any template with:

$ kubectl apply -f agg.yaml

Some templates bundle a ConfigMap (engine configuration) or a PersistentVolumeClaim alongside the deployment. Apply the whole file — every document in it is part of the example.

Aggregated

agg.yaml · Baseline aggregated serving

agg.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: trtllm-agg
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - image: my-registry/tensorrtllm-runtime:my-tag
15           name: main
16     replicas: 1
17     type: frontend
18   - name: TRTLLMWorker
19     podTemplate:
20       spec:
21         containers:
22         - args:
23           - --model-path
24           - Qwen/Qwen3-0.6B
25           - --served-model-name
26           - Qwen/Qwen3-0.6B
27           - --extra-engine-args
28           - ./examples/backends/trtllm/engine_configs/qwen3/agg.yaml
29           command:
30           - python3
31           - -m
32           - dynamo.trtllm
33           envFrom:
34           - secretRef:
35               name: hf-token-secret
36           image: my-registry/tensorrtllm-runtime:my-tag
37           name: main
38           resources:
39             limits:
40               nvidia.com/gpu: "1"
41             requests:
42               # Increase this value for larger models.
43               ephemeral-storage: 2Gi
44           workingDir: /workspace/
45     replicas: 1
46     type: worker

agg_router.yaml · Aggregated with KV-aware routing

agg_router.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: trtllm-agg-router
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - env:
15           - name: DYN_ROUTER_MODE
16             value: kv
17           image: my-registry/tensorrtllm-runtime:my-tag
18           name: main
19     replicas: 1
20     type: frontend
21   - name: TRTLLMWorker
22     podTemplate:
23       spec:
24         containers:
25         - args:
26           - --model-path
27           - Qwen/Qwen3-0.6B
28           - --served-model-name
29           - Qwen/Qwen3-0.6B
30           - --extra-engine-args
31           - ./examples/backends/trtllm/engine_configs/qwen3/agg.yaml
32           - --publish-events-and-metrics
33           command:
34           - python3
35           - -m
36           - dynamo.trtllm
37           envFrom:
38           - secretRef:
39               name: hf-token-secret
40           image: my-registry/tensorrtllm-runtime:my-tag
41           name: main
42           resources:
43             limits:
44               nvidia.com/gpu: "1"
45             requests:
46               # Increase this value for larger models.
47               ephemeral-storage: 2Gi
48           workingDir: /workspace/
49     replicas: 2
50     type: worker

agg-with-config.yaml · Aggregated with an external engine ConfigMap (bundles a ConfigMap)

agg-with-config.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 # configmap that contains the custom trtllm configuration
4 apiVersion: v1
5 kind: ConfigMap
6 metadata:
7   name: nvidia-config
8 data:
9   agg.yaml: |
10     tensor_parallel_size: 1
11     moe_expert_parallel_size: 1
12     enable_attention_dp: false
13     max_num_tokens: 8192
14     max_batch_size: 16
15     trust_remote_code: true
16     backend: pytorch
17     enable_chunked_prefill: true
18     disable_overlap_scheduler: true
19     kv_cache_config:
20       free_gpu_memory_fraction: 0.95
21     cuda_graph_config:
22       max_batch_size: 16
23 ---
24 apiVersion: nvidia.com/v1beta1
25 kind: DynamoGraphDeployment
26 metadata:
27   name: trtllm-agg
28 spec:
29   components:
30   - name: Frontend
31     podTemplate:
32       spec:
33         containers:
34         - image: my-registry/tensorrtllm-runtime:my-tag
35           name: main
36     replicas: 1
37     type: frontend
38   - name: TRTLLMWorker
39     podTemplate:
40       spec:
41         containers:
42         - args:
43           - --model-path
44           - Qwen/Qwen3-0.6B
45           - --served-model-name
46           - Qwen/Qwen3-0.6B
47           - --extra-engine-args
48           - /workspace/examples/backends/trtllm/engine_configs/qwen3/agg.yaml
49           command:
50           - python3
51           - -m
52           - dynamo.trtllm
53           envFrom:
54           - secretRef:
55               name: hf-token-secret
56           image: my-registry/tensorrtllm-runtime:my-tag
57           name: main
58           resources:
59             limits:
60               nvidia.com/gpu: "1"
61           # Mount the ConfigMap key as the engine args file without masking /workspace.
62           volumeMounts:
63           - mountPath: /workspace/examples/backends/trtllm/engine_configs/qwen3/agg.yaml
64             name: nvidia-config
65             readOnly: true
66             subPath: agg.yaml
67           workingDir: /workspace/examples/backends/trtllm
68         # Declare the ConfigMap as a pod volume.
69         volumes:
70         - configMap:
71             name: nvidia-config
72           name: nvidia-config
73     replicas: 1
74     type: worker

Disaggregated

disagg.yaml · Baseline disaggregated prefill/decode

disagg.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: trtllm-disagg
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - image: my-registry/tensorrtllm-runtime:my-tag
15           name: main
16     replicas: 1
17     type: frontend
18   - name: decode
19     podTemplate:
20       spec:
21         containers:
22         - args:
23           - --model-path
24           - Qwen/Qwen3-0.6B
25           - --served-model-name
26           - Qwen/Qwen3-0.6B
27           - --extra-engine-args
28           - ./examples/backends/trtllm/engine_configs/qwen3/decode.yaml
29           - --disaggregation-mode
30           - decode
31           command:
32           - python3
33           - -m
34           - dynamo.trtllm
35           envFrom:
36           - secretRef:
37               name: hf-token-secret
38           image: my-registry/tensorrtllm-runtime:my-tag
39           name: main
40           resources:
41             limits:
42               nvidia.com/gpu: "1"
43           workingDir: /workspace/
44     replicas: 1
45     type: decode
46   - name: prefill
47     podTemplate:
48       spec:
49         containers:
50         - args:
51           - --model-path
52           - Qwen/Qwen3-0.6B
53           - --served-model-name
54           - Qwen/Qwen3-0.6B
55           - --extra-engine-args
56           - ./examples/backends/trtllm/engine_configs/qwen3/prefill.yaml
57           - --disaggregation-mode
58           - prefill
59           command:
60           - python3
61           - -m
62           - dynamo.trtllm
63           envFrom:
64           - secretRef:
65               name: hf-token-secret
66           image: my-registry/tensorrtllm-runtime:my-tag
67           name: main
68           resources:
69             limits:
70               nvidia.com/gpu: "1"
71           workingDir: /workspace/
72     replicas: 1
73     type: prefill

disagg_router.yaml · Disaggregated with KV-aware routing

disagg_router.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: trtllm-v1-disagg-router
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - env:
15           - name: DYN_ROUTER_MODE
16             value: kv
17           image: my-registry/tensorrtllm-runtime:my-tag
18           name: main
19     replicas: 1
20     type: frontend
21   - name: decode
22     podTemplate:
23       spec:
24         containers:
25         - args:
26           - --model-path
27           - Qwen/Qwen3-0.6B
28           - --served-model-name
29           - Qwen/Qwen3-0.6B
30           - --extra-engine-args
31           - ./examples/backends/trtllm/engine_configs/qwen3/decode.yaml
32           - --disaggregation-mode
33           - decode
34           command:
35           - python3
36           - -m
37           - dynamo.trtllm
38           envFrom:
39           - secretRef:
40               name: hf-token-secret
41           image: my-registry/tensorrtllm-runtime:my-tag
42           name: main
43           resources:
44             limits:
45               nvidia.com/gpu: "1"
46           workingDir: /workspace/
47     replicas: 2
48     type: decode
49   - name: prefill
50     podTemplate:
51       spec:
52         containers:
53         - args:
54           - --model-path
55           - Qwen/Qwen3-0.6B
56           - --served-model-name
57           - Qwen/Qwen3-0.6B
58           - --extra-engine-args
59           - ./examples/backends/trtllm/engine_configs/qwen3/prefill.yaml
60           - --disaggregation-mode
61           - prefill
62           - --publish-events-and-metrics
63           command:
64           - python3
65           - -m
66           - dynamo.trtllm
67           envFrom:
68           - secretRef:
69               name: hf-token-secret
70           image: my-registry/tensorrtllm-runtime:my-tag
71           name: main
72           resources:
73             limits:
74               nvidia.com/gpu: "1"
75           workingDir: /workspace/
76     replicas: 2
77     type: prefill

disagg_planner.yaml · Disaggregated with Dynamo Planner autoscaling

disagg_planner.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: trtllm-disagg-planner
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - args:
15           - -m
16           - dynamo.frontend
17           - --http-port
18           - "8000"
19           - --kv-cache-block-size
20           - "128"
21           - --router-mode
22           - kv
23           - --router-kv-overlap-score-credit
24           - "0.0"
25           - --router-temperature
26           - "0.0"
27           - --no-kv-events
28           command:
29           - python3
30           image: my-registry/tensorrtllm-runtime:my-tag
31           name: main
32           workingDir: /workspace/examples/backends/trtllm
33     replicas: 1
34     type: frontend
35   - name: Planner
36     podTemplate:
37       spec:
38         containers:
39         - args:
40           - -m
41           - dynamo.planner
42           - --config
43           - '{"environment": "kubernetes", "backend": "trtllm", "optimization_target": "sla",
44             "enable_throughput_scaling": true, "enable_load_scaling": true, "pre_deployment_sweeping_mode":
45             "none", "throughput_adjustment_interval_seconds": 60, "load_adjustment_interval_seconds": 5}'
46           command:
47           - python3
48           envFrom:
49           - secretRef:
50               name: hf-token-secret
51           # Planner image selection:
52           #   Dynamo >= 1.1.0: use the dedicated planner image
53           #     <registry>/dynamo-planner:<version>
54           #     (backend runtime images no longer ship planner runtime deps
55           #     such as kubernetes_asyncio, pmdarima, prophet, aiconfigurator).
56           #   Dynamo <  1.1.0: use the backend runtime image
57           #     <registry>/tensorrtllm-runtime:<version>.
58           image: my-registry/dynamo-planner:my-tag
59           name: main
60           ports:
61           - containerPort: 9085
62             name: metrics
63     replicas: 1
64     type: planner
65   - name: decode
66     podTemplate:
67       spec:
68         containers:
69         - args:
70           - -m
71           - dynamo.trtllm
72           - --model-path
73           - Qwen/Qwen3-0.6B
74           - --served-model-name
75           - Qwen/Qwen3-0.6B
76           - --extra-engine-args
77           - ./examples/backends/trtllm/engine_configs/qwen3/decode.yaml
78           - --disaggregation-mode
79           - decode
80           command:
81           - python3
82           envFrom:
83           - secretRef:
84               name: hf-token-secret
85           image: my-registry/tensorrtllm-runtime:my-tag
86           livenessProbe:
87             failureThreshold: 1
88             httpGet:
89               path: /live
90               port: 9090
91             periodSeconds: 5
92             timeoutSeconds: 30
93           name: main
94           readinessProbe:
95             failureThreshold: 60
96             httpGet:
97               path: /health
98               port: 9090
99             periodSeconds: 10
100             timeoutSeconds: 30
101           resources:
102             limits:
103               nvidia.com/gpu: "1"
104           workingDir: /workspace/
105         terminationGracePeriodSeconds: 600
106     replicas: 1
107     type: decode
108   - name: prefill
109     podTemplate:
110       spec:
111         containers:
112         - args:
113           - -m
114           - dynamo.trtllm
115           - --model-path
116           - Qwen/Qwen3-0.6B
117           - --served-model-name
118           - Qwen/Qwen3-0.6B
119           - --extra-engine-args
120           - ./examples/backends/trtllm/engine_configs/qwen3/prefill.yaml
121           - --disaggregation-mode
122           - prefill
123           command:
124           - python3
125           envFrom:
126           - secretRef:
127               name: hf-token-secret
128           image: my-registry/tensorrtllm-runtime:my-tag
129           name: main
130           resources:
131             limits:
132               nvidia.com/gpu: "1"
133           workingDir: /workspace/
134         terminationGracePeriodSeconds: 600
135     replicas: 1
136     type: prefill

disagg-multinode.yaml · Disaggregated across multiple nodes (bundles a ConfigMap and PersistentVolumeClaim)

disagg-multinode.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 apiVersion: v1
4 kind: ConfigMap
5 metadata:
6   name: nvidia-config
7 data:
8   prefill.yaml: |
9     # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
10     # SPDX-License-Identifier: Apache-2.0
11     #
12     # Licensed under the Apache License, Version 2.0 (the "License");
13     # you may not use this file except in compliance with the License.
14     # You may obtain a copy of the License at
15     #
16     # http://www.apache.org/licenses/LICENSE-2.0
17     #
18     # Unless required by applicable law or agreed to in writing, software
19     # distributed under the License is distributed on an "AS IS" BASIS,
20     # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
21     # See the License for the specific language governing permissions and
22     # limitations under the License.
23     tensor_parallel_size: 8
24     moe_expert_parallel_size: 1
25     enable_attention_dp: false
26     max_num_tokens: 8192
27     trust_remote_code: true
28     backend: pytorch
29     enable_chunked_prefill: true
30     # Overlap scheduler not currently supported in prefill only workers.
31     disable_overlap_scheduler: true
32     kv_cache_config:
33       free_gpu_memory_fraction: 0.80
34     cache_transceiver_config:
35       backend: DEFAULT
36 
37   decode.yaml: |
38     # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
39     # SPDX-License-Identifier: Apache-2.0
40     #
41     # Licensed under the Apache License, Version 2.0 (the "License");
42     # you may not use this file except in compliance with the License.
43     # You may obtain a copy of the License at
44     #
45     # http://www.apache.org/licenses/LICENSE-2.0
46     #
47     # Unless required by applicable law or agreed to in writing, software
48     # distributed under the License is distributed on an "AS IS" BASIS,
49     # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
50     # See the License for the specific language governing permissions and
51     # limitations under the License.
52     tensor_parallel_size: 8
53     moe_expert_parallel_size: 1
54     enable_attention_dp: false
55     max_num_tokens: 8192
56     trust_remote_code: true
57     backend: pytorch
58     enable_chunked_prefill: true
59     disable_overlap_scheduler: false
60     kv_cache_config:
61       free_gpu_memory_fraction: 0.80
62     cache_transceiver_config:
63       backend: DEFAULT
64 ---
65 
66 apiVersion: v1
67 kind: PersistentVolumeClaim
68 metadata:
69   name: models
70 spec:
71   accessModes:
72     - ReadWriteMany
73   resources:
74     requests:
75       storage: 100Gi
76 ---
77 apiVersion: nvidia.com/v1beta1
78 kind: DynamoGraphDeployment
79 metadata:
80   name: trtllm-disagg-tp8
81 spec:
82   backendFramework: trtllm
83   components:
84   - name: Frontend
85     podTemplate:
86       spec:
87         containers:
88         - args:
89           - --http-port
90           - "8000"
91           command:
92           - python3
93           - -m
94           - dynamo.frontend
95           image: my-registry/tensorrtllm-runtime:my-tag
96           name: main
97           volumeMounts:
98           - mountPath: /models
99             name: models
100           workingDir: /workspace/examples/backends/trtllm
101         volumes:
102         - name: models
103           persistentVolumeClaim:
104             claimName: models
105     replicas: 1
106     type: frontend
107   - multinode:
108       nodeCount: 2
109     name: decode
110     podTemplate:
111       spec:
112         containers:
113         - args:
114           - --model-path
115           - Qwen/Qwen3-0.6B
116           - --served-model-name
117           - Qwen/Qwen3-0.6B
118           - --extra-engine-args
119           - /workspace/engine_configs/decode.yaml
120           - --disaggregation-mode
121           - decode
122           command:
123           - python3
124           - -m
125           - dynamo.trtllm
126           envFrom:
127           - secretRef:
128               name: hf-token-secret
129           image: my-registry/tensorrtllm-runtime:my-tag
130           name: main
131           resources:
132             limits:
133               nvidia.com/gpu: "4"
134           volumeMounts:
135           - mountPath: /models
136             name: models
137           - mountPath: /workspace/engine_configs
138             name: nvidia-config
139             readOnly: true
140           workingDir: /workspace/
141         volumes:
142         - name: models
143           persistentVolumeClaim:
144             claimName: models
145         - configMap:
146             name: nvidia-config
147           name: nvidia-config
148     replicas: 1
149     type: decode
150   - multinode:
151       nodeCount: 2
152     name: prefill
153     podTemplate:
154       spec:
155         containers:
156         - args:
157           - --model-path
158           - Qwen/Qwen3-0.6B
159           - --served-model-name
160           - Qwen/Qwen3-0.6B
161           - --extra-engine-args
162           - /workspace/engine_configs/prefill.yaml
163           - --disaggregation-mode
164           - prefill
165           command:
166           - python3
167           - -m
168           - dynamo.trtllm
169           envFrom:
170           - secretRef:
171               name: hf-token-secret
172           image: my-registry/tensorrtllm-runtime:my-tag
173           name: main
174           resources:
175             limits:
176               nvidia.com/gpu: "4"
177           volumeMounts:
178           - mountPath: /models
179             name: models
180           - mountPath: /workspace/engine_configs
181             name: nvidia-config
182             readOnly: true
183           workingDir: /workspace/
184         volumes:
185         - name: models
186           persistentVolumeClaim:
187             claimName: models
188         - configMap:
189             name: nvidia-config
190           name: nvidia-config
191     replicas: 1
192     type: prefill
193   env:
194   - name: OMPI_ALLOW_RUN_AS_ROOT
195     value: "1"
196   - name: OMPI_ALLOW_RUN_AS_ROOT_CONFIRM
197     value: "1"
198   - name: HF_HOME
199     value: /models

Source

All templates live in examples/backends/trtllm/deploy/. For local launch commands, see TensorRT-LLM Local Deployment Examples.