SGLang Deployment Templates | NVIDIA Dynamo Documentation

Copy-paste DynamoGraphDeployment (nvidia.com/v1beta1) manifests for the SGLang backend, grouped by topology. Each manifest is embedded from examples/backends/sglang/deploy/ — open an entry, use the copy button, then set your image tag and hf-token-secret before applying.

Apply any template with:

$ kubectl apply -f agg.yaml

Aggregated

agg.yaml · Baseline aggregated serving

agg.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-agg
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - image: my-registry/sglang-runtime:my-tag
15           name: main
16     replicas: 1
17     type: frontend
18   - name: decode
19     podTemplate:
20       spec:
21         containers:
22         - args:
23           - --model-path
24           - Qwen/Qwen3-0.6B
25           - --served-model-name
26           - Qwen/Qwen3-0.6B
27           - --page-size
28           - "16"
29           - --tp
30           - "1"
31           - --trust-remote-code
32           - --skip-tokenizer-init
33           command:
34           - python3
35           - -m
36           - dynamo.sglang
37           envFrom:
38           - secretRef:
39               name: hf-token-secret
40           image: my-registry/sglang-runtime:my-tag
41           name: main
42           resources:
43             limits:
44               nvidia.com/gpu: "1"
45             requests:
46               # Increase this value for larger models.
47               ephemeral-storage: 2Gi
48           workingDir: /workspace/examples/backends/sglang
49     replicas: 1
50     type: worker

agg_router.yaml · Aggregated with KV-aware routing

agg_router.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-agg-router
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - env:
15           - name: DYN_ROUTER_MODE
16             value: kv
17           image: my-registry/sglang-runtime:my-tag
18           name: main
19     replicas: 1
20     type: frontend
21   - name: decode
22     podTemplate:
23       spec:
24         containers:
25         - args:
26           - --model-path
27           - Qwen/Qwen3-0.6B
28           - --served-model-name
29           - Qwen/Qwen3-0.6B
30           - --page-size
31           - "16"
32           - --tp
33           - "1"
34           - --trust-remote-code
35           - --skip-tokenizer-init
36           - --kv-events-config
37           - '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}'
38           command:
39           - python3
40           - -m
41           - dynamo.sglang
42           envFrom:
43           - secretRef:
44               name: hf-token-secret
45           image: my-registry/sglang-runtime:my-tag
46           name: main
47           resources:
48             limits:
49               nvidia.com/gpu: "1"
50             requests:
51               # Increase this value for larger models.
52               ephemeral-storage: 2Gi
53           workingDir: /workspace/examples/backends/sglang
54     replicas: 1
55     type: worker

agg_gms.yaml · Aggregated with GPU Memory Service sidecar

agg_gms.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 #
4 # GPU Memory Service (GMS) sidecar example.
5 #
6 # The operator injects a GMS sidecar init container that provides shared GPU
7 # memory access via DRA (Dynamic Resource Allocation). The sidecar runs two GMS
8 # processes per GPU (weights + kv_cache) and communicates with the main container
9 # over UDS sockets on a shared emptyDir volume.
10 #
11 # Requires Kubernetes 1.34+ with DRA v1 enabled and the NVIDIA GPU DRA driver installed.
12 
13 apiVersion: nvidia.com/v1beta1
14 kind: DynamoGraphDeployment
15 metadata:
16   name: sglang-agg-gms
17 spec:
18   components:
19   - name: Frontend
20     podTemplate:
21       spec:
22         containers:
23         - image: my-registry/sglang-runtime:my-tag
24           name: main
25     replicas: 1
26     type: frontend
27   - experimental:
28       gpuMemoryService: {}
29     name: decode
30     podTemplate:
31       spec:
32         containers:
33         - args:
34           - --model-path
35           - Qwen/Qwen3-0.6B
36           - --served-model-name
37           - Qwen/Qwen3-0.6B
38           - --page-size
39           - "16"
40           - --tp
41           - "1"
42           - --trust-remote-code
43           - --skip-tokenizer-init
44           - --load-format
45           - gms
46           command:
47           - python3
48           - -m
49           - dynamo.sglang
50           envFrom:
51           - secretRef:
52               name: hf-token-secret
53           image: my-registry/sglang-runtime:my-tag
54           name: main
55           resources:
56             limits:
57               nvidia.com/gpu: "1"
58             requests:
59               # Increase this value for larger models.
60               ephemeral-storage: 2Gi
61           workingDir: /workspace/examples/backends/sglang
62     replicas: 1
63     type: worker

agg_logging.yaml · Aggregated with structured logging

agg_logging.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-agg-logging
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - image: my-registry/sglang-runtime:my-tag
15           name: main
16     replicas: 1
17     type: frontend
18   - name: decode
19     podTemplate:
20       spec:
21         containers:
22         - args:
23           - --model-path
24           - Qwen/Qwen3-0.6B
25           - --served-model-name
26           - Qwen/Qwen3-0.6B
27           - --page-size
28           - "16"
29           - --tp
30           - "1"
31           - --trust-remote-code
32           - --skip-tokenizer-init
33           command:
34           - python3
35           - -m
36           - dynamo.sglang
37           envFrom:
38           - secretRef:
39               name: hf-token-secret
40           image: my-registry/sglang-runtime:my-tag
41           name: main
42           resources:
43             limits:
44               nvidia.com/gpu: "1"
45           workingDir: /workspace/examples/backends/sglang
46     replicas: 1
47     type: worker
48   env:
49   - name: DYN_LOGGING_JSONL
50     value: "1"

Disaggregated

disagg.yaml · Baseline disaggregated prefill/decode

disagg.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-disagg
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - image: my-registry/sglang-runtime:my-tag
15           name: main
16     replicas: 1
17     type: frontend
18   - name: decode
19     podTemplate:
20       spec:
21         containers:
22         - args:
23           - --model-path
24           - Qwen/Qwen3-0.6B
25           - --served-model-name
26           - Qwen/Qwen3-0.6B
27           - --page-size
28           - "16"
29           - --tp
30           - "1"
31           - --trust-remote-code
32           - --skip-tokenizer-init
33           - --disaggregation-mode
34           - decode
35           - --disaggregation-transfer-backend
36           - nixl
37           - --disaggregation-bootstrap-port
38           - "12345"
39           - --host
40           - 0.0.0.0
41           command:
42           - python3
43           - -m
44           - dynamo.sglang
45           envFrom:
46           - secretRef:
47               name: hf-token-secret
48           image: my-registry/sglang-runtime:my-tag
49           name: main
50           resources:
51             limits:
52               nvidia.com/gpu: "1"
53           workingDir: /workspace/examples/backends/sglang
54     replicas: 1
55     type: decode
56   - name: prefill
57     podTemplate:
58       spec:
59         containers:
60         - args:
61           - --model-path
62           - Qwen/Qwen3-0.6B
63           - --served-model-name
64           - Qwen/Qwen3-0.6B
65           - --page-size
66           - "16"
67           - --tp
68           - "1"
69           - --trust-remote-code
70           - --skip-tokenizer-init
71           - --disaggregation-mode
72           - prefill
73           - --disaggregation-transfer-backend
74           - nixl
75           - --disaggregation-bootstrap-port
76           - "12345"
77           - --host
78           - 0.0.0.0
79           command:
80           - python3
81           - -m
82           - dynamo.sglang
83           envFrom:
84           - secretRef:
85               name: hf-token-secret
86           image: my-registry/sglang-runtime:my-tag
87           name: main
88           resources:
89             limits:
90               nvidia.com/gpu: "1"
91           workingDir: /workspace/examples/backends/sglang
92     replicas: 1
93     type: prefill

disagg_planner.yaml · Disaggregated with Dynamo Planner autoscaling

disagg_planner.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-disagg-planner
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - image: my-registry/sglang-runtime:my-tag
15           name: main
16     replicas: 1
17     type: frontend
18   - name: Planner
19     podTemplate:
20       spec:
21         containers:
22         - args:
23           - --config
24           - '{"environment": "kubernetes", "backend": "sglang", "optimization_target": "sla",
25             "enable_throughput_scaling": true, "enable_load_scaling": true, "pre_deployment_sweeping_mode":
26             "none", "throughput_adjustment_interval_seconds": 60, "load_adjustment_interval_seconds": 5}'
27           command:
28           - python3
29           - -m
30           - dynamo.planner
31           envFrom:
32           - secretRef:
33               name: hf-token-secret
34           # Planner image selection:
35           #   Dynamo >= 1.1.0: use the dedicated planner image
36           #     <registry>/dynamo-planner:<version>
37           #     (backend runtime images no longer ship planner runtime deps
38           #     such as kubernetes_asyncio, pmdarima, prophet, aiconfigurator).
39           #   Dynamo <  1.1.0: use the backend runtime image
40           #     <registry>/sglang-runtime:<version>.
41           image: my-registry/dynamo-planner:my-tag
42           name: main
43     replicas: 1
44     type: planner
45   - name: decode
46     podTemplate:
47       spec:
48         containers:
49         - args:
50           - --model-path
51           - Qwen/Qwen3-0.6B
52           - --served-model-name
53           - Qwen/Qwen3-0.6B
54           - --page-size
55           - "16"
56           - --tp
57           - "1"
58           - --trust-remote-code
59           - --skip-tokenizer-init
60           - --disaggregation-mode
61           - decode
62           - --disaggregation-transfer-backend
63           - nixl
64           - --disaggregation-bootstrap-port
65           - "12345"
66           - --host
67           - 0.0.0.0
68           command:
69           - python3
70           - -m
71           - dynamo.sglang
72           envFrom:
73           - secretRef:
74               name: hf-token-secret
75           image: my-registry/sglang-runtime:my-tag
76           name: main
77           resources:
78             limits:
79               nvidia.com/gpu: "1"
80           workingDir: /workspace/examples/backends/sglang
81     replicas: 2
82     type: decode
83   - name: prefill
84     podTemplate:
85       spec:
86         containers:
87         - args:
88           - --model-path
89           - Qwen/Qwen3-0.6B
90           - --served-model-name
91           - Qwen/Qwen3-0.6B
92           - --page-size
93           - "16"
94           - --tp
95           - "1"
96           - --trust-remote-code
97           - --skip-tokenizer-init
98           - --disaggregation-mode
99           - prefill
100           - --disaggregation-transfer-backend
101           - nixl
102           - --disaggregation-bootstrap-port
103           - "12345"
104           - --host
105           - 0.0.0.0
106           command:
107           - python3
108           - -m
109           - dynamo.sglang
110           envFrom:
111           - secretRef:
112               name: hf-token-secret
113           image: my-registry/sglang-runtime:my-tag
114           name: main
115           resources:
116             limits:
117               nvidia.com/gpu: "1"
118           workingDir: /workspace/examples/backends/sglang
119     replicas: 2
120     type: prefill

disagg-multinode.yaml · Disaggregated across multiple nodes

disagg-multinode.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-disagg-multinode
8 spec:
9   backendFramework: sglang
10   components:
11   - name: Frontend
12     podTemplate:
13       spec:
14         containers:
15         - image: my-registry/sglang-runtime:my-tag
16           name: main
17     replicas: 1
18     type: frontend
19   - multinode:
20       nodeCount: 2
21     name: decode
22     podTemplate:
23       spec:
24         containers:
25         - args:
26           - --model-path
27           - Qwen/Qwen3-0.6B
28           - --served-model-name
29           - Qwen/Qwen3-0.6B
30           - --tp-size
31           - "8"
32           - --trust-remote-code
33           - --skip-tokenizer-init
34           - --disaggregation-mode
35           - decode
36           - --disaggregation-transfer-backend
37           - nixl
38           - --disaggregation-bootstrap-port
39           - "30001"
40           - --host
41           - 0.0.0.0
42           - --mem-fraction-static
43           - "0.82"
44           command:
45           - python3
46           - -m
47           - dynamo.sglang
48           envFrom:
49           - secretRef:
50               name: hf-token-secret
51           image: my-registry/sglang-runtime:my-tag
52           name: main
53           resources:
54             limits:
55               nvidia.com/gpu: "4"
56           workingDir: /workspace/examples/backends/sglang
57     replicas: 1
58     type: decode
59   - multinode:
60       nodeCount: 2
61     name: prefill
62     podTemplate:
63       spec:
64         containers:
65         - args:
66           - --model-path
67           - Qwen/Qwen3-0.6B
68           - --served-model-name
69           - Qwen/Qwen3-0.6B
70           - --tp-size
71           - "8"
72           - --trust-remote-code
73           - --skip-tokenizer-init
74           - --disaggregation-mode
75           - prefill
76           - --disaggregation-transfer-backend
77           - nixl
78           - --disaggregation-bootstrap-port
79           - "30001"
80           - --mem-fraction-static
81           - "0.82"
82           - --host
83           - 0.0.0.0
84           command:
85           - python3
86           - -m
87           - dynamo.sglang
88           envFrom:
89           - secretRef:
90               name: hf-token-secret
91           image: my-registry/sglang-runtime:my-tag
92           name: main
93           resources:
94             limits:
95               nvidia.com/gpu: "4"
96           workingDir: /workspace/examples/backends/sglang
97     replicas: 1
98     type: prefill
99   env:
100   - name: HF_TOKEN
101     valueFrom:
102       secretKeyRef:
103         key: HF_TOKEN
104         name: hf-token-secret
105   - name: GLOO_SOCKET_IFNAME
106     value: eth0

Source

All templates live in examples/backends/sglang/deploy/. For local launch commands, see SGLang Local Deployment Examples.