SGLang Deployment Templates | NVIDIA Dynamo Documentation

Copy-paste DynamoGraphDeployment (nvidia.com/v1beta1) manifests for the SGLang backend, grouped by topology. Each manifest is embedded from examples/backends/sglang/deploy/ — open an entry, use the copy button, then set your image tag and hf-token-secret before applying.

Apply any template with:

$ kubectl apply -f agg.yaml

Aggregated

agg.yaml · Baseline aggregated serving

agg.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-agg
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - image: my-registry/sglang-runtime:my-tag
15           name: main
16     replicas: 1
17     type: frontend
18   - name: decode
19     podTemplate:
20       spec:
21         containers:
22         - args:
23           - --model-path
24           - Qwen/Qwen3-0.6B
25           - --served-model-name
26           - Qwen/Qwen3-0.6B
27           - --page-size
28           - "16"
29           - --tp
30           - "1"
31           - --trust-remote-code
32           command:
33           - python3
34           - -m
35           - dynamo.sglang
36           envFrom:
37           - secretRef:
38               name: hf-token-secret
39           image: my-registry/sglang-runtime:my-tag
40           name: main
41           resources:
42             limits:
43               nvidia.com/gpu: "1"
44             requests:
45               # Increase this value for larger models.
46               ephemeral-storage: 2Gi
47           workingDir: /workspace/examples/backends/sglang
48     replicas: 1
49     type: worker

agg_router.yaml · Aggregated with KV-aware routing

agg_router.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-agg-router
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - env:
15           - name: DYN_ROUTER_MODE
16             value: kv
17           image: my-registry/sglang-runtime:my-tag
18           name: main
19     replicas: 1
20     type: frontend
21   - name: decode
22     podTemplate:
23       spec:
24         containers:
25         - args:
26           - --model-path
27           - Qwen/Qwen3-0.6B
28           - --served-model-name
29           - Qwen/Qwen3-0.6B
30           - --page-size
31           - "16"
32           - --tp
33           - "1"
34           - --trust-remote-code
35           - --kv-events-config
36           - '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}'
37           command:
38           - python3
39           - -m
40           - dynamo.sglang
41           envFrom:
42           - secretRef:
43               name: hf-token-secret
44           image: my-registry/sglang-runtime:my-tag
45           name: main
46           resources:
47             limits:
48               nvidia.com/gpu: "1"
49             requests:
50               # Increase this value for larger models.
51               ephemeral-storage: 2Gi
52           workingDir: /workspace/examples/backends/sglang
53     replicas: 1
54     type: worker

agg_gms.yaml · Aggregated with GPU Memory Service sidecar

agg_gms.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 #
4 # GPU Memory Service (GMS) sidecar example.
5 #
6 # The operator injects a GMS sidecar init container that provides shared GPU
7 # memory access via DRA (Dynamic Resource Allocation). The sidecar runs two GMS
8 # processes per GPU (weights + kv_cache) and communicates with the main container
9 # over UDS sockets on a shared emptyDir volume.
10 #
11 # Requires Kubernetes 1.34+ with DRA v1 enabled and the NVIDIA GPU DRA driver installed.
12 
13 apiVersion: nvidia.com/v1beta1
14 kind: DynamoGraphDeployment
15 metadata:
16   name: sglang-agg-gms
17 spec:
18   components:
19   - name: Frontend
20     podTemplate:
21       spec:
22         containers:
23         - image: my-registry/sglang-runtime:my-tag
24           name: main
25     replicas: 1
26     type: frontend
27   - experimental:
28       gpuMemoryService: {}
29     name: decode
30     podTemplate:
31       spec:
32         containers:
33         - args:
34           - --model-path
35           - Qwen/Qwen3-0.6B
36           - --served-model-name
37           - Qwen/Qwen3-0.6B
38           - --page-size
39           - "16"
40           - --tp
41           - "1"
42           - --trust-remote-code
43           - --load-format
44           - gms
45           command:
46           - python3
47           - -m
48           - dynamo.sglang
49           envFrom:
50           - secretRef:
51               name: hf-token-secret
52           image: my-registry/sglang-runtime:my-tag
53           name: main
54           resources:
55             limits:
56               nvidia.com/gpu: "1"
57             requests:
58               # Increase this value for larger models.
59               ephemeral-storage: 2Gi
60           workingDir: /workspace/examples/backends/sglang
61     replicas: 1
62     type: worker

agg_logging.yaml · Aggregated with structured logging

agg_logging.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-agg-logging
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - image: my-registry/sglang-runtime:my-tag
15           name: main
16     replicas: 1
17     type: frontend
18   - name: decode
19     podTemplate:
20       spec:
21         containers:
22         - args:
23           - --model-path
24           - Qwen/Qwen3-0.6B
25           - --served-model-name
26           - Qwen/Qwen3-0.6B
27           - --page-size
28           - "16"
29           - --tp
30           - "1"
31           - --trust-remote-code
32           command:
33           - python3
34           - -m
35           - dynamo.sglang
36           envFrom:
37           - secretRef:
38               name: hf-token-secret
39           image: my-registry/sglang-runtime:my-tag
40           name: main
41           resources:
42             limits:
43               nvidia.com/gpu: "1"
44           workingDir: /workspace/examples/backends/sglang
45     replicas: 1
46     type: worker
47   env:
48   - name: DYN_LOGGING_JSONL
49     value: "1"

Disaggregated

disagg.yaml · Baseline disaggregated prefill/decode

disagg.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-disagg
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - image: my-registry/sglang-runtime:my-tag
15           name: main
16     replicas: 1
17     type: frontend
18   - name: decode
19     podTemplate:
20       spec:
21         containers:
22         - args:
23           - --model-path
24           - Qwen/Qwen3-0.6B
25           - --served-model-name
26           - Qwen/Qwen3-0.6B
27           - --page-size
28           - "16"
29           - --tp
30           - "1"
31           - --trust-remote-code
32           - --disaggregation-mode
33           - decode
34           - --disaggregation-transfer-backend
35           - nixl
36           - --disaggregation-bootstrap-port
37           - "12345"
38           - --host
39           - 0.0.0.0
40           command:
41           - python3
42           - -m
43           - dynamo.sglang
44           envFrom:
45           - secretRef:
46               name: hf-token-secret
47           image: my-registry/sglang-runtime:my-tag
48           name: main
49           resources:
50             limits:
51               nvidia.com/gpu: "1"
52           workingDir: /workspace/examples/backends/sglang
53     replicas: 1
54     type: decode
55   - name: prefill
56     podTemplate:
57       spec:
58         containers:
59         - args:
60           - --model-path
61           - Qwen/Qwen3-0.6B
62           - --served-model-name
63           - Qwen/Qwen3-0.6B
64           - --page-size
65           - "16"
66           - --tp
67           - "1"
68           - --trust-remote-code
69           - --disaggregation-mode
70           - prefill
71           - --disaggregation-transfer-backend
72           - nixl
73           - --disaggregation-bootstrap-port
74           - "12345"
75           - --host
76           - 0.0.0.0
77           command:
78           - python3
79           - -m
80           - dynamo.sglang
81           envFrom:
82           - secretRef:
83               name: hf-token-secret
84           image: my-registry/sglang-runtime:my-tag
85           name: main
86           resources:
87             limits:
88               nvidia.com/gpu: "1"
89           workingDir: /workspace/examples/backends/sglang
90     replicas: 1
91     type: prefill

disagg_planner.yaml · Disaggregated with Dynamo Planner autoscaling

disagg_planner.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-disagg-planner
8 spec:
9   components:
10   - name: Frontend
11     podTemplate:
12       spec:
13         containers:
14         - image: my-registry/sglang-runtime:my-tag
15           name: main
16     replicas: 1
17     type: frontend
18   - name: Planner
19     podTemplate:
20       spec:
21         containers:
22         - args:
23           - --config
24           - '{"environment": "kubernetes", "backend": "sglang", "optimization_target": "sla",
25             "enable_throughput_scaling": true, "enable_load_scaling": true, "pre_deployment_sweeping_mode":
26             "none", "throughput_adjustment_interval_seconds": 60, "load_adjustment_interval_seconds": 5}'
27           command:
28           - python3
29           - -m
30           - dynamo.planner
31           envFrom:
32           - secretRef:
33               name: hf-token-secret
34           # Planner image selection:
35           #   Dynamo >= 1.1.0: use the dedicated planner image
36           #     <registry>/dynamo-planner:<version>
37           #     (backend runtime images no longer ship planner runtime deps
38           #     such as kubernetes_asyncio, pmdarima, prophet, aiconfigurator).
39           #   Dynamo <  1.1.0: use the backend runtime image
40           #     <registry>/sglang-runtime:<version>.
41           image: my-registry/dynamo-planner:my-tag
42           name: main
43     replicas: 1
44     type: planner
45   - name: decode
46     podTemplate:
47       spec:
48         containers:
49         - args:
50           - --model-path
51           - Qwen/Qwen3-0.6B
52           - --served-model-name
53           - Qwen/Qwen3-0.6B
54           - --page-size
55           - "16"
56           - --tp
57           - "1"
58           - --trust-remote-code
59           - --disaggregation-mode
60           - decode
61           - --disaggregation-transfer-backend
62           - nixl
63           - --disaggregation-bootstrap-port
64           - "12345"
65           - --host
66           - 0.0.0.0
67           command:
68           - python3
69           - -m
70           - dynamo.sglang
71           envFrom:
72           - secretRef:
73               name: hf-token-secret
74           image: my-registry/sglang-runtime:my-tag
75           name: main
76           resources:
77             limits:
78               nvidia.com/gpu: "1"
79           workingDir: /workspace/examples/backends/sglang
80     replicas: 2
81     type: decode
82   - name: prefill
83     podTemplate:
84       spec:
85         containers:
86         - args:
87           - --model-path
88           - Qwen/Qwen3-0.6B
89           - --served-model-name
90           - Qwen/Qwen3-0.6B
91           - --page-size
92           - "16"
93           - --tp
94           - "1"
95           - --trust-remote-code
96           - --disaggregation-mode
97           - prefill
98           - --disaggregation-transfer-backend
99           - nixl
100           - --disaggregation-bootstrap-port
101           - "12345"
102           - --host
103           - 0.0.0.0
104           command:
105           - python3
106           - -m
107           - dynamo.sglang
108           envFrom:
109           - secretRef:
110               name: hf-token-secret
111           image: my-registry/sglang-runtime:my-tag
112           name: main
113           resources:
114             limits:
115               nvidia.com/gpu: "1"
116           workingDir: /workspace/examples/backends/sglang
117     replicas: 2
118     type: prefill

disagg-multinode.yaml · Disaggregated across multiple nodes

disagg-multinode.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1beta1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-disagg-multinode
8 spec:
9   backendFramework: sglang
10   components:
11   - name: Frontend
12     podTemplate:
13       spec:
14         containers:
15         - image: my-registry/sglang-runtime:my-tag
16           name: main
17     replicas: 1
18     type: frontend
19   - multinode:
20       nodeCount: 2
21     name: decode
22     podTemplate:
23       spec:
24         containers:
25         - args:
26           - --model-path
27           - Qwen/Qwen3-0.6B
28           - --served-model-name
29           - Qwen/Qwen3-0.6B
30           - --tp-size
31           - "8"
32           - --trust-remote-code
33           - --disaggregation-mode
34           - decode
35           - --disaggregation-transfer-backend
36           - nixl
37           - --disaggregation-bootstrap-port
38           - "30001"
39           - --host
40           - 0.0.0.0
41           - --mem-fraction-static
42           - "0.82"
43           command:
44           - python3
45           - -m
46           - dynamo.sglang
47           envFrom:
48           - secretRef:
49               name: hf-token-secret
50           image: my-registry/sglang-runtime:my-tag
51           name: main
52           resources:
53             limits:
54               nvidia.com/gpu: "4"
55           workingDir: /workspace/examples/backends/sglang
56     replicas: 1
57     type: decode
58   - multinode:
59       nodeCount: 2
60     name: prefill
61     podTemplate:
62       spec:
63         containers:
64         - args:
65           - --model-path
66           - Qwen/Qwen3-0.6B
67           - --served-model-name
68           - Qwen/Qwen3-0.6B
69           - --tp-size
70           - "8"
71           - --trust-remote-code
72           - --disaggregation-mode
73           - prefill
74           - --disaggregation-transfer-backend
75           - nixl
76           - --disaggregation-bootstrap-port
77           - "30001"
78           - --mem-fraction-static
79           - "0.82"
80           - --host
81           - 0.0.0.0
82           command:
83           - python3
84           - -m
85           - dynamo.sglang
86           envFrom:
87           - secretRef:
88               name: hf-token-secret
89           image: my-registry/sglang-runtime:my-tag
90           name: main
91           resources:
92             limits:
93               nvidia.com/gpu: "4"
94           workingDir: /workspace/examples/backends/sglang
95     replicas: 1
96     type: prefill
97   env:
98   - name: HF_TOKEN
99     valueFrom:
100       secretKeyRef:
101         key: HF_TOKEN
102         name: hf-token-secret
103   - name: GLOO_SOCKET_IFNAME
104     value: eth0

Source

All templates live in examples/backends/sglang/deploy/. For local launch commands, see SGLang Local Deployment Examples.