SGLang Deployment Templates | NVIDIA Dynamo Documentation

Copy-paste DynamoGraphDeployment (nvidia.com/v1alpha1) manifests for the SGLang backend, grouped by topology. Each manifest is embedded from examples/backends/sglang/deploy/ — open an entry, use the copy button, then set your image tag and hf-token-secret before applying.

Apply any template with:

$ kubectl apply -f agg.yaml

Aggregated

agg.yaml · Baseline aggregated serving

agg.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1alpha1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-agg
8 spec:
9   services:
10     Frontend:
11       componentType: frontend
12       replicas: 1
13       extraPodSpec:
14         mainContainer:
15           image: my-registry/sglang-runtime:my-tag
16     decode:
17       envFromSecret: hf-token-secret
18       componentType: worker
19       replicas: 1
20       resources:
21         limits:
22           gpu: "1"
23         requests:
24           custom:
25             # Increase this value for larger models
26             ephemeral-storage: "2Gi"
27       extraPodSpec:
28         mainContainer:
29           image: my-registry/sglang-runtime:my-tag
30           workingDir: /workspace/examples/backends/sglang
31           command:
32           - python3
33           - -m
34           - dynamo.sglang
35           args:
36             - --model-path
37             - Qwen/Qwen3-0.6B
38             - --served-model-name
39             - Qwen/Qwen3-0.6B
40             - --page-size
41             - "16"
42             - --tp
43             - "1"
44             - --trust-remote-code
45             - --skip-tokenizer-init

agg_router.yaml · Aggregated with KV-aware routing

agg_router.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1alpha1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-agg-router
8 spec:
9   services:
10     Frontend:
11       componentType: frontend
12       replicas: 1
13       extraPodSpec:
14         mainContainer:
15           image: my-registry/sglang-runtime:my-tag
16       envs:
17         - name: DYN_ROUTER_MODE
18           value: kv
19     decode:
20       envFromSecret: hf-token-secret
21       componentType: worker
22       replicas: 1
23       resources:
24         limits:
25           gpu: "1"
26         requests:
27           custom:
28             # Increase this value for larger models
29             ephemeral-storage: "2Gi"
30       extraPodSpec:
31         mainContainer:
32           image: my-registry/sglang-runtime:my-tag
33           workingDir: /workspace/examples/backends/sglang
34           command:
35           - python3
36           - -m
37           - dynamo.sglang
38           args:
39             - --model-path
40             - Qwen/Qwen3-0.6B
41             - --served-model-name
42             - Qwen/Qwen3-0.6B
43             - --page-size
44             - "16"
45             - --tp
46             - "1"
47             - --trust-remote-code
48             - --skip-tokenizer-init
49             - --kv-events-config
50             - '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}'

agg_gms.yaml · Aggregated with GPU Memory Service sidecar

agg_gms.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 #
4 # GPU Memory Service (GMS) sidecar example.
5 #
6 # The operator injects a GMS sidecar init container that provides shared GPU
7 # memory access via DRA (Dynamic Resource Allocation). The sidecar runs two GMS
8 # processes per GPU (weights + kv_cache) and communicates with the main container
9 # over UDS sockets on a shared emptyDir volume.
10 #
11 # Requires Kubernetes 1.34+ with DRA v1 enabled and the NVIDIA GPU DRA driver installed.
12 
13 apiVersion: nvidia.com/v1alpha1
14 kind: DynamoGraphDeployment
15 metadata:
16   name: sglang-agg-gms
17 spec:
18   services:
19     Frontend:
20       componentType: frontend
21       replicas: 1
22       extraPodSpec:
23         mainContainer:
24           image: my-registry/sglang-runtime:my-tag
25     decode:
26       envFromSecret: hf-token-secret
27       componentType: worker
28       replicas: 1
29       resources:
30         limits:
31           gpu: "1"
32         requests:
33           custom:
34             # Increase this value for larger models
35             ephemeral-storage: "2Gi"
36       gpuMemoryService:
37         enabled: true
38       extraPodSpec:
39         mainContainer:
40           image: my-registry/sglang-runtime:my-tag
41           workingDir: /workspace/examples/backends/sglang
42           command:
43           - python3
44           - -m
45           - dynamo.sglang
46           args:
47             - --model-path
48             - Qwen/Qwen3-0.6B
49             - --served-model-name
50             - Qwen/Qwen3-0.6B
51             - --page-size
52             - "16"
53             - --tp
54             - "1"
55             - --trust-remote-code
56             - --skip-tokenizer-init
57             - --load-format
58             - gms

agg_logging.yaml · Aggregated with structured logging

agg_logging.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1alpha1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-agg
8 spec:
9   envs:
10   - name: DYN_LOGGING_JSONL
11     value: "1"
12   services:
13     Frontend:
14       componentType: frontend
15       replicas: 1
16       extraPodSpec:
17         mainContainer:
18           image: my-registry/sglang-runtime:my-tag
19     decode:
20       envFromSecret: hf-token-secret
21       componentType: worker
22       replicas: 1
23       resources:
24         limits:
25           gpu: "1"
26       extraPodSpec:
27         mainContainer:
28           image: my-registry/sglang-runtime:my-tag
29           workingDir: /workspace/examples/backends/sglang
30           command:
31           - python3
32           - -m
33           - dynamo.sglang
34           args:
35             - --model-path
36             - Qwen/Qwen3-0.6B
37             - --served-model-name
38             - Qwen/Qwen3-0.6B
39             - --page-size
40             - "16"
41             - --tp
42             - "1"
43             - --trust-remote-code
44             - --skip-tokenizer-init

Disaggregated

disagg.yaml · Baseline disaggregated prefill/decode

disagg.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1alpha1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-disagg
8 spec:
9   services:
10     Frontend:
11       componentType: frontend
12       replicas: 1
13       extraPodSpec:
14         mainContainer:
15           image: my-registry/sglang-runtime:my-tag
16     decode:
17       envFromSecret: hf-token-secret
18       componentType: worker
19       subComponentType: decode
20       replicas: 1
21       resources:
22         limits:
23           gpu: "1"
24       extraPodSpec:
25         mainContainer:
26           image: my-registry/sglang-runtime:my-tag
27           workingDir: /workspace/examples/backends/sglang
28           command:
29           - python3
30           - -m
31           - dynamo.sglang
32           args:
33             - --model-path
34             - Qwen/Qwen3-0.6B
35             - --served-model-name
36             - Qwen/Qwen3-0.6B
37             - --page-size
38             - "16"
39             - --tp
40             - "1"
41             - --trust-remote-code
42             - --skip-tokenizer-init
43             - --disaggregation-mode
44             - decode
45             - --disaggregation-transfer-backend
46             - nixl
47             - --disaggregation-bootstrap-port
48             - "12345"
49             - --host
50             - "0.0.0.0"
51     prefill:
52       envFromSecret: hf-token-secret
53       componentType: worker
54       subComponentType: prefill
55       replicas: 1
56       resources:
57         limits:
58           gpu: "1"
59       extraPodSpec:
60         mainContainer:
61           image: my-registry/sglang-runtime:my-tag
62           workingDir: /workspace/examples/backends/sglang
63           command:
64           - python3
65           - -m
66           - dynamo.sglang
67           args:
68             - --model-path
69             - Qwen/Qwen3-0.6B
70             - --served-model-name
71             - Qwen/Qwen3-0.6B
72             - --page-size
73             - "16"
74             - --tp
75             - "1"
76             - --trust-remote-code
77             - --skip-tokenizer-init
78             - --disaggregation-mode
79             - prefill
80             - --disaggregation-transfer-backend
81             - nixl
82             - --disaggregation-bootstrap-port
83             - "12345"
84             - --host
85             - "0.0.0.0"

disagg_planner.yaml · Disaggregated with Dynamo Planner autoscaling

disagg_planner.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1alpha1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-disagg-planner
8 spec:
9   services:
10     Frontend:
11       componentType: frontend
12       replicas: 1
13       extraPodSpec:
14         mainContainer:
15           image: my-registry/sglang-runtime:my-tag
16     Planner:
17       envFromSecret: hf-token-secret
18       componentType: planner
19       replicas: 1
20       extraPodSpec:
21         mainContainer:
22           # Planner image selection:
23           #   Dynamo >= 1.1.0: use the dedicated planner image
24           #     <registry>/dynamo-planner:<version>
25           #     (backend runtime images no longer ship planner runtime deps
26           #     such as kubernetes_asyncio, pmdarima, prophet, aiconfigurator).
27           #   Dynamo <  1.1.0: use the backend runtime image
28           #     <registry>/sglang-runtime:<version>.
29           image: my-registry/dynamo-planner:my-tag
30           command:
31           - python3
32           - -m
33           - dynamo.planner
34           args:
35             - --config
36             - '{"environment": "kubernetes", "backend": "sglang", "optimization_target": "sla", "enable_throughput_scaling": true, "enable_load_scaling": true, "pre_deployment_sweeping_mode": "none", "throughput_adjustment_interval_seconds": 60, "load_adjustment_interval_seconds": 5}'
37     decode:
38       envFromSecret: hf-token-secret
39       componentType: worker
40       subComponentType: decode
41       replicas: 2
42       resources:
43         limits:
44           gpu: "1"
45       extraPodSpec:
46         mainContainer:
47           image: my-registry/sglang-runtime:my-tag
48           workingDir: /workspace/examples/backends/sglang
49           command:
50             - python3
51           args:
52             - -m
53             - dynamo.sglang
54             - --model-path
55             - Qwen/Qwen3-0.6B
56             - --served-model-name
57             - Qwen/Qwen3-0.6B
58             - --page-size
59             - "16"
60             - --tp
61             - "1"
62             - --trust-remote-code
63             - --skip-tokenizer-init
64             - --disaggregation-mode
65             - decode
66             - --disaggregation-transfer-backend
67             - nixl
68             - --disaggregation-bootstrap-port
69             - "12345"
70             - --host
71             - "0.0.0.0"
72     prefill:
73       envFromSecret: hf-token-secret
74       componentType: worker
75       subComponentType: prefill
76       replicas: 2
77       resources:
78         limits:
79           gpu: "1"
80       extraPodSpec:
81         mainContainer:
82           image: my-registry/sglang-runtime:my-tag
83           workingDir: /workspace/examples/backends/sglang
84           command:
85             - python3
86           args:
87             - -m
88             - dynamo.sglang
89             - --model-path
90             - Qwen/Qwen3-0.6B
91             - --served-model-name
92             - Qwen/Qwen3-0.6B
93             - --page-size
94             - "16"
95             - --tp
96             - "1"
97             - --trust-remote-code
98             - --skip-tokenizer-init
99             - --disaggregation-mode
100             - prefill
101             - --disaggregation-transfer-backend
102             - nixl
103             - --disaggregation-bootstrap-port
104             - "12345"
105             - --host
106             - "0.0.0.0"

disagg-multinode.yaml · Disaggregated across multiple nodes

disagg-multinode.yaml

1 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2 # SPDX-License-Identifier: Apache-2.0
3 
4 apiVersion: nvidia.com/v1alpha1
5 kind: DynamoGraphDeployment
6 metadata:
7   name: sglang-disagg-multinode
8 spec:
9   envs:
10   - name: HF_TOKEN
11     valueFrom:
12       secretKeyRef:
13         name: hf-token-secret
14         key: HF_TOKEN
15   - name: GLOO_SOCKET_IFNAME
16     value: "eth0"
17   backendFramework: sglang
18   services:
19     Frontend:
20       componentType: frontend
21       replicas: 1
22       extraPodSpec:
23         mainContainer:
24           image: my-registry/sglang-runtime:my-tag
25     decode:
26       multinode:
27         nodeCount: 2
28       envFromSecret: hf-token-secret
29       componentType: worker
30       replicas: 1
31       resources:
32         limits:
33           gpu: "4"
34       extraPodSpec:
35         mainContainer:
36           image: my-registry/sglang-runtime:my-tag
37           workingDir: /workspace/examples/backends/sglang
38           command:
39           - python3
40           - -m
41           - dynamo.sglang
42           args:
43             - --model-path
44             - Qwen/Qwen3-0.6B
45             - --served-model-name
46             - Qwen/Qwen3-0.6B
47             - --tp-size
48             - "8"
49             - --trust-remote-code
50             - --skip-tokenizer-init
51             - --disaggregation-mode
52             - decode
53             - --disaggregation-transfer-backend
54             - nixl
55             - --disaggregation-bootstrap-port
56             - "30001"
57             - --host
58             - "0.0.0.0"
59             - --mem-fraction-static
60             - "0.82"
61     prefill:
62       multinode:
63         nodeCount: 2
64       envFromSecret: hf-token-secret
65       componentType: worker
66       replicas: 1
67       resources:
68         limits:
69           gpu: "4"
70       extraPodSpec:
71         mainContainer:
72           image: my-registry/sglang-runtime:my-tag
73           workingDir: /workspace/examples/backends/sglang
74           command:
75           - python3
76           - -m
77           - dynamo.sglang
78           args:
79             - --model-path
80             - Qwen/Qwen3-0.6B
81             - --served-model-name
82             - Qwen/Qwen3-0.6B
83             - --tp-size
84             - "8"
85             - --trust-remote-code
86             - --skip-tokenizer-init
87             - --disaggregation-mode
88             - prefill
89             - --disaggregation-transfer-backend
90             - nixl
91             - --disaggregation-bootstrap-port
92             - "30001"
93             - --mem-fraction-static
94             - "0.82"
95             - --host
96             - "0.0.0.0"

Source

All templates live in examples/backends/sglang/deploy/. For local launch commands, see SGLang Local Deployment Examples.