SGLang Deployment Templates | NVIDIA Dynamo Documentation

Copy-paste DynamoGraphDeployment (nvidia.com/v1beta1) manifests for the SGLang backend, grouped by topology. Each manifest is embedded from examples/backends/sglang/deploy/ — open an entry, use the copy button, then set your image tag and hf-token-secret before applying.

Apply any template with:

kubectl apply -f agg.yaml

Aggregated

agg.yaml · Baseline aggregated serving

agg.yaml

# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
  name: sglang-agg
spec:
  components:
  - name: Frontend
    podTemplate:
      spec:
        containers:
        - image: my-registry/sglang-runtime:my-tag
          name: main
    replicas: 1
    type: frontend
  - name: decode
    podTemplate:
      spec:
        containers:
        - args:
          - --model-path
          - Qwen/Qwen3-0.6B
          - --served-model-name
          - Qwen/Qwen3-0.6B
          - --page-size
          - "16"
          - --tp
          - "1"
          - --trust-remote-code
          command:
          - python3
          - -m
          - dynamo.sglang
          envFrom:
          - secretRef:
              name: hf-token-secret
          image: my-registry/sglang-runtime:my-tag
          name: main
          resources:
            limits:
              nvidia.com/gpu: "1"
            requests:
              # Increase this value for larger models.
              ephemeral-storage: 2Gi
          workingDir: /workspace/examples/backends/sglang
    replicas: 1
    type: worker

agg_router.yaml · Aggregated with KV-aware routing

agg_router.yaml

# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
  name: sglang-agg-router
spec:
  components:
  - name: Frontend
    podTemplate:
      spec:
        containers:
        - env:
          - name: DYN_ROUTER_MODE
            value: kv
          image: my-registry/sglang-runtime:my-tag
          name: main
    replicas: 1
    type: frontend
  - name: decode
    podTemplate:
      spec:
        containers:
        - args:
          - --model-path
          - Qwen/Qwen3-0.6B
          - --served-model-name
          - Qwen/Qwen3-0.6B
          - --page-size
          - "16"
          - --tp
          - "1"
          - --trust-remote-code
          - --kv-events-config
          - '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}'
          command:
          - python3
          - -m
          - dynamo.sglang
          envFrom:
          - secretRef:
              name: hf-token-secret
          image: my-registry/sglang-runtime:my-tag
          name: main
          resources:
            limits:
              nvidia.com/gpu: "1"
            requests:
              # Increase this value for larger models.
              ephemeral-storage: 2Gi
          workingDir: /workspace/examples/backends/sglang
    replicas: 1
    type: worker

agg_gms.yaml · Aggregated with GPU Memory Service sidecar

agg_gms.yaml

# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# GPU Memory Service (GMS) sidecar example.
#
# The operator injects a GMS sidecar init container that provides shared GPU
# memory access via DRA (Dynamic Resource Allocation). The sidecar runs two GMS
# processes per GPU (weights + kv_cache) and communicates with the main container
# over UDS sockets on a shared emptyDir volume.
#
# Requires Kubernetes 1.34+ with DRA v1 enabled and the NVIDIA GPU DRA driver installed.
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
  name: sglang-agg-gms
spec:
  components:
  - name: Frontend
    podTemplate:
      spec:
        containers:
        - image: my-registry/sglang-runtime:my-tag
          name: main
    replicas: 1
    type: frontend
  - experimental:
      gpuMemoryService: {}
    name: decode
    podTemplate:
      spec:
        containers:
        - args:
          - --model-path
          - Qwen/Qwen3-0.6B
          - --served-model-name
          - Qwen/Qwen3-0.6B
          - --page-size
          - "16"
          - --tp
          - "1"
          - --trust-remote-code
          - --load-format
          - gms
          command:
          - python3
          - -m
          - dynamo.sglang
          envFrom:
          - secretRef:
              name: hf-token-secret
          image: my-registry/sglang-runtime:my-tag
          name: main
          resources:
            limits:
              nvidia.com/gpu: "1"
            requests:
              # Increase this value for larger models.
              ephemeral-storage: 2Gi
          workingDir: /workspace/examples/backends/sglang
    replicas: 1
    type: worker

agg_logging.yaml · Aggregated with structured logging

agg_logging.yaml

# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
  name: sglang-agg-logging
spec:
  components:
  - name: Frontend
    podTemplate:
      spec:
        containers:
        - image: my-registry/sglang-runtime:my-tag
          name: main
    replicas: 1
    type: frontend
  - name: decode
    podTemplate:
      spec:
        containers:
        - args:
          - --model-path
          - Qwen/Qwen3-0.6B
          - --served-model-name
          - Qwen/Qwen3-0.6B
          - --page-size
          - "16"
          - --tp
          - "1"
          - --trust-remote-code
          command:
          - python3
          - -m
          - dynamo.sglang
          envFrom:
          - secretRef:
              name: hf-token-secret
          image: my-registry/sglang-runtime:my-tag
          name: main
          resources:
            limits:
              nvidia.com/gpu: "1"
          workingDir: /workspace/examples/backends/sglang
    replicas: 1
    type: worker
  env:
  - name: DYN_LOGGING_JSONL
    value: "1"

Disaggregated

disagg.yaml · Baseline disaggregated prefill/decode

disagg.yaml

# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
  name: sglang-disagg
spec:
  components:
  - name: Frontend
    podTemplate:
      spec:
        containers:
        - image: my-registry/sglang-runtime:my-tag
          name: main
    replicas: 1
    type: frontend
  - name: decode
    podTemplate:
      spec:
        containers:
        - args:
          - --model-path
          - Qwen/Qwen3-0.6B
          - --served-model-name
          - Qwen/Qwen3-0.6B
          - --page-size
          - "16"
          - --tp
          - "1"
          - --trust-remote-code
          - --disaggregation-mode
          - decode
          - --disaggregation-transfer-backend
          - nixl
          - --disaggregation-bootstrap-port
          - "12345"
          - --host
          - 0.0.0.0
          command:
          - python3
          - -m
          - dynamo.sglang
          envFrom:
          - secretRef:
              name: hf-token-secret
          image: my-registry/sglang-runtime:my-tag
          name: main
          resources:
            limits:
              nvidia.com/gpu: "1"
          workingDir: /workspace/examples/backends/sglang
    replicas: 1
    type: decode
  - name: prefill
    podTemplate:
      spec:
        containers:
        - args:
          - --model-path
          - Qwen/Qwen3-0.6B
          - --served-model-name
          - Qwen/Qwen3-0.6B
          - --page-size
          - "16"
          - --tp
          - "1"
          - --trust-remote-code
          - --disaggregation-mode
          - prefill
          - --disaggregation-transfer-backend
          - nixl
          - --disaggregation-bootstrap-port
          - "12345"
          - --host
          - 0.0.0.0
          command:
          - python3
          - -m
          - dynamo.sglang
          envFrom:
          - secretRef:
              name: hf-token-secret
          image: my-registry/sglang-runtime:my-tag
          name: main
          resources:
            limits:
              nvidia.com/gpu: "1"
          workingDir: /workspace/examples/backends/sglang
    replicas: 1
    type: prefill

disagg_planner.yaml · Disaggregated with Dynamo Planner autoscaling

disagg_planner.yaml

# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
  name: sglang-disagg-planner
spec:
  components:
  - name: Frontend
    podTemplate:
      spec:
        containers:
        - image: my-registry/sglang-runtime:my-tag
          name: main
    replicas: 1
    type: frontend
  - name: Planner
    podTemplate:
      spec:
        containers:
        - args:
          - --config
          - '{"environment": "kubernetes", "backend": "sglang", "optimization_target": "sla",
            "enable_throughput_scaling": true, "enable_load_scaling": true, "pre_deployment_sweeping_mode":
            "none", "throughput_adjustment_interval_seconds": 60, "load_adjustment_interval_seconds": 5}'
          command:
          - python3
          - -m
          - dynamo.planner
          envFrom:
          - secretRef:
              name: hf-token-secret
          # Planner image selection:
          #   Dynamo >= 1.1.0: use the dedicated planner image
          #     <registry>/dynamo-planner:<version>
          #     (backend runtime images no longer ship planner runtime deps
          #     such as kubernetes_asyncio, pmdarima, prophet, aiconfigurator).
          #   Dynamo <  1.1.0: use the backend runtime image
          #     <registry>/sglang-runtime:<version>.
          image: my-registry/dynamo-planner:my-tag
          name: main
    replicas: 1
    type: planner
  - name: decode
    podTemplate:
      spec:
        containers:
        - args:
          - --model-path
          - Qwen/Qwen3-0.6B
          - --served-model-name
          - Qwen/Qwen3-0.6B
          - --page-size
          - "16"
          - --tp
          - "1"
          - --trust-remote-code
          - --disaggregation-mode
          - decode
          - --disaggregation-transfer-backend
          - nixl
          - --disaggregation-bootstrap-port
          - "12345"
          - --host
          - 0.0.0.0
          command:
          - python3
          - -m
          - dynamo.sglang
          envFrom:
          - secretRef:
              name: hf-token-secret
          image: my-registry/sglang-runtime:my-tag
          name: main
          resources:
            limits:
              nvidia.com/gpu: "1"
          workingDir: /workspace/examples/backends/sglang
    replicas: 2
    type: decode
  - name: prefill
    podTemplate:
      spec:
        containers:
        - args:
          - --model-path
          - Qwen/Qwen3-0.6B
          - --served-model-name
          - Qwen/Qwen3-0.6B
          - --page-size
          - "16"
          - --tp
          - "1"
          - --trust-remote-code
          - --disaggregation-mode
          - prefill
          - --disaggregation-transfer-backend
          - nixl
          - --disaggregation-bootstrap-port
          - "12345"
          - --host
          - 0.0.0.0
          command:
          - python3
          - -m
          - dynamo.sglang
          envFrom:
          - secretRef:
              name: hf-token-secret
          image: my-registry/sglang-runtime:my-tag
          name: main
          resources:
            limits:
              nvidia.com/gpu: "1"
          workingDir: /workspace/examples/backends/sglang
    replicas: 2
    type: prefill

disagg-multinode.yaml · Disaggregated across multiple nodes

disagg-multinode.yaml

# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
  name: sglang-disagg-multinode
spec:
  backendFramework: sglang
  components:
  - name: Frontend
    podTemplate:
      spec:
        containers:
        - image: my-registry/sglang-runtime:my-tag
          name: main
    replicas: 1
    type: frontend
  - multinode:
      nodeCount: 2
    name: decode
    podTemplate:
      spec:
        containers:
        - args:
          - --model-path
          - Qwen/Qwen3-0.6B
          - --served-model-name
          - Qwen/Qwen3-0.6B
          - --tp-size
          - "8"
          - --trust-remote-code
          - --disaggregation-mode
          - decode
          - --disaggregation-transfer-backend
          - nixl
          - --disaggregation-bootstrap-port
          - "30001"
          - --host
          - 0.0.0.0
          - --mem-fraction-static
          - "0.82"
          command:
          - python3
          - -m
          - dynamo.sglang
          envFrom:
          - secretRef:
              name: hf-token-secret
          image: my-registry/sglang-runtime:my-tag
          name: main
          resources:
            limits:
              nvidia.com/gpu: "4"
          workingDir: /workspace/examples/backends/sglang
    replicas: 1
    type: decode
  - multinode:
      nodeCount: 2
    name: prefill
    podTemplate:
      spec:
        containers:
        - args:
          - --model-path
          - Qwen/Qwen3-0.6B
          - --served-model-name
          - Qwen/Qwen3-0.6B
          - --tp-size
          - "8"
          - --trust-remote-code
          - --disaggregation-mode
          - prefill
          - --disaggregation-transfer-backend
          - nixl
          - --disaggregation-bootstrap-port
          - "30001"
          - --mem-fraction-static
          - "0.82"
          - --host
          - 0.0.0.0
          command:
          - python3
          - -m
          - dynamo.sglang
          envFrom:
          - secretRef:
              name: hf-token-secret
          image: my-registry/sglang-runtime:my-tag
          name: main
          resources:
            limits:
              nvidia.com/gpu: "4"
          workingDir: /workspace/examples/backends/sglang
    replicas: 1
    type: prefill
  env:
  - name: HF_TOKEN
    valueFrom:
      secretKeyRef:
        key: HF_TOKEN
        name: hf-token-secret
  - name: GLOO_SOCKET_IFNAME
    value: eth0

Source

All templates live in examples/backends/sglang/deploy/. For local launch commands, see SGLang Local Deployment Examples.