Kubernetes Templates

DGDR Templates

Ready-to-apply DynamoGraphDeploymentRequest manifests for profiling and generating DynamoGraphDeployments.

View as Markdown

A DynamoGraphDeploymentRequest (DGDR) describes a model, workload, hardware, and optional latency targets. Dynamo profiles that intent and generates the DynamoGraphDeployment (DGD) that serves traffic. Use these templates when you want Dynamo to choose a deployment configuration instead of writing the complete DGD yourself.

Each manifest is embedded from examples/deployments/dgdr/. Open an example, use the copy button, and adjust its model, backend, hardware, and namespace before applying it.

Prerequisites

  • Install the Dynamo Kubernetes platform and operator.
  • Create hf-token-secret in the target namespace when the model requires authentication.
  • For namespace-restricted operator installations, set hardware.gpuSku, hardware.vramMb, and hardware.numGpusPerNode explicitly.

Release installations select the matching profiler image when spec.image is omitted. For a local operator build without a known release version, set spec.image to a compatible dynamo-planner image.

Apply a template with:

export NAMESPACE=dynamo-cloud
kubectl apply -f rapid.yaml -n ${NAMESPACE}

Generate a DGD

Uses simulated performance estimates to generate a DGD without benchmarking candidate engines on real GPUs.

rapid.yaml
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: qwen-rapid
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  searchStrategy: rapid
  autoApply: true

Deploys and benchmarks candidate configurations before selecting and generating the DGD.

thorough.yaml
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: qwen-thorough
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  searchStrategy: thorough
  autoApply: true

Adds a Planner configuration so the generated disaggregated deployment can adjust prefill and decode replicas at runtime.

planner.yaml
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: qwen-planner
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  searchStrategy: rapid
  workload:
    isl: 4000
    osl: 1000
    requestRate: 10
  sla:
    ttft: 500
    itl: 50
  features:
    planner:
      optimization_target: sla
      enable_throughput_scaling: true
      enable_load_scaling: false
      pre_deployment_sweeping_mode: rapid
      mode: disagg
      backend: vllm
  autoApply: true

Uses SGLang and a 32-GPU budget for a large Mixture-of-Experts model. Adapt its hardware, scheduling, model-cache, and SLA settings before applying it.

moe-sglang.yaml
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: deepseek-r1
spec:
  model: deepseek-ai/DeepSeek-R1
  backend: sglang
  searchStrategy: thorough
  hardware:
    totalGpus: 32
    numGpusPerNode: 8
  workload:
    isl: 4000
    osl: 1000
  sla:
    ttft: 2000
    itl: 100
  autoApply: false

Review and Customize the Generated DGD

Sets autoApply: false so the request stops after generation instead of creating the DGD.

review-before-deploy.yaml
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: qwen-review
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  searchStrategy: rapid
  autoApply: false

Merges an environment-variable override into the generated frontend component. Overrides can modify generated components but cannot add a new topology component.

generated-dgd-override.yaml
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: qwen-router-override
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  searchStrategy: rapid
  overrides:
    dgd:
      apiVersion: nvidia.com/v1beta1
      kind: DynamoGraphDeployment
      spec:
        components:
          - name: Frontend
            podTemplate:
              spec:
                containers:
                  - name: main
                    env:
                      - name: DYN_ROUTER_MODE
                        value: kv
  autoApply: false

Mounts a pre-populated model cache into the profiling job and generated workers. Replace the snapshot placeholder and create the PVC before applying the request.

model-cache.yaml
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: qwen-model-cache
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  searchStrategy: rapid
  modelCache:
    pvcName: model-cache
    pvcMountPath: /home/dynamo/.cache/huggingface
    pvcModelPath: hub/models--Qwen--Qwen3-0.6B/snapshots/<commit-hash>
  autoApply: true

Test and Inspect Profiling

Replaces model-serving workers with mock workers for testing routing, profiling, and Planner behavior without serving the model on GPUs.

mocker.yaml
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: qwen-mocker
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  searchStrategy: rapid
  features:
    mocker:
      enabled: true
  autoApply: true

Replaces the profiling output volume with the existing dynamo-pvc claim so plots, logs, configurations, and raw profiling data remain available after the Job completes.

profiling-artifacts.yaml
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: qwen-profiling-artifacts
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  searchStrategy: thorough
  overrides:
    profilingJob:
      template:
        spec:
          containers: []
          volumes:
            - name: profiling-output
              persistentVolumeClaim:
                claimName: dynamo-pvc
  autoApply: false

Inspect the Result

Watch the request until it reaches Deployed or Ready:

kubectl get dgdr <request-name> -n ${NAMESPACE} -w

For a request with autoApply: false, extract the generated DGD:

kubectl get dgdr <request-name> -n ${NAMESPACE} \
  -o jsonpath='{.status.profilingResults.selectedConfig}' > generated-dgd.yaml

See Auto Deploy with DGDR for the task-oriented workflow and the DGDR Reference for field definitions, lifecycle phases, and validation behavior.