Kubernetes DeploymentDeploy Models

DGDR Examples

Practical DynamoGraphDeploymentRequest examples covering AIC estimates, online profiling, and SLA-driven generation.

以 Markdown 格式查看

Practical examples for deploying with DynamoGraphDeploymentRequest (DGDR). The DGDR workflow can use native AIC estimates, optional bootstrap profiling data, or live FPM warmup depending on the model/backend combination. For DGDR concepts, see the DGDR Reference. For profiling concepts, see the Profiler Guide.

DGDR Examples

Minimal DGDR with AIC (Fastest)

The simplest way to generate a deployment from native AIC estimates. Uses AI Configurator for offline profiling (20-30 seconds instead of hours):

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: sla-aic
spec:
  model: Qwen/Qwen3-32B
  backend: vllm
  image: "nvcr.io/nvidia/ai-dynamo/dynamo-planner:1.2.1"  # dynamo-frontend for Dynamo < 1.1.0

Deploy:

export NAMESPACE=your-namespace
# Save the manifest above as sla-aic.yaml first.
kubectl apply -f sla-aic.yaml -n $NAMESPACE

Online Profiling (Real Measurements)

Standard online profiling runs real GPU measurements for more accurate results. Takes 2-4 hours:

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: sla-online
spec:
  model: meta-llama/Llama-3.3-70B-Instruct
  backend: vllm
  image: "nvcr.io/nvidia/ai-dynamo/dynamo-planner:1.2.1"  # dynamo-frontend for Dynamo < 1.1.0
  searchStrategy: thorough

Deploy:

# Save the manifest above as sla-online.yaml first.
kubectl apply -f sla-online.yaml -n $NAMESPACE

Note: Starting with Dynamo 1.0.0 (DGDR API version v1beta1), DGDR fields use structured spec fields (e.g., spec.workload, spec.sla, spec.hardware) instead of the nested profilingConfig.config blob used in v1alpha1.

Planner-Enabled DGDR

Set spec.features.planner to enable Planner generation in the final DGD. DGDR passes this object as PlannerConfig to the Planner service; see the Planner Guide for available fields.

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: qwen3-planner
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  image: "nvcr.io/nvidia/ai-dynamo/dynamo-planner:1.2.1"  # dynamo-frontend for Dynamo < 1.1.0
  features:
    planner:
      mode: disagg
      backend: vllm

spec.overrides.dgd is not required to enable Planner; use it only when the generated DGD needs additional customization.

Additional DGDR Patterns

MoE Models (SGLang)

For Mixture-of-Experts models like DeepSeek-R1, use SGLang backend:

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: sla-moe
spec:
  model: deepseek-ai/DeepSeek-R1
  backend: sglang
  image: "nvcr.io/nvidia/ai-dynamo/dynamo-planner:1.2.1"  # dynamo-frontend for Dynamo < 1.1.0

Deploy:

# Save the manifest above as sla-moe.yaml first.
kubectl apply -f sla-moe.yaml -n $NAMESPACE

Customizing the Generated DGD

Use spec.overrides.dgd to provide a partial DynamoGraphDeployment that is merged into the profiler-generated deployment. Use a v1beta1 override for new DGDRs:

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: deepseek-r1
spec:
  model: deepseek-ai/DeepSeek-R1
  backend: sglang
  image: "nvcr.io/nvidia/ai-dynamo/dynamo-planner:1.2.1"  # dynamo-frontend for Dynamo < 1.1.0
  overrides:
    dgd:
      apiVersion: nvidia.com/v1beta1
      kind: DynamoGraphDeployment
      spec:
        env:
          - name: CUSTOM_WORKER_ENV
            value: "enabled"

DGDR merges the override into the generated DGD after profiling selects a configuration. Both .status.profilingResults.selectedConfig and the DGD created when autoApply: true use nvidia.com/v1beta1.

Existing v1alpha1 overrides remain supported. Their field shape follows the v1alpha1 DGD schema:

spec:
  overrides:
    dgd:
      apiVersion: nvidia.com/v1alpha1
      kind: DynamoGraphDeployment
      spec:
        envs:
          - name: CUSTOM_WORKER_ENV
            value: "enabled"

The override’s API version controls its merge semantics. In particular, the v1beta1 graph-level spec.env list and container args replace their generated lists, while nested container environment variables merge by name. v1alpha1 worker arguments append for compatibility. See Generated DGD Overrides for the complete behavior and direct-profiler requirements.

Inline Configuration (Simple Use Cases)

For simple use cases without a custom DGD config, provide the configuration directly in the v1beta1 DGDR spec fields. The profiler auto-generates a basic DGD configuration:

spec:
  workload:
    isl: 8000
    osl: 200
  sla:
    ttft: 200.0
    itl: 10.0
  hardware:
    gpuSku: h200_sxm
  searchStrategy: rapid

Simulation with Mocker

Deploy a mocker backend that simulates GPU timing behavior without real GPUs. Useful for:

  • Large-scale experiments without GPU resources
  • Testing profiling behavior and infrastructure
  • Validating deployment configurations
spec:
  model: <model-name>
  backend: trtllm  # Real backend for profiling
  features:
    mocker:
      enabled: true  # Deploy mocker instead of real backend
  image: "nvcr.io/nvidia/ai-dynamo/dynamo-planner:1.2.1"  # dynamo-frontend for Dynamo < 1.1.0

Profiling runs against the real backend (via GPUs or AIC). The mocker deployment then uses profiling data to simulate realistic timing.

Model Cache PVC (0.8.1+)

For large models, use a pre-populated PVC instead of downloading from HuggingFace:

See SLA-Driven Profiling for configuration details.

Advanced DGDR Patterns

Review Before Deploy (autoApply: false)

Disable auto-deployment to inspect the generated DGD:

spec:
  autoApply: false

After profiling completes:

# Extract and review generated DGD
kubectl get dgdr sla-aic -n $NAMESPACE \
  -o jsonpath='{.status.profilingResults.selectedConfig}' > my-dgd.yaml
# Review and modify as needed
vi my-dgd.yaml
# Deploy manually
kubectl apply -f my-dgd.yaml -n $NAMESPACE

Profiling Artifacts with PVC

Save detailed profiling artifacts (plots, logs, raw data) to a PVC:

spec:
  workload:
    isl: 3000
    osl: 150
  sla:
    ttft: 200
    itl: 20

Setup:

export NAMESPACE=your-namespace
deploy/utils/setup_benchmarking_resources.sh

Access results:

kubectl apply -f deploy/utils/manifests/pvc-access-pod.yaml -n $NAMESPACE
kubectl wait --for=condition=Ready pod/pvc-access-pod -n $NAMESPACE --timeout=60s
kubectl cp $NAMESPACE/pvc-access-pod:/data ./profiling-results
kubectl delete pod pvc-access-pod -n $NAMESPACE