Auto Deploy with DGDR

Submit a minimal DGDR

Start with the smallest request: a model. With searchStrategy and autoApply at their defaults, the profiler uses rapid simulation (~30 seconds, no GPUs consumed during profiling) and the operator deploys the result. In a release installation, the webhook selects the matching versioned Planner image.

1
apiVersion: nvidia.com/v1beta1
2
kind: DynamoGraphDeploymentRequest
3
metadata:
4
  name: qwen-small
5
spec:
6
  model: Qwen/Qwen3-0.6B

Apply it and watch the request progress through its phases:

$
kubectl apply -f qwen-small.yaml -n <namespace>
$
kubectl get dgdr qwen-small -n <namespace> -w

The DGDR moves through Pending → Profiling → Ready → Deploying → Deployed. Once it reaches Deployed, the generated DGD is running. See Monitor profiling and deployment for the full phase list and log commands.

Describe your workload and SLA targets

The profiler sizes the deployment against the traffic you expect and the latency you need. Provide a workload shape, and optionally sla targets, so the profiler optimizes for your case instead of the defaults.

1
spec:
2
  model: meta-llama/Llama-3.1-8B-Instruct
3
  backend: vllm
4
  workload:
5
    isl: 4000          # average input sequence length (tokens)
6
    osl: 1000          # average output sequence length (tokens)
7
    requestRate: 10    # target requests per second
8
  sla:
9
    ttft: 500          # target Time To First Token (ms)
10
    itl: 50            # target Inter-Token Latency (ms)

Choose a search strategy

The searchStrategy field controls how the profiler explores configurations. The choice trades profiling time against how close to optimal you land.

Rapid (default) uses AIC-backed DynoSim-style performance modeling to search configurations without running real inference. Completes in ~30 seconds with no GPUs consumed during profiling.

1
spec:
2
  searchStrategy: rapid

Use rapid when getting started, iterating quickly, or running in CI/CD — provided your GPU SKU is in the AIC support matrix. If AIC does not support your model/hardware/backend combination, the profiler falls back to a naive memory-fit config that may not be optimal.

Thorough enumerates candidate parallelization configs, deploys each on real GPUs, and benchmarks them with AIPerf. Takes 2–4 hours and produces measured rather than simulated data.

1
spec:
2
  searchStrategy: thorough
3
  backend: vllm      # must specify a concrete backend

Use thorough when tuning for production, when your hardware is not supported by AIC (for example PCIe GPUs), or when you need measured performance. It has constraints:

• Disaggregated mode only — thorough does not run aggregated configurations.
• backend: auto is rejected — specify vllm, sglang, or trtllm.
• Requires GPU resources — the profiler deploys real inference engines during profiling.

For the profiling algorithms, gate checks, and how to pick a mode, see the Profiler Guide.

Set hardware and handle large or multinode models

The operator auto-detects GPU SKU, VRAM, and count (capped at 32). Override any of these under hardware when auto-detection is wrong or you want the profiler to consider more GPUs.

1
spec:
2
  hardware:
3
    gpuSku: h200_sxm   # lowercase underscore format
4
    totalGpus: 64      # raise above the 32-GPU auto-detect cap
5
    numGpusPerNode: 8

GPU SKUs use lowercase underscore format (h100_sxm, not H100-SXM5-80GB). For the full list of accepted values and the PCIe-support caveat, see the DGDR Reference — SKU Format.

Large and MoE models that span nodes. When a model needs more GPUs than one node provides, the deployment is multinode and requires a gang scheduler. Install an orchestrator first — the operator returns a hard error otherwise:

• Multinode Orchestration — install-time prerequisites (Grove + KAI, or LWS + Volcano).
• Grove (default) and LWS — the two orchestration backends.

For Mixture-of-Experts (MoE) models (DeepSeek-R1, Qwen3-MoE), use SGLang for full support — vLLM and TensorRT-LLM have partial MoE support still under development. The profiler sweeps MoE models across up to 4 nodes; beyond that, it selects the best config within range and you may need to adjust replica counts manually. See Backend Selection.

Cache model weights

By default the profiling job and every generated worker pod download the model from HuggingFace. For large models (>70B) this is slow, and many replicas hit HuggingFace rate limits. Point modelCache at a pre-populated ReadWriteMany PVC so weights load from shared storage instead.

1
spec:
2
  model: meta-llama/Llama-3.1-70B-Instruct
3
  modelCache:
4
    pvcName: model-cache
5
    pvcMountPath: /home/dynamo/.cache/huggingface
6
    pvcModelPath: hub/models--meta-llama--Llama-3.1-70B-Instruct/snapshots/<commit-hash>

The operator mounts the PVC read-only into the profiling job and passes it through to the generated DGD, so both profiling and serving use the cached weights.

pvcModelPath must be the HuggingFace snapshot path inside the PVC: hub/models--<org>--<model>/snapshots/<commit-hash>. Substitute / with -- in the model ID, and replace <commit-hash> with the actual snapshot revision. See Model Caching — Find the Snapshot Path for how to look it up.

Setup: create a ReadWriteMany PVC (Installation Guide — Shared Storage), run a one-time download Job to populate it, then reference it here. See Model Caching for the full walkthrough.

For gated models, create the token secret the profiler and pods read automatically:

$
kubectl create secret generic hf-token-secret \
>
  --from-literal=HF_TOKEN=<your-token> \
>
  -n <namespace>

Enable the Planner for runtime autoscaling

The Planner provides runtime autoscaling for disaggregated deployments: it adjusts prefill and decode replica counts to meet your SLA targets as traffic fluctuates. Enable it by setting features.planner to a PlannerConfig; DGDR passes the object through to the Planner service and generates Planner support in the final DGD.

1
spec:
2
  model: Qwen/Qwen3-0.6B
3
  backend: vllm
4
  features:
5
    planner:
6
      mode: disagg
7
      backend: vllm
8
  sla:
9
    ttft: 500
10
    itl: 50

To evaluate Planner recommendations without applying scaling changes, add advisory: true to the same object.

1
spec:
2
  features:
3
    planner:
4
      mode: disagg
5
      backend: vllm
6
      advisory: true

The Planner’s sla optimization target reads live TTFT/ITL from Prometheus, so install Prometheus before creating the DGDR if you want SLA-driven scaling. The throughput and latency modes use internal queue-depth signals and work without Prometheus. For scaling modes and the full PlannerConfig field reference, see the Planner Guide.

Review the generated DGD

For production, inspect the generated DGD before it deploys. Set autoApply: false so the DGDR stops at Ready and stores the config instead of deploying it.

1
spec:
2
  autoApply: false

After profiling completes, extract, review, and apply the DGD yourself:

$
kubectl get dgdr my-model -n <namespace> \
>
  -o jsonpath='{.status.profilingResults.selectedConfig}' > my-dgd.yaml
$
# Review and edit my-dgd.yaml, then:
$
kubectl apply -f my-dgd.yaml -n <namespace>