SLA-Driven Profiling and Planner Deployment Quick Start Guide

Complete workflow to deploy SLA-optimized Dynamo models using DynamoGraphDeploymentRequests (DGDR). This guide shows how to automatically profile models and deploy them with optimal configurations that meet your Service Level Agreements (SLAs).

Overview

The DGDR workflow automates the entire process from SLA specification to deployment:

1. Define SLAs: Specify performance requirements (TTFT, ITL) and model information in a DGDR Custom Resource
2. Automatic Profiling: The Dynamo Operator automatically profiles your model to find optimal configurations
3. Auto-Deploy: The system automatically deploys the optimal configuration that meets your SLAs

What is a DynamoGraphDeploymentRequest (DGDR)?

A DynamoGraphDeploymentRequest (DGDR) is a Kubernetes Custom Resource that serves as the primary interface for users to request model deployments with specific performance and resource constraints. Think of it as a “deployment order” where you specify:

• What model you want to deploy (model)
• How it should perform (SLA targets: ttft, itl)
• Where it should run (optional GPU preferences)
• Which backend to use (backend: vllm, sglang, or trtllm)
• Which images to use (profilingConfig.profilerImage, deploymentOverrides.workersImage)

The Dynamo Operator watches for DGDRs and automatically:

1. Discovers available GPU resources in your cluster
2. Runs profiling (online or offline) to find optimal configurations
3. Generates an optimized DynamoGraphDeployment (DGD) configuration
4. Deploys the DGD to your cluster

Key Benefits:

• Declarative: Specify what you want, not how to achieve it
• Automated: No manual profiling job setup or result processing
• SLA-Driven: Ensures deployments meet your performance requirements
• Integrated: Works seamlessly with the Dynamo Operator

Prerequisites

Before creating a DGDR, ensure:

• Dynamo platform installed with the operator running (see Installation Guide)
• kube-prometheus-stack installed and running (required for SLA planner)
• Image pull secrets configured if using private registries (typically nvcr-imagepullsecret for NVIDIA images)
• Sufficient GPU resources available in your cluster for profiling
• Runtime images available that contain both profiler and runtime components

Container Images

Each DGDR requires you to specify container images for the profiling and deployment process:

profilingConfig.profilerImage (Required): Specifies the container image used for the profiling job itself. This image must contain the profiler code and dependencies needed for SLA-based profiling.

deploymentOverrides.workersImage (Optional): Specifies the container image used for DynamoGraphDeployment worker components (frontend, workers, planner). This image is used for:

• Temporary DGDs created during online profiling (for performance measurements)
• The final DGD deployed after profiling completes

If workersImage is omitted, the image from the base config file (e.g., disagg.yaml) is used. You may use our public images (0.6.1 and later) or build and push your own.

1
spec:
2
  profilingConfig:
3
    profilerImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.6.1"
4
  deploymentOverrides:
5
    workersImage: "nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.6.1"  # Optional

Quick Start: Deploy with DGDR

Step 1: Create Your DGDR

Dynamo provides sample DGDR configurations in benchmarks/profiler/deploy/. You can use these as starting points:

Available Sample DGDRs:

• profile_sla_dgdr.yaml: Standard online profiling for dense models
• profile_sla_aic_dgdr.yaml: Fast offline profiling using AI Configurator
• profile_sla_moe_dgdr.yaml: Online profiling for MoE models (SGLang)

Or, you can create your own DGDR for your own needs.

Important - Profiling Config Cases: Prior to 0.8.1, any fields under profilingConfig.config are represented in snake_case. Starting 0.8.1, fields under profilingConfig.config are represented in camelCase for uniformity. There is backwards compatibility to snake_case, but as all example DGDRs are using camelCase, anyone using a release prior to 0.8.1 must manually update the configs under the examples to have snake_case config fields.

Step 2: Apply the DGDR

The rest of this quickstart will use the DGDR sample that uses AIC profiling. If you use a different DGDR file and/or name, be sure to adjust the commands accordingly.

$
export NAMESPACE=your-namespace
$
kubectl apply -f benchmarks/profiler/deploy/profile_sla_aic_dgdr.yaml -n $NAMESPACE

The Dynamo Operator will immediately begin processing your request.

Step 3: Monitor Progress

Watch the DGDR status:

$
# View status
$
kubectl get dgdr -n $NAMESPACE
$
$
# Detailed status
$
kubectl describe dgdr sla-aic -n $NAMESPACE
$
$
# Watch profiling job logs
$
kubectl logs -f job/profile-sla-aic -n $NAMESPACE

DGDR Status States:

• Pending: Initial state, preparing to profile
• Profiling: Running profiling job (20-30 seconds for AIC, 2-4 hours for online)
• Deploying: Generating and applying DGD configuration
• Ready: DGD successfully deployed and running
• Failed: Error occurred (check events for details)

Step 4: Access Your Deployment

Once the DGDR reaches Ready state, your model is deployed and ready to serve:

$
# Find the frontend service
$
kubectl get svc -n $NAMESPACE | grep trtllm-disagg
$
$
# Port-forward to access locally
$
kubectl port-forward svc/trtllm-disagg-frontend 8000:8000 -n $NAMESPACE
$
$
# Test the endpoint
$
curl http://localhost:8000/v1/models

Step 5 (Optional): Access the Planner Grafana Dashboard

If you want to monitor the SLA Planner’s decision-making in real-time, you can deploy the Planner Grafana dashboard.

$
kubectl apply -n monitoring -f deploy/observability/k8s/grafana-planner-dashboard-configmap.yaml

Follow the instructions in Dynamo Metrics Collection on Kubernetes to access the Grafana UI and select the Dynamo Planner Dashboard.

The dashboard displays:

• Worker Counts & GPU Usage: Current prefill/decode worker counts and cumulative GPU hours
• Observed Metrics: Real-time TTFT, ITL, request rate, and sequence lengths from Prometheus
• Predicted Metrics: Planner’s load predictions and recommended replica counts
• Correction Factors: How the planner adjusts predictions based on observed vs expected performance

DGDR Configuration Details

Required Fields

Optional Fields

SLA Configuration

The sla section defines performance requirements and workload characteristics:

1
sla:
2
  isl: 3000      # Average input sequence length (tokens)
3
  osl: 150       # Average output sequence length (tokens)
4
  ttft: 200      # Target Time To First Token (milliseconds, float)
5
  itl: 20        # Target Inter-Token Latency (milliseconds, float)

Choosing SLA Values:

• ISL/OSL: Based on your expected traffic patterns
• TTFT: First token latency target (lower = more GPUs needed)
• ITL: Token generation latency target (lower = more GPUs needed)
• Trade-offs: Tighter SLAs require more GPU resources

Profiling Methods

Choose between online profiling (real measurements, 2-4 hours) or offline profiling with AI Configurator (estimated, 20-30 seconds):

1
# Online Profiling (Default)
2
sweep:
3
  useAiConfigurator: false
4
5
# Offline Profiling (AI Configurator)
6
sweep:
7
  useAiConfigurator: true
8
  aicSystem: h200_sxm
9
  aicHfId: Qwen/Qwen3-32B
10
  aicBackendVersion: "0.20.0"

Hardware Configuration

For details on hardware configuration and GPU discovery options, see Hardware Configuration in SLA-Driven Profiling.

Advanced Configuration

Using Existing DGD Configs (Recommended for Custom Setups)

If you have an existing DynamoGraphDeployment config (e.g., from examples/backends/*/deploy/disagg.yaml or custom recipes), you can reference it via ConfigMap:

Step 1: Create ConfigMap from your DGD config file:

$
kubectl create configmap deepseek-r1-config \
>
  --from-file=disagg.yaml=/path/to/your/disagg.yaml \
>
  --namespace $NAMESPACE \
>
  --dry-run=client -o yaml | kubectl apply -f -

Step 2: Reference the ConfigMap in your DGDR:

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeploymentRequest
3
metadata:
4
  name: deepseek-r1
5
spec:
6
  model: deepseek-ai/DeepSeek-R1
7
  backend: sglang
8
9
  profilingConfig:
10
    profilerImage: "nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.6.1"
11
    configMapRef:
12
      name: deepseek-r1-config
13
      key: disagg.yaml  # Must match the key used in --from-file
14
    config:
15
      sla:
16
        isl: 4000
17
        osl: 500
18
        ttft: 300
19
        itl: 10
20
      sweep:
21
        useAiConfigurator: true
22
        aicSystem: h200_sxm
23
        aicHfId: deepseek-ai/DeepSeek-V3
24
        aicBackendVersion: "0.20.0"
25
26
  deploymentOverrides:
27
    workersImage: "nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.6.1"
28
29
  autoApply: true

What’s happening: The profiler uses the DGD config from the ConfigMap as a base template, then optimizes it based on your SLA targets. The controller automatically injects spec.model into deployment.model and spec.backend into engine.backend in the final configuration.

Inline Configuration (Simple Use Cases)

For simple use cases without a custom DGD config, provide profiler configuration directly. The profiler will auto-generate a basic DGD configuration from your model and backend:

1
profilingConfig:
2
  config:
3
    # SLA targets (required for profiling)
4
    sla:
5
      isl: 8000   # Input sequence length
6
      osl: 200    # Output sequence length
7
      ttft: 200.0 # Time To First Token (ms)
8
      itl: 10.0   # Inter-Token Latency (ms)
9
10
    # Hardware constraints (optional)
11
    hardware:
12
      minNumGpusPerEngine: 2
13
      maxNumGpusPerEngine: 8
14
      gpuType: h200_sxm
15
16
    # Profiling sweep settings (optional)
17
    sweep:
18
      prefillInterpolationGranularity: 16  # Number of samples for prefill ISL sweep
19
      decodeInterpolationGranularity: 6    # Number of samples for decode sweep

Note: engine.config is a file path to a DGD YAML file, not inline configuration. Use ConfigMapRef (recommended) or leave it unset to auto-generate.

Planner Configuration Passthrough

Add planner-specific settings:

1
profilingConfig:
2
  config:
3
    planner:
4
      plannerMinEndpoint: 2

Understanding Profiling Results

For details about the profiling process, performance plots, and interpolation data, see SLA-Driven Profiling Documentation.

Advanced Topics

Mocker Deployment

Instead of a real DGD that uses GPU resources, you can deploy a mocker deployment that uses simulated engines rather than GPUs. Mocker is available in all backend images and uses profiling data to simulate realistic GPU timing behavior. It is useful for:

• Large-scale experiments without GPU resources
• Testing Planner behavior and infrastructure
• Validating deployment configurations

To deploy mocker instead of the real backend, set useMocker: true:

1
spec:
2
  model: <model-name>
3
  backend: trtllm  # Real backend for profiling (vllm, sglang, or trtllm)
4
  useMocker: true  # Deploy mocker instead of real backend
5
6
  profilingConfig:
7
    profilerImage: "nvcr.io/nvidia/dynamo/trtllm-runtime:<image-tag>"
8
    ...
9
  autoApply: true

Profiling still runs against the real backend (via GPUs or AIC) to collect performance data. The mocker deployment then uses this data to simulate realistic timing behavior.

Using a Model Cache PVC (0.8.1 or later)

Starting in Dynamo 0.8.1, for large models, you can use a pre-populated PVC containing model weights instead of downloading from HuggingFace. See Model Cache PVC for configuration details.

DGDR Immutability

DGDRs are immutable - if you need to update SLAs or configuration:

1. Delete the existing DGDR: kubectl delete dgdr sla-aic
2. Create a new DGDR with updated specifications

Manual Deployment Control

There are two ways to manually control deployment after profiling:

Option 1: Use DGDR-Generated Configuration (Recommended)

Disable auto-deployment to review the generated DGD before applying:

1
spec:
2
  autoApply: false

Then manually extract and apply the generated DGD:

$
# Extract generated DGD from DGDR status
$
kubectl get dgdr sla-aic -n $NAMESPACE -o jsonpath='{.status.generatedDeployment}' | kubectl apply -f -
$
$
# Or save to file first for review/modification
$
kubectl get dgdr sla-aic -n $NAMESPACE -o jsonpath='{.status.generatedDeployment}' > my-dgd.yaml
$
$
vi my-dgd.yaml
$
kubectl apply -f my-dgd.yaml -n $NAMESPACE

The generated DGD includes optimized configurations and the SLA planner component. The required planner-profile-data ConfigMap is automatically created when profiling completes, so the DGD will deploy successfully.

Option 2: Use Standalone Planner Templates (Advanced)

For advanced use cases, you can manually deploy using the standalone planner templates in examples/backends/*/deploy/disagg_planner.yaml:

$
# After profiling completes, profiling data is automatically stored in ConfigMaps
$
$
# OPTIONAL: Inspect profiling results stored in ConfigMaps
$
# View the generated DGD configuration
$
kubectl get configmap dgdr-output-<dgdr-name> -n $NAMESPACE -o yaml
$
$
# View the planner profiling data (JSON format)
$
kubectl get configmap planner-profile-data -n $NAMESPACE -o yaml
$
$
# Update the PROMETHEUS_ENDPOINT environment variable in the planner template
$
# to match your cluster's Prometheus service location (see comments in the template)
>
>
# Update backend planner manifest as needed, then deploy
>
kubectl apply -f examples/backends/<backend>/deploy/disagg_planner.yaml -n $NAMESPACE

Note: The standalone templates are provided as examples and may need customization for your model and requirements. The DGDR-generated configuration (Option 1) is recommended as it’s automatically tuned to your profiling results and SLA targets.

Important - Prometheus Configuration: The planner queries Prometheus to get frontend request metrics for scaling decisions. If you see errors like “Failed to resolve prometheus service”, ensure the PROMETHEUS_ENDPOINT environment variable in your planner configuration correctly points to your Prometheus service. See the comments in the example templates for details.

Relationship to DynamoGraphDeployment (DGD)

• DGDR: High-level “intent” - what you want deployed
• DGD: Low-level “implementation” - how it’s deployed

The DGDR controller generates a DGD that:

• Uses optimal TP configurations from profiling
• Includes SLA planner for autoscaling
• Has deployment and engine settings tuned for your SLAs

The generated DGD is tracked via labels:

1
metadata:
2
  labels:
3
    dgdr.nvidia.com/name: sla-aic
4
    dgdr.nvidia.com/namespace: your-namespace

Accessing Detailed Profiling Artifacts

By default, profiling jobs save essential data to ConfigMaps for planner integration. For advanced users who need access to detailed artifacts (logs, performance plots, AIPerf results, etc), configure the DGDR to use dynamo-pvc. This is optional and will not affect the functionality of profiler or Planner.

What’s available in ConfigMaps (always created):

• Generated DGD configuration
• Profiling data for Planner (.json files)

What’s available in PVC if attached to DGDR (optional):

• Performance plots (PNGs)
• DGD configuration and logs of all services for each profiled deployment
• AIPerf profiling artifacts for each AIPerf run
• Raw profiling data (.npz files)
• Profiler log

Setup:

1. Set up the benchmarking PVC:

$
export NAMESPACE=your-namespace
$
deploy/utils/setup_benchmarking_resources.sh

2. Add outputPVC to your DGDR’s profilingConfig:

1
spec:
2
  profilingConfig:
3
    outputPVC: "dynamo-pvc"
4
    config:
5
      # ... rest of config

3. After profiling completes, access results:

$
kubectl apply -f deploy/utils/manifests/pvc-access-pod.yaml -n $NAMESPACE
$
kubectl wait --for=condition=Ready pod/pvc-access-pod -n $NAMESPACE --timeout=60s
$
kubectl cp $NAMESPACE/pvc-access-pod:/data ./profiling-results
$
kubectl delete pod pvc-access-pod -n $NAMESPACE

Troubleshooting

Quick Diagnostics

$
# Check DGDR status and events
$
kubectl describe dgdr sla-aic -n $NAMESPACE
$
$
# Check operator logs
$
kubectl logs -n $NAMESPACE -l app.kubernetes.io/name=dynamo-operator --tail=100
$
$
# Check profiling job logs
$
kubectl logs -l job-name=profile-sla-aic -n $NAMESPACE

Common Issues

Configuration Reference

For comprehensive documentation of all DGDR configuration options, see the DGDR Configuration Reference.

This includes detailed explanations of:

• SLA Configuration: ISL, OSL, TTFT, ITL with use cases and trade-offs
• Hardware Configuration: GPU constraints and search space control
• Sweep Configuration: Profiling behavior and interpolation settings
• AI Configurator Configuration: System types, model mappings, backend versions
• Planner Configuration: Autoscaling and adjustment parameters
• Complete Examples: Full DGDRs for online, offline (AIC), and MoE profiling

Related Documentation

• DGDR API Reference
• Pre-Deployment Profiling Details
• SLA Planner Architecture
• Dynamo Operator Guide