Installation Guide

This guide walks you through installing everything needed to deploy models with Dynamo on Kubernetes. Follow the steps in order — each builds on the previous one.

Prerequisites

Before you begin, make sure you have Helm v3 or later and choose the accelerator type in your cluster:

Verify the client tools:

kubectl version --client
helm version

Overview

Every Dynamo deployment requires accelerator support and the Dynamo Platform. NVIDIA GPU clusters can use the NVIDIA GPU Operator. Intel GPU clusters use the Intel resource driver with DRA. The Dynamo Platform installation is the same after the cluster exposes its accelerator resources. Everything else is optional.

Grove + KAI Scheduler — Grove is the default multinode orchestrator. The operator returns a hard error on multinode deployments if neither Grove nor LeaderWorkerSet (LWS) is available. KAI Scheduler is optional but recommended alongside Grove for GPU-aware scheduling. See Multinode Orchestration for details.

Network Operator / RDMA — Without RDMA, disaggregated inference falls back to TCP automatically, but with severe performance degradation (~98s TTFT vs ~200-500ms with RDMA). Required for any production disaggregated deployment. Setup is cloud-provider-specific — see RDMA Setup and your cloud provider guide.

kube-prometheus-stack — Required for the Planner’s sla optimization mode (it reads live TTFT/ITL metrics from Prometheus). Also required for KEDA/HPA-based autoscaling. The Planner’s throughput mode can function without it using internal queue depth signals, but metrics-driven features will not work. See Metrics for details.

Shared storage — Prevents each pod from downloading model weights independently. Without it, large models (>70B) take hours to download per pod, and many replicas will hit HuggingFace rate limits. Not enforced by the operator — this is an operational concern. See the Model Storage Overview to choose a storage backend.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace
  # Add a trailing \ above before uncommenting the following setting.
  # --set driver.enabled=false

kubectl get pods --namespace gpu-operator

export NAMESPACE=dynamo-system
export RELEASE_VERSION=1.2.1  # match a version from https://github.com/ai-dynamo/dynamo/releases

helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-$RELEASE_VERSION.tgz
helm install dynamo-platform dynamo-platform-$RELEASE_VERSION.tgz \
  --namespace $NAMESPACE \
  --create-namespace
  # Note: add \ to --create-namespace above when uncommenting any optional flags below
  #
  # Grove + KAI Scheduler — uncomment if using multinode or disaggregated inference.
  # Option A (install=true): Dynamo installs and manages Grove/KAI as bundled subcharts (dev/testing):
  # --set "global.grove.install=true" \
  # --set "global.kai-scheduler.install=true" \
  # --set-string "kai-scheduler.scheduler.args.default-staleness-grace-period=-1s" \
  # Option B (enabled=true): Grove/KAI are already installed externally (production):
  # --set "global.grove.enabled=true" \
  # --set "global.kai-scheduler.enabled=true" \
  #
  # kube-prometheus-stack — uncomment if Prometheus is installed (required for Planner sla mode and autoscaling):
  # --set "dynamo-operator.dynamo.metrics.prometheusEndpoint=http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090"

kubectl get clusterrolebinding -o json | \
  jq -r '.items[] | select(.metadata.name | contains("dynamo-operator-manager")) |
  "Cluster-wide operator found in namespace: \(.subjects[0].namespace)"'

# Check CRDs
kubectl get crd | grep dynamo
# Expected: dynamographdeployments, dynamocomponentdeployments, dynamographdeploymentrequests, etc.
# Check operator and platform pods
kubectl get pods -n $NAMESPACE
# Expected: dynamo-operator-* and nats-* pods all Running

helm repo add volcano-sh https://volcano-sh.github.io/helm-charts
helm repo update
helm install volcano volcano-sh/volcano -n volcano-system --create-namespace

export LWS_VERSION=0.10.0
helm install lws oci://registry.k8s.io/lws/charts/lws \
  --version=$LWS_VERSION \
  --namespace lws-system \
  --create-namespace \
  --set gangSchedulingManagement.schedulerProvider=volcano \
  --wait --timeout 300s

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
  --set-json 'prometheus.prometheusSpec.podMonitorNamespaceSelector={}' \
  --set-json 'prometheus.prometheusSpec.probeNamespaceSelector={}'

./deploy/pre-deployment/pre-deployment-check.sh

Upgrade Dynamo

Dynamo supports Helm upgrades and downgrades within the installed minor release and between adjacent minor releases. Upgrade from N.M to N.(M+1) or downgrade from N.(M+1) to N.M. Move through every intermediate minor release for larger changes. For example, upgrade 1.2.x to 1.3.x, then 1.3.x to 1.4.x; a direct transition from 1.2.x to 1.4.x is unsupported.

1. Check the installed release and operator versions:

   helm list --namespace $NAMESPACE
   kubectl get deployments --namespace $NAMESPACE \
     -l app.kubernetes.io/name=dynamo-operator \
     -L app.kubernetes.io/version

2. Read the chart upgrade notes for every minor release you cross, including the Grove and KAI Scheduler requirements.

3. Set RELEASE_VERSION to the next target release, download its chart, and upgrade with your existing configuration in your-values.yaml. Include all settings from the current installation, including any originally passed with --set. For example, from 1.6.x to 1.7.0:

   export RELEASE_VERSION=1.7.0
   helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-$RELEASE_VERSION.tgz
   helm upgrade dynamo-platform dynamo-platform-$RELEASE_VERSION.tgz \
     --namespace $NAMESPACE \
     -f your-values.yaml

   Wait for the operator rollout before repeating this step for the next minor:

   kubectl rollout status deployment --namespace $NAMESPACE \
     -l app.kubernetes.io/name=dynamo-operator --timeout=120s

The upgrade-path check is enforced by target charts v1.7.0 and later when dynamo-operator.enabled=true. Upgrades or downgrades to earlier charts are not checked. The read-only pre-install,pre-upgrade Job runs before the operator Deployment changes; an initial installation passes when no Deployment exists. Helm and Flux run the hook directly; Argo CD maps it to PreSync. Raw kubectl apply, Argo CD selective sync, or disabled hooks do not enforce the check. A separately managed operator release owns its upgrade policy when the subchart is disabled. See the chart configuration for upgradePreflight.job settings, including images for internal registries and per-check overrides.

Unsupported Transitions

If you have separately validated an unsupported forward upgrade, set unsupportedUpgradeOverride to the target minor version. For example, to upgrade from 1.7.x to 1.9.x, set RELEASE_VERSION to the intended 1.9.x release and download that chart as above, then run:

helm upgrade dynamo-platform dynamo-platform-$RELEASE_VERSION.tgz \
  --namespace $NAMESPACE \
  -f your-values.yaml \
  --set-string dynamo-operator.upgradePreflight.checks.upgradePath.unsupportedUpgradeOverride=1.9

For an unsupported downgrade from 1.9.x to 1.7.x, set RELEASE_VERSION to the intended 1.7.x release and download that chart as above. Authorize the lower target minor with the downgrade override:

helm upgrade dynamo-platform dynamo-platform-$RELEASE_VERSION.tgz \
  --namespace $NAMESPACE \
  -f your-values.yaml \
  --set-string dynamo-operator.upgradePreflight.checks.upgradePath.unsupportedDowngradeOverride=1.7

The override must match the target operator’s N.M version. If you leave the override in your values, it has no effect on later upgrades or downgrades to a different minor. It applies only to the upgrade-path check; other preflight checks still run. It permits the Helm operation but does not make that upgrade or downgrade path supported.

Alternative: Install with AICR

NVIDIA AI Cluster Runtime (AICR) generates a version-locked recipe for the whole GPU and Dynamo stack, then deploys and validates it. Choose AICR when your cluster matches a validated Dynamo recipe and you want one reproducible install. Use the manual steps above for any other cluster. For the full walkthrough, see Install with AICR.

Next Steps

Your cluster is ready. Follow the Model Deployment guide to choose between applying a tuned DGD recipe, creating a DGD directly, or using DGDR to generate one.

Troubleshooting

“Dynamo upgrade-path preflight failed: Unsupported Dynamo upgrade” (or downgrade)

Cause: The target chart rejected a skipped-minor or major-version upgrade or downgrade. For example:

Dynamo upgrade-path preflight failed: Unsupported Dynamo upgrade from 1.5.0 to 1.7.0. ...

Find the retained Job and inspect its logs, replacing <job-name> with the name from the first command:

kubectl get jobs -n $NAMESPACE | grep check-upgrade-path
kubectl logs job/<job-name> -n $NAMESPACE

Move through every intermediate minor release. If you have validated the unsupported path, use the target-minor override under Unsupported Transitions.

“Installed operator Deployment has no app.kubernetes.io/version label”

Cause: The check cannot determine the installed operator version. Target-minor overrides cannot bypass this error.

Inspect the installed release and the Deployment’s metadata and image:

helm list --namespace $NAMESPACE
kubectl get deployments --namespace $NAMESPACE \
  -l app.kubernetes.io/name=dynamo-operator -o yaml

Confirm the version of the running operator from its image and release records. Restore the Deployment’s metadata.labels.app.kubernetes.io/version to that verified version in the Helm or GitOps configuration that manages it, then retry. Do not use the target version as the installed version; custom image tags or digests require checking the corresponding build records.

“VALIDATION ERROR: Cannot install cluster-wide Dynamo operator”

VALIDATION ERROR: Cannot install cluster-wide Dynamo operator.
Found existing namespace-restricted Dynamo operators in namespaces: ...

Cause: Attempting cluster-wide install on a shared cluster with existing namespace-restricted operators.

Solution: Remove the development/test namespace-restricted operators, then install one cluster-wide operator for production use.

CRDs already exist

Cause: Installing CRDs on a cluster where they’re already present (common on shared clusters).

Solution: The cluster-wide operator’s crd-apply init container manages CRDs automatically. If you encounter conflicts, check existing CRDs with kubectl get crd | grep dynamo.

Pods not starting?

kubectl describe pod <pod-name> -n $NAMESPACE
kubectl logs <pod-name> -n $NAMESPACE

Bitnami etcd “unrecognized” image?

ERROR: Original containers have been substituted for unrecognized ones.

Add to the helm install command:

--set "etcd.image.repository=bitnamilegacy/etcd" --set "etcd.global.security.allowInsecureImages=true"

Clean uninstall?

Before uninstalling, record any customized hook names with helm get hooks dynamo-platform -n $NAMESPACE. For the default release and operator names shown in this guide, remove the retained upgrade-path Job and its supporting resources after uninstalling:

# Uninstall the platform
helm uninstall dynamo-platform --namespace $NAMESPACE
# Remove upgrade-path hook resources that Helm does not uninstall
kubectl delete job,serviceaccount,role,rolebinding dynamo-platform-dynamo-operator-check-upgrade-path \
  --namespace $NAMESPACE --ignore-not-found
# List Dynamo CRDs
kubectl get crd | grep "dynamo.*nvidia.com"
# Delete each CRD
kubectl delete crd <crd-name>

Reference

• Helm Chart Configuration
• Deploy with DGD
• Kubernetes API Reference
• ModelExpress Server