Dynamo Graph Deployment on Kubernetes#
A DynamoGraphDeployment (DGD) is a custom resource that describes a
multi-pod Dynamo topology: a Frontend service and one or more Worker services.
The NVIDIA Dynamo Operator reconciles the resource into pods. For NIM 3.0,
deploy a DGD by using the operator and the validated recipe published with the
model in NGC.
Prerequisites#
Prepare a Kubernetes cluster with kubectl, Helm, and the NVIDIA GPU Operator
configured to expose GPUs to workloads. You also need:
GPU nodes that satisfy the SKU and GPU count in the selected recipe.
The NIM 3.0 DGD recipe for your model, GPU SKU, and workload from the model’s collection in the NGC Catalog.
An NGC API key for pulling the recipe’s NIM image and downloading model artifacts.
Important
Use the container image and model profile already specified by the published recipe. The recipe and image are validated together. Replacing the image or profile does not preserve that validated configuration.
Install the Dynamo Operator#
NIM 3.0 DGD recipes are validated with Dynamo Operator 1.2.0. The
dynamo-platform chart includes the required custom resource definitions. Do
not install a separate dynamo-crds chart or add component overrides.
helm upgrade --install dynamo-platform \
https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-1.2.0.tgz \
--namespace dynamo-system \
--create-namespace
Confirm that the operator is running and that the DGD custom resource is registered:
kubectl get pods -n dynamo-system
kubectl wait --for=condition=Established \
crd/dynamographdeployments.nvidia.com --timeout=60s
Prepare the Deployment Namespace#
Create the target namespace and the two secrets referenced by the published recipe. If the recipe uses different secret names, use the names in that recipe.
The Nemotron-3-Super recipes name the NGC API secret
nimcraft-ngc-api-key. The Nemotron-3-Ultra model-free NIM recipes name it
ngc-api. Set the name to match the downloaded recipe:
export NAMESPACE=<your-namespace>
export NGC_API_KEY=<your-ngc-api-key>
export NGC_API_SECRET_NAME=<api-secret-name-from-recipe>
kubectl create namespace "$NAMESPACE" \
--dry-run=client -o yaml | kubectl apply -f -
kubectl create secret docker-registry ngc-secret \
--namespace "$NAMESPACE" \
--docker-server=nvcr.io \
--docker-username='$oauthtoken' \
--docker-password="$NGC_API_KEY" \
--dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic "$NGC_API_SECRET_NAME" \
--namespace "$NAMESPACE" \
--from-literal=NGC_API_KEY="$NGC_API_KEY" \
--dry-run=client -o yaml | kubectl apply -f -
Review the Published Recipe#
The NIM 3.0 Nemotron-3-Super and Nemotron-3-Ultra recipes define release configurations rather than generic templates to customize. Both recipe families provide chat and agentic variants:
Model |
GPU |
Image |
Profile |
GPUs per Worker |
|---|---|---|---|---|
Nemotron-3-Super-120B-A12B |
B200 |
|
|
4 |
Nemotron-3-Super-120B-A12B |
H200 |
|
|
4 |
Nemotron-3-Ultra-550B-A55B |
B200 |
|
|
4 |
Nemotron-3-Ultra-550B-A55B |
H200 |
|
|
8 |
All recipes use nvidia.com/v1beta1, one Frontend replica with KV-aware
routing, and NIM_DYNAMO_WORKER=1 for Workers. The published default is two
Worker replicas. The Super B200 chat recipe explicitly sets four. Super
Workers use 20 GiB shared memory, while Ultra Workers use 64 GiB.
The Super model-specific image selects its baked manifest profile with
NIM_MODEL_PROFILE and does not set NIM_MODEL_PATH. The Ultra model-free
image sets NIM_MODEL_PATH and carries its tensor-parallel and engine tuning
in NIM_PASSTHROUGH_ARGS. Preserve those model-selection fields and all
recipe tuning.
Frontend startup and readiness probes use /v1/health/ready on port 8000.
Worker startup and readiness probes use /v1/health/ready on port 9090, and
Worker liveness uses /live on port 9090.
The published recipes do not define a PersistentVolumeClaim, cache volume,
or cache mount.
Apply the Recipe#
Download the recipe from the model’s NGC collection, then apply that file to the prepared namespace. Do not substitute an internal repository example for the published recipe.
export DGD_RECIPE=<downloaded-recipe.yaml>
kubectl apply -f "$DGD_RECIPE" --namespace "$NAMESPACE"
The release does not publish a LoRA DGD recipe or a public multi-worker Kubernetes LoRA procedure. For the supported single-container LoRA controls, refer to Fine-Tuning with LoRA.
Verify the Deployment#
Get the DGD name from the downloaded recipe and wait for the Frontend and all Worker pods to become ready:
export DGD_NAME=<metadata.name-from-recipe>
kubectl get dynamographdeployment "$DGD_NAME" \
--namespace "$NAMESPACE"
kubectl get pods --namespace "$NAMESPACE" \
-l "nvidia.com/dynamo-graph-deployment-name=$DGD_NAME" --watch
Confirm that each worker launched through Dynamo:
kubectl logs <worker-pod> --namespace "$NAMESPACE" --tail 1000 \
| grep "Starting Dynamo worker: python -m dynamo.vllm"
Forward the operator-created Frontend service:
kubectl port-forward \
"service/$DGD_NAME-frontend" 8000:8000 \
--namespace "$NAMESPACE"
In another terminal, wait for frontend readiness and then list the registered model:
curl --fail http://localhost:8000/v1/health/ready
curl --fail http://localhost:8000/v1/models
Use the id returned by /v1/models in an inference request:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "<model-id>",
"messages": [{"role": "user", "content": "Hello! How are you?"}],
"max_tokens": 100
}'
Remove the Deployment#
Delete the DGD custom resource:
kubectl delete dynamographdeployment "$DGD_NAME" \
--namespace "$NAMESPACE"
To remove the operator when no other DGD uses it:
helm uninstall dynamo-platform --namespace dynamo-system
Note
The NIM-LLM Helm chart does not support DGD mode in this release. Use the published DGD recipe with the Dynamo Operator.
Next Steps#
Refer to the following resources:
For Dynamo-specific failures, refer to Troubleshooting Dynamo Mode.
For router and worker controls, refer to Dynamo Performance Tuning.