跳到导航
Model DeploymentModel Loading

Restore Workers with Dynamo Snapshot

Capture an initialized TensorRT-LLM worker and verify that a restored worker serves requests

以 Markdown 格式查看

NVIDIA Dynamo Snapshot restores initialized workers so new replicas can skip model initialization. This walkthrough captures a TensorRT-LLM worker, restores it, and sends a request through the Dynamo frontend using a single-GPU Qwen3 example.

Experimental. Dynamo 1.5 uses Snapshot 0.1.0, which supports single-GPU workers only. Consult the backend compatibility matrix before using another backend or topology.

Prerequisites

  • A Kubernetes cluster with NVIDIA GPU support and one full GPU on an x86_64 node, with MIG disabled. Review Snapshot’s driver, runtime, and privilege requirements.
  • A storage class that supports ReadWriteMany (RWX) for Snapshot artifacts, and a writable model-cache PersistentVolumeClaim (PVC) in the workload namespace. These are separate volumes. model-cache also needs ReadWriteMany unless the capture and serving Pods run on the same node. See Model Storage.
  • helm, kubectl, curl, and jq, plus access to the runtime image registry and Hugging Face. The capture Pod downloads the model if it is not cached.

The example selects runtimeClassName: nvidia. Use your cluster’s RuntimeClass name, or omit the field if the default runtime provides GPU support.

Example

The Qwen3-0.6B example needs one GPU, a 10 GiB model cache, and 1 TiB of Snapshot artifact storage.

Use the same Dynamo release for the platform chart and runtime image. Replace the storage class with your cluster’s RWX storage class:

export RELEASE_VERSION=1.5.1
export PLATFORM_NAMESPACE=dynamo-system
export NAMESPACE=dynamo-snapshot
export RWX_STORAGE_CLASS=your-rwx-storage-class
export TRTLLM_IMAGE="nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:${RELEASE_VERSION}"
export RECIPE=qwen3-0.6b
export MODEL=Qwen/Qwen3-0.6B
export DGD=qwen3-0-6b-trtllm-snapshot-restore
export SNAPSHOT_STORAGE_SIZE=1Ti
export CAPTURE_TIMEOUT=20m
export RESTORE_TIMEOUT=5m

Provision model-cache in NAMESPACE before deploying. The storage sizes above are starting allocations; check free space before capturing, especially if you retain multiple artifacts.

1

Install the platform with Snapshot

For a test installation, enable the bundled Snapshot chart and the Dynamo integration together. The platform chart selects the matching Snapshot version; no separate Snapshot version is needed.

helm upgrade --install dynamo-platform \
  "https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz" \
  --namespace "$PLATFORM_NAMESPACE" --create-namespace --reuse-values \
  --set global.snapshot.install=true \
  --set dynamo-operator.checkpoint.enabled=true \
  --set-string snapshot.storage.pvc.storageClass="$RWX_STORAGE_CLASS" \
  --set-string snapshot.storage.pvc.size="$SNAPSHOT_STORAGE_SIZE" \
  --wait --timeout 10m
kubectl rollout status deployment/dynamo-platform-snapshot-operator \
  -n "$PLATFORM_NAMESPACE" --timeout=300s
kubectl rollout status daemonset/dynamo-platform-snapshot-agent \
  -n "$PLATFORM_NAMESPACE" --timeout=300s
kubectl get crd snapshotjobs.nvidia.com podsnapshots.nvidia.com

For an existing platform release, use its name, namespace, and version. --reuse-values preserves its settings. The Snapshot chart creates an RWX artifact PVC; use snapshot.storage.pvc values to customize its size or reuse an existing claim. See Snapshot storage configuration.

If Snapshot is already managed separately, keep global.snapshot.install=false and omit the snapshot.storage.pvc settings. Verify that its APIs and agents are ready before enabling dynamo-operator.checkpoint.enabled. Do not install a second cluster-wide Snapshot operator. Separate installation is recommended for production; see Snapshot installation.

2

Deploy the capture and serving configuration

Download the example and set the runtime image:

export EXAMPLE_URL="https://raw.githubusercontent.com/ai-dynamo/dynamo/release/1.5.1/examples/backends/trtllm/deploy/snapshot"
curl --fail --location "$EXAMPLE_URL/$RECIPE.yaml" -o snapshot-template.yaml
sed "s|my-registry/tensorrtllm-runtime:my-tag|${TRTLLM_IMAGE}|g" \
  snapshot-template.yaml > snapshot-restore.yaml
kubectl get pvc model-cache -n "$NAMESPACE"
kubectl apply -n "$NAMESPACE" -f snapshot-restore.yaml

The example sets experimental.checkpoint.enabled: true and startupPolicy: WaitForCheckpoint. Dynamo creates a SnapshotJob to initialize and capture the worker. Serving replicas stay at zero until capture completes, then start from the resulting PodSnapshot.

The default policy, Immediate, starts serving workers cold while capture runs. Only Pods created after capture completes restore; existing workers are not restarted when the snapshot becomes ready.

3

Verify capture and restore

Wait for the DGD to report a usable artifact, then verify the worker itself:

kubectl wait "dgd/$DGD" -n "$NAMESPACE" \
  --for='jsonpath={.status.checkpoints.TRTLLMWorker.ready}=true' --timeout="$CAPTURE_TIMEOUT"
export SNAPSHOT_NAME=$(kubectl get "dgd/$DGD" -n "$NAMESPACE" \
  -o jsonpath='{.status.checkpoints.TRTLLMWorker.checkpointName}')
kubectl get "podsnapshot/$SNAPSHOT_NAME" -n "$NAMESPACE"
kubectl wait "dgd/$DGD" -n "$NAMESPACE" \
  --for=condition=Ready --timeout="$RESTORE_TIMEOUT"
export WORKER_SELECTOR="nvidia.com/dynamo-graph-deployment-name=$DGD,nvidia.com/dynamo-component=TRTLLMWorker,"'!nvidia.com/snapshot-job'
kubectl wait pod -n "$NAMESPACE" -l "$WORKER_SELECTOR" \
  --for=condition=Ready --timeout="$RESTORE_TIMEOUT"
export WORKER_POD=$(kubectl get pod -n "$NAMESPACE" -l "$WORKER_SELECTOR" \
  -o jsonpath='{.items[0].metadata.name}')
kubectl wait "pod/$WORKER_POD" -n "$NAMESPACE" \
  --for=condition=nvidia.com/Restored --timeout="$RESTORE_TIMEOUT"
test "$(kubectl get "pod/$WORKER_POD" -n "$NAMESPACE" \
  -o jsonpath='{.metadata.annotations.nvidia\.com/restore-from}')" = "$SNAPSHOT_NAME"

The selector excludes the capture Pod. Success requires both nvidia.com/Restored=True and the expected nvidia.com/restore-from annotation; artifact readiness alone does not prove a successful restore.

4

Send a request to the restored worker

Forward the frontend port in one terminal:

kubectl port-forward -n "$NAMESPACE" "svc/$DGD-frontend" 8000:8000

In another terminal, set MODEL to the model name and send a request:

curl --fail --retry 6 --retry-all-errors --retry-delay 5 \
  http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d "$(jq -n --arg model "$MODEL" \
    '{model:$model,messages:[{role:"user",content:"What is the capital of France?"}],max_tokens:256,temperature:0,stream:false,chat_template_kwargs:{enable_thinking:false}}')" \
  | jq -e '.choices[0].message.content | select(test("Paris"; "i"))'

If a step fails

Inspect the capture conditions and workload events:

kubectl describe dgd "$DGD" -n "$NAMESPACE"
kubectl get snapshotjobs,podsnapshots -n "$NAMESPACE" \
  -l "nvidia.com/dynamo-graph-deployment-name=$DGD" -o yaml
kubectl get pods -n "$NAMESPACE"
  • Capture is pending: check GPU scheduling, image pulls, the model-cache PVC, and Snapshot agent readiness.
  • Capture fails: inspect the SnapshotJob failure reason and capture Pod logs. Keep the recipe’s OMPI_MCA_pml=ob1 and OMPI_MCA_btl=tcp,self settings.
  • Restore fails: inspect the worker’s nvidia.com/Restored condition and Snapshot troubleshooting.

Clean Up

Stop port forwarding, then delete the example:

kubectl delete -n "$NAMESPACE" -f snapshot-restore.yaml

The default deletionPolicy: Delete removes the DGD-managed capture and artifact. The model-cache PVC and the Snapshot installation remain available for other workloads. See the checkpoint configuration reference for reuse and retention rules.