Fault Tolerance

Graceful Shutdown

Let workers finish in-flight requests and release resources cleanly when a pod is terminated.

View as Markdown

When Kubernetes terminates a pod (rollout, scale-down, node drain), Dynamo workers stop accepting new requests, keep serving in-flight ones through a grace period, then release engine and connection resources before exiting. This is on by default — every component handles SIGTERM/SIGINT and drains automatically. The steps below tune how long it waits and make sure interrupted requests are recovered.

The knobs are three timeouts plus enabling migration. The default flow: endpoints unregister from discovery immediately, workers serve for a short grace period, then endpoints drain (bounded by a timeout) before resources are cleaned up.

How it works: the signal handlers, the graceful_shutdown() sequence, per-backend cleanup() code, and error-initiated shutdown are documented in Graceful Shutdown Architecture.

1

Set the pod termination grace period

Kubernetes gives a terminating pod terminationGracePeriodSeconds to exit before it sends SIGKILL. Dynamo operator-created pods default to 60 seconds. Set a longer value when expected generation time or high utilization can keep admitted requests active beyond that window:

1apiVersion: nvidia.com/v1alpha1
2kind: DynamoGraphDeployment
3spec:
4  services:
5    VllmWorker:
6      extraPodSpec:
7        terminationGracePeriodSeconds: 180  # allow time for request draining

Rough guidance:

WorkloadSuggested terminationGracePeriodSeconds
Short requests (< 10s)60s
Long generation (> 30s) or high utilization120s+
2

Tune the drain windows

Three environment variables control Dynamo’s internal draining. Set the HTTP timeout on the Frontend and the runtime values on worker components:

VariableDefaultPurpose
DYN_HTTP_GRACEFUL_SHUTDOWN_TIMEOUT_SECS5How long the Frontend waits for admitted HTTP and WebSocket inference requests to finish before it cancels runtime state.
DYN_GRACEFUL_SHUTDOWN_GRACE_PERIOD_SECS5How long workers keep serving after endpoints unregister from discovery, before endpoints are invalidated.
DYN_RUNTIME_GRACEFUL_SHUTDOWN_TIMEOUT_SECS900Upper bound on waiting for in-flight requests to finish. If draining exceeds this, Dynamo logs the remaining endpoint count and tears down anyway.

The defaults are sound for most deployments. Raise the relevant timeout only for long generations or sustained high utilization. Keep every internal timeout below terminationGracePeriodSeconds so Dynamo can finish its own cleanup before Kubernetes force-kills the pod.

3

Enable migration so drained requests retry

Draining lets current requests finish, but a request interrupted by an unexpected worker loss still needs somewhere to go. Enable request migration on the Frontend so disconnected streams are retried on healthy workers. Set DYN_MIGRATION_LIMIT in the Frontend env: (or --migration-limit in its args:):

1  - name: Frontend
2    type: frontend
3    replicas: 1
4    podTemplate:
5      spec:
6        containers:
7        - name: main
8          image: ${RUNTIME_IMAGE}
9          env:
10          - name: DYN_MIGRATION_LIMIT
11            value: "3"                 # allow up to 3 migration attempts

Backend workers always drain with graceful_shutdown=True; they don’t need any migration configuration themselves. See Request Migration for the full walkthrough.

4

Verify graceful shutdown

Trigger a shutdown (for example kubectl delete pod <worker-pod> or a rollout) and watch the worker logs for the shutdown sequence:

INFO  Received shutdown signal, shutting down DistributedRuntime
INFO  DistributedRuntime shutdown complete
DEBUG Cleaning up worker

During Frontend shutdown, /health returns 503 so readiness routing stops, while /live remains 200 so Kubernetes does not restart the process during the drain. New OpenAI-compatible requests return 503, but admitted response bodies and accepted /v1/realtime WebSocket sessions continue until they finish or DYN_HTTP_GRACEFUL_SHUTDOWN_TIMEOUT_SECS expires.

During worker shutdown, endpoints unregister and stop receiving new work while admitted requests complete. If a pod receives SIGKILL before draining finishes, increase terminationGracePeriodSeconds (step 1) or lower the relevant internal timeout (step 2).

Custom workers

If you author your own worker with the Dynamo SDK, the graceful_shutdown parameter on serve_endpoint() controls whether that endpoint waits for in-flight requests (True) or returns immediately (False). Backend workers default to True. For the parameter, the shutdown sequence, and per-backend cleanup patterns, see Graceful Shutdown Architecture and the Writing Python Workers guide.