Graceful Shutdown Architecture

This document describes the internals of how Dynamo components handle shutdown signals to ensure in-flight requests complete successfully and resources are properly cleaned up.

This is an architecture reference. For how to tune graceful shutdown for a deployment — grace periods, drain windows, and enabling migration — see the Graceful Shutdown use-case guide.

Overview

Graceful shutdown in Dynamo ensures that:

1. Routing stops quickly - Endpoints are unregistered from discovery first
2. In-flight requests can finish - Workers keep serving during a short grace period
3. Endpoints drain - After the grace period, endpoints are invalidated and optionally wait for in-flight work
4. Resources are cleaned up - Engines, connections, and temporary files are released
5. Pods restart cleanly - Exit codes signal Kubernetes for proper restart behavior

Signal Handling

All Dynamo components handle Unix signals for graceful shutdown:

Implementation

Each component registers signal handlers at startup:

1
def signal_handler():
2
    asyncio.create_task(graceful_shutdown(runtime, endpoints))
3
4
for sig in (signal.SIGTERM, signal.SIGINT):
5
    loop.add_signal_handler(sig, signal_handler)

The graceful_shutdown() function:

1. Logs the shutdown signal
2. Unregisters all endpoints from discovery
3. Waits for a configurable grace period (DYN_GRACEFUL_SHUTDOWN_GRACE_PERIOD_SECS, default 5s)
4. Calls runtime.shutdown() to invalidate endpoints and stop accepting new requests
5. Waits for in-flight requests (based on graceful_shutdown per endpoint)
6. Returns to allow cleanup to proceed

The aggregate wait in runtime.shutdown() is bounded by DYN_RUNTIME_GRACEFUL_SHUTDOWN_TIMEOUT_SECS, which defaults to 900 seconds (15 minutes). If endpoint draining exceeds this timeout, Dynamo logs the remaining graceful endpoint count and proceeds with runtime teardown.

Endpoint Draining

After the grace period, runtime.shutdown() invalidates endpoints so no new requests are accepted. The behavior for in-flight requests depends on the graceful_shutdown parameter when serving the endpoint.

Configuration

When registering an endpoint, the graceful_shutdown parameter controls draining behavior:

1
generate_endpoint.serve_endpoint(
2
    handler.generate,
3
    graceful_shutdown=True,  # Wait for all requests to finish
4
    metrics_labels=[("model", model_name)],
5
    health_check_payload=health_check_payload,
6
)

Component-Specific Behavior

Frontend HTTP Draining

The Frontend uses a separate HTTP drain state before it cancels the distributed runtime:

1. Mark the server as draining.
2. Return 503 Service Unavailable from /health and reject new OpenAI-compatible requests with 503.
3. Keep /live at 200 OK so liveness probes do not restart the process while requests drain.
4. Wait for admitted response bodies, including streaming responses, to complete.
5. Continue tracking accepted /v1/realtime WebSocket sessions until their tasks exit, even though the HTTP upgrade response has completed.
6. After DYN_HTTP_GRACEFUL_SHUTDOWN_TIMEOUT_SECS expires (default 5 seconds), enter the stopping state and cancel runtime state.

This separates readiness from liveness: traffic is removed promptly without turning an intentional drain into a restart loop.

Migration Integration

Backend workers always use graceful_shutdown=True, meaning they wait for in-flight requests to complete until the engine is stopped. Request migration is configured at the frontend level via --migration-limit:

• When migration is enabled at the frontend, disconnected streams from failed workers are automatically retried on healthy workers
• Workers don’t need to know about migration configuration - they simply complete their work or signal incomplete streams
• See Request Migration Architecture for details on how migration works

Resource Cleanup

After endpoint draining, components clean up their resources in finally blocks:

vLLM Worker Cleanup

1
finally:
2
    logger.debug("Cleaning up worker")
3
    handler.cleanup()

The handler’s cleanup() method:

• Removes temporary directories (LoRA adapters, etc.)
• Releases engine resources

SGLang Worker Cleanup

1
def cleanup(self) -> None:
2
    # Cancel pending consume tasks
3
    for task in self._consume_tasks:
4
        if not task.done():
5
            task.cancel()
6
    self._consume_tasks.clear()
7
8
    # Shutdown engine
9
    self.engine.shutdown()

TensorRT-LLM Worker Cleanup

1
async def cleanup(self):
2
    if self._llm:
3
        try:
4
            self._llm.shutdown()
5
        except Exception as e:
6
            logging.error(f"Error during cleanup: {e}")
7
        finally:
8
            self._llm = None

Error-Initiated Shutdown

Workers can initiate graceful shutdown when fatal errors occur:

Engine Health Monitoring (vLLM)

The VllmEngineMonitor continuously checks engine health:

1
async def _check_engine_health(self):
2
    while True:
3
        try:
4
            await self.engine_client.check_health()
5
            await asyncio.sleep(HEALTH_CHECK_INTERVAL)  # 2 seconds
6
        except EngineDeadError as e:
7
            logger.error(f"Health check failed: {e}")
8
            self._shutdown_engine()
9
            self.runtime.shutdown()
10
            os._exit(1)

Configuration:

• HEALTH_CHECK_INTERVAL: 2 seconds between checks
• ENGINE_SHUTDOWN_TIMEOUT: 30 seconds max for engine shutdown

Fatal Error Handling (TensorRT-LLM)

1
async def _initiate_shutdown(self, error: Exception):
2
    logging.warning(f"Initiating graceful shutdown due to: {error}")
3
4
    try:
5
        if self.runtime:
6
            self.runtime.shutdown()
7
        if self.engine:
8
            await self.engine.cleanup()
9
    except Exception as cleanup_error:
10
        logging.error(f"Error during graceful shutdown: {cleanup_error}")
11
    finally:
12
        logging.critical("Forcing process exit for restart")
13
        os._exit(1)

Kubernetes Integration

Pod Termination Flow

1. Kubernetes sends SIGTERM to the pod
2. Dynamo initiates graceful shutdown
3. Dynamo operator-created pods have terminationGracePeriodSeconds to complete (default: 60s)
4. If not terminated, Kubernetes sends SIGKILL

Health Check Integration

Kubernetes uses health endpoints to determine pod readiness:

• During shutdown: Endpoints become unavailable
• Readiness probe fails: Traffic stops routing to the pod
• Graceful draining: Existing requests complete

Related Documentation

• Graceful Shutdown - How to tune grace periods and drain windows (use-case guide)
• Request Migration Architecture - How requests migrate during shutdown
• Request Cancellation Architecture - Canceling in-flight requests
• Health Check Reference - Liveness and readiness endpoints