> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/latest/llms.txt. For section-specific indexes, append /llms.txt to any section URL.

# Fault Tolerance

Dynamo provides comprehensive fault tolerance mechanisms to ensure reliable LLM inference in production deployments. This section covers the various strategies and features that enable Dynamo to handle failures gracefully and maintain service availability.

## Overview

Fault tolerance in Dynamo operates at multiple levels:

| Layer | Mechanism | Purpose |
|-------|-----------|---------|
| **Request** | Migration, Cancellation | Handle in-flight request failures |
| **Worker** | Health Checks, Graceful Shutdown | Detect and recover from worker failures |
| **Engine Process** | [Shadow Engine Failover](/dynamo/dev/kubernetes-deployment/advanced-platform/shadow-engine-failover) | Active/passive recovery for same-node engine-process failures in Kubernetes |
| **System** | Load Shedding, Request Rejection | Prevent system overload |
| **Infrastructure** | etcd HA, NATS resilience | Handle infrastructure component failures |

## Key Features

### Request Migration

When a worker fails during request processing, Dynamo can migrate in-progress requests to healthy workers. The migration system:

- Preserves partial generation state (accumulated tokens)
- Transparently continues generation on a new worker
- Maintains seamless token flow to clients

See [Request Migration](/dynamo/dev/user-guides/fault-tolerance/request-migration) for details.

### Request Cancellation

Dynamo supports canceling in-flight requests to free computational resources:

- Graceful stop signals for clean termination
- Kill signals for immediate termination
- Hierarchical cancellation propagation through request chains

See [Request Cancellation](/dynamo/dev/user-guides/fault-tolerance/request-cancellation) for details.

### Graceful Shutdown

Workers handle shutdown signals (SIGTERM/SIGINT) gracefully:

- Immediately stop accepting new requests
- Optionally drain in-flight requests before terminating
- Clean up resources (engines, connections, temp files)

See [Graceful Shutdown](/dynamo/dev/user-guides/fault-tolerance/graceful-shutdown) for details.

### Request Rejection (Load Shedding)

When workers are overloaded, Dynamo rejects new requests to prevent cascading failures:

- Configurable busy thresholds based on KV cache utilization
- Real-time worker load monitoring
- HTTP 503 responses with retry guidance

See [Request Rejection](/dynamo/dev/user-guides/fault-tolerance/request-rejection) for details.

### Health Checks

Dynamo provides multiple health check mechanisms:

- **HTTP Endpoints**: `/health` and `/live` endpoints for orchestration
- **Canary Health Checks**: Active monitoring via periodic test requests
- **Engine Monitoring**: Automatic shutdown on engine failure detection

See [Health Checks](/dynamo/dev/user-guides/observability-local/health-checks) for details.

### Local Worker Inhibition

After a request-path failure, a runtime client temporarily removes that worker from its local routing set while service discovery propagates the worker's state. `DYN_RUNTIME_INHIBITED_DURATION_SECS` sets the maximum local inhibition window and is read once during client initialization. The default is `5` seconds; set it to `0` to disable local inhibition. Discovery updates remain authoritative and can restore or remove workers sooner. Direct dispatch bypasses local inhibition and honors an upstream-selected worker as long as it remains present in service discovery.

Changes to the environment variable take effect the next time the process starts.

### Shadow Engine Failover

For Kubernetes deployments, [Shadow Engine Failover](/dynamo/dev/kubernetes-deployment/advanced-platform/shadow-engine-failover) can help with same-node recovery from unknown backend engine or software-process failures. It uses GPU Memory Service to keep model weights resident while a standby or replacement engine attaches. It does not preserve in-flight requests or KV cache state, and it does not cover GPU or node loss.

## Configuration Quick Reference

| Feature | Environment Variable | Default |
|---------|---------------------|---------|
| Worker health port | `DYN_SYSTEM_PORT` | `9090` |
| Canary health checks | `DYN_HEALTH_CHECK_ENABLED` | `false` |
| Canary wait time | `DYN_CANARY_WAIT_TIME` | `10` seconds |
| Health check timeout | `DYN_HEALTH_CHECK_REQUEST_TIMEOUT` | `3` seconds |
| Decode blocks threshold | `DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD` | unset |
| Prefill tokens threshold | `DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD` | unset |
| Prefill tokens fraction threshold | `DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD_FRAC` | unset |
| Local worker inhibition | `DYN_RUNTIME_INHIBITED_DURATION_SECS` | `5` seconds (`0` disables) |

## Failure Scenarios and Recovery

### Worker Pod Restart

1. Worker receives SIGTERM from Kubernetes
2. Endpoints are immediately invalidated (no new requests)
3. In-flight requests complete or migrate (based on configuration)
4. Resources are cleaned up
5. Pod restarts with fresh state

### Worker Crash (Unexpected)

1. etcd lease expires (TTL-based detection)
2. Client discovers endpoint removal via etcd watch
3. New requests route to remaining healthy workers
4. In-flight requests on crashed worker are migrated (if enabled)

### Network Partition

1. Worker loses connectivity to etcd/NATS
2. Lease keep-alive fails, lease eventually expires
3. Worker is removed from service discovery
4. Traffic reroutes to reachable workers

### GPU Failure

1. Engine health check detects GPU error (XID, OOM, etc.)
2. Worker initiates graceful shutdown
3. Runtime is shut down, engine cleaned up
4. Process exits with code 1 for pod restart

## Testing Fault Tolerance

Dynamo includes a comprehensive testing framework for validating fault tolerance:

- Request cancellation tests
- Migration tests with worker failures
- etcd HA failover tests
- Hardware fault injection (GPU XID, network partitions)

See [Fault Tolerance Testing](/dynamo/dev/user-guides/fault-tolerance/testing) for details.

## Related Documentation

- [Observability](/dynamo/dev/user-guides/observability-local) - Metrics and monitoring
- [Shadow Engine Failover](/dynamo/dev/kubernetes-deployment/advanced-platform/shadow-engine-failover) - Same-node active/passive engine failover for Kubernetes deployments
- [Distributed Runtime](/dynamo/dev/design-docs/distributed-runtime) - Service discovery architecture
- [Event Plane](/dynamo/dev/design-docs/communication-planes/event-plane) - Pub/sub for KV cache events and worker metrics
- [Discovery Plane](/dynamo/dev/design-docs/communication-planes/discovery-plane) - Service discovery and coordination