Operations

Customize Health Probes

Understand the operator's default liveness, readiness, and startup probes and override them in a DynamoGraphDeployment when defaults do not fit.
View as Markdown

The Dynamo operator attaches sensible health probes to every component, so most deployments need no probe configuration at all. Override them when a large model needs a longer startup window, or when your engine exposes health differently. This page covers the defaults and how to override them in a DynamoGraphDeployment (DGD).

This is a how-to for an existing deployment. For the health endpoints themselves and local-process health signals, see Health Check Reference.

What the operator sets by default

You do not author these — the operator injects them. Knowing them tells you when an override is needed.

Frontend readiness — exec curl against /health, initial delay 60s, period 60s, timeout 30s, failure threshold 10.

Worker probes (served on the system port, 9090):

ProbePathPeriodTimeoutFailure threshold
Liveness/live5s30s1
Readiness/health10s30s60
Startup/live10s5s720 (≈2 hours)

The startup probe’s 2-hour budget (10s × 720) is what tolerates long model downloads and engine warm-up. User-specified probes always take precedence over these defaults.

When to override

  • Model startup exceeds 2 hours (very large models on slow storage). Raise the startup probe’s failureThreshold: failureThreshold = expected_startup_seconds / periodSeconds.
  • Your engine exposes health on a different path or port. Point the probe at it.
  • You want faster failure detection for a small, fast-loading model. Lower the thresholds.

Prefer Model Caching over a longer startup probe when the bottleneck is download time — caching removes the per-pod download instead of just waiting longer for it.

Override in a DGD

Set livenessProbe and readinessProbe on the main container in the component’s podTemplate. They are standard Kubernetes Probe objects:

1spec:
2  components:
3  - name: VllmDecodeWorker
4    type: worker
5    podTemplate:
6      spec:
7        containers:
8        - name: main
9          readinessProbe:
10            httpGet:
11              path: /health
12              port: 9090
13            periodSeconds: 10
14            failureThreshold: 120      # double the default startup tolerance
15          livenessProbe:
16            httpGet:
17              path: /live
18              port: 9090
19            periodSeconds: 10
20            failureThreshold: 3

Multinode note

For multinode deployments the operator adjusts probes by backend and node role — for example, it removes worker-node probes for vLLM Ray workers and replaces the TensorRT-LLM worker readiness probe with a TCP check on the SSH port. See Multinode Deployments for the per-backend behavior before overriding probes on a multinode service.