Request Migration

Enable migration on the Frontend

Set --migration-limit N — the maximum number of times a single request may be migrated to another worker. Configure it on the Frontend component, either as an args: entry or via the DYN_MIGRATION_LIMIT environment variable in env::

1
  - name: Frontend
2
    type: frontend
3
    replicas: 1
4
    podTemplate:
5
      spec:
6
        containers:
7
        - name: main
8
          image: ${RUNTIME_IMAGE}
9
          env:
10
          - name: DYN_MIGRATION_LIMIT
11
            value: "3"                 # allow up to 3 migration attempts

The limit applies to all models served by this frontend. A value of 0 (the default) disables migration. Start with 3 — enough to survive transient worker loss without retrying indefinitely.

Bound memory for long sequences

Optional. Migration caches token state for each in-flight request so it can be replayed on a new worker. For very long sequences this cache grows, so you can cap it with --migration-max-seq-len (or DYN_MIGRATION_MAX_SEQ_LEN) on the Frontend:

1
          env:
2
          - name: DYN_MIGRATION_MAX_SEQ_LEN
3
            value: "32000"

Once a request’s total sequence length (prompt + generated tokens) strictly exceeds this limit, migration is disabled for that request and token tracking stops. Exactly at the limit is still migratable. The check runs both at request start (prompt length) and during generation. Leave it unset for no limit.

Tune temporary worker inhibition

After a request-path failure, each runtime client temporarily removes the failed worker from its local routing set while service discovery propagates the authoritative worker state. The default local inhibition window is 5 seconds. Usually no change is required.

To shorten, extend, or disable that local protection, set DYN_RUNTIME_INHIBITED_DURATION_SECS on the client process, such as the Frontend or another component that dispatches requests:

1
          env:
2
            - name: DYN_RUNTIME_INHIBITED_DURATION_SECS
3
              value: "10"

Set the value to 0 to disable local inhibition. The runtime reads it once when the first client is initialized, so restart the process after changing it. Discovery updates can restore or remove a worker before the window expires. Direct dispatch bypasses local inhibition and continues to honor an upstream-selected worker while that worker remains in service discovery.

See Runtime Configuration for the field reference and Distributed Runtime for the routing behavior.

Verify migrations

Migration counters are exposed on the Frontend’s /metrics endpoint (default port 8000). Port-forward the Frontend and scrape for the migration counter:

$
kubectl port-forward svc/<deployment-name>-frontend 8000:8000 -n ${NAMESPACE}
$
curl -s localhost:8000/metrics | grep dynamo_frontend_model_migration

• dynamo_frontend_model_migration_total — total migrations, labeled by model and migration_type (new_request vs ongoing_request).
• dynamo_frontend_model_migration_max_seq_len_exceeded_total — times migration was disabled because a request exceeded --migration-max-seq-len.

A rising max_seq_len_exceeded counter means your limit may be too low for the workload. For full field definitions and labels, see Migration in the Metrics Catalog.