Request Rejection

Choose the load signals

Use the signals that match the workers you want to protect:

• Set --active-decode-blocks-threshold for decode workers. The value is the fraction of available KV cache blocks in use, from 0.0 through 1.0.
• Set --active-prefill-tokens-threshold for an absolute prefill-token limit.
• Set --active-prefill-tokens-threshold-frac when the prefill limit should scale with the worker’s max_num_batched_tokens value.

The checks use OR logic. If you configure more than one threshold, exceeding any configured threshold marks that data-parallel rank as busy. A worker is rejected only after all of its ranks are busy, and a request is shed only when every eligible worker is busy.

Start conservatively, observe the rejection rate and latency, and then raise the limits. For example, a decode-block threshold of 0.75 sheds earlier than 0.90.

Enable rejection on the Frontend

Configure thresholds on the Frontend component. Decode-block rejection also requires KV router mode because that mode initializes the worker-load metrics path. For long-output workloads, enable output-block tracking so generated tokens contribute to the observed KV cache load.

1
- name: Frontend
2
  type: frontend
3
  replicas: 1
4
  podTemplate:
5
    spec:
6
      containers:
7
        - name: main
8
          image: ${RUNTIME_IMAGE}
9
          command:
10
            - python3
11
            - -m
12
            - dynamo.frontend
13
          args:
14
            - --router-mode
15
            - kv
16
            - --active-decode-blocks-threshold
17
            - "0.85"
18
            - --router-track-output-blocks
19
            - --active-prefill-tokens-threshold
20
            - "10000"

--router-track-output-blocks is especially important for long generations. Without it, a workload can fill KV cache with generated tokens without crossing the load value observed by the router.

You can configure just one signal. For example, a prefill-only deployment can omit the decode-block threshold and KV-specific options. To disable rejection, remove all three threshold settings or set their environment-variable values to None.

See Frontend Configuration for the complete flag, environment-variable, and validation reference.

Adjust thresholds at runtime

Optional. Use the Frontend admin API to change thresholds for a discovered model without redeploying. The API is available when DYN_ENABLE_FRONTEND_ADMIN_API=true, which is the default.

$
kubectl port-forward svc/<deployment-name>-frontend 8000:8000 -n ${NAMESPACE}

Set one or more thresholds:

$
curl -X POST http://localhost:8000/busy_threshold \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "model": "Qwen/Qwen3-0.6B",
>
    "active_decode_blocks_threshold": 0.85,
>
    "active_prefill_tokens_threshold": 10000
>
  }'

Read the stored thresholds:

$
curl http://localhost:8000/busy_threshold

A numeric value enables only its corresponding check. The router applies the new configuration when the next worker-load or runtime-configuration update arrives, so the change might not affect a routing decision immediately. This endpoint does not enable --router-mode kv or --router-track-output-blocks; set those when the Frontend starts.

Add a worker-side hard cap

Optional. A worker can independently cap concurrent engine work and queue only a small burst. Set --engine-request-limit N (or DYN_ENGINE_REQUEST_LIMIT) on the worker component:

1
- name: VllmWorker
2
  type: worker
3
  podTemplate:
4
    spec:
5
      containers:
6
        - name: main
7
          args:
8
            - --engine-request-limit
9
            - "32"

When all N engine slots and the overflow queue are full, the worker rejects the request and the Frontend returns HTTP 529. DYN_DYNAMO_REQUEST_QUEUE_LIMIT controls the advanced overflow-queue size, defaults to 16, must be at least 2, and has an effect only when the engine limit is set. The effective cap is N + Q in-flight requests per worker.

See Runtime Configuration for the exact fields and Worker-Side Request Admission for the queue implementation.

Verify rejection

Inspect the configured thresholds and worker-load metrics:

$
curl -s http://localhost:8000/busy_threshold
$
curl -s http://localhost:8000/metrics \
>
  | grep -E 'dynamo_frontend_worker_active_(decode_blocks|prefill_tokens)'

Generate enough load to exceed the configured threshold, then confirm:

• The client receives HTTP 529.
• dynamo_frontend_model_rejection_total increases for the affected model and endpoint.
• Worker-side hard-cap tests increase dynamo_rejection_request_total when both the engine and queue are full.

For all metric fields and labels, see Cancellation and Rejection.