> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.
# Request Rejection
Request rejection (load shedding) rejects new requests when every eligible worker is overloaded,
rather than accepting work that could exhaust GPU memory or degrade latency for in-flight requests.
Dynamo returns **HTTP 529** for overload by default so clients can distinguish an available but busy
service from **HTTP 503**, which indicates that the service is unavailable.
Rejection is **off by default**. Each threshold is independently opt-in: setting one numeric threshold
enables only that check. You do not need `--admission-control`; that compatibility flag is ignored.
> **How it works:** The busy-detection formulas, data-parallel rank aggregation, worker-load event
> processing, and worker-side overflow queue are documented in
> [Request Rejection Architecture](/dynamo/dev/knowledge-base/design-documents/fault-tolerance/request-rejection-architecture).
Use the signals that match the workers you want to protect:
- Set `--active-decode-blocks-threshold` for decode workers. The value is the fraction of available
KV cache blocks in use, from `0.0` through `1.0`.
- Set `--active-prefill-tokens-threshold` for an absolute prefill-token limit.
- Set `--active-prefill-tokens-threshold-frac` when the prefill limit should scale with the worker's
`max_num_batched_tokens` value.
The checks use OR logic. If you configure more than one threshold, exceeding any configured threshold
marks that data-parallel rank as busy. A worker is rejected only after all of its ranks are busy, and a
request is shed only when every eligible worker is busy.
Start conservatively, observe the rejection rate and latency, and then raise the limits. For example,
a decode-block threshold of `0.75` sheds earlier than `0.90`.
Configure thresholds on the **Frontend** component. Decode-block rejection also requires KV router
mode because that mode initializes the worker-load metrics path. For long-output workloads, enable
output-block tracking so generated tokens contribute to the observed KV cache load.
```yaml
- name: Frontend
type: frontend
replicas: 1
podTemplate:
spec:
containers:
- name: main
image: ${RUNTIME_IMAGE}
command:
- python3
- -m
- dynamo.frontend
args:
- --router-mode
- kv
- --active-decode-blocks-threshold
- "0.85"
- --router-track-output-blocks
- --active-prefill-tokens-threshold
- "10000"
```
`--router-track-output-blocks` is especially important for long generations. Without it, a workload
can fill KV cache with generated tokens without crossing the load value observed by the router.
You can configure just one signal. For example, a prefill-only deployment can omit the decode-block
threshold and KV-specific options. To disable rejection, remove all three threshold settings or set
their environment-variable values to `None`.
See [Frontend Configuration](/dynamo/dev/components/frontend-configuration#fault-tolerance)
for the complete flag, environment-variable, and validation reference.
Optional. Use the Frontend admin API to change thresholds for a discovered model without redeploying.
The API is available when `DYN_ENABLE_FRONTEND_ADMIN_API=true`, which is the default.
```bash
kubectl port-forward svc/-frontend 8000:8000 -n ${NAMESPACE}
```
Set one or more thresholds:
```bash
curl -X POST http://localhost:8000/busy_threshold \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-0.6B",
"active_decode_blocks_threshold": 0.85,
"active_prefill_tokens_threshold": 10000
}'
```
Read the stored thresholds:
```bash
curl http://localhost:8000/busy_threshold
```
A numeric value enables only its corresponding check. The router applies the new configuration when
the next worker-load or runtime-configuration update arrives, so the change might not affect a routing
decision immediately. This endpoint does not enable `--router-mode kv` or
`--router-track-output-blocks`; set those when the Frontend starts.
Optional. A worker can independently cap concurrent engine work and queue only a small burst. Set
`--engine-request-limit N` (or `DYN_ENGINE_REQUEST_LIMIT`) on the **worker** component:
```yaml
- name: VllmWorker
type: worker
podTemplate:
spec:
containers:
- name: main
args:
- --engine-request-limit
- "32"
```
When all `N` engine slots and the overflow queue are full, the worker rejects the request and the
Frontend returns HTTP 529. `DYN_DYNAMO_REQUEST_QUEUE_LIMIT` controls the advanced overflow-queue
size, defaults to `16`, must be at least `2`, and has an effect only when the engine limit is set. The
effective cap is `N + Q` in-flight requests per worker.
See [Runtime Configuration](/dynamo/dev/components/runtime-configuration#operations) for the exact
fields and [Worker-Side Request Admission](/dynamo/dev/knowledge-base/design-documents/fault-tolerance/request-rejection-architecture#worker-side-request-admission)
for the queue implementation.
Inspect the configured thresholds and worker-load metrics:
```bash
curl -s http://localhost:8000/busy_threshold
curl -s http://localhost:8000/metrics \
| grep -E 'dynamo_frontend_worker_active_(decode_blocks|prefill_tokens)'
```
Generate enough load to exceed the configured threshold, then confirm:
- The client receives HTTP 529.
- `dynamo_frontend_model_rejection_total` increases for the affected `model` and `endpoint`.
- Worker-side hard-cap tests increase `dynamo_rejection_request_total` when both the engine and queue
are full.
For all metric fields and labels, see
[Cancellation and Rejection](/dynamo/dev/observability/metrics-catalog#cancellation-and-rejection).
## Troubleshoot Decode-Block Rejection
If decode-block load does not produce HTTP 529 responses:
1. Confirm that `GET /busy_threshold` shows a numeric `active_decode_blocks_threshold` for the model.
2. Confirm that the Frontend started with `--router-mode kv`.
3. For long-output workloads, confirm that the Frontend started with `--router-track-output-blocks`.
4. Check that the Frontend receives worker updates:
```bash
curl -s http://localhost:8000/metrics \
| grep dynamo_frontend_worker_active_decode_blocks
```
5. Check Frontend logs for worker-monitor subscription warnings.
6. Confirm that each worker publishes its total KV block count:
```bash
curl -s http:///metrics \
| grep dynamo_component_total_blocks
```
7. Verify the event-plane configuration and connectivity between the Frontend and workers.
`--router-track-active-blocks` is a separate routing option. Busy rejection depends on configured
thresholds and worker-load events; it does not require that internal router tracking flag.
## Configure Client Retries
Retry HTTP 529 responses with exponential backoff and jitter. Do not retry every HTTP 503 as if it
were overload; 503 indicates that Dynamo currently has no available service path.
```python
import random
import time
def send_with_retry(request, max_retries=5):
for attempt in range(max_retries):
response = send_request(request)
if response.status_code != 529:
return response
wait_time = min(60, (2**attempt) + random.uniform(0, 1))
time.sleep(wait_time)
raise RuntimeError("maximum retries exceeded")
```
If an existing client only understands 503 retry semantics, set
`DYN_HTTP_OVERLOAD_STATUS_CODE=503` on the Frontend. The variable accepts any valid HTTP status code;
an invalid value falls back to 529.
## Related Documentation
- [Request Rejection Architecture](/dynamo/dev/knowledge-base/design-documents/fault-tolerance/request-rejection-architecture) - Busy detection, event flow, and admission internals
- [Frontend Configuration](/dynamo/dev/components/frontend-configuration#fault-tolerance) - Thresholds, overload status, and admin API
- [Runtime Configuration](/dynamo/dev/components/runtime-configuration#operations) - Worker-side engine and queue limits
- [Metrics Catalog](/dynamo/dev/observability/metrics-catalog#cancellation-and-rejection) - Rejection and admission metrics
- [Request Migration](/dynamo/dev/kubernetes/fault-tolerance/request-migration) - Recovering in-flight requests after worker failure