Context-Window Handling

When an upstream rejects a request because the prompt exceeds the model’s context window, Switchyard evicts that target for the current request, reroutes to the configured fallback target, and retries once. If the fallback also overflows, the request fails with a 400 in the client’s inbound wire format.

Any multi-target route (stage-router, random_routing, or deterministic) supports this. Set fallback_target_on_evict on the route. Single-target routes (type: model) have no alternative target, so the original overflow propagates unchanged.

Configuration

fallback_target_on_evict is required on every multi-target route and must match one of the route’s declared target ids:

1
routes:
2
  my-stage-router:
3
    type: stage_router
4
    picker: capable_first
5
    fallback_target_on_evict: strong   # must match strong.id or weak.id
6
    strong:
7
      id: strong
8
      model: anthropic/claude-opus-4.7
9
    weak:
10
      id: weak
11
      model: moonshotai/kimi-k2.6

Scope

Single eviction + single retry per request. Compaction, cool-down, and re-insertion of evicted targets are out of scope.