Routing

Escalation-Router Routing

View as Markdown

Escalation-router routing starts each conversation on a cheaper weak model. An LLM judge watches how the work progresses and moves the conversation to a more capable strong model when it detects sustained, recoverable trouble. After escalation, the conversation stays on the strong tier for the rest of the task.

Use it for multi-turn agent workloads where a weak model can handle routine work but may need rescue after repeated errors, loops, drift, false progress, or premature completion. Unlike LLM Classifier Routing, which predicts how difficult a request looks, escalation routing judges whether the run is actually going well.

How it works

The router keeps one state value per conversation: whether that conversation has escalated. Conversations are identified from the stable system prompt and first user message, using the same bounded in-memory store described in Sticky Routing.

For each turn, Switchyard:

  1. Checks the conversation latch. A latched conversation routes to strong without calling the judge again.
  2. Routes an unlatched conversation to weak. Before judge.min_turn (default 3), it skips the judge because there is too little trajectory to assess.
  3. Gives the judge a bounded summary containing the system and first-user anchors, recent messages, and a coverage note for omitted history.
  4. Parses the judge’s structured escalate decision. When the configured confirmation policy is satisfied, Switchyard pins the conversation to strong and uses the strong model for the current turn.
  5. Fails open to weak when the judge times out, errors, or returns invalid output. A judge failure never creates a strong-tier pin.

The routing decision for one turn is:

Confirmation policy

judge.confirmations controls how many positive verdicts are required before the strong-tier latch fires:

  • confirmations: 1 is the default and escalates on the first positive verdict.
  • confirmations: 2 with confirmation_window: 1 requires positive verdicts on consecutive judged turns.
  • A larger confirmation_window allows recurring trouble to confirm even when negative verdicts occur between positive ones. A positive verdict remains live across at most confirmation_window - 1 intervening negative verdicts.

A judge failure provides no evidence either way: it routes the turn to weak without clearing an existing confirmation streak.

Configure an escalation route

Configure escalation routing in the routes: bundle loaded by --routing-profiles.

1routes:
2  agent-escalation:
3    type: escalation_router
4    fallback_target_on_evict: strong
5    judge:
6      model: google/gemini-3.5-flash
7      api_key: ${OPENROUTER_API_KEY}
8      base_url: https://openrouter.ai/api/v1
9      timeout_secs: 5.0
10      min_turn: 3
11      confirmations: 1
12      confirmation_window: 1
13      recent_turn_window: 14
14    weak:
15      model: moonshotai/kimi-k2.6
16      api_key: ${OPENROUTER_API_KEY}
17      base_url: https://openrouter.ai/api/v1
18    strong:
19      model: anthropic/claude-opus-4.7
20      api_key: ${OPENROUTER_API_KEY}
21      base_url: https://openrouter.ai/api/v1

Run the route as a standalone proxy:

$switchyard serve --routing-profiles routes.yaml --port 4000

The route ID (agent-escalation) is the model ID clients select to use the router. The strong and weak model IDs are also registered as direct passthrough choices. The judge is internal to the route and is not exposed as a client-selectable model.

If the selected tier exceeds its context window, Switchyard retries once on fallback_target_on_evict, which must match one of the configured tier ids (strong / weak unless the targets set their own id). See Context-Window Handling.

Useful options

OptionDefaultUse it when
judge.timeout_secs5.0The judge needs a different wall-clock limit. Timeouts fail open to weak.
judge.min_turn3The judge should start earlier or wait for more trajectory evidence.
judge.confirmations1One positive verdict is too eager and escalation should require repeated evidence.
judge.confirmation_window1Intermittent trouble should remain eligible for confirmation across negative verdicts.
judge.disable_reasoningtrueSet to false when a reasoning judge benefits from thinking despite the added latency. The thinking-off hint is only sent to models that accept it (Claude/Bedrock judges never receive it).
judge.max_completion_tokensautoThe judge’s completion budget needs an explicit value. The default follows the reasoning mode: 128 when the thinking-off hint is sent, 4096 otherwise (reasoning tokens and the JSON verdict share the budget; a too-small cap silently fails open every turn). Judges that never accept the hint (Claude/Bedrock) also get the larger ceiling — it is a cap, not spend.
judge.dump_verdictsfalseBenchmark runs need per-turn escalation_verdict= stderr lines (includes a first-user task_hint snippet). Keep off in production; routing telemetry flows through the stats endpoints.
judge.recent_turn_window14The judge needs a wider or narrower trailing-message window.
judge.window_message_chars300More tool-output detail should survive per-message truncation.
judge.max_request_chars12000The complete judge request needs a different character budget. Oldest recent messages are dropped first.
judge.promptbuilt-inA deployment needs a custom escalation rubric inlined in the YAML.
judge.prompt_pathbuilt-inThe custom rubric lives in a file (mutually exclusive with judge.prompt; relative paths resolve against the server’s working directory).
tier_timeout_s600Strong or weak targets without their own timeout_secs need a different call timeout.
enable_statstrueSet to false only when route-level usage statistics are not needed.
affinity_max_sessions10000A long-lived process needs a different in-memory latch capacity.
affinity_storememorySet to redis to share the escalation latch across workers and keep it across pod churn; with memory, a worker change silently restarts an escalated conversation on weak. Best-effort — a store error never fails a request.
affinity_store_url—Connection URL for the shared latch store (e.g. redis://host:6379/0); required with affinity_store: redis.
affinity_store_ttl_seconds3600A shared latch entry should expire on a different horizon.
affinity_key_prefixswyd:esc:Multiple deployments share one Redis and need namespaced keys.
session_key_depth0Repeated benchmark trials with identical task prefixes need separate latches; keep 0 for normal traffic.

Observability

Read the standard routing stats endpoint:

$curl -s http://localhost:4000/v1/routing/stats

The snapshot reports per-model calls, tokens, latency, and cost for the strong and weak tiers. Judge calls are recorded in the classifier stats bucket so their token cost, latency, and errors remain visible as routing overhead.

With judge.dump_verdicts: true (off by default), each judged turn also writes an escalation_verdict={...} JSON line to server stderr. The record includes the decision, reason, turn, confirmation state, and judge latency. After the latch fires, later turns skip the judge and report the pinned routing source in request metadata.

Repeated benchmark trials

By default, the session key is derived from the system prompt and first user message. Repeated trials of the same task against one long-lived server can therefore share a latch: if the first trial escalates, later trials may start on strong.

Use one of these isolation strategies:

  • Start a fresh Switchyard process for each trial set.
  • Set session_key_depth: N to extend the key with the first N messages after the initial user message. This only separates trials when those early trajectories differ, so it requires nonzero sampling temperature.

Keep session_key_depth: 0 for normal traffic. Context compaction or other mid-session prefix rewrites can change a deep key and lose the existing latch.

When not to use escalation routing

  • One-shot requests. There is no trajectory to judge. Use LLM Classifier Routing when the initial request should determine the tier.
  • Fixed traffic experiments. Use Random Routing for A/B splits and gradual traffic ramps.
  • Per-turn stage optimization. Use Stage-Router Routing when tool-result signals should move individual turns in both directions.
  • Latency-critical traffic. Eligible unlatched turns wait for the judge before the selected backend call.
  • Long-range failure cycles. The judge only sees a bounded recent window; cycles longer than that window may be missed.