> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/switchyard/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/switchyard/_mcp/server.

# Sticky Routing

> Keep multi-turn conversations on one model tier to preserve upstream prompt and KV caches.

By default, routing is recomputed on every request, so a single task can hop
between models mid-conversation. This throws away the upstream prompt/KV cache
and, on the classifier router, pays for an LLM classifier call every turn.

**Session affinity**, also called sticky routing, is an opt-in feature configured
directly on the deterministic LLM-classifier router. It pins the routing
decision for a conversation and reuses it for later turns, so a task stays on
one model tier. The router can optionally delay that pin for a fixed number of
warmup turns before committing to a tier.

## How a conversation is identified

No client session ID is required. A conversation is keyed by hashing the stable
prefix made from the **system prompt + the first user message**. Agent harnesses
do not rewrite that prefix, so every turn of one task maps to the same key while
distinct tasks differ. The key is derived per request and the pin store is a bounded
(least-recently-used) in-process map.

## Configuration

Opt in with `session_affinity: true` on the route. `affinity_max_sessions`
(default `10000`) caps the number of pinned conversations.

These fields are part of the deterministic router configuration, not a
standalone route type. Configure them under a `deterministic` entry in the
`routes:` bundle loaded with `--routing-profiles`.

For deterministic routes, `affinity_warmup_turns` controls how many initial
turns remain non-sticky. The default is `0`, which preserves the historical
"pin the first confident verdict" behavior.

### Deterministic (LLM-classifier) router

```yaml
routes:
  llm-classifier:
    type: deterministic
    profile: coding_agent
    session_affinity: true          # pin a confident tier per conversation
    affinity_max_sessions: 10000
    affinity_warmup_turns: 2        # turns 1-2 classify normally; turn 3 can pin
    fallback_target_on_evict: strong
    classifier:
      model: openai/gpt-4o-mini
      api_key: ${OPENROUTER_API_KEY}
      base_url: https://openrouter.ai/api/v1
    strong:
      model: openai/gpt-4o
      api_key: ${OPENROUTER_API_KEY}
      base_url: https://openrouter.ai/api/v1
    weak:
      model: openai/gpt-4o-mini
      api_key: ${OPENROUTER_API_KEY}
      base_url: https://openrouter.ai/api/v1
```

The first **confident** classifier verdict after the warmup period is pinned and
reused. With `affinity_warmup_turns: 2`, turns 1 and 2 route normally and cannot
read or write a pin; turn 3 is the first turn that can commit a confident tier.
Once pinned, the classifier **skips its LLM call entirely**. It runs only until
the route has a committed tier for the task. Fallback decisions (abstain / low
confidence) are never pinned, so a transient classifier failure can't lock the
task to the default tier.

## Behavior notes

* **Off by default.** With `session_affinity: false`, routing is recomputed per
  turn as before.
* **Warmup.** `affinity_warmup_turns: N` means turns
  `1..N` stay non-sticky; the first eligible commit is turn `N + 1`.
* **`affinity_max_sessions: 0` with affinity on is rejected** at config load. A
  zero-capacity store retains nothing and would silently disable stickiness.
* **In-process, per-worker.** Pins live in memory and are not shared across
  workers or restarts.
* **Degrades safe.** Affinity assumes the system + first-user prefix is stable
  across a task's turns. If a harness injects per-turn churn there, the key
  changes each turn and routing simply reverts to per-turn behavior. No lock,
  no error.