> This page is for version dev. > For other versions, use one of these documentation indexes: > - Latest (v1.6.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt > - dev: https://docs.nvidia.com/dynamo/dev/llms.txt > - v1.6.1: https://docs.nvidia.com/dynamo/v1.6.1/llms.txt > - v1.6.0: https://docs.nvidia.com/dynamo/v1.6.0/llms.txt > - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt > - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt > - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt > - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt > - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt > - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt > - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt > - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt > - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt > - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt > - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt > - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt > - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt > - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt > - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt > - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt > - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt > - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt > - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt > For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt. # Maintain Session Affinity By the end of this guide, the frontend keeps every turn of an agent session on the same worker, so each turn reuses the KV cache that the earlier turns built. An agent turn repeats the whole conversation so far. If the turn lands on the worker that already holds that prefix, it skips most of the prefill. KV-aware routing (`--router-mode kv`) already prefers workers with cache overlap. Session affinity goes further: it binds a session to one worker for a while, so reuse does not depend on the router's scoring on every turn. Use it when your agents run many turns on a growing context and you have more than one worker. Pinning can unbalance load. `soft` mode does not fix that on its own: with the default selector it keeps the bound worker while that worker is eligible. Use it with a custom or [ThunderAgent](/dynamo/dev/agents/thunder-agent-program-scheduler) policy that can choose another worker. ![Each turn of a session repeats a growing prefix. With affinity on, the frontend binds the session to one worker and sends later turns there so they reuse its KV cache. Hard mode always uses the bound worker; soft mode treats it as a preference.](/dynamo/_fern-files/dynamo.docs.buildwithfern.com/d5d0aee09be1b153d070bee021bead60ede45dacbe9df9d110e4a9a7224e528e/assets/img/session-affinity.svg) ## Before You Start - The frontend runs with `--router-mode kv`. - Requests carry a session ID. Packaged harnesses send one automatically, and custom clients send `x-dynamo-session-id`. See [Identify agent sessions](/dynamo/dev/agents/identify-agent-sessions). A session ID alone does not change placement. Affinity is off until you set a TTL. ## Enable Session Affinity Set an idle TTL on the frontend: ```bash python -m dynamo.frontend \ --router-mode kv \ --router-session-affinity-ttl-secs 300 ``` The same setting is available as `DYN_ROUTER_SESSION_AFFINITY_TTL_SECS`. If you started the endpoint with `agg_agent.sh` in [Run an agent on Dynamo](/dynamo/dev/agents/run-an-agent-on-dynamo), stop it and start it again with the TTL in its environment: ```bash DYN_ROUTER_SESSION_AFFINITY_TTL_SECS=300 \ bash examples/backends/sglang/launch/agg_agent.sh ``` The value is the idle timeout in seconds, from `1` through `31536000`. A binding expires when the session has had no active requests for that long, so choose a value longer than the pauses between your agent's turns (tool runs, human review). The first successfully dispatched request binds the session to its selected worker and, when available, its data-parallel rank. Later requests from the same session follow that binding. ## Choose Hard or Soft Binding Set `--router-session-affinity-mode` (or `DYN_ROUTER_SESSION_AFFINITY_MODE`): | Mode | Behavior | Use it when | |---|---|---| | `hard` (default) | Sends the request to the bound worker. If that worker or rank is no longer valid, Dynamo drops the binding and selects a worker normally. | Cache reuse matters more than load balance, and sessions have similar load. | | `soft` | Passes the bound worker to normal worker selection as a preference. The built-in selector keeps it while the worker is eligible. A custom or [ThunderAgent](/dynamo/dev/agents/thunder-agent-program-scheduler) policy can choose another worker. | Sessions vary in length, or a policy needs to move sessions under pressure. | ```bash python -m dynamo.frontend \ --router-mode kv \ --router-session-affinity-ttl-secs 300 \ --router-session-affinity-mode soft ``` ## Check That It Works Enable [tracing](/dynamo/dev/agents/agent-tracing) and list the worker for each request in a session. The default trace sink writes gzip segments, which is also what `agg_agent.sh` produces: ```bash gzip -cd "${DYN_REQUEST_TRACE_FILE_PATH:-/tmp/dynamo-request-trace}".*.jsonl.gz \ | jq -c '.event | select(.event_type=="request_end") | {session: .agent_context.session_id, worker: .request.worker, cached_tokens: .request.cached_tokens}' ``` This check needs at least two eligible workers. With one worker, every row shows that worker whether or not affinity is on, and KV routing alone can raise `cached_tokens`. To confirm affinity, run the same multi-turn session once without the TTL and once with it. The trace writer appends new segments after the existing ones, so clear the old segments before each run to keep the two runs apart: ```bash rm -f "${DYN_REQUEST_TRACE_FILE_PATH:-/tmp/dynamo-request-trace}".*.jsonl.gz ``` Then run the filter above. With `hard` affinity, every row for the session shows the same worker. Without it, rows can spread across workers. ## Subagents A request that carries a parent session ID binds under a key derived from that parent, so the subagents of one parent share a binding while the parent keeps its own. This behavior is experimental. Dynamo reads the parent ID from `X-Dynamo-Parent-Session-ID` and from the native parent headers listed in [Identify agent sessions](/dynamo/dev/agents/identify-agent-sessions#native-agent-headers). - Under `hard` mode, the group stays on its first worker and does not move because of load. The binding resets only when the worker becomes unusable or dispatch fails. - Siblings share one binding, so when several subagents start at once, the others wait for the first sibling's dispatch to finish before they are placed. - With TensorRT-LLM attention data parallelism and `kv_cache_routing_conversation_affinity` enabled, set `--conversation-affinity-dp-rank-source dynamo` so the rank Dynamo selects is the one the engine records. ## SGLang Session Control On SGLang, session control (`nvext.session_control`) also keeps a subagent's KV cache isolated from other sessions. See [Agents on SGLang](/dynamo/dev/knowledge-base/modular-components/backends/sg-lang/agents-on-sg-lang#session-control-for-subagent-kv-isolation-experimental). ## Notes - Affinity state is local to each router and synchronizes across router replicas on a best-effort basis. - The TTL is independent of `--router-ttl-secs` and `--router-predicted-ttl-secs`. - An explicit worker target on a request always wins over a binding, in both modes. For the full binding rules, including how soft rebinding commits and LoRA behavior, see [Session Affinity](/dynamo/dev/knowledge-base/modular-components/router/configuration-and-tuning#session-affinity) in Router Configuration and Tuning. ## Next [Prioritize requests](/dynamo/dev/agents/prioritize-requests) lets interactive requests go ahead of background work on the same workers.