Standalone Selection Service

Overview

The standalone selection service (python -m dynamo.select_service) exposes the KV router’s worker selection and active-load accounting over HTTP. It does not forward model requests or own response streams. External runtimes such as Ray register their worker catalog, request a selection, contact the selected worker, and report the reservation lifecycle.

The service combines:

• KV overlap indexing from worker ZMQ events.
• KV-aware and load-aware worker selection.
• Explicit or atomic selection and reservation.
• Best-effort active-load synchronization between selector replicas.
• Startup KV index recovery from another selector or standalone indexer.

Build And Launch

Build the Python bindings with the select-service feature:

cd lib/bindings/python
VIRTUAL_ENV=../../../.venv ../../../.venv/bin/maturin develop --uv --features select-service

Launch the service from the repository root:

.venv/bin/python -m dynamo.select_service --port 8092

The service binds to 0.0.0.0 and does not provide authentication. Run it on a trusted internal network or place it behind an appropriate network policy.

Embedded Rust API

Use SelectionServiceBuilder to embed selection without the HTTP server. The resulting SelectionService owns worker registration, KV-event listeners, indexer recovery, replica synchronization, and shutdown. It exposes the same worker, selection, bookkeeping, inspection, peer-membership, and recovery operations as the standalone HTTP service.

SelectionCore::try_new_local creates an intentionally unsynchronized core for tests and local-only use while reporting invalid tracking-hash configuration. Production integrations should use SelectionServiceBuilder so startup recovery, readiness, and background-task lifecycle remain consistent with the standalone service.

To inject native Rust scorers and a picker while retaining those service-owned capabilities, see Write Custom Routing Strategies.

The C and Go bindings do not currently expose SelectionService. An EPP integration requires separate FFI lifecycle, error-mapping, worker, and peer APIs. Those bindings should wrap SelectionService rather than construct SelectionCore directly.

CLI

Router scheduling behavior continues to use the standard Dynamo router environment configuration.

KV Indexing Modes

The service resolves its indexer shape from the same router configuration the frontend uses, so the two index the same way:

A routing decision is recorded when a reservation is booked (select_and_reserve, or select followed by POST /reservations), never for a query-only select. Hash-only reservations that supply sequence_hashes without block_hashes or token_ids are not recorded. router_approximate_cache_policy=lru is rejected at startup when use_kv_events=true; with use_kv_events=false it falls back to TTL retention with a warning.

The standalone expiry guard measures absolute age from admission; output progress does not refresh it. Periodic cleanup therefore reclaims stale state approximately five to six minutes after admission by default. The embedded KvRouter uses the same 300-second value as its shared request-liveness CLOCK scan interval and reclaims an idle lease approximately five to ten minutes after the last progress touch.

Worker Registration

Every selector replica must receive the same worker catalog before it serves selection traffic. Replica traffic never creates workers. When the catalog comes from a discovery source (Kubernetes, a workers file), a snapshot whose apply fails is retried after 5 seconds unless a newer snapshot arrives first, so a stale worker can persist for that long after a failed pass.

POST /workers
Content-Type: application/json
{
  "worker_id": 1,
  "model_name": "model",
  "routing_group": "default",
  "endpoint": "http://worker:8000",
  "block_size": 16,
  "data_parallel_start_rank": 0,
  "data_parallel_size": 2,
  "kv_events_endpoints": {
    "0": "tcp://worker:5557",
    "1": "tcp://worker:5558"
  }
}

replay_endpoint (KV event gap recovery) is accepted only when data_parallel_size is 1; a multi-rank worker that sets it stays incomplete because the replay protocol carries no rank.

worker_id is service-wide, not scoped by model or routing group. POST /workers is an upsert and returns 201: reusing an existing ID replaces its catalog record. If the model or routing group changes, the worker is removed from the previous partition and moved to the new one, which can leave the previous partition not ready. Assign a unique ID to every live worker across the entire service.

A worker that can consume router hints registers router_hint_worker_type (its backend role, used to match hint sources to targets) and, when it can also serve as a source, router_hint_source_control_endpoints keyed by global DP rank:

{
  "worker_id": 1,
  "router_hint_worker_type": "decode",
  "router_hint_source_control_endpoints": { "0": "tcp://worker:5600", "1": "tcp://worker:5601" }
}

A worker whose KV events come from a state agent sets "kv_event_source_mode": "state_agent_v2"; it can receive hints but is never chosen as a hint source.

PATCH /workers/{worker_id} updates supplied fields, DELETE /workers/{worker_id} removes the worker, and GET /workers lists catalog state. model_name and routing_group scope selection, indexer, and load state; both default to "default" when omitted.

GET /health is process liveness. GET /ready returns 200 only after at least one worker is schedulable, otherwise 503 with lifecycle details.

Worker Restarts

When a worker or its cache-event publisher restarts, update its registration on every selector replica. A reconnect at the same address does not reset the existing event sequence watermark or invalidate old cache ownership.

With KV events enabled, POST /workers or PATCH /workers/{worker_id} for an existing schedulable worker reconciles its indexer registration rank by rank: a rank whose event endpoint and replay endpoint are unchanged keeps its listener, its sequence watermark, and its indexed blocks, so capacity or topology updates do not drain the cache view. A rank whose endpoint changed gets a new listener with a fresh watermark. Because an unchanged update keeps the old watermark, it does not recover a publisher that restarted at the same address: for that, call DELETE /workers/{worker_id}, wait for the response, then POST /workers with the current record, or register the restarted publisher under a new kv_events_endpoint. Supply the complete current worker record to POST /workers because it replaces the catalog record. Serialize lifecycle operations for the same worker.

This differs from the standalone indexer’s /register, which rejects duplicate registrations and requires /unregister first. See Indexer Worker Restarts for cleanup and replay limits. Neither a successful catalog update nor /ready certifies a complete cache view: missed startup events must still be recovered, and expired replay history requires another source of complete state.

Selection API

POST /select

Select a worker without booking active load:

{
  "selection_id": "select-123",
  "model_name": "model",
  "routing_group": "default",
  "block_hashes": [11, 12, 13, 14, 15, 16, 17, 18],
  "sequence_hashes": [21, 22, 23, 24, 25, 26, 27, 28],
  "isl_tokens": 512,
  "session_id": "session-abc"
}

POST /select_and_reserve

Select and atomically book load in the receiving selector process. Supply a globally unique selection_id, or allow the service to generate one:

{
  "selection_id": "select-123",
  "model_name": "model",
  "routing_group": "default",
  "block_hashes": [11, 12, 13, 14, 15, 16, 17, 18],
  "sequence_hashes": [21, 22, 23, 24, 25, 26, 27, 28],
  "isl_tokens": 512,
  "session_id": "session-abc"
}

select_and_reserve returns the selected worker and the normalized booking inputs:

{
  "selection_id": "select-123",
  "sequence_hashes": [21, 22, 23, 24, 25, 26, 27, 28],
  "isl_tokens": 512,
  "track_prefill_tokens": true,
  "model_name": "model",
  "routing_group": "default",
  "worker_id": 1,
  "dp_rank": 0,
  "endpoint": "http://worker:8000",
  "block_size": 16,
  "overlap": {
    "longest_matched": 128,
    "gpu": 64,
    "dp": {"0": 64, "1": 32},
    "cpu": 96,
    "disk": 128
  },
  "effective_prefill_tokens": 384,
  "potential_decode_blocks": 212,
  "decode_busy": false
}

select returns the same selection fields but omits sequence_hashes, isl_tokens, and track_prefill_tokens. selection_id is omitted when absent.

potential_decode_blocks is the scheduler’s projection of KV blocks on the chosen worker once this request is decoding, including the request’s own blocks. decode_busy compares it against the worker’s total_kv_blocks at conditional_disagg_decode_busy_threshold and is omitted when either the threshold or the capacity is unknown.

A booking response may also carry a kv_hint when another worker of the same router_hint_worker_type (the worker capability key is router_hint) holds a longer cached prefix than the chosen worker and advertises a source control endpoint for the matching DP rank. The hint is a versioned action envelope whose message_id is the selection_id:

{
  "kv_hint": {
    "protocol_version": "0.1",
    "message_id": "select-123",
    "actions": [
      {
        "action_id": "a1",
        "action_type": "kv.fetch",
        "action_version": "1.0",
        "payload": {
          "source_control_endpoint": "tcp://worker-1:5600",
          "block_hashes": [8713492873, 1928374650]
        }
      }
    ]
  }
}

block_hashes are root-aligned external sequence hashes; entry i is request block i, and the target decides which suffix to fetch from the source. Hints require a local event-driven primary indexer (no approximate or remote primary) and are never attached to query-only select responses.

Advisory selection

POST /select accepts "advisory": true to select from current scheduler state without queue admission. The request never waits in the router queue, and the response adds the chosen worker’s projected load:

{
  "worker_id": 1,
  "dp_rank": 0,
  "potential_decode_blocks": 212,
  "decode_busy": false,
  "worker_load": {
    "active_prefill_tokens": 2048,
    "prefill_token_capacity": 8192,
    "total_kv_blocks": 4096,
    "prefill_busy": false
  }
}

prefill_busy compares active_prefill_tokens against prefill_token_capacity at conditional_disagg_prefill_busy_threshold and is omitted when that threshold is unset. This is the probe a disaggregation coordinator uses to decide whether to bypass remote prefill: one advisory select against the prefill pool answers “would the prefill worker I’d get be busy?” without a separate load read. advisory is ignored by select_and_reserve, which always books. All overlap values are matched token counts. gpu, cpu, and disk use the cumulative Mooncake tier semantics documented in the standalone indexer’s per-instance tier breakdown. A zero-overlap response includes the selected dp_rank with value 0.

The overlap summary is raw observability. effective_prefill_tokens is the authoritative weighted prefill-load value computed by the same cache-credit formula used for scheduler booking. It is not derived from longest_matched. When a request waits in the scheduler queue, both fields reflect the final overlap inputs after any dequeue-time refresh rather than the enqueue-time snapshot.

The previous public fields cached_tokens and effective_overlap_blocks have been removed. Their values remain internal scheduler inputs.

When a request supplies token_ids, the selector derives tracking hashes with its configured tracking-hash context while retaining public hashes for indexer lookups. Requests that instead supply block_hashes, sequence_hashes, and isl_tokens remain trusted precomputed inputs. The service does not reject, rewrite, or label those identities in keyed mode. Configure every precomputed hash producer with the same algorithm, key, and key ID as the selector.

session_id

Both POST /select and POST /select_and_reserve accept an optional session_id string. With --session-affinity-ttl-secs enabled, each model and routing group has its own binding table. /select_and_reserve binds a new session to the selected worker and holds its lease until the reservation is released or expires. The idle TTL starts when the last lease is released. /select can use an existing binding but does not create one.

Bindings replicate over the configured replica mesh. An explicit affinity_target or pinned_worker takes precedence over the session binding. Without a configured TTL, session_id is only policy input: custom policies can read it through WorkerSelectionContext::session_id(), while the built-in selector ignores it. See Write Custom Routing Strategies.

The pending-selection cache keeps the session metadata, so a later POST /reservations for that selection binds the session to the booked worker, the same as /select_and_reserve does. The frontend uses the same table implementation for request-header affinity; see Configuration and Tuning.

session_context

Both endpoints also accept an optional session_context object that carries the full session metadata the frontend hands to worker selection. When it is present, the flat session_id field is ignored.

{
  "token_ids": [1, 2, 3, 4],
  "session_context": {
    "session_id": "child-session",
    "parent_session_id": "root-session",
    "session_final": false,
    "input_trigger": "tool_result",
    "agent_headers": {
      "x-claude-code-request-class": ["subagent"]
    }
  }
}

Only session_id is required inside the object. input_trigger is one of user_message, tool_result, or other. The affinity table uses this session ID when a TTL is configured. Custom policies can read the remaining values through WorkerSelectionContext::session_context().

The optional agent_headers map carries upstream-captured header names and ordered lists of string values to the policy’s SessionContext.agent_headers() accessor. Both selection endpoints preserve these values without interpreting them. Supply the map in the request body; the selection API does not extract coding-agent metadata from its own HTTP headers.

Ray Select-Then-Reserve Flow

Ray can keep model invocation separate from selector admission:

1. Call POST /select with a selection_id.
2. Send the request to the returned endpoint and dp_rank.
3. Call POST /reservations using the cached selection replay form, or the explicit form with the full worker identity and prompt.
4. Report prefill completion and request completion through the lifecycle API.

Cached selection replay form

A /select that carries a selection_id caches its booking inputs (the chosen worker, the normalized prompt, effective_prefill_tokens, expected_output_tokens, and the prefill-tracking decision) on the selector that served it. A reservation that passes the same selection_id, model_name, and routing_group replays the cached selection, booked under that selection_id, without re-sending the prompt:

POST /reservations
Content-Type: application/json
{
  "selection_id": "select-123",
  "model_name": "model",
  "routing_group": "default"
}

• Id namespace: selection_id is client-chosen and scoped per (model_name, routing_group); use a distinct id per in-flight select. A new select reusing a pending id replaces it (latest wins), and an explicit booking discards the cached selection for its selection_id.
• Required id: selection_id is always required (the replay key and the booking id); a request without it is rejected.
• Single-use: The first successful booking consumes the entry; a repeat replay returns 404 (no pending selection). Concurrent replays of the same id collide at the scheduler, so only one books.
• Retryable on failure: A booking that fails before landing (worker no longer schedulable, service not ready) leaves the entry in place, so the same call can be retried once the condition clears.
• Bounded window: By default, entries expire after 120 seconds and each selector retains at most 4096 pending selections within a 256 MiB budget, evicting oldest first. All three limits are configurable.
• Replica-local: The cache lives in the selector process that served the /select. With multiple selector replicas, route the reservation to the same replica or use the explicit form.
• Pure replay: The booking uses exactly what select captured; other request fields are ignored. Supplying worker_id switches to the explicit form.

On any miss (expired, already consumed, wrong model or routing group, or a different replica) the call returns 404; fall back to the explicit form.

Explicit form

The self-contained form carries the worker identity and prompt and needs no cached selection; it wins whenever worker_id is present. It discards the cached selection for its selection_id, so a later replay of the same id cannot book stale state:

POST /reservations
Content-Type: application/json
{
  "selection_id": "request-123",
  "model_name": "model",
  "routing_group": "default",
  "worker_id": 1,
  "dp_rank": 0,
  "sequence_hashes": [21, 22, 23, 24, 25, 26, 27, 28],
  "isl_tokens": 512,
  "effective_prefill_tokens": 384,
  "track_prefill_tokens": true
}

When supplied, effective_prefill_tokens is authoritative and directly enables prefill-load tracking unless track_prefill_tokens is false. It must not exceed the normalized input sequence length. When track_prefill_tokens is omitted, existing behavior applies: effective_prefill_tokens enables tracking, otherwise router configuration controls it. The reservation API does not accept or derive accounting from overlap fields.

Reservation Lifecycle

POST /reservations/{selection_id}/prefill_complete
POST /reservations/{selection_id}/output_block
DELETE /reservations/{selection_id}

prefill_complete clears active prefill load. output_block updates only the receiving selector’s local decode-block accounting and accepts an optional decay_fraction in [0.0, 1.0]. DELETE frees the reservation.

NOTE: Output-block updates are intentionally not replica-synchronized. They can occur at high frequency, and broadcasting them would consume disproportionate network bandwidth.

Peer Planes

The selector has two independent peer configurations:

Example:

.venv/bin/python -m dynamo.select_service \
  --port 8092 \
  --indexer-peers http://selector-b:8092 \
  --replica-sync-port 9092 \
  --replica-sync-peers 'tcp://selector-b:9092'

Configure the reverse peer direction on selector B for bidirectional lifecycle synchronization. GET /dump exposes the selector’s current indexer snapshot in the same recovery format as the standalone indexer.

Replica-sync peers may also be changed without restarting the selector:

POST /replica_sync/register_peer
Content-Type: application/json
{"endpoint":"tcp://selector-b:9092"}

The same body is accepted by POST /replica_sync/deregister_peer. GET /replica_sync/peers returns the sorted configured endpoints. Dynamic membership is in-memory; after restart, only peers supplied through --replica-sync-peers are restored. These routes only manage live ZMQ replica-sync peers. They do not alter the HTTP indexer-recovery peers.

Consistency Invariants

• Replica synchronization is bounded and best-effort. Delays, reordering, dropped events, and temporary active-load divergence are accepted.
• There is no sequencing, acknowledgement, replay, backpressure, or resynchronization for replica lifecycle events.
• Unknown worker, model, routing-group, DP-rank, and block-size events are dropped. Register the same worker catalog on every selector before routing traffic.
• Replica messages carry opaque tracking hashes and do not negotiate the tracking algorithm or key epoch. Configure all hash producers consistently. For key rotation, stop traffic, change the key and key ID together, restart all producers, and recreate derived tracker state before resuming traffic.
• The v1 replica envelope uses routing_group and is incompatible with binaries that send tenant_id. Drain active reservations and lifecycle traffic, upgrade all connected selectors together, re-register worker catalogs, and then resume traffic. Active advisory state is not migrated.
• Admission, prefill-complete, and free are synchronized. Output-block growth remains local to avoid excessive network bandwidth.
• Startup recovery waits for recovered events to be submitted to the indexer, not for complete processing. Early selections may temporarily miss recovered KV state.
• /select followed by /reservations provides eventual, not atomic, cross-replica admission, and the pending-selection cache behind the minimal reservation form is local to the selector that served the /select. Use /select_and_reserve for atomic local booking.
• Reservation IDs must be globally unique. Duplicate bookings for the same ID conflict (409), regardless of the target worker. An explicit booking that carries a selection_id also discards that cached selection.

Inspection APIs

• GET /loads returns active-load snapshots, optionally filtered by model_name and routing_group.
• POST /potential_loads estimates worker load for a prompt without selection.
• POST /overlap_scores returns per-worker/per-rank tiered overlap rows.
• GET /dump returns the compatible indexer recovery snapshot.