> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.
# dynamo.llm
`dynamo.llm` publishes 36 classes and 11 functions. Source: [`lib/bindings/python/src/dynamo/llm/__init__.py`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/llm/__init__.py)
#### AicPerfConfig (class)
No summary available.
```python
from dynamo.llm import AicPerfConfig
```
```python
AicPerfConfig(aic_backend: str, aic_system: str, aic_model_path: str, aic_tp_size: int = 1, aic_backend_version: Optional[str] = None, aic_moe_tp_size: Optional[int] = None, aic_moe_ep_size: Optional[int] = None, aic_attention_dp_size: Optional[int] = None, aic_nextn: Optional[int] = None, aic_nextn_accept_rates: Optional[str] = None, aic_gemm_dtype: Optional[str] = None, aic_moe_dtype: Optional[str] = None, aic_fmha_dtype: Optional[str] = None, aic_kv_cache_dtype: Optional[str] = None, aic_comm_dtype: Optional[str] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1722`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1722)
**Public methods**
**init**
```python
__init__(aic_backend: str, aic_system: str, aic_model_path: str, aic_tp_size: int = 1, aic_backend_version: Optional[str] = None, aic_moe_tp_size: Optional[int] = None, aic_moe_ep_size: Optional[int] = None, aic_attention_dp_size: Optional[int] = None, aic_nextn: Optional[int] = None, aic_nextn_accept_rates: Optional[str] = None, aic_gemm_dtype: Optional[str] = None, aic_moe_dtype: Optional[str] = None, aic_fmha_dtype: Optional[str] = None, aic_kv_cache_dtype: Optional[str] = None, aic_comm_dtype: Optional[str] = None) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1723)
#### EngineType (class)
Engine type for Dynamo workers
```python
from dynamo.llm import EngineType
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3069`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3069)
#### EntrypointArgs (class)
Settings to connect an input to a worker and run them. Use by `dynamo run`.
```python
from dynamo.llm import EntrypointArgs
```
```python
EntrypointArgs(engine_type: EngineType, model_path: Optional[str] = None, model_name: Optional[str] = None, endpoint_id: Optional[str] = None, template_file: Optional[str] = None, router_config: Optional[RouterConfig] = None, kv_cache_block_size: Optional[int] = None, http_host: Optional[str] = None, http_port: Optional[int] = None, http_metrics_port: Optional[int] = None, tls_cert_path: Optional[str] = None, tls_key_path: Optional[str] = None, extra_engine_args: Optional[str] = None, mocker_engine_args: Optional[MockEngineArgs] = None, runtime_config: Optional[ModelRuntimeConfig] = None, namespace: Optional[str] = None, namespace_prefix: Optional[str] = None, is_prefill: bool = False, is_decode: bool = False, migration_limit: int = 0, migration_max_seq_len: Optional[int] = None, chat_engine_factory: Optional[Callable] = None, aic_perf_config: Optional[AicPerfConfig] = None, *, metrics_prefix: Optional[str] = None, enable_anthropic_api: Optional[bool] = None, strip_anthropic_preamble: Optional[bool] = None, enable_streaming_tool_dispatch: Optional[bool] = None, enable_streaming_reasoning_dispatch: Optional[bool] = None, tokenizer_backend: Optional[str] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3076`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3076)
**Public methods**
**init**
```python
__init__(engine_type: EngineType, model_path: Optional[str] = None, model_name: Optional[str] = None, endpoint_id: Optional[str] = None, template_file: Optional[str] = None, router_config: Optional[RouterConfig] = None, kv_cache_block_size: Optional[int] = None, http_host: Optional[str] = None, http_port: Optional[int] = None, http_metrics_port: Optional[int] = None, tls_cert_path: Optional[str] = None, tls_key_path: Optional[str] = None, extra_engine_args: Optional[str] = None, mocker_engine_args: Optional[MockEngineArgs] = None, runtime_config: Optional[ModelRuntimeConfig] = None, namespace: Optional[str] = None, namespace_prefix: Optional[str] = None, is_prefill: bool = False, is_decode: bool = False, migration_limit: int = 0, migration_max_seq_len: Optional[int] = None, chat_engine_factory: Optional[Callable] = None, aic_perf_config: Optional[AicPerfConfig] = None, *, metrics_prefix: Optional[str] = None, enable_anthropic_api: Optional[bool] = None, strip_anthropic_preamble: Optional[bool] = None, enable_streaming_tool_dispatch: Optional[bool] = None, enable_streaming_reasoning_dispatch: Optional[bool] = None, tokenizer_backend: Optional[str] = None) -> None
```
Create EntrypointArgs.
**Parameters**
**`engine_type`** `EngineType`
The type of engine to use
---
**`model_path`** `Optional[str]`
Path to the model directory on disk
---
**`model_name`** `Optional[str]`
Model name or dynamo endpoint (e.g. 'dyn://namespace.component.endpoint')
---
**`endpoint_id`** `Optional[str]`
Optional endpoint ID
---
**`template_file`** `Optional[str]`
Optional path to a prompt template file
---
**`router_config`** `Optional[RouterConfig]`
Optional router configuration
---
**`kv_cache_block_size`** `Optional[int]`
Optional KV cache block size
---
**`http_host`** `Optional[str]`
HTTP host to bind to
---
**`http_port`** `Optional[int]`
HTTP port to bind to
---
**`http_metrics_port`** `Optional[int]`
HTTP metrics port (for gRPC service)
---
**`tls_cert_path`** `Optional[str]`
TLS certificate path (PEM format)
---
**`tls_key_path`** `Optional[str]`
TLS key path (PEM format)
---
**`extra_engine_args`** `Optional[str]`
Optional path to mocker engine arguments JSON
---
**`mocker_engine_args`** `Optional[MockEngineArgs]`
Typed mocker engine arguments
---
**`runtime_config`** `Optional[ModelRuntimeConfig]`
Optional runtime configuration for discovery registration
---
**`namespace`** `Optional[str]`
Dynamo namespace for model discovery scoping
---
**`namespace_prefix`** `Optional[str]`
Optional namespace prefix
---
**`is_prefill`** `bool`
Whether this is a prefill worker
---
**`is_decode`** `bool`
Whether this is a decode worker (disaggregated); pairs with a prefill peer for readiness
---
**`migration_limit`** `int`
Maximum number of request migrations (0=disabled)
---
**`migration_max_seq_len`** `Optional[int]`
Optional max sequence length for migration
---
**`chat_engine_factory`** `Optional[Callable]`
Optional Python chat completions engine factory callback
---
**`aic_perf_config`** `Optional[AicPerfConfig]`
Optional AIC perf-model configuration for default KV routing
---
**`metrics_prefix`** `Optional[str]`
Optional Prometheus metrics prefix override
---
**`enable_anthropic_api`** `Optional[bool]`
Optional Anthropic Messages API override
---
**`strip_anthropic_preamble`** `Optional[bool]`
Optional Anthropic preamble stripping override
---
**`enable_streaming_tool_dispatch`** `Optional[bool]`
Optional streaming tool dispatch override
---
**`enable_streaming_reasoning_dispatch`** `Optional[bool]`
Optional streaming reasoning dispatch override
---
**`tokenizer_backend`** `Optional[str]`
Optional tokenizer backend override ("default" or "fastokens")
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3082)
#### FpmDirectPublisher (class)
Direct Forward Pass Metrics publisher used by in-process producers such as the TRT-LLM adapter. The underlying Rust publisher owns per-DP-rank serialization tasks (each with its own 1s idle heartbeat timer) and a single event-plane publisher task. Python callers do not manage heartbeat: when `publish` is not called for `IDLE_HEARTBEAT_INTERVAL` (1.0s, matching vLLM's `HEARTBEAT_INTERVAL`), the Rust side emits a zeroed snapshot on that rank's channel.
```python
from dynamo.llm import FpmDirectPublisher
```
```python
FpmDirectPublisher(endpoint: Endpoint, worker_id: str, dp_size: int = 1) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1285`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1285)
**Public methods**
**init**
```python
__init__(endpoint: Endpoint, worker_id: str, dp_size: int = 1) -> None
```
Create a publisher with `dp_size` per-DP-rank channels.
**Parameters**
**`endpoint`** `Endpoint`
Dynamo component endpoint (provides runtime + discovery).
---
**`worker_id`** `str`
Unique worker identifier stamped on every emitted FPM.
---
**`dp_size`** `int`
Number of DP ranks to allocate channels for. Use `1`
when attention DP is disabled.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1296)
publish
```python
publish(*, dp_rank: int, scheduled_num_prefill_requests: int, scheduled_sum_prefill_tokens: int, scheduled_sum_prefill_kv_tokens: int, scheduled_num_decode_requests: int, scheduled_sum_decode_kv_tokens: int, queued_num_prefill_requests: int, queued_sum_prefill_tokens: int, queued_num_decode_requests: int, queued_sum_decode_kv_tokens: int, wall_time_secs: float) -> None
```
Publish one iteration's FPM snapshot for the given DP rank.
All parameters are keyword-only on the Python side: adjacent ints
with similar units (`scheduled_*` vs `queued_*`, `*_prefill_*`
vs `*_decode_*`) cannot be distinguished by the type system, so
a transposition would silently corrupt every published snapshot.
Variance fields (var\_prefill\_length, var\_decode\_kv\_tokens,
var\_queued\_prefill\_length, var\_queued\_decode\_kv\_tokens) are defaulted
to 0.0 per the MVP scope; a follow-up PR can add Welford-based
variance computation.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1313)
shutdown
```python
shutdown() -> None
```
Shut down the publisher and its per-rank serialization tasks.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1343)
#### FpmEventRelay (class)
Relay that bridges ForwardPassMetrics from a local raw ZMQ PUB socket (InstrumentedScheduler in EngineCore child process) to the Dynamo event plane with automatic discovery registration.
```python
from dynamo.llm import FpmEventRelay
```
```python
FpmEventRelay(endpoint: Endpoint, zmq_endpoint: str) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1258`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1258)
**Public methods**
**init**
```python
__init__(endpoint: Endpoint, zmq_endpoint: str) -> None
```
Create a relay.
**Parameters**
**`endpoint`** `Endpoint`
Dynamo component endpoint (provides runtime + discovery).
---
**`zmq_endpoint`** `str`
Local ZMQ PUB address to subscribe to
(e.g., "tcp\://127.0.0.1:20380").
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1265)
shutdown
```python
shutdown() -> None
```
Shut down the relay task.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1280)
#### FpmEventSubscriber (class)
Subscriber for ForwardPassMetrics from the Dynamo event plane. Auto-discovers engine publishers via the discovery plane.
```python
from dynamo.llm import FpmEventSubscriber
```
```python
FpmEventSubscriber(endpoint: Endpoint) -> None
```
Two mutually exclusive usage modes:
1. **recv mode** (default): call `recv()` to pull individual messages.
2. **tracking mode**: call `start_tracking()` once, then poll
`get_recent_stats()` to retrieve the latest FPM bytes keyed by
`(worker_id, dp_rank)`. Stale entries are cleaned up when
workers are removed (via discovery watch).
[`lib/bindings/python/src/dynamo/_core.pyi#L1348`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1348)
**Public methods**
**init**
```python
__init__(endpoint: Endpoint) -> None
```
Create a subscriber that auto-discovers FPM publishers.
No background tasks are started until `recv()` or
`start_tracking()` is called.
**Parameters**
**`endpoint`** `Endpoint`
Dynamo component endpoint (provides runtime + discovery).
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1362)
recv
```python
recv() -> Optional[bytes]
```
Blocking receive of the next message (raw msgspec bytes). Releases the GIL while waiting.
On the first call a background subscriber task is spawned (recv mode).
Cannot be used after `start_tracking()`.
**Returns**
* `Optional[bytes]` — Raw msgspec payload, or None if the stream is closed.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1374)
start_tracking
```python
start_tracking() -> None
```
Start background tracking of the latest FPM per (worker\_id, dp\_rank).
Spawns two background tasks:
1. Event consumption: subscribes to FPM events, extracts the composite
key (worker\_id, dp\_rank) from the msgpack payload, stores latest
raw bytes in an internal map.
2. MDC discovery watch: monitors ComponentModels for the target
component. When a model is removed, all entries whose
worker\_id matches the removed instance\_id are purged.
After calling this, `recv()` will raise RuntimeError.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1387)
get_recent_stats
```python
get_recent_stats() -> dict[tuple[str, int], bytes]
```
Return the latest FPM bytes for every tracked (worker\_id, dp\_rank).
Cleanup of removed engines is handled by the MDC discovery watch
task spawned by `start_tracking()`.
Raises RuntimeError if `start_tracking()` has not been called.
**Returns**
* `dict[tuple[str, int], bytes]` — dict mapping `(worker_id, dp_rank)` to raw msgspec bytes.
* `dict[tuple[str, int], bytes]` — Decode each value with `forward_pass_metrics.decode(data)`.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1404)
get_model_cards
```python
get_model_cards() -> dict[str, str]
```
Snapshot of model deployment cards keyed by worker id.
The snapshot is filtered against the known-workers set so entries
for already-removed workers are not returned. Values are the raw
`ModelDeploymentCard` serialized as a JSON string; callers parse
whichever fields they need (e.g. `runtime_config`,
`display_name`).
Raises RuntimeError if `start_tracking()` has not been called.
**Returns**
* `dict[str, str]` — dict mapping `worker_id` to `card_json` (JSON string).
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1419)
shutdown
```python
shutdown() -> None
```
Shut down the subscriber (all background tasks).
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1436)
#### FrontendExtensionContext (class)
Read-only, live view of frontend state passed to extension route handlers.
```python
from dynamo.llm import FrontendExtensionContext
```
Handlers receive this and answer from current state. The surface is
intentionally narrow (typed read-only accessors only); it does not expose
the internal service state.
[`lib/bindings/python/src/dynamo/_core.pyi#L2356`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2356)
**Public methods**
is_ready
```python
is_ready() -> bool
```
Whether the HTTP service has finished startup and is ready to serve.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2364)
is_cancelled
```python
is_cancelled() -> bool
```
Whether the frontend is shutting down (draining).
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2368)
has_any_ready_model
```python
has_any_ready_model() -> bool
```
Whether at least one model is registered and ready to serve.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2372)
is_model_ready_to_serve
```python
is_model_ready_to_serve(model: str) -> bool
```
Whether the named model is registered and ready to serve.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2376)
model_display_names
```python
model_display_names() -> list[str]
```
Sorted display names of all registered models.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2380)
serving_ready_display_names
```python
serving_ready_display_names() -> list[str]
```
Sorted display names of models ready to serve.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2384)
#### FrontendResponse (class)
Explicit status-code override returned by a `FrontendRoute` handler.
```python
from dynamo.llm import FrontendResponse
```
```python
FrontendResponse(status_code: int, body: object) -> None
```
Return this to set a non-200 status (e.g. `FrontendResponse(503, body)`);
return a plain JSON-serializable value for the default 200.
[`lib/bindings/python/src/dynamo/_core.pyi#L2409`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2409)
**Public methods**
**init**
```python
__init__(status_code: int, body: object) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2416)
#### FrontendRoute (class)
A trusted extension route served on the Dynamo HTTP frontend.
```python
from dynamo.llm import FrontendRoute
```
```python
FrontendRoute(method: str, path: str, handler: Callable[[FrontendExtensionContext], object]) -> None
```
Currently restricted to static-path `GET` routes. `handler` is a
synchronous callable that receives a `FrontendExtensionContext` and
returns a JSON-serializable body (implies HTTP 200) or a `FrontendResponse`
to set the status code. Async handlers and path parameters are rejected at
construction.
[`lib/bindings/python/src/dynamo/_core.pyi#L2388`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2388)
**Public methods**
**init**
```python
__init__(method: str, path: str, handler: Callable[[FrontendExtensionContext], object]) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2398)
#### HttpAsyncEngine (class)
An async engine for a distributed Dynamo http service. This is an extension of the python based AsyncEngine that handles HttpError exceptions from Python and converts them to the Rust version of HttpError
```python
from dynamo.llm import HttpAsyncEngine
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1482`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1482)
#### HttpError (class)
No summary available.
```python
from dynamo.llm import HttpError
```
```python
HttpError(code: int, message: str)
```
[`lib/bindings/python/src/dynamo/llm/exceptions.py#L25`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/llm/exceptions.py#L25)
**Public methods**
**init**
```python
__init__(code: int, message: str)
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/llm/exceptions.py#L26)
#### HttpService (class)
A HTTP service for dynamo applications. It is a OpenAI compatible http ingress into the Dynamo Distributed Runtime.
```python
from dynamo.llm import HttpService
```
```python
HttpService(port: Optional[int] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1441`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1441)
**Public methods**
**init**
```python
__init__(port: Optional[int] = None) -> None
```
Create a new HTTP service.
**Parameters**
**`port`** `Optional[int]`
Optional port number to bind the service to (default: 8080)
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1447)
run
```python
run(runtime: DistributedRuntime) -> None
```
Run the HTTP service.
**Parameters**
**`runtime`** `DistributedRuntime`
DistributedRuntime instance for token management
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1456)
shutdown
```python
shutdown() -> None
```
Shutdown the HTTP service by cancelling its internal token.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1465)
#### KserveGrpcService (class)
A gRPC service implementing the KServe protocol for dynamo applications. Provides model management for completions, chat completions, and tensor-based models.
```python
from dynamo.llm import KserveGrpcService
```
```python
KserveGrpcService(port: Optional[int] = None, host: Optional[str] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1491`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1491)
**Public methods**
**init**
```python
__init__(port: Optional[int] = None, host: Optional[str] = None) -> None
```
Create a new KServe gRPC service.
**Parameters**
**`port`** `Optional[int]`
Optional port number to bind the service to
---
**`host`** `Optional[str]`
Optional host address to bind the service to
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1497)
add_completions_model
```python
add_completions_model(model: str, checksum: str, engine: PythonAsyncEngine) -> None
```
Register a completions model with the service.
**Parameters**
**`model`** `str`
The model name
---
**`checksum`** `str`
The model checksum
---
**`engine`** `PythonAsyncEngine`
The async engine to handle requests
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1507)
add_chat_completions_model
```python
add_chat_completions_model(model: str, checksum: str, engine: PythonAsyncEngine) -> None
```
Register a chat completions model with the service.
**Parameters**
**`model`** `str`
The model name
---
**`checksum`** `str`
The model checksum
---
**`engine`** `PythonAsyncEngine`
The async engine to handle requests
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1523)
add_tensor_model
```python
add_tensor_model(model: str, checksum: str, engine: PythonAsyncEngine, *, runtime_config: Optional[ModelRuntimeConfig] = None, tensor_model_config: Optional[Dict[str, Any]] = None) -> None
```
Register a tensor-based model with the service.
**Parameters**
**`model`** `str`
The model name
---
**`checksum`** `str`
The model checksum
---
**`engine`** `PythonAsyncEngine`
The async engine to handle requests
---
**`runtime_config`** `Optional[ModelRuntimeConfig]`
Optional runtime-resolved worker metadata
---
**`tensor_model_config`** `Optional[Dict[str, Any]]`
Optional tensor protocol model metadata
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1539)
remove_completions_model
```python
remove_completions_model(model: str) -> None
```
Remove a completions model from the service.
**Parameters**
**`model`** `str`
The model name to remove
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1560)
remove_chat_completions_model
```python
remove_chat_completions_model(model: str) -> None
```
Remove a chat completions model from the service.
**Parameters**
**`model`** `str`
The model name to remove
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1569)
remove_tensor_model
```python
remove_tensor_model(model: str) -> None
```
Remove a tensor model from the service.
**Parameters**
**`model`** `str`
The model name to remove
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1578)
list_chat_completions_models
```python
list_chat_completions_models() -> List[str]
```
List all registered chat completions models.
**Returns**
* `List[str]` — List of model names
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1587)
list_completions_models
```python
list_completions_models() -> List[str]
```
List all registered completions models.
**Returns**
* `List[str]` — List of model names
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1596)
list_tensor_models
```python
list_tensor_models() -> List[str]
```
List all registered tensor models.
**Returns**
* `List[str]` — List of model names
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1605)
run
```python
run(runtime: DistributedRuntime) -> None
```
Run the KServe gRPC service.
**Parameters**
**`runtime`** `DistributedRuntime`
DistributedRuntime instance for token management
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1614)
shutdown
```python
shutdown() -> None
```
Shutdown the KServe gRPC service by cancelling its internal token.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1623)
#### KvDcRelay (class)
No summary available.
```python
from dynamo.llm import KvDcRelay
```
```python
KvDcRelay(endpoint: Endpoint, dc_id: str, namespace_filter: Optional[str] = None, endpoint_prefix: Optional[str] = None, publication_threshold: int = 16, publication_delay_ms: int = 1, recovery_attempt_timeout_ms: int = 30000) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2801`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2801)
**Public methods**
**init**
```python
__init__(endpoint: Endpoint, dc_id: str, namespace_filter: Optional[str] = None, endpoint_prefix: Optional[str] = None, publication_threshold: int = 16, publication_delay_ms: int = 1, recovery_attempt_timeout_ms: int = 30000) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2802)
start
```python
start() -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2814)
health
```python
health() -> Dict[str, Any]
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2817)
flush
```python
flush() -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2820)
shutdown
```python
shutdown() -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2823)
#### KvEventPublisher (class)
A KV event publisher will publish KV events corresponding to the component.
```python
from dynamo.llm import KvEventPublisher
```
```python
KvEventPublisher(endpoint: Endpoint, worker_id: Optional[int] = None, kv_block_size: int = 0, dp_rank: int = 0, enable_local_indexer: bool = False, zmq_endpoint: Optional[str] = None, zmq_topic: Optional[str] = None, batching_timeout_ms: Optional[int] = None, image_token_id: Optional[int] = None, kv_state_endpoint: Optional[str] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1156`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1156)
**Public methods**
**init**
```python
__init__(endpoint: Endpoint, worker_id: Optional[int] = None, kv_block_size: int = 0, dp_rank: int = 0, enable_local_indexer: bool = False, zmq_endpoint: Optional[str] = None, zmq_topic: Optional[str] = None, batching_timeout_ms: Optional[int] = None, image_token_id: Optional[int] = None, kv_state_endpoint: Optional[str] = None) -> None
```
Create a `KvEventPublisher` object.
When zmq\_endpoint is provided, the publisher subscribes to a ZMQ socket for
incoming engine events (e.g. from SGLang/vLLM) and relays them to NATS.
When zmq\_endpoint is None, events are pushed manually via publish\_batch,
publish\_stored, or publish\_removed.
**Parameters**
**`endpoint`** `Endpoint`
The endpoint to extract component information from for event publishing
---
**`worker_id`** `Optional[int]`
Optional worker ID override. Use None to infer from endpoint.
---
**`kv_block_size`** `int`
The KV block size (must be > 0)
---
**`dp_rank`** `int`
The data parallel rank (defaults to 0)
---
**`enable_local_indexer`** `bool`
Enable worker-local KV indexer
---
**`zmq_endpoint`** `Optional[str]`
Optional ZMQ endpoint for relay mode (e.g. "tcp\://127.0.0.1:5557")
---
**`zmq_topic`** `Optional[str]`
ZMQ topic to subscribe to (defaults to "" when zmq\_endpoint is set)
---
**`batching_timeout_ms`** `Optional[int]`
Cross-list batching timeout in milliseconds. None/0
flushes at each submitted source-list boundary.
---
**`kv_state_endpoint`** `Optional[str]`
KV event ownership endpoint; defaults to endpoint.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1163)
publish_stored
```python
publish_stored(token_ids: List[int], num_block_tokens: List[int], block_hashes: List[int], parent_hash: Optional[int] = None, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, is_eagle: Optional[bool] = None, cache_salt: Optional[str] = None) -> None
```
Publish a KV stored event.
Event IDs are managed internally by the publisher using a monotonic counter.
**Parameters**
**`token_ids`** `List[int]`
List of token IDs
---
**`num_block_tokens`** `List[int]`
Number of tokens per block
---
**`block_hashes`** `List[int]`
List of block hashes (signed 64-bit integers)
---
**`parent_hash`** `Optional[int]`
Optional parent hash (signed 64-bit integer)
---
**`block_mm_infos`** `Optional[List[Optional[Dict[str, Any]]]]`
Optional list of multimodal info for each block.
Each item is either None or a dict with "mm\_objects" key containing
a list of \{"mm\_hash": int, "offsets": \[\[start, end], ...]} dicts.
---
**`lora_name`** `Optional[str]`
Optional LoRA adapter name for adapter-aware block hashing.
---
**`is_eagle`** `Optional[bool]`
Optional Eagle mode flag. When true, stored blocks are
reconstructed using overlapping `kv_block_size + 1` token windows.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1198)
publish_removed
```python
publish_removed(block_hashes: List[int]) -> None
```
Publish a KV removed event.
Event IDs are managed internally by the publisher using a monotonic counter.
**Parameters**
**`block_hashes`** `List[int]`
List of block hashes to remove (signed 64-bit integers)
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1228)
publish_batch
```python
publish_batch(events: Sequence[KvStoredEventInput | KvRemovedEventInput]) -> None
```
Publish an ordered list of KV events as one processor input.
The complete list is validated before it is enqueued. Compatible
events are coalesced while preserving source order and the processor's
existing block-count limits.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1239)
shutdown
```python
shutdown() -> None
```
Shuts down the event publisher, stopping any background tasks.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1251)
#### KvRouter (class)
A KV-aware router that performs intelligent routing based on KV cache overlap.
```python
from dynamo.llm import KvRouter
```
```python
KvRouter(endpoint: Endpoint, block_size: int, kv_router_config: KvRouterConfig, aic_perf_config: Optional[AicPerfConfig] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2826`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2826)
**Public methods**
**init**
```python
__init__(endpoint: Endpoint, block_size: int, kv_router_config: KvRouterConfig, aic_perf_config: Optional[AicPerfConfig] = None) -> None
```
Create a new KvRouter instance.
**Parameters**
**`endpoint`** `Endpoint`
The endpoint to connect to for routing requests
---
**`block_size`** `int`
The KV cache block size
---
**`kv_router_config`** `KvRouterConfig`
Configuration for the KV router
---
**`aic_perf_config`** `Optional[AicPerfConfig]`
Optional AIC perf-model config for effective prefill load tracking
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2831)
generate
```python
generate(token_ids: List[int], model: str, stop_conditions: Optional[JsonLike] = None, sampling_options: Optional[JsonLike] = None, output_options: Optional[JsonLike] = None, router_config_override: Optional[JsonLike] = None, worker_id: Optional[int] = None, dp_rank: Optional[int] = None, extra_args: Optional[JsonLike] = None, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, multi_modal_data: Optional[JsonLike] = None, mm_routing_info: Optional[JsonLike] = None, routing_constraints: Optional[RoutingConstraints] = None, response_buffer_size: int = 100) -> AsyncIterator[JsonLike]
```
Generate text using the KV-aware router.
**Parameters**
**`token_ids`** `List[int]`
Input token IDs
---
**`model`** `str`
Model name to use for generation
---
**`stop_conditions`** `Optional[JsonLike]`
Optional stop conditions for generation
---
**`sampling_options`** `Optional[JsonLike]`
Optional sampling configuration
---
**`output_options`** `Optional[JsonLike]`
Optional output configuration
---
**`router_config_override`** `Optional[JsonLike]`
Optional router configuration override
---
**`worker_id`** `Optional[int]`
Optional worker ID to route to directly. If set, the request
will be sent to this specific worker and router states will be
updated accordingly.
---
**`dp_rank`** `Optional[int]`
Optional data parallel rank to route to. If set along with worker\_id,
the request will be routed to the specific (worker\_id, dp\_rank) pair.
If only dp\_rank is set, the router will select the best worker but
force routing to the specified dp\_rank.
---
**`extra_args`** `Optional[JsonLike]`
Optional extra request arguments to include in the
PreprocessedRequest.
---
**`block_mm_infos`** `Optional[List[Optional[Dict[str, Any]]]]`
Optional block-level multimodal metadata aligned to
request blocks. Backward-compatible shortcut; this is
converted to mm\_routing\_info with routing\_token\_ids=token\_ids.
---
**`multi_modal_data`** `Optional[JsonLike]`
Optional multimodal payload map to preserve image/video
data for downstream model execution.
---
**`mm_routing_info`** `Optional[JsonLike]`
Optional structured routing-only multimodal payload
(e.g., \{"routing\_token\_ids": \[...], "block\_mm\_infos": \[...]})
used by router selection without changing execution token\_ids.
---
**`routing_constraints`** `Optional[RoutingConstraints]`
Optional request routing constraints used to constrain or prefer tainted workers.
---
**`response_buffer_size`** `int`
Maximum number of responses buffered by the Python
adapter. Set to 0 for demand-driven direct Python
consumption; negative values are rejected.
---
**Returns**
* `AsyncIterator[JsonLike]` — An async iterator yielding generation responses
- If worker\_id is set, the request bypasses KV matching and routes directly
to the specified worker while still updating router states.
- dp\_rank allows targeting a specific data parallel replica when workers have
multiple replicas (data\_parallel\_size > 1).
- This is different from query\_instance\_id which doesn't route the request.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2849)
generate_from_request
```python
generate_from_request(request: JsonLike, response_buffer_size: int = 100) -> AsyncIterator[JsonLike]
```
Generate from a preprocessed request dict (PreprocessedRequest format).
Accepts a full request dict with token\_ids, model, stop\_conditions, etc.
Set response\_buffer\_size to 0 for demand-driven direct Python consumption;
negative values are rejected.
Returns an async iterator yielding generation responses.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2910)
best_worker
```python
best_worker(token_ids: List[int], router_config_override: Optional[JsonLike] = None, request_id: Optional[str] = None, update_indexer: bool = False, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, routing_constraints: Optional[RoutingConstraints] = None, strict_priority: int = 0, policy_class: Optional[str] = None, cache_namespace: Optional[str] = None) -> Tuple[int, int, int]
```
Find the best matching worker for the given tokens.
**Parameters**
**`token_ids`** `List[int]`
List of token IDs to find matches for
---
**`router_config_override`** `Optional[JsonLike]`
Optional router configuration override
---
**`request_id`** `Optional[str]`
Optional request ID. If provided, router states will be updated
to track this request (active blocks, lifecycle events). If not
provided, this is a query-only operation that doesn't affect state.
---
**`update_indexer`** `bool`
Whether to record the selected worker in the router's
approximate indexer. This is only meaningful when
`use_kv_events=False` and is independent from lifecycle
state tracking via `request_id`.
---
**`block_mm_infos`** `Optional[List[Optional[Dict[str, Any]]]]`
Optional block-level multimodal metadata aligned to request
blocks. When provided, this is used in block hash computation
to enable MM-aware worker selection.
---
**`cache_namespace`** `Optional[str]`
Optional cache namespace used in block hash computation.
---
**`policy_class`** `Optional[str]`
Requested policy family, or an exact explicit class.
Missing, unknown, and ordinary physical-class names use the
configured default family before cache-bucket resolution.
---
**Returns**
* `Tuple[int, int, int]` — A tuple of (worker\_id, dp\_rank, overlap\_blocks) where: - worker\_id: The ID of the best matching worker - dp\_rank: The data parallel rank of the selected worker - overlap\_blocks: The number of overlapping blocks found
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2925)
get_potential_loads
```python
get_potential_loads(token_ids: List[int], block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, cache_namespace: Optional[str] = None) -> List[Dict[str, int]]
```
Get potential prefill and decode loads for all workers.
**Parameters**
**`token_ids`** `List[int]`
List of token IDs to evaluate
---
**`block_mm_infos`** `Optional[List[Optional[Dict[str, Any]]]]`
Optional block-level multimodal metadata aligned to request
blocks. When provided, this is used in hash computation
for MM-aware potential-load estimation.
---
**`lora_name`** `Optional[str]`
Optional LoRA adapter name used in block hash computation.
---
**Returns**
* `List[Dict[str, int]]` — A list of dictionaries, each containing: - worker\_id: The worker ID - dp\_rank: The data parallel rank - potential\_prefill\_tokens: Number of tokens that would need prefill - potential\_decode\_blocks: Number of blocks currently in decode phase - active\_requests: Number of active requests tracked on the worker
Each (worker\_id, dp\_rank) pair is returned as a separate entry.
If you need aggregated loads per worker\_id, sum the values manually.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2967)
get_overlap_scores
```python
get_overlap_scores(token_ids: List[int], router_config_override: Optional[JsonLike] = None, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, include_shared: bool = True, cache_namespace: Optional[str] = None) -> Dict[str, Any]
```
Get per-worker KV overlap by storage tier.
**Parameters**
**`token_ids`** `List[int]`
List of token IDs to evaluate.
---
**`router_config_override`** `Optional[JsonLike]`
Optional router configuration override for
score-credit fields.
---
**`block_mm_infos`** `Optional[List[Optional[Dict[str, Any]]]]`
Optional block-level multimodal metadata aligned to
request blocks.
---
**`lora_name`** `Optional[str]`
Optional LoRA adapter name for adapter-aware matching.
---
**`include_shared`** `bool`
Whether to query the configured shared cache.
---
**Returns**
* `Dict[str, Any]` — A dictionary containing block\_size, num\_blocks, shared\_cache, and
* `Dict[str, Any]` — workers. Each worker row is keyed by worker\_id and dp\_rank and
* `Dict[str, Any]` — reports device, host-pinned, disk, and shared-cache overlap blocks.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2998)
dump_events
```python
dump_events() -> str
```
Dump all events from the KV router's indexer.
**Returns**
* `str` — A JSON string containing all indexer events
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3026)
mark_prefill_complete
```python
mark_prefill_complete(request_id: str) -> None
```
Mark prefill as completed for a request.
This signals that the request has finished its prefill phase and is now
in the decode phase. Used to update router state for accurate load tracking.
**Parameters**
**`request_id`** `str`
The ID of the request that completed prefill
---
This is typically called automatically by the router when using the
`generate()` method. Only call this manually if you're using
`best_worker()` with `request_id` for custom routing.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3035)
free
```python
free(request_id: str) -> None
```
Free a request by its ID, signaling the router to release resources.
This should be called when a request completes to update the router's
tracking of active blocks and ensure accurate load balancing.
**Parameters**
**`request_id`** `str`
The ID of the request to free
---
This is typically called automatically by the router when using the
`generate()` method. Only call this manually if you're using
`best_worker()` with `request_id` for custom routing.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3052)
#### KvRouterConfig (class)
Values for KV router
```python
from dynamo.llm import KvRouterConfig
```
```python
KvRouterConfig(overlap_score_weight: Optional[float] = None, host_cache_hit_weight: float = 0.75, disk_cache_hit_weight: float = 0.25, router_temperature: float = 0.0, use_kv_events: bool = True, *, router_replica_sync: bool = False, router_track_active_blocks: bool = True, router_track_output_blocks: bool = False, router_assume_kv_reuse: bool = True, router_track_prefill_tokens: bool = True, router_prefill_load_model: str = 'none', router_ttl_secs: float = 120.0, router_queue_threshold: Optional[float] = None, router_event_threads: int = 4, router_queue_policy: str = 'fcfs', use_remote_indexer: bool = False, serve_indexer: bool = False, shared_cache_multiplier: float = 0.0, shared_cache_type: str = 'none', router_predicted_ttl_secs: Optional[float] = None, overlap_score_credit: float = 1.0, overlap_score_credit_decay: float = 0.0, prefill_load_scale: float = 1.0, decode_active_request_weight: float = 0.0, router_policy_config: Optional[str] = None, router_tracking_hash: Literal['public-xxh3-v1', 'keyed-xxh3-v1'] = 'public-xxh3-v1', router_tracking_key_file: Optional[str | os.PathLike[str]] = None, router_tracking_key_id: Optional[str] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1743`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1743)
**Public methods**
**init**
```python
__init__(overlap_score_weight: Optional[float] = None, host_cache_hit_weight: float = 0.75, disk_cache_hit_weight: float = 0.25, router_temperature: float = 0.0, use_kv_events: bool = True, *, router_replica_sync: bool = False, router_track_active_blocks: bool = True, router_track_output_blocks: bool = False, router_assume_kv_reuse: bool = True, router_track_prefill_tokens: bool = True, router_prefill_load_model: str = 'none', router_ttl_secs: float = 120.0, router_queue_threshold: Optional[float] = None, router_event_threads: int = 4, router_queue_policy: str = 'fcfs', use_remote_indexer: bool = False, serve_indexer: bool = False, shared_cache_multiplier: float = 0.0, shared_cache_type: str = 'none', router_predicted_ttl_secs: Optional[float] = None, overlap_score_credit: float = 1.0, overlap_score_credit_decay: float = 0.0, prefill_load_scale: float = 1.0, decode_active_request_weight: float = 0.0, router_policy_config: Optional[str] = None, router_tracking_hash: Literal['public-xxh3-v1', 'keyed-xxh3-v1'] = 'public-xxh3-v1', router_tracking_key_file: Optional[str | os.PathLike[str]] = None, router_tracking_key_id: Optional[str] = None) -> None
```
Create a KV router configuration.
**Parameters**
**`overlap_score_weight`** `Optional[float]`
Deprecated positional/keyword alias for prefill\_load\_scale. When present, it takes precedence over prefill\_load\_scale; a value of 0 also sets overlap\_score\_credit to 0.
---
**`overlap_score_credit`** `float`
Finite, non-negative credit multiplier for device-local prefix overlap (default: 1.0). Values above 1.0 give device overlap extra credit, with adjusted prefill cost clamped at zero.
---
**`prefill_load_scale`** `float`
Scale for adjusted prompt-side prefill load after cache-hit credits (default: 1.0)
---
**`decode_active_request_weight`** `float`
Experimental block-equivalent decode cost added for each active request on a candidate worker (default: 0.0)
---
**`host_cache_hit_weight`** `float`
Credit multiplier for host-pinned cache hits (default: 0.75)
---
**`disk_cache_hit_weight`** `float`
Credit multiplier for disk/external cache hits (default: 0.25)
---
**`router_temperature`** `float`
Temperature for normalized worker sampling via softmax (default: 0.0)
---
**`use_kv_events`** `bool`
Whether to use KV events from workers (default: True)
---
**`router_replica_sync`** `bool`
Enable replica synchronization (default: False)
---
**`router_track_active_blocks`** `bool`
Track active blocks for load balancing (default: True)
---
**`router_track_output_blocks`** `bool`
Track output blocks during generation (default: False).
When enabled, the router adds placeholder blocks as tokens are generated
and, with expected output sequence length (agent\_hints.osl in nvext),
applies fractional decay to output blocks and the structurally exclusive
prompt suffix. Shared prompt blocks retain full weight.
---
**`router_assume_kv_reuse`** `bool`
Assume KV cache reuse when tracking active blocks (default: True).
When True, computes actual block hashes. When False, generates random hashes.
---
**`router_track_prefill_tokens`** `bool`
Include prompt-side prefill tokens in active load accounting (default: True).
---
**`router_tracking_hash`** `Literal['public-xxh3-v1', 'keyed-xxh3-v1']`
Tracking identity algorithm, "public-xxh3-v1" or
"keyed-xxh3-v1" (default: "public-xxh3-v1").
---
**`router_tracking_key_file`** `Optional[str | os.PathLike[str]]`
File containing exactly 32 raw provider-key bytes.
Required only for keyed tracking mode.
---
**`router_tracking_key_id`** `Optional[str]`
Provider-managed key epoch mixed into keyed scope
derivation. Required only for keyed tracking mode.
---
**`router_prefill_load_model`** `str`
Prompt-side prefill load model (default: "none").
"none" keeps static prompt load accounting.
"aic" decays the oldest active prefill request using AIC-predicted duration.
---
**`router_ttl_secs`** `float`
TTL for blocks in seconds when not using KV events (default: 120.0)
---
**`router_queue_threshold`** `Optional[float]`
Optional queue threshold fraction for prefill token capacity (default: None).
Requests are queued if all workers exceed this fraction of max\_num\_batched\_tokens.
Enables priority scheduling via request priority hints.
Set a numeric value to enable queueing.
---
**`router_policy_config`** `Optional[str]`
Startup-only policy-family and cache-bucket queue
YAML path. When omitted, router\_queue\_threshold and
router\_queue\_policy define one synthetic policy class.
---
**`router_event_threads`** `int`
Number of KV indexer worker threads (default: 4).
When > 1, uses a concurrent radix tree with a thread pool,
including for approximate routing when KV events are disabled.
---
**`router_queue_policy`** `str`
Scheduling policy for the router queue (default: "fcfs").
"fcfs": first-come first-served with priority bumps — optimizes tail TTFT.
"lcfs": last-come first-served with priority bumps — intentionally worsens tail behavior for policy comparisons.
"wspt": weighted shortest processing time (Smith's rule) — optimizes average TTFT.
---
**`use_remote_indexer`** `bool`
Query a remote KV indexer served from the worker component (default: False).
---
**`serve_indexer`** `bool`
Serve this router's local indexer from the worker component (default: False).
---
**`shared_cache_multiplier`** `float`
Credit multiplier for shared cache hits beyond the device prefix (default: 0.0).
---
**`shared_cache_type`** `str`
External shared KV cache type, "none" or "hicache" (default: "none").
---
**`router_predicted_ttl_secs`** `Optional[float]`
Enables predict-on-route when set. This TTL
applies to entries in the local side indexer and requires
use\_kv\_events=True. Set to None to disable. Independent of
router\_ttl\_secs, which covers pure approximate mode.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1746)
from_json
```python
from_json(config_json: str) -> KvRouterConfig
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1835)
copy
```python
copy() -> KvRouterConfig
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1839)
with_overrides
```python
with_overrides(overlap_score_weight: Optional[float] = None, *, overlap_score_credit: Optional[float] = None, overlap_score_credit_decay: Optional[float] = None, prefill_load_scale: Optional[float] = None, decode_active_request_weight: Optional[float] = None) -> KvRouterConfig
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1865)
#### LoRADownloader (class)
Unified interface for LoRA downloading and caching (local file:// and S3 s3:// URIs).
```python
from dynamo.llm import LoRADownloader
```
```python
LoRADownloader(cache_path: Optional[str] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2305`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2305)
**Public methods**
**init**
```python
__init__(cache_path: Optional[str] = None) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2308)
download_if_needed
```python
download_if_needed(lora_uri: str) -> Awaitable[str]
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2309)
get_cache_path
```python
get_cache_path(cache_key: str) -> str
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2310)
is_cached
```python
is_cached(lora_uri: str) -> bool
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2311)
validate_cached
```python
validate_cached(cache_key: str) -> bool
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2312)
uri_to_cache_key
```python
uri_to_cache_key(uri: str) -> str
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2314)
#### MediaDecoder (class)
Media decoder for image and video preprocessing.
```python
from dynamo.llm import MediaDecoder
```
```python
MediaDecoder() -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2318`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2318)
**Public methods**
```python
__init__() -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2321)
```python
enable_image(decoder_options: Dict[str, Any]) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2322)
#### MediaFetcher (class)
Media fetcher for loading remote image/video URLs.
```python
from dynamo.llm import MediaFetcher
```
```python
MediaFetcher() -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2325`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2325)
**Public methods**
```python
__init__() -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2328)
```python
user_agent(user_agent: str) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2329)
```python
allow_direct_ip(allow: bool) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2330)
```python
allow_direct_port(allow: bool) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2331)
allowed_media_domains
```python
allowed_media_domains(domains: List[str]) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2332)
```python
timeout_ms(timeout_ms: int) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2333)
#### ModelCardInstanceId (class)
Unique identifier for a worker instance: namespace, component, endpoint and instance\_id. The instance\_id is not currently exposed in the Python bindings.
```python
from dynamo.llm import ModelCardInstanceId
```
[`lib/bindings/python/src/dynamo/_core.pyi#L384`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L384)
**Public methods**
triple
```python
triple() -> Tuple[str, str, str]
```
Triple of namespace, component and endpoint this worker is serving.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L389)
#### ModelInput (class)
What type of request this model needs: Text, Tokens or Tensor
```python
from dynamo.llm import ModelInput
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1629`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1629)
#### ModelRuntimeConfig (class)
A model runtime configuration is a collection of runtime information
```python
from dynamo.llm import ModelRuntimeConfig
```
```python
ModelRuntimeConfig() -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L855`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L855)
**Public methods**
**init**
```python
__init__() -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L884)
set_engine_specific
```python
set_engine_specific(key: str, value: Any) -> None
```
Set an engine-specific runtime configuration value
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L886)
get_engine_specific
```python
get_engine_specific(key: str) -> Any | None
```
Get an engine-specific runtime configuration value
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L890)
set_structural_tag_mode
```python
set_structural_tag_mode(mode: str) -> None
```
Set structural tag mode ("off" or "on").
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L894)
set_structural_tag_scope
```python
set_structural_tag_scope(scope: str) -> None
```
Set structural tag scope ("auto" or "always").
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L898)
set_structural_tag_schema
```python
set_structural_tag_schema(schema: str) -> None
```
Set structural tag schema mode ("auto" or "strict").
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L902)
set_disaggregated_endpoint
```python
set_disaggregated_endpoint(bootstrap_host: str | None = None, bootstrap_port: int | None = None) -> None
```
Set the disaggregated endpoint for the model
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L906)
#### ModelType (class)
OpenAI-style surfaces supported by a model.
```python
from dynamo.llm import ModelType
```
Values are Chat, Completions, Embedding, Classify, Pooling, TensorBased,
Images, Audios, Videos, Realtime, and Empty (no OpenAI surface).
[`lib/bindings/python/src/dynamo/_core.pyi#L1636`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1636)
**Public methods**
supports_chat
```python
supports_chat() -> bool
```
Return True if this model type supports chat.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1666)
supports_embedding
```python
supports_embedding() -> bool
```
Return True if this model type supports /v1/embeddings.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1670)
supports_classify
```python
supports_classify() -> bool
```
Return True if this model type supports /v1/classify.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1674)
supports_pooling
```python
supports_pooling() -> bool
```
Return True if this model type supports /v1/pooling.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1678)
#### MultimodalEmbeddingCachePublisher (class)
A publisher for multimodal encode-worker cache state.
```python
from dynamo.llm import MultimodalEmbeddingCachePublisher
```
```python
MultimodalEmbeddingCachePublisher() -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L684`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L684)
**Public methods**
**init**
```python
__init__() -> None
```
Create a `MultimodalEmbeddingCachePublisher` object.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L691)
create_endpoint
```python
create_endpoint(endpoint: Endpoint) -> None
```
Initialize event-plane publishing for multimodal cache state.
**Parameters**
**`endpoint`** `Endpoint`
The endpoint to extract component information from.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L696)
publish_delta
```python
publish_delta(added_keys: list[str], removed_keys: list[str]) -> None
```
Publish an incremental cache mutation for this worker.
**Parameters**
**`added_keys`** `list[str]`
Newly cached embedding keys.
---
**`removed_keys`** `list[str]`
Cache keys no longer present on the worker.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L704)
#### OverlapScores (class)
A collection of prefix matching scores of workers for a given token ids. 'scores' is a map of worker id to the score which is the number of matching blocks.
```python
from dynamo.llm import OverlapScores
```
[`lib/bindings/python/src/dynamo/_core.pyi#L934`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L934)
#### PythonAsyncEngine (class)
Bridge a Python async generator onto Dynamo's AsyncEngine interface.
```python
from dynamo.llm import PythonAsyncEngine
```
```python
PythonAsyncEngine(generator: Any, event_loop: Any) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1471`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1471)
**Public methods**
**init**
```python
__init__(generator: Any, event_loop: Any) -> None
```
Wrap a Python generator and event loop for use with Dynamo services.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1476)
#### RadixTree (class)
A RadixTree that tracks KV cache blocks and can find prefix matches for sequences.
```python
from dynamo.llm import RadixTree
```
```python
RadixTree() -> None
```
Thread-safe: operations route to a dedicated background thread and long calls
release the Python GIL.
[`lib/bindings/python/src/dynamo/_core.pyi#L961`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L961)
**Public methods**
**init**
```python
__init__() -> None
```
Create a new RadixTree instance.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L969)
find_matches
```python
find_matches(sequence: List[int], early_exit: bool = False) -> OverlapScores
```
Find prefix matches for the given sequence of block hashes.
**Parameters**
**`sequence`** `List[int]`
List of block hashes to find matches for
---
**`early_exit`** `bool`
If True, stop searching after finding the first match
---
**Returns**
* `OverlapScores` — OverlapScores containing worker matching scores and frequencies
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L975)
apply_event
```python
apply_event(worker_id: int, kv_cache_event_bytes: bytes) -> None
```
Apply a KV cache event to update the RadixTree state.
**Parameters**
**`worker_id`** `int`
ID of the worker that generated the event
---
**`kv_cache_event_bytes`** `bytes`
Serialized KV cache event as bytes
---
**Raises**
* `ValueError` — If the event bytes cannot be deserialized
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L990)
remove_worker
```python
remove_worker(worker_id: int) -> None
```
Remove all blocks associated with a specific worker.
**Parameters**
**`worker_id`** `int`
ID of the worker to remove
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1003)
clear_all_blocks
```python
clear_all_blocks(worker_id: int) -> None
```
Clear all blocks for a specific worker.
**Parameters**
**`worker_id`** `int`
ID of the worker whose blocks should be cleared
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1012)
dump_tree_as_events
```python
dump_tree_as_events() -> List[str]
```
Dump the current RadixTree state as a list of JSON-serialized KV cache events.
**Returns**
* `List[str]` — List of JSON-serialized KV cache events as strings
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1021)
#### RoutedEngine (class)
No summary available.
```python
from dynamo.llm import RoutedEngine
```
[`lib/bindings/python/src/dynamo/llm/__init__.py#L65`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/llm/__init__.py#L65)
**Public methods**
generate
```python
generate(request: Any, **kwargs: Any) -> AsyncIterator[Any]
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/llm/__init__.py#L66)
#### RouterConfig (class)
How to route the request
```python
from dynamo.llm import RouterConfig
```
```python
RouterConfig(mode: RouterMode, config: Optional[KvRouterConfig] = None, active_decode_blocks_threshold: Optional[float] = None, active_prefill_tokens_threshold: Optional[int] = None, active_prefill_tokens_threshold_frac: Optional[float] = None, enforce_disagg: bool = False, session_affinity_ttl_secs: Optional[int] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1693`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1693)
**Public methods**
**init**
```python
__init__(mode: RouterMode, config: Optional[KvRouterConfig] = None, active_decode_blocks_threshold: Optional[float] = None, active_prefill_tokens_threshold: Optional[int] = None, active_prefill_tokens_threshold_frac: Optional[float] = None, enforce_disagg: bool = False, session_affinity_ttl_secs: Optional[int] = None) -> None
```
Create a RouterConfig.
**Parameters**
**`mode`** `RouterMode`
The router mode (RoundRobin, Random, KV, Direct, LeastLoaded, or DeviceAwareWeighted)
---
**`config`** `Optional[KvRouterConfig]`
Optional KV router configuration (used when mode is KV)
---
**`active_decode_blocks_threshold`** `Optional[float]`
Threshold percentage (0.0-1.0) for decode blocks busy detection
---
**`active_prefill_tokens_threshold`** `Optional[int]`
Literal token count threshold for prefill busy detection
---
**`active_prefill_tokens_threshold_frac`** `Optional[float]`
Fraction of max\_num\_batched\_tokens for busy detection
---
**`enforce_disagg`** `bool`
Deprecated and ignored. Routing topology and readiness come from registered worker types.
---
**`session_affinity_ttl_secs`** `Optional[int]`
Router-local session-affinity idle TTL in seconds.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1698)
#### RouterMode (class)
Router mode for load balancing requests across workers
```python
from dynamo.llm import RouterMode
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1682`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1682)
#### RouterQueueLimitExceeded (class)
A policy-class queue cap rejected the request.
```python
from dynamo.llm import RouterQueueLimitExceeded
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3215`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3215)
#### RoutingConstraints (class)
Request-side routing constraints.
```python
from dynamo.llm import RoutingConstraints
```
```python
RoutingConstraints(required_taints: Optional[Set[str]] = None, preferred_taints: Optional[Dict[str, float]] = None) -> None
```
`required_taints` is a hard eligibility filter.
`preferred_taints` maps taint -> signed weight.
Positive weights prefer matching workers, negative weights avoid them,
and `0.0` is neutral. Matching weights are summed and squashed with
`tanh`, so opposite preferences cancel before Dynamo converts the
bounded bias into a strictly positive score multiplier.
[`lib/bindings/python/src/dynamo/_core.pyi#L914`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L914)
**Public methods**
**init**
```python
__init__(required_taints: Optional[Set[str]] = None, preferred_taints: Optional[Dict[str, float]] = None) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L928)
#### SelectionService (class)
In-process handle to a runtime-free Dynamo selection core.
```python
from dynamo.llm import SelectionService
```
```python
SelectionService(*, indexer_threads: int = 4, indexer_peers: Optional[list[str]] = None, replica_sync_port: Optional[int] = None, replica_sync_peers: Optional[list[str]] = None, selection_cache: Optional[SelectionCacheConfig] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L728`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L728)
**Public methods**
**init**
```python
__init__(*, indexer_threads: int = 4, indexer_peers: Optional[list[str]] = None, replica_sync_port: Optional[int] = None, replica_sync_peers: Optional[list[str]] = None, selection_cache: Optional[SelectionCacheConfig] = None) -> None
```
Create a selection service. `indexer_threads` sizes the KV indexer pool.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L733)
shutdown
```python
shutdown() -> None
```
Stop the service: cancel KV-event listeners and scheduling so that in-flight and queued selections fail fast.
The KV indexer thread pool is released when the handle is dropped.
Idempotent, and also runs automatically on drop.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L745)
upsert_worker
```python
upsert_worker(worker: JsonLike) -> JsonLike
```
Upsert a worker and subscribe to its live KV events; returns its catalog record.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L755)
delete_worker
```python
delete_worker(worker_id: int) -> JsonLike
```
Remove a worker and tear down its KV-event listener; returns its catalog record.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L759)
list_workers
```python
list_workers(*, model_name: Optional[str] = None, routing_group: Optional[str] = None) -> JsonLike
```
List catalog records, optionally filtered by model and routing group.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L763)
ready
```python
ready() -> JsonLike
```
Readiness: whether at least one worker is schedulable, plus catalog state.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L769)
overlap_scores
```python
overlap_scores(request: JsonLike) -> JsonLike
```
Per-worker KV-overlap scores for a prompt.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L773)
select
```python
select(request: JsonLike) -> JsonLike
```
Select the best worker by KV-overlap + load, without booking.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L777)
select_and_reserve
```python
select_and_reserve(request: JsonLike) -> JsonLike
```
Select the best worker and book its load.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L781)
create_reservation
```python
create_reservation(request: JsonLike) -> JsonLike
```
Book a request's load against a worker, keyed by `selection_id`.
Without a `worker_id`, replays the matching `select`'s cached
selection (same model/routing-group), booked under `selection_id`;
other request fields are ignored. With a `worker_id` and the prompt,
books explicitly under `selection_id` on that worker and discards any
cached selection for the id. `selection_id` is required.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L785)
prefill_complete
```python
prefill_complete(selection_id: str) -> None
```
Mark a reservation's prefill complete; its load shifts prefill -> decode.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L796)
add_output_block
```python
add_output_block(selection_id: str, *, decay_fraction: Optional[float] = None) -> None
```
Record one decode output block for a reservation, advancing its decode load.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L800)
free_reservation
```python
free_reservation(selection_id: str) -> None
```
Free a finished reservation, releasing its tracked load.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L806)
loads
```python
loads(*, model_name: Optional[str] = None, routing_group: Optional[str] = None) -> JsonLike
```
Current per-model active load (pending counts + per-worker potential loads).
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L810)
potential_loads
```python
potential_loads(request: JsonLike) -> JsonLike
```
Per-worker potential loads for a prompt, without booking.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L816)
#### WorkerMetricsPublisher (class)
A metrics publisher will provide metrics to the router for load monitoring.
```python
from dynamo.llm import WorkerMetricsPublisher
```
```python
WorkerMetricsPublisher() -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L645`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L645)
**Public methods**
**init**
```python
__init__() -> None
```
Create a `WorkerMetricsPublisher` object
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L652)
create_endpoint
```python
create_endpoint(endpoint: Endpoint) -> None
```
Initialize event-plane publishing for worker metrics. Must be awaited.
Extracts component information from the endpoint to set up metrics publishing
on the endpoint-scoped event subject used for routing decisions.
**Parameters**
**`endpoint`** `Endpoint`
The endpoint to extract component information from for metrics publishing
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L657)
publish
```python
publish(dp_rank: Optional[int] = None, active_decode_blocks: int | None = None, kv_used_blocks: int | None = None) -> None
```
Publish worker metrics for load monitoring.
**Parameters**
**`dp_rank`** `Optional[int]`
Data parallel rank of the worker (None defaults to 0)
---
**`active_decode_blocks`** `int | None`
Optional scheduler-compatible decode-block signal
---
**`kv_used_blocks`** `int | None`
Optional authoritative total KV blocks currently in use
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L668)
#### WorkerType (class)
Processing stage a worker handles.
```python
from dynamo.llm import WorkerType
```
Each worker has exactly one role; values are not combinable. Use the
`needs` argument on register\_model to express dependencies in DNF form
(a list of alternative AND-sets) — for example, an encode worker that
needs (Prefill AND Decode) OR a single Aggregated peer is expressed as
`[[WorkerType.Prefill, WorkerType.Decode], [WorkerType.Aggregated]]`.
[`lib/bindings/python/src/dynamo/_core.pyi#L2205`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2205)
#### compute\_block\_hash\_for\_seq (function)
Compute block hashes for a sequence of tokens, optionally including multimodal metadata.
```python
from dynamo.llm import compute_block_hash_for_seq
```
```python
compute_block_hash_for_seq(tokens: List[int], kv_block_size: int, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, is_eagle: Optional[bool] = None, cache_namespace: Optional[str] = None) -> List[int]
```
When block\_mm\_infos is provided, the mm\_hashes are included in the hash computation
to ensure that blocks with identical tokens but different multimodal objects produce
different hashes.
**Parameters**
**`tokens`** `List[int]`
List of token IDs
---
**`kv_block_size`** `int`
Size of each block in tokens
---
**`block_mm_infos`** `Optional[List[Optional[Dict[str, Any]]]]`
Optional per-block multimodal metadata. Each element corresponds to a block
and should be None or a dict with structure:
\{
"mm\_objects": \[
\{
"mm\_hash": int, # Hash of the MM object
}
]
}
---
**`lora_name`** `Optional[str]`
Optional LoRA adapter name for adapter-aware block hashing.
---
**`is_eagle`** `Optional[bool]`
Optional Eagle mode flag. When true, hashes use overlapping
`kv_block_size + 1` token windows with `kv_block_size` stride.
---
**Returns**
* `List[int]` — List of block hashes (one per block)
\>>> tokens = \[1, 2, 3, 4] \* 8 # 32 tokens = 1 block
\>>> mm\_info = \{
... "mm\_objects": \[\{
... "mm\_hash": 0xDEADBEEF,
... }]
... }
\>>> hashes = compute\_block\_hash\_for\_seq(tokens, 32, \[mm\_info])
[`lib/bindings/python/src/dynamo/_core.pyi#L396`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L396)
#### fetch\_model (function)
Download a model from Hugging Face, returning its local path. If `ignore_weights` is True, only fetches tokenizer and config files. Example: `model_path = await fetch_model("Qwen/Qwen3-0.6B")`
```python
from dynamo.llm import fetch_model
```
```python
fetch_model(remote_name: str, ignore_weights: bool = False) -> str
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2335`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2335)
#### lora\_name\_to\_id (function)
Generate a deterministic integer ID from a LoRA name using blake3 hash.
```python
from dynamo.llm import lora_name_to_id
```
```python
lora_name_to_id(lora_name: str) -> int
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2293`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2293)
#### make\_engine (function)
Make an engine matching the args
```python
from dynamo.llm import make_engine
```
```python
make_engine(distributed_runtime: DistributedRuntime, args: EntrypointArgs) -> EngineConfig
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2352`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2352)
#### register\_model (function)
Attach the model at path to the given endpoint, and advertise it as model\_type. LoRA Registration: The `lora_name` and `base_model_path` parameters must be provided together or not at all. Providing only one of these parameters will raise a ValueError. - `lora_name`: The served model name for the LoRA model - `base_model_path`: Path to the base model that the LoRA extends
```python
from dynamo.llm import register_model
```
```python
register_model(model_input: ModelInput, model_type: ModelType, endpoint: Endpoint, model_path: str, model_name: Optional[str] = None, *, worker_type: WorkerType, kv_cache_block_size: Optional[int] = None, router_mode: Optional[RouterMode] = None, runtime_config: Optional[ModelRuntimeConfig] = None, tensor_model_config: Optional[Dict[str, Any]] = None, user_data: Optional[Dict[str, Any]] = None, custom_template_path: Optional[str] = None, media_decoder: Optional[MediaDecoder] = None, media_fetcher: Optional[MediaFetcher] = None, lora_name: Optional[str] = None, base_model_path: Optional[str] = None, needs: Optional[List[List[WorkerType]]] = None, self_host_metadata: Optional[bool] = None, ignore_weights: bool = False, max_gpu_lora_count: Optional[int] = None, model_aliases: Optional[List[str]] = None) -> None
```
For TensorBased models (using ModelInput.Tensor), HuggingFace downloads are skipped
and a minimal model card is registered directly. Use model\_path as the display name
for these models. Pass tensor protocol metadata through `tensor_model_config`.
Model serving readiness:
`worker_type` and `needs` describe the worker's processing stage and
peer dependencies. `needs` is a DNF list — each inner list is an
AND-set, the outer list is OR. `worker_type` is required; backends
declare it literally at each call site.
When `ignore_weights` is true, remote HuggingFace model resolution skips
weight files and downloads only the metadata needed for registration.
[`lib/bindings/python/src/dynamo/_core.pyi#L2224`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2224)
#### run\_input (function)
Start an engine, connect it to an input, and run until stopped.
```python
from dynamo.llm import run_input
```
```python
run_input(distributed_runtime: DistributedRuntime, input: str, engine_config: EngineConfig, frontend_route_extensions: Optional[Sequence[FrontendRoute]] = None) -> None
```
`frontend_route_extensions` supplies additional HTTP routes to the
frontend (HTTP input only); see `FrontendRoute`.
[`lib/bindings/python/src/dynamo/_core.pyi#L2418`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2418)
#### run\_kv\_indexer (function)
Run the KV indexer with the given arguments.
```python
from dynamo.llm import run_kv_indexer
```
```python
run_kv_indexer(args: List[str]) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L41`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L41)
#### run\_select\_service (function)
Run the Dynamo selection service with the given arguments.
```python
from dynamo.llm import run_select_service
```
```python
run_select_service(args: List[str]) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L49`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L49)
#### run\_slot\_tracker (function)
Run the KV router slot tracker with the given arguments.
```python
from dynamo.llm import run_slot_tracker
```
```python
run_slot_tracker(args: List[str]) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L45`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L45)
#### unregister\_model (function)
Unregister a model from the discovery system.
```python
from dynamo.llm import unregister_model
```
```python
unregister_model(endpoint: Endpoint, lora_name: Optional[str] = None) -> None
```
If lora\_name is provided, unregisters a LoRA adapter instead of a base model.
[`lib/bindings/python/src/dynamo/_core.pyi#L2271`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2271)
#### update\_model\_taints (function)
Replace caller-managed taints on this worker's registered model.
```python
from dynamo.llm import update_model_taints
```
```python
update_model_taints(endpoint: Endpoint, taints: Set[str]) -> None
```
Reserved 'dynamo.topology/' taints are derived from the model's topology
metadata and cannot be supplied by callers.
[`lib/bindings/python/src/dynamo/_core.pyi#L2282`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2282)