> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.
# dynamo._core
`dynamo._core` publishes 74 classes and 15 functions. Source: [`lib/bindings/python/src/dynamo/_core.pyi`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi)
#### AicPerfConfig (class)
No summary available.
```python
from dynamo._core import AicPerfConfig
```
```python
AicPerfConfig(aic_backend: str, aic_system: str, aic_model_path: str, aic_tp_size: int = 1, aic_backend_version: Optional[str] = None, aic_moe_tp_size: Optional[int] = None, aic_moe_ep_size: Optional[int] = None, aic_attention_dp_size: Optional[int] = None, aic_nextn: Optional[int] = None, aic_nextn_accept_rates: Optional[str] = None, aic_gemm_dtype: Optional[str] = None, aic_moe_dtype: Optional[str] = None, aic_fmha_dtype: Optional[str] = None, aic_kv_cache_dtype: Optional[str] = None, aic_comm_dtype: Optional[str] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1722`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1722)
**Public methods**
**init**
```python
__init__(aic_backend: str, aic_system: str, aic_model_path: str, aic_tp_size: int = 1, aic_backend_version: Optional[str] = None, aic_moe_tp_size: Optional[int] = None, aic_moe_ep_size: Optional[int] = None, aic_attention_dp_size: Optional[int] = None, aic_nextn: Optional[int] = None, aic_nextn_accept_rates: Optional[str] = None, aic_gemm_dtype: Optional[str] = None, aic_moe_dtype: Optional[str] = None, aic_fmha_dtype: Optional[str] = None, aic_kv_cache_dtype: Optional[str] = None, aic_comm_dtype: Optional[str] = None) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1723)
#### ApproxKvIndexer (class)
An approximate KV Indexer that doesn't receive KV cache events from workers. Instead, it relies on routing decisions with TTL-based expiration and pruning to estimate which blocks are cached on which workers.
```python
from dynamo._core import ApproxKvIndexer
```
```python
ApproxKvIndexer(endpoint: Endpoint, kv_block_size: int, router_ttl_secs: float = 120.0) -> None
```
This is useful when:
* Backend engines don't emit KV events
* You want to reduce event processing overhead
* Lower routing accuracy is acceptable
[`lib/bindings/python/src/dynamo/_core.pyi#L1068`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1068)
**Public methods**
**init**
```python
__init__(endpoint: Endpoint, kv_block_size: int, router_ttl_secs: float = 120.0) -> None
```
Create an `ApproxKvIndexer` object
**Parameters**
**`component`**
The component to associate with this indexer
---
**`kv_block_size`** `int`
The KV cache block size
---
**`router_ttl_secs`** `float`
TTL for blocks in seconds (default: 120.0)
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1082)
find_matches_for_request
```python
find_matches_for_request(token_ids: List[int], lora_name: Optional[str] = None, is_eagle: Optional[bool] = None) -> OverlapScores
```
Return the overlapping scores of workers for the given token ids.
**Parameters**
**`token_ids`** `List[int]`
List of token IDs to find matches for
---
**`lora_name`** `Optional[str]`
Optional LoRA adapter name for adapter-aware matching
---
**Returns**
* `OverlapScores` — OverlapScores containing worker matching scores and frequencies
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1098)
block_size
```python
block_size() -> int
```
Return the block size of the ApproxKvIndexer.
**Returns**
* `int` — The KV cache block size
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1113)
process_routing_decision_for_request
```python
process_routing_decision_for_request(tokens: List[int], worker_id: int, dp_rank: int = 0) -> None
```
Notify the indexer that a token sequence has been routed to a specific worker.
This updates the indexer's internal state to track which blocks are likely
cached on which workers based on routing decisions.
**Parameters**
**`tokens`** `List[int]`
List of token IDs that were routed
---
**`worker_id`** `int`
The worker ID the request was routed to
---
**`dp_rank`** `int`
The data parallel rank (default: 0)
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1122)
#### Block (class)
A KV cache block
```python
from dynamo._core import Block
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2602`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2602)
**Public methods**
to_list
```python
to_list() -> List[Layer]
```
Get a list of layers
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2633)
#### BlockList (class)
A list of KV cache blocks
```python
from dynamo._core import BlockList
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2652`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2652)
**Public methods**
to_list
```python
to_list() -> List[Block]
```
Get a list of blocks
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2683)
#### BlockManager (class)
A KV cache block manager
```python
from dynamo._core import BlockManager
```
```python
BlockManager(worker_id: int, num_layer: int, page_size: int, inner_dim: int, dtype: Optional[str] = None, host_num_blocks: Optional[int] = None, device_num_blocks: Optional[int] = None, device_id: int = 0) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2689`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2689)
**Public methods**
**init**
```python
__init__(worker_id: int, num_layer: int, page_size: int, inner_dim: int, dtype: Optional[str] = None, host_num_blocks: Optional[int] = None, device_num_blocks: Optional[int] = None, device_id: int = 0) -> None
```
Create a `BlockManager` object
## Parameters:
worker\_id: int
The worker ID for this block manager
num\_layer: int
Number of layers in the model
page\_size: int
Page size for blocks
inner\_dim: int
Inner dimension size
dtype: Optional\[str]
Data type (e.g., 'fp16', 'bf16', 'fp32'), defaults to 'fp16' if None
host\_num\_blocks: Optional\[int]
Number of host blocks to allocate, None means no host blocks
device\_num\_blocks: Optional\[int]
Number of device blocks to allocate, None means no device blocks
device\_id: int
CUDA device ID, defaults to 0
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2694)
allocate_host_blocks_blocking
```python
allocate_host_blocks_blocking(count: int) -> BlockList
```
Allocate a list of host blocks (blocking call)
## Parameters:
count: int
Number of blocks to allocate
## Returns:
BlockList
List of allocated blocks
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2729)
allocate_host_blocks
```python
allocate_host_blocks(count: int) -> BlockList
```
Allocate a list of host blocks
## Parameters:
count: int
Number of blocks to allocate
## Returns:
BlockList
List of allocated blocks
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2745)
allocate_device_blocks_blocking
```python
allocate_device_blocks_blocking(count: int) -> BlockList
```
Allocate a list of device blocks (blocking call)
## Parameters:
count: int
Number of blocks to allocate
## Returns:
BlockList
List of allocated blocks
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2761)
allocate_device_blocks
```python
allocate_device_blocks(count: int) -> BlockList
```
Allocate a list of device blocks
## Parameters:
count: int
Number of blocks to allocate
## Returns:
BlockList
List of allocated blocks
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2777)
#### Cancelled (class)
The request was cancelled.
```python
from dynamo._core import Cancelled
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3248`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3248)
#### CannotConnect (class)
Failed to establish a connection.
```python
from dynamo._core import CannotConnect
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3233`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3233)
#### Client (class)
A client capable of calling served instances of an endpoint
```python
from dynamo._core import Client
```
[`lib/bindings/python/src/dynamo/_core.pyi#L288`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L288)
**Public methods**
instance_ids
```python
instance_ids() -> List[int]
```
Get list of current instance IDs.
**Returns**
* `List[int]` — A list of currently available instance IDs
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L295)
instances
```python
instances() -> List[Instance]
```
Get a snapshot of the current instances with full transport details.
Like `instance_ids()`, the result is a snapshot of the watched
instance set; pair with `wait_for_instances()` to block until
instances exist.
**Returns**
* `List[Instance]` — A list of `Instance` for the currently available instances,
* `List[Instance]` — across all transports (TCP, NATS, ...).
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L304)
wait_for_instances
```python
wait_for_instances() -> List[int]
```
Wait for instances to be available for work and return their IDs.
**Returns**
* `List[int]` — A list of instance IDs that are available for work
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L318)
wait_for_instance_by_runtime_data
```python
wait_for_instance_by_runtime_data(key: str, value: str, timeout_s: float | None = None) -> int
```
Wait for exactly one instance whose MDC runtime\_data contains the given string value.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L327)
random
```python
random(request: JsonLike, annotated: bool | None = True, context: Context | None = None) -> AsyncIterator[JsonLike]
```
Pick a random instance of the endpoint and issue the request
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L338)
round_robin
```python
round_robin(request: JsonLike, annotated: bool | None = True, context: Context | None = None) -> AsyncIterator[JsonLike]
```
Pick the next instance of the endpoint in a round-robin fashion
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L349)
direct
```python
direct(request: JsonLike, instance_id: int, annotated: bool | None = True, context: Context | None = None) -> AsyncIterator[JsonLike]
```
Pick a specific instance of the endpoint
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L360)
generate
```python
generate(request: JsonLike, annotated: bool | None = True, context: Context | None = None) -> AsyncIterator[JsonLike]
```
Generate a response from the endpoint
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L372)
#### ConnectionTimeout (class)
A connection or request timed out.
```python
from dynamo._core import ConnectionTimeout
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3243`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3243)
#### Context (class)
Context wrapper around AsyncEngineContext for Python bindings. Provides tracing and cancellation capabilities for request handling.
```python
from dynamo._core import Context
```
```python
Context(id: Optional[str] = None, metadata: Optional[Dict[str, str]] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L460`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L460)
**Public methods**
**init**
```python
__init__(id: Optional[str] = None, metadata: Optional[Dict[str, str]] = None) -> None
```
Create a new Context instance.
**Parameters**
**`id`** `Optional[str]`
Optional request ID. If None, a default ID will be generated.
---
**`metadata`** `Optional[Dict[str, str]]`
Optional propagated metadata map.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L466)
is_stopped
```python
is_stopped() -> bool
```
Check if the context has been stopped (synchronous).
**Returns**
* `bool` — True if the context is stopped, False otherwise.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L480)
is_killed
```python
is_killed() -> bool
```
Check if the context has been killed (synchronous).
**Returns**
* `bool` — True if the context is killed, False otherwise.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L489)
stop_generating
```python
stop_generating() -> None
```
Issue a stop generating signal to the context.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L498)
id
```python
id() -> str
```
Get the context ID.
**Returns**
* `str` — The context identifier string.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L504)
detached
```python
detached(id: str) -> Context
```
Create a context with a fresh cancellation controller and request ID while preserving trace parentage and a metadata snapshot.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L513)
async_killed_or_stopped
```python
async_killed_or_stopped() -> asyncio.Future[bool]
```
Asynchronously wait until the context is killed or stopped.
**Returns**
* `asyncio.Future[bool]` — True when the context is killed or stopped.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L520)
notify_first_token
```python
notify_first_token() -> None
```
Fire the first-token signal so the framework can release any deferred `engine.abort()`. Idempotent; no-op on non-decode requests. Engines normally don't need this — the framework auto-fires on the first non-empty chunk in the response stream.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L529)
```python
trace_headers() -> Optional[dict[str, str]]
```
Build W3C trace headers for propagating to downstream inference engines.
**Returns**
* `Optional[dict[str, str]]` — `{"traceparent": "00---"}` when this
* `Optional[dict[str, str]]` — request carries trace context, `None` otherwise. Also emits `tracestate`,
* `Optional[dict[str, str]]` — `x-request-id`, `request-id` when upstream propagated them.
* `Optional[dict[str, str]]` — Forward unchanged to the inference engine's `trace_headers` kwarg.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L576)
current_span
```python
current_span() -> SpanProxy
```
Handle on the framework's `engine.generate` span. Use it to `set_attribute` / `add_event` / `set_status` on the parent span. Returns a silent no-op proxy when no parent was plumbed in (test contexts) or the OTel bridge isn't installed.
Engines normally reach this through
`dynamo.common.backend.telemetry.current_span(context)`.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L588)
start_span
```python
start_span(name: str, attrs: Optional[dict[str, Any]] = None) -> SpanProxy
```
Open a child span under `engine.generate` with a dynamic name. The returned `SpanProxy` is a context manager — the span ends on `__exit__` / `close()` / drop.
Engines normally reach this through
`dynamo.common.backend.telemetry.start_span(context, name)`.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L600)
#### ContextMetadata (class)
Live mutable view over propagated context metadata.
```python
from dynamo._core import ContextMetadata
```
[`lib/bindings/python/src/dynamo/_core.pyi#L442`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L442)
**Public methods**
get
```python
get(key: str, default: Optional[str] = None) -> Optional[str]
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L452)
pop
```python
pop(key: str, default: Optional[str] = None) -> Optional[str]
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L453)
keys
```python
keys() -> List[str]
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L454)
values
```python
values() -> List[str]
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L455)
items
```python
items() -> List[Tuple[str, str]]
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L456)
clear
```python
clear() -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L457)
copy
```python
copy() -> Dict[str, str]
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L458)
#### Disconnected (class)
An established connection was lost.
```python
from dynamo._core import Disconnected
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3238`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3238)
#### DistributedRuntime (class)
The runtime object for dynamo applications
```python
from dynamo._core import DistributedRuntime
```
[`lib/bindings/python/src/dynamo/_core.pyi#L58`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L58)
**Public methods**
endpoint
```python
endpoint(path: str) -> Endpoint
```
Get an endpoint directly by path.
**Parameters**
**`path`** `str`
Endpoint path in format 'namespace.component.endpoint'
or 'dyn://namespace.component.endpoint'
---
**Returns**
* `Endpoint` — The requested endpoint
**Raises**
* `ValueError` — If path format is invalid (not 3 parts separated by dots)
* `Exception` — If namespace or component creation fails
endpoint = runtime.endpoint("demo.backend.generate")
endpoint = runtime.endpoint("dyn://demo.backend.generate")
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L84)
shutdown
```python
shutdown() -> None
```
Shutdown the runtime by triggering the cancellation token
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L105)
set_health_status
```python
set_health_status(ready: bool) -> None
```
Explicitly set the system-level health status (Ready / NotReady).
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L111)
register_engine_route
```python
register_engine_route(route_name: str, callback: Callable[[dict], Awaitable[dict]]) -> None
```
Register an async callback for /engine/\{route\_name} on the system status server.
**Parameters**
**`route_name`** `str`
The route path (e.g., "control/start\_profile" creates /engine/control/start\_profile)
---
**`callback`** `Callable[[dict], Awaitable[dict]]`
Async function with signature: async def(body: dict) -> dict
---
async def start\_profile(body: dict) -> dict:
await engine.start\_profile(\*\*body)
return \{"status": "ok", "message": "Profiling started"}
runtime.register\_engine\_route("control/start\_profile", start\_profile)
The callback receives the JSON request body as a dict and should return
a dict that will be serialized as the JSON response.
For GET requests or empty bodies, an empty dict \{} is passed.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L117)
#### DynamoException (class)
Base exception for all Dynamo error types.
```python
from dynamo._core import DynamoException
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3210`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3210)
#### Endpoint (class)
An Endpoint is a single API endpoint
```python
from dynamo._core import Endpoint
```
[`lib/bindings/python/src/dynamo/_core.pyi#L144`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L144)
**Public methods**
serve_endpoint
```python
serve_endpoint(handler: RequestHandler, graceful_shutdown: bool = True, metrics_labels: Optional[List[Tuple[str, str]]] = None, health_check_payload: Optional[Dict[str, Any]] = None) -> None
```
Serve an endpoint discoverable by all connected clients at `{{ namespace }}/components/{{ component_name }}/endpoints/{{ endpoint_name }}`
**Parameters**
**`handler`** `RequestHandler`
The request handler function
---
**`graceful_shutdown`** `bool`
Whether to wait for inflight requests to complete during shutdown (default: True)
---
**`metrics_labels`** `Optional[List[Tuple[str, str]]]`
Optional list of metrics labels to add to the metrics
---
**`health_check_payload`** `Optional[Dict[str, Any]]`
Optional dict containing the health check request payload
that will be used to verify endpoint health
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L151)
serve_bidirectional_endpoint
```python
serve_bidirectional_endpoint(handler: Callable[..., AsyncIterator[JsonLike]], graceful_shutdown: bool = True, metrics_labels: Optional[List[Tuple[str, str]]] = None) -> None
```
Serve a bidirectional (streaming-input, streaming-output) endpoint.
The handler is an async generator function — `async def
generate(request_stream)` or `async def generate(request_stream,
context)` — so calling it returns an async iterator of response frames
directly (it is not awaited). `request_stream` is a
`PyAsyncRequestStream` yielding inbound frames as JSON-like Python
objects; the generator yields response frames as JSON-like Python
objects.
Request-stream end (when `__anext__` raises `StopAsyncIteration`)
is not a cancellation signal: the caller has merely stopped sending
input. The engine must keep yielding response chunks until it
chooses to return or observes `context.is_stopped()`.
**Parameters**
**`handler`** `Callable[..., AsyncIterator[JsonLike]]`
The async generator factory described above
---
**`graceful_shutdown`** `bool`
Whether to wait for inflight requests to complete during shutdown (default: True)
---
**`metrics_labels`** `Optional[List[Tuple[str, str]]]`
Optional list of metrics labels to add to the metrics
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L165)
client
```python
client(router_mode: Optional[RouterMode] = None) -> Client
```
Create a `Client` capable of calling served instances of this endpoint.
By default this uses round-robin routing when `router_mode` is not provided.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L194)
connection_id
```python
connection_id() -> int
```
Opaque unique ID for this worker. May change over worker lifetime.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L202)
unregister_endpoint_instance
```python
unregister_endpoint_instance() -> None
```
Unregister this endpoint instance from discovery.
This removes the endpoint from the instances bucket, preventing the router
from sending requests to this worker. Use this when a worker is sleeping
and should not receive any requests.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L218)
register_endpoint_instance
```python
register_endpoint_instance() -> None
```
Re-register this endpoint instance to discovery.
This adds the endpoint back to the instances bucket, allowing the router
to send requests to this worker again. Use this when a worker wakes up
and should start receiving requests.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L228)
#### EngineConfig (class)
Holds internal configuration for a Dynamo engine.
```python
from dynamo._core import EngineConfig
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2348`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2348)
#### EngineShutdown (class)
The engine process has shut down or crashed.
```python
from dynamo._core import EngineShutdown
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3253`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3253)
#### EngineType (class)
Engine type for Dynamo workers
```python
from dynamo._core import EngineType
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3069`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3069)
#### EntrypointArgs (class)
Settings to connect an input to a worker and run them. Use by `dynamo run`.
```python
from dynamo._core import EntrypointArgs
```
```python
EntrypointArgs(engine_type: EngineType, model_path: Optional[str] = None, model_name: Optional[str] = None, endpoint_id: Optional[str] = None, template_file: Optional[str] = None, router_config: Optional[RouterConfig] = None, kv_cache_block_size: Optional[int] = None, http_host: Optional[str] = None, http_port: Optional[int] = None, http_metrics_port: Optional[int] = None, tls_cert_path: Optional[str] = None, tls_key_path: Optional[str] = None, extra_engine_args: Optional[str] = None, mocker_engine_args: Optional[MockEngineArgs] = None, runtime_config: Optional[ModelRuntimeConfig] = None, namespace: Optional[str] = None, namespace_prefix: Optional[str] = None, is_prefill: bool = False, is_decode: bool = False, migration_limit: int = 0, migration_max_seq_len: Optional[int] = None, chat_engine_factory: Optional[Callable] = None, aic_perf_config: Optional[AicPerfConfig] = None, *, metrics_prefix: Optional[str] = None, enable_anthropic_api: Optional[bool] = None, strip_anthropic_preamble: Optional[bool] = None, enable_streaming_tool_dispatch: Optional[bool] = None, enable_streaming_reasoning_dispatch: Optional[bool] = None, tokenizer_backend: Optional[str] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3076`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3076)
**Public methods**
**init**
```python
__init__(engine_type: EngineType, model_path: Optional[str] = None, model_name: Optional[str] = None, endpoint_id: Optional[str] = None, template_file: Optional[str] = None, router_config: Optional[RouterConfig] = None, kv_cache_block_size: Optional[int] = None, http_host: Optional[str] = None, http_port: Optional[int] = None, http_metrics_port: Optional[int] = None, tls_cert_path: Optional[str] = None, tls_key_path: Optional[str] = None, extra_engine_args: Optional[str] = None, mocker_engine_args: Optional[MockEngineArgs] = None, runtime_config: Optional[ModelRuntimeConfig] = None, namespace: Optional[str] = None, namespace_prefix: Optional[str] = None, is_prefill: bool = False, is_decode: bool = False, migration_limit: int = 0, migration_max_seq_len: Optional[int] = None, chat_engine_factory: Optional[Callable] = None, aic_perf_config: Optional[AicPerfConfig] = None, *, metrics_prefix: Optional[str] = None, enable_anthropic_api: Optional[bool] = None, strip_anthropic_preamble: Optional[bool] = None, enable_streaming_tool_dispatch: Optional[bool] = None, enable_streaming_reasoning_dispatch: Optional[bool] = None, tokenizer_backend: Optional[str] = None) -> None
```
Create EntrypointArgs.
**Parameters**
**`engine_type`** `EngineType`
The type of engine to use
---
**`model_path`** `Optional[str]`
Path to the model directory on disk
---
**`model_name`** `Optional[str]`
Model name or dynamo endpoint (e.g. 'dyn://namespace.component.endpoint')
---
**`endpoint_id`** `Optional[str]`
Optional endpoint ID
---
**`template_file`** `Optional[str]`
Optional path to a prompt template file
---
**`router_config`** `Optional[RouterConfig]`
Optional router configuration
---
**`kv_cache_block_size`** `Optional[int]`
Optional KV cache block size
---
**`http_host`** `Optional[str]`
HTTP host to bind to
---
**`http_port`** `Optional[int]`
HTTP port to bind to
---
**`http_metrics_port`** `Optional[int]`
HTTP metrics port (for gRPC service)
---
**`tls_cert_path`** `Optional[str]`
TLS certificate path (PEM format)
---
**`tls_key_path`** `Optional[str]`
TLS key path (PEM format)
---
**`extra_engine_args`** `Optional[str]`
Optional path to mocker engine arguments JSON
---
**`mocker_engine_args`** `Optional[MockEngineArgs]`
Typed mocker engine arguments
---
**`runtime_config`** `Optional[ModelRuntimeConfig]`
Optional runtime configuration for discovery registration
---
**`namespace`** `Optional[str]`
Dynamo namespace for model discovery scoping
---
**`namespace_prefix`** `Optional[str]`
Optional namespace prefix
---
**`is_prefill`** `bool`
Whether this is a prefill worker
---
**`is_decode`** `bool`
Whether this is a decode worker (disaggregated); pairs with a prefill peer for readiness
---
**`migration_limit`** `int`
Maximum number of request migrations (0=disabled)
---
**`migration_max_seq_len`** `Optional[int]`
Optional max sequence length for migration
---
**`chat_engine_factory`** `Optional[Callable]`
Optional Python chat completions engine factory callback
---
**`aic_perf_config`** `Optional[AicPerfConfig]`
Optional AIC perf-model configuration for default KV routing
---
**`metrics_prefix`** `Optional[str]`
Optional Prometheus metrics prefix override
---
**`enable_anthropic_api`** `Optional[bool]`
Optional Anthropic Messages API override
---
**`strip_anthropic_preamble`** `Optional[bool]`
Optional Anthropic preamble stripping override
---
**`enable_streaming_tool_dispatch`** `Optional[bool]`
Optional streaming tool dispatch override
---
**`enable_streaming_reasoning_dispatch`** `Optional[bool]`
Optional streaming reasoning dispatch override
---
**`tokenizer_backend`** `Optional[str]`
Optional tokenizer backend override ("default" or "fastokens")
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3082)
#### FpmDirectPublisher (class)
Direct Forward Pass Metrics publisher used by in-process producers such as the TRT-LLM adapter. The underlying Rust publisher owns per-DP-rank serialization tasks (each with its own 1s idle heartbeat timer) and a single event-plane publisher task. Python callers do not manage heartbeat: when `publish` is not called for `IDLE_HEARTBEAT_INTERVAL` (1.0s, matching vLLM's `HEARTBEAT_INTERVAL`), the Rust side emits a zeroed snapshot on that rank's channel.
```python
from dynamo._core import FpmDirectPublisher
```
```python
FpmDirectPublisher(endpoint: Endpoint, worker_id: str, dp_size: int = 1) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1285`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1285)
**Public methods**
**init**
```python
__init__(endpoint: Endpoint, worker_id: str, dp_size: int = 1) -> None
```
Create a publisher with `dp_size` per-DP-rank channels.
**Parameters**
**`endpoint`** `Endpoint`
Dynamo component endpoint (provides runtime + discovery).
---
**`worker_id`** `str`
Unique worker identifier stamped on every emitted FPM.
---
**`dp_size`** `int`
Number of DP ranks to allocate channels for. Use `1`
when attention DP is disabled.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1296)
publish
```python
publish(*, dp_rank: int, scheduled_num_prefill_requests: int, scheduled_sum_prefill_tokens: int, scheduled_sum_prefill_kv_tokens: int, scheduled_num_decode_requests: int, scheduled_sum_decode_kv_tokens: int, queued_num_prefill_requests: int, queued_sum_prefill_tokens: int, queued_num_decode_requests: int, queued_sum_decode_kv_tokens: int, wall_time_secs: float) -> None
```
Publish one iteration's FPM snapshot for the given DP rank.
All parameters are keyword-only on the Python side: adjacent ints
with similar units (`scheduled_*` vs `queued_*`, `*_prefill_*`
vs `*_decode_*`) cannot be distinguished by the type system, so
a transposition would silently corrupt every published snapshot.
Variance fields (var\_prefill\_length, var\_decode\_kv\_tokens,
var\_queued\_prefill\_length, var\_queued\_decode\_kv\_tokens) are defaulted
to 0.0 per the MVP scope; a follow-up PR can add Welford-based
variance computation.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1313)
shutdown
```python
shutdown() -> None
```
Shut down the publisher and its per-rank serialization tasks.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1343)
#### FpmEventRelay (class)
Relay that bridges ForwardPassMetrics from a local raw ZMQ PUB socket (InstrumentedScheduler in EngineCore child process) to the Dynamo event plane with automatic discovery registration.
```python
from dynamo._core import FpmEventRelay
```
```python
FpmEventRelay(endpoint: Endpoint, zmq_endpoint: str) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1258`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1258)
**Public methods**
**init**
```python
__init__(endpoint: Endpoint, zmq_endpoint: str) -> None
```
Create a relay.
**Parameters**
**`endpoint`** `Endpoint`
Dynamo component endpoint (provides runtime + discovery).
---
**`zmq_endpoint`** `str`
Local ZMQ PUB address to subscribe to
(e.g., "tcp\://127.0.0.1:20380").
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1265)
shutdown
```python
shutdown() -> None
```
Shut down the relay task.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1280)
#### FpmEventSubscriber (class)
Subscriber for ForwardPassMetrics from the Dynamo event plane. Auto-discovers engine publishers via the discovery plane.
```python
from dynamo._core import FpmEventSubscriber
```
```python
FpmEventSubscriber(endpoint: Endpoint) -> None
```
Two mutually exclusive usage modes:
1. **recv mode** (default): call `recv()` to pull individual messages.
2. **tracking mode**: call `start_tracking()` once, then poll
`get_recent_stats()` to retrieve the latest FPM bytes keyed by
`(worker_id, dp_rank)`. Stale entries are cleaned up when
workers are removed (via discovery watch).
[`lib/bindings/python/src/dynamo/_core.pyi#L1348`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1348)
**Public methods**
**init**
```python
__init__(endpoint: Endpoint) -> None
```
Create a subscriber that auto-discovers FPM publishers.
No background tasks are started until `recv()` or
`start_tracking()` is called.
**Parameters**
**`endpoint`** `Endpoint`
Dynamo component endpoint (provides runtime + discovery).
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1362)
recv
```python
recv() -> Optional[bytes]
```
Blocking receive of the next message (raw msgspec bytes). Releases the GIL while waiting.
On the first call a background subscriber task is spawned (recv mode).
Cannot be used after `start_tracking()`.
**Returns**
* `Optional[bytes]` — Raw msgspec payload, or None if the stream is closed.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1374)
start_tracking
```python
start_tracking() -> None
```
Start background tracking of the latest FPM per (worker\_id, dp\_rank).
Spawns two background tasks:
1. Event consumption: subscribes to FPM events, extracts the composite
key (worker\_id, dp\_rank) from the msgpack payload, stores latest
raw bytes in an internal map.
2. MDC discovery watch: monitors ComponentModels for the target
component. When a model is removed, all entries whose
worker\_id matches the removed instance\_id are purged.
After calling this, `recv()` will raise RuntimeError.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1387)
get_recent_stats
```python
get_recent_stats() -> dict[tuple[str, int], bytes]
```
Return the latest FPM bytes for every tracked (worker\_id, dp\_rank).
Cleanup of removed engines is handled by the MDC discovery watch
task spawned by `start_tracking()`.
Raises RuntimeError if `start_tracking()` has not been called.
**Returns**
* `dict[tuple[str, int], bytes]` — dict mapping `(worker_id, dp_rank)` to raw msgspec bytes.
* `dict[tuple[str, int], bytes]` — Decode each value with `forward_pass_metrics.decode(data)`.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1404)
get_model_cards
```python
get_model_cards() -> dict[str, str]
```
Snapshot of model deployment cards keyed by worker id.
The snapshot is filtered against the known-workers set so entries
for already-removed workers are not returned. Values are the raw
`ModelDeploymentCard` serialized as a JSON string; callers parse
whichever fields they need (e.g. `runtime_config`,
`display_name`).
Raises RuntimeError if `start_tracking()` has not been called.
**Returns**
* `dict[str, str]` — dict mapping `worker_id` to `card_json` (JSON string).
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1419)
shutdown
```python
shutdown() -> None
```
Shut down the subscriber (all background tasks).
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1436)
#### FrontendExtensionContext (class)
Read-only, live view of frontend state passed to extension route handlers.
```python
from dynamo._core import FrontendExtensionContext
```
Handlers receive this and answer from current state. The surface is
intentionally narrow (typed read-only accessors only); it does not expose
the internal service state.
[`lib/bindings/python/src/dynamo/_core.pyi#L2356`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2356)
**Public methods**
is_ready
```python
is_ready() -> bool
```
Whether the HTTP service has finished startup and is ready to serve.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2364)
is_cancelled
```python
is_cancelled() -> bool
```
Whether the frontend is shutting down (draining).
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2368)
has_any_ready_model
```python
has_any_ready_model() -> bool
```
Whether at least one model is registered and ready to serve.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2372)
is_model_ready_to_serve
```python
is_model_ready_to_serve(model: str) -> bool
```
Whether the named model is registered and ready to serve.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2376)
model_display_names
```python
model_display_names() -> list[str]
```
Sorted display names of all registered models.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2380)
serving_ready_display_names
```python
serving_ready_display_names() -> list[str]
```
Sorted display names of models ready to serve.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2384)
#### FrontendResponse (class)
Explicit status-code override returned by a `FrontendRoute` handler.
```python
from dynamo._core import FrontendResponse
```
```python
FrontendResponse(status_code: int, body: object) -> None
```
Return this to set a non-200 status (e.g. `FrontendResponse(503, body)`);
return a plain JSON-serializable value for the default 200.
[`lib/bindings/python/src/dynamo/_core.pyi#L2409`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2409)
**Public methods**
**init**
```python
__init__(status_code: int, body: object) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2416)
#### FrontendRoute (class)
A trusted extension route served on the Dynamo HTTP frontend.
```python
from dynamo._core import FrontendRoute
```
```python
FrontendRoute(method: str, path: str, handler: Callable[[FrontendExtensionContext], object]) -> None
```
Currently restricted to static-path `GET` routes. `handler` is a
synchronous callable that receives a `FrontendExtensionContext` and
returns a JSON-serializable body (implies HTTP 200) or a `FrontendResponse`
to set the status code. Async handlers and path parameters are rejected at
construction.
[`lib/bindings/python/src/dynamo/_core.pyi#L2388`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2388)
**Public methods**
**init**
```python
__init__(method: str, path: str, handler: Callable[[FrontendExtensionContext], object]) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2398)
#### HttpAsyncEngine (class)
An async engine for a distributed Dynamo http service. This is an extension of the python based AsyncEngine that handles HttpError exceptions from Python and converts them to the Rust version of HttpError
```python
from dynamo._core import HttpAsyncEngine
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1482`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1482)
#### HttpService (class)
A HTTP service for dynamo applications. It is a OpenAI compatible http ingress into the Dynamo Distributed Runtime.
```python
from dynamo._core import HttpService
```
```python
HttpService(port: Optional[int] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1441`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1441)
**Public methods**
**init**
```python
__init__(port: Optional[int] = None) -> None
```
Create a new HTTP service.
**Parameters**
**`port`** `Optional[int]`
Optional port number to bind the service to (default: 8080)
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1447)
run
```python
run(runtime: DistributedRuntime) -> None
```
Run the HTTP service.
**Parameters**
**`runtime`** `DistributedRuntime`
DistributedRuntime instance for token management
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1456)
shutdown
```python
shutdown() -> None
```
Shutdown the HTTP service by cancelling its internal token.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1465)
#### Instance (class)
A read-only view of a single registered instance of an endpoint, wrapping a snapshot of the runtime `Instance`. `str(instance)` yields `"namespace/component/endpoint/instance_id"`.
```python
from dynamo._core import Instance
```
[`lib/bindings/python/src/dynamo/_core.pyi#L266`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L266)
#### InvalidArgument (class)
Invalid input (e.g., prompt exceeds context length).
```python
from dynamo._core import InvalidArgument
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3228`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3228)
#### KserveGrpcService (class)
A gRPC service implementing the KServe protocol for dynamo applications. Provides model management for completions, chat completions, and tensor-based models.
```python
from dynamo._core import KserveGrpcService
```
```python
KserveGrpcService(port: Optional[int] = None, host: Optional[str] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1491`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1491)
**Public methods**
**init**
```python
__init__(port: Optional[int] = None, host: Optional[str] = None) -> None
```
Create a new KServe gRPC service.
**Parameters**
**`port`** `Optional[int]`
Optional port number to bind the service to
---
**`host`** `Optional[str]`
Optional host address to bind the service to
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1497)
add_completions_model
```python
add_completions_model(model: str, checksum: str, engine: PythonAsyncEngine) -> None
```
Register a completions model with the service.
**Parameters**
**`model`** `str`
The model name
---
**`checksum`** `str`
The model checksum
---
**`engine`** `PythonAsyncEngine`
The async engine to handle requests
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1507)
add_chat_completions_model
```python
add_chat_completions_model(model: str, checksum: str, engine: PythonAsyncEngine) -> None
```
Register a chat completions model with the service.
**Parameters**
**`model`** `str`
The model name
---
**`checksum`** `str`
The model checksum
---
**`engine`** `PythonAsyncEngine`
The async engine to handle requests
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1523)
add_tensor_model
```python
add_tensor_model(model: str, checksum: str, engine: PythonAsyncEngine, *, runtime_config: Optional[ModelRuntimeConfig] = None, tensor_model_config: Optional[Dict[str, Any]] = None) -> None
```
Register a tensor-based model with the service.
**Parameters**
**`model`** `str`
The model name
---
**`checksum`** `str`
The model checksum
---
**`engine`** `PythonAsyncEngine`
The async engine to handle requests
---
**`runtime_config`** `Optional[ModelRuntimeConfig]`
Optional runtime-resolved worker metadata
---
**`tensor_model_config`** `Optional[Dict[str, Any]]`
Optional tensor protocol model metadata
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1539)
remove_completions_model
```python
remove_completions_model(model: str) -> None
```
Remove a completions model from the service.
**Parameters**
**`model`** `str`
The model name to remove
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1560)
remove_chat_completions_model
```python
remove_chat_completions_model(model: str) -> None
```
Remove a chat completions model from the service.
**Parameters**
**`model`** `str`
The model name to remove
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1569)
remove_tensor_model
```python
remove_tensor_model(model: str) -> None
```
Remove a tensor model from the service.
**Parameters**
**`model`** `str`
The model name to remove
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1578)
list_chat_completions_models
```python
list_chat_completions_models() -> List[str]
```
List all registered chat completions models.
**Returns**
* `List[str]` — List of model names
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1587)
list_completions_models
```python
list_completions_models() -> List[str]
```
List all registered completions models.
**Returns**
* `List[str]` — List of model names
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1596)
list_tensor_models
```python
list_tensor_models() -> List[str]
```
List all registered tensor models.
**Returns**
* `List[str]` — List of model names
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1605)
run
```python
run(runtime: DistributedRuntime) -> None
```
Run the KServe gRPC service.
**Parameters**
**`runtime`** `DistributedRuntime`
DistributedRuntime instance for token management
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1614)
shutdown
```python
shutdown() -> None
```
Shutdown the KServe gRPC service by cancelling its internal token.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1623)
#### KvDcRelay (class)
No summary available.
```python
from dynamo._core import KvDcRelay
```
```python
KvDcRelay(endpoint: Endpoint, dc_id: str, namespace_filter: Optional[str] = None, endpoint_prefix: Optional[str] = None, publication_threshold: int = 16, publication_delay_ms: int = 1, recovery_attempt_timeout_ms: int = 30000) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2801`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2801)
**Public methods**
**init**
```python
__init__(endpoint: Endpoint, dc_id: str, namespace_filter: Optional[str] = None, endpoint_prefix: Optional[str] = None, publication_threshold: int = 16, publication_delay_ms: int = 1, recovery_attempt_timeout_ms: int = 30000) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2802)
start
```python
start() -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2814)
health
```python
health() -> Dict[str, Any]
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2817)
flush
```python
flush() -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2820)
shutdown
```python
shutdown() -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2823)
#### KvEventPublisher (class)
A KV event publisher will publish KV events corresponding to the component.
```python
from dynamo._core import KvEventPublisher
```
```python
KvEventPublisher(endpoint: Endpoint, worker_id: Optional[int] = None, kv_block_size: int = 0, dp_rank: int = 0, enable_local_indexer: bool = False, zmq_endpoint: Optional[str] = None, zmq_topic: Optional[str] = None, batching_timeout_ms: Optional[int] = None, image_token_id: Optional[int] = None, kv_state_endpoint: Optional[str] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1156`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1156)
**Public methods**
**init**
```python
__init__(endpoint: Endpoint, worker_id: Optional[int] = None, kv_block_size: int = 0, dp_rank: int = 0, enable_local_indexer: bool = False, zmq_endpoint: Optional[str] = None, zmq_topic: Optional[str] = None, batching_timeout_ms: Optional[int] = None, image_token_id: Optional[int] = None, kv_state_endpoint: Optional[str] = None) -> None
```
Create a `KvEventPublisher` object.
When zmq\_endpoint is provided, the publisher subscribes to a ZMQ socket for
incoming engine events (e.g. from SGLang/vLLM) and relays them to NATS.
When zmq\_endpoint is None, events are pushed manually via publish\_batch,
publish\_stored, or publish\_removed.
**Parameters**
**`endpoint`** `Endpoint`
The endpoint to extract component information from for event publishing
---
**`worker_id`** `Optional[int]`
Optional worker ID override. Use None to infer from endpoint.
---
**`kv_block_size`** `int`
The KV block size (must be > 0)
---
**`dp_rank`** `int`
The data parallel rank (defaults to 0)
---
**`enable_local_indexer`** `bool`
Enable worker-local KV indexer
---
**`zmq_endpoint`** `Optional[str]`
Optional ZMQ endpoint for relay mode (e.g. "tcp\://127.0.0.1:5557")
---
**`zmq_topic`** `Optional[str]`
ZMQ topic to subscribe to (defaults to "" when zmq\_endpoint is set)
---
**`batching_timeout_ms`** `Optional[int]`
Cross-list batching timeout in milliseconds. None/0
flushes at each submitted source-list boundary.
---
**`kv_state_endpoint`** `Optional[str]`
KV event ownership endpoint; defaults to endpoint.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1163)
publish_stored
```python
publish_stored(token_ids: List[int], num_block_tokens: List[int], block_hashes: List[int], parent_hash: Optional[int] = None, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, is_eagle: Optional[bool] = None, cache_salt: Optional[str] = None) -> None
```
Publish a KV stored event.
Event IDs are managed internally by the publisher using a monotonic counter.
**Parameters**
**`token_ids`** `List[int]`
List of token IDs
---
**`num_block_tokens`** `List[int]`
Number of tokens per block
---
**`block_hashes`** `List[int]`
List of block hashes (signed 64-bit integers)
---
**`parent_hash`** `Optional[int]`
Optional parent hash (signed 64-bit integer)
---
**`block_mm_infos`** `Optional[List[Optional[Dict[str, Any]]]]`
Optional list of multimodal info for each block.
Each item is either None or a dict with "mm\_objects" key containing
a list of \{"mm\_hash": int, "offsets": \[\[start, end], ...]} dicts.
---
**`lora_name`** `Optional[str]`
Optional LoRA adapter name for adapter-aware block hashing.
---
**`is_eagle`** `Optional[bool]`
Optional Eagle mode flag. When true, stored blocks are
reconstructed using overlapping `kv_block_size + 1` token windows.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1198)
publish_removed
```python
publish_removed(block_hashes: List[int]) -> None
```
Publish a KV removed event.
Event IDs are managed internally by the publisher using a monotonic counter.
**Parameters**
**`block_hashes`** `List[int]`
List of block hashes to remove (signed 64-bit integers)
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1228)
publish_batch
```python
publish_batch(events: Sequence[KvStoredEventInput | KvRemovedEventInput]) -> None
```
Publish an ordered list of KV events as one processor input.
The complete list is validated before it is enqueued. Compatible
events are coalesced while preserving source order and the processor's
existing block-count limits.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1239)
shutdown
```python
shutdown() -> None
```
Shuts down the event publisher, stopping any background tasks.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1251)
#### KvIndexer (class)
A KV Indexer that tracks KV Events emitted by workers. Events include add\_block and remove\_block.
```python
from dynamo._core import KvIndexer
```
```python
KvIndexer(endpoint: Endpoint, block_size: int) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1030`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1030)
**Public methods**
**init**
```python
__init__(endpoint: Endpoint, block_size: int) -> None
```
Create a `KvIndexer` object
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1037)
find_matches
```python
find_matches(sequence: List[int]) -> OverlapScores
```
Find prefix matches for the given sequence of block hashes.
**Parameters**
**`sequence`** `List[int]`
List of block hashes to find matches for
---
**Returns**
* `OverlapScores` — OverlapScores containing worker matching scores and frequencies
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1042)
find_matches_for_request
```python
find_matches_for_request(token_ids: List[int], lora_name: Optional[str] = None, is_eagle: Optional[bool] = None) -> OverlapScores
```
Return the overlapping scores of workers for the given token ids.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1054)
block_size
```python
block_size() -> int
```
Return the block size of the KV Indexer.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1062)
#### KvRemovedEventInput (class)
No summary available.
```python
from dynamo._core import KvRemovedEventInput
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1151`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1151)
#### KvRouter (class)
A KV-aware router that performs intelligent routing based on KV cache overlap.
```python
from dynamo._core import KvRouter
```
```python
KvRouter(endpoint: Endpoint, block_size: int, kv_router_config: KvRouterConfig, aic_perf_config: Optional[AicPerfConfig] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2826`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2826)
**Public methods**
**init**
```python
__init__(endpoint: Endpoint, block_size: int, kv_router_config: KvRouterConfig, aic_perf_config: Optional[AicPerfConfig] = None) -> None
```
Create a new KvRouter instance.
**Parameters**
**`endpoint`** `Endpoint`
The endpoint to connect to for routing requests
---
**`block_size`** `int`
The KV cache block size
---
**`kv_router_config`** `KvRouterConfig`
Configuration for the KV router
---
**`aic_perf_config`** `Optional[AicPerfConfig]`
Optional AIC perf-model config for effective prefill load tracking
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2831)
generate
```python
generate(token_ids: List[int], model: str, stop_conditions: Optional[JsonLike] = None, sampling_options: Optional[JsonLike] = None, output_options: Optional[JsonLike] = None, router_config_override: Optional[JsonLike] = None, worker_id: Optional[int] = None, dp_rank: Optional[int] = None, extra_args: Optional[JsonLike] = None, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, multi_modal_data: Optional[JsonLike] = None, mm_routing_info: Optional[JsonLike] = None, routing_constraints: Optional[RoutingConstraints] = None, response_buffer_size: int = 100) -> AsyncIterator[JsonLike]
```
Generate text using the KV-aware router.
**Parameters**
**`token_ids`** `List[int]`
Input token IDs
---
**`model`** `str`
Model name to use for generation
---
**`stop_conditions`** `Optional[JsonLike]`
Optional stop conditions for generation
---
**`sampling_options`** `Optional[JsonLike]`
Optional sampling configuration
---
**`output_options`** `Optional[JsonLike]`
Optional output configuration
---
**`router_config_override`** `Optional[JsonLike]`
Optional router configuration override
---
**`worker_id`** `Optional[int]`
Optional worker ID to route to directly. If set, the request
will be sent to this specific worker and router states will be
updated accordingly.
---
**`dp_rank`** `Optional[int]`
Optional data parallel rank to route to. If set along with worker\_id,
the request will be routed to the specific (worker\_id, dp\_rank) pair.
If only dp\_rank is set, the router will select the best worker but
force routing to the specified dp\_rank.
---
**`extra_args`** `Optional[JsonLike]`
Optional extra request arguments to include in the
PreprocessedRequest.
---
**`block_mm_infos`** `Optional[List[Optional[Dict[str, Any]]]]`
Optional block-level multimodal metadata aligned to
request blocks. Backward-compatible shortcut; this is
converted to mm\_routing\_info with routing\_token\_ids=token\_ids.
---
**`multi_modal_data`** `Optional[JsonLike]`
Optional multimodal payload map to preserve image/video
data for downstream model execution.
---
**`mm_routing_info`** `Optional[JsonLike]`
Optional structured routing-only multimodal payload
(e.g., \{"routing\_token\_ids": \[...], "block\_mm\_infos": \[...]})
used by router selection without changing execution token\_ids.
---
**`routing_constraints`** `Optional[RoutingConstraints]`
Optional request routing constraints used to constrain or prefer tainted workers.
---
**`response_buffer_size`** `int`
Maximum number of responses buffered by the Python
adapter. Set to 0 for demand-driven direct Python
consumption; negative values are rejected.
---
**Returns**
* `AsyncIterator[JsonLike]` — An async iterator yielding generation responses
- If worker\_id is set, the request bypasses KV matching and routes directly
to the specified worker while still updating router states.
- dp\_rank allows targeting a specific data parallel replica when workers have
multiple replicas (data\_parallel\_size > 1).
- This is different from query\_instance\_id which doesn't route the request.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2849)
generate_from_request
```python
generate_from_request(request: JsonLike, response_buffer_size: int = 100) -> AsyncIterator[JsonLike]
```
Generate from a preprocessed request dict (PreprocessedRequest format).
Accepts a full request dict with token\_ids, model, stop\_conditions, etc.
Set response\_buffer\_size to 0 for demand-driven direct Python consumption;
negative values are rejected.
Returns an async iterator yielding generation responses.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2910)
best_worker
```python
best_worker(token_ids: List[int], router_config_override: Optional[JsonLike] = None, request_id: Optional[str] = None, update_indexer: bool = False, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, routing_constraints: Optional[RoutingConstraints] = None, strict_priority: int = 0, policy_class: Optional[str] = None, cache_namespace: Optional[str] = None) -> Tuple[int, int, int]
```
Find the best matching worker for the given tokens.
**Parameters**
**`token_ids`** `List[int]`
List of token IDs to find matches for
---
**`router_config_override`** `Optional[JsonLike]`
Optional router configuration override
---
**`request_id`** `Optional[str]`
Optional request ID. If provided, router states will be updated
to track this request (active blocks, lifecycle events). If not
provided, this is a query-only operation that doesn't affect state.
---
**`update_indexer`** `bool`
Whether to record the selected worker in the router's
approximate indexer. This is only meaningful when
`use_kv_events=False` and is independent from lifecycle
state tracking via `request_id`.
---
**`block_mm_infos`** `Optional[List[Optional[Dict[str, Any]]]]`
Optional block-level multimodal metadata aligned to request
blocks. When provided, this is used in block hash computation
to enable MM-aware worker selection.
---
**`cache_namespace`** `Optional[str]`
Optional cache namespace used in block hash computation.
---
**`policy_class`** `Optional[str]`
Requested policy family, or an exact explicit class.
Missing, unknown, and ordinary physical-class names use the
configured default family before cache-bucket resolution.
---
**Returns**
* `Tuple[int, int, int]` — A tuple of (worker\_id, dp\_rank, overlap\_blocks) where: - worker\_id: The ID of the best matching worker - dp\_rank: The data parallel rank of the selected worker - overlap\_blocks: The number of overlapping blocks found
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2925)
get_potential_loads
```python
get_potential_loads(token_ids: List[int], block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, cache_namespace: Optional[str] = None) -> List[Dict[str, int]]
```
Get potential prefill and decode loads for all workers.
**Parameters**
**`token_ids`** `List[int]`
List of token IDs to evaluate
---
**`block_mm_infos`** `Optional[List[Optional[Dict[str, Any]]]]`
Optional block-level multimodal metadata aligned to request
blocks. When provided, this is used in hash computation
for MM-aware potential-load estimation.
---
**`lora_name`** `Optional[str]`
Optional LoRA adapter name used in block hash computation.
---
**Returns**
* `List[Dict[str, int]]` — A list of dictionaries, each containing: - worker\_id: The worker ID - dp\_rank: The data parallel rank - potential\_prefill\_tokens: Number of tokens that would need prefill - potential\_decode\_blocks: Number of blocks currently in decode phase - active\_requests: Number of active requests tracked on the worker
Each (worker\_id, dp\_rank) pair is returned as a separate entry.
If you need aggregated loads per worker\_id, sum the values manually.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2967)
get_overlap_scores
```python
get_overlap_scores(token_ids: List[int], router_config_override: Optional[JsonLike] = None, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, include_shared: bool = True, cache_namespace: Optional[str] = None) -> Dict[str, Any]
```
Get per-worker KV overlap by storage tier.
**Parameters**
**`token_ids`** `List[int]`
List of token IDs to evaluate.
---
**`router_config_override`** `Optional[JsonLike]`
Optional router configuration override for
score-credit fields.
---
**`block_mm_infos`** `Optional[List[Optional[Dict[str, Any]]]]`
Optional block-level multimodal metadata aligned to
request blocks.
---
**`lora_name`** `Optional[str]`
Optional LoRA adapter name for adapter-aware matching.
---
**`include_shared`** `bool`
Whether to query the configured shared cache.
---
**Returns**
* `Dict[str, Any]` — A dictionary containing block\_size, num\_blocks, shared\_cache, and
* `Dict[str, Any]` — workers. Each worker row is keyed by worker\_id and dp\_rank and
* `Dict[str, Any]` — reports device, host-pinned, disk, and shared-cache overlap blocks.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2998)
dump_events
```python
dump_events() -> str
```
Dump all events from the KV router's indexer.
**Returns**
* `str` — A JSON string containing all indexer events
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3026)
mark_prefill_complete
```python
mark_prefill_complete(request_id: str) -> None
```
Mark prefill as completed for a request.
This signals that the request has finished its prefill phase and is now
in the decode phase. Used to update router state for accurate load tracking.
**Parameters**
**`request_id`** `str`
The ID of the request that completed prefill
---
This is typically called automatically by the router when using the
`generate()` method. Only call this manually if you're using
`best_worker()` with `request_id` for custom routing.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3035)
free
```python
free(request_id: str) -> None
```
Free a request by its ID, signaling the router to release resources.
This should be called when a request completes to update the router's
tracking of active blocks and ensure accurate load balancing.
**Parameters**
**`request_id`** `str`
The ID of the request to free
---
This is typically called automatically by the router when using the
`generate()` method. Only call this manually if you're using
`best_worker()` with `request_id` for custom routing.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3052)
#### KvRouterConfig (class)
Values for KV router
```python
from dynamo._core import KvRouterConfig
```
```python
KvRouterConfig(overlap_score_weight: Optional[float] = None, host_cache_hit_weight: float = 0.75, disk_cache_hit_weight: float = 0.25, router_temperature: float = 0.0, use_kv_events: bool = True, *, router_replica_sync: bool = False, router_track_active_blocks: bool = True, router_track_output_blocks: bool = False, router_assume_kv_reuse: bool = True, router_track_prefill_tokens: bool = True, router_prefill_load_model: str = 'none', router_ttl_secs: float = 120.0, router_queue_threshold: Optional[float] = None, router_event_threads: int = 4, router_queue_policy: str = 'fcfs', use_remote_indexer: bool = False, serve_indexer: bool = False, shared_cache_multiplier: float = 0.0, shared_cache_type: str = 'none', router_predicted_ttl_secs: Optional[float] = None, overlap_score_credit: float = 1.0, overlap_score_credit_decay: float = 0.0, prefill_load_scale: float = 1.0, decode_active_request_weight: float = 0.0, router_policy_config: Optional[str] = None, router_tracking_hash: Literal['public-xxh3-v1', 'keyed-xxh3-v1'] = 'public-xxh3-v1', router_tracking_key_file: Optional[str | os.PathLike[str]] = None, router_tracking_key_id: Optional[str] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1743`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1743)
**Public methods**
**init**
```python
__init__(overlap_score_weight: Optional[float] = None, host_cache_hit_weight: float = 0.75, disk_cache_hit_weight: float = 0.25, router_temperature: float = 0.0, use_kv_events: bool = True, *, router_replica_sync: bool = False, router_track_active_blocks: bool = True, router_track_output_blocks: bool = False, router_assume_kv_reuse: bool = True, router_track_prefill_tokens: bool = True, router_prefill_load_model: str = 'none', router_ttl_secs: float = 120.0, router_queue_threshold: Optional[float] = None, router_event_threads: int = 4, router_queue_policy: str = 'fcfs', use_remote_indexer: bool = False, serve_indexer: bool = False, shared_cache_multiplier: float = 0.0, shared_cache_type: str = 'none', router_predicted_ttl_secs: Optional[float] = None, overlap_score_credit: float = 1.0, overlap_score_credit_decay: float = 0.0, prefill_load_scale: float = 1.0, decode_active_request_weight: float = 0.0, router_policy_config: Optional[str] = None, router_tracking_hash: Literal['public-xxh3-v1', 'keyed-xxh3-v1'] = 'public-xxh3-v1', router_tracking_key_file: Optional[str | os.PathLike[str]] = None, router_tracking_key_id: Optional[str] = None) -> None
```
Create a KV router configuration.
**Parameters**
**`overlap_score_weight`** `Optional[float]`
Deprecated positional/keyword alias for prefill\_load\_scale. When present, it takes precedence over prefill\_load\_scale; a value of 0 also sets overlap\_score\_credit to 0.
---
**`overlap_score_credit`** `float`
Finite, non-negative credit multiplier for device-local prefix overlap (default: 1.0). Values above 1.0 give device overlap extra credit, with adjusted prefill cost clamped at zero.
---
**`prefill_load_scale`** `float`
Scale for adjusted prompt-side prefill load after cache-hit credits (default: 1.0)
---
**`decode_active_request_weight`** `float`
Experimental block-equivalent decode cost added for each active request on a candidate worker (default: 0.0)
---
**`host_cache_hit_weight`** `float`
Credit multiplier for host-pinned cache hits (default: 0.75)
---
**`disk_cache_hit_weight`** `float`
Credit multiplier for disk/external cache hits (default: 0.25)
---
**`router_temperature`** `float`
Temperature for normalized worker sampling via softmax (default: 0.0)
---
**`use_kv_events`** `bool`
Whether to use KV events from workers (default: True)
---
**`router_replica_sync`** `bool`
Enable replica synchronization (default: False)
---
**`router_track_active_blocks`** `bool`
Track active blocks for load balancing (default: True)
---
**`router_track_output_blocks`** `bool`
Track output blocks during generation (default: False).
When enabled, the router adds placeholder blocks as tokens are generated
and, with expected output sequence length (agent\_hints.osl in nvext),
applies fractional decay to output blocks and the structurally exclusive
prompt suffix. Shared prompt blocks retain full weight.
---
**`router_assume_kv_reuse`** `bool`
Assume KV cache reuse when tracking active blocks (default: True).
When True, computes actual block hashes. When False, generates random hashes.
---
**`router_track_prefill_tokens`** `bool`
Include prompt-side prefill tokens in active load accounting (default: True).
---
**`router_tracking_hash`** `Literal['public-xxh3-v1', 'keyed-xxh3-v1']`
Tracking identity algorithm, "public-xxh3-v1" or
"keyed-xxh3-v1" (default: "public-xxh3-v1").
---
**`router_tracking_key_file`** `Optional[str | os.PathLike[str]]`
File containing exactly 32 raw provider-key bytes.
Required only for keyed tracking mode.
---
**`router_tracking_key_id`** `Optional[str]`
Provider-managed key epoch mixed into keyed scope
derivation. Required only for keyed tracking mode.
---
**`router_prefill_load_model`** `str`
Prompt-side prefill load model (default: "none").
"none" keeps static prompt load accounting.
"aic" decays the oldest active prefill request using AIC-predicted duration.
---
**`router_ttl_secs`** `float`
TTL for blocks in seconds when not using KV events (default: 120.0)
---
**`router_queue_threshold`** `Optional[float]`
Optional queue threshold fraction for prefill token capacity (default: None).
Requests are queued if all workers exceed this fraction of max\_num\_batched\_tokens.
Enables priority scheduling via request priority hints.
Set a numeric value to enable queueing.
---
**`router_policy_config`** `Optional[str]`
Startup-only policy-family and cache-bucket queue
YAML path. When omitted, router\_queue\_threshold and
router\_queue\_policy define one synthetic policy class.
---
**`router_event_threads`** `int`
Number of KV indexer worker threads (default: 4).
When > 1, uses a concurrent radix tree with a thread pool,
including for approximate routing when KV events are disabled.
---
**`router_queue_policy`** `str`
Scheduling policy for the router queue (default: "fcfs").
"fcfs": first-come first-served with priority bumps — optimizes tail TTFT.
"lcfs": last-come first-served with priority bumps — intentionally worsens tail behavior for policy comparisons.
"wspt": weighted shortest processing time (Smith's rule) — optimizes average TTFT.
---
**`use_remote_indexer`** `bool`
Query a remote KV indexer served from the worker component (default: False).
---
**`serve_indexer`** `bool`
Serve this router's local indexer from the worker component (default: False).
---
**`shared_cache_multiplier`** `float`
Credit multiplier for shared cache hits beyond the device prefix (default: 0.0).
---
**`shared_cache_type`** `str`
External shared KV cache type, "none" or "hicache" (default: "none").
---
**`router_predicted_ttl_secs`** `Optional[float]`
Enables predict-on-route when set. This TTL
applies to entries in the local side indexer and requires
use\_kv\_events=True. Set to None to disable. Independent of
router\_ttl\_secs, which covers pure approximate mode.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1746)
from_json
```python
from_json(config_json: str) -> KvRouterConfig
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1835)
copy
```python
copy() -> KvRouterConfig
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1839)
with_overrides
```python
with_overrides(overlap_score_weight: Optional[float] = None, *, overlap_score_credit: Optional[float] = None, overlap_score_credit_decay: Optional[float] = None, prefill_load_scale: Optional[float] = None, decode_active_request_weight: Optional[float] = None) -> KvRouterConfig
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1865)
#### KvStoredEventInput (class)
No summary available.
```python
from dynamo._core import KvStoredEventInput
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1139`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1139)
#### KvbmRequest (class)
A request for KV cache
```python
from dynamo._core import KvbmRequest
```
```python
KvbmRequest(request_id: int, tokens: List[int], block_size: int) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2793`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2793)
**Public methods**
**init**
```python
__init__(request_id: int, tokens: List[int], block_size: int) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2798)
#### Layer (class)
A KV cache block layer
```python
from dynamo._core import Layer
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2583`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2583)
#### LoRADownloader (class)
Unified interface for LoRA downloading and caching (local file:// and S3 s3:// URIs).
```python
from dynamo._core import LoRADownloader
```
```python
LoRADownloader(cache_path: Optional[str] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2305`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2305)
**Public methods**
**init**
```python
__init__(cache_path: Optional[str] = None) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2308)
download_if_needed
```python
download_if_needed(lora_uri: str) -> Awaitable[str]
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2309)
get_cache_path
```python
get_cache_path(cache_key: str) -> str
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2310)
is_cached
```python
is_cached(lora_uri: str) -> bool
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2311)
validate_cached
```python
validate_cached(cache_key: str) -> bool
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2312)
uri_to_cache_key
```python
uri_to_cache_key(uri: str) -> str
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2314)
#### MediaDecoder (class)
Media decoder for image and video preprocessing.
```python
from dynamo._core import MediaDecoder
```
```python
MediaDecoder() -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2318`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2318)
**Public methods**
```python
__init__() -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2321)
```python
enable_image(decoder_options: Dict[str, Any]) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2322)
#### MediaFetcher (class)
Media fetcher for loading remote image/video URLs.
```python
from dynamo._core import MediaFetcher
```
```python
MediaFetcher() -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2325`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2325)
**Public methods**
```python
__init__() -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2328)
```python
user_agent(user_agent: str) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2329)
```python
allow_direct_ip(allow: bool) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2330)
```python
allow_direct_port(allow: bool) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2331)
allowed_media_domains
```python
allowed_media_domains(domains: List[str]) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2332)
```python
timeout_ms(timeout_ms: int) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2333)
#### MockEngineArgs (class)
No summary available.
```python
from dynamo._core import MockEngineArgs
```
```python
MockEngineArgs(engine_type: str = 'vllm', num_gpu_blocks: Optional[int] = None, block_size: int = 0, max_num_seqs: Optional[int] = 256, max_num_batched_tokens: Optional[int] = 8192, enable_prefix_caching: bool = True, enable_chunked_prefill: bool = True, speedup_ratio: float = 1.0, decode_speedup_ratio: float = 1.0, dp_size: int = 1, startup_time: Optional[float] = None, worker_type: str = 'aggregated', planner_profile_data: Optional[str | os.PathLike[str]] = None, aic_backend: Optional[str] = None, aic_system: Optional[str] = None, aic_backend_version: Optional[str] = None, aic_tp_size: Optional[int] = None, aic_model_path: Optional[str] = None, aic_moe_tp_size: Optional[int] = None, aic_moe_ep_size: Optional[int] = None, aic_attention_dp_size: Optional[int] = None, aic_nextn: Optional[int] = None, aic_nextn_accept_rates: Optional[str] = None, aic_mtp_seed: int = 42, aic_gemm_dtype: Optional[str] = None, aic_moe_dtype: Optional[str] = None, aic_fmha_dtype: Optional[str] = None, aic_kv_cache_dtype: Optional[str] = None, aic_comm_dtype: Optional[str] = None, gpu_memory_utilization: Optional[float] = None, mem_fraction_static: Optional[float] = None, free_gpu_memory_fraction: Optional[float] = None, enable_local_indexer: bool = False, bootstrap_port: Optional[int] = None, handoff_session_timeout_ms: int = 300000, kv_bytes_per_token: Optional[int] = None, kv_transfer_bandwidth: Optional[float] = None, kv_transfer_timing_mode: str = 'full_prompt', reasoning: Optional[ReasoningConfig] = None, response_replay_trace_path: Optional[str | os.PathLike[str]] = None, zmq_kv_events_port: Optional[int] = None, zmq_replay_port: Optional[int] = None, preemption_mode: str = 'lifo', router_queue_policy: Optional[str] = None, sglang: Optional[SglangArgs] = None, trtllm: Optional[TrtllmArgs] = None, num_g2_blocks: Optional[int] = None, num_g3_blocks: Optional[int] = None, offload_batch_size: Optional[int] = None, bandwidth_g1_to_g2_gbps: Optional[float] = None, bandwidth_g2_to_g1_gbps: Optional[float] = None, bandwidth_g2_to_g3_gbps: Optional[float] = None, bandwidth_g3_to_g2_gbps: Optional[float] = None, enable_g4_storage: bool = False, bandwidth_g2_to_g4_gbps: Optional[float] = None, bandwidth_g4_to_g2_gbps: Optional[float] = None, max_model_len: Optional[int] = None, g1_backend: Optional[str] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1903`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1903)
**Public methods**
**init**
```python
__init__(engine_type: str = 'vllm', num_gpu_blocks: Optional[int] = None, block_size: int = 0, max_num_seqs: Optional[int] = 256, max_num_batched_tokens: Optional[int] = 8192, enable_prefix_caching: bool = True, enable_chunked_prefill: bool = True, speedup_ratio: float = 1.0, decode_speedup_ratio: float = 1.0, dp_size: int = 1, startup_time: Optional[float] = None, worker_type: str = 'aggregated', planner_profile_data: Optional[str | os.PathLike[str]] = None, aic_backend: Optional[str] = None, aic_system: Optional[str] = None, aic_backend_version: Optional[str] = None, aic_tp_size: Optional[int] = None, aic_model_path: Optional[str] = None, aic_moe_tp_size: Optional[int] = None, aic_moe_ep_size: Optional[int] = None, aic_attention_dp_size: Optional[int] = None, aic_nextn: Optional[int] = None, aic_nextn_accept_rates: Optional[str] = None, aic_mtp_seed: int = 42, aic_gemm_dtype: Optional[str] = None, aic_moe_dtype: Optional[str] = None, aic_fmha_dtype: Optional[str] = None, aic_kv_cache_dtype: Optional[str] = None, aic_comm_dtype: Optional[str] = None, gpu_memory_utilization: Optional[float] = None, mem_fraction_static: Optional[float] = None, free_gpu_memory_fraction: Optional[float] = None, enable_local_indexer: bool = False, bootstrap_port: Optional[int] = None, handoff_session_timeout_ms: int = 300000, kv_bytes_per_token: Optional[int] = None, kv_transfer_bandwidth: Optional[float] = None, kv_transfer_timing_mode: str = 'full_prompt', reasoning: Optional[ReasoningConfig] = None, response_replay_trace_path: Optional[str | os.PathLike[str]] = None, zmq_kv_events_port: Optional[int] = None, zmq_replay_port: Optional[int] = None, preemption_mode: str = 'lifo', router_queue_policy: Optional[str] = None, sglang: Optional[SglangArgs] = None, trtllm: Optional[TrtllmArgs] = None, num_g2_blocks: Optional[int] = None, num_g3_blocks: Optional[int] = None, offload_batch_size: Optional[int] = None, bandwidth_g1_to_g2_gbps: Optional[float] = None, bandwidth_g2_to_g1_gbps: Optional[float] = None, bandwidth_g2_to_g3_gbps: Optional[float] = None, bandwidth_g3_to_g2_gbps: Optional[float] = None, enable_g4_storage: bool = False, bandwidth_g2_to_g4_gbps: Optional[float] = None, bandwidth_g4_to_g2_gbps: Optional[float] = None, max_model_len: Optional[int] = None, g1_backend: Optional[str] = None) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1904)
from_json
```python
from_json(config_json: str) -> MockEngineArgs
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1967)
copy
```python
copy() -> MockEngineArgs
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1971)
is_prefill
```python
is_prefill() -> bool
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2171)
is_decode
```python
is_decode() -> bool
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2173)
with_overrides
```python
with_overrides(bootstrap_port: Optional[int] = None, zmq_kv_events_port: Optional[int] = None, zmq_replay_port: Optional[int] = None, kv_bytes_per_token: Optional[int] = None, num_gpu_blocks: Optional[int] = None, aic_backend: Optional[str] = None, aic_system: Optional[str] = None, aic_backend_version: Optional[str] = None, aic_tp_size: Optional[int] = None, aic_model_path: Optional[str] = None, aic_moe_tp_size: Optional[int] = None, aic_moe_ep_size: Optional[int] = None, aic_attention_dp_size: Optional[int] = None, aic_nextn: Optional[int] = None, aic_nextn_accept_rates: Optional[str] = None, aic_mtp_seed: Optional[int] = None, aic_gemm_dtype: Optional[str] = None, aic_moe_dtype: Optional[str] = None, aic_fmha_dtype: Optional[str] = None, aic_kv_cache_dtype: Optional[str] = None, aic_comm_dtype: Optional[str] = None, gpu_memory_utilization: Optional[float] = None, mem_fraction_static: Optional[float] = None, free_gpu_memory_fraction: Optional[float] = None, enable_prefix_caching: Optional[bool] = None, worker_type: Optional[str] = None) -> MockEngineArgs
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2175)
#### ModelCardInstanceId (class)
Unique identifier for a worker instance: namespace, component, endpoint and instance\_id. The instance\_id is not currently exposed in the Python bindings.
```python
from dynamo._core import ModelCardInstanceId
```
[`lib/bindings/python/src/dynamo/_core.pyi#L384`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L384)
**Public methods**
triple
```python
triple() -> Tuple[str, str, str]
```
Triple of namespace, component and endpoint this worker is serving.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L389)
#### ModelDeploymentCard (class)
A model deployment card is a collection of model information
```python
from dynamo._core import ModelDeploymentCard
```
[`lib/bindings/python/src/dynamo/_core.pyi#L820`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L820)
**Public methods**
to_json_str
```python
to_json_str() -> str
```
Serialize the model deployment card to a JSON string.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L825)
from_json_str
```python
from_json_str(json: str) -> ModelDeploymentCard
```
Deserialize a model deployment card from a JSON string.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L829)
model_type
```python
model_type() -> ModelType
```
Return the model type of this deployment card.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L834)
source_path
```python
source_path() -> str
```
Return the source path of this deployment card.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L838)
local_dir
```python
local_dir() -> str
```
Resolved metadata directory (post-`download_config`). Raises ValueError if the path contains non-UTF-8 bytes.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L842)
name
```python
name() -> str
```
Return the model name.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L847)
runtime_config
```python
runtime_config() -> Any
```
Return the runtime configuration as a dict.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L851)
#### ModelInput (class)
What type of request this model needs: Text, Tokens or Tensor
```python
from dynamo._core import ModelInput
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1629`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1629)
#### ModelRuntimeConfig (class)
A model runtime configuration is a collection of runtime information
```python
from dynamo._core import ModelRuntimeConfig
```
```python
ModelRuntimeConfig() -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L855`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L855)
**Public methods**
**init**
```python
__init__() -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L884)
set_engine_specific
```python
set_engine_specific(key: str, value: Any) -> None
```
Set an engine-specific runtime configuration value
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L886)
get_engine_specific
```python
get_engine_specific(key: str) -> Any | None
```
Get an engine-specific runtime configuration value
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L890)
set_structural_tag_mode
```python
set_structural_tag_mode(mode: str) -> None
```
Set structural tag mode ("off" or "on").
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L894)
set_structural_tag_scope
```python
set_structural_tag_scope(scope: str) -> None
```
Set structural tag scope ("auto" or "always").
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L898)
set_structural_tag_schema
```python
set_structural_tag_schema(schema: str) -> None
```
Set structural tag schema mode ("auto" or "strict").
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L902)
set_disaggregated_endpoint
```python
set_disaggregated_endpoint(bootstrap_host: str | None = None, bootstrap_port: int | None = None) -> None
```
Set the disaggregated endpoint for the model
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L906)
#### ModelType (class)
OpenAI-style surfaces supported by a model.
```python
from dynamo._core import ModelType
```
Values are Chat, Completions, Embedding, Classify, Pooling, TensorBased,
Images, Audios, Videos, Realtime, and Empty (no OpenAI surface).
[`lib/bindings/python/src/dynamo/_core.pyi#L1636`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1636)
**Public methods**
supports_chat
```python
supports_chat() -> bool
```
Return True if this model type supports chat.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1666)
supports_embedding
```python
supports_embedding() -> bool
```
Return True if this model type supports /v1/embeddings.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1670)
supports_classify
```python
supports_classify() -> bool
```
Return True if this model type supports /v1/classify.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1674)
supports_pooling
```python
supports_pooling() -> bool
```
Return True if this model type supports /v1/pooling.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1678)
#### MultimodalEmbeddingCachePublisher (class)
A publisher for multimodal encode-worker cache state.
```python
from dynamo._core import MultimodalEmbeddingCachePublisher
```
```python
MultimodalEmbeddingCachePublisher() -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L684`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L684)
**Public methods**
**init**
```python
__init__() -> None
```
Create a `MultimodalEmbeddingCachePublisher` object.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L691)
create_endpoint
```python
create_endpoint(endpoint: Endpoint) -> None
```
Initialize event-plane publishing for multimodal cache state.
**Parameters**
**`endpoint`** `Endpoint`
The endpoint to extract component information from.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L696)
publish_delta
```python
publish_delta(added_keys: list[str], removed_keys: list[str]) -> None
```
Publish an incremental cache mutation for this worker.
**Parameters**
**`added_keys`** `list[str]`
Newly cached embedding keys.
---
**`removed_keys`** `list[str]`
Cache keys no longer present on the worker.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L704)
#### OverlapScores (class)
A collection of prefix matching scores of workers for a given token ids. 'scores' is a map of worker id to the score which is the number of matching blocks.
```python
from dynamo._core import OverlapScores
```
[`lib/bindings/python/src/dynamo/_core.pyi#L934`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L934)
#### PlannerDecision (class)
A request from planner to client to perform a scaling action. Fields: num\_prefill\_workers, num\_decode\_workers, decision\_id. -1 in any of those fields mean not set, usually because planner hasn't decided anything yet. Call VirtualConnectorClient.complete(event) when action is completed.
```python
from dynamo._core import PlannerDecision
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3151`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3151)
#### PyAsyncRequestStream (class)
Python-visible inbound iterator handed to bidirectional engine handlers as the first positional argument. Yields request frames as JSON-like Python objects.
```python
from dynamo._core import PyAsyncRequestStream
```
Request-stream end is not a cancellation signal: when this iterator
raises `StopAsyncIteration`, the caller has merely stopped sending
input. The engine should keep yielding response chunks until it
chooses to return or observes `context.is_stopped()`.
[`lib/bindings/python/src/dynamo/_core.pyi#L238`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L238)
#### PyRuntimeMetrics (class)
Helper class for registering Prometheus metrics callbacks on an Endpoint.
```python
from dynamo._core import PyRuntimeMetrics
```
Provides utilities for integrating external metrics (e.g., from vLLM, SGLang, TensorRT-LLM).
[`lib/bindings/python/src/dynamo/prometheus_metrics.pyi#L12`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/prometheus_metrics.pyi#L12)
**Public methods**
register_prometheus_expfmt_callback
```python
register_prometheus_expfmt_callback(callback: Callable[[], str]) -> None
```
Register a Python callback that returns Prometheus exposition text. The returned text will be appended to the /metrics endpoint output.
This allows you to integrate external Prometheus metrics (e.g. from vLLM)
directly into the endpoint's metrics output.
**Parameters**
**`callback`** `Callable[[], str]`
A callable that takes no arguments and returns a string
in Prometheus text exposition format
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/prometheus_metrics.pyi#L19)
#### PythonAsyncEngine (class)
Bridge a Python async generator onto Dynamo's AsyncEngine interface.
```python
from dynamo._core import PythonAsyncEngine
```
```python
PythonAsyncEngine(generator: Any, event_loop: Any) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1471`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1471)
**Public methods**
**init**
```python
__init__(generator: Any, event_loop: Any) -> None
```
Wrap a Python generator and event loop for use with Dynamo services.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1476)
#### RadixTree (class)
A RadixTree that tracks KV cache blocks and can find prefix matches for sequences.
```python
from dynamo._core import RadixTree
```
```python
RadixTree() -> None
```
Thread-safe: operations route to a dedicated background thread and long calls
release the Python GIL.
[`lib/bindings/python/src/dynamo/_core.pyi#L961`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L961)
**Public methods**
**init**
```python
__init__() -> None
```
Create a new RadixTree instance.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L969)
find_matches
```python
find_matches(sequence: List[int], early_exit: bool = False) -> OverlapScores
```
Find prefix matches for the given sequence of block hashes.
**Parameters**
**`sequence`** `List[int]`
List of block hashes to find matches for
---
**`early_exit`** `bool`
If True, stop searching after finding the first match
---
**Returns**
* `OverlapScores` — OverlapScores containing worker matching scores and frequencies
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L975)
apply_event
```python
apply_event(worker_id: int, kv_cache_event_bytes: bytes) -> None
```
Apply a KV cache event to update the RadixTree state.
**Parameters**
**`worker_id`** `int`
ID of the worker that generated the event
---
**`kv_cache_event_bytes`** `bytes`
Serialized KV cache event as bytes
---
**Raises**
* `ValueError` — If the event bytes cannot be deserialized
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L990)
remove_worker
```python
remove_worker(worker_id: int) -> None
```
Remove all blocks associated with a specific worker.
**Parameters**
**`worker_id`** `int`
ID of the worker to remove
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1003)
clear_all_blocks
```python
clear_all_blocks(worker_id: int) -> None
```
Clear all blocks for a specific worker.
**Parameters**
**`worker_id`** `int`
ID of the worker whose blocks should be cleared
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1012)
dump_tree_as_events
```python
dump_tree_as_events() -> List[str]
```
Dump the current RadixTree state as a list of JSON-serialized KV cache events.
**Returns**
* `List[str]` — List of JSON-serialized KV cache events as strings
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1021)
#### ReasoningConfig (class)
No summary available.
```python
from dynamo._core import ReasoningConfig
```
```python
ReasoningConfig(start_thinking_token_id: int, end_thinking_token_id: int, thinking_ratio: float) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1875`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1875)
**Public methods**
**init**
```python
__init__(start_thinking_token_id: int, end_thinking_token_id: int, thinking_ratio: float) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1876)
#### RouterConfig (class)
How to route the request
```python
from dynamo._core import RouterConfig
```
```python
RouterConfig(mode: RouterMode, config: Optional[KvRouterConfig] = None, active_decode_blocks_threshold: Optional[float] = None, active_prefill_tokens_threshold: Optional[int] = None, active_prefill_tokens_threshold_frac: Optional[float] = None, enforce_disagg: bool = False, session_affinity_ttl_secs: Optional[int] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1693`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1693)
**Public methods**
**init**
```python
__init__(mode: RouterMode, config: Optional[KvRouterConfig] = None, active_decode_blocks_threshold: Optional[float] = None, active_prefill_tokens_threshold: Optional[int] = None, active_prefill_tokens_threshold_frac: Optional[float] = None, enforce_disagg: bool = False, session_affinity_ttl_secs: Optional[int] = None) -> None
```
Create a RouterConfig.
**Parameters**
**`mode`** `RouterMode`
The router mode (RoundRobin, Random, KV, Direct, LeastLoaded, or DeviceAwareWeighted)
---
**`config`** `Optional[KvRouterConfig]`
Optional KV router configuration (used when mode is KV)
---
**`active_decode_blocks_threshold`** `Optional[float]`
Threshold percentage (0.0-1.0) for decode blocks busy detection
---
**`active_prefill_tokens_threshold`** `Optional[int]`
Literal token count threshold for prefill busy detection
---
**`active_prefill_tokens_threshold_frac`** `Optional[float]`
Fraction of max\_num\_batched\_tokens for busy detection
---
**`enforce_disagg`** `bool`
Deprecated and ignored. Routing topology and readiness come from registered worker types.
---
**`session_affinity_ttl_secs`** `Optional[int]`
Router-local session-affinity idle TTL in seconds.
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1698)
#### RouterMode (class)
Router mode for load balancing requests across workers
```python
from dynamo._core import RouterMode
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1682`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1682)
#### RouterQueueLimitExceeded (class)
A policy-class queue cap rejected the request.
```python
from dynamo._core import RouterQueueLimitExceeded
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3215`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3215)
#### RoutingConstraints (class)
Request-side routing constraints.
```python
from dynamo._core import RoutingConstraints
```
```python
RoutingConstraints(required_taints: Optional[Set[str]] = None, preferred_taints: Optional[Dict[str, float]] = None) -> None
```
`required_taints` is a hard eligibility filter.
`preferred_taints` maps taint -> signed weight.
Positive weights prefer matching workers, negative weights avoid them,
and `0.0` is neutral. Matching weights are summed and squashed with
`tanh`, so opposite preferences cancel before Dynamo converts the
bounded bias into a strictly positive score multiplier.
[`lib/bindings/python/src/dynamo/_core.pyi#L914`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L914)
**Public methods**
**init**
```python
__init__(required_taints: Optional[Set[str]] = None, preferred_taints: Optional[Dict[str, float]] = None) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L928)
#### SelectionCacheConfig (class)
Bounds for the in-flight selection cache. Each field defaults to the service default when omitted.
```python
from dynamo._core import SelectionCacheConfig
```
```python
SelectionCacheConfig(*, ttl_secs: Optional[float] = None, max_entries: Optional[int] = None, max_bytes: Optional[int] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L714`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L714)
**Public methods**
**init**
```python
__init__(*, ttl_secs: Optional[float] = None, max_entries: Optional[int] = None, max_bytes: Optional[int] = None) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L720)
#### SelectionService (class)
In-process handle to a runtime-free Dynamo selection core.
```python
from dynamo._core import SelectionService
```
```python
SelectionService(*, indexer_threads: int = 4, indexer_peers: Optional[list[str]] = None, replica_sync_port: Optional[int] = None, replica_sync_peers: Optional[list[str]] = None, selection_cache: Optional[SelectionCacheConfig] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L728`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L728)
**Public methods**
**init**
```python
__init__(*, indexer_threads: int = 4, indexer_peers: Optional[list[str]] = None, replica_sync_port: Optional[int] = None, replica_sync_peers: Optional[list[str]] = None, selection_cache: Optional[SelectionCacheConfig] = None) -> None
```
Create a selection service. `indexer_threads` sizes the KV indexer pool.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L733)
shutdown
```python
shutdown() -> None
```
Stop the service: cancel KV-event listeners and scheduling so that in-flight and queued selections fail fast.
The KV indexer thread pool is released when the handle is dropped.
Idempotent, and also runs automatically on drop.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L745)
upsert_worker
```python
upsert_worker(worker: JsonLike) -> JsonLike
```
Upsert a worker and subscribe to its live KV events; returns its catalog record.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L755)
delete_worker
```python
delete_worker(worker_id: int) -> JsonLike
```
Remove a worker and tear down its KV-event listener; returns its catalog record.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L759)
list_workers
```python
list_workers(*, model_name: Optional[str] = None, routing_group: Optional[str] = None) -> JsonLike
```
List catalog records, optionally filtered by model and routing group.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L763)
ready
```python
ready() -> JsonLike
```
Readiness: whether at least one worker is schedulable, plus catalog state.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L769)
overlap_scores
```python
overlap_scores(request: JsonLike) -> JsonLike
```
Per-worker KV-overlap scores for a prompt.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L773)
select
```python
select(request: JsonLike) -> JsonLike
```
Select the best worker by KV-overlap + load, without booking.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L777)
select_and_reserve
```python
select_and_reserve(request: JsonLike) -> JsonLike
```
Select the best worker and book its load.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L781)
create_reservation
```python
create_reservation(request: JsonLike) -> JsonLike
```
Book a request's load against a worker, keyed by `selection_id`.
Without a `worker_id`, replays the matching `select`'s cached
selection (same model/routing-group), booked under `selection_id`;
other request fields are ignored. With a `worker_id` and the prompt,
books explicitly under `selection_id` on that worker and discards any
cached selection for the id. `selection_id` is required.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L785)
prefill_complete
```python
prefill_complete(selection_id: str) -> None
```
Mark a reservation's prefill complete; its load shifts prefill -> decode.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L796)
add_output_block
```python
add_output_block(selection_id: str, *, decay_fraction: Optional[float] = None) -> None
```
Record one decode output block for a reservation, advancing its decode load.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L800)
free_reservation
```python
free_reservation(selection_id: str) -> None
```
Free a finished reservation, releasing its tracked load.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L806)
loads
```python
loads(*, model_name: Optional[str] = None, routing_group: Optional[str] = None) -> JsonLike
```
Current per-model active load (pending counts + per-worker potential loads).
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L810)
potential_loads
```python
potential_loads(request: JsonLike) -> JsonLike
```
Per-worker potential loads for a prompt, without booking.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L816)
#### SelectionServiceError (class)
Raised by `SelectionService` for selector failures that are not malformed input.
```python
from dynamo._core import SelectionServiceError
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3263`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3263)
#### SglangArgs (class)
No summary available.
```python
from dynamo._core import SglangArgs
```
```python
SglangArgs(schedule_policy: Optional[str] = None, page_size: Optional[int] = None, max_prefill_tokens: Optional[int] = None, chunked_prefill_size: Optional[int] = None, clip_max_new_tokens: Optional[int] = None, schedule_conservativeness: Optional[float] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1884`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1884)
**Public methods**
**init**
```python
__init__(schedule_policy: Optional[str] = None, page_size: Optional[int] = None, max_prefill_tokens: Optional[int] = None, chunked_prefill_size: Optional[int] = None, clip_max_new_tokens: Optional[int] = None, schedule_conservativeness: Optional[float] = None) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1885)
#### SpanProxy (class)
Unified span handle returned by `Context.current_span()` (the framework auto-span) and `Context.start_span()` (child spans). Mirrors the OTel `Span` API: `set_attribute` / `add_event` / `set_status`. Usable as a Python context manager (closes on `__exit__`). All methods are silent no-ops when the underlying span is absent.
```python
from dynamo._core import SpanProxy
```
[`lib/bindings/python/src/dynamo/_core.pyi#L613`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L613)
**Public methods**
set_attribute
```python
set_attribute(key: str, value: Any) -> None
```
Set an attribute on the span. Any key is accepted; OTel imposes no pre-declaration constraint.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L623)
add_event
```python
add_event(name: str, attrs: Optional[dict[str, Any]] = None) -> None
```
Emit a structured event on the span.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L628)
set_status
```python
set_status(status: str, description: Optional[str] = None) -> None
```
Set the span's status. `status` is `"ok"` or `"error"`; `description` is optional context (typically a short error name).
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L632)
close
```python
close() -> None
```
End the underlying span (child spans only — no-op for the auto-span). Idempotent.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L637)
#### StreamIncomplete (class)
The response stream was terminated before completion.
```python
from dynamo._core import StreamIncomplete
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3258`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3258)
#### TransportType (class)
A read-only view of an instance's transport, wrapping the runtime `TransportType`. `kind` is the transport variant ("tcp" / "nats\_tcp") and `address` is its (transport-specific) address. The address format is not a stable parse target.
```python
from dynamo._core import TransportType
```
[`lib/bindings/python/src/dynamo/_core.pyi#L253`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L253)
#### TrtllmArgs (class)
No summary available.
```python
from dynamo._core import TrtllmArgs
```
```python
TrtllmArgs(capacity_scheduler_policy: Optional[str] = None) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L1896`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1896)
**Public methods**
**init**
```python
__init__(capacity_scheduler_policy: Optional[str] = None) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L1897)
#### Unknown (class)
Uncategorized or unknown error.
```python
from dynamo._core import Unknown
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3223`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3223)
#### VirtualConnectorClient (class)
How a client discovers planner requests and marks them complete
```python
from dynamo._core import VirtualConnectorClient
```
```python
VirtualConnectorClient(runtime: DistributedRuntime, dynamo_namespace: str) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3185`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3185)
**Public methods**
**init**
```python
__init__(runtime: DistributedRuntime, dynamo_namespace: str) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3188)
get
```python
get() -> PlannerDecision
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3191)
complete
```python
complete(decision: PlannerDecision) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3194)
wait
```python
wait() -> None
```
Blocks until there is a new decision to fetch using 'get'
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3197)
#### VirtualConnectorCoordinator (class)
Internal planner virtual connector component
```python
from dynamo._core import VirtualConnectorCoordinator
```
```python
VirtualConnectorCoordinator(runtime: DistributedRuntime, dynamo_namespace: str, check_interval_secs: int, max_wait_time_secs: int, max_retries: int) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3161`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3161)
**Public methods**
**init**
```python
__init__(runtime: DistributedRuntime, dynamo_namespace: str, check_interval_secs: int, max_wait_time_secs: int, max_retries: int) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3164)
async_init
```python
async_init() -> None
```
Call this before using the object
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3167)
read_state
```python
read_state() -> PlannerDecision
```
Get the current values. Most for test / debug.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3171)
update_scaling_decision
```python
update_scaling_decision(num_prefill: Optional[int] = None, num_decode: Optional[int] = None) -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3175)
wait_for_scaling_completion
```python
wait_for_scaling_completion() -> None
```
No summary available.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3178)
is_scaling_ready
```python
is_scaling_ready() -> bool
```
Return whether the client acknowledged the current scaling decision.
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3181)
#### WorkerMetricsPublisher (class)
A metrics publisher will provide metrics to the router for load monitoring.
```python
from dynamo._core import WorkerMetricsPublisher
```
```python
WorkerMetricsPublisher() -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L645`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L645)
**Public methods**
**init**
```python
__init__() -> None
```
Create a `WorkerMetricsPublisher` object
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L652)
create_endpoint
```python
create_endpoint(endpoint: Endpoint) -> None
```
Initialize event-plane publishing for worker metrics. Must be awaited.
Extracts component information from the endpoint to set up metrics publishing
on the endpoint-scoped event subject used for routing decisions.
**Parameters**
**`endpoint`** `Endpoint`
The endpoint to extract component information from for metrics publishing
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L657)
publish
```python
publish(dp_rank: Optional[int] = None, active_decode_blocks: int | None = None, kv_used_blocks: int | None = None) -> None
```
Publish worker metrics for load monitoring.
**Parameters**
**`dp_rank`** `Optional[int]`
Data parallel rank of the worker (None defaults to 0)
---
**`active_decode_blocks`** `int | None`
Optional scheduler-compatible decode-block signal
---
**`kv_used_blocks`** `int | None`
Optional authoritative total KV blocks currently in use
---
[source](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L668)
#### WorkerType (class)
Processing stage a worker handles.
```python
from dynamo._core import WorkerType
```
Each worker has exactly one role; values are not combinable. Use the
`needs` argument on register\_model to express dependencies in DNF form
(a list of alternative AND-sets) — for example, an encode worker that
needs (Prefill AND Decode) OR a single Aggregated peer is expressed as
`[[WorkerType.Prefill, WorkerType.Decode], [WorkerType.Aggregated]]`.
[`lib/bindings/python/src/dynamo/_core.pyi#L2205`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2205)
#### backend (class)
No summary available.
```python
from dynamo._core import backend
```
[`lib/bindings/python/src/dynamo/_core.pyi#L3283`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L3283)
#### compute\_block\_hash\_for\_seq (function)
Compute block hashes for a sequence of tokens, optionally including multimodal metadata.
```python
from dynamo._core import compute_block_hash_for_seq
```
```python
compute_block_hash_for_seq(tokens: List[int], kv_block_size: int, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, is_eagle: Optional[bool] = None, cache_namespace: Optional[str] = None) -> List[int]
```
When block\_mm\_infos is provided, the mm\_hashes are included in the hash computation
to ensure that blocks with identical tokens but different multimodal objects produce
different hashes.
**Parameters**
**`tokens`** `List[int]`
List of token IDs
---
**`kv_block_size`** `int`
Size of each block in tokens
---
**`block_mm_infos`** `Optional[List[Optional[Dict[str, Any]]]]`
Optional per-block multimodal metadata. Each element corresponds to a block
and should be None or a dict with structure:
\{
"mm\_objects": \[
\{
"mm\_hash": int, # Hash of the MM object
}
]
}
---
**`lora_name`** `Optional[str]`
Optional LoRA adapter name for adapter-aware block hashing.
---
**`is_eagle`** `Optional[bool]`
Optional Eagle mode flag. When true, hashes use overlapping
`kv_block_size + 1` token windows with `kv_block_size` stride.
---
**Returns**
* `List[int]` — List of block hashes (one per block)
\>>> tokens = \[1, 2, 3, 4] \* 8 # 32 tokens = 1 block
\>>> mm\_info = \{
... "mm\_objects": \[\{
... "mm\_hash": 0xDEADBEEF,
... }]
... }
\>>> hashes = compute\_block\_hash\_for\_seq(tokens, 32, \[mm\_info])
[`lib/bindings/python/src/dynamo/_core.pyi#L396`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L396)
#### fetch\_model (function)
Download a model from Hugging Face, returning its local path. If `ignore_weights` is True, only fetches tokenizer and config files. Example: `model_path = await fetch_model("Qwen/Qwen3-0.6B")`
```python
from dynamo._core import fetch_model
```
```python
fetch_model(remote_name: str, ignore_weights: bool = False) -> str
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2335`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2335)
#### get\_reasoning\_parser\_names (function)
Get list of available reasoning parser names.
```python
from dynamo._core import get_reasoning_parser_names
```
```python
get_reasoning_parser_names() -> list[str]
```
[`lib/bindings/python/src/dynamo/_core.pyi#L37`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L37)
#### get\_tool\_parser\_names (function)
Get list of available tool parser names.
```python
from dynamo._core import get_tool_parser_names
```
```python
get_tool_parser_names() -> list[str]
```
[`lib/bindings/python/src/dynamo/_core.pyi#L33`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L33)
#### log\_message (function)
Log a message from Python with file and line info
```python
from dynamo._core import log_message
```
```python
log_message(level: str, message: str, module: str, file: str, line: int) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L27`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L27)
#### lora\_name\_to\_id (function)
Generate a deterministic integer ID from a LoRA name using blake3 hash.
```python
from dynamo._core import lora_name_to_id
```
```python
lora_name_to_id(lora_name: str) -> int
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2293`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2293)
#### make\_engine (function)
Make an engine matching the args
```python
from dynamo._core import make_engine
```
```python
make_engine(distributed_runtime: DistributedRuntime, args: EntrypointArgs) -> EngineConfig
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2352`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2352)
#### register\_model (function)
Attach the model at path to the given endpoint, and advertise it as model\_type. LoRA Registration: The `lora_name` and `base_model_path` parameters must be provided together or not at all. Providing only one of these parameters will raise a ValueError. - `lora_name`: The served model name for the LoRA model - `base_model_path`: Path to the base model that the LoRA extends
```python
from dynamo._core import register_model
```
```python
register_model(model_input: ModelInput, model_type: ModelType, endpoint: Endpoint, model_path: str, model_name: Optional[str] = None, *, worker_type: WorkerType, kv_cache_block_size: Optional[int] = None, router_mode: Optional[RouterMode] = None, runtime_config: Optional[ModelRuntimeConfig] = None, tensor_model_config: Optional[Dict[str, Any]] = None, user_data: Optional[Dict[str, Any]] = None, custom_template_path: Optional[str] = None, media_decoder: Optional[MediaDecoder] = None, media_fetcher: Optional[MediaFetcher] = None, lora_name: Optional[str] = None, base_model_path: Optional[str] = None, needs: Optional[List[List[WorkerType]]] = None, self_host_metadata: Optional[bool] = None, ignore_weights: bool = False, max_gpu_lora_count: Optional[int] = None, model_aliases: Optional[List[str]] = None) -> None
```
For TensorBased models (using ModelInput.Tensor), HuggingFace downloads are skipped
and a minimal model card is registered directly. Use model\_path as the display name
for these models. Pass tensor protocol metadata through `tensor_model_config`.
Model serving readiness:
`worker_type` and `needs` describe the worker's processing stage and
peer dependencies. `needs` is a DNF list — each inner list is an
AND-set, the outer list is OR. `worker_type` is required; backends
declare it literally at each call site.
When `ignore_weights` is true, remote HuggingFace model resolution skips
weight files and downloads only the metadata needed for registration.
[`lib/bindings/python/src/dynamo/_core.pyi#L2224`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2224)
#### resolve\_routing\_image\_token\_id (function)
Routing-side image-placeholder token id for a model, resolved with the same per-family logic the frontend's MM-aware KV routing uses. Returns None when the model isn't in the MM-routing registry or its config can't be read. Only present when the bindings are built with the `mm-routing` feature.
```python
from dynamo._core import resolve_routing_image_token_id
```
```python
resolve_routing_image_token_id(model_id: str, model_dir: str) -> Optional[int]
```
[`lib/bindings/python/src/dynamo/_core.pyi#L2297`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2297)
#### run\_input (function)
Start an engine, connect it to an input, and run until stopped.
```python
from dynamo._core import run_input
```
```python
run_input(distributed_runtime: DistributedRuntime, input: str, engine_config: EngineConfig, frontend_route_extensions: Optional[Sequence[FrontendRoute]] = None) -> None
```
`frontend_route_extensions` supplies additional HTTP routes to the
frontend (HTTP input only); see `FrontendRoute`.
[`lib/bindings/python/src/dynamo/_core.pyi#L2418`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2418)
#### run\_kv\_indexer (function)
Run the KV indexer with the given arguments.
```python
from dynamo._core import run_kv_indexer
```
```python
run_kv_indexer(args: List[str]) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L41`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L41)
#### run\_select\_service (function)
Run the Dynamo selection service with the given arguments.
```python
from dynamo._core import run_select_service
```
```python
run_select_service(args: List[str]) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L49`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L49)
#### run\_slot\_tracker (function)
Run the KV router slot tracker with the given arguments.
```python
from dynamo._core import run_slot_tracker
```
```python
run_slot_tracker(args: List[str]) -> None
```
[`lib/bindings/python/src/dynamo/_core.pyi#L45`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L45)
#### unregister\_model (function)
Unregister a model from the discovery system.
```python
from dynamo._core import unregister_model
```
```python
unregister_model(endpoint: Endpoint, lora_name: Optional[str] = None) -> None
```
If lora\_name is provided, unregisters a LoRA adapter instead of a base model.
[`lib/bindings/python/src/dynamo/_core.pyi#L2271`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2271)
#### update\_model\_taints (function)
Replace caller-managed taints on this worker's registered model.
```python
from dynamo._core import update_model_taints
```
```python
update_model_taints(endpoint: Endpoint, taints: Set[str]) -> None
```
Reserved 'dynamo.topology/' taints are derived from the model's topology
metadata and cannot be supplied by callers.
[`lib/bindings/python/src/dynamo/_core.pyi#L2282`](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/src/dynamo/_core.pyi#L2282)