dynamo.llm | NVIDIA Dynamo Documentation

dynamo.llm publishes 36 classes and 11 functions. Source: lib/bindings/python/src/dynamo/llm/__init__.py

AicPerfConfig (class)

No summary available.

1 from dynamo.llm import AicPerfConfig

1 AicPerfConfig(aic_backend: str, aic_system: str, aic_model_path: str, aic_tp_size: int = 1, aic_backend_version: Optional[str] = None, aic_moe_tp_size: Optional[int] = None, aic_moe_ep_size: Optional[int] = None, aic_attention_dp_size: Optional[int] = None, aic_nextn: Optional[int] = None, aic_nextn_accept_rates: Optional[str] = None, aic_gemm_dtype: Optional[str] = None, aic_moe_dtype: Optional[str] = None, aic_fmha_dtype: Optional[str] = None, aic_kv_cache_dtype: Optional[str] = None, aic_comm_dtype: Optional[str] = None) -> None

lib/bindings/python/src/dynamo/_core.pyi#L1722

Public methods

init

1 __init__(aic_backend: str, aic_system: str, aic_model_path: str, aic_tp_size: int = 1, aic_backend_version: Optional[str] = None, aic_moe_tp_size: Optional[int] = None, aic_moe_ep_size: Optional[int] = None, aic_attention_dp_size: Optional[int] = None, aic_nextn: Optional[int] = None, aic_nextn_accept_rates: Optional[str] = None, aic_gemm_dtype: Optional[str] = None, aic_moe_dtype: Optional[str] = None, aic_fmha_dtype: Optional[str] = None, aic_kv_cache_dtype: Optional[str] = None, aic_comm_dtype: Optional[str] = None) -> None

No summary available.

source

EngineType (class)

Engine type for Dynamo workers

1 from dynamo.llm import EngineType

lib/bindings/python/src/dynamo/_core.pyi#L3069

EntrypointArgs (class)

Settings to connect an input to a worker and run them. Use by dynamo run.

1 from dynamo.llm import EntrypointArgs

1 EntrypointArgs(engine_type: EngineType, model_path: Optional[str] = None, model_name: Optional[str] = None, endpoint_id: Optional[str] = None, template_file: Optional[str] = None, router_config: Optional[RouterConfig] = None, kv_cache_block_size: Optional[int] = None, http_host: Optional[str] = None, http_port: Optional[int] = None, http_metrics_port: Optional[int] = None, tls_cert_path: Optional[str] = None, tls_key_path: Optional[str] = None, extra_engine_args: Optional[str] = None, mocker_engine_args: Optional[MockEngineArgs] = None, runtime_config: Optional[ModelRuntimeConfig] = None, namespace: Optional[str] = None, namespace_prefix: Optional[str] = None, is_prefill: bool = False, is_decode: bool = False, migration_limit: int = 0, migration_max_seq_len: Optional[int] = None, chat_engine_factory: Optional[Callable] = None, aic_perf_config: Optional[AicPerfConfig] = None, *, metrics_prefix: Optional[str] = None, enable_anthropic_api: Optional[bool] = None, strip_anthropic_preamble: Optional[bool] = None, enable_streaming_tool_dispatch: Optional[bool] = None, enable_streaming_reasoning_dispatch: Optional[bool] = None, tokenizer_backend: Optional[str] = None) -> None

lib/bindings/python/src/dynamo/_core.pyi#L3076

Public methods

init

1 __init__(engine_type: EngineType, model_path: Optional[str] = None, model_name: Optional[str] = None, endpoint_id: Optional[str] = None, template_file: Optional[str] = None, router_config: Optional[RouterConfig] = None, kv_cache_block_size: Optional[int] = None, http_host: Optional[str] = None, http_port: Optional[int] = None, http_metrics_port: Optional[int] = None, tls_cert_path: Optional[str] = None, tls_key_path: Optional[str] = None, extra_engine_args: Optional[str] = None, mocker_engine_args: Optional[MockEngineArgs] = None, runtime_config: Optional[ModelRuntimeConfig] = None, namespace: Optional[str] = None, namespace_prefix: Optional[str] = None, is_prefill: bool = False, is_decode: bool = False, migration_limit: int = 0, migration_max_seq_len: Optional[int] = None, chat_engine_factory: Optional[Callable] = None, aic_perf_config: Optional[AicPerfConfig] = None, *, metrics_prefix: Optional[str] = None, enable_anthropic_api: Optional[bool] = None, strip_anthropic_preamble: Optional[bool] = None, enable_streaming_tool_dispatch: Optional[bool] = None, enable_streaming_reasoning_dispatch: Optional[bool] = None, tokenizer_backend: Optional[str] = None) -> None

Create EntrypointArgs.

Parameters

engine_type

EngineType

The type of engine to use

model_path

Optional[str]

Path to the model directory on disk

model_name

Optional[str]

Model name or dynamo endpoint (e.g. ‘dyn://namespace.component.endpoint’)

endpoint_id

Optional[str]

Optional endpoint ID

template_file

Optional[str]

Optional path to a prompt template file

router_config

Optional[RouterConfig]

Optional router configuration

kv_cache_block_size

Optional[int]

Optional KV cache block size

http_host

Optional[str]

HTTP host to bind to

http_port

Optional[int]

HTTP port to bind to

http_metrics_port

Optional[int]

HTTP metrics port (for gRPC service)

tls_cert_path

Optional[str]

TLS certificate path (PEM format)

tls_key_path

Optional[str]

TLS key path (PEM format)

extra_engine_args

Optional[str]

Optional path to mocker engine arguments JSON

mocker_engine_args

Optional[MockEngineArgs]

Typed mocker engine arguments

runtime_config

Optional[ModelRuntimeConfig]

Optional runtime configuration for discovery registration

namespace

Optional[str]

Dynamo namespace for model discovery scoping

namespace_prefix

Optional[str]

Optional namespace prefix

is_prefill

bool

Whether this is a prefill worker

is_decode

bool

Whether this is a decode worker (disaggregated); pairs with a prefill peer for readiness

migration_limit

int

Maximum number of request migrations (0=disabled)

migration_max_seq_len

Optional[int]

Optional max sequence length for migration

chat_engine_factory

Optional[Callable]

Optional Python chat completions engine factory callback

aic_perf_config

Optional[AicPerfConfig]

Optional AIC perf-model configuration for default KV routing

metrics_prefix

Optional[str]

Optional Prometheus metrics prefix override

enable_anthropic_api

Optional[bool]

Optional Anthropic Messages API override

strip_anthropic_preamble

Optional[bool]

Optional Anthropic preamble stripping override

enable_streaming_tool_dispatch

Optional[bool]

Optional streaming tool dispatch override

enable_streaming_reasoning_dispatch

Optional[bool]

Optional streaming reasoning dispatch override

tokenizer_backend

Optional[str]

Optional tokenizer backend override (“default” or “fastokens”)

source

FpmDirectPublisher (class)

Direct Forward Pass Metrics publisher used by in-process producers such as the TRT-LLM adapter. The underlying Rust publisher owns per-DP-rank serialization tasks (each with its own 1s idle heartbeat timer) and a single event-plane publisher task. Python callers do not manage heartbeat: when publish is not called for IDLE_HEARTBEAT_INTERVAL (1.0s, matching vLLM’s HEARTBEAT_INTERVAL), the Rust side emits a zeroed snapshot on that rank’s channel.

1 from dynamo.llm import FpmDirectPublisher

1 FpmDirectPublisher(endpoint: Endpoint, worker_id: str, dp_size: int = 1) -> None

lib/bindings/python/src/dynamo/_core.pyi#L1285

Public methods

init

1 __init__(endpoint: Endpoint, worker_id: str, dp_size: int = 1) -> None

Create a publisher with dp_size per-DP-rank channels.

Parameters

endpoint

Endpoint

Dynamo component endpoint (provides runtime + discovery).

worker_id

str

Unique worker identifier stamped on every emitted FPM.

dp_size

int

Number of DP ranks to allocate channels for. Use 1 when attention DP is disabled.

source

publish

1 publish(*, dp_rank: int, scheduled_num_prefill_requests: int, scheduled_sum_prefill_tokens: int, scheduled_sum_prefill_kv_tokens: int, scheduled_num_decode_requests: int, scheduled_sum_decode_kv_tokens: int, queued_num_prefill_requests: int, queued_sum_prefill_tokens: int, queued_num_decode_requests: int, queued_sum_decode_kv_tokens: int, wall_time_secs: float) -> None

Publish one iteration’s FPM snapshot for the given DP rank.

All parameters are keyword-only on the Python side: adjacent ints with similar units (scheduled_* vs queued_*, *_prefill_* vs *_decode_*) cannot be distinguished by the type system, so a transposition would silently corrupt every published snapshot.

Variance fields (var_prefill_length, var_decode_kv_tokens, var_queued_prefill_length, var_queued_decode_kv_tokens) are defaulted to 0.0 per the MVP scope; a follow-up PR can add Welford-based variance computation.

source

shutdown

1 shutdown() -> None

Shut down the publisher and its per-rank serialization tasks.

source

FpmEventRelay (class)

Relay that bridges ForwardPassMetrics from a local raw ZMQ PUB socket (InstrumentedScheduler in EngineCore child process) to the Dynamo event plane with automatic discovery registration.

1 from dynamo.llm import FpmEventRelay

1 FpmEventRelay(endpoint: Endpoint, zmq_endpoint: str) -> None

lib/bindings/python/src/dynamo/_core.pyi#L1258

Public methods

init

1 __init__(endpoint: Endpoint, zmq_endpoint: str) -> None

Create a relay.

Parameters

endpoint

Endpoint

Dynamo component endpoint (provides runtime + discovery).

zmq_endpoint

str

Local ZMQ PUB address to subscribe to (e.g., “tcp://127.0.0.1:20380”).

source

shutdown

1 shutdown() -> None

Shut down the relay task.

source

FpmEventSubscriber (class)

Subscriber for ForwardPassMetrics from the Dynamo event plane. Auto-discovers engine publishers via the discovery plane.

1 from dynamo.llm import FpmEventSubscriber

1 FpmEventSubscriber(endpoint: Endpoint) -> None

Two mutually exclusive usage modes:

recv mode (default): call recv() to pull individual messages.
tracking mode: call start_tracking() once, then poll get_recent_stats() to retrieve the latest FPM bytes keyed by (worker_id, dp_rank). Stale entries are cleaned up when workers are removed (via discovery watch).

lib/bindings/python/src/dynamo/_core.pyi#L1348

Public methods

init

1 __init__(endpoint: Endpoint) -> None

Create a subscriber that auto-discovers FPM publishers.

No background tasks are started until recv() or start_tracking() is called.

Parameters

endpoint

Endpoint

Dynamo component endpoint (provides runtime + discovery).

source

recv

1 recv() -> Optional[bytes]

Blocking receive of the next message (raw msgspec bytes). Releases the GIL while waiting.

On the first call a background subscriber task is spawned (recv mode). Cannot be used after start_tracking().

Returns

Optional[bytes] — Raw msgspec payload, or None if the stream is closed.

source

start_tracking

1 start_tracking() -> None

Start background tracking of the latest FPM per (worker_id, dp_rank).

Spawns two background tasks:

Event consumption: subscribes to FPM events, extracts the composite key (worker_id, dp_rank) from the msgpack payload, stores latest raw bytes in an internal map.
MDC discovery watch: monitors ComponentModels for the target component. When a model is removed, all entries whose worker_id matches the removed instance_id are purged.

After calling this, recv() will raise RuntimeError.

source

get_recent_stats

1 get_recent_stats() -> dict[tuple[str, int], bytes]

Return the latest FPM bytes for every tracked (worker_id, dp_rank).

Cleanup of removed engines is handled by the MDC discovery watch task spawned by start_tracking().

Raises RuntimeError if start_tracking() has not been called.

Returns

dict[tuple[str, int], bytes] — dict mapping (worker_id, dp_rank) to raw msgspec bytes.
dict[tuple[str, int], bytes] — Decode each value with forward_pass_metrics.decode(data).

source

get_model_cards

1 get_model_cards() -> dict[str, str]

Snapshot of model deployment cards keyed by worker id.

The snapshot is filtered against the known-workers set so entries for already-removed workers are not returned. Values are the raw ModelDeploymentCard serialized as a JSON string; callers parse whichever fields they need (e.g. runtime_config, display_name).

Raises RuntimeError if start_tracking() has not been called.

Returns

dict[str, str] — dict mapping worker_id to card_json (JSON string).

source

shutdown

1 shutdown() -> None

Shut down the subscriber (all background tasks).

source

FrontendExtensionContext (class)

Read-only, live view of frontend state passed to extension route handlers.

1 from dynamo.llm import FrontendExtensionContext

Handlers receive this and answer from current state. The surface is intentionally narrow (typed read-only accessors only); it does not expose the internal service state.

lib/bindings/python/src/dynamo/_core.pyi#L2356

Public methods

is_ready

1 is_ready() -> bool

Whether the HTTP service has finished startup and is ready to serve.

source

is_cancelled

1 is_cancelled() -> bool

Whether the frontend is shutting down (draining).

source

has_any_ready_model

1 has_any_ready_model() -> bool

Whether at least one model is registered and ready to serve.

source

is_model_ready_to_serve

1 is_model_ready_to_serve(model: str) -> bool

Whether the named model is registered and ready to serve.

source

model_display_names

1 model_display_names() -> list[str]

Sorted display names of all registered models.

source

serving_ready_display_names

1 serving_ready_display_names() -> list[str]

Sorted display names of models ready to serve.

source

FrontendResponse (class)

Explicit status-code override returned by a FrontendRoute handler.

1 from dynamo.llm import FrontendResponse

1 FrontendResponse(status_code: int, body: object) -> None

Return this to set a non-200 status (e.g. FrontendResponse(503, body)); return a plain JSON-serializable value for the default 200.

lib/bindings/python/src/dynamo/_core.pyi#L2409

Public methods

init

1 __init__(status_code: int, body: object) -> None

No summary available.

source

FrontendRoute (class)

A trusted extension route served on the Dynamo HTTP frontend.

1 from dynamo.llm import FrontendRoute

1 FrontendRoute(method: str, path: str, handler: Callable[[FrontendExtensionContext], object]) -> None

Currently restricted to static-path GET routes. handler is a synchronous callable that receives a FrontendExtensionContext and returns a JSON-serializable body (implies HTTP 200) or a FrontendResponse to set the status code. Async handlers and path parameters are rejected at construction.

lib/bindings/python/src/dynamo/_core.pyi#L2388

Public methods

init

1 __init__(method: str, path: str, handler: Callable[[FrontendExtensionContext], object]) -> None

No summary available.

source

HttpAsyncEngine (class)

An async engine for a distributed Dynamo http service. This is an extension of the python based AsyncEngine that handles HttpError exceptions from Python and converts them to the Rust version of HttpError

1 from dynamo.llm import HttpAsyncEngine

lib/bindings/python/src/dynamo/_core.pyi#L1482

HttpError (class)

No summary available.

1 from dynamo.llm import HttpError

1 HttpError(code: int, message: str)

lib/bindings/python/src/dynamo/llm/exceptions.py#L25

Public methods

init

1 __init__(code: int, message: str)

No summary available.

source

HttpService (class)

A HTTP service for dynamo applications. It is a OpenAI compatible http ingress into the Dynamo Distributed Runtime.

1 from dynamo.llm import HttpService

1 HttpService(port: Optional[int] = None) -> None

lib/bindings/python/src/dynamo/_core.pyi#L1441

Public methods

init

1 __init__(port: Optional[int] = None) -> None

Create a new HTTP service.

Parameters

port

Optional[int]

Optional port number to bind the service to (default: 8080)

source

run

1 run(runtime: DistributedRuntime) -> None

Run the HTTP service.

Parameters

runtime

DistributedRuntime

DistributedRuntime instance for token management

source

shutdown

1 shutdown() -> None

Shutdown the HTTP service by cancelling its internal token.

source

KserveGrpcService (class)

A gRPC service implementing the KServe protocol for dynamo applications. Provides model management for completions, chat completions, and tensor-based models.

1 from dynamo.llm import KserveGrpcService

1 KserveGrpcService(port: Optional[int] = None, host: Optional[str] = None) -> None

lib/bindings/python/src/dynamo/_core.pyi#L1491

Public methods

init

1 __init__(port: Optional[int] = None, host: Optional[str] = None) -> None

Create a new KServe gRPC service.

Parameters

port

Optional[int]

Optional port number to bind the service to

host

Optional[str]

Optional host address to bind the service to

source

add_completions_model

1 add_completions_model(model: str, checksum: str, engine: PythonAsyncEngine) -> None

Parameters

model

str

The model name

checksum

str

The model checksum

engine

PythonAsyncEngine

The async engine to handle requests

source

add_chat_completions_model

1 add_chat_completions_model(model: str, checksum: str, engine: PythonAsyncEngine) -> None

Parameters

model

str

The model name

checksum

str

The model checksum

engine

PythonAsyncEngine

The async engine to handle requests

source

add_tensor_model

1 add_tensor_model(model: str, checksum: str, engine: PythonAsyncEngine, *, runtime_config: Optional[ModelRuntimeConfig] = None, tensor_model_config: Optional[Dict[str, Any]] = None) -> None

Parameters

model

str

The model name

checksum

str

The model checksum

engine

PythonAsyncEngine

The async engine to handle requests

runtime_config

Optional[ModelRuntimeConfig]

Optional runtime-resolved worker metadata

tensor_model_config

Optional[Dict[str, Any]]

Optional tensor protocol model metadata

source

remove_completions_model

1 remove_completions_model(model: str) -> None

Remove a completions model from the service.

Parameters

model

str

The model name to remove

source

remove_chat_completions_model

1 remove_chat_completions_model(model: str) -> None

Remove a chat completions model from the service.

Parameters

model

str

The model name to remove

source

remove_tensor_model

1 remove_tensor_model(model: str) -> None

Remove a tensor model from the service.

Parameters

model

str

The model name to remove

source

list_chat_completions_models

1 list_chat_completions_models() -> List[str]

List all registered chat completions models.

Returns

List[str] — List of model names

source

list_completions_models

1 list_completions_models() -> List[str]

List all registered completions models.

Returns

List[str] — List of model names

source

list_tensor_models

1 list_tensor_models() -> List[str]

List all registered tensor models.

Returns

List[str] — List of model names

source

run

1 run(runtime: DistributedRuntime) -> None

Run the KServe gRPC service.

Parameters

runtime

DistributedRuntime

DistributedRuntime instance for token management

source

shutdown

1 shutdown() -> None

Shutdown the KServe gRPC service by cancelling its internal token.

source

KvDcRelay (class)

No summary available.

1 from dynamo.llm import KvDcRelay

1 KvDcRelay(endpoint: Endpoint, dc_id: str, namespace_filter: Optional[str] = None, endpoint_prefix: Optional[str] = None, publication_threshold: int = 16, publication_delay_ms: int = 1, recovery_attempt_timeout_ms: int = 30000) -> None

lib/bindings/python/src/dynamo/_core.pyi#L2801

Public methods

init

1 __init__(endpoint: Endpoint, dc_id: str, namespace_filter: Optional[str] = None, endpoint_prefix: Optional[str] = None, publication_threshold: int = 16, publication_delay_ms: int = 1, recovery_attempt_timeout_ms: int = 30000) -> None

No summary available.

source

start

1 start() -> None

No summary available.

source

health

1 health() -> Dict[str, Any]

No summary available.

source

flush

1 flush() -> None

No summary available.

source

shutdown

1 shutdown() -> None

No summary available.

source

KvEventPublisher (class)

A KV event publisher will publish KV events corresponding to the component.

1 from dynamo.llm import KvEventPublisher

1 KvEventPublisher(endpoint: Endpoint, worker_id: Optional[int] = None, kv_block_size: int = 0, dp_rank: int = 0, enable_local_indexer: bool = False, zmq_endpoint: Optional[str] = None, zmq_topic: Optional[str] = None, batching_timeout_ms: Optional[int] = None, image_token_id: Optional[int] = None, kv_state_endpoint: Optional[str] = None) -> None

lib/bindings/python/src/dynamo/_core.pyi#L1156

Public methods

init

1 __init__(endpoint: Endpoint, worker_id: Optional[int] = None, kv_block_size: int = 0, dp_rank: int = 0, enable_local_indexer: bool = False, zmq_endpoint: Optional[str] = None, zmq_topic: Optional[str] = None, batching_timeout_ms: Optional[int] = None, image_token_id: Optional[int] = None, kv_state_endpoint: Optional[str] = None) -> None

Create a KvEventPublisher object.

When zmq_endpoint is provided, the publisher subscribes to a ZMQ socket for incoming engine events (e.g. from SGLang/vLLM) and relays them to NATS.

When zmq_endpoint is None, events are pushed manually via publish_batch, publish_stored, or publish_removed.

Parameters

endpoint

Endpoint

The endpoint to extract component information from for event publishing

worker_id

Optional[int]

Optional worker ID override. Use None to infer from endpoint.

kv_block_size

int

The KV block size (must be > 0)

dp_rank

int

The data parallel rank (defaults to 0)

enable_local_indexer

bool

Enable worker-local KV indexer

zmq_endpoint

Optional[str]

Optional ZMQ endpoint for relay mode (e.g. “tcp://127.0.0.1:5557”)

zmq_topic

Optional[str]

ZMQ topic to subscribe to (defaults to "" when zmq_endpoint is set)

batching_timeout_ms

Optional[int]

Cross-list batching timeout in milliseconds. None/0 flushes at each submitted source-list boundary.

kv_state_endpoint

Optional[str]

KV event ownership endpoint; defaults to endpoint.

source

publish_stored

1 publish_stored(token_ids: List[int], num_block_tokens: List[int], block_hashes: List[int], parent_hash: Optional[int] = None, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, is_eagle: Optional[bool] = None, cache_salt: Optional[str] = None) -> None

Publish a KV stored event.

Event IDs are managed internally by the publisher using a monotonic counter.

Parameters

token_ids

List[int]

List of token IDs

num_block_tokens

List[int]

Number of tokens per block

block_hashes

List[int]

List of block hashes (signed 64-bit integers)

parent_hash

Optional[int]

Optional parent hash (signed 64-bit integer)

block_mm_infos

Optional[List[Optional[Dict[str, Any]]]]

Optional list of multimodal info for each block. Each item is either None or a dict with “mm_objects” key containing a list of {“mm_hash”: int, “offsets”: [[start, end], …]} dicts.

lora_name

Optional[str]

Optional LoRA adapter name for adapter-aware block hashing.

is_eagle

Optional[bool]

Optional Eagle mode flag. When true, stored blocks are reconstructed using overlapping kv_block_size + 1 token windows.

source

publish_removed

1 publish_removed(block_hashes: List[int]) -> None

Publish a KV removed event.

Event IDs are managed internally by the publisher using a monotonic counter.

Parameters

block_hashes

List[int]

List of block hashes to remove (signed 64-bit integers)

source

publish_batch

1 publish_batch(events: Sequence[KvStoredEventInput | KvRemovedEventInput]) -> None

Publish an ordered list of KV events as one processor input.

The complete list is validated before it is enqueued. Compatible events are coalesced while preserving source order and the processor’s existing block-count limits.

source

shutdown

1 shutdown() -> None

Shuts down the event publisher, stopping any background tasks.

source

KvRouter (class)

A KV-aware router that performs intelligent routing based on KV cache overlap.

1 from dynamo.llm import KvRouter

1 KvRouter(endpoint: Endpoint, block_size: int, kv_router_config: KvRouterConfig, aic_perf_config: Optional[AicPerfConfig] = None) -> None

lib/bindings/python/src/dynamo/_core.pyi#L2826

Public methods

init

1 __init__(endpoint: Endpoint, block_size: int, kv_router_config: KvRouterConfig, aic_perf_config: Optional[AicPerfConfig] = None) -> None

Create a new KvRouter instance.

Parameters

endpoint

Endpoint

The endpoint to connect to for routing requests

block_size

int

The KV cache block size

kv_router_config

KvRouterConfig

Configuration for the KV router

aic_perf_config

Optional[AicPerfConfig]

Optional AIC perf-model config for effective prefill load tracking

source

generate

1 generate(token_ids: List[int], model: str, stop_conditions: Optional[JsonLike] = None, sampling_options: Optional[JsonLike] = None, output_options: Optional[JsonLike] = None, router_config_override: Optional[JsonLike] = None, worker_id: Optional[int] = None, dp_rank: Optional[int] = None, extra_args: Optional[JsonLike] = None, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, multi_modal_data: Optional[JsonLike] = None, mm_routing_info: Optional[JsonLike] = None, routing_constraints: Optional[RoutingConstraints] = None, response_buffer_size: int = 100) -> AsyncIterator[JsonLike]

Generate text using the KV-aware router.

Parameters

token_ids

List[int]

Input token IDs

model

str

Model name to use for generation

stop_conditions

Optional[JsonLike]

Optional stop conditions for generation

sampling_options

Optional[JsonLike]

Optional sampling configuration

output_options

Optional[JsonLike]

Optional output configuration

router_config_override

Optional[JsonLike]

Optional router configuration override

worker_id

Optional[int]

Optional worker ID to route to directly. If set, the request will be sent to this specific worker and router states will be updated accordingly.

dp_rank

Optional[int]

Optional data parallel rank to route to. If set along with worker_id, the request will be routed to the specific (worker_id, dp_rank) pair. If only dp_rank is set, the router will select the best worker but force routing to the specified dp_rank.

extra_args

Optional[JsonLike]

Optional extra request arguments to include in the PreprocessedRequest.

block_mm_infos

Optional[List[Optional[Dict[str, Any]]]]

Optional block-level multimodal metadata aligned to request blocks. Backward-compatible shortcut; this is converted to mm_routing_info with routing_token_ids=token_ids.

multi_modal_data

Optional[JsonLike]

Optional multimodal payload map to preserve image/video data for downstream model execution.

mm_routing_info

Optional[JsonLike]

Optional structured routing-only multimodal payload (e.g., {“routing_token_ids”: […], “block_mm_infos”: […]}) used by router selection without changing execution token_ids.

routing_constraints

Optional[RoutingConstraints]

Optional request routing constraints used to constrain or prefer tainted workers.

response_buffer_size

int

Maximum number of responses buffered by the Python adapter. Set to 0 for demand-driven direct Python consumption; negative values are rejected.

Returns

AsyncIterator[JsonLike] — An async iterator yielding generation responses

If worker_id is set, the request bypasses KV matching and routes directly to the specified worker while still updating router states.
dp_rank allows targeting a specific data parallel replica when workers have multiple replicas (data_parallel_size > 1).
This is different from query_instance_id which doesn’t route the request.

source

generate_from_request

1 generate_from_request(request: JsonLike, response_buffer_size: int = 100) -> AsyncIterator[JsonLike]

Generate from a preprocessed request dict (PreprocessedRequest format).

Accepts a full request dict with token_ids, model, stop_conditions, etc. Set response_buffer_size to 0 for demand-driven direct Python consumption; negative values are rejected. Returns an async iterator yielding generation responses.

source

best_worker

1 best_worker(token_ids: List[int], router_config_override: Optional[JsonLike] = None, request_id: Optional[str] = None, update_indexer: bool = False, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, routing_constraints: Optional[RoutingConstraints] = None, strict_priority: int = 0, policy_class: Optional[str] = None, cache_namespace: Optional[str] = None) -> Tuple[int, int, int]

Find the best matching worker for the given tokens.

Parameters

token_ids

List[int]

List of token IDs to find matches for

router_config_override

Optional[JsonLike]

Optional router configuration override

request_id

Optional[str]

Optional request ID. If provided, router states will be updated to track this request (active blocks, lifecycle events). If not provided, this is a query-only operation that doesn’t affect state.

update_indexer

bool

Whether to record the selected worker in the router’s approximate indexer. This is only meaningful when use_kv_events=False and is independent from lifecycle state tracking via request_id.

block_mm_infos

Optional[List[Optional[Dict[str, Any]]]]

Optional block-level multimodal metadata aligned to request blocks. When provided, this is used in block hash computation to enable MM-aware worker selection.

cache_namespace

Optional[str]

Optional cache namespace used in block hash computation.

policy_class

Optional[str]

Requested policy family, or an exact explicit class. Missing, unknown, and ordinary physical-class names use the configured default family before cache-bucket resolution.

Returns

Tuple[int, int, int] — A tuple of (worker_id, dp_rank, overlap_blocks) where: - worker_id: The ID of the best matching worker - dp_rank: The data parallel rank of the selected worker - overlap_blocks: The number of overlapping blocks found

source

get_potential_loads

1 get_potential_loads(token_ids: List[int], block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, cache_namespace: Optional[str] = None) -> List[Dict[str, int]]

Get potential prefill and decode loads for all workers.

Parameters

token_ids

List[int]

List of token IDs to evaluate

block_mm_infos

Optional[List[Optional[Dict[str, Any]]]]

Optional block-level multimodal metadata aligned to request blocks. When provided, this is used in hash computation for MM-aware potential-load estimation.

lora_name

Optional[str]

Optional LoRA adapter name used in block hash computation.

Returns

List[Dict[str, int]] — A list of dictionaries, each containing: - worker_id: The worker ID - dp_rank: The data parallel rank - potential_prefill_tokens: Number of tokens that would need prefill - potential_decode_blocks: Number of blocks currently in decode phase - active_requests: Number of active requests tracked on the worker

Each (worker_id, dp_rank) pair is returned as a separate entry. If you need aggregated loads per worker_id, sum the values manually.

source

get_overlap_scores

1 get_overlap_scores(token_ids: List[int], router_config_override: Optional[JsonLike] = None, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, include_shared: bool = True, cache_namespace: Optional[str] = None) -> Dict[str, Any]

Get per-worker KV overlap by storage tier.

Parameters

token_ids

List[int]

List of token IDs to evaluate.

router_config_override

Optional[JsonLike]

Optional router configuration override for score-credit fields.

block_mm_infos

Optional[List[Optional[Dict[str, Any]]]]

Optional block-level multimodal metadata aligned to request blocks.

lora_name

Optional[str]

Optional LoRA adapter name for adapter-aware matching.

include_shared

bool

Whether to query the configured shared cache.

Returns

Dict[str, Any] — A dictionary containing block_size, num_blocks, shared_cache, and
Dict[str, Any] — workers. Each worker row is keyed by worker_id and dp_rank and
Dict[str, Any] — reports device, host-pinned, disk, and shared-cache overlap blocks.

source

dump_events

1 dump_events() -> str

Dump all events from the KV router’s indexer.

Returns

str — A JSON string containing all indexer events

source

mark_prefill_complete

1 mark_prefill_complete(request_id: str) -> None

Mark prefill as completed for a request.

This signals that the request has finished its prefill phase and is now in the decode phase. Used to update router state for accurate load tracking.

Parameters

request_id

str

The ID of the request that completed prefill

This is typically called automatically by the router when using the generate() method. Only call this manually if you’re using best_worker() with request_id for custom routing.

source

free

1 free(request_id: str) -> None

Free a request by its ID, signaling the router to release resources.

This should be called when a request completes to update the router’s tracking of active blocks and ensure accurate load balancing.

Parameters

request_id

str

The ID of the request to free

This is typically called automatically by the router when using the generate() method. Only call this manually if you’re using best_worker() with request_id for custom routing.

source

KvRouterConfig (class)

Values for KV router

1 from dynamo.llm import KvRouterConfig

1 KvRouterConfig(overlap_score_weight: Optional[float] = None, host_cache_hit_weight: float = 0.75, disk_cache_hit_weight: float = 0.25, router_temperature: float = 0.0, use_kv_events: bool = True, *, router_replica_sync: bool = False, router_track_active_blocks: bool = True, router_track_output_blocks: bool = False, router_assume_kv_reuse: bool = True, router_track_prefill_tokens: bool = True, router_prefill_load_model: str = 'none', router_ttl_secs: float = 120.0, router_queue_threshold: Optional[float] = None, router_event_threads: int = 4, router_queue_policy: str = 'fcfs', use_remote_indexer: bool = False, serve_indexer: bool = False, shared_cache_multiplier: float = 0.0, shared_cache_type: str = 'none', router_predicted_ttl_secs: Optional[float] = None, overlap_score_credit: float = 1.0, overlap_score_credit_decay: float = 0.0, prefill_load_scale: float = 1.0, decode_active_request_weight: float = 0.0, router_policy_config: Optional[str] = None, router_tracking_hash: Literal['public-xxh3-v1', 'keyed-xxh3-v1'] = 'public-xxh3-v1', router_tracking_key_file: Optional[str | os.PathLike[str]] = None, router_tracking_key_id: Optional[str] = None) -> None

lib/bindings/python/src/dynamo/_core.pyi#L1743

Public methods

init

1 __init__(overlap_score_weight: Optional[float] = None, host_cache_hit_weight: float = 0.75, disk_cache_hit_weight: float = 0.25, router_temperature: float = 0.0, use_kv_events: bool = True, *, router_replica_sync: bool = False, router_track_active_blocks: bool = True, router_track_output_blocks: bool = False, router_assume_kv_reuse: bool = True, router_track_prefill_tokens: bool = True, router_prefill_load_model: str = 'none', router_ttl_secs: float = 120.0, router_queue_threshold: Optional[float] = None, router_event_threads: int = 4, router_queue_policy: str = 'fcfs', use_remote_indexer: bool = False, serve_indexer: bool = False, shared_cache_multiplier: float = 0.0, shared_cache_type: str = 'none', router_predicted_ttl_secs: Optional[float] = None, overlap_score_credit: float = 1.0, overlap_score_credit_decay: float = 0.0, prefill_load_scale: float = 1.0, decode_active_request_weight: float = 0.0, router_policy_config: Optional[str] = None, router_tracking_hash: Literal['public-xxh3-v1', 'keyed-xxh3-v1'] = 'public-xxh3-v1', router_tracking_key_file: Optional[str | os.PathLike[str]] = None, router_tracking_key_id: Optional[str] = None) -> None

Create a KV router configuration.

Parameters

overlap_score_weight

Optional[float]

Deprecated positional/keyword alias for prefill_load_scale. When present, it takes precedence over prefill_load_scale; a value of 0 also sets overlap_score_credit to 0.

overlap_score_credit

float

Finite, non-negative credit multiplier for device-local prefix overlap (default: 1.0). Values above 1.0 give device overlap extra credit, with adjusted prefill cost clamped at zero.

prefill_load_scale

float

Scale for adjusted prompt-side prefill load after cache-hit credits (default: 1.0)

decode_active_request_weight

float

Experimental block-equivalent decode cost added for each active request on a candidate worker (default: 0.0)

host_cache_hit_weight

float

Credit multiplier for host-pinned cache hits (default: 0.75)

disk_cache_hit_weight

float

Credit multiplier for disk/external cache hits (default: 0.25)

router_temperature

float

Temperature for normalized worker sampling via softmax (default: 0.0)

use_kv_events

bool

Whether to use KV events from workers (default: True)

router_replica_sync

bool

Enable replica synchronization (default: False)

router_track_active_blocks

bool

Track active blocks for load balancing (default: True)

router_track_output_blocks

bool

Track output blocks during generation (default: False). When enabled, the router adds placeholder blocks as tokens are generated and, with expected output sequence length (agent_hints.osl in nvext), applies fractional decay to output blocks and the structurally exclusive prompt suffix. Shared prompt blocks retain full weight.

router_assume_kv_reuse

bool

Assume KV cache reuse when tracking active blocks (default: True). When True, computes actual block hashes. When False, generates random hashes.

router_track_prefill_tokens

bool

Include prompt-side prefill tokens in active load accounting (default: True).

router_tracking_hash

Literal['public-xxh3-v1', 'keyed-xxh3-v1']

Tracking identity algorithm, “public-xxh3-v1” or “keyed-xxh3-v1” (default: “public-xxh3-v1”).

router_tracking_key_file

Optional[str | os.PathLike[str]]

File containing exactly 32 raw provider-key bytes. Required only for keyed tracking mode.

router_tracking_key_id

Optional[str]

Provider-managed key epoch mixed into keyed scope derivation. Required only for keyed tracking mode.

router_prefill_load_model

str

Prompt-side prefill load model (default: “none”). “none” keeps static prompt load accounting. “aic” decays the oldest active prefill request using AIC-predicted duration.

router_ttl_secs

float

TTL for blocks in seconds when not using KV events (default: 120.0)

router_queue_threshold

Optional[float]

Optional queue threshold fraction for prefill token capacity (default: None). Requests are queued if all workers exceed this fraction of max_num_batched_tokens. Enables priority scheduling via request priority hints. Set a numeric value to enable queueing.

router_policy_config

Optional[str]

Startup-only policy-family and cache-bucket queue YAML path. When omitted, router_queue_threshold and router_queue_policy define one synthetic policy class.

router_event_threads

int

Number of KV indexer worker threads (default: 4). When > 1, uses a concurrent radix tree with a thread pool, including for approximate routing when KV events are disabled.

router_queue_policy

str

Scheduling policy for the router queue (default: “fcfs”). “fcfs”: first-come first-served with priority bumps — optimizes tail TTFT. “lcfs”: last-come first-served with priority bumps — intentionally worsens tail behavior for policy comparisons. “wspt”: weighted shortest processing time (Smith’s rule) — optimizes average TTFT.

use_remote_indexer

bool

Query a remote KV indexer served from the worker component (default: False).

serve_indexer

bool

Serve this router’s local indexer from the worker component (default: False).

shared_cache_multiplier

float

Credit multiplier for shared cache hits beyond the device prefix (default: 0.0).

shared_cache_type

str

External shared KV cache type, “none” or “hicache” (default: “none”).

router_predicted_ttl_secs

Optional[float]

Enables predict-on-route when set. This TTL applies to entries in the local side indexer and requires use_kv_events=True. Set to None to disable. Independent of router_ttl_secs, which covers pure approximate mode.

source

from_json

1 from_json(config_json: str) -> KvRouterConfig

No summary available.

source

copy

1 copy() -> KvRouterConfig

No summary available.

source

with_overrides

1 with_overrides(overlap_score_weight: Optional[float] = None, *, overlap_score_credit: Optional[float] = None, overlap_score_credit_decay: Optional[float] = None, prefill_load_scale: Optional[float] = None, decode_active_request_weight: Optional[float] = None) -> KvRouterConfig

No summary available.

source

LoRADownloader (class)

Unified interface for LoRA downloading and caching (local file:// and S3 s3:// URIs).

1 from dynamo.llm import LoRADownloader

1 LoRADownloader(cache_path: Optional[str] = None) -> None

lib/bindings/python/src/dynamo/_core.pyi#L2305

Public methods

init

1 __init__(cache_path: Optional[str] = None) -> None

No summary available.

source

download_if_needed

1 download_if_needed(lora_uri: str) -> Awaitable[str]

No summary available.

source

get_cache_path

1 get_cache_path(cache_key: str) -> str

No summary available.

source

is_cached

1 is_cached(lora_uri: str) -> bool

No summary available.

source

validate_cached

1 validate_cached(cache_key: str) -> bool

No summary available.

source

uri_to_cache_key

1 uri_to_cache_key(uri: str) -> str

No summary available.

source

MediaDecoder (class)

Media decoder for image and video preprocessing.

1 from dynamo.llm import MediaDecoder

1 MediaDecoder() -> None

lib/bindings/python/src/dynamo/_core.pyi#L2318

Public methods

init

1 __init__() -> None

No summary available.

source

enable_image

1 enable_image(decoder_options: Dict[str, Any]) -> None

No summary available.

source

MediaFetcher (class)

Media fetcher for loading remote image/video URLs.

1 from dynamo.llm import MediaFetcher

1 MediaFetcher() -> None

lib/bindings/python/src/dynamo/_core.pyi#L2325

Public methods

init

1 __init__() -> None

No summary available.

source

user_agent

1 user_agent(user_agent: str) -> None

No summary available.

source

allow_direct_ip

1 allow_direct_ip(allow: bool) -> None

No summary available.

source

allow_direct_port

1 allow_direct_port(allow: bool) -> None

No summary available.

source

allowed_media_domains

1 allowed_media_domains(domains: List[str]) -> None

No summary available.

source

timeout_ms

1 timeout_ms(timeout_ms: int) -> None

No summary available.

source

ModelCardInstanceId (class)

Unique identifier for a worker instance: namespace, component, endpoint and instance_id. The instance_id is not currently exposed in the Python bindings.

1 from dynamo.llm import ModelCardInstanceId

lib/bindings/python/src/dynamo/_core.pyi#L384

Public methods

triple

1 triple() -> Tuple[str, str, str]

Triple of namespace, component and endpoint this worker is serving.

source

ModelInput (class)

What type of request this model needs: Text, Tokens or Tensor

1 from dynamo.llm import ModelInput

lib/bindings/python/src/dynamo/_core.pyi#L1629

ModelRuntimeConfig (class)

A model runtime configuration is a collection of runtime information

1 from dynamo.llm import ModelRuntimeConfig

1 ModelRuntimeConfig() -> None

lib/bindings/python/src/dynamo/_core.pyi#L855

Public methods

init

1 __init__() -> None

No summary available.

source

set_engine_specific

1 set_engine_specific(key: str, value: Any) -> None

Set an engine-specific runtime configuration value

source

get_engine_specific

1 get_engine_specific(key: str) -> Any | None

Get an engine-specific runtime configuration value

source

set_structural_tag_mode

1 set_structural_tag_mode(mode: str) -> None

Set structural tag mode (“off” or “on”).

source

set_structural_tag_scope

1 set_structural_tag_scope(scope: str) -> None

Set structural tag scope (“auto” or “always”).

source

set_structural_tag_schema

1 set_structural_tag_schema(schema: str) -> None

Set structural tag schema mode (“auto” or “strict”).

source

set_disaggregated_endpoint

1 set_disaggregated_endpoint(bootstrap_host: str | None = None, bootstrap_port: int | None = None) -> None

Set the disaggregated endpoint for the model

source

ModelType (class)

OpenAI-style surfaces supported by a model.

1 from dynamo.llm import ModelType

Values are Chat, Completions, Embedding, Classify, Pooling, TensorBased, Images, Audios, Videos, Realtime, and Empty (no OpenAI surface).

lib/bindings/python/src/dynamo/_core.pyi#L1636

Public methods

supports_chat

1 supports_chat() -> bool

Return True if this model type supports chat.

source

supports_embedding

1 supports_embedding() -> bool

Return True if this model type supports /v1/embeddings.

source

supports_classify

1 supports_classify() -> bool

Return True if this model type supports /v1/classify.

source

supports_pooling

1 supports_pooling() -> bool

Return True if this model type supports /v1/pooling.

source

MultimodalEmbeddingCachePublisher (class)

A publisher for multimodal encode-worker cache state.

1 from dynamo.llm import MultimodalEmbeddingCachePublisher

1 MultimodalEmbeddingCachePublisher() -> None

lib/bindings/python/src/dynamo/_core.pyi#L684

Public methods

init

1 __init__() -> None

Create a MultimodalEmbeddingCachePublisher object.

source

create_endpoint

1 create_endpoint(endpoint: Endpoint) -> None

Initialize event-plane publishing for multimodal cache state.

Parameters

endpoint

Endpoint

The endpoint to extract component information from.

source

publish_delta

1 publish_delta(added_keys: list[str], removed_keys: list[str]) -> None

Publish an incremental cache mutation for this worker.

Parameters

added_keys

list[str]

Newly cached embedding keys.

removed_keys

list[str]

Cache keys no longer present on the worker.

source

OverlapScores (class)

A collection of prefix matching scores of workers for a given token ids. ‘scores’ is a map of worker id to the score which is the number of matching blocks.

1 from dynamo.llm import OverlapScores

lib/bindings/python/src/dynamo/_core.pyi#L934

PythonAsyncEngine (class)

Bridge a Python async generator onto Dynamo’s AsyncEngine interface.

1 from dynamo.llm import PythonAsyncEngine

1 PythonAsyncEngine(generator: Any, event_loop: Any) -> None

lib/bindings/python/src/dynamo/_core.pyi#L1471

Public methods

init

1 __init__(generator: Any, event_loop: Any) -> None

Wrap a Python generator and event loop for use with Dynamo services.

source

RadixTree (class)

A RadixTree that tracks KV cache blocks and can find prefix matches for sequences.

1 from dynamo.llm import RadixTree

1 RadixTree() -> None

Thread-safe: operations route to a dedicated background thread and long calls release the Python GIL.

lib/bindings/python/src/dynamo/_core.pyi#L961

Public methods

init

1 __init__() -> None

Create a new RadixTree instance.

source

find_matches

1 find_matches(sequence: List[int], early_exit: bool = False) -> OverlapScores

Find prefix matches for the given sequence of block hashes.

Parameters

sequence

List[int]

List of block hashes to find matches for

early_exit

bool

If True, stop searching after finding the first match

Returns

OverlapScores — OverlapScores containing worker matching scores and frequencies

source

apply_event

1 apply_event(worker_id: int, kv_cache_event_bytes: bytes) -> None

Apply a KV cache event to update the RadixTree state.

Parameters

worker_id

int

ID of the worker that generated the event

kv_cache_event_bytes

bytes

Serialized KV cache event as bytes

Raises

ValueError — If the event bytes cannot be deserialized

source

remove_worker

1 remove_worker(worker_id: int) -> None

Remove all blocks associated with a specific worker.

Parameters

worker_id

int

ID of the worker to remove

source

clear_all_blocks

1 clear_all_blocks(worker_id: int) -> None

Clear all blocks for a specific worker.

Parameters

worker_id

int

ID of the worker whose blocks should be cleared

source

dump_tree_as_events

1 dump_tree_as_events() -> List[str]

Dump the current RadixTree state as a list of JSON-serialized KV cache events.

Returns

List[str] — List of JSON-serialized KV cache events as strings

source

RoutedEngine (class)

No summary available.

1 from dynamo.llm import RoutedEngine

lib/bindings/python/src/dynamo/llm/__init__.py#L65

Public methods

generate

1 generate(request: Any, **kwargs: Any) -> AsyncIterator[Any]

No summary available.

source

RouterConfig (class)

How to route the request

1 from dynamo.llm import RouterConfig

1 RouterConfig(mode: RouterMode, config: Optional[KvRouterConfig] = None, active_decode_blocks_threshold: Optional[float] = None, active_prefill_tokens_threshold: Optional[int] = None, active_prefill_tokens_threshold_frac: Optional[float] = None, enforce_disagg: bool = False, session_affinity_ttl_secs: Optional[int] = None) -> None

lib/bindings/python/src/dynamo/_core.pyi#L1693

Public methods

init

1 __init__(mode: RouterMode, config: Optional[KvRouterConfig] = None, active_decode_blocks_threshold: Optional[float] = None, active_prefill_tokens_threshold: Optional[int] = None, active_prefill_tokens_threshold_frac: Optional[float] = None, enforce_disagg: bool = False, session_affinity_ttl_secs: Optional[int] = None) -> None

Create a RouterConfig.

Parameters

mode

RouterMode

The router mode (RoundRobin, Random, KV, Direct, LeastLoaded, or DeviceAwareWeighted)

config

Optional[KvRouterConfig]

Optional KV router configuration (used when mode is KV)

active_decode_blocks_threshold

Optional[float]

Threshold percentage (0.0-1.0) for decode blocks busy detection

active_prefill_tokens_threshold

Optional[int]

Literal token count threshold for prefill busy detection

active_prefill_tokens_threshold_frac

Optional[float]

Fraction of max_num_batched_tokens for busy detection

enforce_disagg

bool

Deprecated and ignored. Routing topology and readiness come from registered worker types.

session_affinity_ttl_secs

Optional[int]

Router-local session-affinity idle TTL in seconds.

source

RouterMode (class)

Router mode for load balancing requests across workers

1 from dynamo.llm import RouterMode

lib/bindings/python/src/dynamo/_core.pyi#L1682

RouterQueueLimitExceeded (class)

A policy-class queue cap rejected the request.

1 from dynamo.llm import RouterQueueLimitExceeded

lib/bindings/python/src/dynamo/_core.pyi#L3215

RoutingConstraints (class)

Request-side routing constraints.

1 from dynamo.llm import RoutingConstraints

1 RoutingConstraints(required_taints: Optional[Set[str]] = None, preferred_taints: Optional[Dict[str, float]] = None) -> None

required_taints is a hard eligibility filter. preferred_taints maps taint -> signed weight. Positive weights prefer matching workers, negative weights avoid them, and 0.0 is neutral. Matching weights are summed and squashed with tanh, so opposite preferences cancel before Dynamo converts the bounded bias into a strictly positive score multiplier.

lib/bindings/python/src/dynamo/_core.pyi#L914

Public methods

init

1 __init__(required_taints: Optional[Set[str]] = None, preferred_taints: Optional[Dict[str, float]] = None) -> None

No summary available.

source

SelectionService (class)

In-process handle to a runtime-free Dynamo selection core.

1 from dynamo.llm import SelectionService

1 SelectionService(*, indexer_threads: int = 4, indexer_peers: Optional[list[str]] = None, replica_sync_port: Optional[int] = None, replica_sync_peers: Optional[list[str]] = None, selection_cache: Optional[SelectionCacheConfig] = None) -> None

lib/bindings/python/src/dynamo/_core.pyi#L728

Public methods

init

1 __init__(*, indexer_threads: int = 4, indexer_peers: Optional[list[str]] = None, replica_sync_port: Optional[int] = None, replica_sync_peers: Optional[list[str]] = None, selection_cache: Optional[SelectionCacheConfig] = None) -> None

Create a selection service. indexer_threads sizes the KV indexer pool.

source

shutdown

1 shutdown() -> None

Stop the service: cancel KV-event listeners and scheduling so that in-flight and queued selections fail fast.

The KV indexer thread pool is released when the handle is dropped. Idempotent, and also runs automatically on drop.

source

upsert_worker

1 upsert_worker(worker: JsonLike) -> JsonLike

Upsert a worker and subscribe to its live KV events; returns its catalog record.

source

delete_worker

1 delete_worker(worker_id: int) -> JsonLike

Remove a worker and tear down its KV-event listener; returns its catalog record.

source

list_workers

1 list_workers(*, model_name: Optional[str] = None, routing_group: Optional[str] = None) -> JsonLike

List catalog records, optionally filtered by model and routing group.

source

ready

1 ready() -> JsonLike

Readiness: whether at least one worker is schedulable, plus catalog state.

source

overlap_scores

1 overlap_scores(request: JsonLike) -> JsonLike

Per-worker KV-overlap scores for a prompt.

source

select

1 select(request: JsonLike) -> JsonLike

Select the best worker by KV-overlap + load, without booking.

source

select_and_reserve

1 select_and_reserve(request: JsonLike) -> JsonLike

Select the best worker and book its load.

source

create_reservation

1 create_reservation(request: JsonLike) -> JsonLike

Book a request’s load against a worker, keyed by selection_id.

Without a worker_id, replays the matching select’s cached selection (same model/routing-group), booked under selection_id; other request fields are ignored. With a worker_id and the prompt, books explicitly under selection_id on that worker and discards any cached selection for the id. selection_id is required.

source

prefill_complete

1 prefill_complete(selection_id: str) -> None

Mark a reservation’s prefill complete; its load shifts prefill -> decode.

source

add_output_block

1 add_output_block(selection_id: str, *, decay_fraction: Optional[float] = None) -> None

Record one decode output block for a reservation, advancing its decode load.

source

free_reservation

1 free_reservation(selection_id: str) -> None

Free a finished reservation, releasing its tracked load.

source

loads

1 loads(*, model_name: Optional[str] = None, routing_group: Optional[str] = None) -> JsonLike

Current per-model active load (pending counts + per-worker potential loads).

source

potential_loads

1 potential_loads(request: JsonLike) -> JsonLike

Per-worker potential loads for a prompt, without booking.

source

WorkerMetricsPublisher (class)

A metrics publisher will provide metrics to the router for load monitoring.

1 from dynamo.llm import WorkerMetricsPublisher

1 WorkerMetricsPublisher() -> None

lib/bindings/python/src/dynamo/_core.pyi#L645

Public methods

init

1 __init__() -> None

Create a WorkerMetricsPublisher object

source

create_endpoint

1 create_endpoint(endpoint: Endpoint) -> None

Initialize event-plane publishing for worker metrics. Must be awaited.

Extracts component information from the endpoint to set up metrics publishing on the endpoint-scoped event subject used for routing decisions.

Parameters

endpoint

Endpoint

The endpoint to extract component information from for metrics publishing

source

publish

1 publish(dp_rank: Optional[int] = None, active_decode_blocks: int | None = None, kv_used_blocks: int | None = None) -> None

Publish worker metrics for load monitoring.

Parameters

dp_rank

Optional[int]

Data parallel rank of the worker (None defaults to 0)

active_decode_blocks

int | None

Optional scheduler-compatible decode-block signal

kv_used_blocks

int | None

Optional authoritative total KV blocks currently in use

source

WorkerType (class)

Processing stage a worker handles.

1 from dynamo.llm import WorkerType

Each worker has exactly one role; values are not combinable. Use the needs argument on register_model to express dependencies in DNF form (a list of alternative AND-sets) — for example, an encode worker that needs (Prefill AND Decode) OR a single Aggregated peer is expressed as [[WorkerType.Prefill, WorkerType.Decode], [WorkerType.Aggregated]].

lib/bindings/python/src/dynamo/_core.pyi#L2205

compute_block_hash_for_seq (function)

Compute block hashes for a sequence of tokens, optionally including multimodal metadata.

1 from dynamo.llm import compute_block_hash_for_seq

1 compute_block_hash_for_seq(tokens: List[int], kv_block_size: int, block_mm_infos: Optional[List[Optional[Dict[str, Any]]]] = None, lora_name: Optional[str] = None, is_eagle: Optional[bool] = None, cache_namespace: Optional[str] = None) -> List[int]

When block_mm_infos is provided, the mm_hashes are included in the hash computation to ensure that blocks with identical tokens but different multimodal objects produce different hashes.

Parameters

tokens

List[int]

List of token IDs

kv_block_size

int

Size of each block in tokens

block_mm_infos

Optional[List[Optional[Dict[str, Any]]]]

Optional per-block multimodal metadata. Each element corresponds to a block and should be None or a dict with structure: { “mm_objects”: [ { “mm_hash”: int, # Hash of the MM object } ] }

lora_name

Optional[str]

Optional LoRA adapter name for adapter-aware block hashing.

is_eagle

Optional[bool]

Optional Eagle mode flag. When true, hashes use overlapping kv_block_size + 1 token windows with kv_block_size stride.

Returns

List[int] — List of block hashes (one per block)

>>> tokens = [1, 2, 3, 4] * 8 # 32 tokens = 1 block >>> mm_info = { … “mm_objects”: [{ … “mm_hash”: 0xDEADBEEF, … }] … } >>> hashes = compute_block_hash_for_seq(tokens, 32, [mm_info])

lib/bindings/python/src/dynamo/_core.pyi#L396

fetch_model (function)

Download a model from Hugging Face, returning its local path. If ignore_weights is True, only fetches tokenizer and config files. Example: model_path = await fetch_model("Qwen/Qwen3-0.6B")

1 from dynamo.llm import fetch_model

1 fetch_model(remote_name: str, ignore_weights: bool = False) -> str

lib/bindings/python/src/dynamo/_core.pyi#L2335

lora_name_to_id (function)

Generate a deterministic integer ID from a LoRA name using blake3 hash.

1 from dynamo.llm import lora_name_to_id

1 lora_name_to_id(lora_name: str) -> int

lib/bindings/python/src/dynamo/_core.pyi#L2293

make_engine (function)

Make an engine matching the args

1 from dynamo.llm import make_engine

1 make_engine(distributed_runtime: DistributedRuntime, args: EntrypointArgs) -> EngineConfig

lib/bindings/python/src/dynamo/_core.pyi#L2352

register_model (function)

Attach the model at path to the given endpoint, and advertise it as model_type. LoRA Registration: The lora_name and base_model_path parameters must be provided together or not at all. Providing only one of these parameters will raise a ValueError. - lora_name: The served model name for the LoRA model - base_model_path: Path to the base model that the LoRA extends

1 from dynamo.llm import register_model

1 register_model(model_input: ModelInput, model_type: ModelType, endpoint: Endpoint, model_path: str, model_name: Optional[str] = None, *, worker_type: WorkerType, kv_cache_block_size: Optional[int] = None, router_mode: Optional[RouterMode] = None, runtime_config: Optional[ModelRuntimeConfig] = None, tensor_model_config: Optional[Dict[str, Any]] = None, user_data: Optional[Dict[str, Any]] = None, custom_template_path: Optional[str] = None, media_decoder: Optional[MediaDecoder] = None, media_fetcher: Optional[MediaFetcher] = None, lora_name: Optional[str] = None, base_model_path: Optional[str] = None, needs: Optional[List[List[WorkerType]]] = None, self_host_metadata: Optional[bool] = None, ignore_weights: bool = False, max_gpu_lora_count: Optional[int] = None, model_aliases: Optional[List[str]] = None) -> None

For TensorBased models (using ModelInput.Tensor), HuggingFace downloads are skipped and a minimal model card is registered directly. Use model_path as the display name for these models. Pass tensor protocol metadata through tensor_model_config.

Model serving readiness: worker_type and needs describe the worker’s processing stage and peer dependencies. needs is a DNF list — each inner list is an AND-set, the outer list is OR. worker_type is required; backends declare it literally at each call site.

When ignore_weights is true, remote HuggingFace model resolution skips weight files and downloads only the metadata needed for registration.

lib/bindings/python/src/dynamo/_core.pyi#L2224

run_input (function)

Start an engine, connect it to an input, and run until stopped.

1 from dynamo.llm import run_input

1 run_input(distributed_runtime: DistributedRuntime, input: str, engine_config: EngineConfig, frontend_route_extensions: Optional[Sequence[FrontendRoute]] = None) -> None

frontend_route_extensions supplies additional HTTP routes to the frontend (HTTP input only); see FrontendRoute.

lib/bindings/python/src/dynamo/_core.pyi#L2418

run_kv_indexer (function)

Run the KV indexer with the given arguments.

1 from dynamo.llm import run_kv_indexer

1 run_kv_indexer(args: List[str]) -> None

lib/bindings/python/src/dynamo/_core.pyi#L41

run_select_service (function)

Run the Dynamo selection service with the given arguments.

1 from dynamo.llm import run_select_service

1 run_select_service(args: List[str]) -> None

lib/bindings/python/src/dynamo/_core.pyi#L49

run_slot_tracker (function)

Run the KV router slot tracker with the given arguments.

1 from dynamo.llm import run_slot_tracker

1 run_slot_tracker(args: List[str]) -> None

lib/bindings/python/src/dynamo/_core.pyi#L45

unregister_model (function)

Unregister a model from the discovery system.

1 from dynamo.llm import unregister_model

1 unregister_model(endpoint: Endpoint, lora_name: Optional[str] = None) -> None

If lora_name is provided, unregisters a LoRA adapter instead of a base model.

lib/bindings/python/src/dynamo/_core.pyi#L2271

update_model_taints (function)

Replace caller-managed taints on this worker’s registered model.

1 from dynamo.llm import update_model_taints

1 update_model_taints(endpoint: Endpoint, taints: Set[str]) -> None

Reserved ‘dynamo.topology/’ taints are derived from the model’s topology metadata and cannot be supplied by callers.

lib/bindings/python/src/dynamo/_core.pyi#L2282