core.inference.engine_endpoint

Typed description of a running dynamic-inference engine.

Module Contents

Classes

InferenceEngineCapabilities	Static capabilities needed by an external inference control plane.
InferenceEngineEndpoint	Address and capabilities of an already-running inference engine.

API

class core.inference.engine_endpoint.InferenceEngineCapabilities

Static capabilities needed by an external inference control plane.

This record deliberately contains no framework-specific fields. A serving integration can translate it to its own registration type without reaching into DynamicInferenceContext or depending on Megatron’s CLI package.

context_length: int

None

kv_cache_block_size: int

None

total_kv_blocks: int

None

max_num_seqs: int

None

max_num_batched_tokens: int

None

bos_token_id: int

None

enable_prefix_caching: bool

None

logical_data_parallel_size: int

1

prefix_caching_coordinator_policy: megatron.core.inference.config.PrefixCachingCoordinatorPolicy | None

None

Routing policy for client-side hashing; None supports older endpoint records.

__post_init__() → None

classmethod from_engine(

engine: megatron.core.inference.engines.dynamic_engine.DynamicInferenceEngine,

*,

logical_data_parallel_size: int = 1,

) → core.inference.engine_endpoint.InferenceEngineCapabilities

Inspect a constructed dynamic engine once at registration time.

classmethod from_dict(

value: collections.abc.Mapping[str, Any],

) → core.inference.engine_endpoint.InferenceEngineCapabilities

Deserialize a capabilities mapping received across a process boundary.

to_dict() → dict[str, int | bool | str | None]

Return a serialization-friendly representation.

class core.inference.engine_endpoint.InferenceEngineEndpoint

Address and capabilities of an already-running inference engine.

coordinator_address: str

None

capabilities: core.inference.engine_endpoint.InferenceEngineCapabilities

None

__post_init__() → None

classmethod from_engine(

coordinator_address: str,

engine: megatron.core.inference.engines.dynamic_engine.DynamicInferenceEngine,

*,

logical_data_parallel_size: int = 1,

) → core.inference.engine_endpoint.InferenceEngineEndpoint

Describe a running engine without transferring ownership of it.

classmethod from_dict(

value: collections.abc.Mapping[str, Any],

) → core.inference.engine_endpoint.InferenceEngineEndpoint

Deserialize an endpoint received across a process boundary.

to_dict() → dict[str, Any]

Return a serialization-friendly representation.