`core.inference.disaggregation.pending_handoff_imports`#

State records for KV-cache and SSM-state imports awaiting completion.

Module Contents#

Classes#

`HandoffRequestMetadata`	Request identity and RL history carried across the internal prefill/decode hop.
`DeferredKvHandoff`	Decode handoff waiting for local cache capacity before transfer starts.
`PendingSSMImport`	Exact SSM state being transferred into a reserved live request slot.
`PendingKvImport`	Decode request waiting for an asynchronous KV-cache import.

API#

class core.inference.disaggregation.pending_handoff_imports.HandoffRequestMetadata#

Request identity and RL history carried across the internal prefill/decode hop.

uid: str#: None

policy_epoch: list[tuple[int, int]] | None#: None

kv_cache_epoch: list[tuple[int, int]] | None#: None

eviction_timestamps: list[float]#: None

classmethod from_request( request: megatron.core.inference.inference_request.DynamicInferenceRequest, ) → core.inference.disaggregation.pending_handoff_imports.HandoffRequestMetadata#: Capture the merged source history, including prefill evictions.

restore( request: megatron.core.inference.inference_request.DynamicInferenceRequest, generation_epoch: int | None, ) → None#: Preserve source history; decode owns positions after the source’s sampled token.

class core.inference.disaggregation.pending_handoff_imports.DeferredKvHandoff#

Decode handoff waiting for local cache capacity before transfer starts.

request_id: int#: None

prompt: list#: None

sampling_params: megatron.core.inference.sampling_params.SamplingParams#: None

kv_meta: dict#: None

resume_tokens: List[int]#: None

resume_log_probs: List[float]#: None

src_block_ids: List[int]#: None

hashes: List[int]#: None

num_blocks: int#: None

future: asyncio.Future#: None

offload_params: dict | None#: None

request_metadata: core.inference.disaggregation.pending_handoff_imports.HandoffRequestMetadata | None#: None

class core.inference.disaggregation.pending_handoff_imports.PendingSSMImport#

Exact SSM state being transferred into a reserved live request slot.

handles: list[Any]#: None

live_slot: int#: None

class core.inference.disaggregation.pending_handoff_imports.PendingKvImport#

Decode request waiting for an asynchronous KV-cache import.

request_id: int#: None

prompt: list#: None

sampling_params: megatron.core.inference.sampling_params.SamplingParams#: None

local_blocks: List[int]#: None

hashes: List[int]#: None

cached_prefix_block_count: int#: None

handle: Any#: None

future: asyncio.Future#: None

ssm: core.inference.disaggregation.pending_handoff_imports.PendingSSMImport | None#: None

block_hash_salt: str | None#: None

request_metadata: core.inference.disaggregation.pending_handoff_imports.HandoffRequestMetadata | None#: None

offload_params: dict | None#: None

resume_tokens: List[int]#: ‘field(…)’

resume_log_probs: List[float]#: ‘field(…)’

continuation_blocks: List[int]#: ‘field(…)’

local_error: Exception | None#: None

destinations_safe: bool#: True

terminal_state_reported: bool#: False

cancel_requested: bool#: False

core.inference.disaggregation.pending_handoff_imports#

Module Contents#

Classes#

API#

`core.inference.disaggregation.pending_handoff_imports`#