core.inference.disaggregation.engine#
Dynamic inference engine composed with the KV/state hand-off behavior.
Module Contents#
Classes#
Dynamic inference engine with prefill/decode state hand-off support. |
API#
- class core.inference.disaggregation.engine.DisaggDynamicInferenceEngine(
- controller: megatron.core.inference.text_generation_controllers.text_generation_controller.TextGenerationController,
- context: megatron.core.inference.contexts.dynamic_context.DynamicInferenceContext,
Bases:
megatron.core.inference.disaggregation.inference_state_handoff.InferenceStateHandoffMixin,megatron.core.inference.engines.dynamic_engine.DynamicInferenceEngineDynamic inference engine with prefill/decode state hand-off support.
Used by both control planes: the Dynamo integration and the coordinator-native 2-hop mode.
Initialization