`core.inference.inference_request`#

Module Contents#

Classes#

`_PreparedMultimodalData`	Multimodal wire data whose content identity has already been computed.
`Status`	Enum for status
`InferenceRequest`	Class for one inference request
`DynamicInferenceEventType`	Dynamic inference event type.
`DynamicInferenceEvent`	A lifecycle event for a dynamic inference requests.
`DynamicInferenceRequest`	Class for one inference request
`DynamicInferenceRequestRecord`	Internal engine history across request checkpoints.
`FinishedRequestRecord`	Stores per-request metadata that is not meant to be passed through the RESTful server.
`OffloadedRequestPayload`	A finished request’s per-token payload, kept off the RESTful reply by payload offload.
`RequestPayloadStageResult`	A custody acknowledgement and opaque metadata for the served response.
`RequestPayloadStager`	Protocol to handle offloading of request payloads to a storage backend.
`RequestPromptPreparationResult`	The resolved engine prompt and opaque parameters that describe it.
`RequestPromptPreparer`	Protocol for resolving an exact prompt before engine admission.
`VLMInferenceRequest`	Class for a VLM inference request
`DynamicVLMInferenceRequest`	Dynamic inference request for VLM models.

Functions#

`serialize_tensor`	Serialize a tensor as contiguous binary data.
`deserialize_tensor`	Deserialize binary tensor data or the legacy nested-list representation.
`_normalize_raw_media_items`	Normalize supported raw-media inputs, or return None for preprocessed data.
`_media_tensor_keys`	Return the tensor fields that define a preprocessed media input.
`compute_media_cache_key`	Return a stable content key for raw or preprocessed media.
`prepare_multimodal_data`	Serialize and hash media once for reuse across equivalent submissions.
`serialize_multimodal_data`	Serialize one request’s vLLM-style multimodal dictionary.
`split_multimodal_data`	Split serialized media into a bounded descriptor and its payload.
`merge_multimodal_data`	Rebuild serialized media from its descriptor and payload frames.
`resolve_multimodal_data_for_engine`	Resolve wire-format multimodal data into dynamic-engine arguments.
`serialize_ndarray`	Serialize numpy array to a JSON-compatible dict.
`deserialize_ndarray`	Deserialize numpy array from dict.
`unwrap_serialized_tensors`	Unwrap serialized tensor tuples produced by serialize() into plain lists.
`compute_block_hashes_batched`	Compute SHA-256 based hashes for all complete blocks in a prompt.

Data#

`PREFIX_TEMPLATE_TOKEN_IDS_FIELD`
`PREFIX_EOS_TOKEN_ID_FIELD`

API#

core.inference.inference_request.serialize_tensor(tensor: torch.Tensor) → Dict[str, Any]#

Serialize a tensor as contiguous binary data.

Parameters:: tensor (Tensor) – Tensor.
Returns:: Dictionary containing dtype, shape, and raw bytes.

core.inference.inference_request.deserialize_tensor(tensor_data: Any) → torch.Tensor#

Deserialize binary tensor data or the legacy nested-list representation.

Parameters:: tensor_data – Binary tensor dictionary or legacy nested list.
Returns:: (Tensor) Tensor.

core.inference.inference_request._normalize_raw_media_items( modality_data: Any, ) → Optional[List[bytes]]#: Normalize supported raw-media inputs, or return None for preprocessed data.

core.inference.inference_request._media_tensor_keys(modality: str) → Tuple[str, ...]#: Return the tensor fields that define a preprocessed media input.

core.inference.inference_request.compute_media_cache_key( modality: str, modality_data: Any, ) → str#

Return a stable content key for raw or preprocessed media.

The key is generated inside the inference stack so callers do not need to understand vision-embedding cache identity. Tensor metadata is included to prevent equal byte streams with different shapes or dtypes from colliding.

class core.inference.inference_request._PreparedMultimodalData#

Multimodal wire data whose content identity has already been computed.

serialized: Dict[str, Any]#: None

core.inference.inference_request.prepare_multimodal_data( multi_modal_data: Any, ) → Optional[core.inference.inference_request._PreparedMultimodalData]#: Serialize and hash media once for reuse across equivalent submissions.

core.inference.inference_request.serialize_multimodal_data( multi_modal_data: Any, ) → Optional[Dict[str, Any]]#

Serialize one request’s vLLM-style multimodal dictionary.

Supported modalities:

Images: "image" accepts raw image bytes, a list of raw image bytes, or a preprocessed tensor dictionary containing imgs / imgs_sizes or imgs / num_tiles. Video: "video" accepts raw video bytes, a list of raw video bytes, or a preprocessed tensor dictionary containing imgs, imgs_sizes, and num_frames. Audio: Audio does not yet have any supported data preprocessing or modeling formats.

core.inference.inference_request.split_multimodal_data( serialized: Optional[Dict[str, Any]], ) → Tuple[Optional[Dict[str, Any]], Any]#

Split serialized media into a bounded descriptor and its payload.

The descriptor rides in the metadata frame, which the coordinator decodes and repacks for every request; the payload rides in a body frame it forwards untouched. Keeping them apart is what bounds that per-request cost: the descriptor is a 64-character key plus two small flags, while the payload is raw image or video bytes, or serialized preprocessed tensors.

Parameters:: serialized – The output of :func:serialize_multimodal_data, or None.
Returns:: (media_meta, payload), both None for a text-only request.

core.inference.inference_request.merge_multimodal_data( media_meta: Optional[Dict[str, Any]], payload: Any, ) → Optional[Dict[str, Any]]#

Rebuild serialized media from its descriptor and payload frames.

Inverse of :func:split_multimodal_data, returning the shape

Func:

resolve_multimodal_data_for_engine consumes.

Parameters:

media_meta – The bounded descriptor from the metadata frame, or None.
payload – The media payload from its own body frame.

Returns:

The serialized multimodal dictionary, or None for a text-only request.

core.inference.inference_request.resolve_multimodal_data_for_engine( multi_modal_data: Any, *, image_preprocessing_config: Optional[megatron.core.inference.config.ImageProcessingConfig] = None, video_preprocessing_config: Optional[megatron.core.inference.config.VideoProcessingConfig] = None, ) → Dict[str, Any]#

Resolve wire-format multimodal data into dynamic-engine arguments.

Supported modalities:

Images: Raw image bytes are preprocessed into model inputs. Serialized or in-process preprocessed image tensor dictionaries are passed through as dynamic-engine image arguments. Video: Raw video bytes are decoded and sampled into model inputs. Serialized or in-process preprocessed tensor dictionaries are passed through. Audio: Audio does not yet have any supported data preprocessing or modeling formats.

core.inference.inference_request.serialize_ndarray(arr: numpy.ndarray) → dict#: Serialize numpy array to a JSON-compatible dict.

core.inference.inference_request.deserialize_ndarray(obj: dict) → numpy.ndarray#: Deserialize numpy array from dict.

core.inference.inference_request.unwrap_serialized_tensors(serialized_request: dict) → dict#

Unwrap serialized tensor tuples produced by serialize() into plain lists.

Parameters:: serialized_request (dict) – A dict produced by serialize().
Returns:: A shallow copy with tensor wrapper tuples replaced by plain lists.
Return type:: dict

class core.inference.inference_request.Status(*args, **kwds)#

Bases: enum.Enum

Enum for status

Initialization

WAITING_IN_QUEUE#: 1

ACTIVE_AND_GENERATING_TOKENS#: 2

ACTIVE_BUT_NOT_GENERATING_TOKENS#: 3

COMPLETED#: 4

FAILED#: 5

core.inference.inference_request.compute_block_hashes_batched( prompt_tokens: torch.Tensor, block_size: int, cache_salt: Optional[str] = None, ) → List[int]#

Compute SHA-256 based hashes for all complete blocks in a prompt.

Each block hash is computed as SHA-256(parent_digest || block_bytes), where parent_digest chains from the previous block (starting from a zero digest). This provides cryptographic collision resistance with no exploitable algebraic structure.

Parameters:

prompt_tokens – All prompt token IDs, shape [seq_len].
block_size – Number of tokens per block.
cache_salt – Optional request-input identity mixed into every chained block hash. Multimodal requests use their generated media key so equal token placeholders backed by different media cannot share KV.

Returns:

List of positive integer hash values in [1, 2^63-1], one per complete block.

class core.inference.inference_request.InferenceRequest#

Class for one inference request

Containing relevant data for an inference request

request_id: int#: None

prompt: str#: None

sampling_params: Optional[megatron.core.inference.sampling_params.SamplingParams]#: None

inference_parameters: Optional[megatron.core.inference.sampling_params.SamplingParams]#: None

prompt_tokens: Optional[List[int]]#: None

prompt_length: Optional[int]#: None

arrival_time: Optional[float]#: None

status: Optional[core.inference.inference_request.Status]#: None

encoder_prompt: Optional[str]#: None

generated_text: Optional[str]#: None

segments: Optional[List[str]]#: None

generated_segments: Optional[List[str]]#: None

generated_sequence_lengths: Optional[List[int]]#: None

generated_tokens: Optional[torch.Tensor]#: None

prompt_log_probs: Optional[torch.Tensor]#: None

generated_log_probs: Optional[torch.Tensor]#: None

prompt_top_n_logprobs: Optional[List[Dict[str, float]]]#: None

generated_top_n_logprobs: Optional[List[Dict[str, float]]]#: None

generated_length: Optional[int]#: None

tpot: List[float]#: ‘field(…)’

__post_init__()#

serialize() → dict#

Converts the instance into a serializable dictionary.

Returns:: (dict) A dictionary representation of the instance suitable for serialization.

classmethod deserialize( obj: dict, ) → core.inference.inference_request.InferenceRequest#

Deserialize request.

Parameters:: obj (dict) – Serialized request data.
Returns:: (InferenceRequest) Deserialized request.

_post_deserialize(obj: dict)#: This is called after the dataclass is initialized to handle any special deserialization logic.

class core.inference.inference_request.DynamicInferenceEventType(*args, **kwds)#

Bases: enum.Enum

Dynamic inference event type.

Initialization

ADD_ENGINE#: ‘auto(…)’

ADD_CONTEXT#: ‘auto(…)’

GENERATED_TOKEN#: ‘auto(…)’

PAUSE#: ‘auto(…)’

EVICT#: ‘auto(…)’

FINISH#: ‘auto(…)’

FAIL#: ‘auto(…)’

ERROR_TRANSIENT#: ‘auto(…)’

ERROR_NONTRANSIENT#: ‘auto(…)’

class core.inference.inference_request.DynamicInferenceEvent#

A lifecycle event for a dynamic inference requests.

An event is currently one of the following:

request added
request paused
request evicted
request finished
request failed
request error (transient)
request error (non-transient, i.e. fatal)

timestamp: Optional[float]#: None

type: core.inference.inference_request.DynamicInferenceEventType#: None

payload: Optional[Any]#: None

__post_init__()#

__str__()#

serialize() → dict#

Converts the instance into a serializable dictionary.

Returns:: Full event dict.
Return type:: dict

classmethod deserialize( obj: dict, ) → core.inference.inference_request.DynamicInferenceEvent#

Deserialize event.

Parameters:: obj – Serialized event data dict.
Returns:: (DynamicInferenceEvent) Deserialized event.

class core.inference.inference_request.DynamicInferenceRequest#

Bases: core.inference.inference_request.InferenceRequest

Class for one inference request

Containing relevant data for an dynamic inference request

request_id: int#: None

uid: str#: ‘field(…)’

prompt: Optional[str]#: None

prompt_tokens: Optional[torch.Tensor]#: None

compact_prompt_tokens: Optional[torch.Tensor]#: None

offload_params: Optional[Dict[str, Any]]#: None

remaining_prompt_tokens: Optional[torch.Tensor]#: None

policy_epoch: Optional[list[tuple[int, int]]]#: None

kv_cache_epoch: Optional[list[tuple[int, int]]]#: None

latency: Optional[float]#: None

routing_indices: Optional[numpy.ndarray]#: None

finished_chunk_token_count: int#: 0

stop_word_ids: Optional[List[List[int]]]#: None

cg_wait_iters: int#: 0

block_size_tokens: Optional[int]#: None

enable_prefix_caching: bool#: False

num_cached_tokens: int#: 0

num_matched_prefix_blocks: int#: 0

mtp_private_suffix_start: Optional[int]#: None

block_hash_salt: Optional[str]#: None

precomputed_block_hashes: List[int]#: ‘field(…)’

disaggregated_params: Optional[dict]#: None

payload_offloaded: bool#: False

payload_stage_metadata: Dict[str, Any]#: ‘field(…)’

__post_init__()#

_compute_block_hashes() → None#

Compute hashes for all complete blocks in the prompt.

After this call:

precomputed_block_hashes is [] if prompt < block_size (no complete blocks)
precomputed_block_hashes is [hash1, …] for N complete blocks

property remaining_prompt_length#: Get the length of the remaining prompt tokens.

ttft: Optional[float]#: None

events: List[core.inference.inference_request.DynamicInferenceEvent]#: ‘field(…)’

event_add_engine: Optional[core.inference.inference_request.DynamicInferenceEvent]#: ‘field(…)’

generated_tokens: List[int]#: ‘field(…)’

acceptance_step_lengths: List[int]#: ‘field(…)’

finalize_text( tokenizer: Any, ) → core.inference.inference_request.DynamicInferenceRequest#

Populate generated text by decoding the complete generated token stream.

Parameters:: tokenizer – Tokenizer used to decode generated_tokens.
Returns:: This request, with generated_text populated.
Raises:: ValueError – If tokenizer is None.

__str__()#

serialize( payload_offloaded: bool = False, payload_stage_metadata: Optional[Dict[str, Any]] = None, )#

Converts the instance into a serializable dictionary.

Parameters:

payload_offloaded (bool) – Drop the per-token payload (log probs, MoE routing indices) from the wire; the engine’s RequestPayloadStager has custody of it.
payload_stage_metadata – Opaque metadata returned by the stager for the REST endpoint to attach to its response.

Returns:

(dict) A dictionary representation of the instance suitable for serialization.

_post_deserialize(obj)#

property tracked_metadata: List[Any]#

Obtain an ordered list of all request metadata to be tracked by the context.

This consists of metadata that is used to inform text generation. The values of such fields are tensorized and kept aligned with the current active batch.

Note that while the general request object is mutable, this metadata is inherently assumed to remain immutable once the request becomes active.

static get_metadata_types() → List[Tuple[str, torch.dtype]]#

Keeps track of all request metadata names and dtypes.

Returns:: Mapping from metadata name to: name (str) - The name of the metadata field. dtype (torch.dtype) - The datatype of the metadata.
Return type:: List[Tuple[str, torch.dtype]]

add_event( type: core.inference.inference_request.DynamicInferenceEventType, payload: Optional[Any] = None, ) → core.inference.inference_request.DynamicInferenceEvent#: Add event.

add_event_add_engine()#: Add ‘add_engine’ event - called when request enters the engine queue.

add_event_add_context()#: Add ‘add_context’ event - called when request is added to context for prefill.

add_event_generated_token( token: int, blocks_total: Optional[int] = None, blocks_hashed_total: Optional[int] = None, blocks_hashed_active: Optional[int] = None, blocks_ref_count: Optional[int] = None, pre_fwd_active_token_count: Optional[int] = None, pre_fwd_step_count: Optional[int] = None, )#

Add ‘generated_token’ event - records each generated token.

Parameters:

token (int) – The token ID that was generated.
blocks_total (int) – Total block capacity from allocator.
blocks_hashed_total (int) – All allocated (hashed) blocks.
blocks_hashed_active (int) – Blocks with ref_count > 0.
blocks_ref_count (int) – Sum of block ref counts from allocator.
pre_fwd_active_token_count (int) – Active token count before forward pass.
pre_fwd_step_count (int) – Step count before forward pass.

add_event_pause()#: Add ‘pause’ event.

add_event_evict()#: Add ‘evict’ event.

add_event_finish()#: Add ‘finish’ event.

add_event_fail()#: Add ‘fail’ event.

add_event_error_transient(error: Exception)#: Add transient error event.

add_event_error_nontransient(error: Exception)#: Add non-transient error event.

succeeded() → bool#: Request experienced no non-transient errors.

failed() → bool#: Request experienced non-transient error.

class core.inference.inference_request.DynamicInferenceRequestRecord#

Internal engine history across request checkpoints.

requests: list[core.inference.inference_request.DynamicInferenceRequest]#: ‘field(…)’

latency: Optional[float]#: None

classmethod from_request( request: core.inference.inference_request.DynamicInferenceRequest, ) → core.inference.inference_request.DynamicInferenceRequestRecord#

Initialize record from a single request.

Parameters:: request (DynamicInferenceRequest) – Initial request.
Returns:: (DynamicInferenceRequestRecord) A record.

__getitem__( idx: int, ) → core.inference.inference_request.DynamicInferenceRequest#

Get request by index.

Parameters:: idx (int) – Request index.
Returns:: (DynamicInferenceRequest) Request object.

property request_id: int#

Get request id.

Returns:: (int) Request id.

checkpoint() → None#: Maintain reference to previous request, and then append a new request that concatenates the previous prompt and generations.

merge() → core.inference.inference_request.DynamicInferenceRequest#

Merge requests into a single checkpoint-agnostic request object.

Returns:: (DynamicInferenceRequest) Merged request.

class core.inference.inference_request.FinishedRequestRecord#

Stores per-request metadata that is not meant to be passed through the RESTful server.

policy_epoch: Optional[list[tuple[int, int]]]#: None

kv_cache_epoch: Optional[list[tuple[int, int]]]#: None

num_evictions: int#: None

classmethod from_request( request: core.inference.inference_request.DynamicInferenceRequest, ) → core.inference.inference_request.FinishedRequestRecord#: Build the request’s non-RESTful metadata from a finished request.

class core.inference.inference_request.OffloadedRequestPayload#

A finished request’s per-token payload, kept off the RESTful reply by payload offload.

Self-contained: a consumer can rebuild the served sequence from it alone, keyed by the request uid (the OpenAI response id).

prompt_token_ids: Optional[list[int]]#: None

generated_token_ids: list[int]#: None

generated_log_probs: Optional[list[float]]#: None

prompt_log_probs: Optional[list[float]]#: None

routing_indices: Optional[numpy.ndarray]#: None

classmethod from_request( request: core.inference.inference_request.DynamicInferenceRequest, ) → core.inference.inference_request.OffloadedRequestPayload#

Copy the payload off a finished (merged) request as plain host-side values.

Side effect: replaces request.prompt_tokens with its host copy when it is a tensor, so one device-to-host copy serves both this payload and every later host-side read of the finished request (its reply and the caller’s result).

class core.inference.inference_request.RequestPayloadStageResult#

A custody acknowledgement and opaque metadata for the served response.

response_metadata: Dict[str, Any]#: ‘field(…)’

class core.inference.inference_request.RequestPayloadStager#

Bases: typing.Protocol

Protocol to handle offloading of request payloads to a storage backend.

stage( uid: str, payload: core.inference.inference_request.OffloadedRequestPayload, *, finished_metadata: core.inference.inference_request.FinishedRequestRecord, offload_params: Optional[Dict[str, Any]] = None, ) → Optional[core.inference.inference_request.RequestPayloadStageResult]#: Stage a payload, or return None to keep it on the normal reply path.

core.inference.inference_request.PREFIX_TEMPLATE_TOKEN_IDS_FIELD#: ‘template_prefix_token_ids’

core.inference.inference_request.PREFIX_EOS_TOKEN_ID_FIELD#: ‘eos_token_id’

class core.inference.inference_request.RequestPromptPreparationResult#

The resolved engine prompt and opaque parameters that describe it.

prompt: Union[str, List[int], torch.Tensor]#: None

offload_params: Optional[Dict[str, Any]]#: None

class core.inference.inference_request.RequestPromptPreparer#

Bases: typing.Protocol

Protocol for resolving an exact prompt before engine admission.

prepare_prompt( prompt: Union[str, List[int], torch.Tensor], *, offload_params: Optional[Dict[str, Any]] = None, ) → core.inference.inference_request.RequestPromptPreparationResult#: Return the engine prompt and metadata that describe that prompt.

class core.inference.inference_request.VLMInferenceRequest#

Bases: core.inference.inference_request.InferenceRequest

Class for a VLM inference request

num_img_embeddings_per_tile: int#: None

imgs: torch.Tensor#: None

num_tiles: torch.Tensor#: None

decoder_seq_length: int#: None

class core.inference.inference_request.DynamicVLMInferenceRequest#

Bases: core.inference.inference_request.DynamicInferenceRequest, core.inference.inference_request.VLMInferenceRequest

Dynamic inference request for VLM models.

Combines DynamicInferenceRequest (for dynamic batching) with VLMInferenceRequest (for multimodal fields). Also stores pre-computed image embeddings and the image token mask produced by expand_image_tokens.

image_embeddings: Optional[torch.Tensor]#: None

image_token_mask: Optional[torch.Tensor]#: None

imgs_sizes: Optional[torch.Tensor]#: None

num_frames: Optional[torch.Tensor]#: None

media_tokens_preexpanded: bool#: False

media_cache_key: Optional[str]#: None

core.inference.inference_request#

Module Contents#

Classes#

Functions#

Data#

API#

`core.inference.inference_request`#