`core.inference.inference_request`#

Module Contents#

Classes#

`_PreparedMultimodalData`	Multimodal wire data whose content identity has already been computed.
`Status`	Enum for status
`InferenceRequest`	Class for one inference request
`DynamicInferenceEventType`	Dynamic inference event type.
`DynamicInferenceEvent`	A lifecycle event for a dynamic inference requests.
`DynamicInferenceRequest`	Class for one inference request
`DynamicInferenceRequestRecord`	Internal engine history across request checkpoints.
`FinishedRequestRecord`	Per-request metadata, outside of the RESTful reply, used by the stager and snapshots.
`OffloadedRequestPayload`	A finished request’s per-token payload, kept off the RESTful reply by payload offload.
`RequestPayloadStageResult`	A custody acknowledgement and opaque metadata for the served response.
`RequestPayloadStager`	Protocol to handle offloading of request payloads to a storage backend.
`LiveRequestSnapshot`	Snapshot of a live request, allowing for its later resumption.
`ResumedGeneration`	Generated tokens a request already holds when it is admitted.
`RequestPromptPreparationResult`	The resolved engine prompt and opaque parameters that describe it.
`RequestPromptPreparer`	Protocol for resolving an exact prompt before engine admission.
`VLMInferenceRequest`	Class for a VLM inference request
`DynamicVLMInferenceRequest`	Dynamic inference request for VLM models.

Functions#

`serialize_tensor`	Serialize a tensor as contiguous binary data.
`deserialize_tensor`	Deserialize binary tensor data or the legacy nested-list representation.
`_normalize_raw_media_items`	Normalize supported raw-media inputs, or return None for preprocessed data.
`_media_tensor_keys`	Return the tensor fields that define a preprocessed media input.
`compute_media_cache_key`	Return a stable content key for raw or preprocessed media.
`prepare_multimodal_data`	Serialize and hash media once for reuse across equivalent submissions.
`serialize_multimodal_data`	Serialize one request’s vLLM-style multimodal dictionary.
`split_multimodal_data`	Split serialized media into a bounded descriptor and its payload.
`merge_multimodal_data`	Rebuild serialized media from its descriptor and payload frames.
`resolve_multimodal_data_for_engine`	Resolve wire-format multimodal data into dynamic-engine arguments.
`serialize_ndarray`	Serialize numpy array to a JSON-compatible dict.
`deserialize_ndarray`	Deserialize numpy array from dict.
`_merge_routing`	Combine the routing of a sequence’s segments, each recorded from the sequence’s first row.
`unwrap_serialized_tensors`	Unwrap serialized tensor tuples produced by serialize() into plain lists.
`compute_block_hashes_batched`	Compute SHA-256 based hashes for all complete blocks in a prompt.

Data#

`PREFIX_TEMPLATE_TOKEN_IDS_FIELD`
`PREFIX_EOS_TOKEN_ID_FIELD`
`PREFIX_EXPANDED_TOKEN_COUNT_FIELD`
`PREFIX_MEDIA_COUNT_FIELD`
`RESUMED_GENERATION_FIELD`

API#

core.inference.inference_request.serialize_tensor(tensor: torch.Tensor) → Dict[str, Any]#

Serialize a tensor as contiguous binary data.

Parameters:: tensor (Tensor) – Tensor.
Returns:: Dictionary containing dtype, shape, and raw bytes.

core.inference.inference_request.deserialize_tensor(tensor_data: Any) → torch.Tensor#

Deserialize binary tensor data or the legacy nested-list representation.

Parameters:: tensor_data – Binary tensor dictionary or legacy nested list.
Returns:: (Tensor) Tensor.

core.inference.inference_request._normalize_raw_media_items( modality_data: Any, ) → Optional[List[bytes]]#: Normalize supported raw-media inputs, or return None for preprocessed data.

core.inference.inference_request._media_tensor_keys(modality: str) → Tuple[str, ...]#: Return the tensor fields that define a preprocessed media input.

core.inference.inference_request.compute_media_cache_key( modality: str, modality_data: Any, ) → str#

Return a stable content key for raw or preprocessed media.

The key is generated inside the inference stack so callers do not need to understand vision-embedding cache identity. Tensor metadata is included to prevent equal byte streams with different shapes or dtypes from colliding.

class core.inference.inference_request._PreparedMultimodalData#

Multimodal wire data whose content identity has already been computed.

serialized: Dict[str, Any]#: None

core.inference.inference_request.prepare_multimodal_data( multi_modal_data: Any, ) → Optional[core.inference.inference_request._PreparedMultimodalData]#: Serialize and hash media once for reuse across equivalent submissions.

core.inference.inference_request.serialize_multimodal_data( multi_modal_data: Any, ) → Optional[Dict[str, Any]]#

Serialize one request’s vLLM-style multimodal dictionary.

Supported modalities:

Images: "image" accepts raw image bytes, a list of raw image bytes, or a preprocessed tensor dictionary containing imgs / imgs_sizes or imgs / num_tiles. Video: "video" accepts raw video bytes, a list of raw video bytes, or a preprocessed tensor dictionary containing imgs, imgs_sizes, and num_frames. Preprocessed video dictionaries may also include video_frame_indices and video_fps timing metadata. Audio: Audio does not yet have any supported data preprocessing or modeling formats.

core.inference.inference_request.split_multimodal_data( serialized: Optional[Dict[str, Any]], ) → Tuple[Optional[Dict[str, Any]], Any]#

Split serialized media into a bounded descriptor and its payload.

The descriptor rides in the metadata frame, which the coordinator decodes and repacks for every request; the payload rides in a body frame it forwards untouched. Keeping them apart is what bounds that per-request cost: the descriptor is a 64-character key plus two small flags, while the payload is raw image or video bytes, or serialized preprocessed tensors.

Parameters:: serialized – The output of :func:serialize_multimodal_data, or None.
Returns:: (media_meta, payload), both None for a text-only request.

core.inference.inference_request.merge_multimodal_data( media_meta: Optional[Dict[str, Any]], payload: Any, ) → Optional[Dict[str, Any]]#

Rebuild serialized media from its descriptor and payload frames.

Inverse of :func:split_multimodal_data, returning the shape

Func:

resolve_multimodal_data_for_engine consumes.

Parameters:

media_meta – The bounded descriptor from the metadata frame, or None.
payload – The media payload from its own body frame.

Returns:

The serialized multimodal dictionary, or None for a text-only request.

core.inference.inference_request.resolve_multimodal_data_for_engine( multi_modal_data: Any, *, image_preprocessing_config: Optional[megatron.core.inference.config.ImageProcessingConfig] = None, video_preprocessing_config: Optional[megatron.core.inference.config.VideoProcessingConfig] = None, ) → Dict[str, Any]#

Resolve wire-format multimodal data into dynamic-engine arguments.

Supported modalities:

Images: Raw image bytes are preprocessed into model inputs. Serialized or in-process preprocessed image tensor dictionaries are passed through as dynamic-engine image arguments. Video: Raw video bytes are decoded and sampled into model inputs. Serialized or in-process preprocessed tensor dictionaries are passed through. Audio: Audio does not yet have any supported data preprocessing or modeling formats.

core.inference.inference_request.serialize_ndarray(arr: numpy.ndarray) → dict#: Serialize numpy array to a JSON-compatible dict.

core.inference.inference_request.deserialize_ndarray(obj: dict) → numpy.ndarray#: Deserialize numpy array from dict.

core.inference.inference_request._merge_routing( parts: List[numpy.ndarray], total_rows: int, ) → Optional[numpy.ndarray]#: Combine the routing of a sequence’s segments, each recorded from the sequence’s first row.

core.inference.inference_request.unwrap_serialized_tensors(serialized_request: dict) → dict#

Unwrap serialized tensor tuples produced by serialize() into plain lists.

Parameters:: serialized_request (dict) – A dict produced by serialize().
Returns:: A shallow copy with tensor wrapper tuples replaced by plain lists.
Return type:: dict

class core.inference.inference_request.Status(*args, **kwds)#

Bases: enum.Enum

Enum for status

Initialization

WAITING_IN_QUEUE#: 1

ACTIVE_AND_GENERATING_TOKENS#: 2

ACTIVE_BUT_NOT_GENERATING_TOKENS#: 3

COMPLETED#: 4

FAILED#: 5

core.inference.inference_request.compute_block_hashes_batched( prompt_tokens: torch.Tensor, block_size: int, cache_salt: Optional[str] = None, ) → List[int]#

Compute SHA-256 based hashes for all complete blocks in a prompt.

Each block hash is computed as SHA-256(parent_digest || block_bytes), where parent_digest chains from the previous block (starting from a zero digest). This provides cryptographic collision resistance with no exploitable algebraic structure.

Parameters:

prompt_tokens – All prompt token IDs, shape [seq_len].
block_size – Number of tokens per block.
cache_salt – Optional request-input identity mixed into every chained block hash. Multimodal requests use their generated media key so equal token placeholders backed by different media cannot share KV.

Returns:

List of positive integer hash values in [1, 2^63-1], one per complete block.

class core.inference.inference_request.InferenceRequest#

Class for one inference request

Containing relevant data for an inference request

request_id: int#: None

prompt: str#: None

sampling_params: Optional[megatron.core.inference.sampling_params.SamplingParams]#: None

inference_parameters: Optional[megatron.core.inference.sampling_params.SamplingParams]#: None

prompt_tokens: Optional[List[int]]#: None

prompt_length: Optional[int]#: None

arrival_time: Optional[float]#: None

status: Optional[core.inference.inference_request.Status]#: None

encoder_prompt: Optional[str]#: None

generated_text: Optional[str]#: None

segments: Optional[List[str]]#: None

generated_segments: Optional[List[str]]#: None

generated_sequence_lengths: Optional[List[int]]#: None

generated_tokens: Optional[torch.Tensor]#: None

prompt_log_probs: Optional[torch.Tensor]#: None

generated_log_probs: Optional[torch.Tensor]#: None

prompt_top_n_logprobs: Optional[List[Dict[str, float]]]#: None

generated_top_n_logprobs: Optional[List[Dict[str, float]]]#: None

generated_length: Optional[int]#: None

finish_reason: Optional[str]#: None

tpot: List[float]#: ‘field(…)’

__post_init__()#

serialize() → dict#

Converts the instance into a serializable dictionary.

Returns:: (dict) A dictionary representation of the instance suitable for serialization.

classmethod deserialize( obj: dict, ) → core.inference.inference_request.InferenceRequest#

Deserialize request.

Parameters:: obj (dict) – Serialized request data.
Returns:: (InferenceRequest) Deserialized request.

_post_deserialize(obj: dict)#: This is called after the dataclass is initialized to handle any special deserialization logic.

class core.inference.inference_request.DynamicInferenceEventType(*args, **kwds)#

Bases: enum.Enum

Dynamic inference event type.

Initialization

ADD_ENGINE#: ‘auto(…)’

ADD_CONTEXT#: ‘auto(…)’

GENERATED_TOKEN#: ‘auto(…)’

PAUSE#: ‘auto(…)’

EVICT#: ‘auto(…)’

FINISH#: ‘auto(…)’

FAIL#: ‘auto(…)’

ERROR_TRANSIENT#: ‘auto(…)’

ERROR_NONTRANSIENT#: ‘auto(…)’

class core.inference.inference_request.DynamicInferenceEvent#

A lifecycle event for a dynamic inference requests.

An event is currently one of the following:

request added
request paused
request evicted
request finished
request failed
request error (transient)
request error (non-transient, i.e. fatal)

timestamp: Optional[float]#: None

type: core.inference.inference_request.DynamicInferenceEventType#: None

payload: Optional[Any]#: None

__post_init__()#

__str__()#

serialize() → dict#

Converts the instance into a serializable dictionary.

Returns:: Full event dict.
Return type:: dict

classmethod deserialize( obj: dict, ) → core.inference.inference_request.DynamicInferenceEvent#

Deserialize event.

Parameters:: obj – Serialized event data dict.
Returns:: (DynamicInferenceEvent) Deserialized event.

class core.inference.inference_request.DynamicInferenceRequest#

Bases: core.inference.inference_request.InferenceRequest

Class for one inference request

Containing relevant data for an dynamic inference request

request_id: int#: None

uid: str#: ‘field(…)’

prompt: Optional[str]#: None

prompt_tokens: Optional[torch.Tensor]#: None

media_tensors: Optional[Dict[str, torch.Tensor]]#: None

offload_params: Optional[Dict[str, Any]]#: None

remaining_prompt_tokens: Optional[torch.Tensor]#: None

policy_epoch: Optional[list[tuple[int, int]]]#: None

kv_cache_epoch: Optional[list[tuple[int, int]]]#: None

latency: Optional[float]#: None

routing_indices: Optional[numpy.ndarray]#: None

finished_chunk_token_count: int#: 0

stop_word_ids: Optional[List[List[int]]]#: None

cg_wait_iters: int#: 0

block_size_tokens: Optional[int]#: None

enable_prefix_caching: bool#: False

num_cached_tokens: int#: 0

cached_prompt_log_probs: List[Tuple[int, numpy.ndarray, numpy.ndarray, numpy.ndarray]]#: ‘field(…)’

num_matched_prefix_blocks: int#: 0

mtp_private_suffix_start: Optional[int]#: None

block_hash_salt: Optional[str]#: None

precomputed_block_hashes: List[int]#: ‘field(…)’

disaggregated_params: Optional[dict]#: None

payload_offloaded: bool#: False

payload_stage_metadata: Dict[str, Any]#: ‘field(…)’

__post_init__()#

_compute_block_hashes() → None#

Compute hashes for all complete blocks in the prompt.

After this call:

precomputed_block_hashes is [] if prompt < block_size (no complete blocks)
precomputed_block_hashes is [hash1, …] for N complete blocks

resalt_block_hashes(block_hash_salt: Optional[str]) → None#

Replace the block-hash salt and recompute the prompt’s block hashes.

Only valid for a request that holds no KV yet (nothing prefilled, matched, or generated), since its hash chain must not change under blocks it already owns. Clears any recorded prefix-match state.

property remaining_prompt_length#: Get the length of the remaining prompt tokens.

ttft: Optional[float]#: None

events: List[core.inference.inference_request.DynamicInferenceEvent]#: ‘field(…)’

event_add_engine: Optional[core.inference.inference_request.DynamicInferenceEvent]#: ‘field(…)’

generated_tokens: List[int]#: ‘field(…)’

acceptance_step_lengths: List[int]#: ‘field(…)’

finalize_text( tokenizer: Any, ) → core.inference.inference_request.DynamicInferenceRequest#

Populate generated text by decoding the complete generated token stream.

Parameters:: tokenizer – Tokenizer used to decode generated_tokens.
Returns:: This request, with generated_text populated.
Raises:: ValueError – If tokenizer is None.

__str__()#

serialize( payload_offloaded: bool = False, payload_stage_metadata: Optional[Dict[str, Any]] = None, )#

Converts the instance into a serializable dictionary.

Parameters:

payload_offloaded (bool) – Drop the per-token payload (log probs, MoE routing indices) from the wire; the engine’s RequestPayloadStager has custody of it.
payload_stage_metadata – Opaque metadata returned by the stager for the REST endpoint to attach to its response.

Returns:

(dict) A dictionary representation of the instance suitable for serialization.

_post_deserialize(obj)#

property tracked_metadata: List[Any]#

Obtain an ordered list of all request metadata to be tracked by the context.

This consists of metadata that is used to inform text generation. The values of such fields are tensorized and kept aligned with the current active batch.

Note that while the general request object is mutable, this metadata is inherently assumed to remain immutable once the request becomes active.

static get_metadata_types() → List[Tuple[str, torch.dtype]]#

Keeps track of all request metadata names and dtypes.

Returns:: Mapping from metadata name to: name (str) - The name of the metadata field. dtype (torch.dtype) - The datatype of the metadata.
Return type:: List[Tuple[str, torch.dtype]]

add_event( type: core.inference.inference_request.DynamicInferenceEventType, payload: Optional[Any] = None, ) → core.inference.inference_request.DynamicInferenceEvent#: Add event.

add_event_add_engine()#: Add ‘add_engine’ event - called when request enters the engine queue.

add_event_add_context()#: Add ‘add_context’ event - called when request is added to context for prefill.

add_event_generated_token( token: int, blocks_total: Optional[int] = None, blocks_hashed_total: Optional[int] = None, blocks_hashed_active: Optional[int] = None, blocks_ref_count: Optional[int] = None, pre_fwd_active_token_count: Optional[int] = None, pre_fwd_step_count: Optional[int] = None, )#

Add ‘generated_token’ event - records each generated token.

Parameters:

token (int) – The token ID that was generated.
blocks_total (int) – Total block capacity from allocator.
blocks_hashed_total (int) – All allocated (hashed) blocks.
blocks_hashed_active (int) – Blocks with ref_count > 0.
blocks_ref_count (int) – Sum of block ref counts from allocator.
pre_fwd_active_token_count (int) – Active token count before forward pass.
pre_fwd_step_count (int) – Step count before forward pass.

add_event_pause()#: Add ‘pause’ event.

add_event_evict()#: Add ‘evict’ event.

add_event_finish()#: Add ‘finish’ event.

add_event_fail()#: Add ‘fail’ event.

add_event_error_transient(error: Exception)#: Add transient error event.

add_event_error_nontransient(error: Exception)#: Add non-transient error event.

succeeded() → bool#: Request experienced no non-transient errors.

failed() → bool#: Request experienced non-transient error.

class core.inference.inference_request.DynamicInferenceRequestRecord#

Internal engine history across request checkpoints.

requests: list[core.inference.inference_request.DynamicInferenceRequest]#: ‘field(…)’

latency: Optional[float]#: None

classmethod from_request( request: core.inference.inference_request.DynamicInferenceRequest, ) → core.inference.inference_request.DynamicInferenceRequestRecord#

Initialize record from a single request.

Parameters:: request (DynamicInferenceRequest) – Initial request.
Returns:: (DynamicInferenceRequestRecord) A record.

classmethod from_resumed_generation( first: core.inference.inference_request.DynamicInferenceRequest, resumed: core.inference.inference_request.ResumedGeneration, *, generation_epoch: Optional[int], ) → core.inference.inference_request.DynamicInferenceRequestRecord#

Build the record the engine admits for a request that resumes a generation.

Parameters:

first – The request built from the new submission, on the prompt as this engine expanded it. It becomes the record’s first segment and takes over the resumed generation’s tokens, scores, stamps, evictions and budget.
resumed – The generation the prompt preparer handed back, decoded from its wire form: what an earlier engine generated for this request and whether it ended.
generation_epoch – This engine’s current epoch, stamped on the continuation when it differs from the resumed generation’s last.

Returns:

A completed one-segment record when the generation already ended; otherwise two segments, the resumed prefix evicted and the continuation the engine schedules.

Raises:

ValueError – When the resumed generation does not fit the request it resumes.

__getitem__( idx: int, ) → core.inference.inference_request.DynamicInferenceRequest#

Get request by index.

Parameters:: idx (int) – Request index.
Returns:: (DynamicInferenceRequest) Request object.

property request_id: int#

Get request id.

Returns:: (int) Request id.

checkpoint() → None#: Maintain reference to previous request, and then append a new request that concatenates the previous prompt and generations.

merge() → core.inference.inference_request.DynamicInferenceRequest#

Merge requests into a single checkpoint-agnostic request object.

Returns:: (DynamicInferenceRequest) Merged request.

class core.inference.inference_request.FinishedRequestRecord#

Per-request metadata, outside of the RESTful reply, used by the stager and snapshots.

policy_epoch: Optional[list[tuple[int, int]]]#: None

kv_cache_epoch: Optional[list[tuple[int, int]]]#: None

num_evictions: int#: None

num_tokens_to_generate: Optional[int]#: None

finish_reason: Optional[str]#: None

payload: Optional[OffloadedRequestPayload]#: None

classmethod from_request( request: core.inference.inference_request.DynamicInferenceRequest, ) → core.inference.inference_request.FinishedRequestRecord#: Build the request’s non-RESTful metadata from a finished request.

class core.inference.inference_request.OffloadedRequestPayload#

A finished request’s per-token payload, kept off the RESTful reply by payload offload.

Self-contained: a consumer can rebuild the served sequence from it alone, keyed by the request uid (the OpenAI response id). The top-N scores and the per-step token counts also stay on the reply; they ride here for a resume.

prompt_token_ids is the prompt the model ran on. For a VLM request that is the expanded sequence (one media token per projected embedding), which is what a trainer needs. media_tensors is None for text-only requests; otherwise it holds host copies of what the vision encoder consumed (imgs as packed patches [1, total_patches, C*P*P] on the HTTP path, imgs_sizes, and num_frames / num_tiles when present), so a trainer can project the same media the policy generated against.

prompt_token_ids: Optional[list[int]]#: None

generated_token_ids: list[int]#: None

generated_log_probs: Optional[list[float]]#: None

prompt_log_probs: Optional[list[float]]#: None

routing_indices: Optional[numpy.ndarray]#: None

media_tensors: Optional[Dict[str, torch.Tensor]]#: None

prompt_top_n_logprobs: Optional[List[Dict[str, float]]]#: None

generated_top_n_logprobs: Optional[List[Dict[str, float]]]#: None

acceptance_step_lengths: List[int]#: ‘field(…)’

classmethod from_request( request: core.inference.inference_request.DynamicInferenceRequest, ) → core.inference.inference_request.OffloadedRequestPayload#

Copy the payload off a finished (merged) request as plain host-side values.

Side effect: replaces request.prompt_tokens and request.media_tensors with their host copies, so one device-to-host copy serves both this payload and every later host-side read of the finished request (its reply and the caller’s result), and a finished request kept alive while its payload is staged holds no device memory.

class core.inference.inference_request.RequestPayloadStageResult#

A custody acknowledgement and opaque metadata for the served response.

response_metadata: Dict[str, Any]#: ‘field(…)’

class core.inference.inference_request.RequestPayloadStager#

Bases: typing.Protocol

Protocol to handle offloading of request payloads to a storage backend.

The engine calls stage() on a worker thread (asyncio.to_thread), one call per completed request, so implementations must follow these rules:

Calls run concurrently with each other, so stage() must be thread-safe.
Calls share the event loop’s default thread pool. A write that blocks for a long time holds a pool thread, so many slow writes can use up the pool and delay other to_thread work.
stage() must use only its arguments and the stager’s own state. It must not touch engine state, the engine’s ZMQ sockets, or anything else owned by the engine loop’s thread.

stage( uid: str, payload: core.inference.inference_request.OffloadedRequestPayload, *, finished_metadata: core.inference.inference_request.FinishedRequestRecord, offload_params: Optional[Dict[str, Any]] = None, ) → Optional[core.inference.inference_request.RequestPayloadStageResult]#: Stage a payload, or return None to keep it on the normal reply path.

class core.inference.inference_request.LiveRequestSnapshot#

Snapshot of a live request, allowing for its later resumption.

A snapshot carries no media; resuming a multimodal request requires resubmitting the same media so the expanded prompt the boundaries were counted on is reproduced.

uid: str#: None

request_id: int#: None

offload_params: Optional[Dict[str, Any]]#: None

payload: core.inference.inference_request.OffloadedRequestPayload#: None

record: core.inference.inference_request.FinishedRequestRecord#: None

classmethod from_record( record: core.inference.inference_request.DynamicInferenceRequestRecord, *, routing_indices: Optional[numpy.ndarray] = None, ) → core.inference.inference_request.LiveRequestSnapshot#: Copy a live record’s merged state; the record itself is left untouched.

core.inference.inference_request.PREFIX_TEMPLATE_TOKEN_IDS_FIELD#: ‘template_prefix_token_ids’

core.inference.inference_request.PREFIX_EOS_TOKEN_ID_FIELD#: ‘eos_token_id’

core.inference.inference_request.PREFIX_EXPANDED_TOKEN_COUNT_FIELD#: ‘_prefix_expanded_token_count’

core.inference.inference_request.PREFIX_MEDIA_COUNT_FIELD#: ‘_prefix_media_count’

core.inference.inference_request.RESUMED_GENERATION_FIELD#: ‘_resumed_generation’

class core.inference.inference_request.ResumedGeneration#

Generated tokens a request already holds when it is admitted.

token_ids: List[int]#: None

num_tokens_to_generate: int#: None

log_probs: Optional[List[float]]#: None

prompt_log_probs: Optional[List[float]]#: None

policy_epoch: Optional[List[Tuple[int, int]]]#: None

kv_cache_epoch: Optional[List[Tuple[int, int]]]#: None

num_evictions: int#: 0

finish_reason: Optional[str]#: None

prompt_top_n_logprobs: Optional[List[Dict[str, float]]]#: None

generated_top_n_logprobs: Optional[List[Dict[str, float]]]#: None

acceptance_step_lengths: List[int]#: ‘field(…)’

routing_indices: Optional[numpy.ndarray]#: None

__post_init__()#

classmethod from_snapshot( snapshot: core.inference.inference_request.LiveRequestSnapshot, ) → core.inference.inference_request.ResumedGeneration#: The resume of a live request from the engine’s snapshot of it.

to_wire() → Dict[str, Any]#: The msgpack form MP rank zero broadcasts.

classmethod from_wire( obj: Dict[str, Any], ) → core.inference.inference_request.ResumedGeneration#: Rebuild the broadcast form.

class core.inference.inference_request.RequestPromptPreparationResult#

The resolved engine prompt and opaque parameters that describe it.

prompt: Union[str, List[int], torch.Tensor]#: None

offload_params: Optional[Dict[str, Any]]#: None

class core.inference.inference_request.RequestPromptPreparer#

Bases: typing.Protocol

Protocol for resolving an exact prompt before engine admission.

The preparer runs on the model-parallel coordinator before the request is broadcast, i.e. before DynamicInferenceEngine.add_request tokenizes or expands anything. For a multimodal request prompt is therefore the compact render (one media token per image/video, as the chat endpoint tokenized it) and the returned prompt must stay in that space: _build_vlm_request expands every media token it finds, so a prefix spliced in already-expanded form would be expanded a second time and fail the placeholder-count check. Consumers that store previous turns splice their exact tokens (OffloadedRequestPayload. prompt_token_ids plus generated_token_ids) the same way for text and multimodal requests. When PREFIX_MEDIA_COUNT_FIELD is present, they must also return the length of that exact prefix as PREFIX_EXPANDED_TOKEN_COUNT_FIELD so the engine expands only the tokens after it. A preparer that resumes a generation puts ResumedGeneration.to_wire() under RESUMED_GENERATION_FIELD; its epoch boundaries are the engine’s own, counted on the expanded prompt the engine ran.

prepare_prompt( prompt: Union[str, List[int], torch.Tensor], *, offload_params: Optional[Dict[str, Any]] = None, ) → core.inference.inference_request.RequestPromptPreparationResult#: Return the engine prompt and metadata that describe that prompt.

class core.inference.inference_request.VLMInferenceRequest#

Bases: core.inference.inference_request.InferenceRequest

Class for a VLM inference request

num_img_embeddings_per_tile: int#: None

imgs: torch.Tensor#: None

num_tiles: torch.Tensor#: None

decoder_seq_length: int#: None

class core.inference.inference_request.DynamicVLMInferenceRequest#

Bases: core.inference.inference_request.DynamicInferenceRequest, core.inference.inference_request.VLMInferenceRequest

Dynamic inference request for VLM models.

Combines DynamicInferenceRequest (for dynamic batching) with VLMInferenceRequest (for multimodal fields). Also stores pre-computed image embeddings and the image token mask produced by expand_image_tokens.

image_embeddings: Optional[torch.Tensor]#: None

image_token_mask: Optional[torch.Tensor]#: None

imgs_sizes: Optional[torch.Tensor]#: None

num_frames: Optional[torch.Tensor]#: None

video_frame_indices: Optional[List[List[int]]]#: None

video_fps: Optional[List[float]]#: None

media_tokens_preexpanded: bool#: False

media_cache_key: Optional[str]#: None

deferred_image_embedding_count: Optional[int]#

None

Embedding count of an image request admitted without encoding; the engine encodes it together with the other waiting requests right before prefill and clears this afterwards.

core.inference.inference_request#

Module Contents#

Classes#

Functions#

Data#

API#

`core.inference.inference_request`#