core.inference.text_generation_server.dynamic_text_gen_server.image_preprocessing

Image preprocessing for multimodal inference servers.

Shared between vlm_server.py and the coordinator/engine image dispatch in run_dynamic_text_generation_server.py. Lives in core/inference so the engine can import it without circular dependencies.

Module Contents

Functions

_resolve_pixel_stats	Return (pixel_mean, pixel_std) for a vision encoder.
_load_frame_sequence_manifest	Load PIL images from a configured frame-sequence manifest.
dynamic_res_preprocess	Resize image to fit within [min_patches, max_patches] preserving aspect ratio.
preprocess_image	Convert one PIL image into packed vision patches and its resized shape.
preprocess_image_bytes	Decode image bytes and return packed vision patches and its resized shape.
preprocess_image_bytes_list	Preprocess a list of raw image bytes into engine.add_request image kwargs.
_video_sample_indices	Return the existing uniformly spaced sample indices for a video.
_decode_sampled_video_frames	Decode the stream while converting only uniformly sampled frames to RGB.
preprocess_video_bytes_list	Decode videos and return packed dynamic-resolution engine inputs.

API

core.inference.text_generation_server.dynamic_text_gen_server.image_preprocessing._resolve_pixel_stats(vision_model_type: str)

Return (pixel_mean, pixel_std) for a vision encoder.

Reads from the canonical encoder registry so training and inference share one source of truth. Falls back to CLIP-style stats for unknown encoders (matching the registry’s own EncoderSpec defaults).

core.inference.text_generation_server.dynamic_text_gen_server.image_preprocessing._load_frame_sequence_manifest(

payload: bytes,

frame_manifest_magic: Optional[bytes],

)

Load PIL images from a configured frame-sequence manifest.

core.inference.text_generation_server.dynamic_text_gen_server.image_preprocessing.dynamic_res_preprocess(

image,

min_patches=1,

max_patches=128,

res_step=16,

factor_max=1.0,

pixel_shuffle=False,

spatial_merge_size=1,

video_maintain_aspect_ratio=None,

)

Resize image to fit within [min_patches, max_patches] preserving aspect ratio.

When video_maintain_aspect_ratio is not None, use a fixed per-frame max_patches budget. True preserves the source aspect ratio; False uses a square grid. Images leave this as None and use adaptive resizing. For pixel_shuffle, patch grid dimensions are rounded to even numbers for compatibility.

NOTE: Training uses DynamicResolutionImageTilingStrategy._process_single (in megatron.energon.task_encoder.multimodal.image_tiling) as the canonical resize. The math here is intentionally a subset of that strategy and could drift if energon’s implementation changes (e.g. min_side floor, tiling augmentation). For full parity, inference should call into the energon strategy directly — TODO once we have a clean way to import it that doesn’t require energon at engine-drain time.

core.inference.text_generation_server.dynamic_text_gen_server.image_preprocessing.preprocess_image(

image,

config: megatron.core.inference.config.ImageProcessingConfig,

target_hw=None,

device: Optional[torch.device] = None,

) → tuple

Convert one PIL image into packed vision patches and its resized shape.

core.inference.text_generation_server.dynamic_text_gen_server.image_preprocessing.preprocess_image_bytes(

image_bytes: bytes,

config: megatron.core.inference.config.ImageProcessingConfig,

target_hw=None,

device: Optional[torch.device] = None,

) → tuple

Decode image bytes and return packed vision patches and its resized shape.

core.inference.text_generation_server.dynamic_text_gen_server.image_preprocessing.preprocess_image_bytes_list(

image_bytes_list,

config: megatron.core.inference.config.ImageProcessingConfig,

device: Optional[torch.device] = None,

) → dict

Preprocess a list of raw image bytes into engine.add_request image kwargs.

Selects the dynamic-resolution or tiling path from the inference config. Each image is preprocessed independently so its aspect ratio is preserved.

Parameters:

• image_bytes_list – List of raw image bytes (one entry per image).

• config – Image preprocessing configuration.

• device – Optional target device for the returned tensors. If None, tensors are returned on CPU and the caller is responsible for transfer.

Returns:

dict suitable for **kwargs to DynamicInferenceEngine.add_request.

core.inference.text_generation_server.dynamic_text_gen_server.image_preprocessing._video_sample_indices(

total_frames: int,

config: megatron.core.inference.config.VideoProcessingConfig,

) → list[int]

Return the existing uniformly spaced sample indices for a video.

core.inference.text_generation_server.dynamic_text_gen_server.image_preprocessing._decode_sampled_video_frames(

encoded_video: bytes,

config: megatron.core.inference.config.VideoProcessingConfig,

)

Decode the stream while converting only uniformly sampled frames to RGB.

core.inference.text_generation_server.dynamic_text_gen_server.image_preprocessing.preprocess_video_bytes_list(

video_bytes_list,

config: megatron.core.inference.config.VideoProcessingConfig,

device: Optional[torch.device] = None,

) → dict

Decode videos and return packed dynamic-resolution engine inputs.

Frames are sampled uniformly, resized using the same image preprocessing configuration as still images, and kept grouped through num_frames.