`bridge.models.qwen_vl.data.energon`#

Module Contents#

Classes#

`QwenVLTaskSample`	HF-style Qwen VLM sample produced from an Energon `ChatMLSample`.
`QwenVLTaskBatch`	Batched Qwen VLM tensors produced by the shared HF collate function.
`QwenVLTaskEncoder`	Energon task encoder for Qwen VL samples.

Functions#

`process_vision`	Minimal vision preprocessing wrapper using the provided processor (e.g., HF AutoProcessor).
`_resolve_hf_mm_token_ids`	Resolve HF tokenizer ids for and
`_visual_token_count`	Return merged visual token count for THW grid metadata.
`convert_to_qwenvl_content`	Split user input into format QwenVL tokenizer accepts.

API#

bridge.models.qwen_vl.data.energon.process_vision( processor, images, videos, fps=None, model_version: str = 'qwen-vl', min_pixels=None, max_pixels=None, )#: Minimal vision preprocessing wrapper using the provided processor (e.g., HF AutoProcessor).

bridge.models.qwen_vl.data.energon._resolve_hf_mm_token_ids(hf_tokenizer)#: Resolve HF tokenizer ids for and

bridge.models.qwen_vl.data.energon._visual_token_count(grid_thw: Any, merge_size: int) → int#: Return merged visual token count for THW grid metadata.

class bridge.models.qwen_vl.data.energon.QwenVLTaskSample#

HF-style Qwen VLM sample produced from an Energon ChatMLSample.

Expected input format: Produced by QwenVLTaskEncoder.encode_sample from an Energon ChatMLSample. example follows the HF VLM collate schema: {"conversation": [{"role": ..., "content": [...]}, ...]} with inline {"type": "image"|"video", ...} media parts.

Output format: Consumed by QwenVLTaskEncoder.batch, which passes the example dictionaries to the same Qwen collate function used by HF-style VLM datasets.

__key__: str#: None

__subflavors__: Dict#: None

example: Dict[str, Any]#: None

class bridge.models.qwen_vl.data.energon.QwenVLTaskBatch#

Bases: megatron.energon.Batch

Batched Qwen VLM tensors produced by the shared HF collate function.

__keys__: List[str]#: None

__subflavors__: List[Dict]#: None

input_ids: torch.Tensor#: None

position_ids: torch.Tensor#: None

labels: torch.Tensor#: None

loss_mask: torch.Tensor#: None

visual_inputs: megatron.bridge.training.utils.visual_inputs.GenericVisualInputs | None#: None

attention_mask: torch.Tensor | None#: None

cu_seqlens_q: torch.Tensor | None#: None

cu_seqlens_kv: torch.Tensor | None#: None

cu_seqlens_q_padded: torch.Tensor | None#: None

cu_seqlens_kv_padded: torch.Tensor | None#: None

max_seqlen_q: torch.Tensor | None#: None

max_seqlen_kv: torch.Tensor | None#: None

bridge.models.qwen_vl.data.energon.convert_to_qwenvl_content( user_input: str, image_pattern: str = '<image>', video_pattern: str = '<video>', )#: Split user input into format QwenVL tokenizer accepts.

class bridge.models.qwen_vl.data.energon.QwenVLTaskEncoder( tokenizer, image_processor, temporal_patch_size: int = 2, spatial_merge_size: int = 2, patch_size: int = 14, max_padding_length: int = 4096, min_pixels: int = 200704, max_pixels: int = 1003520, max_num_images: int | None = 10, max_num_frames: int | None = 60, max_visual_tokens: int | None = 16384, pad_to_max_length: bool = False, pad_to_multiple_of: int = 128, enable_in_batch_packing: bool = False, in_batch_packing_pad_to_multiple_of: int = 1, )#

Bases: megatron.energon.DefaultTaskEncoder[megatron.bridge.data.energon.task_encoder_utils.ChatMLSample, bridge.models.qwen_vl.data.energon.QwenVLTaskSample, bridge.models.qwen_vl.data.energon.QwenVLTaskBatch, dict]

Energon task encoder for Qwen VL samples.

Parameters:

tokenizer – HF tokenizer for resolving multimodal token ids.
image_processor – HF processor passed to the Qwen collate function.
temporal_patch_size – Temporal patch size used for video token accounting.
spatial_merge_size – Spatial merge size used for visual token accounting.
patch_size – Vision patch size.
max_padding_length – Maximum sequence length accepted after collation.
min_pixels – Minimum pixel constraint forwarded to Qwen vision processing.
max_pixels – Maximum pixel constraint forwarded to Qwen vision processing.
max_num_images – Optional per-sample image count limit.
max_num_frames – Optional per-video frame limit.
max_visual_tokens – Optional per-sample visual token budget.
pad_to_max_length – Whether collate-time padding should pad non-packed batches to max_padding_length.
pad_to_multiple_of – Non-packed collate-time padding multiple used when pad_to_max_length is false.
enable_in_batch_packing – Whether the Qwen collate should do in-batch sequence packing.
in_batch_packing_pad_to_multiple_of – Per-sample padding multiple used only by the in-batch packed path, typically to satisfy CP/SP divisibility.

Initialization

encode_sample( sample: megatron.bridge.data.energon.task_encoder_utils.ChatMLSample, )#

Normalize one Energon sample into the HF-style Qwen collate schema.

Expected input format: sample is an Energon ChatMLSample with JSON conversation and optional decoded imgs / videos media payloads.

Output format: Returns QwenVLTaskSample whose example is a HF-style VLM collate dictionary: {"conversation": [{"role": ..., "content": [{"type": ...}, ...]}]}. Tokenization, image/video preprocessing, labels, and loss masks are intentionally deferred to self.collate_fn so HF and Energon data paths share the same Qwen model processing.

collate_fn( examples: List[Dict[str, Any]], ) → dict[str, torch.Tensor]#

Collate Qwen HF-style examples with the shared HF dataset collator.

Expected input format: examples is a list of dictionaries in the same schema returned by HF-style VLM datasets: each item has conversation messages with multimodal content parts.

Output format: Returns the exact dictionary produced by qwen2_5_collate_fn: input_ids, labels, loss_mask, position_ids, optional attention_mask, and visual_inputs.

batch( samples: List[bridge.models.qwen_vl.data.energon.QwenVLTaskSample], ) → bridge.models.qwen_vl.data.energon.QwenVLTaskBatch#

Collate normalized Energon samples with the shared Qwen HF collator.

Expected input format: samples are QwenVLTaskSample objects whose example fields follow the HF VLM collate schema.

Output format: Returns QwenVLTaskBatch carrying the same tensors as qwen2_5_collate_fn plus Energon batch metadata.

encode_batch( batch: bridge.models.qwen_vl.data.energon.QwenVLTaskBatch, ) → dict#: Encode batch in dict

bridge.models.qwen_vl.data.energon#

Module Contents#

Classes#

Functions#

API#

`bridge.models.qwen_vl.data.energon`#