bridge.models.qwen_vl.modelling_qwen3_vl.text_model

Copied from https://github.com/Thaurun/mbridge/blob/4462d1e284626d2ed9d3e3e 3e5a40f2ee42a2c74/mbridge/models/qwen3_vl/gpt_model.py

Module Contents

Classes

Qwen3VLGPTModel

Qwen3-VL GPT model with vision-language capabilities.

Functions

_get_mtp_packed_seq_params

Use physical padded offsets for MTP token rolling without changing attention metadata.

API

bridge.models.qwen_vl.modelling_qwen3_vl.text_model._get_mtp_packed_seq_params(

packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams | None,

) → megatron.core.packed_seq_params.PackedSeqParams | None

Use physical padded offsets for MTP token rolling without changing attention metadata.

class bridge.models.qwen_vl.modelling_qwen3_vl.text_model.Qwen3VLGPTModel(

config: megatron.bridge.models.transformer_config.TransformerConfig,

transformer_layer_spec: megatron.core.transformer.spec_utils.ModuleSpec,

vocab_size: int,

max_sequence_length: int,

pre_process: bool = True,

post_process: bool = True,

fp16_lm_cross_entropy: bool = False,

parallel_output: bool = True,

share_embeddings_and_output_weights: bool = False,

position_embedding_type: Literal[learned_absolute, rope, mrope, none] = 'learned_absolute',

rotary_percent: float = 1.0,

rotary_base: int = 10000,

rope_scaling: bool = False,

rope_scaling_factor: float = 8.0,

scatter_embedding_sequence_parallel: bool = True,

seq_len_interpolation_factor: Optional[float] = None,

mtp_block_spec: Optional[megatron.core.transformer.spec_utils.ModuleSpec] = None,

vp_stage: Optional[int] = None,

pg_collection: megatron.core.process_groups_config.ProcessGroupCollection = None,

)

Bases: megatron.core.models.gpt.gpt_model.GPTModel

Qwen3-VL GPT model with vision-language capabilities.

Initialization

tie_embeddings_and_output_weights_state_dict(

sharded_state_dict: megatron.core.dist_checkpointing.mapping.ShardedStateDict,

output_layer_weight_key: str,

first_stage_word_emb_key: str,

metadata: dict | None = None,

) → None

Tie embedding/output checkpoint entries for Qwen3-VL MTP pipeline stages.

forward(

input_ids: torch.Tensor,

position_ids: torch.Tensor,

attention_mask: torch.Tensor,

decoder_input: torch.Tensor = None,

labels: torch.Tensor = None,

inference_context: megatron.core.inference.contexts.BaseInferenceContext = None,

packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams = None,

extra_block_kwargs: dict = None,

runtime_gather_output: Optional[bool] = None,

*,

inference_params: Optional[megatron.core.inference.contexts.BaseInferenceContext] = None,

loss_mask: Optional[torch.Tensor] = None,

visual_pos_masks: Optional[torch.Tensor] = None,

deepstack_visual_embeds: Optional[list[torch.Tensor]] = None,

) → torch.Tensor

Forward function of the GPT Model This function passes the input tensors through the embedding layer, and then the decoeder and finally into the post processing layer (optional).

forward pass is overridden to add support for deepstack visual embeddings.

It either returns the Loss values if labels are given or the final hidden units

Parameters:

runtime_gather_output (bool) – Gather output at runtime. Default None means parallel_output arg in the constructor will be used.