Skip to navigation
DevelopmentAPI ReferenceFull Library ReferenceNemo AutomodelNemo AutomodelComponentsModelsQwen3 Vl Moenemo_automodel.components.models.qwen3_vl_moe.model

nemo_automodel.components.models.qwen3_vl_moe.model

View as Markdown

Module Contents

Classes

NameDescription
Fp32SafeQwen3VLMoeTextRotaryEmbeddingEnsure inv_freq stays in float32
Fp32SafeQwen3VLMoeVisionRotaryEmbeddingEnsure the vision rotary inv_freq buffer remains float32.
Qwen3VLMoeBlockQwen3-VL block adapter that accepts HF-style position embeddings.
Qwen3VLMoeForConditionalGenerationQwen3-VL conditional generation model using the Qwen3-MoE backend components.
Qwen3VLMoeModel-
Qwen3VLMoeTextModelBackendQwen3-VL text decoder rebuilt on top of the Qwen3-MoE block implementation.

Data

ModelClass

API

class nemo_automodel.components.models.qwen3_vl_moe.model.Fp32SafeQwen3VLMoeTextRotaryEmbedding()

Bases: Qwen3VLMoeTextRotaryEmbedding

Ensure inv_freq stays in float32

nemo_automodel.components.models.qwen3_vl_moe.model.Fp32SafeQwen3VLMoeTextRotaryEmbedding._apply(
    fn: typing.Any,
    recurse: bool = True
)
class nemo_automodel.components.models.qwen3_vl_moe.model.Fp32SafeQwen3VLMoeVisionRotaryEmbedding()

Bases: Qwen3VLMoeVisionRotaryEmbedding

Ensure the vision rotary inv_freq buffer remains float32.

nemo_automodel.components.models.qwen3_vl_moe.model.Fp32SafeQwen3VLMoeVisionRotaryEmbedding._apply(
    fn: typing.Any,
    recurse: bool = True
)
class nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeBlock()

Bases: Block

Qwen3-VL block adapter that accepts HF-style position embeddings.

nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeBlock.forward(
    x: torch.Tensor,
    freqs_cis: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
    attn_kwargs: typing.Any = {}
) -> torch.Tensor
class nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration(
    config: transformers.models.qwen3_vl_moe.configuration_qwen3_vl_moe.Qwen3VLMoeConfig,
    moe_config: nemo_automodel.components.moe.config.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    kwargs = {}
)

Bases: HFCheckpointingMixin, HFQwen3VLMoeForConditionalGeneration, MoEFSDPSyncMixin

Qwen3-VL conditional generation model using the Qwen3-MoE backend components.

_pp_keep_self_forward
bool = True
lm_head
pad_token_id
= pad_token_id if pad_token_id is not None else -1
state_dict_adapter
tie_word_embeddings_support
TieSupport = TieSupport.UNTIED_ONLY
vocab_size
= text_config.vocab_size
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.forward(
    input_ids: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    inputs_embeds: torch.Tensor | None = None,
    cache_position: torch.Tensor | None = None,
    logits_to_keep: int | torch.Tensor = 0,
    output_hidden_states: bool | None = None,
    kwargs: typing.Any = {}
)

Run text and media inputs while preserving their distinct leading axes.

Parameters:

input_ids
torch.Tensor | NoneDefaults to None

Token IDs of shape [batch, sequence] or [tokens] for THD.

position_ids
torch.Tensor | NoneDefaults to None

Rotary coordinates of shape [3, batch, sequence], with a placeholder batch axis of size one for THD.

attention_mask
torch.Tensor | NoneDefaults to None

Optional binary padding mask of shape [batch, sequence]. Dense NEAT masks are unsupported; packed inputs use THD with qkv_format="thd" and attention_mask=None.

padding_mask
torch.Tensor | NoneDefaults to None

Optional padding flags of shape [batch, sequence] or [tokens] for THD.

inputs_embeds
torch.Tensor | NoneDefaults to None

Optional embeddings of shape [batch, sequence, hidden] supplied by a preceding pipeline stage.

cache_position
torch.Tensor | NoneDefaults to None

Optional token positions of shape [sequence].

logits_to_keep
int | torch.TensorDefaults to 0

Number of trailing logits, or token indices of shape [selected_tokens], to retain.

output_hidden_states
bool | NoneDefaults to None

Return hidden states for fused loss computation.

**kwargs
AnyDefaults to {}

Attention metadata, including cu_seqlens and cu_seqlens_padded of shape [documents + 1] (optionally with a leading placeholder batch axis), and scalar max_seqlen. Media tensors retain pixel_values/pixel_values_videos of shape [patches, patch_features] and image/video grid_thw of shape [media_items, 3], including when media_items is one.

Returns:

Model output with logits of shape [batch, sequence, vocab] or

nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.from_config(
    config: transformers.models.qwen3_vl_moe.configuration_qwen3_vl_moe.Qwen3VLMoeConfig,
    moe_config: nemo_automodel.components.moe.config.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    kwargs = {}
)
classmethod
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.from_pretrained(
    pretrained_model_name_or_path: str,
    model_args = (),
    kwargs = {}
)
classmethod
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.get_input_embeddings()
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.get_output_embeddings()
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.initialize_weights(
    buffer_device: torch.device | None = None,
    dtype: torch.dtype = torch.bfloat16
) -> None
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.set_input_embeddings(
    value
)
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.set_output_embeddings(
    new_embeddings
)
class nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeModel()

Bases: HFQwen3VLMoeModel

nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeModel.forward(
    input_ids = None,
    attention_mask = None,
    position_ids = None,
    past_key_values = None,
    inputs_embeds = None,
    pixel_values = None,
    pixel_values_videos = None,
    image_grid_thw = None,
    video_grid_thw = None,
    cache_position = None,
    kwargs = {}
)

Fuse packed vision patches into padded text sequences.

Parameters:

input_ids
Defaults to None

Integer tensor of shape [batch, sequence].

attention_mask
Defaults to None

Padding mask of shape [batch, sequence], or per-layer masks.

position_ids
Defaults to None

Rotary coordinates of shape [3, batch, sequence].

past_key_values
Defaults to None

Hugging Face decoder cache, unused by the native text backend.

inputs_embeds
Defaults to None

Optional embeddings of shape [batch, sequence, hidden].

pixel_values
Defaults to None

Image patches of shape [image_patches, patch_features], where patch_features flattens channels, temporal depth, height, and width.

pixel_values_videos
Defaults to None

Video patches of shape [video_patches, patch_features].

image_grid_thw
Defaults to None

Integer tensor of shape [images, 3] with temporal/height/width grids.

video_grid_thw
Defaults to None

Integer tensor of shape [videos, 3] with temporal/height/width grids.

cache_position
Defaults to None

Optional positions of shape [sequence].

**kwargs
Defaults to {}

Backend options, including mm_token_type_ids of shape [batch, sequence].

Returns:

Model output with last_hidden_state of shape [batch, sequence, hidden].

class nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeTextModelBackend(
    config: transformers.models.qwen3_vl_moe.configuration_qwen3_vl_moe.Qwen3VLMoeTextConfig,
    backend: nemo_automodel.components.models.common.BackendConfig,
    moe_config: nemo_automodel.components.moe.config.MoEConfig | None = None,
    moe_overrides: dict | None = None
)

Bases: Module

Qwen3-VL text decoder rebuilt on top of the Qwen3-MoE block implementation.

embed_tokens
layers
moe_config
= moe_config or MoEConfig(**moe_defaults)
norm
padding_idx
= getattr(config, 'pad_token_id', None)
rotary_emb
vocab_size
= config.vocab_size
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeTextModelBackend._deepstack_process(
    hidden_states: torch.Tensor,
    visual_pos_masks: torch.Tensor | None,
    visual_embeds: torch.Tensor
) -> torch.Tensor
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeTextModelBackend.forward(
    input_ids: torch.Tensor | None = None,
    inputs_embeds: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    cache_position: torch.Tensor | None = None,
    visual_pos_masks: torch.Tensor | None = None,
    deepstack_visual_embeds: list[torch.Tensor] | None = None,
    padding_mask: torch.Tensor | None = None,
    past_key_values: typing.Any | None = None,
    use_cache: bool | None = None,
    attn_kwargs: typing.Any = {}
) -> transformers.models.qwen3_vl_moe.modeling_qwen3_vl_moe.Qwen3VLMoeModelOutputWithPast
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeTextModelBackend.get_input_embeddings() -> torch.nn.Module
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeTextModelBackend.init_weights(
    buffer_device: torch.device | None = None
) -> None
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeTextModelBackend.set_input_embeddings(
    value: torch.nn.Module
) -> None
nemo_automodel.components.models.qwen3_vl_moe.model.ModelClass = Qwen3VLMoeForConditionalGeneration