nemo_automodel.components.models.qwen3_vl.model

Dense Qwen3-VL integration with in-forward context-parallel multimodal sharding.

Module Contents

Classes

Name	Description
`Qwen3VLForConditionalGeneration`	Dense Qwen3-VL with a DeepStack-aware context-parallel forward path.

Data

ModelClass

API

class nemo_automodel.components.models.qwen3_vl.model.Qwen3VLForConditionalGeneration(
    config: transformers.models.qwen3_vl.configuration_qwen3_vl.Qwen3VLConfig
)

Bases: HFCheckpointingMixin, HFQwen3VLForConditionalGeneration

Dense Qwen3-VL with a DeepStack-aware context-parallel forward path.

_tied_weights_keys

tie_word_embeddings_support

TieSupport = TieSupport.BOTH

nemo_automodel.components.models.qwen3_vl.model.Qwen3VLForConditionalGeneration._prepare_visual_inputs_for_cp(
    input_ids: torch.Tensor,
    inputs_embeds: torch.Tensor,
    pixel_values: torch.Tensor | None,
    pixel_values_videos: torch.Tensor | None,
    image_grid_thw: torch.Tensor | None,
    video_grid_thw: torch.Tensor | None
) -> tuple[torch.Tensor, torch.Tensor | None, list[torch.Tensor] | None]

Encode and scatter Qwen3-VL visual features before sequence sharding.

Images and videos share one vision-tower call when both are present. This preserves collective ordering under FSDP and lets CP vision frame sharding partition their combined frame stream once.

Parameters:

input_ids

torch.Tensor

Token ids of shape [batch, sequence] containing image and video placeholder ids.

inputs_embeds

torch.Tensor

Text embeddings of shape [batch, sequence, hidden].

pixel_values

torch.Tensor | None

Optional image patch rows of shape [image_patch_rows, patch_dim] in image-entry order.

pixel_values_videos

torch.Tensor | None

Optional video patch rows of shape [video_patch_rows, patch_dim] in video-entry/frame order.

image_grid_thw

torch.Tensor | None

Optional image grid tensor of shape [num_images, 3].

video_grid_thw

torch.Tensor | None

Optional video grid tensor of shape [num_videos, 3].

Returns: torch.Tensor

A tuple containing updated embeddings of shape “[batch, sequence,

Raises:

ValueError: If patch rows are provided without matching grid metadata, or grid metadata is provided without patch rows.

nemo_automodel.components.models.qwen3_vl.model.Qwen3VLForConditionalGeneration._shard_multimodal_inputs_for_cp(
    inputs_embeds: torch.Tensor,
    visual_pos_masks: torch.Tensor | None,
    deepstack_visual_embeds: list[torch.Tensor] | None
) -> tuple[torch.Tensor, torch.Tensor | None, list[torch.Tensor] | None]

Shard token embeddings and ragged DeepStack inputs in matching order.

Parameters:

inputs_embeds

torch.Tensor

Full-sequence embeddings [batch, sequence, hidden].

visual_pos_masks

torch.Tensor | None

Optional full-sequence visual mask [batch, sequence].

deepstack_visual_embeds

list[torch.Tensor] | None

Optional tensors shaped [visual_tokens, hidden], one per DeepStack injection layer.

Returns: torch.Tensor

Local embeddings [batch, local_sequence, hidden], the matching

Raises:

TypeError: If visual_pos_masks is not boolean.
ValueError: If mask and DeepStack shapes are inconsistent.

nemo_automodel.components.models.qwen3_vl.model.Qwen3VLForConditionalGeneration.forward(
    input_ids: torch.LongTensor | None = None,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.LongTensor | None = None,
    past_key_values: typing.Any | None = None,
    inputs_embeds: torch.FloatTensor | None = None,
    labels: torch.LongTensor | None = None,
    pixel_values: torch.Tensor | None = None,
    pixel_values_videos: torch.FloatTensor | None = None,
    image_grid_thw: torch.LongTensor | None = None,
    video_grid_thw: torch.LongTensor | None = None,
    mm_token_type_ids: torch.IntTensor | None = None,
    use_cache: bool | None = None,
    cache_position: torch.LongTensor | None = None,
    logits_to_keep: int | torch.Tensor = 0,
    kwargs: typing.Any = {}
) -> tuple | transformers.models.qwen3_vl.modeling_qwen3_vl.Qwen3VLCausalLMOutputWithPast

Run Qwen3-VL, including its in-forward CP multimodal path.

Named tensor arguments preserve the Hugging Face Qwen3VLForConditionalGeneration.forward layouts. The internal CP path consumes inputs_embeds of shape [batch, local_sequence, hidden], visual_pos_masks of shape [batch, local_sequence], and DeepStack tensors of shape [local_visual_tokens, hidden].

nemo_automodel.components.models.qwen3_vl.model.Qwen3VLForConditionalGeneration.prepare_model_inputs_for_cp(
    batch: dict[str, typing.Any],
    num_chunks: int = 1
) -> dict[str, typing.Any]

Return Qwen3-VL’s aux-only sharder and full-sequence mRoPE positions.

Embedding, vision encoding, DeepStack construction, and differentiable sequence sharding run inside :meth:forward per microbatch. This hook only computes metadata and selects the round-robin CP strategy, so it does not touch model weights or create an autograd graph shared across microbatches.

Parameters:

batch

dict[str, Any]

Full-sequence input mapping. input_ids has shape [batch, sequence]; optional grid metadata has shape [num_media, 3].

num_chunks

intDefaults to 1

Accepted for the common model-hook contract; unused by the round-robin Qwen3-VL strategy.

Returns: dict[str, Any]

Mapping containing a :class:ContextParallelSharder, optional

Raises:

ValueError: If input_ids is missing or multimodal position metadata is incomplete.

nemo_automodel.components.models.qwen3_vl.model.Qwen3VLForConditionalGeneration.tie_weights(
    _args: object = (),
    _kwargs: object = {}
) -> None

Tie the language head to the active text embedding when configured.

nemo_automodel.components.models.qwen3_vl.model.ModelClass = Qwen3VLForConditionalGeneration