> This page is for version 0.5.0 · 26.06.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.models.qwen3_vl_moe.model

## Module Contents

### Classes

| Name                                                                                                                                      | Description                                                                   |
| ----------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| [`Fp32SafeQwen3VLMoeTextRotaryEmbedding`](#nemo_automodel-components-models-qwen3_vl_moe-model-Fp32SafeQwen3VLMoeTextRotaryEmbedding)     | Ensure inv\_freq stays in float32                                             |
| [`Fp32SafeQwen3VLMoeVisionRotaryEmbedding`](#nemo_automodel-components-models-qwen3_vl_moe-model-Fp32SafeQwen3VLMoeVisionRotaryEmbedding) | Ensure the vision rotary inv\_freq buffer remains float32.                    |
| [`Qwen3VLMoeBlock`](#nemo_automodel-components-models-qwen3_vl_moe-model-Qwen3VLMoeBlock)                                                 | Qwen3-VL block adapter that accepts HF-style position embeddings.             |
| [`Qwen3VLMoeForConditionalGeneration`](#nemo_automodel-components-models-qwen3_vl_moe-model-Qwen3VLMoeForConditionalGeneration)           | Qwen3-VL conditional generation model using the Qwen3-MoE backend components. |
| [`Qwen3VLMoeModel`](#nemo_automodel-components-models-qwen3_vl_moe-model-Qwen3VLMoeModel)                                                 | -                                                                             |
| [`Qwen3VLMoeTextModelBackend`](#nemo_automodel-components-models-qwen3_vl_moe-model-Qwen3VLMoeTextModelBackend)                           | Qwen3-VL text decoder rebuilt on top of the Qwen3-MoE block implementation.   |

### Data

[`ModelClass`](#nemo_automodel-components-models-qwen3_vl_moe-model-ModelClass)

### API

```python
class nemo_automodel.components.models.qwen3_vl_moe.model.Fp32SafeQwen3VLMoeTextRotaryEmbedding()
```

**Bases:** `Qwen3VLMoeTextRotaryEmbedding`

Ensure inv\_freq stays in float32

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Fp32SafeQwen3VLMoeTextRotaryEmbedding._apply(
    fn: typing.Any,
    recurse: bool = True
)
```

```python
class nemo_automodel.components.models.qwen3_vl_moe.model.Fp32SafeQwen3VLMoeVisionRotaryEmbedding()
```

**Bases:** `Qwen3VLMoeVisionRotaryEmbedding`

Ensure the vision rotary inv\_freq buffer remains float32.

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Fp32SafeQwen3VLMoeVisionRotaryEmbedding._apply(
    fn: typing.Any,
    recurse: bool = True
)
```

```python
class nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeBlock()
```

**Bases:** [Block](/nemo-automodel/nemo_automodel/components/models/qwen3_moe/model#nemo_automodel-components-models-qwen3_moe-model-Block)

Qwen3-VL block adapter that accepts HF-style position embeddings.

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeBlock.forward(
    x: torch.Tensor,
    freqs_cis: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
    attn_kwargs: typing.Any = {}
) -> torch.Tensor
```

```python
class nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration(
    config: transformers.models.qwen3_vl_moe.configuration_qwen3_vl_moe.Qwen3VLMoeConfig,
    moe_config: nemo_automodel.components.moe.config.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    kwargs = {}
)
```

**Bases:** [HFCheckpointingMixin](/nemo-automodel/nemo_automodel/components/models/common/hf_checkpointing_mixin#nemo_automodel-components-models-common-hf_checkpointing_mixin-HFCheckpointingMixin), `HFQwen3VLMoeForConditionalGeneration`, [MoEFSDPSyncMixin](/nemo-automodel/nemo_automodel/components/moe/fsdp_mixin#nemo_automodel-components-moe-fsdp_mixin-MoEFSDPSyncMixin)

Qwen3-VL conditional generation model using the Qwen3-MoE backend components.

**`_pp_keep_self_forward`** `bool = True`

---

**`lm_head`**

---

**`pad_token_id`** `= pad_token_id if pad_token_id is not None else -1`

---

**`state_dict_adapter`**

---

**`tie_word_embeddings_support`** `TieSupport = TieSupport.UNTIED_ONLY`

---

**`vocab_size`** `= text_config.vocab_size`

---

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.forward(
    input_ids: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    inputs_embeds: torch.Tensor | None = None,
    cache_position: torch.Tensor | None = None,
    logits_to_keep: int | torch.Tensor = 0,
    output_hidden_states: bool | None = None,
    kwargs: typing.Any = {}
)
```

Run text and media inputs while preserving their distinct leading axes.

**Parameters:**

**`input_ids`** `torch.Tensor | None` — default: None

Token IDs of shape \[batch, sequence] or \[tokens] for THD.

---

**`position_ids`** `torch.Tensor | None` — default: None

Rotary coordinates of shape \[3, batch, sequence],
with a placeholder batch axis of size one for THD.

---

**`attention_mask`** `torch.Tensor | None` — default: None

Optional binary padding mask of shape \[batch, sequence].
Dense NEAT masks are unsupported; packed inputs use THD with
`qkv_format="thd"` and `attention_mask=None`.

---

**`padding_mask`** `torch.Tensor | None` — default: None

Optional padding flags of shape \[batch, sequence]
or \[tokens] for THD.

---

**`inputs_embeds`** `torch.Tensor | None` — default: None

Optional embeddings of shape \[batch, sequence, hidden]
supplied by a preceding pipeline stage.

---

**`cache_position`** `torch.Tensor | None` — default: None

Optional token positions of shape \[sequence].

---

**`logits_to_keep`** `int | torch.Tensor` — default: 0

Number of trailing logits, or token indices of shape
\[selected\_tokens], to retain.

---

**`output_hidden_states`** `bool | None` — default: None

Return hidden states for fused loss computation.

---

**`**kwargs`** `Any` — default: \{}

Attention metadata, including `cu_seqlens` and
`cu_seqlens_padded` of shape \[documents + 1] (optionally with
a leading placeholder batch axis), and scalar `max_seqlen`.
Media tensors retain `pixel_values`/`pixel_values_videos`
of shape \[patches, patch\_features] and image/video `grid_thw`
of shape \[media\_items, 3], including when media\_items is one.

---

**Returns:**

Model output with logits of shape \[batch, sequence, vocab] or

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.from_config(
    config: transformers.models.qwen3_vl_moe.configuration_qwen3_vl_moe.Qwen3VLMoeConfig,
    moe_config: nemo_automodel.components.moe.config.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    kwargs = {}
)
```

classmethod

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.from_pretrained(
    pretrained_model_name_or_path: str,
    model_args = (),
    kwargs = {}
)
```

classmethod

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.get_input_embeddings()
```

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.get_output_embeddings()
```

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.initialize_weights(
    buffer_device: torch.device | None = None,
    dtype: torch.dtype = torch.bfloat16
) -> None
```

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.set_input_embeddings(
    value
)
```

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeForConditionalGeneration.set_output_embeddings(
    new_embeddings
)
```

```python
class nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeModel()
```

**Bases:** `HFQwen3VLMoeModel`

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeModel.forward(
    input_ids = None,
    attention_mask = None,
    position_ids = None,
    past_key_values = None,
    inputs_embeds = None,
    pixel_values = None,
    pixel_values_videos = None,
    image_grid_thw = None,
    video_grid_thw = None,
    cache_position = None,
    kwargs = {}
)
```

Fuse packed vision patches into padded text sequences.

**Parameters:**

**`input_ids`** — default: None

Integer tensor of shape \[batch, sequence].

---

**`attention_mask`** — default: None

Padding mask of shape \[batch, sequence], or per-layer masks.

---

**`position_ids`** — default: None

Rotary coordinates of shape \[3, batch, sequence].

---

**`past_key_values`** — default: None

Hugging Face decoder cache, unused by the native text backend.

---

**`inputs_embeds`** — default: None

Optional embeddings of shape \[batch, sequence, hidden].

---

**`pixel_values`** — default: None

Image patches of shape \[image\_patches, patch\_features], where
patch\_features flattens channels, temporal depth, height, and width.

---

**`pixel_values_videos`** — default: None

Video patches of shape \[video\_patches, patch\_features].

---

**`image_grid_thw`** — default: None

Integer tensor of shape \[images, 3] with temporal/height/width grids.

---

**`video_grid_thw`** — default: None

Integer tensor of shape \[videos, 3] with temporal/height/width grids.

---

**`cache_position`** — default: None

Optional positions of shape \[sequence].

---

**`**kwargs`** — default: \{}

Backend options, including mm\_token\_type\_ids of shape \[batch, sequence].

---

**Returns:**

Model output with last\_hidden\_state of shape \[batch, sequence, hidden].

```python
class nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeTextModelBackend(
    config: transformers.models.qwen3_vl_moe.configuration_qwen3_vl_moe.Qwen3VLMoeTextConfig,
    backend: nemo_automodel.components.models.common.BackendConfig,
    moe_config: nemo_automodel.components.moe.config.MoEConfig | None = None,
    moe_overrides: dict | None = None
)
```

**Bases:** `Module`

Qwen3-VL text decoder rebuilt on top of the Qwen3-MoE block implementation.

**`embed_tokens`**

---

**`layers`**

---

**`moe_config`** `= moe_config or MoEConfig(**moe_defaults)`

---

**`norm`**

---

**`padding_idx`** `= getattr(config, 'pad_token_id', None)`

---

**`rotary_emb`**

---

**`vocab_size`** `= config.vocab_size`

---

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeTextModelBackend._deepstack_process(
    hidden_states: torch.Tensor,
    visual_pos_masks: torch.Tensor | None,
    visual_embeds: torch.Tensor
) -> torch.Tensor
```

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeTextModelBackend.forward(
    input_ids: torch.Tensor | None = None,
    inputs_embeds: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    cache_position: torch.Tensor | None = None,
    visual_pos_masks: torch.Tensor | None = None,
    deepstack_visual_embeds: list[torch.Tensor] | None = None,
    padding_mask: torch.Tensor | None = None,
    past_key_values: typing.Any | None = None,
    use_cache: bool | None = None,
    attn_kwargs: typing.Any = {}
) -> transformers.models.qwen3_vl_moe.modeling_qwen3_vl_moe.Qwen3VLMoeModelOutputWithPast
```

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeTextModelBackend.get_input_embeddings() -> torch.nn.Module
```

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeTextModelBackend.init_weights(
    buffer_device: torch.device | None = None
) -> None
```

```python
nemo_automodel.components.models.qwen3_vl_moe.model.Qwen3VLMoeTextModelBackend.set_input_embeddings(
    value: torch.nn.Module
) -> None
```

```python
nemo_automodel.components.models.qwen3_vl_moe.model.ModelClass = Qwen3VLMoeForConditionalGeneration
```