> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.models.qwen3_5_moe.model

Qwen3.5-MoE (VL) NeMo Automodel support.

## Module Contents

### Classes

| Name                                                                                                                                     | Description                                                                   |
| ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| [`Fp32SafeQwen3_5MoeTextRotaryEmbedding`](#nemo_automodel-components-models-qwen3_5_moe-model-Fp32SafeQwen3_5MoeTextRotaryEmbedding)     | Ensure inv\_freq stays in float32 across `.to(dtype)` calls.                  |
| [`Fp32SafeQwen3_5MoeVisionRotaryEmbedding`](#nemo_automodel-components-models-qwen3_5_moe-model-Fp32SafeQwen3_5MoeVisionRotaryEmbedding) | Ensure the vision rotary inv\_freq buffer remains float32.                    |
| [`Qwen3_5MoeBlock`](#nemo_automodel-components-models-qwen3_5_moe-model-Qwen3_5MoeBlock)                                                 | Block that uses the Qwen3.5-MoE native GatedDeltaNet (separate in\_proj\_qkv, |
| [`Qwen3_5MoeCausalLMOutputWithPast`](#nemo_automodel-components-models-qwen3_5_moe-model-Qwen3_5MoeCausalLMOutputWithPast)               | Qwen3.5-MoE output extended with MTP auxiliary hidden states.                 |
| [`Qwen3_5MoeForCausalLM`](#nemo_automodel-components-models-qwen3_5_moe-model-Qwen3_5MoeForCausalLM)                                     | Text-only Qwen3.5-MoE causal language model.                                  |
| [`Qwen3_5MoeForConditionalGeneration`](#nemo_automodel-components-models-qwen3_5_moe-model-Qwen3_5MoeForConditionalGeneration)           | Qwen3.5-MoE VL conditional generation model using NeMo backend components.    |
| [`Qwen3_5MoeMTPSublayer`](#nemo_automodel-components-models-qwen3_5_moe-model-Qwen3_5MoeMTPSublayer)                                     | One full-attention Qwen3.5-MoE MTP sublayer.                                  |
| [`Qwen3_5MoeModel`](#nemo_automodel-components-models-qwen3_5_moe-model-Qwen3_5MoeModel)                                                 | Thin wrapper that exposes `language_model` internals as properties            |
| [`Qwen3_5MoeTextModelBackend`](#nemo_automodel-components-models-qwen3_5_moe-model-Qwen3_5MoeTextModelBackend)                           | Qwen3.5-MoE text decoder rebuilt on top of the Qwen3-Next Block.              |
| [`_Qwen3_5MoeAttention`](#nemo_automodel-components-models-qwen3_5_moe-model-_Qwen3_5MoeAttention)                                       | Qwen3.5-MoE full attention with packed block-diagonal CP dispatch.            |

### Functions

| Name                                                                                                                     | Description                                                       |
| ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- |
| [`_default_init_device`](#nemo_automodel-components-models-qwen3_5_moe-model-_default_init_device)                       | -                                                                 |
| [`_freqs_cis_from_rotary`](#nemo_automodel-components-models-qwen3_5_moe-model-_freqs_cis_from_rotary)                   | -                                                                 |
| [`_make_missing`](#nemo_automodel-components-models-qwen3_5_moe-model-_make_missing)                                     | -                                                                 |
| [`_make_mtp_block_config`](#nemo_automodel-components-models-qwen3_5_moe-model-_make_mtp_block_config)                   | -                                                                 |
| [`_mask_mtp_embed_inputs`](#nemo_automodel-components-models-qwen3_5_moe-model-_mask_mtp_embed_inputs)                   | Zero invalid future-token embeddings in the local CP layout.      |
| [`_qwen3_5_moe_backend`](#nemo_automodel-components-models-qwen3_5_moe-model-_qwen3_5_moe_backend)                       | Return a Qwen3.5-MoE backend with TE fused RoPE disabled.         |
| [`_resolve_mtp_num_layers`](#nemo_automodel-components-models-qwen3_5_moe-model-_resolve_mtp_num_layers)                 | -                                                                 |
| [`_rolled_embed_inputs`](#nemo_automodel-components-models-qwen3_5_moe-model-_rolled_embed_inputs)                       | -                                                                 |
| [`_split_qwen3_5_moe_position_ids`](#nemo_automodel-components-models-qwen3_5_moe-model-_split_qwen3_5_moe_position_ids) | -                                                                 |
| [`build_mtp_config_from_hf`](#nemo_automodel-components-models-qwen3_5_moe-model-build_mtp_config_from_hf)               | Build Qwen3.5-MoE MTP runtime config from HF-style config fields. |
| [`build_qwen3_5_moe_mtp`](#nemo_automodel-components-models-qwen3_5_moe-model-build_qwen3_5_moe_mtp)                     | Construct Qwen3.5-MoE MTP blocks.                                 |

### Data

[`ModelClass`](#nemo_automodel-components-models-qwen3_5_moe-model-ModelClass)

[`_QWEN3_5_MOE_HF_AVAILABLE`](#nemo_automodel-components-models-qwen3_5_moe-model-_QWEN3_5_MOE_HF_AVAILABLE)

### API

```python
class nemo_automodel.components.models.qwen3_5_moe.model.Fp32SafeQwen3_5MoeTextRotaryEmbedding()
```

**Bases:** `Qwen3_5MoeTextRotaryEmbedding`

Ensure inv\_freq stays in float32 across `.to(dtype)` calls.

```python
nemo_automodel.components.models.qwen3_5_moe.model.Fp32SafeQwen3_5MoeTextRotaryEmbedding._apply(
    fn: typing.Any,
    recurse: bool = True
)
```

```python
class nemo_automodel.components.models.qwen3_5_moe.model.Fp32SafeQwen3_5MoeVisionRotaryEmbedding()
```

**Bases:** `Qwen3_5MoeVisionRotaryEmbedding`

Ensure the vision rotary inv\_freq buffer remains float32.

```python
nemo_automodel.components.models.qwen3_5_moe.model.Fp32SafeQwen3_5MoeVisionRotaryEmbedding._apply(
    fn: typing.Any,
    recurse: bool = True
)
```

```python
class nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeBlock(
    layer_idx,
    config,
    moe_config,
    backend
)
```

**Bases:** [Block](/nemo-automodel/nemo_automodel/components/models/qwen3_next/model#nemo_automodel-components-models-qwen3_next-model-Block)

Block that uses the Qwen3.5-MoE native GatedDeltaNet (separate in\_proj\_qkv,
in\_proj\_z, in\_proj\_b, in\_proj\_a)

**`linear_attn`** `= CPAwareGatedDeltaNet(config, layer_idx)`

---

**`self_attn`** `= _Qwen3_5MoeAttention(config, layer_idx, backend)`

---

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeBlock.forward(
    x: torch.Tensor,
    freqs_cis: torch.Tensor,
    attention_mask: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    attn_kwargs: typing.Any = {}
) -> torch.Tensor
```

Mirror :meth:`Block.forward` but thread NEAT-packing kwargs into
`CPAwareGatedDeltaNet`.

The parent `Block.forward` calls `linear_attn` with only
`hidden_states` and `attention_mask`; for packed sequences the
gated\_delta\_rule kernel additionally needs `cu_seqlens` /
`indices` to reset state at document boundaries (issue #2131).
Derived once per forward from the indexed attention mask.

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeBlock.init_weights(
    buffer_device: torch.device
)
```

```python
class nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeCausalLMOutputWithPast(
    mtp_per_depth_h: list[torch.Tensor] | None = None,
    mtp_loss_scaling_factor: float | None = None
)
```

Dataclass

**Bases:** `CausalLMOutputWithPast`

Qwen3.5-MoE output extended with MTP auxiliary hidden states.

**`mtp_loss_scaling_factor`** `float | None = None`

---

**`mtp_per_depth_h`** `list[Tensor] | None = None`

---

```python
class nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForCausalLM(
    config: transformers.models.qwen3_5_moe.configuration_qwen3_5_moe.Qwen3_5MoeTextConfig,
    moe_config: nemo_automodel.components.moe.layers.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    mtp_loss_scaling_factor: float = 0.1,
    num_nextn_predict_layers: int | None = None,
    kwargs: typing.Any = {}
)
```

**Bases:** [HFCheckpointingMixin](/nemo-automodel/nemo_automodel/components/models/common/hf_checkpointing_mixin#nemo_automodel-components-models-common-hf_checkpointing_mixin-HFCheckpointingMixin), `Module`, [MoEFSDPSyncMixin](/nemo-automodel/nemo_automodel/components/moe/fsdp_mixin#nemo_automodel-components-moe-fsdp_mixin-MoEFSDPSyncMixin)

Text-only Qwen3.5-MoE causal language model.

**`_pp_keep_self_forward`** `bool = True`

---

**`backend`** `= _qwen3_5_moe_backend(backend)`

---

**`lm_head`**

---

**`model`**

---

**`moe_config`** `= self.model.moe_config`

---

**`mtp`**

---

**`mtp_config`**

---

**`pad_token_id`**

---

**`state_dict_adapter`**

---

**`tie_word_embeddings_support`** `TieSupport = TieSupport.UNTIED_ONLY`

---

**`vocab_size`** `= config.vocab_size`

---

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForCausalLM.forward(
    input_ids: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.LongTensor | None = None,
    past_key_values: typing.Any | None = None,
    inputs_embeds: torch.FloatTensor | None = None,
    labels: torch.LongTensor | None = None,
    mtp_per_depth_input_ids: tuple[torch.LongTensor, ...] | None = None,
    mtp_per_depth_position_ids: tuple[torch.LongTensor, ...] | None = None,
    use_cache: bool | None = None,
    logits_to_keep: int | torch.Tensor = 0,
    output_hidden_states: bool | None = None,
    cache_position: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    kwargs: typing.Any = {}
) -> nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeCausalLMOutputWithPast
```

Run the text-only causal LM forward pass.

**Parameters:**

**`input_ids`** `torch.Tensor | None` — default: None

Token ids of shape `[batch, sequence]`. On a non-first
pipeline stage where `embed_tokens` is absent, this argument
instead carries hidden states of shape
`[batch, sequence, hidden]` and is routed as `inputs_embeds`.

---

**`attention_mask`** `torch.Tensor | None` — default: None

Optional mask of shape `[batch, sequence]` or
`[batch, 1, sequence, sequence]`.

---

**`position_ids`** `torch.LongTensor | None` — default: None

Optional M-RoPE positions of shape
`[axes, batch, sequence]` or `[batch, sequence]`.

---

**`past_key_values`** `Any | None` — default: None

Optional KV-cache state; caching is not supported
by this backend and a non-`None` value raises.

---

**`inputs_embeds`** `torch.FloatTensor | None` — default: None

Optional hidden inputs of shape
`[batch, sequence, hidden]`.

---

**`labels`** `torch.LongTensor | None` — default: None

Optional labels of shape `[batch, sequence]`. Accepted for
Hugging Face compatibility; loss is computed outside this model.

---

**`mtp_per_depth_input_ids`** `tuple[torch.LongTensor, ...] | None` — default: None

Optional globally shifted then CP-sharded
token IDs, one tensor of shape `[batch, sequence]` per depth.

---

**`mtp_per_depth_position_ids`** `tuple[torch.LongTensor, ...] | None` — default: None

Optional globally shifted then CP-sharded
positions, one tensor of shape `[batch, sequence]` per depth.

---

**`use_cache`** `bool | None` — default: None

Whether to use a KV cache; `True` is unsupported.

---

**`logits_to_keep`** `int | torch.Tensor` — default: 0

`0` to project every position, a positive integer
to keep the last positions, or indices of shape
`[kept_sequence]`.

---

**`output_hidden_states`** `bool | None` — default: None

Whether to include final decoder states in the
output for the fused loss path.

---

**`cache_position`** `torch.Tensor | None` — default: None

Optional token positions of shape `[sequence]`.

---

**`padding_mask`** `torch.Tensor | None` — default: None

Optional padding mask of shape `[batch, sequence]`
where `True` marks padding.

---

**`**kwargs`** `Any` — default: \{}

Additional attention and packed-sequence metadata forwarded
to the decoder and optional MTP layers.

---

**Returns:** `Qwen3_5MoeCausalLMOutputWithPast`

A causal-LM output whose logits have shape

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForCausalLM.from_config(
    config: transformers.models.qwen3_5_moe.configuration_qwen3_5_moe.Qwen3_5MoeTextConfig,
    moe_config: nemo_automodel.components.moe.layers.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    kwargs: typing.Any = {}
)
```

classmethod

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForCausalLM.from_pretrained(
    pretrained_model_name_or_path: str,
    model_args: typing.Any = (),
    kwargs: typing.Any = {}
)
```

classmethod

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForCausalLM.get_input_embeddings() -> torch.nn.Module
```

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForCausalLM.get_output_embeddings() -> torch.nn.Module
```

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForCausalLM.initialize_weights(
    buffer_device: torch.device | None = None,
    dtype: torch.dtype = torch.bfloat16
) -> None
```

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForCausalLM.prepare_mtp_inputs_for_cp(
    batch: dict[str, typing.Any],
    ignore_index: int = -100
) -> nemo_automodel.components.models.common.mtp.MTPContextParallelInputs
```

Prepare text-model future-token streams before CP sharding.

**Parameters:**

**`batch`** `dict[str, Any]`

Full-sequence batch whose `input_ids`, `labels`, and
optional `position_ids` tensors have shape `[batch, sequence]`.

---

**`ignore_index`** `int` — default: -100

Fill value for invalid targets at trailing and packed
document-boundary positions.

---

**Returns:** `MTPContextParallelInputs`

Global per-depth token IDs, position IDs, targets, and validity masks.

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForCausalLM.set_input_embeddings(
    value: torch.nn.Module
) -> None
```

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForCausalLM.set_output_embeddings(
    new_embeddings: torch.nn.Module
) -> None
```

```python
class nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForConditionalGeneration(
    config: transformers.models.qwen3_5_moe.configuration_qwen3_5_moe.Qwen3_5MoeConfig,
    moe_config: nemo_automodel.components.moe.layers.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    mtp_loss_scaling_factor: float = 0.1,
    num_nextn_predict_layers: int | None = None,
    kwargs = {}
)
```

**Bases:** [HFCheckpointingMixin](/nemo-automodel/nemo_automodel/components/models/common/hf_checkpointing_mixin#nemo_automodel-components-models-common-hf_checkpointing_mixin-HFCheckpointingMixin), `HFQwen3_5MoeForConditionalGeneration`, [MoEFSDPSyncMixin](/nemo-automodel/nemo_automodel/components/moe/fsdp_mixin#nemo_automodel-components-moe-fsdp_mixin-MoEFSDPSyncMixin)

Qwen3.5-MoE VL conditional generation model using NeMo backend components.

**`_keep_in_fp32_modules_strict`** `= ['_fp32_params']`

---

**`_packed_cp_attn_backends`** `= ('sdpa',)`

---

**`_pp_keep_self_forward`** `bool = True`

---

**`lm_head`**

---

**`mtp`**

---

**`mtp_config`**

---

**`pad_token_id`** `= pad_token_id if pad_token_id is not None else -1`

---

**`state_dict_adapter`**

---

**`tie_word_embeddings_support`** `TieSupport = TieSupport.UNTIED_ONLY`

---

**`vocab_size`** `= text_config.vocab_size`

---

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForConditionalGeneration._embed_and_splice_for_cp(
    input_ids: torch.Tensor,
    pixel_values: torch.Tensor | None,
    pixel_values_videos: torch.Tensor | None,
    image_grid_thw: torch.Tensor | None,
    video_grid_thw: torch.Tensor | None
) -> torch.Tensor
```

Embed token ids and splice image/video features into the embeddings.

The VLM->LM multimodal scatter runs on the full (unsharded) sequence
inside the forward before the CP sequence shard, identical to the
pre-CP-refactor pre-embed.

**Parameters:**

**`input_ids`** `torch.Tensor`

Token ids `[batch, sequence]` (full, unsharded).

---

**`pixel_values`** `torch.Tensor | None`

Optional packed image patches for HF vision encoding.

---

**`pixel_values_videos`** `torch.Tensor | None`

Optional packed video patches.

---

**`image_grid_thw`** `torch.Tensor | None`

Per-image `[num_images, 3]` T/H/W grid.

---

**`video_grid_thw`** `torch.Tensor | None`

Per-video `[num_videos, 3]` T/H/W grid.

---

**Returns:** `torch.Tensor`

`inputs_embeds` of shape `[batch, sequence, hidden]` with image /

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForConditionalGeneration._encode_vision_for_cp(
    pixel_values: torch.Tensor,
    grid_thw: torch.Tensor,
    is_video: bool
) -> torch.Tensor
```

Encode one modality into flat, entry-ordered visual tokens under CP.

**Parameters:**

**`pixel_values`** `torch.Tensor`

Patch rows of shape `[total_patch_rows, patch_dim]`
for all image or video entries, frame-contiguous in entry order.

---

**`grid_thw`** `torch.Tensor`

Per-entry temporal/height/width grids of shape
`[num_entries, 3]`.

---

**`is_video`** `bool`

Whether the replicated fallback uses the video feature
helper instead of the image feature helper.

---

**Returns:** `torch.Tensor`

Flat merged-token embeddings of shape `[visual_tokens, hidden]`

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForConditionalGeneration.forward(
    input_ids: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    inputs_embeds: torch.Tensor | None = None,
    cache_position: torch.Tensor | None = None,
    mtp_per_depth_input_ids: tuple[torch.LongTensor, ...] | None = None,
    mtp_per_depth_position_ids: tuple[torch.LongTensor, ...] | None = None,
    mtp_per_depth_valid_masks: tuple[torch.BoolTensor, ...] | None = None,
    logits_to_keep: typing.Union[int, torch.Tensor] = 0,
    output_hidden_states: bool | None = None,
    kwargs: typing.Any = {}
)
```

Run Qwen3.5-MoE VLM generation with model-owned CP embedding.

**Parameters:**

**`input_ids`** `torch.Tensor | None` — default: None

Token ids of shape `[batch, sequence]`. Under CP this
remains full-length until multimodal features are spliced.

---

**`position_ids`** `torch.Tensor | None` — default: None

Optional mRoPE positions of shape
`[axes, batch, sequence]`; under CP, `sequence` is local.

---

**`attention_mask`** `torch.Tensor | None` — default: None

Optional mask of shape `[batch, sequence]` or
`[batch, 1, sequence, sequence]`.

---

**`padding_mask`** `torch.Tensor | None` — default: None

Optional local padding mask of shape
`[batch, sequence]` where `True` marks padding.

---

**`inputs_embeds`** `torch.Tensor | None` — default: None

Optional embeddings of shape
`[batch, sequence, hidden]`.

---

**`cache_position`** `torch.Tensor | None` — default: None

Optional token positions of shape `[sequence]`.

---

**`mtp_per_depth_input_ids`** `tuple[torch.LongTensor, ...] | None` — default: None

Optional globally shifted then CP-sharded
token IDs, one tensor of shape `[batch, sequence]` per depth.

---

**`mtp_per_depth_position_ids`** `tuple[torch.LongTensor, ...] | None` — default: None

Optional globally shifted then CP-sharded
mRoPE positions, one tensor of shape
`[axes, batch, sequence]` per depth.

---

**`mtp_per_depth_valid_masks`** `tuple[torch.BoolTensor, ...] | None` — default: None

Optional CP-sharded masks, one tensor of
shape `[batch, sequence]` per depth, where `True` marks a
valid same-document future token.

---

**`logits_to_keep`** `Union[int, torch.Tensor]` — default: 0

Number of trailing logits to retain, or indices of
shape `[kept_sequence]`.

---

**`output_hidden_states`** `bool | None` — default: None

Whether to expose final decoder states for the
fused loss path.

---

**`**kwargs`** `Any` — default: \{}

Model inputs including image/video patch rows of shape
`[total_patch_rows, patch_dim]`, their grids of shape
`[num_entries, 3]`, and optional packed document ids of shape
`[batch, sequence]`.

---

**Returns:**

A causal-LM output whose logits have shape

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForConditionalGeneration.from_config(
    config: transformers.models.qwen3_5_moe.configuration_qwen3_5_moe.Qwen3_5MoeConfig,
    moe_config: nemo_automodel.components.moe.layers.MoEConfig | None = None,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    kwargs = {}
)
```

classmethod

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForConditionalGeneration.from_pretrained(
    pretrained_model_name_or_path: str,
    model_args = (),
    kwargs = {}
)
```

classmethod

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForConditionalGeneration.get_pipeline_kwargs_chunk_dims(
    kwargs: dict[str, typing.Any]
) -> dict[str, int]
```

Return model-owned PP microbatch split dims for keyword inputs.

Qwen3.5-MoE mRoPE position ids can arrive as `[T/H/W, batch, seq]` or
`[text/T/H/W, batch, seq]`. The PP schedule must split that tensor on
the batch axis, while all standard batch-major kwargs keep the framework
default.

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForConditionalGeneration.get_pipeline_stage_metas(
    is_first: bool,
    microbatch_size: int,
    seq_len: int,
    dtype: torch.dtype
) -> tuple[tuple[torch.Tensor, ...], tuple[torch.Tensor, ...]]
```

Per-stage input/output meta tensors for the PP schedule's shape inference.

Matches the framework default except that under context parallelism the
first stage embeds the full sequence and shards it in forward, so every
stage output and later-stage input carries the LOCAL (padded-to-`2*cp`
then `//cp`) sequence length while the first-stage input stays
full-length. At `cp_size == 1` this reduces to the default symmetric
shapes.

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForConditionalGeneration.initialize_weights(
    buffer_device: torch.device | None = None,
    dtype: torch.dtype = torch.bfloat16
) -> None
```

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForConditionalGeneration.prepare_model_inputs_for_cp(
    batch: dict[str, typing.Any],
    num_chunks: int = 1
) -> dict[str, typing.Any]
```

Return a sharder-only CP backend plus the full-sequence mRoPE positions.

Embedding and the VLM->LM multimodal scatter now run inside `forward`
per microbatch (see :meth:`_embed_and_splice_for_cp`), so this hook only
computes the mRoPE `position_ids` on the full (unsharded) sequence via
`get_rope_index` and returns them with a
:class:`ContextParallelSharder`. Packed SDPA batches select contiguous
block-diagonal CP; ordinary batches retain the round-robin sharder.
`input_ids` and media stay in the batch for the forward, while
`mm_token_type_ids` is consumed here (only `get_rope_index` needs it).

**Parameters:**

**`batch`** `dict[str, Any]`

The full-sequence batch. Tensor fields consumed here are
`input_ids` of shape `[batch, sequence]`, optional
`attention_mask` of shape `[batch, sequence]` or
`[batch, 1, sequence, sequence]`, optional `position_ids`
of shape `[axes, batch, sequence]`, and image/video grids of
shape `[num_entries, 2 or 3]`.

---

**`num_chunks`** `int` — default: 1

Accepted for hook-signature parity; unused.

---

**Returns:** `dict[str, Any]`

A mapping containing the selected `cp_sharder`, full-sequence

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeForConditionalGeneration.prepare_mtp_inputs_for_cp(
    batch: dict[str, typing.Any],
    ignore_index: int = -100
) -> nemo_automodel.components.models.common.mtp.MTPContextParallelInputs
```

Prepare Qwen VLM future-token streams before CP sharding.

Call this after :meth:`prepare_model_inputs_for_cp` has materialized
full-sequence mRoPE positions.

**Parameters:**

**`batch`** `dict[str, Any]`

Full-sequence batch whose `input_ids` and `labels` tensors
have shape `[batch, sequence]` and whose `position_ids` tensor
has shape `[axes, batch, sequence]`.

---

**`ignore_index`** `int` — default: -100

Fill value for invalid targets at trailing and packed
document-boundary positions.

---

**Returns:** `MTPContextParallelInputs`

Global per-depth token IDs, mRoPE position IDs, targets, and validity

```python
class nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeMTPSublayer(
    layer_idx: int,
    config: transformers.models.qwen3_5_moe.configuration_qwen3_5_moe.Qwen3_5MoeTextConfig,
    moe_config: nemo_automodel.components.moe.layers.MoEConfig,
    backend: nemo_automodel.components.models.common.BackendConfig,
    has_fusion: bool = False,
    has_final_norm: bool = False,
    dtype: torch.dtype = torch.bfloat16
)
```

**Bases:** [Qwen3\_5MoeBlock](#nemo_automodel-components-models-qwen3_5_moe-model-Qwen3_5MoeBlock)

One full-attention Qwen3.5-MoE MTP sublayer.

**`eh_proj`**

---

**`enorm`**

---

**`final_layernorm`**

---

**`hnorm`**

---

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeMTPSublayer.forward(
    hidden_states: torch.Tensor,
    embed_input: torch.Tensor | None = None,
    rotary_emb: torch.nn.Module,
    position_ids: torch.Tensor,
    attention_mask: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    attn_kwargs: typing.Any = {}
) -> torch.Tensor
```

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeMTPSublayer.init_weights(
    buffer_device: torch.device
) -> None
```

```python
class nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeModel()
```

**Bases:** `HFQwen3_5MoeModel`

Thin wrapper that exposes `language_model` internals as properties
expected by the NeMo training loop (e.g. `model.layers`).

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeModel.forward(
    input_ids = None,
    attention_mask = None,
    position_ids = None,
    past_key_values = None,
    inputs_embeds = None,
    pixel_values = None,
    pixel_values_videos = None,
    image_grid_thw = None,
    video_grid_thw = None,
    cache_position = None,
    kwargs = {}
)
```

```python
class nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeTextModelBackend(
    config: transformers.models.qwen3_5_moe.configuration_qwen3_5_moe.Qwen3_5MoeTextConfig,
    backend: nemo_automodel.components.models.common.BackendConfig,
    moe_config: nemo_automodel.components.moe.layers.MoEConfig | None = None,
    moe_overrides: dict | None = None
)
```

**Bases:** `Module`

Qwen3.5-MoE text decoder rebuilt on top of the Qwen3-Next Block.

**`embed_tokens`**

---

**`layers`**

---

**`moe_config`** `= moe_config or MoEConfig(**moe_defaults)`

---

**`norm`**

---

**`padding_idx`** `= getattr(config, 'pad_token_id', None)`

---

**`rotary_emb`**

---

**`vocab_size`** `= config.vocab_size`

---

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeTextModelBackend.forward(
    input_ids: torch.Tensor | None = None,
    inputs_embeds: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    cache_position: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    past_key_values: typing.Any | None = None,
    use_cache: bool | None = None,
    attn_kwargs: typing.Any = {}
) -> transformers.models.qwen3_5_moe.modeling_qwen3_5_moe.Qwen3_5MoeModelOutputWithPast
```

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeTextModelBackend.get_input_embeddings() -> torch.nn.Module
```

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeTextModelBackend.init_weights(
    buffer_device: torch.device | None = None
) -> None
```

```python
nemo_automodel.components.models.qwen3_5_moe.model.Qwen3_5MoeTextModelBackend.set_input_embeddings(
    value: torch.nn.Module
) -> None
```

```python
class nemo_automodel.components.models.qwen3_5_moe.model._Qwen3_5MoeAttention(
    args: typing.Any = (),
    kwargs: typing.Any = {}
)
```

**Bases:** [Qwen3NextAttention](/nemo-automodel/nemo_automodel/components/models/qwen3_next/layers#nemo_automodel-components-models-qwen3_next-layers-Qwen3NextAttention)

Qwen3.5-MoE full attention with packed block-diagonal CP dispatch.

**`_base_attn_func`** `= self.attn_func`

---

**`attn_func`** `= self._dispatch_attention`

---

```python
nemo_automodel.components.models.qwen3_5_moe.model._Qwen3_5MoeAttention._dispatch_attention(
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    attn_kwargs: typing.Any = {}
) -> torch.Tensor
```

Route preprocessed QKV through packed CP or the configured backend.

**Parameters:**

**`query`** `torch.Tensor`

Query states of shape `[batch, heads, local_sequence, head_dim]`.

---

**`key`** `torch.Tensor`

Key states of shape `[batch, kv_heads, local_sequence, head_dim]`.

---

**`value`** `torch.Tensor`

Value states with the same layout as `key`.

---

**`**attn_kwargs`** `Any` — default: \{}

Keyword arguments produced by the parent attention's
backend preprocessing.

---

**Returns:** `torch.Tensor`

Attention output of shape

```python
nemo_automodel.components.models.qwen3_5_moe.model._default_init_device() -> torch.device
```

```python
nemo_automodel.components.models.qwen3_5_moe.model._freqs_cis_from_rotary(
    rotary_emb: torch.nn.Module,
    hidden_states: torch.Tensor,
    position_ids: torch.Tensor
) -> torch.Tensor
```

```python
nemo_automodel.components.models.qwen3_5_moe.model._make_missing(
    name: str
)
```

```python
nemo_automodel.components.models.qwen3_5_moe.model._make_mtp_block_config(
    config: transformers.models.qwen3_5_moe.configuration_qwen3_5_moe.Qwen3_5MoeTextConfig,
    layer_idx: int
) -> transformers.models.qwen3_5_moe.configuration_qwen3_5_moe.Qwen3_5MoeTextConfig
```

```python
nemo_automodel.components.models.qwen3_5_moe.model._mask_mtp_embed_inputs(
    embed_inputs: tuple[torch.Tensor, ...],
    valid_masks: tuple[torch.BoolTensor, ...]
) -> tuple[torch.Tensor, ...]
```

Zero invalid future-token embeddings in the local CP layout.

**Parameters:**

**`embed_inputs`** `tuple[torch.Tensor, ...]`

Per-depth embeddings of shape `[batch, sequence, hidden]`.

---

**`valid_masks`** `tuple[torch.BoolTensor, ...]`

Per-depth masks of shape `[batch, sequence]` in the same
local CP token layout as `embed_inputs`.

---

**Returns:** `torch.Tensor`

Per-depth embeddings of shape `[batch, sequence, hidden]` with invalid

```python
nemo_automodel.components.models.qwen3_5_moe.model._qwen3_5_moe_backend(
    backend: nemo_automodel.components.models.common.BackendConfig | None = None
) -> nemo_automodel.components.models.common.BackendConfig
```

Return a Qwen3.5-MoE backend with TE fused RoPE disabled.

The Qwen3.5 full-attention blocks reuse Qwen3-Next attention, and VLM/packed
execution can present THD-shaped q/k tensors. TE fused RoPE expects 4D inputs
in this path, so use non-fused RoPE while preserving the rest of the backend.

```python
nemo_automodel.components.models.qwen3_5_moe.model._resolve_mtp_num_layers(
    config: typing.Any,
    override: int | None = None
) -> int
```

```python
nemo_automodel.components.models.qwen3_5_moe.model._rolled_embed_inputs(
    inputs_embeds: torch.Tensor,
    num_depths: int
) -> tuple[torch.Tensor, ...]
```

```python
nemo_automodel.components.models.qwen3_5_moe.model._split_qwen3_5_moe_position_ids(
    position_ids: torch.Tensor | None,
    batch_size: int,
    seq_len: int,
    device: torch.device,
    cache_position: torch.Tensor | None = None
) -> torch.Tensor
```

```python
nemo_automodel.components.models.qwen3_5_moe.model.build_mtp_config_from_hf(
    config: typing.Any,
    loss_scaling_factor: float = 0.1,
    num_nextn_predict_layers: int | None = None
) -> nemo_automodel.components.models.common.mtp.MTPConfig
```

Build Qwen3.5-MoE MTP runtime config from HF-style config fields.

```python
nemo_automodel.components.models.qwen3_5_moe.model.build_qwen3_5_moe_mtp(
    config: transformers.models.qwen3_5_moe.configuration_qwen3_5_moe.Qwen3_5MoeTextConfig,
    mtp_config: nemo_automodel.components.models.common.mtp.MTPConfig,
    backend: nemo_automodel.components.models.common.BackendConfig,
    moe_config: nemo_automodel.components.moe.layers.MoEConfig,
    dtype: torch.dtype
) -> nemo_automodel.components.models.common.mtp.MTPModule
```

Construct Qwen3.5-MoE MTP blocks.

```python
nemo_automodel.components.models.qwen3_5_moe.model.ModelClass = Qwen3_5MoeForConditionalGeneration
```

```python
nemo_automodel.components.models.qwen3_5_moe.model._QWEN3_5_MOE_HF_AVAILABLE = True
```