> This page is for version 0.5.0 · 26.06.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.models.qwen3_5.model

Qwen3.5 dense causal LM with Megatron-style MTP support.

## Module Contents

### Classes

| Name                                                                                                                       | Description                                                            |
| -------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| [`Fp32SafeQwen3_5TextRotaryEmbedding`](#nemo_automodel-components-models-qwen3_5-model-Fp32SafeQwen3_5TextRotaryEmbedding) | Ensure inv\_freq stays in float32 across `.to(dtype)` calls.           |
| [`Qwen3_5CausalLMOutputWithPast`](#nemo_automodel-components-models-qwen3_5-model-Qwen3_5CausalLMOutputWithPast)           | Qwen3.5 causal-LM output extended with MTP auxiliary hidden states.    |
| [`Qwen3_5DenseBlock`](#nemo_automodel-components-models-qwen3_5-model-Qwen3_5DenseBlock)                                   | Qwen3.5 dense decoder block on top of the Qwen3-Next `Block`.          |
| [`Qwen3_5DenseMTPSublayer`](#nemo_automodel-components-models-qwen3_5-model-Qwen3_5DenseMTPSublayer)                       | One full-attention Qwen3.5 dense MTP sublayer.                         |
| [`Qwen3_5DenseTextBackbone`](#nemo_automodel-components-models-qwen3_5-model-Qwen3_5DenseTextBackbone)                     | Qwen3.5 dense text decoder rebuilt on the Qwen3-Next `Block`.          |
| [`Qwen3_5ForCausalLM`](#nemo_automodel-components-models-qwen3_5-model-Qwen3_5ForCausalLM)                                 | Qwen3.5 dense causal LM with optional Megatron-style MTP head.         |
| [`Qwen3_5ForConditionalGeneration`](#nemo_automodel-components-models-qwen3_5-model-Qwen3_5ForConditionalGeneration)       | Qwen3.5/Qwen3.6 dense VLM with optional Megatron-style MTP head.       |
| [`Qwen3_5Model`](#nemo_automodel-components-models-qwen3_5-model-Qwen3_5Model)                                             | Thin VLM wrapper exposing `language_model` internals as properties and |

### Functions

| Name                                                                                                         | Description                                                          |
| ------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------- |
| [`_default_init_device`](#nemo_automodel-components-models-qwen3_5-model-_default_init_device)               | -                                                                    |
| [`_dense_moe_config`](#nemo_automodel-components-models-qwen3_5-model-_dense_moe_config)                     | Trivial MoEConfig for the dense Qwen3.5 backbone.                    |
| [`_make_full_attention_config`](#nemo_automodel-components-models-qwen3_5-model-_make_full_attention_config) | -                                                                    |
| [`_mask_mtp_embed_inputs`](#nemo_automodel-components-models-qwen3_5-model-_mask_mtp_embed_inputs)           | Zero invalid future-token embeddings in the local CP layout.         |
| [`_mtp_block_causal_mask`](#nemo_automodel-components-models-qwen3_5-model-_mtp_block_causal_mask)           | Build a 4D block-causal attention mask from an indexed packing mask. |
| [`_qwen3_5_backend`](#nemo_automodel-components-models-qwen3_5-model-_qwen3_5_backend)                       | Return a Qwen3.5 backend with TE fused RoPE disabled.                |
| [`_resolve_mtp_num_layers`](#nemo_automodel-components-models-qwen3_5-model-_resolve_mtp_num_layers)         | -                                                                    |
| [`_rolled_embed_inputs`](#nemo_automodel-components-models-qwen3_5-model-_rolled_embed_inputs)               | -                                                                    |
| [`_split_qwen3_5_position_ids`](#nemo_automodel-components-models-qwen3_5-model-_split_qwen3_5_position_ids) | -                                                                    |
| [`build_mtp_config_from_hf`](#nemo_automodel-components-models-qwen3_5-model-build_mtp_config_from_hf)       | Build Qwen3.5 MTP runtime config from HF-style config fields.        |
| [`build_qwen3_5_dense_mtp`](#nemo_automodel-components-models-qwen3_5-model-build_qwen3_5_dense_mtp)         | Construct dense Qwen3.5 MTP blocks.                                  |

### Data

[`ModelClass`](#nemo_automodel-components-models-qwen3_5-model-ModelClass)

### API

```python
class nemo_automodel.components.models.qwen3_5.model.Fp32SafeQwen3_5TextRotaryEmbedding()
```

**Bases:** `Qwen3_5TextRotaryEmbedding`

Ensure inv\_freq stays in float32 across `.to(dtype)` calls.

```python
nemo_automodel.components.models.qwen3_5.model.Fp32SafeQwen3_5TextRotaryEmbedding._apply(
    fn: typing.Any,
    recurse: bool = True
)
```

```python
class nemo_automodel.components.models.qwen3_5.model.Qwen3_5CausalLMOutputWithPast(
    rope_deltas: torch.Tensor | None = None,
    mtp_per_depth_h: list[torch.Tensor] | None = None,
    mtp_loss_scaling_factor: float | None = None
)
```

Dataclass

**Bases:** `CausalLMOutputWithPast`

Qwen3.5 causal-LM output extended with MTP auxiliary hidden states.

**`mtp_loss_scaling_factor`** `float | None = None`

---

**`mtp_per_depth_h`** `list[Tensor] | None = None`

---

**`rope_deltas`** `Tensor | None = None`

---

```python
class nemo_automodel.components.models.qwen3_5.model.Qwen3_5DenseBlock(
    layer_idx,
    config,
    moe_config,
    backend
)
```

**Bases:** [Block](/nemo-automodel/nemo_automodel/components/models/qwen3_next/model#nemo_automodel-components-models-qwen3_next-model-Block)

Qwen3.5 dense decoder block on top of the Qwen3-Next `Block`.

Identical to `Qwen3_5MoeBlock` except the MLP degrades to a dense `MLP`
(no experts). The CP-aware GatedDeltaNet is built natively for
linear-attention layers, and the forward threads NEAT-packing kwargs.

**`linear_attn`** `= CPAwareGatedDeltaNet(config, layer_idx)`

---

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5DenseBlock.forward(
    x: torch.Tensor,
    freqs_cis: torch.Tensor,
    attention_mask: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    packed_gdn_metadata: nemo_automodel.components.models.qwen3_5.packing.GatedDeltaPackedMetadata | None = None,
    attn_kwargs: typing.Any = {}
) -> torch.Tensor
```

Run one dense Qwen3.5 decoder block.

**Parameters:**

**`x`** `torch.Tensor`

Hidden states of shape \[batch, sequence, hidden].

---

**`freqs_cis`** `torch.Tensor`

Rotary cosine/sine values of shape \[batch, sequence,
rotary\_dim].

---

**`attention_mask`** `torch.Tensor | None` — default: None

Optional validity, indexed document, or backend mask
of shape \[batch, sequence] or \[batch, 1, sequence, sequence].

---

**`padding_mask`** `torch.Tensor | None` — default: None

Optional padding mask of shape \[batch, sequence].

---

**`position_ids`** `torch.Tensor | None` — default: None

Optional positions of shape \[batch, sequence] or
\[axes, batch, sequence].

---

**`packed_gdn_metadata`** `GatedDeltaPackedMetadata | None` — default: None

Optional model-forward-owned packing metadata;
tensor layouts are documented by `GatedDeltaPackedMetadata`.

---

**`**attn_kwargs`** `Any` — default: \{}

Packed or THD metadata: `_packed_seq_ids` has shape
\[batch, sequence] with 1-based document IDs and zero padding.
`packed_token_indices` has shape \[batch, sequence] with row-local
positions and -1 padding, or \[tokens] indexing flattened batch
and sequence axes. `cu_seqlens` has shape \[batch, max\_documents + 1]
with row-local boundaries and -1 padding, or \[documents + 1] for
flattened/THD inputs. `max_seqlen` is an integer document length.

---

**Returns:** `torch.Tensor`

Hidden states of shape \[batch, sequence, hidden].

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5DenseBlock.init_weights(
    buffer_device: torch.device
)
```

```python
class nemo_automodel.components.models.qwen3_5.model.Qwen3_5DenseMTPSublayer(
    config: transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5TextConfig,
    layer_idx: int,
    has_fusion: bool = False,
    has_final_norm: bool = False,
    dtype: torch.dtype = torch.bfloat16
)
```

**Bases:** `Qwen3_5DecoderLayer`

One full-attention Qwen3.5 dense MTP sublayer.

**`eh_proj`**

---

**`enorm`**

---

**`final_layernorm`**

---

**`hnorm`**

---

**`layer_type`** `= self.block_type`

---

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5DenseMTPSublayer.forward(
    hidden_states: torch.Tensor,
    embed_input: torch.Tensor | None = None,
    rotary_emb: torch.nn.Module,
    position_ids: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    past_key_values: typing.Any | None = None,
    kwargs: typing.Any = {}
) -> torch.Tensor
```

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5DenseMTPSublayer.init_weights(
    buffer_device: torch.device | None = None
) -> None
```

```python
class nemo_automodel.components.models.qwen3_5.model.Qwen3_5DenseTextBackbone(
    config: transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5TextConfig,
    backend: nemo_automodel.components.models.common.BackendConfig
)
```

**Bases:** `Module`

Qwen3.5 dense text decoder rebuilt on the Qwen3-Next `Block`.

Native counterpart of `Qwen3_5MoeTextModelBackend` for the dense model:
reuses the same blocks/GatedDeltaNet/norm/rotary so dense and MoE share one
code path, with the fp32 `SSMGate` built at construction (no runtime patch).

**`embed_tokens`**

---

**`layers`**

---

**`norm`**

---

**`padding_idx`** `= getattr(config, 'pad_token_id', None)`

---

**`rotary_emb`** `= Fp32SafeQwen3_5TextRotaryEmbedding(config=config)`

---

**`vocab_size`** `= config.vocab_size`

---

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5DenseTextBackbone.forward(
    input_ids: torch.Tensor | None = None,
    inputs_embeds: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    cache_position: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    past_key_values: typing.Any | None = None,
    use_cache: bool | None = None,
    output_hidden_states: bool | None = None,
    attn_kwargs: typing.Any = {}
) -> transformers.modeling_outputs.BaseModelOutputWithPast
```

Decode text tokens with model-forward-owned packed GDN metadata.

**Parameters:**

**`input_ids`** `torch.Tensor | None` — default: None

Optional token IDs of shape \[batch, sequence].

---

**`inputs_embeds`** `torch.Tensor | None` — default: None

Optional token embeddings of shape \[batch, sequence,
hidden].

---

**`attention_mask`** `torch.Tensor | None` — default: None

Optional validity, indexed document, or backend mask
of shape \[batch, sequence] or \[batch, 1, sequence, sequence].

---

**`position_ids`** `torch.Tensor | None` — default: None

Optional positions of shape \[batch, sequence] or
\[axes, batch, sequence].

---

**`cache_position`** `torch.Tensor | None` — default: None

Optional token positions of shape \[sequence].

---

**`padding_mask`** `torch.Tensor | None` — default: None

Optional padding mask of shape \[batch, sequence].

---

**`past_key_values`** `Any | None` — default: None

Unsupported recurrent or KV cache.

---

**`use_cache`** `bool | None` — default: None

Whether to use a cache; only `False` or `None` is
supported.

---

**`output_hidden_states`** `bool | None` — default: None

Accepted for Hugging Face compatibility and
ignored.

---

**`**attn_kwargs`** `Any` — default: \{}

Backend-specific attention arguments, including optional
`_packed_seq_ids` of shape \[batch, sequence], `packed_token_indices`
of shape \[batch, sequence] or \[tokens], `cu_seqlens` of shape
\[batch, max\_documents + 1] or \[documents + 1], and integer
`max_seqlen`. Batch-major metadata uses -1 padding. Explicit
token metadata is supported without context parallelism.
The multimodal wrapper temporarily prefixes `cu_seqlens` and
`max_seqlen` with `_text_` while crossing the HF vision path;
their layouts and values are unchanged.

---

**Returns:** `BaseModelOutputWithPast`

Model output whose `last_hidden_state` has shape \[batch, sequence,

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5DenseTextBackbone.get_input_embeddings() -> torch.nn.Module
```

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5DenseTextBackbone.init_weights(
    buffer_device: torch.device | None = None
) -> None
```

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5DenseTextBackbone.set_input_embeddings(
    value: torch.nn.Module
) -> None
```

```python
class nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForCausalLM(
    config: transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5TextConfig,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    mtp_loss_scaling_factor: float = 0.1,
    num_nextn_predict_layers: int | None = None,
    kwargs: typing.Any = {}
)
```

**Bases:** [HFCheckpointingMixin](/nemo-automodel/nemo_automodel/components/models/common/hf_checkpointing_mixin#nemo_automodel-components-models-common-hf_checkpointing_mixin-HFCheckpointingMixin), `Module`

Qwen3.5 dense causal LM with optional Megatron-style MTP head.

**`_tied_weights_keys`** `= {'lm_head.weight': 'model.embed_tokens.weight'}`

---

**`backend`** `= _qwen3_5_backend(backend)`

---

**`lm_head`**

---

**`model`** `= Qwen3_5DenseTextBackbone(config, self.backend)`

---

**`mtp`**

---

**`mtp_config`**

---

**`packed_mask_type`** `PackedMaskType`

Request compact NEAT document IDs when TE consumes packed offsets.

---

**`state_dict_adapter`**

---

**`tie_word_embeddings_support`** `TieSupport = TieSupport.BOTH`

---

**`vocab_size`** `= config.vocab_size`

---

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForCausalLM.forward(
    input_ids: torch.LongTensor | None = None,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.LongTensor | None = None,
    past_key_values: typing.Any | None = None,
    inputs_embeds: torch.FloatTensor | None = None,
    labels: torch.LongTensor | None = None,
    use_cache: bool | None = None,
    logits_to_keep: int | torch.Tensor = 0,
    kwargs: typing.Any = {}
) -> nemo_automodel.components.models.qwen3_5.model.Qwen3_5CausalLMOutputWithPast
```

Compute causal-LM logits and optional multi-token prediction states.

**Parameters:**

**`input_ids`** `torch.LongTensor | None` — default: None

Optional token IDs of shape \[batch, sequence].

---

**`attention_mask`** `torch.Tensor | None` — default: None

Optional document/validity mask of shape \[batch, sequence]
or block-causal mask of shape \[batch, 1, sequence, sequence].

---

**`position_ids`** `torch.LongTensor | None` — default: None

Optional text positions of shape \[batch, sequence] or
multi-axis positions of shape \[3, batch, sequence] or
\[4, batch, sequence] including a leading text-position axis.

---

**`past_key_values`** `Any | None` — default: None

Optional HF cache; cached generation is unsupported.

---

**`inputs_embeds`** `torch.FloatTensor | None` — default: None

Optional embeddings of shape \[batch, sequence, hidden].

---

**`labels`** `torch.LongTensor | None` — default: None

Optional targets of shape \[batch, sequence], ignored here.

---

**`use_cache`** `bool | None` — default: None

Whether to cache states; only False or None is supported.

---

**`logits_to_keep`** `int | torch.Tensor` — default: 0

Trailing token count (zero means all), or position
indices of shape \[selected\_tokens] indexing the sequence axis.

---

**`**kwargs`** `Any` — default: \{}

Backbone arguments, including `_packed_seq_ids` of shape
\[batch, sequence], `packed_token_indices` of shape \[batch, sequence]
or \[tokens], `cu_seqlens` of shape \[batch, max\_documents + 1] or
\[documents + 1], and integer `max_seqlen`. Batch-major metadata
uses -1 padding and is removed before the separate MTP sublayers.

---

**Returns:** `Qwen3_5CausalLMOutputWithPast`

Logits of shape \[batch, selected\_sequence, vocab], final hidden states

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForCausalLM.from_config(
    config: transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5TextConfig,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    kwargs: typing.Any = {}
)
```

classmethod

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForCausalLM.from_pretrained(
    pretrained_model_name_or_path: str,
    model_args: typing.Any = (),
    kwargs: typing.Any = {}
)
```

classmethod

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForCausalLM.get_input_embeddings() -> torch.nn.Module
```

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForCausalLM.get_output_embeddings() -> torch.nn.Module
```

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForCausalLM.initialize_weights(
    buffer_device: torch.device | None = None,
    dtype: torch.dtype = torch.bfloat16
) -> None
```

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForCausalLM.set_input_embeddings(
    value: torch.nn.Module
) -> None
```

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForCausalLM.set_output_embeddings(
    new_embeddings: torch.nn.Module
) -> None
```

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForCausalLM.tie_weights(
    _args: object = (),
    _kwargs: object = {}
) -> None
```

```python
class nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForConditionalGeneration(
    config: transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5Config,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    mtp_loss_scaling_factor: float = 0.1,
    num_nextn_predict_layers: int | None = None,
    kwargs: typing.Any = {}
)
```

**Bases:** [HFCheckpointingMixin](/nemo-automodel/nemo_automodel/components/models/common/hf_checkpointing_mixin#nemo_automodel-components-models-common-hf_checkpointing_mixin-HFCheckpointingMixin), `HFQwen3_5ForConditionalGeneration`

Qwen3.5/Qwen3.6 dense VLM with optional Megatron-style MTP head.

The base VLM stays on the upstream HF implementation so image/video feature
insertion, M-RoPE position handling, and generation helpers remain intact.
MTP is added as an auxiliary train-time module over the final language
hidden states, matching the dense text-only MTP architecture.

**`_keep_in_fp32_modules`** `list[str] = keep_fp32`

---

**`_pp_keep_self_forward`** `bool = True`

---

**`_tied_weights_keys`**

---

**`backend`** `= _qwen3_5_backend(backend)`

---

**`lm_head`** `= self.lm_head.to(dtype)`

---

**`mtp`**

---

**`mtp_config`**

---

**`packed_mask_type`** `PackedMaskType`

Request compact NEAT document IDs when TE consumes packed offsets.

---

**`state_dict_adapter`**

---

**`tie_word_embeddings_support`** `TieSupport = TieSupport.BOTH`

---

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForConditionalGeneration._embed_and_splice_for_cp(
    input_ids: torch.Tensor,
    pixel_values: torch.Tensor | None,
    pixel_values_videos: torch.Tensor | None,
    image_grid_thw: torch.Tensor | None,
    video_grid_thw: torch.Tensor | None
) -> torch.Tensor
```

Embed token ids and splice image/video features into the embeddings.

The VLM->LM multimodal scatter runs on the full (unsharded) sequence
inside the forward before the CP sequence shard, so it is identical to
the pre-CP-refactor pre-embed. Vision features may be frame-sharded
across the CP group before `get_placeholder_mask` scatters them into
the full `input_ids` sequence.

**Parameters:**

**`input_ids`** `torch.Tensor`

Token ids `[batch, sequence]` (full, unsharded).

---

**`pixel_values`** `torch.Tensor | None`

Optional packed image patches for HF vision encoding.

---

**`pixel_values_videos`** `torch.Tensor | None`

Optional packed video patches.

---

**`image_grid_thw`** `torch.Tensor | None`

Per-image `[num_images, 3]` T/H/W grid.

---

**`video_grid_thw`** `torch.Tensor | None`

Per-video `[num_videos, 3]` T/H/W grid.

---

**Returns:** `torch.Tensor`

`inputs_embeds` of shape `[batch, sequence, hidden]` with image /

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForConditionalGeneration._encode_vision_for_cp(
    pixel_values: torch.Tensor,
    grid_thw: torch.Tensor,
    is_video: bool
) -> torch.Tensor
```

Encode one modality's patches into flat, entry-ordered visual tokens under CP.

Under context parallelism the vision tower otherwise runs redundantly on the full,
un-sharded patch set on every CP rank. When the CP vision-frame sharding group is published
(see `set_cp_vision_group`),
route the single `self.model.visual` call through
`maybe_distribute_visual`,
which shards the frames across the CP group and all-gathers the per-frame embeds in
original entry order. Its `pooler_output` is the flat, already-concatenated tensor,
identical to `torch.cat(get_image_features(...).pooler_output, dim=0)` (HF
`get_image_features` splits that same flat tensor per entry). When sharding is
disabled / inactive this is the exact replicated `get_image_features` /
`get_video_features` path.

**Parameters:**

**`pixel_values`** `torch.Tensor`

Patch rows of shape `[total_patch_rows, patch_dim]` for ALL entries
of one modality, frame-contiguous in entry order.

---

**`grid_thw`** `torch.Tensor`

`[num_entries, 3]` tensor of per-entry `(t, h, w)` grid sizes.

---

**`is_video`** `bool`

Select the video (`True`) vs image (`False`) replicated fallback.

---

**Returns:** `torch.Tensor`

`[total_patch_rows / spatial_merge_size**2, hidden]` flat merged-token embeddings

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForConditionalGeneration._pop_staged_vlm_media(
    input_ids: torch.Tensor | None,
    kwargs: dict[str, typing.Any]
) -> tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]
```

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForConditionalGeneration.forward(
    input_ids: torch.LongTensor | None = None,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.LongTensor | None = None,
    past_key_values: typing.Any | None = None,
    inputs_embeds: torch.FloatTensor | None = None,
    labels: torch.LongTensor | None = None,
    pixel_values: torch.Tensor | None = None,
    pixel_values_videos: torch.FloatTensor | None = None,
    image_grid_thw: torch.LongTensor | None = None,
    video_grid_thw: torch.LongTensor | None = None,
    mm_token_type_ids: torch.IntTensor | None = None,
    use_cache: bool | None = None,
    logits_to_keep: int | torch.Tensor = 0,
    padding_mask: torch.Tensor | None = None,
    mtp_per_depth_input_ids: tuple[torch.LongTensor, ...] | None = None,
    mtp_per_depth_position_ids: tuple[torch.LongTensor, ...] | None = None,
    mtp_per_depth_valid_masks: tuple[torch.BoolTensor, ...] | None = None,
    kwargs: typing.Any = {}
) -> nemo_automodel.components.models.qwen3_5.model.Qwen3_5CausalLMOutputWithPast
```

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForConditionalGeneration.from_config(
    config: transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5Config,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    kwargs: typing.Any = {}
)
```

classmethod

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForConditionalGeneration.from_pretrained(
    pretrained_model_name_or_path: str,
    model_args: typing.Any = (),
    kwargs: typing.Any = {}
)
```

classmethod

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForConditionalGeneration.get_pipeline_stage_metas(
    is_first: bool,
    microbatch_size: int,
    seq_len: int,
    dtype: torch.dtype
) -> tuple[tuple[torch.Tensor, ...], tuple[torch.Tensor, ...]]
```

Per-stage input/output meta tensors for the PP schedule's shape inference.

Matches the framework default (first stage consumes full token ids
`[mb, seq]`; later stages consume hidden states; the last stage owning
`lm_head` emits logits, earlier stages emit hidden states) except that
under context parallelism the first stage embeds the full sequence and
shards it in forward, so every stage output and later-stage input carries
the LOCAL (padded-to-`2*cp` then `//cp`) sequence length while the
first-stage input stays full-length. At `cp_size == 1` this reduces to
the default symmetric shapes.

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForConditionalGeneration.initialize_weights(
    buffer_device: torch.device | None = None,
    dtype: torch.dtype = torch.bfloat16
) -> None
```

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForConditionalGeneration.prepare_model_inputs_for_cp(
    batch: dict[str, typing.Any],
    num_chunks: int = 1
) -> dict[str, typing.Any]
```

Return a sharder-only CP backend plus the full-sequence mRoPE positions.

Embedding and the VLM->LM multimodal scatter now run inside `forward`
per microbatch (see `_embed_and_splice_for_cp`), so this hook only
(a) computes the mRoPE `position_ids` on the *full* (unsharded) sequence
via `get_rope_index` and returns them for `shard_batch_aux_only`
to round-robin-shard on the mRoPE axis, and (b) returns the
`ContextParallelSharder`. `input_ids` and the media inputs are
left in the batch for the forward; `mm_token_type_ids` is consumed here
(only `get_rope_index` needs it) so the sharded forward never sees a
full-length copy.

**Parameters:**

**`batch`** `dict[str, Any]`

The batch dict (with `input_ids` and optional multimodal
keys). `input_ids` is `[batch, sequence]`.

---

**`num_chunks`** `int` — default: 1

Accepted for hook-signature parity; unused (round-robin CP).

---

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForConditionalGeneration.prepare_mtp_inputs_for_cp(
    batch: dict[str, typing.Any],
    ignore_index: int = -100
) -> nemo_automodel.components.models.common.mtp.MTPContextParallelInputs
```

Prepare dense Qwen VLM future-token streams before CP sharding.

Call this after `prepare_model_inputs_for_cp` has materialized
full-sequence mRoPE positions.

**Parameters:**

**`batch`** `dict[str, Any]`

Full-sequence batch whose `input_ids` and `labels` tensors
have shape `[batch, sequence]` and whose `position_ids` tensor
has shape `[axes, batch, sequence]`.

---

**`ignore_index`** `int` — default: -100

Fill value for invalid targets at trailing and packed
document-boundary positions.

---

**Returns:** `MTPContextParallelInputs`

Global per-depth token IDs, mRoPE position IDs, targets, and validity

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5ForConditionalGeneration.tie_weights(
    _args: object = (),
    _kwargs: object = {}
) -> None
```

Tie `lm_head` to the active VLM text embedding when requested.

```python
class nemo_automodel.components.models.qwen3_5.model.Qwen3_5Model()
```

**Bases:** `HFQwen3_5Model`

Thin VLM wrapper exposing `language_model` internals as properties and
routing the forward: HF vision+scatter path when media is present, else the
NeMo dense backbone directly. Mirrors `Qwen3_5MoeModel`.

```python
nemo_automodel.components.models.qwen3_5.model.Qwen3_5Model.forward(
    input_ids = None,
    attention_mask = None,
    position_ids = None,
    past_key_values = None,
    inputs_embeds = None,
    pixel_values = None,
    pixel_values_videos = None,
    image_grid_thw = None,
    video_grid_thw = None,
    cache_position = None,
    kwargs = {}
)
```

Route multimodal inputs without exposing text boundaries to vision.

**Parameters:**

**`input_ids`** — default: None

Token IDs of shape \[batch, sequence], or floating-point
pipeline hidden states of shape \[batch, sequence, hidden].

---

**`attention_mask`** — default: None

Padding/document IDs of shape \[batch, sequence] or
a dense mask of shape \[batch, 1, sequence, sequence].

---

**`position_ids`** — default: None

Text positions of shape \[batch, sequence] or mRoPE
coordinates of shape \[3 or 4, batch, sequence].

---

**`past_key_values`** — default: None

Unsupported recurrent or KV cache.

---

**`inputs_embeds`** — default: None

Optional embeddings of shape \[batch, sequence, hidden].

---

**`pixel_values`** — default: None

Optional image patches of shape \[patches, patch\_features].

---

**`pixel_values_videos`** — default: None

Optional video patches of shape \[patches, patch\_features].

---

**`image_grid_thw`** — default: None

Image grids of shape \[images, 3] in temporal/height/width order.

---

**`video_grid_thw`** — default: None

Video grids of shape \[videos, 3] in temporal/height/width order.

---

**`cache_position`** — default: None

Optional token positions of shape \[sequence].

---

**`**kwargs`** — default: \{}

Text attention metadata, including `cu_seqlens` of shape
\[batch, max\_documents + 1] or \[documents + 1], integer
`max_seqlen`, and `packed_token_indices` of shape \[batch,
sequence] or \[tokens]. Image/video-prefixed metadata retains
the inherited HF precomputed-vision contract.

---

**Returns:**

Model output with `last_hidden_state` of shape \[batch, sequence,

```python
nemo_automodel.components.models.qwen3_5.model._default_init_device() -> torch.device
```

```python
nemo_automodel.components.models.qwen3_5.model._dense_moe_config(
    config: transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5TextConfig,
    dtype: torch.dtype
) -> nemo_automodel.components.moe.layers.MoEConfig
```

Trivial MoEConfig for the dense Qwen3.5 backbone.

The dense model has no experts (`num_experts` is 0/absent), so `Block`
builds a dense `MLP` and never consults this config; it is only required to
satisfy `Block.__init__`'s signature.

```python
nemo_automodel.components.models.qwen3_5.model._make_full_attention_config(
    config: transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5TextConfig,
    layer_idx: int
) -> transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5TextConfig
```

```python
nemo_automodel.components.models.qwen3_5.model._mask_mtp_embed_inputs(
    embed_inputs: tuple[torch.Tensor, ...],
    valid_masks: tuple[torch.BoolTensor, ...]
) -> tuple[torch.Tensor, ...]
```

Zero invalid future-token embeddings in the local CP layout.

**Parameters:**

**`embed_inputs`** `tuple[torch.Tensor, ...]`

Per-depth embeddings of shape `[batch, sequence, hidden]`.

---

**`valid_masks`** `tuple[torch.BoolTensor, ...]`

Per-depth masks of shape `[batch, sequence]` in the same
local CP token layout as `embed_inputs`.

---

**Returns:** `torch.Tensor`

Per-depth embeddings with invalid trailing or packed-boundary positions

```python
nemo_automodel.components.models.qwen3_5.model._mtp_block_causal_mask(
    packing_mask: torch.Tensor,
    inputs_embeds: torch.Tensor
) -> torch.Tensor
```

Build a 4D block-causal attention mask from an indexed packing mask.

`packing_mask` is `[B, S]` with the 1-based document index per token
(0 = padding). The returned bool mask `[B, 1, S, S]` (`True` = attend)
keeps attention causal *and* within each packed document, matching the
backbone's packed-sequence semantics. Used for the MTP sublayers, which run
SDPA self-attention over the same packed batch (NVBugs 6330129).

```python
nemo_automodel.components.models.qwen3_5.model._qwen3_5_backend(
    backend: nemo_automodel.components.models.common.BackendConfig | None = None
) -> nemo_automodel.components.models.common.BackendConfig
```

Return a Qwen3.5 backend with TE fused RoPE disabled.

Qwen3.5 VLM training can feed full-attention layers in packed/THD shape via
the shared Qwen3-Next attention block. TE fused RoPE expects 4D inputs there,
so keep the non-fused RoPE path while preserving the rest of the backend
selection (TE Linear, attention backend, etc.).

```python
nemo_automodel.components.models.qwen3_5.model._resolve_mtp_num_layers(
    config: typing.Any,
    override: int | None = None
) -> int
```

```python
nemo_automodel.components.models.qwen3_5.model._rolled_embed_inputs(
    inputs_embeds: torch.Tensor,
    num_depths: int
) -> tuple[torch.Tensor, ...]
```

```python
nemo_automodel.components.models.qwen3_5.model._split_qwen3_5_position_ids(
    position_ids: torch.Tensor | None,
    batch_size: int,
    seq_len: int,
    device: torch.device,
    past_key_values: typing.Any | None = None
) -> tuple[torch.Tensor, torch.Tensor | None]
```

```python
nemo_automodel.components.models.qwen3_5.model.build_mtp_config_from_hf(
    config: typing.Any,
    loss_scaling_factor: float = 0.1,
    num_nextn_predict_layers: int | None = None
) -> nemo_automodel.components.models.common.mtp.MTPConfig
```

Build Qwen3.5 MTP runtime config from HF-style config fields.

```python
nemo_automodel.components.models.qwen3_5.model.build_qwen3_5_dense_mtp(
    config: transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5TextConfig,
    mtp_config: nemo_automodel.components.models.common.mtp.MTPConfig,
    dtype: torch.dtype
) -> nemo_automodel.components.models.common.mtp.MTPModule
```

Construct dense Qwen3.5 MTP blocks.

Qwen3.5 MTP follows Megatron Bridge: each depth is one full-attention
Qwen3.5 decoder block, regardless of the backbone's GatedDeltaNet layers.

```python
nemo_automodel.components.models.qwen3_5.model.ModelClass = Qwen3_5ForCausalLM
```