> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.models.qwen3_8_flash_next.layers

Qwen3.8-Flash-Next model-local layers.

The HyperConnection equations in this module follow the Qwen3.8-Flash-Next reference
implementation. They are intentionally not shared with DeepSeek-V4: that model
uses a different Sinkhorn-based HyperConnection parameterization.

## Module Contents

### Classes

| Name                                                                                                                                               | Description                                                                    |
| -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| [`Qwen3_8_FlashNextDecoderLayer`](#nemo_automodel-components-models-qwen3_8_flash_next-layers-Qwen3_8_FlashNextDecoderLayer)                       | One Qwen3.8-Flash-Next decoder layer with two learned HyperConnections.        |
| [`Qwen3_8_FlashNextGatedDeltaNet`](#nemo_automodel-components-models-qwen3_8_flash_next-layers-Qwen3_8_FlashNextGatedDeltaNet)                     | CP-aware GatedDeltaNet with Qwen3.8-Flash-Next's sigmoid output gate.          |
| [`Qwen3_8_FlashNextGroupedRMSNorm`](#nemo_automodel-components-models-qwen3_8_flash_next-layers-Qwen3_8_FlashNextGroupedRMSNorm)                   | Gemma-style RMS normalization over fixed-width HyperConnection branches.       |
| [`Qwen3_8_FlashNextHyperConnection`](#nemo_automodel-components-models-qwen3_8_flash_next-layers-Qwen3_8_FlashNextHyperConnection)                 | Qwen3.8-Flash-Next gated HyperConnection read/write transform.                 |
| [`Qwen3_8_FlashNextHyperConnectionResidual`](#nemo_automodel-components-models-qwen3_8_flash_next-layers-Qwen3_8_FlashNextHyperConnectionResidual) | Residual tensors retained between a HyperConnection read and write.            |
| [`Qwen3_8_FlashNextQSAAttention`](#nemo_automodel-components-models-qwen3_8_flash_next-layers-Qwen3_8_FlashNextQSAAttention)                       | Qwen3.8-Flash-Next gated attention with compressed-block QSA routing.          |
| [`Qwen3_8_FlashNextRMSNormGated`](#nemo_automodel-components-models-qwen3_8_flash_next-layers-Qwen3_8_FlashNextRMSNormGated)                       | Qwen3.8-Flash-Next GDN output normalization with its checkpoint-selected gate. |

### Functions

| Name                                                                                                           | Description                                                  |
| -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| [`_grouped_rms_norm_fp32`](#nemo_automodel-components-models-qwen3_8_flash_next-layers-_grouped_rms_norm_fp32) | FP32 grouped RMSNorm as one fusable elementwise chain.       |
| [`_rms_norm_gated_fp32`](#nemo_automodel-components-models-qwen3_8_flash_next-layers-_rms_norm_gated_fp32)     | FP32 RMSNorm + output gate as one fusable elementwise chain. |

### Data

[`_grouped_rms_norm_fp32_compiled`](#nemo_automodel-components-models-qwen3_8_flash_next-layers-_grouped_rms_norm_fp32_compiled)

[`_rms_norm_gated_fp32_compiled`](#nemo_automodel-components-models-qwen3_8_flash_next-layers-_rms_norm_gated_fp32_compiled)

### API

```python
class nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextDecoderLayer(
    layer_idx: int,
    config: object,
    moe_config: object,
    backend: nemo_automodel.components.models.common.BackendConfig,
    ple: torch.nn.Module | None = None
)
```

**Bases:** `Module`

One Qwen3.8-Flash-Next decoder layer with two learned HyperConnections.

**Parameters:**

**`layer_idx`** `int`

Zero-based decoder index.

---

**`config`** `object`

Qwen3.8-Flash-Next text configuration.

---

**`moe_config`** `object`

Native MoE configuration.

---

**`backend`** `BackendConfig`

Attention, linear, and expert backend configuration.

---

**`ple`** `nn.Module | None` — default: None

Optional Engram-derived PLE module. The checkpoint installs it
only on decoder index 1.

---

**`_nemo_disable_activation_checkpointing`** `= ple is not None`

---

**`attn_hyper_connection`** `= Qwen3_8_FlashNextHyperConnection(**hc_kwargs)`

---

**`hc_count`** `= int(getattr(config, 'hc_count'))`

---

**`hidden_size`** `= int(getattr(config, 'hidden_size'))`

---

**`layer_type`** `= str(layer_types[layer_idx])`

---

**`linear_attn`** `= Qwen3_8_FlashNextGatedDeltaNet(config, layer_idx)`

---

**`mlp`** `= MoE(moe_config, backend)`

---

**`mlp_hyper_connection`** `= Qwen3_8_FlashNextHyperConnection(**hc_kwargs)`

---

**`self_attn`**

---

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextDecoderLayer._expand_initial_streams(
    hidden_states: torch.Tensor
) -> torch.Tensor
```

Expand a one-stream decoder input into the persistent HC layout.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape `[batch, sequence, hidden_size]`
or `[batch, sequence, hc_count * hidden_size]`.

---

**Returns:** `torch.Tensor`

Tensor of shape

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextDecoderLayer.forward(
    hidden_states: torch.Tensor,
    input_ids: torch.Tensor,
    freqs_cis: torch.Tensor,
    attention_mask: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    cp_context: nemo_automodel.components.models.qwen3_8_flash_next.cp.Qwen3_8_FlashNextCPContext | None = None,
    attn_kwargs: object = {}
) -> torch.Tensor
```

Run PLE, attention/GDN, and top-10 MoE updates.

**Parameters:**

**`hidden_states`** `torch.Tensor`

One-stream input `[batch, sequence, hidden]` on
the first layer, otherwise flattened HC streams
`[batch, sequence, hc_count * hidden]`.

---

**`input_ids`** `torch.Tensor`

Raw tokenizer IDs of shape `[batch, sequence]` used by
the PLE hash path.

---

**`freqs_cis`** `torch.Tensor`

Composed rotary values `[batch, sequence, rotary_dim]`
whose final axis stores concatenated cosine and sine values.

---

**`attention_mask`** `torch.Tensor | None` — default: None

Optional token mask of shape `[batch, sequence]`
or backend-specific causal attention mask.

---

**`padding_mask`** `torch.Tensor | None` — default: None

Optional `[batch, sequence]` mask where `True`
marks padding for MoE dispatch.

---

**`position_ids`** `torch.Tensor | None` — default: None

Optional positions of shape `[batch, sequence]` or
`[axes, batch, sequence]`.

---

**`cp_context`** `Qwen3_8_FlashNextCPContext | None` — default: None

Optional contiguous CP metadata. Tensor-bearing fields
contain replicated global raw IDs/padding of shape `[batch,
global_sequence]` and identify this rank's local interval.

---

**`**attn_kwargs`** `object` — default: \{}

Attention backend metadata.

---

**Returns:** `torch.Tensor`

Flattened HC streams of shape

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextDecoderLayer.init_weights(
    buffer_device: torch.device,
    init_std: float = 0.02
) -> None
```

Initialize this decoder layer for training from scratch.

**Parameters:**

**`buffer_device`** `torch.device`

Device used by attention/MoE initializers.

---

**`init_std`** `float` — default: 0.02

Standard deviation for dense projection weights.

---

```python
class nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextGatedDeltaNet(
    config: object,
    layer_idx: int
)
```

**Bases:** [CPAwareGatedDeltaNet](/nemo-automodel/nemo_automodel/components/models/qwen3_5_moe/cp_linear_attn#nemo_automodel-components-models-qwen3_5_moe-cp_linear_attn-CPAwareGatedDeltaNet)

CP-aware GatedDeltaNet with Qwen3.8-Flash-Next's sigmoid output gate.

**`_packed_global_cu_seqlens`** `Tensor | None = None`

---

**`norm`**

---

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextGatedDeltaNet._forward_with_cp(
    hidden_states: torch.Tensor,
    position_ids: torch.Tensor | None,
    seq_index: torch.Tensor | None,
    blockdiag_state: nemo_automodel.components.distributed.blockdiag_cp.BlockdiagCpModelState | None = None
) -> torch.Tensor
```

Run the inherited FLA CP core in explicit contiguous sequence order.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Local contiguous states of shape `[batch,
local_sequence, hidden]`. Rank `r` owns global positions
`[r * local_sequence, (r + 1) * local_sequence)`.

---

**`position_ids`** `torch.Tensor | None`

Local global positions of shape `[batch,
local_sequence]` or repeated text-RoPE axes of shape `[axes,
batch, local_sequence]`.

---

**`seq_index`** `torch.Tensor | None`

Optional local positions of shape `[local_sequence]`
or `[batch, local_sequence]`; ignored because Qwen3.8-Flash-Next's
model-owned layout is always contiguous.

---

**`blockdiag_state`** `BlockdiagCpModelState | None` — default: None

External packed CP metadata. Qwen3.8-Flash-Next builds its
own state and rejects an externally supplied one.

---

**Returns:** `torch.Tensor`

Local GDN output of shape `[batch, local_sequence, hidden]` in

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextGatedDeltaNet.forward(
    hidden_states: torch.Tensor,
    kwargs: object = {}
) -> torch.Tensor
```

Route packed boundaries around the inherited non-CP varlen path.

Without CP, `cu_seqlens` flows to the inherited FLA varlen forward
unchanged. With an active CP mesh the boundaries describe the global
packed row, so they are stashed for :meth:`_forward_with_cp` and the
inherited dispatcher sees no `cu_seqlens`.

```python
class nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextGroupedRMSNorm(
    hidden_size: int,
    group_size: int,
    eps: float = 1e-06
)
```

**Bases:** `Module`

Gemma-style RMS normalization over fixed-width HyperConnection branches.

**Parameters:**

**`hidden_size`** `int`

Flattened feature width.

---

**`group_size`** `int`

Number of features normalized together. A flattened input
of shape `[..., hidden_size]` is viewed as
`[..., hidden_size // group_size, group_size]`.

---

**`eps`** `float` — default: 1e-06

Variance epsilon.

---

**`weight`** `= nn.Parameter(torch.zeros(hidden_size))`

---

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextGroupedRMSNorm.forward(
    hidden_states: torch.Tensor
) -> torch.Tensor
```

Normalize each branch independently.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Flattened branch states of shape
`[..., hidden_size]`.

---

**Returns:** `torch.Tensor`

Normalized states of shape `[..., hidden_size]`.

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextGroupedRMSNorm.reset_parameters() -> None
```

Reset the additive Gemma-style scale to zero.

```python
class nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextHyperConnection(
    hidden_size: int,
    hc_count: int,
    lowrank_size: int,
    rms_norm_eps: float,
    backend: nemo_automodel.components.models.common.BackendConfig,
    use_combine: bool = True,
    dtype: torch.dtype | str | None = None
)
```

**Bases:** `Module`

Qwen3.8-Flash-Next gated HyperConnection read/write transform.

A read normalizes `hc_count` streams independently, predicts a feature
gate, and averages the gated streams to one block input. A write predicts
one injection gate per stream and adds the block output back to every
stream. The final decoder mixer uses only the read side.

**Parameters:**

**`hidden_size`** `int`

Width of one HC stream.

---

**`hc_count`** `int`

Number of streams.

---

**`lowrank_size`** `int`

Bottleneck width used to predict read gates.

---

**`rms_norm_eps`** `float`

Variance epsilon for branch normalization.

---

**`backend`** `BackendConfig`

Linear backend configuration.

---

**`use_combine`** `bool` — default: True

Whether to instantiate the write-side injection weight.

---

**`dtype`** `torch.dtype | str | None` — default: None

Parameter dtype override. If omitted, the backend model dtype is
resolved by the caller and should be passed explicitly.

---

**`block_inject_weight`**

---

**`flat_hidden_size`** `= hidden_size * hc_count`

---

**`hc_norm`**

---

**`input_mix_weight_down`**

---

**`input_mix_weight_up`**

---

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextHyperConnection.combine(
    block_output: torch.Tensor,
    residual: nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextHyperConnectionResidual
) -> torch.Tensor
```

Inject one block output into every HC stream.

**Parameters:**

**`block_output`** `torch.Tensor`

Attention or MoE output of shape
`[..., hidden_size]`.

---

**`residual`** `Qwen3_8_FlashNextHyperConnectionResidual`

Flattened pre-block streams and their normalized values,
each shaped `[..., hc_count * hidden_size]`.

---

**Returns:** `torch.Tensor`

Updated flattened streams of shape

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextHyperConnection.init_weights(
    init_std: float = 0.02
) -> None
```

Initialize HC weights for training from scratch.

**Parameters:**

**`init_std`** `float` — default: 0.02

Standard deviation for all HC linear weights.

---

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextHyperConnection.mix(
    hidden_states: torch.Tensor
) -> tuple[torch.Tensor, nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextHyperConnectionResidual]
```

Collapse HC streams to the input of an attention or MoE block.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Flattened HC streams of shape
`[..., hc_count * hidden_size]`.

---

**Returns:** `torch.Tensor`

A pair containing the mixed block input of shape

```python
class nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextHyperConnectionResidual(
    hidden_states: torch.Tensor,
    normalized_states: torch.Tensor
)
```

Dataclass

Residual tensors retained between a HyperConnection read and write.

**`hidden_states`** `Tensor`

---

**`normalized_states`** `Tensor`

---

```python
class nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextQSAAttention(
    config: object,
    layer_idx: int,
    backend: nemo_automodel.components.models.common.BackendConfig
)
```

**Bases:** [Qwen3NextAttention](/nemo-automodel/nemo_automodel/components/models/qwen3_next/layers#nemo_automodel-components-models-qwen3_next-layers-Qwen3NextAttention)

Qwen3.8-Flash-Next gated attention with compressed-block QSA routing.

The main query/key/value, output gate, and output projection retain the
Qwen3-Next equations.  A separate frozen indexer returns logical token IDs,
then the model-owned QSA dispatcher evaluates only those IDs. CUDA BF16
training uses FlexAttention over a route-membership BlockMask; CPU and
explicit reference backends use the PyTorch oracle. Main Q/K/V remain
differentiable.

**`_cp_mesh`** `DeviceMesh | None = None`

---

**`indexer`** `= Qwen3_8_FlashNextQSAIndexer(config, backend)`

---

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextQSAAttention.forward(
    x: torch.Tensor,
    freqs_cis: torch.Tensor,
    attention_mask: torch.Tensor | None = None,
    cp_context: nemo_automodel.components.models.qwen3_8_flash_next.cp.Qwen3_8_FlashNextCPContext | None = None,
    attn_kwargs: object = {}
) -> torch.Tensor
```

Select compressed blocks and run model-owned sparse GQA.

**Parameters:**

**`x`** `torch.Tensor`

Block input of shape `[batch, sequence, hidden_size]`.

---

**`freqs_cis`** `torch.Tensor`

Rotary values `[batch, sequence, rotary_dim]` whose
final axis stores concatenated cosine and sine values.

---

**`attention_mask`** `torch.Tensor | None` — default: None

Optional token mask of shape `[batch, sequence]`
or backend-specific causal attention mask.

---

**`cp_context`** `Qwen3_8_FlashNextCPContext | None` — default: None

Optional contiguous CP metadata. When present, `x`
contains local queries while compressed/main K/V are gathered
to global rank order.

---

**`**attn_kwargs`** `object` — default: \{}

Backend attention metadata. Packed layouts provide
global document boundaries through `cu_seqlens` or
`cp_context`.

---

**Returns:** `torch.Tensor`

Attention output with the same shape as `x`.

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextQSAAttention.init_weights(
    buffer_device: torch.device,
    init_std: float = 0.02
) -> None
```

Initialize attention and indexer weights.

**Parameters:**

**`buffer_device`** `torch.device`

Device retained for the common attention initializer contract.

---

**`init_std`** `float` — default: 0.02

Projection initialization standard deviation.

---

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextQSAAttention.setup_cp_attention(
    cp_mesh: torch.distributed.device_mesh.DeviceMesh
) -> None
```

Install the contiguous CP mesh used for QSA K/V exchange.

```python
class nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextRMSNormGated(
    hidden_size: int,
    eps: float = 1e-06,
    activation: str = 'sigmoid',
    dtype: torch.dtype | None = None
)
```

**Bases:** `Module`

Qwen3.8-Flash-Next GDN output normalization with its checkpoint-selected gate.

Transformers' Qwen3.5 GatedDeltaNet hard-codes a SiLU output gate.  Qwen3.8-Flash-Next
keeps the same projections and delta-rule core but sets
`output_gate_type='sigmoid'`.  Keeping this model-local module avoids
changing the shared Qwen3.5 execution contract.

**`weight`** `= nn.Parameter(torch.ones(hidden_size, dtype=dtype))`

---

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextRMSNormGated.forward(
    hidden_states: torch.Tensor,
    gate: torch.Tensor
) -> torch.Tensor
```

Normalize in fp32, then apply the configured gate in fp32.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape `[..., hidden_size]` containing
GatedDeltaNet values.

---

**`gate`** `torch.Tensor`

Tensor of shape `[..., hidden_size]` containing the
elementwise output-gate logits.

---

**Returns:** `torch.Tensor`

Tensor of shape `[..., hidden_size]` in the input dtype.

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers.Qwen3_8_FlashNextRMSNormGated.reset_parameters() -> None
```

Match the multiplicative RMSNorm checkpoint convention.

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers._grouped_rms_norm_fp32(
    hidden_states: torch.Tensor,
    weight: torch.Tensor,
    group_size: int,
    eps: float
) -> torch.Tensor
```

FP32 grouped RMSNorm as one fusable elementwise chain.

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers._rms_norm_gated_fp32(
    hidden_states: torch.Tensor,
    gate: torch.Tensor,
    weight: torch.Tensor,
    eps: float,
    use_sigmoid: bool
) -> torch.Tensor
```

FP32 RMSNorm + output gate as one fusable elementwise chain.

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers._grouped_rms_norm_fp32_compiled = torch.compile(_grouped_rms_norm_fp32, dynamic=True)
```

```python
nemo_automodel.components.models.qwen3_8_flash_next.layers._rms_norm_gated_fp32_compiled = torch.compile(_rms_norm_gated_fp32, dynamic=True)
```