> This page is for version 0.7.0 · 26.10.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.models.ernie5_5.layers

ERNIE 5.5 normalization, activation and residual-stream layers.

This module holds the leaf layers of the ERNIE 5.5 decoder:

* `Ernie5_5RMSNorm`: RMSNorm that rounds its input to the weight dtype before computing FP32 statistics.
* `ernie_situ`: the SiTU activation on a packed gate/up tensor, with its checkpoint-compatible rounding points.
* `virtual_head_mix`: the low-rank virtual-head mix `I + V U^T` applied to per-head attention output.
* `Ernie5_5HyperConnection`: the multi-stream residual mixer (mHC) with FP32 stream arithmetic.
* `Ernie5_5HyperHead`: the learned contraction of the residual streams into one output stream.
* `Ernie5_5MultiMax`: the learned piecewise correction of vocabulary logits.
* `Ernie5_5DenseMLP`: the bias-free dense feed-forward block built on `ernie_situ`.

It also defines `DENSE_ATTENTION_BACKENDS`, the `backend.attn` names that the two attention modules of the
package, `dsa` and `hca`, accept for their dense PyTorch path, and `shard_start`, the row offset of a
context-parallel shard.

Provenance: the definitions below follow the ERNIE 5.5 implementation in Megatron-Bridge (Apache License 2.0) at
revision `a38833622b88969564fe8bfa053c5e4e4989c6cd`.

* `Ernie5_5RMSNorm` (arithmetic and input conversion):
  [https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/norm.py](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/norm.py)
* `ernie_situ` (operation and rounding order):
  [https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/moe.py](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/moe.py)
* `virtual_head_mix` (mathematical contract of `vha_postmix`):
  [https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/ops.py](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/ops.py)
* `Ernie5_5HyperConnection` (arithmetic):
  [https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/hyper_connection.py](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/hyper_connection.py)
* `Ernie5_5MultiMax` (formula and 512-row accumulation order):
  [https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/output_layer.py](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/output_layer.py)

## Module Contents

### Classes

| Name                                                                                                   | Description                                                               |
| ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------- |
| [`Ernie5_5DenseMLP`](#nemo_automodel-components-models-ernie5_5-layers-Ernie5_5DenseMLP)               | Bias-free gate/up/down projections and ERNIE SiTU.                        |
| [`Ernie5_5HyperConnection`](#nemo_automodel-components-models-ernie5_5-layers-Ernie5_5HyperConnection) | Use ERNIE normalization and FP32 stream mixing with eager Sinkhorn.       |
| [`Ernie5_5HyperHead`](#nemo_automodel-components-models-ernie5_5-layers-Ernie5_5HyperHead)             | Collapse multiple residual streams using FP32 learned coefficients.       |
| [`Ernie5_5MultiMax`](#nemo_automodel-components-models-ernie5_5-layers-Ernie5_5MultiMax)               | Apply learned MultiMax correction to materialized vocabulary logits.      |
| [`Ernie5_5RMSNorm`](#nemo_automodel-components-models-ernie5_5-layers-Ernie5_5RMSNorm)                 | Round inputs to the weight dtype, normalize in FP32 and round the result. |
| [`_MultiMaxFunction`](#nemo_automodel-components-models-ernie5_5-layers-_MultiMaxFunction)             | Chunk temporary logits and accumulate parameter gradients in FP32.        |

### Functions

| Name                                                                                     | Description                                                                     |
| ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| [`_correction`](#nemo_automodel-components-models-ernie5_5-layers-_correction)           | Compute piecewise correction in FP32.                                           |
| [`ernie_situ`](#nemo_automodel-components-models-ernie5_5-layers-ernie_situ)             | Apply SiTU with rounding before optional routing-weight multiplication.         |
| [`shard_start`](#nemo_automodel-components-models-ernie5_5-layers-shard_start)           | Return the row offset of the first token that this context-parallel rank holds. |
| [`virtual_head_mix`](#nemo_automodel-components-models-ernie5_5-layers-virtual_head_mix) | Apply the low-rank head mix I + V U^T to attention output.                      |

### Data

[`DENSE_ATTENTION_BACKENDS`](#nemo_automodel-components-models-ernie5_5-layers-DENSE_ATTENTION_BACKENDS)

[`_CHUNK_ROWS`](#nemo_automodel-components-models-ernie5_5-layers-_CHUNK_ROWS)

### API

```python
class nemo_automodel.components.models.ernie5_5.layers.Ernie5_5DenseMLP(
    hidden: int,
    intermediate: int,
    dtype: torch.dtype,
    beta: float = 4.0,
    linear_beta: float = 25.0
)
```

**Bases:** `Module`

Bias-free gate/up/down projections and ERNIE SiTU.

**`down_proj`**

---

**`gate_proj`**

---

**`up_proj`**

---

```python
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5DenseMLP.forward(
    x: torch.Tensor
) -> torch.Tensor
```

Compute the dense FFN on normalized activations.

**Parameters:**

**`x`** `torch.Tensor`

Tensor of shape \[..., hidden], with arbitrary leading dimensions.

---

**Returns:** `torch.Tensor`

Tensor of shape \[..., hidden].

```python
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5DenseMLP.init_weights(
    init_std: float = 0.02
) -> None
```

Initialize projection weights after meta materialization.

**Parameters:**

**`init_std`** `float` — default: 0.02

Standard deviation of the normal initialization of the three projection weights.

---

**Raises:**

* `ValueError`: If `init_std` is not positive.

```python
class nemo_automodel.components.models.ernie5_5.layers.Ernie5_5HyperConnection(
    hc_mult: int,
    hidden_size: int,
    hc_sinkhorn_iters: int,
    hc_eps: float,
    rms_norm_eps: float
)
```

**Bases:** [DeepseekV4HyperConnection](/nemo/automodel/v0.7/nemo-automodel/nemo_automodel/components/models/deepseek_v4/layers#nemo_automodel-components-models-deepseek_v4-layers-DeepseekV4HyperConnection)

Use ERNIE normalization and FP32 stream mixing with eager Sinkhorn.

The parameters are those of `DeepseekV4HyperConnection`: `fn` of shape \[streams \* (streams + 2),
streams \* hidden], `base` of shape \[streams \* (streams + 2)] and `scale` of shape \[3], all FP32. The class
implements the mixing arithmetic only; checkpoint key mapping, the lifetime of the residual streams, the
mixed-precision policy, activation recomputation and distributed placement belong to the code that owns the
module.

```python
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5HyperConnection._project(
    hidden_streams: torch.Tensor,
    fn: torch.Tensor,
    base: torch.Tensor,
    scale: torch.Tensor,
    hc: int,
    norm_eps: float,
    hc_eps: float
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]
```

staticmethod

Predict the mixing coefficients before Sinkhorn normalization, in FP32 with autocast disabled.

**Parameters:**

**`hidden_streams`** `torch.Tensor`

Tensor of shape \[batch, sequence, streams, hidden].

---

**`fn`** `torch.Tensor`

Projection of shape \[mix_components, streams \* hidden], where mix_components is
streams \* (streams + 2).

---

**`base`** `torch.Tensor`

Offsets of shape \[mix_components], ordered pre, post, combination.

---

**`scale`** `torch.Tensor`

Gains of shape \[3], ordered pre, post, combination.

---

**`hc`** `int`

Number of residual streams.

---

**`norm_eps`** `float`

Epsilon added to the RMS of the flattened streams.

---

**`hc_eps`** `float`

Epsilon added to the pre coefficients.

---

**Returns:** `torch.Tensor`

FP32 pre and post coefficients of shape \[batch, sequence, streams] and FP32 combination logits of

```python
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5HyperConnection.collapse(
    hidden_streams: torch.Tensor,
    pre: torch.Tensor
) -> torch.Tensor
```

staticmethod

Sum streams in FP32; the following layer owns any downcast.

**Parameters:**

**`hidden_streams`** `torch.Tensor`

Tensor of shape \[batch, sequence, streams, hidden].

---

**`pre`** `torch.Tensor`

Coefficients of shape \[batch, sequence, streams].

---

**Returns:** `torch.Tensor`

FP32 tensor of shape \[batch, sequence, hidden]. Unlike the parent class, the result is not converted

```python
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5HyperConnection.expand(
    output: torch.Tensor,
    residual: torch.Tensor,
    post: torch.Tensor,
    comb: torch.Tensor
) -> torch.Tensor
```

staticmethod

Mix the residual and layer output in FP32, rounding once.

**Parameters:**

**`output`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden].

---

**`residual`** `torch.Tensor`

Tensor of shape \[batch, sequence, streams, hidden].

---

**`post`** `torch.Tensor`

Coefficients of shape \[batch, sequence, streams].

---

**`comb`** `torch.Tensor`

Coefficients of shape \[batch, sequence, input_streams, output_streams].

---

**Returns:** `torch.Tensor`

Tensor of shape \[batch, sequence, streams, hidden] in the residual dtype.

```python
class nemo_automodel.components.models.ernie5_5.layers.Ernie5_5HyperHead(
    streams: int,
    hidden: int,
    eps: float = 1e-06
)
```

**Bases:** `Module`

Collapse multiple residual streams using FP32 learned coefficients.

The parameters `hc_fn` of shape \[streams, streams \* hidden], `hc_base` of shape \[streams] and `hc_scale`
of shape \[1] are created in FP32, and `forward` raises if any of them has another dtype. Full-model and FSDP
precision policies remain the caller's responsibility.

**`hc_base`**

---

**`hc_fn`**

---

**`hc_scale`** `= nn.Parameter(torch.ones(1, dtype=(torch.float32)))`

---

```python
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5HyperHead.forward(
    x: torch.Tensor
) -> torch.Tensor
```

Contract \[batch, sequence, streams, hidden] into \[batch, sequence, hidden].

**Parameters:**

**`x`** `torch.Tensor`

Tensor of shape \[batch, sequence, streams, hidden].

---

**Returns:** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden] in the input dtype. The stream weights are computed and

**Raises:**

* `ValueError`: If `x` does not have shape \[batch, sequence, streams, hidden].
* `RuntimeError`: If `hc_fn`, `hc_base` or `hc_scale` is not FP32.

```python
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5HyperHead.init_weights(
    init_std: float = 0.02
) -> None
```

Initialize parameters after meta materialization.

**Parameters:**

**`init_std`** `float` — default: 0.02

Standard deviation of the normal initialization of `hc_fn`.

---

**Raises:**

* `ValueError`: If `init_std` is not positive.

```python
class nemo_automodel.components.models.ernie5_5.layers.Ernie5_5MultiMax(
    dtype: torch.dtype = torch.float32
)
```

**Bases:** `Module`

Apply learned MultiMax correction to materialized vocabulary logits.

The thresholds `multimax_ranges` and the scales `multimax_ts` each have shape \[4].

**`multimax_ranges`** `= nn.Parameter(torch.zeros(4, dtype=dtype))`

---

**`multimax_ts`** `= nn.Parameter(torch.zeros(4, dtype=dtype))`

---

```python
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5MultiMax.forward(
    logits: torch.Tensor
) -> torch.Tensor
```

Correct logits while retaining their shape and dtype.

**Parameters:**

**`logits`** `torch.Tensor`

Floating tensor of shape \[..., vocab], with arbitrary
leading dimensions and nonempty `vocab`.

---

**Returns:** `torch.Tensor`

Corrected tensor of shape \[..., vocab], with the input dtype and

**Raises:**

* `ValueError`: If `logits` has no dimensions, an empty final dimension, or a dtype that is not floating
  point.

```python
class nemo_automodel.components.models.ernie5_5.layers.Ernie5_5RMSNorm(
    hidden_size: int,
    eps: float = 1e-06,
    dtype: torch.dtype = torch.float32
)
```

**Bases:** `Module`

Round inputs to the weight dtype, normalize in FP32 and round the result.

The scale parameter `weight` has shape \[hidden] and the configured dtype.

**`weight`** `= nn.Parameter(torch.ones(hidden_size, dtype=dtype))`

---

```python
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5RMSNorm.forward(
    x: torch.Tensor
) -> torch.Tensor
```

Normalize arbitrary leading dimensions over the final hidden axis.

**Parameters:**

**`x`** `torch.Tensor`

Floating tensor of shape \[..., hidden], with arbitrary leading
dimensions and `hidden` equal to the scale parameter width.

---

**Returns:** `torch.Tensor`

Tensor of shape \[..., hidden] on the input device, in the scale

**Raises:**

* `ValueError`: If `x` has no dimensions, its final dimension differs from the width of the scale
  parameter, or its dtype is not floating point.

```python
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5RMSNorm.reset_parameters() -> None
```

Initialize the scale, including after meta-device materialization.

```python
class nemo_automodel.components.models.ernie5_5.layers._MultiMaxFunction()
```

**Bases:** `Function`

Chunk temporary logits and accumulate parameter gradients in FP32.

```python
nemo_automodel.components.models.ernie5_5.layers._MultiMaxFunction.backward(
    ctx: torch.autograd.function.FunctionCtx,
    grad_output: torch.Tensor
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]
```

staticmethod

Compute first-order gradients with FP32 parameter accumulation.

**Parameters:**

**`ctx`** `torch.autograd.function.FunctionCtx`

Autograd context with the forward inputs.

---

**`grad_output`** `torch.Tensor`

Incoming gradient of shape \[..., vocab], matching forward logits.

---

**Returns:** `torch.Tensor`

Logits gradient of shape \[..., vocab], threshold gradient \[4] and scale

```python
nemo_automodel.components.models.ernie5_5.layers._MultiMaxFunction.forward(
    ctx: torch.autograd.function.FunctionCtx,
    logits: torch.Tensor,
    ranges: torch.Tensor,
    scales: torch.Tensor
) -> torch.Tensor
```

staticmethod

Correct logits using learned thresholds and scales.

**Parameters:**

**`ctx`** `torch.autograd.function.FunctionCtx`

Autograd context retaining inputs for first-order gradients.

---

**`logits`** `torch.Tensor`

Floating tensor of shape \[..., vocab], with arbitrary leading dimensions.

---

**`ranges`** `torch.Tensor`

Floating threshold tensor of shape \[4], on the logits device.

---

**`scales`** `torch.Tensor`

Floating scale tensor of shape \[4], on the logits device.

---

**Returns:** `torch.Tensor`

Corrected tensor of shape \[..., vocab], with logits dtype and device.

```python
nemo_automodel.components.models.ernie5_5.layers._correction(
    logits: torch.Tensor,
    ranges: torch.Tensor,
    scales: torch.Tensor
) -> torch.Tensor
```

Compute piecewise correction in FP32.

**Parameters:**

**`logits`** `torch.Tensor`

FP32 tensor of shape \[rows, vocab].

---

**`ranges`** `torch.Tensor`

FP32 threshold tensor of shape \[4].

---

**`scales`** `torch.Tensor`

FP32 scale tensor of shape \[4].

---

**Returns:** `torch.Tensor`

FP32 corrected logits of shape \[rows, vocab].

```python
nemo_automodel.components.models.ernie5_5.layers.ernie_situ(
    gate_up: torch.Tensor,
    beta: float,
    linear_beta: float | None,
    routing_weight: torch.Tensor | None = None
) -> torch.Tensor
```

Apply SiTU with rounding before optional routing-weight multiplication.

**Parameters:**

**`gate_up`** `torch.Tensor`

Floating tensor of shape \[..., 2 \* intermediate], with
arbitrary leading dimensions and the gate followed by the up
projection on the final axis.

---

**`beta`** `float`

Positive finite gate-clamp parameter.

---

**`linear_beta`** `float | None`

Optional positive finite up-projection clamp parameter.

---

**`routing_weight`** `torch.Tensor | None` — default: None

Optional tensor broadcastable with \[..., intermediate],
typically a routing weight of shape \[..., 1].

---

**Returns:** `torch.Tensor`

Activated tensor with the broadcast shape of \[..., intermediate] and

**Raises:**

* `ValueError`: If the final dimension of `gate_up` is odd, `gate_up` is not floating point, or `beta` or
  `linear_beta` is not a positive finite number.

```python
nemo_automodel.components.models.ernie5_5.layers.shard_start(
    cp_group: torch.distributed.ProcessGroup | None,
    sequence: int
) -> int
```

Return the row offset of the first token that this context-parallel rank holds.

**Parameters:**

**`cp_group`** `dist.ProcessGroup | None`

Context-parallel group whose ranks hold consecutive shards of a row, or None.

---

**`sequence`** `int`

Number of tokens in every shard.

---

```python
nemo_automodel.components.models.ernie5_5.layers.virtual_head_mix(
    attention: torch.Tensor,
    u: torch.Tensor,
    v: torch.Tensor
) -> torch.Tensor
```

Apply the low-rank head mix I + V U^T to attention output.

Only the activation-dtype path is implemented: `u` and `v` are converted to the dtype of `attention` and
the mix runs in that dtype with autocast disabled.

**Parameters:**

**`attention`** `torch.Tensor`

Tensor of shape \[..., heads, channels], with arbitrary
leading dimensions. The final two axes are head and head channel.

---

**`u`** `torch.Tensor`

Input-head factor of shape \[heads, rank] on the same device.

---

**`v`** `torch.Tensor`

Output-head factor of shape \[heads, rank] on the same device.

---

**Returns:** `torch.Tensor`

Newly allocated tensor with the same shape and dtype as attention.

**Raises:**

* `ValueError`: If `attention` has fewer than two dimensions, or `u` and `v` are not equal-shaped
  matrices with one row per attention head.

```python
nemo_automodel.components.models.ernie5_5.layers.DENSE_ATTENTION_BACKENDS: tuple[str, ...] = ('eager', 'torch', 'sdpa')
```

```python
nemo_automodel.components.models.ernie5_5.layers._CHUNK_ROWS = 512
```