> This page is for version 0.5.0 · 26.06.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.models.ernie5_5.moe

ERNIE 5.5 latent mixture-of-experts: router, gate and MoE block.

* `route`: the dual-projection sigmoid router. It returns the selected routing weights and expert IDs for local
  token states. It is a pure function of its inputs: it does no load tracking, no correction-bias update and no
  communication between ranks.
* `Ernie5_5Gate`: `route` behind AutoModel's MoE `Gate` interface. It owns the two FP32 gate matrices, the
  learned per-expert scale and the selection-only correction bias, and records expert loads for the bias update.
* `Ernie5_5SharedExpert`: the shared expert with the SiTU activation.
* `Ernie5_5MoE`: the MoE block: FP32 latent projections around the routed experts, a latent RMSNorm and the shared
  expert.

Supported execution: the torch linear backend with one of two expert and dispatcher pairs.

* `experts="torch"` with `dispatcher="torch"`: the routed experts run in `GroupedExperts`, which gathers the
  tokens of the expert-parallel group, loops over the local experts and sums the selected expert outputs of a
  token in FP32.
* `experts="torch_mm"` with `dispatcher="hybridep"`: with more than one process the routed experts run in
  `GroupedExpertsDeepEP`, which exchanges the tokens through HybridEP and runs two grouped GEMMs. The HybridEP
  combine also sums the selected expert outputs of a token in FP32, but it rounds the partial sum of every
  expert-parallel rank to the activation dtype before it adds the partial sums. The result differs from the
  first pair by BF16 rounding and depends on `ep_size`. Only BF16 activations are supported there. In a single
  process the shared `MoE` builds `GroupedExperts` on its grouped-GEMM path.

The two pairs store the expert weights in the same parameters with the same placements. In a single process there
is no placement requirement. With more than one process the expert weights must be `DTensor` objects on an
`("ep",)` mesh with placements `(Shard(0),)` or on an `("ep_shard", "ep")` mesh with placements
`(Shard(1), Shard(0))`; `Ernie5_5MoE.forward` raises `RuntimeError` for any other layout. Routing uses the
local token states, so a context-parallel shard is routed like any other row.

Provenance: the expert-selection and routing-weight contracts of `route` are those of the ERNIE 5.5 implementation
in Megatron-Bridge (Apache License 2.0) at revision `a38833622b88969564fe8bfa053c5e4e4989c6cd`:
[https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/moe.py](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/moe.py)

## Module Contents

### Classes

| Name                                                                                          | Description                                                             |
| --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
| [`Ernie5_5Gate`](#nemo_automodel-components-models-ernie5_5-moe-Ernie5_5Gate)                 | ERNIE 5.5 gate using AutoModel's MoE gate interface.                    |
| [`Ernie5_5MoE`](#nemo_automodel-components-models-ernie5_5-moe-Ernie5_5MoE)                   | ERNIE dual-router latent experts and shared SiTU expert.                |
| [`Ernie5_5SharedExpert`](#nemo_automodel-components-models-ernie5_5-moe-Ernie5_5SharedExpert) | Preserve ERNIE shared-expert arithmetic at a callable FSDP boundary.    |
| [`_FP32LatentLinear`](#nemo_automodel-components-models-ernie5_5-moe-_FP32LatentLinear)       | Project in the activation dtype while preserving an FP32 master weight. |

### Functions

| Name                                                                                                    | Description                                                                          |
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| [`_make_callable_projection`](#nemo_automodel-components-models-ernie5_5-moe-_make_callable_projection) | Preserve the initialized parameter while adding a callable FSDP boundary.            |
| [`_routed_situ`](#nemo_automodel-components-models-ernie5_5-moe-_routed_situ)                           | Apply SiTU and the routing weight to the packed gate/up output of one routed expert. |
| [`route`](#nemo_automodel-components-models-ernie5_5-moe-route)                                         | Return selected routing weights and expert IDs.                                      |

### Data

[`_EXPERT_BACKENDS`](#nemo_automodel-components-models-ernie5_5-moe-_EXPERT_BACKENDS)

### API

```python
class nemo_automodel.components.models.ernie5_5.moe.Ernie5_5Gate(
    config: nemo_automodel.components.moe.config.MoEConfig
)
```

**Bases:** [Gate](/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/moe/layers#nemo_automodel-components-moe-layers-Gate)

ERNIE 5.5 gate using AutoModel's MoE gate interface.

The gate supports one expert group, sigmoid scores with a selection-only correction bias and FP32 routing
weights. The auxiliary loss, a gate bias and routing replay are not supported; the constructor raises
`ValueError` for such a configuration. Routing uses the local token states of the calling process. Expert
loads for the correction-bias update are accumulated only in training mode with gradients enabled, so a
no-grad forward followed by its recomputation with gradients is counted once.

The parameters `weight` and `weight_1` of shape \[experts, hidden] and `routed_scaling_factor_param` of
shape \[experts] are FP32, and `forward` raises if any of them has another dtype; preserving FP32 under FSDP
mixed precision is the responsibility of the model that owns the gate.

**`routed_scaling_factor_param`**

---

**`weight_1`**

---

```python
nemo_automodel.components.models.ernie5_5.moe.Ernie5_5Gate.forward(
    x: torch.Tensor,
    token_mask: torch.Tensor,
    cp_mesh: torch.distributed.device_mesh.DeviceMesh | None
) -> tuple[torch.Tensor, torch.Tensor, None]
```

Route local token states and record valid-token loads.

**Parameters:**

**`x`** `torch.Tensor`

Local token states of shape \[tokens, hidden].

---

**`token_mask`** `torch.Tensor`

Boolean valid-token mask of shape \[tokens].

---

**`cp_mesh`** `DeviceMesh | None`

Context mesh; auxiliary loss is unsupported and it is unused.

---

**Returns:** `torch.Tensor`

Tuple of FP32 routing weights \[tokens, topk], integer expert IDs

**Raises:**

* `ValueError`: If `token_mask` does not have one entry per token, the correction bias is not on the
  device of `x`, or `route` rejects the shapes or dtypes.
* `RuntimeError`: If no correction bias is available.

```python
nemo_automodel.components.models.ernie5_5.moe.Ernie5_5Gate.init_weights(
    buffer_device: torch.device,
    init_std: float = 0.02
) -> None
```

Initialize both gates and learned expert scales after meta materialization.

**Parameters:**

**`buffer_device`** `torch.device`

Device passed to the parent initializer.

---

**`init_std`** `float` — default: 0.02

Standard deviation of the normal initialization of both gate matrices.

---

```python
class nemo_automodel.components.models.ernie5_5.moe.Ernie5_5MoE(
    config: nemo_automodel.components.moe.config.MoEConfig,
    backend: nemo_automodel.components.models.common.BackendConfig,
    beta: float = 4.0,
    linear_beta: float | None = 25.0,
    norm_eps: float = 1e-06
)
```

**Bases:** [MoE](/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/moe/layers#nemo_automodel-components-moe-layers-MoE)

ERNIE dual-router latent experts and shared SiTU expert.

The routed experts are the shared `GroupedExperts` (torch dispatcher, or one process) or the shared
`GroupedExpertsDeepEP` (HybridEP dispatcher in a job of several processes), with SiTU installed as their
activation. Distributed execution is restricted to the expert-parallel layouts that `forward` checks, and
with the HybridEP dispatcher to BF16 activations.

**`_dispatches_through_hybridep`** `= isinstance(self.experts, GroupedExpertsDeepEP)`

---

**`fc1_latent_proj`** `= _make_callable_projection(self.fc1_latent_proj)`

---

**`fc2_latent_proj`** `= _make_callable_projection(self.fc2_latent_proj)`

---

**`gate`** `= Ernie5_5Gate(config)`

---

**`latent_norm`**

---

**`shared_experts`**

---

```python
nemo_automodel.components.models.ernie5_5.moe.Ernie5_5MoE.forward(
    hidden: torch.Tensor,
    padding_mask: torch.Tensor | None = None,
    cp_mesh: torch.distributed.device_mesh.DeviceMesh | None = None
) -> torch.Tensor
```

Run the MoE.

**Parameters:**

**`hidden`** `torch.Tensor`

Tensor of shape \[..., hidden] with arbitrary leading dimensions.

---

**`padding_mask`** `torch.Tensor | None` — default: None

Boolean tensor of shape \[...] where true marks padding.

---

**`cp_mesh`** `DeviceMesh | None` — default: None

Context-parallel mesh; unused, because the gate computes no auxiliary loss.

---

**Returns:** `torch.Tensor`

Output of shape \[..., hidden].

**Raises:**

* `RuntimeError`: If, with more than one
  process, the expert weights are not `DTensor` objects with the same mesh and placements in a
  supported layout (mesh `("ep",)` with placements `(Shard(0),)`, or mesh `("ep_shard", "ep")`
  with placements `(Shard(1), Shard(0))`, with more than one expert-parallel rank); if, with more
  than one process, the HybridEP dispatcher was requested but the module was built for one process;
  if a latent projection weight is not FP32; or if the routed experts dispatch through HybridEP and
  the latent activations are not BF16.
* `ValueError`: If `hidden` is empty, has fewer than two dimensions or does not end in the configured
  hidden size, or `padding_mask` does not have the shape of `hidden` without its final axis.

```python
nemo_automodel.components.models.ernie5_5.moe.Ernie5_5MoE.init_weights(
    buffer_device: torch.device,
    init_std: float = 0.02
) -> None
```

Initialize standard MoE modules plus both ERNIE router projections.

**Parameters:**

**`buffer_device`** `torch.device`

Device passed to the parent initializer.

---

**`init_std`** `float` — default: 0.02

Standard deviation of the normal initializations.

---

```python
class nemo_automodel.components.models.ernie5_5.moe.Ernie5_5SharedExpert(
    old: nemo_automodel.components.moe.layers.MLP,
    beta: float,
    linear_beta: float | None
)
```

**Bases:** [MLP](/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/moe/layers#nemo_automodel-components-moe-layers-MLP)

Preserve ERNIE shared-expert arithmetic at a callable FSDP boundary.

The module has the gate, up and down projections of the shared `MLP` and computes SiTU between them. Its
`activation` attribute is `"situ"`: code that selects a kernel from that attribute, such as the fused LoRA
MLP path, finds no SwiGLU module here and leaves `forward` in place.

**`activation`** `= 'situ'`

---

```python
nemo_automodel.components.models.ernie5_5.moe.Ernie5_5SharedExpert.forward(
    x: torch.Tensor
) -> torch.Tensor
```

Run SiTU and the output projection inside the FSDP-wrapped module.

**Parameters:**

**`x`** `torch.Tensor`

Tensor of shape \[..., hidden], with arbitrary leading dimensions.

---

**Returns:** `torch.Tensor`

Tensor of shape \[..., hidden].

```python
class nemo_automodel.components.models.ernie5_5.moe._FP32LatentLinear()
```

**Bases:** `Linear`

Project in the activation dtype while preserving an FP32 master weight.

```python
nemo_automodel.components.models.ernie5_5.moe._FP32LatentLinear.forward(
    hidden: torch.Tensor
) -> torch.Tensor
```

Project \[..., in_features] to \[..., out_features].

**Parameters:**

**`hidden`** `torch.Tensor`

Tensor of shape \[..., in_features], with arbitrary leading dimensions.

---

**Returns:** `torch.Tensor`

Tensor of shape \[..., out_features] in the input dtype; the FP32 weight is converted to that dtype

**Raises:**

* `RuntimeError`: If the weight is not FP32.

```python
nemo_automodel.components.models.ernie5_5.moe._make_callable_projection(
    projection: torch.nn.Linear
) -> nemo_automodel.components.models.ernie5_5.moe._FP32LatentLinear
```

Preserve the initialized parameter while adding a callable FSDP boundary.

**Parameters:**

**`projection`** `nn.Linear`

Bias-free torch linear module.

---

**Returns:** `_FP32LatentLinear`

Projection module whose `weight` is the same parameter object as `projection.weight`.

**Raises:**

* `TypeError`: If `projection` is not an `nn.Linear` or has a bias.

```python
nemo_automodel.components.models.ernie5_5.moe._routed_situ(
    x: torch.Tensor,
    weights: torch.Tensor,
    beta: float,
    linear_beta: float | None
) -> torch.Tensor
```

Apply SiTU and the routing weight to the packed gate/up output of one routed expert.

**Parameters:**

**`x`** `torch.Tensor`

Tensor of shape \[routed_tokens, 2 \* intermediate], with the gate followed by the up projection on the
final axis. `routed_tokens` is the number of (token, top-k slot) pairs routed to the expert.

---

**`weights`** `torch.Tensor`

Routing weights of shape \[routed_tokens, 1].

---

**`beta`** `float`

Gate-clamp parameter of SiTU.

---

**`linear_beta`** `float | None`

Optional up-projection clamp parameter of SiTU.

---

**Returns:** `torch.Tensor`

Tensor of shape \[routed_tokens, intermediate] in the dtype of `x`.

```python
nemo_automodel.components.models.ernie5_5.moe.route(
    hidden: torch.Tensor,
    gate_a: torch.Tensor,
    gate_b: torch.Tensor,
    correction: torch.Tensor,
    expert_scale: torch.Tensor,
    top_k: int,
    normalize: bool = True
) -> tuple[torch.Tensor, torch.Tensor]
```

Return selected routing weights and expert IDs.

**Parameters:**

**`hidden`** `torch.Tensor`

Local token states of shape \[tokens, hidden].

---

**`gate_a`** `torch.Tensor`

First FP32 gate matrix of shape \[experts, hidden].

---

**`gate_b`** `torch.Tensor`

Second FP32 gate matrix of shape \[experts, hidden].

---

**`correction`** `torch.Tensor`

FP32 selection-only bias of shape \[experts].

---

**`expert_scale`** `torch.Tensor`

Learnable FP32 per-expert scale of shape \[experts].

---

**`top_k`** `int`

Number of distinct experts selected per token.

---

**`normalize`** `bool` — default: True

Normalize selected unbiased scores when top_k exceeds one.

---

**Returns:** `torch.Tensor`

FP32 weights and integer expert IDs, each of shape \[tokens, top_k].

**Raises:**

* `ValueError`: If the tensor shapes do not agree, `top_k` is not between 1 and the number of experts, or a
  gate matrix, `correction` or `expert_scale` is not FP32.

```python
nemo_automodel.components.models.ernie5_5.moe._EXPERT_BACKENDS = (('torch', 'torch'), ('torch_mm', 'hybridep'))
```