Skip to navigation

nemo_automodel.components.models.ernie5_5.moe

View as MarkdownOpen in Claude

ERNIE 5.5 latent mixture-of-experts: router, gate and MoE block.

  • route: the dual-projection sigmoid router. It returns the selected routing weights and expert IDs for local token states. It is a pure function of its inputs: it does no load tracking, no correction-bias update and no communication between ranks.
  • Ernie5_5Gate: route behind AutoModel’s MoE Gate interface. It owns the two FP32 gate matrices, the learned per-expert scale and the selection-only correction bias, and records expert loads for the bias update.
  • Ernie5_5SharedExpert: the shared expert with the SiTU activation.
  • Ernie5_5MoE: the MoE block: FP32 latent projections around the routed experts, a latent RMSNorm and the shared expert.

Supported execution: the torch linear backend with one of two expert and dispatcher pairs.

  • experts="torch" with dispatcher="torch": the routed experts run in GroupedExperts, which gathers the tokens of the expert-parallel group, loops over the local experts and sums the selected expert outputs of a token in FP32.
  • experts="torch_mm" with dispatcher="hybridep": with more than one process the routed experts run in GroupedExpertsDeepEP, which exchanges the tokens through HybridEP and runs two grouped GEMMs. The HybridEP combine also sums the selected expert outputs of a token in FP32, but it rounds the partial sum of every expert-parallel rank to the activation dtype before it adds the partial sums. The result differs from the first pair by BF16 rounding and depends on ep_size. Only BF16 activations are supported there. In a single process the shared MoE builds GroupedExperts on its grouped-GEMM path.

The two pairs store the expert weights in the same parameters with the same placements. In a single process there is no placement requirement. With more than one process the expert weights must be DTensor objects on an ("ep",) mesh with placements (Shard(0),) or on an ("ep_shard", "ep") mesh with placements (Shard(1), Shard(0)); Ernie5_5MoE.forward raises RuntimeError for any other layout. Routing uses the local token states, so a context-parallel shard is routed like any other row.

Provenance: the expert-selection and routing-weight contracts of route are those of the ERNIE 5.5 implementation in Megatron-Bridge (Apache License 2.0) at revision a38833622b88969564fe8bfa053c5e4e4989c6cd: https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/moe.py

Module Contents

Classes

NameDescription
Ernie5_5GateERNIE 5.5 gate using AutoModel’s MoE gate interface.
Ernie5_5MoEERNIE dual-router latent experts and shared SiTU expert.
Ernie5_5SharedExpertPreserve ERNIE shared-expert arithmetic at a callable FSDP boundary.
_FP32LatentLinearProject in the activation dtype while preserving an FP32 master weight.

Functions

NameDescription
_make_callable_projectionPreserve the initialized parameter while adding a callable FSDP boundary.
_routed_situApply SiTU and the routing weight to the packed gate/up output of one routed expert.
routeReturn selected routing weights and expert IDs.

Data

_EXPERT_BACKENDS

API

class nemo_automodel.components.models.ernie5_5.moe.Ernie5_5Gate(
)

Bases: Gate

ERNIE 5.5 gate using AutoModel’s MoE gate interface.

The gate supports one expert group, sigmoid scores with a selection-only correction bias and FP32 routing weights. The auxiliary loss, a gate bias and routing replay are not supported; the constructor raises ValueError for such a configuration. Routing uses the local token states of the calling process. Expert loads for the correction-bias update are accumulated only in training mode with gradients enabled, so a no-grad forward followed by its recomputation with gradients is counted once.

The parameters weight and weight_1 of shape [experts, hidden] and routed_scaling_factor_param of shape [experts] are FP32, and forward raises if any of them has another dtype; preserving FP32 under FSDP mixed precision is the responsibility of the model that owns the gate.

routed_scaling_factor_param
weight_1
nemo_automodel.components.models.ernie5_5.moe.Ernie5_5Gate.forward(
x: torch.Tensor,
token_mask: torch.Tensor,
cp_mesh: torch.distributed.device_mesh.DeviceMesh | None
) -> tuple[torch.Tensor, torch.Tensor, None]

Route local token states and record valid-token loads.

Parameters:

x
torch.Tensor

Local token states of shape [tokens, hidden].

token_mask
torch.Tensor

Boolean valid-token mask of shape [tokens].

cp_mesh
DeviceMesh | None

Context mesh; auxiliary loss is unsupported and it is unused.

Returns: torch.Tensor

Tuple of FP32 routing weights [tokens, topk], integer expert IDs

Raises:

  • ValueError: If token_mask does not have one entry per token, the correction bias is not on the device of x, or route rejects the shapes or dtypes.
  • RuntimeError: If no correction bias is available.
nemo_automodel.components.models.ernie5_5.moe.Ernie5_5Gate.init_weights(
buffer_device: torch.device,
init_std: float = 0.02
) -> None

Initialize both gates and learned expert scales after meta materialization.

Parameters:

buffer_device
torch.device

Device passed to the parent initializer.

init_std
floatDefaults to 0.02

Standard deviation of the normal initialization of both gate matrices.

class nemo_automodel.components.models.ernie5_5.moe.Ernie5_5MoE(
beta: float = 4.0,
linear_beta: float | None = 25.0,
norm_eps: float = 1e-06
)

Bases: MoE

ERNIE dual-router latent experts and shared SiTU expert.

The routed experts are the shared GroupedExperts (torch dispatcher, or one process) or the shared GroupedExpertsDeepEP (HybridEP dispatcher in a job of several processes), with SiTU installed as their activation. Distributed execution is restricted to the expert-parallel layouts that forward checks, and with the HybridEP dispatcher to BF16 activations.

_dispatches_through_hybridep
= isinstance(self.experts, GroupedExpertsDeepEP)
fc1_latent_proj
= _make_callable_projection(self.fc1_latent_proj)
fc2_latent_proj
= _make_callable_projection(self.fc2_latent_proj)
gate
= Ernie5_5Gate(config)
latent_norm
shared_experts
nemo_automodel.components.models.ernie5_5.moe.Ernie5_5MoE.forward(
hidden: torch.Tensor,
padding_mask: torch.Tensor | None = None,
cp_mesh: torch.distributed.device_mesh.DeviceMesh | None = None
) -> torch.Tensor

Run the MoE.

Parameters:

hidden
torch.Tensor

Tensor of shape […, hidden] with arbitrary leading dimensions.

padding_mask
torch.Tensor | NoneDefaults to None

Boolean tensor of shape […] where true marks padding.

cp_mesh
DeviceMesh | NoneDefaults to None

Context-parallel mesh; unused, because the gate computes no auxiliary loss.

Returns: torch.Tensor

Output of shape […, hidden].

Raises:

  • RuntimeError: If, with more than one process, the expert weights are not DTensor objects with the same mesh and placements in a supported layout (mesh ("ep",) with placements (Shard(0),), or mesh ("ep_shard", "ep") with placements (Shard(1), Shard(0)), with more than one expert-parallel rank); if, with more than one process, the HybridEP dispatcher was requested but the module was built for one process; if a latent projection weight is not FP32; or if the routed experts dispatch through HybridEP and the latent activations are not BF16.
  • ValueError: If hidden is empty, has fewer than two dimensions or does not end in the configured hidden size, or padding_mask does not have the shape of hidden without its final axis.
nemo_automodel.components.models.ernie5_5.moe.Ernie5_5MoE.init_weights(
buffer_device: torch.device,
init_std: float = 0.02
) -> None

Initialize standard MoE modules plus both ERNIE router projections.

Parameters:

buffer_device
torch.device

Device passed to the parent initializer.

init_std
floatDefaults to 0.02

Standard deviation of the normal initializations.

class nemo_automodel.components.models.ernie5_5.moe.Ernie5_5SharedExpert(
beta: float,
linear_beta: float | None
)

Bases: MLP

Preserve ERNIE shared-expert arithmetic at a callable FSDP boundary.

The module has the gate, up and down projections of the shared MLP and computes SiTU between them. Its activation attribute is "situ": code that selects a kernel from that attribute, such as the fused LoRA MLP path, finds no SwiGLU module here and leaves forward in place.

activation
= 'situ'
nemo_automodel.components.models.ernie5_5.moe.Ernie5_5SharedExpert.forward(
x: torch.Tensor
) -> torch.Tensor

Run SiTU and the output projection inside the FSDP-wrapped module.

Parameters:

x
torch.Tensor

Tensor of shape […, hidden], with arbitrary leading dimensions.

Returns: torch.Tensor

Tensor of shape […, hidden].

class nemo_automodel.components.models.ernie5_5.moe._FP32LatentLinear()

Bases: Linear

Project in the activation dtype while preserving an FP32 master weight.

nemo_automodel.components.models.ernie5_5.moe._FP32LatentLinear.forward(
hidden: torch.Tensor
) -> torch.Tensor

Project […, in_features] to […, out_features].

Parameters:

hidden
torch.Tensor

Tensor of shape […, in_features], with arbitrary leading dimensions.

Returns: torch.Tensor

Tensor of shape […, out_features] in the input dtype; the FP32 weight is converted to that dtype

Raises:

  • RuntimeError: If the weight is not FP32.
nemo_automodel.components.models.ernie5_5.moe._make_callable_projection(
projection: torch.nn.Linear

Preserve the initialized parameter while adding a callable FSDP boundary.

Parameters:

projection
nn.Linear

Bias-free torch linear module.

Returns: _FP32LatentLinear

Projection module whose weight is the same parameter object as projection.weight.

Raises:

  • TypeError: If projection is not an nn.Linear or has a bias.
nemo_automodel.components.models.ernie5_5.moe._routed_situ(
x: torch.Tensor,
weights: torch.Tensor,
beta: float,
linear_beta: float | None
) -> torch.Tensor

Apply SiTU and the routing weight to the packed gate/up output of one routed expert.

Parameters:

x
torch.Tensor

Tensor of shape [routed_tokens, 2 * intermediate], with the gate followed by the up projection on the final axis. routed_tokens is the number of (token, top-k slot) pairs routed to the expert.

weights
torch.Tensor

Routing weights of shape [routed_tokens, 1].

beta
float

Gate-clamp parameter of SiTU.

linear_beta
float | None

Optional up-projection clamp parameter of SiTU.

Returns: torch.Tensor

Tensor of shape [routed_tokens, intermediate] in the dtype of x.

nemo_automodel.components.models.ernie5_5.moe.route(
hidden: torch.Tensor,
gate_a: torch.Tensor,
gate_b: torch.Tensor,
correction: torch.Tensor,
expert_scale: torch.Tensor,
top_k: int,
normalize: bool = True
) -> tuple[torch.Tensor, torch.Tensor]

Return selected routing weights and expert IDs.

Parameters:

hidden
torch.Tensor

Local token states of shape [tokens, hidden].

gate_a
torch.Tensor

First FP32 gate matrix of shape [experts, hidden].

gate_b
torch.Tensor

Second FP32 gate matrix of shape [experts, hidden].

correction
torch.Tensor

FP32 selection-only bias of shape [experts].

expert_scale
torch.Tensor

Learnable FP32 per-expert scale of shape [experts].

top_k
int

Number of distinct experts selected per token.

normalize
boolDefaults to True

Normalize selected unbiased scores when top_k exceeds one.

Returns: torch.Tensor

FP32 weights and integer expert IDs, each of shape [tokens, top_k].

Raises:

  • ValueError: If the tensor shapes do not agree, top_k is not between 1 and the number of experts, or a gate matrix, correction or expert_scale is not FP32.
nemo_automodel.components.models.ernie5_5.moe._EXPERT_BACKENDS = (('torch', 'torch'), ('torch_mm', 'hybridep'))