nemo_automodel.components.models.ernie5_5.moe
nemo_automodel.components.models.ernie5_5.moe
ERNIE 5.5 latent mixture-of-experts: router, gate and MoE block.
route: the dual-projection sigmoid router. It returns the selected routing weights and expert IDs for local token states. It is a pure function of its inputs: it does no load tracking, no correction-bias update and no communication between ranks.Ernie5_5Gate:routebehind AutoModel’s MoEGateinterface. It owns the two FP32 gate matrices, the learned per-expert scale and the selection-only correction bias, and records expert loads for the bias update.Ernie5_5SharedExpert: the shared expert with the SiTU activation.Ernie5_5MoE: the MoE block: FP32 latent projections around the routed experts, a latent RMSNorm and the shared expert.
Supported execution: the torch linear backend with one of two expert and dispatcher pairs.
experts="torch"withdispatcher="torch": the routed experts run inGroupedExperts, which gathers the tokens of the expert-parallel group, loops over the local experts and sums the selected expert outputs of a token in FP32.experts="torch_mm"withdispatcher="hybridep": with more than one process the routed experts run inGroupedExpertsDeepEP, which exchanges the tokens through HybridEP and runs two grouped GEMMs. The HybridEP combine also sums the selected expert outputs of a token in FP32, but it rounds the partial sum of every expert-parallel rank to the activation dtype before it adds the partial sums. The result differs from the first pair by BF16 rounding and depends onep_size. Only BF16 activations are supported there. In a single process the sharedMoEbuildsGroupedExpertson its grouped-GEMM path.
The two pairs store the expert weights in the same parameters with the same placements. In a single process there
is no placement requirement. With more than one process the expert weights must be DTensor objects on an
("ep",) mesh with placements (Shard(0),) or on an ("ep_shard", "ep") mesh with placements
(Shard(1), Shard(0)); Ernie5_5MoE.forward raises RuntimeError for any other layout. Routing uses the
local token states, so a context-parallel shard is routed like any other row.
Provenance: the expert-selection and routing-weight contracts of route are those of the ERNIE 5.5 implementation
in Megatron-Bridge (Apache License 2.0) at revision a38833622b88969564fe8bfa053c5e4e4989c6cd:
https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/moe.py
Module Contents
Classes
Functions
Data
API
Bases: Gate
ERNIE 5.5 gate using AutoModel’s MoE gate interface.
The gate supports one expert group, sigmoid scores with a selection-only correction bias and FP32 routing
weights. The auxiliary loss, a gate bias and routing replay are not supported; the constructor raises
ValueError for such a configuration. Routing uses the local token states of the calling process. Expert
loads for the correction-bias update are accumulated only in training mode with gradients enabled, so a
no-grad forward followed by its recomputation with gradients is counted once.
The parameters weight and weight_1 of shape [experts, hidden] and routed_scaling_factor_param of
shape [experts] are FP32, and forward raises if any of them has another dtype; preserving FP32 under FSDP
mixed precision is the responsibility of the model that owns the gate.
Route local token states and record valid-token loads.
Parameters:
Local token states of shape [tokens, hidden].
Boolean valid-token mask of shape [tokens].
Context mesh; auxiliary loss is unsupported and it is unused.
Returns: torch.Tensor
Tuple of FP32 routing weights [tokens, topk], integer expert IDs
Raises:
ValueError: Iftoken_maskdoes not have one entry per token, the correction bias is not on the device ofx, orrouterejects the shapes or dtypes.RuntimeError: If no correction bias is available.
Initialize both gates and learned expert scales after meta materialization.
Parameters:
Device passed to the parent initializer.
Standard deviation of the normal initialization of both gate matrices.
Bases: MoE
ERNIE dual-router latent experts and shared SiTU expert.
The routed experts are the shared GroupedExperts (torch dispatcher, or one process) or the shared
GroupedExpertsDeepEP (HybridEP dispatcher in a job of several processes), with SiTU installed as their
activation. Distributed execution is restricted to the expert-parallel layouts that forward checks, and
with the HybridEP dispatcher to BF16 activations.
Run the MoE.
Parameters:
Tensor of shape […, hidden] with arbitrary leading dimensions.
Boolean tensor of shape […] where true marks padding.
Context-parallel mesh; unused, because the gate computes no auxiliary loss.
Returns: torch.Tensor
Output of shape […, hidden].
Raises:
RuntimeError: If, with more than one process, the expert weights are notDTensorobjects with the same mesh and placements in a supported layout (mesh("ep",)with placements(Shard(0),), or mesh("ep_shard", "ep")with placements(Shard(1), Shard(0)), with more than one expert-parallel rank); if, with more than one process, the HybridEP dispatcher was requested but the module was built for one process; if a latent projection weight is not FP32; or if the routed experts dispatch through HybridEP and the latent activations are not BF16.ValueError: Ifhiddenis empty, has fewer than two dimensions or does not end in the configured hidden size, orpadding_maskdoes not have the shape ofhiddenwithout its final axis.
Initialize standard MoE modules plus both ERNIE router projections.
Parameters:
Device passed to the parent initializer.
Standard deviation of the normal initializations.
Bases: MLP
Preserve ERNIE shared-expert arithmetic at a callable FSDP boundary.
The module has the gate, up and down projections of the shared MLP and computes SiTU between them. Its
activation attribute is "situ": code that selects a kernel from that attribute, such as the fused LoRA
MLP path, finds no SwiGLU module here and leaves forward in place.
Run SiTU and the output projection inside the FSDP-wrapped module.
Parameters:
Tensor of shape […, hidden], with arbitrary leading dimensions.
Returns: torch.Tensor
Tensor of shape […, hidden].
Bases: Linear
Project in the activation dtype while preserving an FP32 master weight.
Project […, in_features] to […, out_features].
Parameters:
Tensor of shape […, in_features], with arbitrary leading dimensions.
Returns: torch.Tensor
Tensor of shape […, out_features] in the input dtype; the FP32 weight is converted to that dtype
Raises:
RuntimeError: If the weight is not FP32.
Preserve the initialized parameter while adding a callable FSDP boundary.
Parameters:
Bias-free torch linear module.
Returns: _FP32LatentLinear
Projection module whose weight is the same parameter object as projection.weight.
Raises:
TypeError: Ifprojectionis not annn.Linearor has a bias.
Apply SiTU and the routing weight to the packed gate/up output of one routed expert.
Parameters:
Tensor of shape [routed_tokens, 2 * intermediate], with the gate followed by the up projection on the
final axis. routed_tokens is the number of (token, top-k slot) pairs routed to the expert.
Routing weights of shape [routed_tokens, 1].
Gate-clamp parameter of SiTU.
Optional up-projection clamp parameter of SiTU.
Returns: torch.Tensor
Tensor of shape [routed_tokens, intermediate] in the dtype of x.
Return selected routing weights and expert IDs.
Parameters:
Local token states of shape [tokens, hidden].
First FP32 gate matrix of shape [experts, hidden].
Second FP32 gate matrix of shape [experts, hidden].
FP32 selection-only bias of shape [experts].
Learnable FP32 per-expert scale of shape [experts].
Number of distinct experts selected per token.
Normalize selected unbiased scores when top_k exceeds one.
Returns: torch.Tensor
FP32 weights and integer expert IDs, each of shape [tokens, top_k].
Raises:
ValueError: If the tensor shapes do not agree,top_kis not between 1 and the number of experts, or a gate matrix,correctionorexpert_scaleis not FP32.