nemo_automodel.components.models.ernie5_5.layers
nemo_automodel.components.models.ernie5_5.layers
ERNIE 5.5 normalization, activation and residual-stream layers.
This module holds the leaf layers of the ERNIE 5.5 decoder:
Ernie5_5RMSNorm: RMSNorm that rounds its input to the weight dtype before computing FP32 statistics.ernie_situ: the SiTU activation on a packed gate/up tensor, with its checkpoint-compatible rounding points.virtual_head_mix: the low-rank virtual-head mixI + V U^Tapplied to per-head attention output.Ernie5_5HyperConnection: the multi-stream residual mixer (mHC) with FP32 stream arithmetic.Ernie5_5HyperHead: the learned contraction of the residual streams into one output stream.Ernie5_5MultiMax: the learned piecewise correction of vocabulary logits.Ernie5_5DenseMLP: the bias-free dense feed-forward block built onernie_situ.
It also defines DENSE_ATTENTION_BACKENDS, the backend.attn names that the two attention modules of the
package, dsa and hca, accept for their dense PyTorch path, and shard_start, the row offset of a
context-parallel shard.
Provenance: the definitions below follow the ERNIE 5.5 implementation in Megatron-Bridge (Apache License 2.0) at
revision a38833622b88969564fe8bfa053c5e4e4989c6cd.
Ernie5_5RMSNorm(arithmetic and input conversion): https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/norm.pyernie_situ(operation and rounding order): https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/moe.pyvirtual_head_mix(mathematical contract ofvha_postmix): https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/ops.pyErnie5_5HyperConnection(arithmetic): https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/hyper_connection.pyErnie5_5MultiMax(formula and 512-row accumulation order): https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/output_layer.py
Module Contents
Classes
Functions
Data
API
Bases: Module
Bias-free gate/up/down projections and ERNIE SiTU.
Compute the dense FFN on normalized activations.
Parameters:
Tensor of shape […, hidden], with arbitrary leading dimensions.
Returns: torch.Tensor
Tensor of shape […, hidden].
Initialize projection weights after meta materialization.
Parameters:
Standard deviation of the normal initialization of the three projection weights.
Raises:
ValueError: Ifinit_stdis not positive.
Bases: DeepseekV4HyperConnection
Use ERNIE normalization and FP32 stream mixing with eager Sinkhorn.
The parameters are those of DeepseekV4HyperConnection: fn of shape [streams * (streams + 2),
streams * hidden], base of shape [streams * (streams + 2)] and scale of shape [3], all FP32. The class
implements the mixing arithmetic only; checkpoint key mapping, the lifetime of the residual streams, the
mixed-precision policy, activation recomputation and distributed placement belong to the code that owns the
module.
Predict the mixing coefficients before Sinkhorn normalization, in FP32 with autocast disabled.
Parameters:
Tensor of shape [batch, sequence, streams, hidden].
Projection of shape [mix_components, streams * hidden], where mix_components is streams * (streams + 2).
Offsets of shape [mix_components], ordered pre, post, combination.
Gains of shape [3], ordered pre, post, combination.
Number of residual streams.
Epsilon added to the RMS of the flattened streams.
Epsilon added to the pre coefficients.
Returns: torch.Tensor
FP32 pre and post coefficients of shape [batch, sequence, streams] and FP32 combination logits of
Sum streams in FP32; the following layer owns any downcast.
Parameters:
Tensor of shape [batch, sequence, streams, hidden].
Coefficients of shape [batch, sequence, streams].
Returns: torch.Tensor
FP32 tensor of shape [batch, sequence, hidden]. Unlike the parent class, the result is not converted
Mix the residual and layer output in FP32, rounding once.
Parameters:
Tensor of shape [batch, sequence, hidden].
Tensor of shape [batch, sequence, streams, hidden].
Coefficients of shape [batch, sequence, streams].
Coefficients of shape [batch, sequence, input_streams, output_streams].
Returns: torch.Tensor
Tensor of shape [batch, sequence, streams, hidden] in the residual dtype.
Bases: Module
Collapse multiple residual streams using FP32 learned coefficients.
The parameters hc_fn of shape [streams, streams * hidden], hc_base of shape [streams] and hc_scale
of shape [1] are created in FP32, and forward raises if any of them has another dtype. Full-model and FSDP
precision policies remain the caller’s responsibility.
Contract [batch, sequence, streams, hidden] into [batch, sequence, hidden].
Parameters:
Tensor of shape [batch, sequence, streams, hidden].
Returns: torch.Tensor
Tensor of shape [batch, sequence, hidden] in the input dtype. The stream weights are computed and
Raises:
ValueError: Ifxdoes not have shape [batch, sequence, streams, hidden].RuntimeError: Ifhc_fn,hc_baseorhc_scaleis not FP32.
Initialize parameters after meta materialization.
Parameters:
Standard deviation of the normal initialization of hc_fn.
Raises:
ValueError: Ifinit_stdis not positive.
Bases: Module
Apply learned MultiMax correction to materialized vocabulary logits.
The thresholds multimax_ranges and the scales multimax_ts each have shape [4].
Correct logits while retaining their shape and dtype.
Parameters:
Floating tensor of shape […, vocab], with arbitrary
leading dimensions and nonempty vocab.
Returns: torch.Tensor
Corrected tensor of shape […, vocab], with the input dtype and
Raises:
ValueError: Iflogitshas no dimensions, an empty final dimension, or a dtype that is not floating point.
Bases: Module
Round inputs to the weight dtype, normalize in FP32 and round the result.
The scale parameter weight has shape [hidden] and the configured dtype.
Normalize arbitrary leading dimensions over the final hidden axis.
Parameters:
Floating tensor of shape […, hidden], with arbitrary leading
dimensions and hidden equal to the scale parameter width.
Returns: torch.Tensor
Tensor of shape […, hidden] on the input device, in the scale
Raises:
ValueError: Ifxhas no dimensions, its final dimension differs from the width of the scale parameter, or its dtype is not floating point.
Initialize the scale, including after meta-device materialization.
Bases: Function
Chunk temporary logits and accumulate parameter gradients in FP32.
Compute first-order gradients with FP32 parameter accumulation.
Parameters:
Autograd context with the forward inputs.
Incoming gradient of shape […, vocab], matching forward logits.
Returns: torch.Tensor
Logits gradient of shape […, vocab], threshold gradient [4] and scale
Correct logits using learned thresholds and scales.
Parameters:
Autograd context retaining inputs for first-order gradients.
Floating tensor of shape […, vocab], with arbitrary leading dimensions.
Floating threshold tensor of shape [4], on the logits device.
Floating scale tensor of shape [4], on the logits device.
Returns: torch.Tensor
Corrected tensor of shape […, vocab], with logits dtype and device.
Compute piecewise correction in FP32.
Parameters:
FP32 tensor of shape [rows, vocab].
FP32 threshold tensor of shape [4].
FP32 scale tensor of shape [4].
Returns: torch.Tensor
FP32 corrected logits of shape [rows, vocab].
Apply SiTU with rounding before optional routing-weight multiplication.
Parameters:
Floating tensor of shape […, 2 * intermediate], with arbitrary leading dimensions and the gate followed by the up projection on the final axis.
Positive finite gate-clamp parameter.
Optional positive finite up-projection clamp parameter.
Optional tensor broadcastable with […, intermediate], typically a routing weight of shape […, 1].
Returns: torch.Tensor
Activated tensor with the broadcast shape of […, intermediate] and
Raises:
ValueError: If the final dimension ofgate_upis odd,gate_upis not floating point, orbetaorlinear_betais not a positive finite number.
Return the row offset of the first token that this context-parallel rank holds.
Parameters:
Context-parallel group whose ranks hold consecutive shards of a row, or None.
Number of tokens in every shard.
Apply the low-rank head mix I + V U^T to attention output.
Only the activation-dtype path is implemented: u and v are converted to the dtype of attention and
the mix runs in that dtype with autocast disabled.
Parameters:
Tensor of shape […, heads, channels], with arbitrary leading dimensions. The final two axes are head and head channel.
Input-head factor of shape [heads, rank] on the same device.
Output-head factor of shape [heads, rank] on the same device.
Returns: torch.Tensor
Newly allocated tensor with the same shape and dtype as attention.
Raises:
ValueError: Ifattentionhas fewer than two dimensions, oruandvare not equal-shaped matrices with one row per attention head.