Skip to navigation

nemo_automodel.components.models.ernie5_5.layers

View as MarkdownOpen in Claude

ERNIE 5.5 normalization, activation and residual-stream layers.

This module holds the leaf layers of the ERNIE 5.5 decoder:

  • Ernie5_5RMSNorm: RMSNorm that rounds its input to the weight dtype before computing FP32 statistics.
  • ernie_situ: the SiTU activation on a packed gate/up tensor, with its checkpoint-compatible rounding points.
  • virtual_head_mix: the low-rank virtual-head mix I + V U^T applied to per-head attention output.
  • Ernie5_5HyperConnection: the multi-stream residual mixer (mHC) with FP32 stream arithmetic.
  • Ernie5_5HyperHead: the learned contraction of the residual streams into one output stream.
  • Ernie5_5MultiMax: the learned piecewise correction of vocabulary logits.
  • Ernie5_5DenseMLP: the bias-free dense feed-forward block built on ernie_situ.

It also defines DENSE_ATTENTION_BACKENDS, the backend.attn names that the two attention modules of the package, dsa and hca, accept for their dense PyTorch path, and shard_start, the row offset of a context-parallel shard.

Provenance: the definitions below follow the ERNIE 5.5 implementation in Megatron-Bridge (Apache License 2.0) at revision a38833622b88969564fe8bfa053c5e4e4989c6cd.

Module Contents

Classes

NameDescription
Ernie5_5DenseMLPBias-free gate/up/down projections and ERNIE SiTU.
Ernie5_5HyperConnectionUse ERNIE normalization and FP32 stream mixing with eager Sinkhorn.
Ernie5_5HyperHeadCollapse multiple residual streams using FP32 learned coefficients.
Ernie5_5MultiMaxApply learned MultiMax correction to materialized vocabulary logits.
Ernie5_5RMSNormRound inputs to the weight dtype, normalize in FP32 and round the result.
_MultiMaxFunctionChunk temporary logits and accumulate parameter gradients in FP32.

Functions

NameDescription
_correctionCompute piecewise correction in FP32.
ernie_situApply SiTU with rounding before optional routing-weight multiplication.
shard_startReturn the row offset of the first token that this context-parallel rank holds.
virtual_head_mixApply the low-rank head mix I + V U^T to attention output.

Data

DENSE_ATTENTION_BACKENDS

_CHUNK_ROWS

API

class nemo_automodel.components.models.ernie5_5.layers.Ernie5_5DenseMLP(
hidden: int,
intermediate: int,
dtype: torch.dtype,
beta: float = 4.0,
linear_beta: float = 25.0
)

Bases: Module

Bias-free gate/up/down projections and ERNIE SiTU.

down_proj
gate_proj
up_proj
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5DenseMLP.forward(
x: torch.Tensor
) -> torch.Tensor

Compute the dense FFN on normalized activations.

Parameters:

x
torch.Tensor

Tensor of shape […, hidden], with arbitrary leading dimensions.

Returns: torch.Tensor

Tensor of shape […, hidden].

nemo_automodel.components.models.ernie5_5.layers.Ernie5_5DenseMLP.init_weights(
init_std: float = 0.02
) -> None

Initialize projection weights after meta materialization.

Parameters:

init_std
floatDefaults to 0.02

Standard deviation of the normal initialization of the three projection weights.

Raises:

  • ValueError: If init_std is not positive.
class nemo_automodel.components.models.ernie5_5.layers.Ernie5_5HyperConnection(
hc_mult: int,
hidden_size: int,
hc_sinkhorn_iters: int,
hc_eps: float,
rms_norm_eps: float
)

Bases: DeepseekV4HyperConnection

Use ERNIE normalization and FP32 stream mixing with eager Sinkhorn.

The parameters are those of DeepseekV4HyperConnection: fn of shape [streams * (streams + 2), streams * hidden], base of shape [streams * (streams + 2)] and scale of shape [3], all FP32. The class implements the mixing arithmetic only; checkpoint key mapping, the lifetime of the residual streams, the mixed-precision policy, activation recomputation and distributed placement belong to the code that owns the module.

nemo_automodel.components.models.ernie5_5.layers.Ernie5_5HyperConnection._project(
hidden_streams: torch.Tensor,
fn: torch.Tensor,
base: torch.Tensor,
scale: torch.Tensor,
hc: int,
norm_eps: float,
hc_eps: float
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]
staticmethod

Predict the mixing coefficients before Sinkhorn normalization, in FP32 with autocast disabled.

Parameters:

hidden_streams
torch.Tensor

Tensor of shape [batch, sequence, streams, hidden].

fn
torch.Tensor

Projection of shape [mix_components, streams * hidden], where mix_components is streams * (streams + 2).

base
torch.Tensor

Offsets of shape [mix_components], ordered pre, post, combination.

scale
torch.Tensor

Gains of shape [3], ordered pre, post, combination.

hc
int

Number of residual streams.

norm_eps
float

Epsilon added to the RMS of the flattened streams.

hc_eps
float

Epsilon added to the pre coefficients.

Returns: torch.Tensor

FP32 pre and post coefficients of shape [batch, sequence, streams] and FP32 combination logits of

nemo_automodel.components.models.ernie5_5.layers.Ernie5_5HyperConnection.collapse(
hidden_streams: torch.Tensor,
pre: torch.Tensor
) -> torch.Tensor
staticmethod

Sum streams in FP32; the following layer owns any downcast.

Parameters:

hidden_streams
torch.Tensor

Tensor of shape [batch, sequence, streams, hidden].

pre
torch.Tensor

Coefficients of shape [batch, sequence, streams].

Returns: torch.Tensor

FP32 tensor of shape [batch, sequence, hidden]. Unlike the parent class, the result is not converted

nemo_automodel.components.models.ernie5_5.layers.Ernie5_5HyperConnection.expand(
output: torch.Tensor,
residual: torch.Tensor,
post: torch.Tensor,
comb: torch.Tensor
) -> torch.Tensor
staticmethod

Mix the residual and layer output in FP32, rounding once.

Parameters:

output
torch.Tensor

Tensor of shape [batch, sequence, hidden].

residual
torch.Tensor

Tensor of shape [batch, sequence, streams, hidden].

post
torch.Tensor

Coefficients of shape [batch, sequence, streams].

comb
torch.Tensor

Coefficients of shape [batch, sequence, input_streams, output_streams].

Returns: torch.Tensor

Tensor of shape [batch, sequence, streams, hidden] in the residual dtype.

class nemo_automodel.components.models.ernie5_5.layers.Ernie5_5HyperHead(
streams: int,
hidden: int,
eps: float = 1e-06
)

Bases: Module

Collapse multiple residual streams using FP32 learned coefficients.

The parameters hc_fn of shape [streams, streams * hidden], hc_base of shape [streams] and hc_scale of shape [1] are created in FP32, and forward raises if any of them has another dtype. Full-model and FSDP precision policies remain the caller’s responsibility.

hc_base
hc_fn
hc_scale
= nn.Parameter(torch.ones(1, dtype=(torch.float32)))
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5HyperHead.forward(
x: torch.Tensor
) -> torch.Tensor

Contract [batch, sequence, streams, hidden] into [batch, sequence, hidden].

Parameters:

x
torch.Tensor

Tensor of shape [batch, sequence, streams, hidden].

Returns: torch.Tensor

Tensor of shape [batch, sequence, hidden] in the input dtype. The stream weights are computed and

Raises:

  • ValueError: If x does not have shape [batch, sequence, streams, hidden].
  • RuntimeError: If hc_fn, hc_base or hc_scale is not FP32.
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5HyperHead.init_weights(
init_std: float = 0.02
) -> None

Initialize parameters after meta materialization.

Parameters:

init_std
floatDefaults to 0.02

Standard deviation of the normal initialization of hc_fn.

Raises:

  • ValueError: If init_std is not positive.
class nemo_automodel.components.models.ernie5_5.layers.Ernie5_5MultiMax(
dtype: torch.dtype = torch.float32
)

Bases: Module

Apply learned MultiMax correction to materialized vocabulary logits.

The thresholds multimax_ranges and the scales multimax_ts each have shape [4].

multimax_ranges
= nn.Parameter(torch.zeros(4, dtype=dtype))
multimax_ts
= nn.Parameter(torch.zeros(4, dtype=dtype))
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5MultiMax.forward(
logits: torch.Tensor
) -> torch.Tensor

Correct logits while retaining their shape and dtype.

Parameters:

logits
torch.Tensor

Floating tensor of shape […, vocab], with arbitrary leading dimensions and nonempty vocab.

Returns: torch.Tensor

Corrected tensor of shape […, vocab], with the input dtype and

Raises:

  • ValueError: If logits has no dimensions, an empty final dimension, or a dtype that is not floating point.
class nemo_automodel.components.models.ernie5_5.layers.Ernie5_5RMSNorm(
hidden_size: int,
eps: float = 1e-06,
dtype: torch.dtype = torch.float32
)

Bases: Module

Round inputs to the weight dtype, normalize in FP32 and round the result.

The scale parameter weight has shape [hidden] and the configured dtype.

weight
= nn.Parameter(torch.ones(hidden_size, dtype=dtype))
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5RMSNorm.forward(
x: torch.Tensor
) -> torch.Tensor

Normalize arbitrary leading dimensions over the final hidden axis.

Parameters:

x
torch.Tensor

Floating tensor of shape […, hidden], with arbitrary leading dimensions and hidden equal to the scale parameter width.

Returns: torch.Tensor

Tensor of shape […, hidden] on the input device, in the scale

Raises:

  • ValueError: If x has no dimensions, its final dimension differs from the width of the scale parameter, or its dtype is not floating point.
nemo_automodel.components.models.ernie5_5.layers.Ernie5_5RMSNorm.reset_parameters() -> None

Initialize the scale, including after meta-device materialization.

class nemo_automodel.components.models.ernie5_5.layers._MultiMaxFunction()

Bases: Function

Chunk temporary logits and accumulate parameter gradients in FP32.

nemo_automodel.components.models.ernie5_5.layers._MultiMaxFunction.backward(
ctx: torch.autograd.function.FunctionCtx,
grad_output: torch.Tensor
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]
staticmethod

Compute first-order gradients with FP32 parameter accumulation.

Parameters:

ctx
torch.autograd.function.FunctionCtx

Autograd context with the forward inputs.

grad_output
torch.Tensor

Incoming gradient of shape […, vocab], matching forward logits.

Returns: torch.Tensor

Logits gradient of shape […, vocab], threshold gradient [4] and scale

nemo_automodel.components.models.ernie5_5.layers._MultiMaxFunction.forward(
ctx: torch.autograd.function.FunctionCtx,
logits: torch.Tensor,
ranges: torch.Tensor,
scales: torch.Tensor
) -> torch.Tensor
staticmethod

Correct logits using learned thresholds and scales.

Parameters:

ctx
torch.autograd.function.FunctionCtx

Autograd context retaining inputs for first-order gradients.

logits
torch.Tensor

Floating tensor of shape […, vocab], with arbitrary leading dimensions.

ranges
torch.Tensor

Floating threshold tensor of shape [4], on the logits device.

scales
torch.Tensor

Floating scale tensor of shape [4], on the logits device.

Returns: torch.Tensor

Corrected tensor of shape […, vocab], with logits dtype and device.

nemo_automodel.components.models.ernie5_5.layers._correction(
logits: torch.Tensor,
ranges: torch.Tensor,
scales: torch.Tensor
) -> torch.Tensor

Compute piecewise correction in FP32.

Parameters:

logits
torch.Tensor

FP32 tensor of shape [rows, vocab].

ranges
torch.Tensor

FP32 threshold tensor of shape [4].

scales
torch.Tensor

FP32 scale tensor of shape [4].

Returns: torch.Tensor

FP32 corrected logits of shape [rows, vocab].

nemo_automodel.components.models.ernie5_5.layers.ernie_situ(
gate_up: torch.Tensor,
beta: float,
linear_beta: float | None,
routing_weight: torch.Tensor | None = None
) -> torch.Tensor

Apply SiTU with rounding before optional routing-weight multiplication.

Parameters:

gate_up
torch.Tensor

Floating tensor of shape […, 2 * intermediate], with arbitrary leading dimensions and the gate followed by the up projection on the final axis.

beta
float

Positive finite gate-clamp parameter.

linear_beta
float | None

Optional positive finite up-projection clamp parameter.

routing_weight
torch.Tensor | NoneDefaults to None

Optional tensor broadcastable with […, intermediate], typically a routing weight of shape […, 1].

Returns: torch.Tensor

Activated tensor with the broadcast shape of […, intermediate] and

Raises:

  • ValueError: If the final dimension of gate_up is odd, gate_up is not floating point, or beta or linear_beta is not a positive finite number.
nemo_automodel.components.models.ernie5_5.layers.shard_start(
cp_group: torch.distributed.ProcessGroup | None,
sequence: int
) -> int

Return the row offset of the first token that this context-parallel rank holds.

Parameters:

cp_group
dist.ProcessGroup | None

Context-parallel group whose ranks hold consecutive shards of a row, or None.

sequence
int

Number of tokens in every shard.

nemo_automodel.components.models.ernie5_5.layers.virtual_head_mix(
attention: torch.Tensor,
u: torch.Tensor,
v: torch.Tensor
) -> torch.Tensor

Apply the low-rank head mix I + V U^T to attention output.

Only the activation-dtype path is implemented: u and v are converted to the dtype of attention and the mix runs in that dtype with autocast disabled.

Parameters:

attention
torch.Tensor

Tensor of shape […, heads, channels], with arbitrary leading dimensions. The final two axes are head and head channel.

u
torch.Tensor

Input-head factor of shape [heads, rank] on the same device.

v
torch.Tensor

Output-head factor of shape [heads, rank] on the same device.

Returns: torch.Tensor

Newly allocated tensor with the same shape and dtype as attention.

Raises:

  • ValueError: If attention has fewer than two dimensions, or u and v are not equal-shaped matrices with one row per attention head.
nemo_automodel.components.models.ernie5_5.layers.DENSE_ATTENTION_BACKENDS: tuple[str, ...] = ('eager', 'torch', 'sdpa')
nemo_automodel.components.models.ernie5_5.layers._CHUNK_ROWS = 512