Skip to navigation

nemo_automodel.components.models.ernie5_5.model

View as MarkdownOpen in Claude

ERNIE 5.5 causal language model for AutoModel training.

Forward contract:

  1. Embed the tokens and copy the embedding into hc_mult FP32 residual streams.
  2. Run the decoder blocks. Each block has two hyper-connection sites: attention, which is heavily compressed attention (HCA) or dynamic sparse attention (DSA) as compress_ratios selects per layer, and feed-forward, which is a dense SiTU MLP in the first first_k_dense_replace layers and the latent mixture of experts elsewhere.
  3. Contract the streams with the hyper-head, then apply the final RMSNorm, the output projection and the MultiMax correction of the logits.

A row is one unpadded document whose positions start at zero, or a packed row: several documents with their lengths in seq_lens. Under context parallelism a rank holds one contiguous shard of a row, prepared by the sharder of prepare_model_inputs_for_cp. A packed row and its shards are laid out so that every document starts at a multiple of the HCA compression ratio, and no attention layer reads across a document boundary. Without seq_lens, padding, a row whose masks or positions mark several documents, and a KV cache are rejected; several documents concatenated in one row without such metadata are not detected. The model returns logits only; the recipe computes the loss.

The MTP block of the released checkpoint is not built: the model accepts num_nextn_predict_layers == 0 only and does not load the MTP weights. The DSA indexers only select keys, receive no gradient and stay frozen.

Module Contents

Classes

NameDescription
Ernie5_5AttentionInputsPosition and mask inputs of one forward call, shared by all decoder blocks.
Ernie5_5BlockOne decoder layer: attention and feed-forward, each mixed into the residual streams by a hyper-connection.
Ernie5_5ModelERNIE 5.5 decoder: embedding, hyper-connected blocks, hyper-head, final norm, output head and MultiMax.
Ernie5_5_MoeForCausalLMERNIE 5.5 causal language model with expert parallelism.

Functions

NameDescription
_hca_attention_configDescribe the HCA layers in the configuration type of their DeepSeek V4 base class.
_require_full_row_spansReject packed rows whose document spans do not add up to the row length.
_require_supervision_within_documentsReject supervised positions in the padding of a packed row: the layout drops them after they were counted.

Data

ModelClass

_VHA_RANK

API

class nemo_automodel.components.models.ernie5_5.model.Ernie5_5AttentionInputs(
position_ids: torch.Tensor,
document_ids: torch.Tensor,
cu_seqlens: torch.Tensor | None,
hca_rope: tuple[torch.Tensor, torch.Tensor],
hca_mask: torch.Tensor | None,
)
Dataclass

Position and mask inputs of one forward call, shared by all decoder blocks.

attention_inputs builds them for rows that are one unpadded document starting at position zero, and packed_attention_inputs for a packed row or one context-parallel shard of it.

cu_seqlens
Tensor | None

Int32 tensor of shape [documents + 1] with the cumulative document lengths of the row, read by the cuDNN DSA path; None for a packed row on the eager path.

document_ids
Tensor

Integer tensor of shape [batch, sequence] with the document of every token, numbered from zero along the row; -1 marks padding.

dsa_metadata
CudnnDsaPackedMetadata | None = None

Layout of a packed row and of the local queries in it for the cuDNN DSA path; None otherwise.

hca_mask
Tensor | None

Additive causal sliding-window mask of shape [batch, 1, sequence, sequence] in the dtype of the token embedding: zero for visible keys, the most negative finite value for hidden keys. None for a packed row or a shard, whose HCA layers attend over key indices.

hca_rope
tuple[Tensor, Tensor]

Cosine and sine of the HCA compress rotary embedding, each of shape [batch, sequence, qk_rope_head_dim], in the dtype of the token embedding.

hca_rotary
DeepseekV4RotaryEmbedding

The rotary module that produced hca_rope; the HCA compressor calls it for the positions of the compressed keys.

position_ids
Tensor

Integer tensor of shape [batch, sequence] with the position of every token in its document.

Bases: Module

One decoder layer: attention and feed-forward, each mixed into the residual streams by a hyper-connection.

The feed-forward child is registered as moe or mlp, the names the MoE parallelizer looks up, and forward reads that attribute on every call, so a wrapper installed in its place takes effect.

attention
attn_hc
ffn_hc
input_layernorm
is_dsa
= layer.attention == 'dsa'
is_moe
= layer.feed_forward == 'moe'
mlp
moe
post_attention_layernorm
self_attn
Module

The attention layer, under the name that the shared context-parallel setup looks up.

nemo_automodel.components.models.ernie5_5.model.Ernie5_5Block._attend(
hidden_states: torch.Tensor,
) -> torch.Tensor

Run the attention layer with the inputs of its kind.

Parameters:

hidden_states
torch.Tensor

Normalized tensor of shape [batch, sequence, hidden] in the model dtype.

inputs
Ernie5_5AttentionInputs

Position and mask inputs of the forward call.

Returns: torch.Tensor

Attention output of shape [batch, sequence, hidden].

nemo_automodel.components.models.ernie5_5.model.Ernie5_5Block.forward(
hidden_streams: torch.Tensor,
) -> torch.Tensor

Apply the attention site and the feed-forward site to the residual streams.

Parameters:

hidden_streams
torch.Tensor

FP32 tensor of shape [batch, sequence, streams, hidden], where streams is hc_mult.

inputs
Ernie5_5AttentionInputs

Position and mask inputs of the forward call.

Returns: torch.Tensor

FP32 tensor of shape [batch, sequence, streams, hidden].

Raises:

  • ValueError: If hidden_streams does not have that shape or is not FP32.
nemo_automodel.components.models.ernie5_5.model.Ernie5_5Block.init_weights(
buffer_device: torch.device,
init_std: float
) -> None

Initialize every parameter of the layer after meta-device materialization.

Parameters:

buffer_device
torch.device

Device of the materialized parameters, passed to the HCA and MoE initializers.

init_std
float

Standard deviation of the weight draws.

nemo_automodel.components.models.ernie5_5.model.Ernie5_5Block.nemo_checkpoint_context_fn(
context_fn: collections.abc.Callable[[], tuple[contextlib.AbstractContextManager, contextlib.AbstractContextManager]] | None
) -> collections.abc.Callable[[], tuple[contextlib.AbstractContextManager, contextlib.AbstractContextManager]] | None

Make the recomputation of a checkpointed DSA layer call reuse the key selection of its forward.

Parameters:

context_fn
Callable[[], tuple[AbstractContextManager, AbstractContextManager]] | None

Factory of the forward and recomputation contexts of activation checkpointing, or None.

Returns: Callable[[], tuple[AbstractContextManager, AbstractContextManager]] | None

The factory with the selection record of one call bound for a DSA layer, otherwise context_fn.

Bases: Module

ERNIE 5.5 decoder: embedding, hyper-connected blocks, hyper-head, final norm, output head and MultiMax.

dsa_uses_cudnn
embed_tokens
hyper_head
layers
lm_head
multimax
= Ernie5_5MultiMax(dtype=dtype)
norm
sliding_window
= config.sliding_window
nemo_automodel.components.models.ernie5_5.model.Ernie5_5Model._hca_rotary(
device: torch.device

Create the FP32 rotary module of the HCA layers on device.

nemo_automodel.components.models.ernie5_5.model.Ernie5_5Model.attention_inputs(
input_ids: torch.Tensor

Build the position and mask inputs of the attention layers for unpadded single-document rows.

The HCA rotary module is created here, in FP32 and on the input device, so that the model stores no frequency buffer that a dtype cast could round. The mask uses the sliding window captured at construction.

Parameters:

input_ids
torch.Tensor

Integer tensor of shape [batch, sequence]; every row is one unpadded document.

Returns: Ernie5_5AttentionInputs

The inputs shared by all decoder blocks, on the device of input_ids.

nemo_automodel.components.models.ernie5_5.model.Ernie5_5Model.forward(
input_ids: torch.Tensor,
position_ids: torch.Tensor | None = None,
document_ids: torch.Tensor | None = None,
cp_group: torch.distributed.ProcessGroup | None = None
) -> torch.Tensor

Compute the logits of unpadded single-document rows, or of a packed row or shard.

Parameters:

input_ids
torch.Tensor

Integer tensor of shape [batch, sequence]. Without document_ids every row is one unpadded document that starts at position zero.

position_ids
torch.Tensor | NoneDefaults to None

Position of every token in its document, of shape [batch, sequence]; required with document_ids.

document_ids
torch.Tensor | NoneDefaults to None

Document of every token of a packed row or shard, of shape [batch, sequence]; see packed_attention_inputs.

cp_group
dist.ProcessGroup | NoneDefaults to None

Context-parallel group whose ranks hold consecutive shards of the row, or None.

Returns: torch.Tensor

Logits of shape [batch, sequence, vocab] in the model dtype.

nemo_automodel.components.models.ernie5_5.model.Ernie5_5Model.init_weights(
buffer_device: torch.device,
init_std: float
) -> None

Initialize every parameter of the decoder after meta-device materialization.

Parameters:

buffer_device
torch.device

Device of the materialized parameters.

init_std
float

Standard deviation of the weight draws.

nemo_automodel.components.models.ernie5_5.model.Ernie5_5Model.packed_attention_inputs(
position_ids: torch.Tensor,
document_ids: torch.Tensor,
cp_group: torch.distributed.ProcessGroup | None

Build the inputs of the attention layers for a packed row or one context-parallel shard of it.

Parameters:

position_ids
torch.Tensor

Integer tensor of shape [batch, sequence] with the position of every token in its document.

document_ids
torch.Tensor

Integer tensor of shape [batch, sequence] with the document of every token, numbered from zero along the row; -1 marks padding. Every document starts at a row offset that is a multiple of the HCA compression ratio.

cp_group
dist.ProcessGroup | None

Context-parallel group whose ranks hold consecutive shards of the row, or None.

Returns: Ernie5_5AttentionInputs

The inputs shared by all decoder blocks, on the device of position_ids.

class nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM(
)

Bases: HFCheckpointingMixin, Module, MoEFSDPSyncMixin

ERNIE 5.5 causal language model with expert parallelism.

The model is built from a decoder-only configuration (num_nextn_predict_layers == 0); self.config is a normalized copy of the configuration passed in, see decoder_only_config.

backend.attn selects the attention path. "cudnn" runs the DSA layers on the cuDNN sparse-attention kernels and the HCA layers on eager attention, or, for a packed row or a context-parallel shard, on the same kernels; it requires nvidia-cudnn-frontend[cutedsl] and FlashMLA, BF16 and the released latent and indexer dimensions, and was validated on SM100-class GPUs only. "eager", "torch" and "sdpa" run both kinds of layer on eager PyTorch attention on any device; the eager DSA path is limited to 512 tokens. Without a backend argument the model selects "cudnn"; pass attn="eager" where those kernels are not available.

Packed rows and context parallelism: forward takes the document lengths of a packed row in seq_lens, and prepare_model_inputs_for_cp returns the sharder that gives every context-parallel rank one contiguous shard of a row. Under context parallelism the HCA layers exchange the window keys at the shard boundaries and share the compressed keys, and the DSA layers gather the latent and indexer keys of the row.

The model reads one more BackendConfig field: enable_hf_state_dict_adapter attaches the checkpoint adapter. Every other field must hold a value the layers implement. linear is "torch". experts and dispatcher are "torch" and "torch", or "torch_mm" and "hybridep". With the second pair the routed experts of a job of several processes exchange their tokens through HybridEP, which is supported for BF16 compute only (the parallelizer rejects another compute dtype). The remaining fields, the tuning fields of HybridEP among them, keep the BackendConfig default. Other values are rejected at construction. This includes enable_fsdp_optimizations: MoEFSDPSyncMixin would honour it, but it has not been validated with this model.

The DSA indexers only select keys and are not trained. The constructor freezes their parameters, and a forward in training mode raises if an indexer holds a trainable parameter: one that a freeze configuration enabled again, or an adapter tensor of a LoRA adapter on an indexer projection.

LoRA is validated on the plain linear projections that an attention module owns directly (not those of its indexer or compressor), on the routed experts, and on the projections of the dense FFN and of the shared expert.

_keep_in_fp32_modules_strict
backend
config
= decoder_only_config(config)
cp_mesh
DeviceMesh | None = None
generation_config
= generation_config_from_model_config(self.config)
model
packed_alignment
state_dict_adapter
= self._create_state_dict_adapter(self.config)
tie_word_embeddings_support
TieSupport = TieSupport.UNTIED_ONLY
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM._create_state_dict_adapter(
staticmethod

Create the adapter between the native state dict and the released checkpoint layout.

This is the only place that names the adapter class.

Parameters:

config
Ernie5_5Config

Decoder-only model configuration.

Returns: StateDictAdapter

The adapter that the checkpointer uses to load and save Hugging Face format weights.

nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM._require_frozen_indexers() -> None

Reject training while a DSA indexer parameter requires gradients.

The loss gives the indexers no gradient, and an optimizer entry that never receives one has no state to restore when a run resumes. The constructor therefore freezes them, and nothing in the model enables them again. A freeze configuration whose unfreeze selectors cover an indexer does, and a LoRA adapter on an indexer projection adds trainable tensors of its own to the indexer. Both are rejected here at the first training step.

Raises:

  • ValueError: If a parameter of a DSA indexer requires gradients.
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM._validate_backend(
dtype: torch.dtype
) -> None

Reject backends and dtypes that the layers do not implement.

Parameters:

dtype
torch.dtype

Model dtype taken from the configuration.

Raises:

  • ValueError: If backend.attn is neither "cudnn" nor an eager name, another BackendConfig field holds a value that the model does not implement, the dtype is neither FP32 nor BF16, or the cuDNN path is selected without BF16 and the released latent and indexer dimensions.
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM._validate_packed_layout(
position_ids: torch.Tensor,
packed_seq_ids: torch.Tensor,
first: int
) -> None

Reject a laid-out row or shard whose documents are not numbered, aligned and counted as the model reads them.

Parameters:

position_ids
torch.Tensor

Integer tensor of shape [batch, sequence] with the position of every token in its document.

packed_seq_ids
torch.Tensor

Integer tensor of shape [batch, sequence] with the document of every token; zero marks padding.

first
int

Row offset of the first token of this shard; zero for a whole row.

Raises:

  • ValueError: If a document does not start at a multiple of the HCA compression ratio, its positions do not count up from zero at its first token, or a document id returns after another document.
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM._validate_single_document(
input_ids: torch.Tensor,
attention_mask: torch.Tensor | None,
padding_mask: torch.Tensor | None,
position_ids: torch.Tensor | None
) -> None

Reject every input that is not one unpadded document starting at position zero.

Several documents concatenated in one row without position or mask metadata cannot be detected here; the data pipeline must supply one document per row.

Parameters:

input_ids
torch.Tensor

Integer tensor of shape [1, sequence].

attention_mask
torch.Tensor | None

Optional tensor of shape [1, sequence] with one for every token.

padding_mask
torch.Tensor | None

Optional tensor of shape [1, sequence] with zero (False) for every token.

position_ids
torch.Tensor | None

Optional integer tensor of shape [1, sequence] holding 0..sequence-1.

Raises:

  • ValueError: If input_ids is not a nonempty integer tensor of shape [1, sequence] on the device of the model, contains the padding token, a mask marks padding or packed documents, or the positions do not count up from zero.
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM.forward(
input_ids: torch.Tensor,
attention_mask: torch.Tensor | None = None,
padding_mask: torch.Tensor | None = None,
position_ids: torch.Tensor | None = None,
packed_seq_ids: torch.Tensor | None = None,
seq_lens: torch.Tensor | None = None,
seq_lens_padded: torch.Tensor | None = None,
use_cache: bool = False
) -> transformers.modeling_outputs.CausalLMOutputWithPast

Compute the logits of one unpadded document, of a packed row, or of this rank’s shard of a row.

Without seq_lens, packed_seq_ids and context parallelism the row is one unpadded document: a row whose masks or positions mark padding or several documents is rejected by the mask and position checks, and several documents concatenated in one row without such metadata cannot be detected.

seq_lens marks a packed row. Its documents are moved to multiples of the HCA compression ratio for the forward, and the logits are returned in the layout of input_ids. packed_seq_ids marks a row or a shard that the sharder of prepare_model_inputs_for_cp has laid out already; the logits keep that layout. A shard of unpacked rows is one document per row, with attention_mask marking its tokens.

Parameters:

input_ids
torch.Tensor

Integer tensor of shape [batch, sequence]; [1, sequence] for one unpadded document.

attention_mask
torch.Tensor | NoneDefaults to None

Optional tensor of shape [batch, sequence]. For one unpadded document it must be one for every token; for a shard it is true for tokens and false for padding.

padding_mask
torch.Tensor | NoneDefaults to None

Optional tensor of shape [1, sequence]; must be zero (False) for every token.

position_ids
torch.Tensor | NoneDefaults to None

Optional integer tensor of shape [batch, sequence]. For one unpadded document it must hold 0..sequence-1; with packed_seq_ids or for a shard it is required and holds the position of every token in its document.

packed_seq_ids
torch.Tensor | NoneDefaults to None

Optional integer tensor of shape [batch, sequence] with the document of every token, counted from one along the row; zero marks padding.

seq_lens
torch.Tensor | NoneDefaults to None

Optional integer tensor of shape [batch, documents] with the document lengths of a packed row; -1000 marks an unused entry.

seq_lens_padded
torch.Tensor | NoneDefaults to None

Optional integer tensor of shape [batch, documents] with the length of every document including its padding in input_ids; these spans, or seq_lens without them, add up to the row length.

use_cache
boolDefaults to False

Must be False: the model keeps no KV cache.

Returns: CausalLMOutputWithPast

CausalLMOutputWithPast whose logits have shape [batch, sequence, vocab] in the model dtype. No

Raises:

  • ValueError: If a row without packing metadata is not one unpadded document starting at position zero, seq_lens comes with packed_seq_ids or under context parallelism, the document spans of seq_lens do not add up to the row length, a shard or a row with packed_seq_ids lacks its positions or its documents, its positions do not count up from zero in every document, its documents are not numbered in row order or do not start at multiples of the HCA compression ratio, use_cache is set, or the model is in training mode and a DSA indexer parameter requires gradients.
classmethod

Construct the model through AutoModel’s configuration entry point.

Parameters:

config
Ernie5_5Config

Model configuration with num_nextn_predict_layers == 0. It is not modified.

backend
BackendConfig | NoneDefaults to None

Backend selection. None selects cuDNN DSA attention (the class docstring states what it requires) with the torch expert, dispatcher and linear backends.

Returns: Ernie5_5_MoeForCausalLM

The model, with parameters allocated and not initialized from a checkpoint.

nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM.get_input_embeddings() -> torch.nn.Embedding

Return the untied token embedding.

nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM.get_output_embeddings() -> torch.nn.Linear

Return the untied output projection.

nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM.initialize_weights(
buffer_device: torch.device | None = None,
dtype: torch.dtype | None = None
) -> None

Initialize every parameter after meta-device materialization.

Weights are drawn with standard deviation config.initializer_range. The attention sinks, the second virtual-head factors, the HCA position embeddings of the compressor, the hyper-connection and hyper-head offsets, the router correction bias and the MultiMax parameters start at zero; norms and mixer gains start at one; the per-expert routing scales start at routed_scaling_factor.

Parameters:

buffer_device
torch.device | NoneDefaults to None

Device of the materialized parameters; defaults to the device of the token embedding.

dtype
torch.dtype | NoneDefaults to None

Model dtype; None selects the dtype of the configuration. Storage dtypes are fixed at construction; an unsharded model is cast again, which keeps the strict FP32 names in FP32.

nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM.prepare_model_inputs_for_cp(
batch: dict[str, torch.Tensor | str],
num_chunks: int = 1

Return the sharder that gives every context-parallel rank one contiguous shard of a row.

The sharder starts every document of a packed row at a multiple of the HCA compression ratio, pads the row so that every shard holds whole compression windows, and adds packed_seq_ids and the document-local position_ids that forward reads.

Parameters:

batch
dict[str, torch.Tensor | str]

Full rows: input_ids and labels of shape [batch, sequence], and seq_lens of shape [batch, documents] for packed rows, with optional seq_lens_padded. The hook checks the spans of packed rows and leaves the batch unchanged.

num_chunks
intDefaults to 1

Chunk count of the framework; this model uses one chunk.

Returns: dict[str, ContextParallelSharder]

The sharder, under the key cp_sharder.

Raises:

  • ValueError: If the document spans of a packed row do not add up to its length, or a supervised label lies in the padding of a document.
nemo_automodel.components.models.ernie5_5.model._hca_attention_config(
dtype: torch.dtype

Describe the HCA layers in the configuration type of their DeepSeek V4 base class.

Ernie5_5HCAAttention extends DeepseekV4Attention, whose constructor reads a DeepseekV4Config. The result is only passed to the HCA layers; the model keeps its own Ernie5_5Config.

Parameters:

config
Ernie5_5Config

Decoder-only model configuration.

dtype
torch.dtype

Dtype of the attention projections.

Returns: DeepseekV4Config

Configuration with the HCA dimensions of config, no hash layers, no MTP layers and no attention dropout.

nemo_automodel.components.models.ernie5_5.model._require_full_row_spans(
seq_lens: torch.Tensor,
seq_lens_padded: torch.Tensor | None,
row_length: int
) -> None

Reject packed rows whose document spans do not add up to the row length.

nemo_automodel.components.models.ernie5_5.model._require_supervision_within_documents(
seq_lens: torch.Tensor,
seq_lens_padded: torch.Tensor | None,
supervised: torch.Tensor
) -> None

Reject supervised positions in the padding of a packed row: the layout drops them after they were counted.

Parameters:

seq_lens
torch.Tensor

Real document lengths of shape [batch, documents]; negative entries are unused.

seq_lens_padded
torch.Tensor | None

Optional physical spans of the same shape.

supervised
torch.Tensor

Boolean tensor of shape [batch, sequence], True where the loss counts a target.

nemo_automodel.components.models.ernie5_5.model.ModelClass = Ernie5_5_MoeForCausalLM
nemo_automodel.components.models.ernie5_5.model._VHA_RANK = 16