> This page is for version 0.4.0 · 26.04.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.models.ernie5_5.model

ERNIE 5.5 causal language model for AutoModel training.

Forward contract:

1. Embed the tokens and copy the embedding into `hc_mult` FP32 residual streams.
2. Run the decoder blocks. Each block has two hyper-connection sites: attention, which is heavily compressed
   attention (HCA) or dynamic sparse attention (DSA) as `compress_ratios` selects per layer, and feed-forward, which
   is a dense SiTU MLP in the first `first_k_dense_replace` layers and the latent mixture of experts elsewhere.
3. Contract the streams with the hyper-head, then apply the final RMSNorm, the output projection and the MultiMax
   correction of the logits.

A row is one unpadded document whose positions start at zero, or a packed row: several documents with their lengths
in `seq_lens`. Under context parallelism a rank holds one contiguous shard of a row, prepared by the sharder of
`prepare_model_inputs_for_cp`. A packed row and its shards are laid out so that every document starts at a multiple
of the HCA compression ratio, and no attention layer reads across a document boundary. Without `seq_lens`, padding,
a row whose masks or positions mark several documents, and a KV cache are rejected; several documents concatenated in
one row without such metadata are not detected. The model returns logits only; the recipe computes the loss.

The MTP block of the released checkpoint is not built: the model accepts `num_nextn_predict_layers == 0` only and
does not load the MTP weights. The DSA indexers only select keys, receive no gradient and stay frozen.

## Module Contents

### Classes

| Name                                                                                                  | Description                                                                                                |
| ----------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| [`Ernie5_5AttentionInputs`](#nemo_automodel-components-models-ernie5_5-model-Ernie5_5AttentionInputs) | Position and mask inputs of one forward call, shared by all decoder blocks.                                |
| [`Ernie5_5Block`](#nemo_automodel-components-models-ernie5_5-model-Ernie5_5Block)                     | One decoder layer: attention and feed-forward, each mixed into the residual streams by a hyper-connection. |
| [`Ernie5_5Model`](#nemo_automodel-components-models-ernie5_5-model-Ernie5_5Model)                     | ERNIE 5.5 decoder: embedding, hyper-connected blocks, hyper-head, final norm, output head and MultiMax.    |
| [`Ernie5_5_MoeForCausalLM`](#nemo_automodel-components-models-ernie5_5-model-Ernie5_5_MoeForCausalLM) | ERNIE 5.5 causal language model with expert parallelism.                                                   |

### Functions

| Name                                                                                                                              | Description                                                                                                |
| --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| [`_hca_attention_config`](#nemo_automodel-components-models-ernie5_5-model-_hca_attention_config)                                 | Describe the HCA layers in the configuration type of their DeepSeek V4 base class.                         |
| [`_require_full_row_spans`](#nemo_automodel-components-models-ernie5_5-model-_require_full_row_spans)                             | Reject packed rows whose document spans do not add up to the row length.                                   |
| [`_require_supervision_within_documents`](#nemo_automodel-components-models-ernie5_5-model-_require_supervision_within_documents) | Reject supervised positions in the padding of a packed row: the layout drops them after they were counted. |

### Data

[`ModelClass`](#nemo_automodel-components-models-ernie5_5-model-ModelClass)

[`_VHA_RANK`](#nemo_automodel-components-models-ernie5_5-model-_VHA_RANK)

### API

```python
class nemo_automodel.components.models.ernie5_5.model.Ernie5_5AttentionInputs(
    position_ids: torch.Tensor,
    document_ids: torch.Tensor,
    cu_seqlens: torch.Tensor | None,
    hca_rope: tuple[torch.Tensor, torch.Tensor],
    hca_rotary: nemo_automodel.components.models.deepseek_v4.layers.DeepseekV4RotaryEmbedding,
    hca_mask: torch.Tensor | None,
    dsa_metadata: nemo_automodel.components.models.glm_moe_dsa.kernels.cudnn_dsa.CudnnDsaPackedMetadata | None = None
)
```

Dataclass

Position and mask inputs of one forward call, shared by all decoder blocks.

`attention_inputs` builds them for rows that are one unpadded document starting at position zero, and
`packed_attention_inputs` for a packed row or one context-parallel shard of it.

**`cu_seqlens`** `Tensor | None`

Int32 tensor of shape \[documents + 1] with the cumulative document lengths of the row, read by
the cuDNN DSA path; None for a packed row on the eager path.

---

**`document_ids`** `Tensor`

Integer tensor of shape \[batch, sequence] with the document of every token, numbered from
zero along the row; -1 marks padding.

---

**`dsa_metadata`** `CudnnDsaPackedMetadata | None = None`

Layout of a packed row and of the local queries in it for the cuDNN DSA path; None otherwise.

---

**`hca_mask`** `Tensor | None`

Additive causal sliding-window mask of shape \[batch, 1, sequence, sequence] in the dtype of the
token embedding: zero for visible keys, the most negative finite value for hidden keys. None for a
packed row or a shard, whose HCA layers attend over key indices.

---

**`hca_rope`** `tuple[Tensor, Tensor]`

Cosine and sine of the HCA compress rotary embedding, each of shape
\[batch, sequence, qk_rope_head_dim], in the dtype of the token embedding.

---

**`hca_rotary`** `DeepseekV4RotaryEmbedding`

The rotary module that produced `hca_rope`; the HCA compressor calls it for the positions of
the compressed keys.

---

**`position_ids`** `Tensor`

Integer tensor of shape \[batch, sequence] with the position of every token in its document.

---

```python
class nemo_automodel.components.models.ernie5_5.model.Ernie5_5Block(
    config: nemo_automodel.components.models.ernie5_5.config.Ernie5_5Config,
    layer: nemo_automodel.components.models.ernie5_5.config.Ernie5_5LayerPlan,
    backend: nemo_automodel.components.models.common.BackendConfig,
    moe_config: nemo_automodel.components.moe.config.MoEConfig,
    hca_config: nemo_automodel.components.models.deepseek_v4.config.DeepseekV4Config
)
```

**Bases:** `Module`

One decoder layer: attention and feed-forward, each mixed into the residual streams by a hyper-connection.

The feed-forward child is registered as `moe` or `mlp`, the names the MoE parallelizer looks up, and
`forward` reads that attribute on every call, so a wrapper installed in its place takes effect.

**`attention`**

---

**`attn_hc`**

---

**`ffn_hc`**

---

**`input_layernorm`**

---

**`is_dsa`** `= layer.attention == 'dsa'`

---

**`is_moe`** `= layer.feed_forward == 'moe'`

---

**`mlp`**

---

**`moe`**

---

**`post_attention_layernorm`**

---

**`self_attn`** `Module`

The attention layer, under the name that the shared context-parallel setup looks up.

---

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5Block._attend(
    hidden_states: torch.Tensor,
    inputs: nemo_automodel.components.models.ernie5_5.model.Ernie5_5AttentionInputs
) -> torch.Tensor
```

Run the attention layer with the inputs of its kind.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Normalized tensor of shape \[batch, sequence, hidden] in the model dtype.

---

**`inputs`** `Ernie5_5AttentionInputs`

Position and mask inputs of the forward call.

---

**Returns:** `torch.Tensor`

Attention output of shape \[batch, sequence, hidden].

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5Block.forward(
    hidden_streams: torch.Tensor,
    inputs: nemo_automodel.components.models.ernie5_5.model.Ernie5_5AttentionInputs
) -> torch.Tensor
```

Apply the attention site and the feed-forward site to the residual streams.

**Parameters:**

**`hidden_streams`** `torch.Tensor`

FP32 tensor of shape \[batch, sequence, streams, hidden], where `streams` is
`hc_mult`.

---

**`inputs`** `Ernie5_5AttentionInputs`

Position and mask inputs of the forward call.

---

**Returns:** `torch.Tensor`

FP32 tensor of shape \[batch, sequence, streams, hidden].

**Raises:**

* `ValueError`: If `hidden_streams` does not have that shape or is not FP32.

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5Block.init_weights(
    buffer_device: torch.device,
    init_std: float
) -> None
```

Initialize every parameter of the layer after meta-device materialization.

**Parameters:**

**`buffer_device`** `torch.device`

Device of the materialized parameters, passed to the HCA and MoE initializers.

---

**`init_std`** `float`

Standard deviation of the weight draws.

---

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5Block.nemo_checkpoint_context_fn(
    context_fn: collections.abc.Callable[[], tuple[contextlib.AbstractContextManager, contextlib.AbstractContextManager]] | None
) -> collections.abc.Callable[[], tuple[contextlib.AbstractContextManager, contextlib.AbstractContextManager]] | None
```

Make the recomputation of a checkpointed DSA layer call reuse the key selection of its forward.

**Parameters:**

**`context_fn`** `Callable[[], tuple[AbstractContextManager, AbstractContextManager]] | None`

Factory of the forward and recomputation contexts of activation checkpointing, or None.

---

**Returns:** `Callable[[], tuple[AbstractContextManager, AbstractContextManager]] | None`

The factory with the selection record of one call bound for a DSA layer, otherwise `context_fn`.

```python
class nemo_automodel.components.models.ernie5_5.model.Ernie5_5Model(
    config: nemo_automodel.components.models.ernie5_5.config.Ernie5_5Config,
    backend: nemo_automodel.components.models.common.BackendConfig,
    moe_config: nemo_automodel.components.moe.config.MoEConfig
)
```

**Bases:** `Module`

ERNIE 5.5 decoder: embedding, hyper-connected blocks, hyper-head, final norm, output head and MultiMax.

**`dsa_uses_cudnn`**

---

**`embed_tokens`**

---

**`hyper_head`**

---

**`layers`**

---

**`lm_head`**

---

**`multimax`** `= Ernie5_5MultiMax(dtype=dtype)`

---

**`norm`**

---

**`sliding_window`** `= config.sliding_window`

---

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5Model._hca_rotary(
    device: torch.device
) -> nemo_automodel.components.models.deepseek_v4.layers.DeepseekV4RotaryEmbedding
```

Create the FP32 rotary module of the HCA layers on `device`.

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5Model.attention_inputs(
    input_ids: torch.Tensor
) -> nemo_automodel.components.models.ernie5_5.model.Ernie5_5AttentionInputs
```

Build the position and mask inputs of the attention layers for unpadded single-document rows.

The HCA rotary module is created here, in FP32 and on the input device, so that the model stores no
frequency buffer that a dtype cast could round. The mask uses the sliding window captured at
construction.

**Parameters:**

**`input_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence]; every row is one unpadded document.

---

**Returns:** `Ernie5_5AttentionInputs`

The inputs shared by all decoder blocks, on the device of `input_ids`.

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5Model.forward(
    input_ids: torch.Tensor,
    position_ids: torch.Tensor | None = None,
    document_ids: torch.Tensor | None = None,
    cp_group: torch.distributed.ProcessGroup | None = None
) -> torch.Tensor
```

Compute the logits of unpadded single-document rows, or of a packed row or shard.

**Parameters:**

**`input_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence]. Without `document_ids` every row is one
unpadded document that starts at position zero.

---

**`position_ids`** `torch.Tensor | None` — default: None

Position of every token in its document, of shape \[batch, sequence]; required with
`document_ids`.

---

**`document_ids`** `torch.Tensor | None` — default: None

Document of every token of a packed row or shard, of shape \[batch, sequence]; see
`packed_attention_inputs`.

---

**`cp_group`** `dist.ProcessGroup | None` — default: None

Context-parallel group whose ranks hold consecutive shards of the row, or None.

---

**Returns:** `torch.Tensor`

Logits of shape \[batch, sequence, vocab] in the model dtype.

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5Model.init_weights(
    buffer_device: torch.device,
    init_std: float
) -> None
```

Initialize every parameter of the decoder after meta-device materialization.

**Parameters:**

**`buffer_device`** `torch.device`

Device of the materialized parameters.

---

**`init_std`** `float`

Standard deviation of the weight draws.

---

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5Model.packed_attention_inputs(
    position_ids: torch.Tensor,
    document_ids: torch.Tensor,
    cp_group: torch.distributed.ProcessGroup | None
) -> nemo_automodel.components.models.ernie5_5.model.Ernie5_5AttentionInputs
```

Build the inputs of the attention layers for a packed row or one context-parallel shard of it.

**Parameters:**

**`position_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence] with the position of every token in its
document.

---

**`document_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence] with the document of every token, numbered
from zero along the row; -1 marks padding. Every document starts at a row offset that is a
multiple of the HCA compression ratio.

---

**`cp_group`** `dist.ProcessGroup | None`

Context-parallel group whose ranks hold consecutive shards of the row, or None.

---

**Returns:** `Ernie5_5AttentionInputs`

The inputs shared by all decoder blocks, on the device of `position_ids`.

```python
class nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM(
    config: nemo_automodel.components.models.ernie5_5.config.Ernie5_5Config,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None
)
```

**Bases:** [HFCheckpointingMixin](/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/models/common/hf_checkpointing_mixin#nemo_automodel-components-models-common-hf_checkpointing_mixin-HFCheckpointingMixin), `Module`, [MoEFSDPSyncMixin](/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/moe/fsdp_mixin#nemo_automodel-components-moe-fsdp_mixin-MoEFSDPSyncMixin)

ERNIE 5.5 causal language model with expert parallelism.

The model is built from a decoder-only configuration (`num_nextn_predict_layers == 0`); `self.config` is a
normalized copy of the configuration passed in, see `decoder_only_config`.

`backend.attn` selects the attention path. `"cudnn"` runs the DSA layers on the cuDNN sparse-attention
kernels and the HCA layers on eager attention, or, for a packed row or a context-parallel shard, on the same
kernels; it requires `nvidia-cudnn-frontend[cutedsl]` and FlashMLA, BF16 and the released latent and indexer
dimensions, and was validated on SM100-class GPUs only. `"eager"`, `"torch"` and `"sdpa"` run both kinds
of layer on eager PyTorch attention on any device; the eager DSA path is limited to 512 tokens. Without a
`backend` argument the model selects `"cudnn"`; pass `attn="eager"` where those kernels are not
available.

Packed rows and context parallelism: `forward` takes the document lengths of a packed row in `seq_lens`,
and `prepare_model_inputs_for_cp` returns the sharder that gives every context-parallel rank one contiguous
shard of a row. Under context parallelism the HCA layers exchange the window keys at the shard boundaries and
share the compressed keys, and the DSA layers gather the latent and indexer keys of the row.

The model reads one more `BackendConfig` field: `enable_hf_state_dict_adapter` attaches the checkpoint
adapter. Every other field must hold a value the layers implement. `linear` is `"torch"`. `experts` and
`dispatcher` are `"torch"` and `"torch"`, or `"torch_mm"` and `"hybridep"`. With the second pair the
routed experts of a job of several processes exchange their tokens through HybridEP, which is supported for
BF16 compute only (the parallelizer rejects another compute dtype). The remaining fields, the tuning fields
of HybridEP among them, keep the `BackendConfig` default. Other values are rejected at construction. This
includes `enable_fsdp_optimizations`: `MoEFSDPSyncMixin` would honour it, but it has
not been validated with this model.

The DSA indexers only select keys and are not trained. The constructor freezes their parameters, and a
forward in training mode raises if an indexer holds a trainable parameter: one that a freeze configuration
enabled again, or an adapter tensor of a LoRA adapter on an indexer projection.

LoRA is validated on the plain linear projections that an attention module owns directly (not those of its
indexer or compressor), on the routed experts, and on the projections of the dense FFN and of the shared
expert.

**`_keep_in_fp32_modules_strict`**

---

**`backend`**

---

**`config`** `= decoder_only_config(config)`

---

**`cp_mesh`** `DeviceMesh | None = None`

---

**`generation_config`** `= generation_config_from_model_config(self.config)`

---

**`model`**

---

**`packed_alignment`**

---

**`state_dict_adapter`** `= self._create_state_dict_adapter(self.config)`

---

**`tie_word_embeddings_support`** `TieSupport = TieSupport.UNTIED_ONLY`

---

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM._create_state_dict_adapter(
    config: nemo_automodel.components.models.ernie5_5.config.Ernie5_5Config
) -> nemo_automodel.components.checkpoint.state_dict_adapter.StateDictAdapter
```

staticmethod

Create the adapter between the native state dict and the released checkpoint layout.

This is the only place that names the adapter class.

**Parameters:**

**`config`** `Ernie5_5Config`

Decoder-only model configuration.

---

**Returns:** `StateDictAdapter`

The adapter that the checkpointer uses to load and save Hugging Face format weights.

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM._require_frozen_indexers() -> None
```

Reject training while a DSA indexer parameter requires gradients.

The loss gives the indexers no gradient, and an optimizer entry that never receives one has no state to
restore when a run resumes. The constructor therefore freezes them, and nothing in the model enables
them again. A freeze configuration whose unfreeze selectors cover an indexer does, and a LoRA adapter on
an indexer projection adds trainable tensors of its own to the indexer. Both are rejected here at the
first training step.

**Raises:**

* `ValueError`: If a parameter of a DSA indexer requires gradients.

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM._validate_backend(
    dtype: torch.dtype
) -> None
```

Reject backends and dtypes that the layers do not implement.

**Parameters:**

**`dtype`** `torch.dtype`

Model dtype taken from the configuration.

---

**Raises:**

* `ValueError`: If `backend.attn` is neither `"cudnn"` nor an eager name, another `BackendConfig`
  field holds a value that the model does not implement, the dtype is neither FP32 nor BF16, or the
  cuDNN path is selected without BF16 and the released latent and indexer dimensions.

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM._validate_packed_layout(
    position_ids: torch.Tensor,
    packed_seq_ids: torch.Tensor,
    first: int
) -> None
```

Reject a laid-out row or shard whose documents are not numbered, aligned and counted as the model reads them.

**Parameters:**

**`position_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence] with the position of every token in its document.

---

**`packed_seq_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence] with the document of every token; zero marks
padding.

---

**`first`** `int`

Row offset of the first token of this shard; zero for a whole row.

---

**Raises:**

* `ValueError`: If a document does not start at a multiple of the HCA compression ratio, its positions do
  not count up from zero at its first token, or a document id returns after another document.

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM._validate_single_document(
    input_ids: torch.Tensor,
    attention_mask: torch.Tensor | None,
    padding_mask: torch.Tensor | None,
    position_ids: torch.Tensor | None
) -> None
```

Reject every input that is not one unpadded document starting at position zero.

Several documents concatenated in one row without position or mask metadata cannot be detected here; the
data pipeline must supply one document per row.

**Parameters:**

**`input_ids`** `torch.Tensor`

Integer tensor of shape \[1, sequence].

---

**`attention_mask`** `torch.Tensor | None`

Optional tensor of shape \[1, sequence] with one for every token.

---

**`padding_mask`** `torch.Tensor | None`

Optional tensor of shape \[1, sequence] with zero (False) for every token.

---

**`position_ids`** `torch.Tensor | None`

Optional integer tensor of shape \[1, sequence] holding `0..sequence-1`.

---

**Raises:**

* `ValueError`: If `input_ids` is not a nonempty integer tensor of shape \[1, sequence] on the device of
  the model, contains the padding token, a mask marks padding or packed documents, or the positions
  do not count up from zero.

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM.forward(
    input_ids: torch.Tensor,
    attention_mask: torch.Tensor | None = None,
    padding_mask: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    packed_seq_ids: torch.Tensor | None = None,
    seq_lens: torch.Tensor | None = None,
    seq_lens_padded: torch.Tensor | None = None,
    use_cache: bool = False
) -> transformers.modeling_outputs.CausalLMOutputWithPast
```

Compute the logits of one unpadded document, of a packed row, or of this rank's shard of a row.

Without `seq_lens`, `packed_seq_ids` and context parallelism the row is one unpadded document: a row
whose masks or positions mark padding or several documents is rejected by the mask and position checks, and
several documents concatenated in one row without such metadata cannot be detected.

`seq_lens` marks a packed row. Its documents are moved to multiples of the HCA compression ratio for the
forward, and the logits are returned in the layout of `input_ids`. `packed_seq_ids` marks a row or a
shard that the sharder of `prepare_model_inputs_for_cp` has laid out already; the logits keep that
layout. A shard of unpacked rows is one document per row, with `attention_mask` marking its tokens.

**Parameters:**

**`input_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence]; \[1, sequence] for one unpadded document.

---

**`attention_mask`** `torch.Tensor | None` — default: None

Optional tensor of shape \[batch, sequence]. For one unpadded document it must be one
for every token; for a shard it is true for tokens and false for padding.

---

**`padding_mask`** `torch.Tensor | None` — default: None

Optional tensor of shape \[1, sequence]; must be zero (False) for every token.

---

**`position_ids`** `torch.Tensor | None` — default: None

Optional integer tensor of shape \[batch, sequence]. For one unpadded document it must
hold `0..sequence-1`; with `packed_seq_ids` or for a shard it is required and holds the
position of every token in its document.

---

**`packed_seq_ids`** `torch.Tensor | None` — default: None

Optional integer tensor of shape \[batch, sequence] with the document of every token,
counted from one along the row; zero marks padding.

---

**`seq_lens`** `torch.Tensor | None` — default: None

Optional integer tensor of shape \[batch, documents] with the document lengths of a packed
row; -1000 marks an unused entry.

---

**`seq_lens_padded`** `torch.Tensor | None` — default: None

Optional integer tensor of shape \[batch, documents] with the length of every document
including its padding in `input_ids`; these spans, or `seq_lens` without them, add up to the
row length.

---

**`use_cache`** `bool` — default: False

Must be False: the model keeps no KV cache.

---

**Returns:** `CausalLMOutputWithPast`

`CausalLMOutputWithPast` whose `logits` have shape \[batch, sequence, vocab] in the model dtype. No

**Raises:**

* `ValueError`: If a row without packing metadata is not one unpadded document starting at position zero,
  `seq_lens` comes with `packed_seq_ids` or under context parallelism, the document spans of
  `seq_lens` do not add up to the row length, a shard or a row with `packed_seq_ids` lacks its
  positions or its documents, its positions do not count up from zero in every document, its
  documents are not numbered in row order or do not start at multiples of the HCA compression ratio,
  `use_cache` is set, or the
  model is in training mode and a DSA indexer parameter requires gradients.

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM.from_config(
    config: nemo_automodel.components.models.ernie5_5.config.Ernie5_5Config,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None
) -> nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM
```

classmethod

Construct the model through AutoModel's configuration entry point.

**Parameters:**

**`config`** `Ernie5_5Config`

Model configuration with `num_nextn_predict_layers == 0`. It is not modified.

---

**`backend`** `BackendConfig | None` — default: None

Backend selection. None selects cuDNN DSA attention (the class docstring states what it
requires) with the torch expert, dispatcher and linear backends.

---

**Returns:** `Ernie5_5_MoeForCausalLM`

The model, with parameters allocated and not initialized from a checkpoint.

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM.get_input_embeddings() -> torch.nn.Embedding
```

Return the untied token embedding.

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM.get_output_embeddings() -> torch.nn.Linear
```

Return the untied output projection.

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM.initialize_weights(
    buffer_device: torch.device | None = None,
    dtype: torch.dtype | None = None
) -> None
```

Initialize every parameter after meta-device materialization.

Weights are drawn with standard deviation `config.initializer_range`. The attention sinks, the second
virtual-head factors, the HCA position embeddings of the compressor, the hyper-connection and hyper-head
offsets, the router correction bias and the MultiMax parameters start at zero; norms and mixer gains start
at one; the per-expert routing scales start at `routed_scaling_factor`.

**Parameters:**

**`buffer_device`** `torch.device | None` — default: None

Device of the materialized parameters; defaults to the device of the token embedding.

---

**`dtype`** `torch.dtype | None` — default: None

Model dtype; None selects the dtype of the configuration. Storage dtypes are fixed at
construction; an unsharded model is cast again, which keeps the strict FP32 names in FP32.

---

```python
nemo_automodel.components.models.ernie5_5.model.Ernie5_5_MoeForCausalLM.prepare_model_inputs_for_cp(
    batch: dict[str, torch.Tensor | str],
    num_chunks: int = 1
) -> dict[str, nemo_automodel.components.distributed.context_parallel.sharder.ContextParallelSharder]
```

Return the sharder that gives every context-parallel rank one contiguous shard of a row.

The sharder starts every document of a packed row at a multiple of the HCA compression ratio, pads the row
so that every shard holds whole compression windows, and adds `packed_seq_ids` and the document-local
`position_ids` that `forward` reads.

**Parameters:**

**`batch`** `dict[str, torch.Tensor | str]`

Full rows: `input_ids` and `labels` of shape \[batch, sequence], and `seq_lens` of shape
\[batch, documents] for packed rows, with optional `seq_lens_padded`. The hook checks the
spans of packed rows and leaves the batch unchanged.

---

**`num_chunks`** `int` — default: 1

Chunk count of the framework; this model uses one chunk.

---

**Returns:** `dict[str, ContextParallelSharder]`

The sharder, under the key `cp_sharder`.

**Raises:**

* `ValueError`: If the document spans of a packed row do not add up to its length, or a supervised
  label lies in the padding of a document.

```python
nemo_automodel.components.models.ernie5_5.model._hca_attention_config(
    config: nemo_automodel.components.models.ernie5_5.config.Ernie5_5Config,
    dtype: torch.dtype
) -> nemo_automodel.components.models.deepseek_v4.config.DeepseekV4Config
```

Describe the HCA layers in the configuration type of their DeepSeek V4 base class.

`Ernie5_5HCAAttention` extends `DeepseekV4Attention`, whose constructor reads a `DeepseekV4Config`. The
result is only passed to the HCA layers; the model keeps its own `Ernie5_5Config`.

**Parameters:**

**`config`** `Ernie5_5Config`

Decoder-only model configuration.

---

**`dtype`** `torch.dtype`

Dtype of the attention projections.

---

**Returns:** `DeepseekV4Config`

Configuration with the HCA dimensions of `config`, no hash layers, no MTP layers and no attention dropout.

```python
nemo_automodel.components.models.ernie5_5.model._require_full_row_spans(
    seq_lens: torch.Tensor,
    seq_lens_padded: torch.Tensor | None,
    row_length: int
) -> None
```

Reject packed rows whose document spans do not add up to the row length.

```python
nemo_automodel.components.models.ernie5_5.model._require_supervision_within_documents(
    seq_lens: torch.Tensor,
    seq_lens_padded: torch.Tensor | None,
    supervised: torch.Tensor
) -> None
```

Reject supervised positions in the padding of a packed row: the layout drops them after they were counted.

**Parameters:**

**`seq_lens`** `torch.Tensor`

Real document lengths of shape \[batch, documents]; negative entries are unused.

---

**`seq_lens_padded`** `torch.Tensor | None`

Optional physical spans of the same shape.

---

**`supervised`** `torch.Tensor`

Boolean tensor of shape \[batch, sequence], True where the loss counts a target.

---

```python
nemo_automodel.components.models.ernie5_5.model.ModelClass = Ernie5_5_MoeForCausalLM
```

```python
nemo_automodel.components.models.ernie5_5.model._VHA_RANK = 16
```