> This page is for version 0.7.0 · 26.10.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.models.ernie5_5.hca

ERNIE 5.5 heavily compressed attention (HCA).

HCA is DeepSeek V4's ratio-128 attention with two additions: a low-rank
virtual-head mix of the per-head attention output, and a sigmoid gate on the
grouped output projection that is driven by the query latent. DeepSeek V4's
projections, norms, compressor, rotary helper and eager attention are reused
unchanged. Only the order in which they are called is restated here, because
both additions sit between two steps of `DeepseekV4Attention.forward`.

Provenance: the operation order follows the ERNIE 5.5 HCA of Megatron-Bridge (Apache License 2.0) at revision
`a38833622b88969564fe8bfa053c5e4e4989c6cd`:
[https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/hca_attention.py](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/hca_attention.py)

Two kinds of input are supported, both with the torch rotary backend. A row
that holds one document starting at position zero runs the eager attention under
an explicit causal sliding-window mask. A packed row, or one rank's shard of it
under context parallelism, runs sparse attention over the window keys and the
complete compressed keys of each query's document; neighbouring ranks exchange
the window keys at the shard boundary and all ranks share the compressed keys. A
KV cache raises an error.

## Module Contents

### Classes

| Name                                                                                                | Description                                                                       |
| --------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| [`Ernie5_5HCAAttention`](#nemo_automodel-components-models-ernie5_5-hca-Ernie5_5HCAAttention)       | DeepSeek V4 ratio-128 attention with ERNIE virtual-head mixing and output gating. |
| [`Ernie5_5HCAVirtualHeads`](#nemo_automodel-components-models-ernie5_5-hca-Ernie5_5HCAVirtualHeads) | Keep the FP32 virtual-head parameters in a callable FSDP unit.                    |

### Functions

| Name                                                                                          | Description                                                                        |
| --------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| [`_compressed_key_mask`](#nemo_automodel-components-models-ernie5_5-hca-_compressed_key_mask) | Build the additive mask over compressed keys for rows that start at position zero. |

### Data

[`_HCA_COMPRESS_RATIO`](#nemo_automodel-components-models-ernie5_5-hca-_HCA_COMPRESS_RATIO)

### API

```python
class nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAAttention(
    config: nemo_automodel.components.models.deepseek_v4.config.DeepseekV4Config,
    layer_idx: int,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None,
    vha_rank: int = 16
)
```

**Bases:** [DeepseekV4Attention](/nemo/automodel/v0.7/nemo-automodel/nemo_automodel/components/models/deepseek_v4/layers#nemo_automodel-components-models-deepseek_v4-layers-DeepseekV4Attention)

DeepSeek V4 ratio-128 attention with ERNIE virtual-head mixing and output gating.

Construction, parameter names and the initialization of the DeepSeek V4
parameters are inherited. `forward` restates the eager, single-document
path of `DeepseekV4Attention.forward` as of NVIDIA-NeMo/Automodel commit
6f44dbf2390a4eb31bcdc67898bde98ce408f915
(`nemo_automodel/components/models/deepseek_v4/layers.py`), with the
virtual-head mix inserted after the inverse rotation and the gate after the
grouped projection. With a zero `vha_postmix_V` and a zero gate weight the
output is exactly half of the parent's output; the unit tests compare the two
so that a change to the parent's forward which this class does not follow
fails a test.

The config must describe a compression-ratio-128 layer. It is not sufficient
by itself to construct a full ERNIE 5.5 model.

**`cp_group`** `ProcessGroup | None = None`

---

**`gate_proj`**

---

**`vha`**

---

**`vha_postmix_U`** `Tensor`

FP32 virtual-head input factor of shape \[heads, vha_rank], owned by `vha`.

---

**`vha_postmix_V`** `Tensor`

FP32 virtual-head output factor of shape \[heads, vha_rank], owned by `vha`.

---

```python
nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAAttention._attend_packed(
    hidden_states: torch.Tensor,
    q_residual: torch.Tensor,
    query: torch.Tensor,
    keys: torch.Tensor,
    rope: tuple[torch.Tensor, torch.Tensor],
    rotary: torch.nn.Module,
    position_ids: torch.Tensor,
    packed_seq_ids: torch.Tensor
) -> torch.Tensor
```

Attend every query to the window keys and the complete compressed keys of its own document.

Every document starts at a row offset that is a multiple of the compression ratio, and so does every
context-parallel shard, so each compressed key pools the tokens of one document. A rank receives the last
`sliding_window - 1` keys of the preceding rank and the compressed keys of all ranks.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden].

---

**`q_residual`** `torch.Tensor`

Query latent of shape \[batch, sequence, q_lora_rank].

---

**`query`** `torch.Tensor`

Rotated query of shape \[batch, heads, sequence, head_dim].

---

**`keys`** `torch.Tensor`

Rotated keys of shape \[batch, sequence, head_dim].

---

**`rope`** `tuple[torch.Tensor, torch.Tensor]`

Compress-rope cosine and sine, each of shape \[batch, sequence, rotary_channels].

---

**`rotary`** `nn.Module`

Rotary module for the compressed keys.

---

**`position_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence] with the position of every token in its
document.

---

**`packed_seq_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence] with the document of every token, counted
from one along the row; zero marks padding.

---

**Returns:** `torch.Tensor`

Attention output of shape \[batch, sequence, heads, head_dim].

```python
nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAAttention._project_output(
    attended: torch.Tensor,
    q_residual: torch.Tensor,
    cos: torch.Tensor,
    sin: torch.Tensor
) -> torch.Tensor
```

Undo the key rotation, mix the heads and apply the gated grouped output projection.

**Parameters:**

**`attended`** `torch.Tensor`

Attention output of shape \[batch, sequence, heads, head_dim].

---

**`q_residual`** `torch.Tensor`

Query latent of shape \[batch, sequence, q_lora_rank].

---

**`cos`** `torch.Tensor`

Compress-rope cosine of shape \[batch, sequence, rotary_channels].

---

**`sin`** `torch.Tensor`

Compress-rope sine of shape \[batch, sequence, rotary_channels].

---

**Returns:** `torch.Tensor`

Output of shape \[batch, sequence, hidden].

```python
nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAAttention._validate_inputs(
    hidden_states: torch.Tensor,
    attention_mask: torch.Tensor | None,
    start_pos: int,
    position_ids: torch.Tensor | None,
    extra: dict[str, typing.Any]
) -> torch.Tensor
```

Reject every input outside the single-document eager path.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden].

---

**`attention_mask`** `torch.Tensor | None`

Additive mask of shape \[batch, 1 or heads, sequence, sequence].

---

**`start_pos`** `int`

First query position; must be zero.

---

**`position_ids`** `torch.Tensor | None`

Must be None; positions are the row offsets.

---

**`extra`** `dict[str, Any]`

Remaining keyword arguments of `forward`.

---

**Returns:** `torch.Tensor`

The validated `attention_mask`, unchanged.

**Raises:**

* `ValueError`: If the mask is missing or misshaped.
* `NotImplementedError`: For a nonzero start position, explicit position ids
  or a context-parallel group.
* `TypeError`: For any other keyword argument.

```python
nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAAttention.forward(
    hidden_states: torch.Tensor,
    position_embeddings: tuple[torch.Tensor, torch.Tensor],
    attention_mask: torch.Tensor | None = None,
    position_embeddings_compress: tuple[torch.Tensor, torch.Tensor] | None = None,
    rotary_compress: torch.nn.Module | None = None,
    start_pos: int = 0,
    position_ids: torch.Tensor | None = None,
    kwargs: typing.Any = {}
) -> tuple[torch.Tensor, torch.Tensor | None]
```

Run windowed attention over local and compressed keys, then mix heads and gate.

Without `packed_seq_ids` every row is one document starting at position
zero, and the caller is responsible for the mask contents: a causal mask
limited to the sliding window. Only its shape and dtype kind are checked.
With `packed_seq_ids` the row is packed and takes no mask; see
`_attend_packed`.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden]. With context
parallelism, `sequence` is this rank's shard of a packed row.

---

**`position_embeddings`** `tuple[torch.Tensor, torch.Tensor]`

Main-rope cosine and sine, each of shape
\[batch, sequence, rotary_channels]. Unused: a ratio-128 layer rotates
with the compress tables. Kept for the parent's signature.

---

**`attention_mask`** `torch.Tensor | None` — default: None

Additive mask of shape
\[batch, 1 or heads, sequence, sequence] over the uncompressed keys;
required for an unpacked row and None for a packed one.

---

**`position_embeddings_compress`** `tuple[torch.Tensor, torch.Tensor] | None` — default: None

Required compress-rope cosine and sine, each
of shape \[batch, sequence, rotary_channels], for the position of every
token in its document.

---

**`rotary_compress`** `nn.Module | None` — default: None

Required rotary module for the compressed keys.

---

**`start_pos`** `int` — default: 0

First query position; must be zero.

---

**`position_ids`** `torch.Tensor | None` — default: None

Integer tensor of shape \[batch, sequence] with the position
of every token in its document; required for a packed row and None
for an unpacked one.

---

**`**kwargs`** `Any` — default: \{}

`packed_seq_ids`, an integer tensor of shape \[batch, sequence]
with the document of every token, selects the packed path.
`_dsv4_cp_group` is accepted only as None; any other name raises.

---

**Returns:** `torch.Tensor`

Output of shape \[batch, sequence, hidden], and the attention

**Raises:**

* `ValueError`: If `position_embeddings_compress` or `rotary_compress` is None, or a packed row comes
  with a mask, without `position_ids`, with a nonzero start position or with another keyword
  argument. Every other rejected input of an unpacked row raises in `_validate_inputs`:
  `ValueError` for a missing or misshaped mask, `NotImplementedError` for a nonzero start
  position, explicit position ids or a context-parallel group, and `TypeError` for any other
  keyword argument.

```python
nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAAttention.init_weights(
    buffer_device: torch.device,
    init_std: float = 0.02
) -> None
```

Initialize inherited and ERNIE parameters after meta materialization.

`vha_postmix_V` is set to zero, so the virtual-head mix starts as the identity.

**Parameters:**

**`buffer_device`** `torch.device`

Forwarded to `DeepseekV4Attention.init_weights`.

---

**`init_std`** `float` — default: 0.02

Standard deviation of the weight draws: truncated normal for the inherited projections and
the gate, normal for `vha_postmix_U`.

---

```python
nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAAttention.setup_cp_attention(
    cp_mesh: torch.distributed.device_mesh.DeviceMesh
) -> None
```

Record the context-parallel group whose ranks hold consecutive shards of a row.

**Parameters:**

**`cp_mesh`** `DeviceMesh`

Context-parallel device mesh offered by the parallelizer.

---

```python
class nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAVirtualHeads(
    num_heads: int,
    rank: int
)
```

**Bases:** `Module`

Keep the FP32 virtual-head parameters in a callable FSDP unit.

Parameters, both FP32: `vha_postmix_U` of shape \[heads, rank], allocated without initialization and filled
by the owning layer, and `vha_postmix_V` of shape \[heads, rank], initialized to zeros.

**`vha_postmix_U`**

---

**`vha_postmix_V`**

---

```python
nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAVirtualHeads.forward(
    attention: torch.Tensor
) -> torch.Tensor
```

Mix the attention output across heads with `I + V U^T`.

**Parameters:**

**`attention`** `torch.Tensor`

Tensor of shape \[batch, sequence, heads, head_dim].

---

**Returns:** `torch.Tensor`

Newly allocated tensor with the shape and dtype of `attention`.

```python
nemo_automodel.components.models.ernie5_5.hca._compressed_key_mask(
    attention_mask: torch.Tensor,
    pooled: int,
    ratio: int
) -> torch.Tensor
```

Build the additive mask over compressed keys for rows that start at position zero.

Query `q` sees compressed key `p` when all `ratio` tokens pooled into
that key are the query itself or precede it: `p &lt; (q + 1) // ratio`.

**Parameters:**

**`attention_mask`** `torch.Tensor`

Additive mask of shape \[batch, mask_heads, sequence, sequence]
over the uncompressed keys, where mask_heads is 1 or the head count. Only
its shape, dtype and device are used.

---

**`pooled`** `int`

Number of compressed keys.

---

**`ratio`** `int`

Number of tokens pooled into one compressed key.

---

**Returns:** `torch.Tensor`

Tensor of shape \[batch, mask_heads, sequence, pooled] with the dtype and

```python
nemo_automodel.components.models.ernie5_5.hca._HCA_COMPRESS_RATIO = 128
```