Skip to navigation

nemo_automodel.components.models.ernie5_5.hca

View as MarkdownOpen in Claude

ERNIE 5.5 heavily compressed attention (HCA).

HCA is DeepSeek V4’s ratio-128 attention with two additions: a low-rank virtual-head mix of the per-head attention output, and a sigmoid gate on the grouped output projection that is driven by the query latent. DeepSeek V4’s projections, norms, compressor, rotary helper and eager attention are reused unchanged. Only the order in which they are called is restated here, because both additions sit between two steps of DeepseekV4Attention.forward.

Provenance: the operation order follows the ERNIE 5.5 HCA of Megatron-Bridge (Apache License 2.0) at revision a38833622b88969564fe8bfa053c5e4e4989c6cd: https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/hca_attention.py

Two kinds of input are supported, both with the torch rotary backend. A row that holds one document starting at position zero runs the eager attention under an explicit causal sliding-window mask. A packed row, or one rank’s shard of it under context parallelism, runs sparse attention over the window keys and the complete compressed keys of each query’s document; neighbouring ranks exchange the window keys at the shard boundary and all ranks share the compressed keys. A KV cache raises an error.

Module Contents

Classes

NameDescription
Ernie5_5HCAAttentionDeepSeek V4 ratio-128 attention with ERNIE virtual-head mixing and output gating.
Ernie5_5HCAVirtualHeadsKeep the FP32 virtual-head parameters in a callable FSDP unit.

Functions

NameDescription
_compressed_key_maskBuild the additive mask over compressed keys for rows that start at position zero.

Data

_HCA_COMPRESS_RATIO

API

class nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAAttention(
layer_idx: int,
vha_rank: int = 16
)

Bases: DeepseekV4Attention

DeepSeek V4 ratio-128 attention with ERNIE virtual-head mixing and output gating.

Construction, parameter names and the initialization of the DeepSeek V4 parameters are inherited. forward restates the eager, single-document path of DeepseekV4Attention.forward as of NVIDIA-NeMo/Automodel commit 6f44dbf2390a4eb31bcdc67898bde98ce408f915 (nemo_automodel/components/models/deepseek_v4/layers.py), with the virtual-head mix inserted after the inverse rotation and the gate after the grouped projection. With a zero vha_postmix_V and a zero gate weight the output is exactly half of the parent’s output; the unit tests compare the two so that a change to the parent’s forward which this class does not follow fails a test.

The config must describe a compression-ratio-128 layer. It is not sufficient by itself to construct a full ERNIE 5.5 model.

cp_group
ProcessGroup | None = None
gate_proj
vha
vha_postmix_U
Tensor

FP32 virtual-head input factor of shape [heads, vha_rank], owned by vha.

vha_postmix_V
Tensor

FP32 virtual-head output factor of shape [heads, vha_rank], owned by vha.

nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAAttention._attend_packed(
hidden_states: torch.Tensor,
q_residual: torch.Tensor,
query: torch.Tensor,
keys: torch.Tensor,
rope: tuple[torch.Tensor, torch.Tensor],
rotary: torch.nn.Module,
position_ids: torch.Tensor,
packed_seq_ids: torch.Tensor
) -> torch.Tensor

Attend every query to the window keys and the complete compressed keys of its own document.

Every document starts at a row offset that is a multiple of the compression ratio, and so does every context-parallel shard, so each compressed key pools the tokens of one document. A rank receives the last sliding_window - 1 keys of the preceding rank and the compressed keys of all ranks.

Parameters:

hidden_states
torch.Tensor

Tensor of shape [batch, sequence, hidden].

q_residual
torch.Tensor

Query latent of shape [batch, sequence, q_lora_rank].

query
torch.Tensor

Rotated query of shape [batch, heads, sequence, head_dim].

keys
torch.Tensor

Rotated keys of shape [batch, sequence, head_dim].

rope
tuple[torch.Tensor, torch.Tensor]

Compress-rope cosine and sine, each of shape [batch, sequence, rotary_channels].

rotary
nn.Module

Rotary module for the compressed keys.

position_ids
torch.Tensor

Integer tensor of shape [batch, sequence] with the position of every token in its document.

packed_seq_ids
torch.Tensor

Integer tensor of shape [batch, sequence] with the document of every token, counted from one along the row; zero marks padding.

Returns: torch.Tensor

Attention output of shape [batch, sequence, heads, head_dim].

nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAAttention._project_output(
attended: torch.Tensor,
q_residual: torch.Tensor,
cos: torch.Tensor,
sin: torch.Tensor
) -> torch.Tensor

Undo the key rotation, mix the heads and apply the gated grouped output projection.

Parameters:

attended
torch.Tensor

Attention output of shape [batch, sequence, heads, head_dim].

q_residual
torch.Tensor

Query latent of shape [batch, sequence, q_lora_rank].

cos
torch.Tensor

Compress-rope cosine of shape [batch, sequence, rotary_channels].

sin
torch.Tensor

Compress-rope sine of shape [batch, sequence, rotary_channels].

Returns: torch.Tensor

Output of shape [batch, sequence, hidden].

nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAAttention._validate_inputs(
hidden_states: torch.Tensor,
attention_mask: torch.Tensor | None,
start_pos: int,
position_ids: torch.Tensor | None,
extra: dict[str, typing.Any]
) -> torch.Tensor

Reject every input outside the single-document eager path.

Parameters:

hidden_states
torch.Tensor

Tensor of shape [batch, sequence, hidden].

attention_mask
torch.Tensor | None

Additive mask of shape [batch, 1 or heads, sequence, sequence].

start_pos
int

First query position; must be zero.

position_ids
torch.Tensor | None

Must be None; positions are the row offsets.

extra
dict[str, Any]

Remaining keyword arguments of forward.

Returns: torch.Tensor

The validated attention_mask, unchanged.

Raises:

  • ValueError: If the mask is missing or misshaped.
  • NotImplementedError: For a nonzero start position, explicit position ids or a context-parallel group.
  • TypeError: For any other keyword argument.
nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAAttention.forward(
hidden_states: torch.Tensor,
position_embeddings: tuple[torch.Tensor, torch.Tensor],
attention_mask: torch.Tensor | None = None,
position_embeddings_compress: tuple[torch.Tensor, torch.Tensor] | None = None,
rotary_compress: torch.nn.Module | None = None,
start_pos: int = 0,
position_ids: torch.Tensor | None = None,
kwargs: typing.Any = {}
) -> tuple[torch.Tensor, torch.Tensor | None]

Run windowed attention over local and compressed keys, then mix heads and gate.

Without packed_seq_ids every row is one document starting at position zero, and the caller is responsible for the mask contents: a causal mask limited to the sliding window. Only its shape and dtype kind are checked. With packed_seq_ids the row is packed and takes no mask; see _attend_packed.

Parameters:

hidden_states
torch.Tensor

Tensor of shape [batch, sequence, hidden]. With context parallelism, sequence is this rank’s shard of a packed row.

position_embeddings
tuple[torch.Tensor, torch.Tensor]

Main-rope cosine and sine, each of shape [batch, sequence, rotary_channels]. Unused: a ratio-128 layer rotates with the compress tables. Kept for the parent’s signature.

attention_mask
torch.Tensor | NoneDefaults to None

Additive mask of shape [batch, 1 or heads, sequence, sequence] over the uncompressed keys; required for an unpacked row and None for a packed one.

position_embeddings_compress
tuple[torch.Tensor, torch.Tensor] | NoneDefaults to None

Required compress-rope cosine and sine, each of shape [batch, sequence, rotary_channels], for the position of every token in its document.

rotary_compress
nn.Module | NoneDefaults to None

Required rotary module for the compressed keys.

start_pos
intDefaults to 0

First query position; must be zero.

position_ids
torch.Tensor | NoneDefaults to None

Integer tensor of shape [batch, sequence] with the position of every token in its document; required for a packed row and None for an unpacked one.

**kwargs
AnyDefaults to {}

packed_seq_ids, an integer tensor of shape [batch, sequence] with the document of every token, selects the packed path. _dsv4_cp_group is accepted only as None; any other name raises.

Returns: torch.Tensor

Output of shape [batch, sequence, hidden], and the attention

Raises:

  • ValueError: If position_embeddings_compress or rotary_compress is None, or a packed row comes with a mask, without position_ids, with a nonzero start position or with another keyword argument. Every other rejected input of an unpacked row raises in _validate_inputs: ValueError for a missing or misshaped mask, NotImplementedError for a nonzero start position, explicit position ids or a context-parallel group, and TypeError for any other keyword argument.
nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAAttention.init_weights(
buffer_device: torch.device,
init_std: float = 0.02
) -> None

Initialize inherited and ERNIE parameters after meta materialization.

vha_postmix_V is set to zero, so the virtual-head mix starts as the identity.

Parameters:

buffer_device
torch.device

Forwarded to DeepseekV4Attention.init_weights.

init_std
floatDefaults to 0.02

Standard deviation of the weight draws: truncated normal for the inherited projections and the gate, normal for vha_postmix_U.

nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAAttention.setup_cp_attention(
cp_mesh: torch.distributed.device_mesh.DeviceMesh
) -> None

Record the context-parallel group whose ranks hold consecutive shards of a row.

Parameters:

cp_mesh
DeviceMesh

Context-parallel device mesh offered by the parallelizer.

class nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAVirtualHeads(
num_heads: int,
rank: int
)

Bases: Module

Keep the FP32 virtual-head parameters in a callable FSDP unit.

Parameters, both FP32: vha_postmix_U of shape [heads, rank], allocated without initialization and filled by the owning layer, and vha_postmix_V of shape [heads, rank], initialized to zeros.

vha_postmix_U
vha_postmix_V
nemo_automodel.components.models.ernie5_5.hca.Ernie5_5HCAVirtualHeads.forward(
attention: torch.Tensor
) -> torch.Tensor

Mix the attention output across heads with I + V U^T.

Parameters:

attention
torch.Tensor

Tensor of shape [batch, sequence, heads, head_dim].

Returns: torch.Tensor

Newly allocated tensor with the shape and dtype of attention.

nemo_automodel.components.models.ernie5_5.hca._compressed_key_mask(
attention_mask: torch.Tensor,
pooled: int,
ratio: int
) -> torch.Tensor

Build the additive mask over compressed keys for rows that start at position zero.

Query q sees compressed key p when all ratio tokens pooled into that key are the query itself or precede it: p < (q + 1) // ratio.

Parameters:

attention_mask
torch.Tensor

Additive mask of shape [batch, mask_heads, sequence, sequence] over the uncompressed keys, where mask_heads is 1 or the head count. Only its shape, dtype and device are used.

pooled
int

Number of compressed keys.

ratio
int

Number of tokens pooled into one compressed key.

Returns: torch.Tensor

Tensor of shape [batch, mask_heads, sequence, pooled] with the dtype and

nemo_automodel.components.models.ernie5_5.hca._HCA_COMPRESS_RATIO = 128