nemo_automodel.components.models.ernie5_5.hca
nemo_automodel.components.models.ernie5_5.hca
ERNIE 5.5 heavily compressed attention (HCA).
HCA is DeepSeek V4’s ratio-128 attention with two additions: a low-rank
virtual-head mix of the per-head attention output, and a sigmoid gate on the
grouped output projection that is driven by the query latent. DeepSeek V4’s
projections, norms, compressor, rotary helper and eager attention are reused
unchanged. Only the order in which they are called is restated here, because
both additions sit between two steps of DeepseekV4Attention.forward.
Provenance: the operation order follows the ERNIE 5.5 HCA of Megatron-Bridge (Apache License 2.0) at revision
a38833622b88969564fe8bfa053c5e4e4989c6cd:
https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/a38833622b88969564fe8bfa053c5e4e4989c6cd/src/megatron/bridge/models/ernie/modeling_ernie55/hca_attention.py
Two kinds of input are supported, both with the torch rotary backend. A row that holds one document starting at position zero runs the eager attention under an explicit causal sliding-window mask. A packed row, or one rank’s shard of it under context parallelism, runs sparse attention over the window keys and the complete compressed keys of each query’s document; neighbouring ranks exchange the window keys at the shard boundary and all ranks share the compressed keys. A KV cache raises an error.
Module Contents
Classes
Functions
Data
API
Bases: DeepseekV4Attention
DeepSeek V4 ratio-128 attention with ERNIE virtual-head mixing and output gating.
Construction, parameter names and the initialization of the DeepSeek V4
parameters are inherited. forward restates the eager, single-document
path of DeepseekV4Attention.forward as of NVIDIA-NeMo/Automodel commit
6f44dbf2390a4eb31bcdc67898bde98ce408f915
(nemo_automodel/components/models/deepseek_v4/layers.py), with the
virtual-head mix inserted after the inverse rotation and the gate after the
grouped projection. With a zero vha_postmix_V and a zero gate weight the
output is exactly half of the parent’s output; the unit tests compare the two
so that a change to the parent’s forward which this class does not follow
fails a test.
The config must describe a compression-ratio-128 layer. It is not sufficient by itself to construct a full ERNIE 5.5 model.
FP32 virtual-head input factor of shape [heads, vha_rank], owned by vha.
FP32 virtual-head output factor of shape [heads, vha_rank], owned by vha.
Attend every query to the window keys and the complete compressed keys of its own document.
Every document starts at a row offset that is a multiple of the compression ratio, and so does every
context-parallel shard, so each compressed key pools the tokens of one document. A rank receives the last
sliding_window - 1 keys of the preceding rank and the compressed keys of all ranks.
Parameters:
Tensor of shape [batch, sequence, hidden].
Query latent of shape [batch, sequence, q_lora_rank].
Rotated query of shape [batch, heads, sequence, head_dim].
Rotated keys of shape [batch, sequence, head_dim].
Compress-rope cosine and sine, each of shape [batch, sequence, rotary_channels].
Rotary module for the compressed keys.
Integer tensor of shape [batch, sequence] with the position of every token in its document.
Integer tensor of shape [batch, sequence] with the document of every token, counted from one along the row; zero marks padding.
Returns: torch.Tensor
Attention output of shape [batch, sequence, heads, head_dim].
Undo the key rotation, mix the heads and apply the gated grouped output projection.
Parameters:
Attention output of shape [batch, sequence, heads, head_dim].
Query latent of shape [batch, sequence, q_lora_rank].
Compress-rope cosine of shape [batch, sequence, rotary_channels].
Compress-rope sine of shape [batch, sequence, rotary_channels].
Returns: torch.Tensor
Output of shape [batch, sequence, hidden].
Reject every input outside the single-document eager path.
Parameters:
Tensor of shape [batch, sequence, hidden].
Additive mask of shape [batch, 1 or heads, sequence, sequence].
First query position; must be zero.
Must be None; positions are the row offsets.
Remaining keyword arguments of forward.
Returns: torch.Tensor
The validated attention_mask, unchanged.
Raises:
ValueError: If the mask is missing or misshaped.NotImplementedError: For a nonzero start position, explicit position ids or a context-parallel group.TypeError: For any other keyword argument.
Run windowed attention over local and compressed keys, then mix heads and gate.
Without packed_seq_ids every row is one document starting at position
zero, and the caller is responsible for the mask contents: a causal mask
limited to the sliding window. Only its shape and dtype kind are checked.
With packed_seq_ids the row is packed and takes no mask; see
_attend_packed.
Parameters:
Tensor of shape [batch, sequence, hidden]. With context
parallelism, sequence is this rank’s shard of a packed row.
Main-rope cosine and sine, each of shape [batch, sequence, rotary_channels]. Unused: a ratio-128 layer rotates with the compress tables. Kept for the parent’s signature.
Additive mask of shape [batch, 1 or heads, sequence, sequence] over the uncompressed keys; required for an unpacked row and None for a packed one.
Required compress-rope cosine and sine, each of shape [batch, sequence, rotary_channels], for the position of every token in its document.
Required rotary module for the compressed keys.
First query position; must be zero.
Integer tensor of shape [batch, sequence] with the position of every token in its document; required for a packed row and None for an unpacked one.
packed_seq_ids, an integer tensor of shape [batch, sequence]
with the document of every token, selects the packed path.
_dsv4_cp_group is accepted only as None; any other name raises.
Returns: torch.Tensor
Output of shape [batch, sequence, hidden], and the attention
Raises:
ValueError: Ifposition_embeddings_compressorrotary_compressis None, or a packed row comes with a mask, withoutposition_ids, with a nonzero start position or with another keyword argument. Every other rejected input of an unpacked row raises in_validate_inputs:ValueErrorfor a missing or misshaped mask,NotImplementedErrorfor a nonzero start position, explicit position ids or a context-parallel group, andTypeErrorfor any other keyword argument.
Initialize inherited and ERNIE parameters after meta materialization.
vha_postmix_V is set to zero, so the virtual-head mix starts as the identity.
Parameters:
Forwarded to DeepseekV4Attention.init_weights.
Standard deviation of the weight draws: truncated normal for the inherited projections and
the gate, normal for vha_postmix_U.
Record the context-parallel group whose ranks hold consecutive shards of a row.
Parameters:
Context-parallel device mesh offered by the parallelizer.
Bases: Module
Keep the FP32 virtual-head parameters in a callable FSDP unit.
Parameters, both FP32: vha_postmix_U of shape [heads, rank], allocated without initialization and filled
by the owning layer, and vha_postmix_V of shape [heads, rank], initialized to zeros.
Mix the attention output across heads with I + V U^T.
Parameters:
Tensor of shape [batch, sequence, heads, head_dim].
Returns: torch.Tensor
Newly allocated tensor with the shape and dtype of attention.
Build the additive mask over compressed keys for rows that start at position zero.
Query q sees compressed key p when all ratio tokens pooled into
that key are the query itself or precede it: p < (q + 1) // ratio.
Parameters:
Additive mask of shape [batch, mask_heads, sequence, sequence] over the uncompressed keys, where mask_heads is 1 or the head count. Only its shape, dtype and device are used.
Number of compressed keys.
Number of tokens pooled into one compressed key.
Returns: torch.Tensor
Tensor of shape [batch, mask_heads, sequence, pooled] with the dtype and