nemo_automodel.components.models.ernie5_5.dsa
nemo_automodel.components.models.ernie5_5.dsa
ERNIE 5.5 dynamic sparse attention (DSA).
A DSA layer attends, for every query, to a causal local window of the query’s document and to the remote keys of the same document that an indexer selects. This module holds the layer and its parts:
apply_indexer_ropeandapply_main_rope: the two rotary channel layouts of a DSA layer.rotate_indexer_activation: the scaled Hadamard rotation of the indexer query and key.Ernie5_5Indexer: the projections that produce the indexer query, key and per-head weights.ernie5_5_index_scoresandselect_window_and_remote: the index scores and the dense key selection of the eager path.latent_mqa_core: the eager attention core over a dense [sequence, sequence] selection.merge_remote_and_windowandcudnn_dsa_core: the sparse key indices and the attention core of the cuDNN path.cudnn_dsa_layout: the document layout of a packed row, or of a context-parallel shard of it, for the cuDNN path.Ernie5_5DSAFP32ParametersandErnie5_5DSASparseFP32Parameters: the FP32 sink and virtual-head parameters, each with the core that consumes them.Ernie5_5DSAAttention: the layer.backend.attnselects the eager path for short sequences or the cuDNN path; both paths share all parameters, and both attend within each document of a packed row and across the context-parallel shards of a row.freeze_ernie5_5_indexer_params: disables gradients for the indexer parameters.
Provenance: the projection layout, the indexer precision contract and the operation order follow two sources.
- The ERNIE 5.5 DSA-layer attention of Megatron-Bridge (Apache License 2.0) at revision
7c55987a027084e87e36277c9711e87bc52a2116: https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/7c55987a027084e87e36277c9711e87bc52a2116/src/megatron/bridge/models/ernie/modeling_ernie55/latent_mqa_attention.py - The DSA of Megatron-LM (3-clause BSD terms of the repository
LICENSEfile; its copyright notices, conditions and disclaimer are retained in the comment block above) at revisionf78809f6d9c88f8638ea7822966f35057a63fd5c: https://github.com/NVIDIA/Megatron-LM/blob/f78809f6d9c88f8638ea7822966f35057a63fd5c/megatron/core/transformer/experimental_attention_variant/dsa.py
Module Contents
Classes
Functions
Data
API
Bases: Module
ERNIE 5.5 dynamic sparse attention (DSA) layer.
Latent multi-query attention over a causal local window plus the remote keys an indexer selects, followed by the value up-projection, virtual-head mixing, the query gate and the output projection.
The attention backend selects one of two paths that share all parameters:
backend.attn == "cudnn": cuDNN top-k selection and sparse attention for one row: an unpadded document, or a packed row with the metadata ofcudnn_dsa_layout. The forward runs with autocast disabled, and the kernels require BF16 inputs and the released latent and indexer dimensions.backend.attnin("eager", "torch", "sdpa"), or no backend: eager PyTorch attention over a dense [sequence, sequence] mask, limited tomax_eager_sequencetokens. The three names select this same path; no scaled-dot-product kernel is called.
Every other value of backend.attn is rejected.
Under context parallelism a rank holds one contiguous shard of the row as queries and gathers the latent and indexer keys of all ranks. Neither path supports a KV cache.
The indexer only selects keys. Its projections receive no gradient from the
attention output, and this layer computes no indexer loss, so the indexer is
not trained here; freeze_ernie5_5_indexer_params keeps it frozen. This
differs from the reference training recipe, which trains the indexer with a
KL term against the attention distribution.
FP32 per-head sink logits of shape [heads], owned by fp32_params.
FP32 virtual-head input factor of shape [heads, vha_rank], owned by fp32_params.
FP32 virtual-head output factor of shape [heads, vha_rank], owned by fp32_params.
Run the cuDNN path on one row, or on this rank’s shard of it.
Parameters:
Tensor of shape [1, sequence, hidden].
Integer tensor of shape [1, sequence] with the position of each token in its document.
Int32 cumulative document lengths of the row; see cudnn_dsa_core.
Length of the document, equal to sequence. Not read with metadata.
Layout of a packed row and of the queries in it, or None for an unpadded row that holds one document.
Returns: torch.Tensor
Output of shape [1, sequence, hidden].
Raises:
ValueError: Ifhiddenis not [1, sequence, hidden] orposition_idsis not [1, sequence].
Run the eager path over a dense [sequence, sequence] selection.
Parameters:
Tensor of shape [batch, sequence, hidden], with at most max_eager_sequence tokens per row.
Integer tensor of shape [batch, sequence] that identifies each contiguous document of a row; negative ids are padding.
Integer tensor of shape [batch, sequence]; positions restart for each document.
Returns: torch.Tensor
Output of shape [batch, sequence, hidden].
Raises:
ValueError: Ifhiddenis not rank 3, the ids do not have shape [batch, sequence], or the sequence is empty or longer thanmax_eager_sequence.
Project hidden states to the attention and indexer inputs.
Parameters:
Tensor of shape [batch, sequence, hidden].
Integer tensor of shape [batch, sequence].
Returns: torch.Tensor
The query latent [batch, sequence, q_rank], the attention query
Return the attention output.
The eager path takes document_ids; the cuDNN path takes cu_seqlens
and max_seqlen, and metadata for a packed row. Passing the metadata
of the other path raises. On the cuDNN path a row that holds several
documents without metadata is rejected here, before the projections run;
cudnn_dsa_core describes the remaining checks of cu_seqlens and
max_seqlen.
Parameters:
Tensor of shape [batch, sequence, hidden]. The cuDNN path requires batch 1. Under
context parallelism, sequence is this rank’s shard of the row.
Integer tensor of shape [batch, sequence]; positions restart for each document.
Eager path only. Integer tensor of shape [batch, sequence] that identifies each contiguous document of a row; negative ids are padding.
cuDNN path only. Int32 cumulative document lengths of the row: [0, sequence]
of shape [2] for an unpadded row that holds one document, and of shape
[documents + 1] with metadata.
cuDNN path only. Length of the document, equal to sequence. Not read with
metadata.
cuDNN path only. Layout of a packed row and of the queries in it, as
cudnn_dsa_layout returns it.
Returns: torch.Tensor
Output of shape [batch, sequence, hidden], and None in the place where
Raises:
ValueError: If shapes disagree, the metadata does not match the path, or the eager path receives no token or more thanmax_eager_sequencetokens.NotImplementedError: Ifcu_seqlenshas more than two entries withoutmetadata.
Initialize all parameters, including after meta-device materialization.
The projections, the two separate K/V parameters and vha_postmix_U are drawn from a normal
distribution, the norms are reset, and vha_postmix_V and the sink logits are set to zero.
Parameters:
Positive finite standard deviation of the normal draws.
Raises:
ValueError: Ifinit_stdis not a positive finite number. No parameter is changed in that case.
Record the context-parallel group whose ranks hold consecutive shards of a row.
Parameters:
Context-parallel device mesh offered by the parallelizer.
Bases: Module
Hold the FP32 sink and virtual-head parameters of a DSA layer and run the eager core with them.
The three parameters form one module so that a dtype-aware FSDP wrap can treat them as one FP32 unit.
Parameters, all FP32: softmax_offset of shape [heads], the per-head sink logits, initialized to zeros;
vha_postmix_U of shape [heads, rank], allocated without initialization and filled by the owning layer;
vha_postmix_V of shape [heads, rank], initialized to zeros.
Run latent_mqa_core with the parameters of this module.
The core runs inside this module’s forward because returning the parameters themselves would allow an FSDP
post-forward reshard to free their storage before the caller uses them. latent_mqa_core validates the
inputs and raises ValueError for shapes that disagree.
Parameters:
Tensor of shape [batch, sequence, heads, latent + rope], the queries of this rank.
Tensor of shape [batch, keys, latent + rope] shared by all heads: the keys of the
whole row in row order. keys equals sequence without context parallelism.
Finite index scores of shape [batch, sequence, keys], indexed [batch, query, key].
Integer tensor of shape [batch, sequence]; negative ids are padding.
Value up-projection of shape [heads, value_dim, latent].
Tensor of shape [batch, sequence, heads * value_dim].
Positive number of local causal keys per query.
Non-negative number of remote keys per query.
Factor applied to the query-key products.
Optional integer tensor of shape [batch, keys]; see select_window_and_remote.
Index of the first query among the keys.
Returns: torch.Tensor
Core output of shape [batch, sequence, heads * value_dim] in the dtype of query.
Raises:
RuntimeError: If one of the three parameters of this module is not FP32.
Bases: Ernie5_5DSAFP32Parameters
Hold the same FP32 parameters as the parent class and run the cuDNN core with them.
The core runs inside this module for the reason given in Ernie5_5DSAFP32Parameters.forward.
Run cudnn_dsa_core with the parameters of this module.
cudnn_dsa_core validates the inputs: it raises ValueError for shapes that disagree and
NotImplementedError for a row that holds several documents without metadata.
Parameters:
BF16 tensor of shape [tokens, heads, latent + rope], the queries of this rank.
BF16 tensor of shape [keys, latent + rope] shared by all heads: the keys of the whole
row in row order. keys equals tokens without context parallelism.
BF16 indexer query of shape [tokens, index_heads, index_dim].
BF16 indexer key of shape [keys, index_dim], in the order of latent_kv.
Indexer head weights of shape [tokens, index_heads].
Value up-projection of shape [heads, value_dim, latent].
Tensor of shape [tokens, heads * value_dim].
Int32 cumulative document lengths of the row; see cudnn_dsa_core.
Length of the document, equal to tokens. Not read with metadata.
Positive number of local causal keys per query.
Positive number of remote keys the indexer selects per query.
Factor applied to the query-key products.
Layout of a packed row and of the queries in it, as cudnn_dsa_layout returns it.
Returns: torch.Tensor
Core output of shape [tokens, heads * value_dim] in the dtype of query.
Raises:
RuntimeError: If one of the three parameters of this module is not FP32.
Bases: Module
Project hidden states and the query latent to the indexer query, key and per-head weights.
The outputs feed the index scores, which only decide which remote keys the attention reads. Both inputs are detached, so no gradient reaches the rest of the model through the indexer.
Parameters: wq_b.weight of shape [heads * head_dim, q_rank], wk.weight of shape [head_dim, hidden] and
weights_proj.weight of shape [heads, hidden] in the configured dtype, and k_norm.weight and
k_norm.bias of shape [head_dim], which must stay FP32.
Project and rotate the indexer inputs without propagating gradients into the backbone.
Parameters:
Tensor of shape [batch, sequence, hidden]. It is detached before the projections.
Tensor of shape [batch, sequence, q_rank]. It is detached before the projection.
Position rotation. It is called with the query of shape [batch, sequence, heads, head_dim] and with
the key of shape [batch, sequence, 1, head_dim], and must return a tensor of the same shape and dtype.
Each result is then rotated by rotate_indexer_activation.
Returns: torch.Tensor
A tuple of the rotated query of shape [batch, sequence, heads, head_dim], the rotated key of shape
Raises:
RuntimeError: If the weight or the bias of the key LayerNorm is not FP32.
Initialize the projections and the key norm, including after meta-device materialization.
Parameters:
Positive finite standard deviation of the projection weights.
Raises:
ValueError: Ifinit_stdis not a positive finite number.
Return the key selection of select, or the one that the forward of this checkpointed layer call recorded.
Two calls can select different keys among equal index scores, so the recomputation of a checkpointed decoder
layer reuses the selection of its forward. Ernie5_5Block.nemo_checkpoint_context_fn binds the record of one
call; outside a checkpointed call select runs unchanged.
Parameters:
Computes the selection without gradient tracking: a boolean tensor of shape [batch, queries, keys] on the eager path, or int32 row offsets of shape [queries, remote_topk] on the cuDNN path.
Returns: torch.Tensor
The selection of the forward of this call.
Rotate the leading channels of every head in the half-split rotary layout.
The first rope channels of a head are rotated, pairing channel i with channel i + rope / 2; the
remaining channels pass through. The cosine and sine are computed in the dtype of frequencies and then cast
to the dtype of x.
Parameters:
Tensor of shape [batch, sequence, heads, channels].
Rotation angles in radians, of shape [batch, sequence, rope], on the device of x. rope is
the number of rotated channels: positive, even and at most channels. Entries i and
i + rope / 2 are the angle of one channel pair and are expected to be equal; this is not checked.
Returns: torch.Tensor
Newly allocated tensor with the shape and dtype of x: the first rope channels rotated, the other
Raises:
ValueError: Ifxis not rank 4,frequenciesis not rank 3, their batch and sequence dimensions differ, orropeis zero, odd or larger thanchannels.
Rotate adjacent channel pairs and return them in the half-split layout.
The input pairs channel 2i with channel 2i + 1. The output holds the rotated even channels in its first
half and the rotated odd channels in its second half, which is the layout of apply_indexer_rope.
Parameters:
Tensor of shape [batch, sequence, heads, rope], with rope even. Every channel is rotated.
Rotation angles in radians, of shape [batch, sequence, rope], on the device of x; see
apply_indexer_rope.
Returns: torch.Tensor
Newly allocated tensor with the shape and dtype of x, in the half-split layout.
Raises:
ValueError: Ifxis not rank 4,frequenciesis not rank 3, their final dimensions differ or are odd or zero, or their batch and sequence dimensions differ.
Run the cuDNN DSA core on the tokens of one row.
cudnn_indexer_topk selects the remote keys of every query, merge_remote_and_window adds the local
window, and cudnn_sparse_attention attends over the union with the per-head sink. The attended latent is
then projected to the value channels of each head in FP32, mixed across heads and gated in the dtype of
query, as in latent_mqa_core. The indexer inputs are detached, so the selection receives no gradient.
Without metadata the row is unpadded and holds one document, and queries and keys are the same tokens
rows: cu_seqlens with more than two entries raises NotImplementedError, and before the first kernel is
called prepare_cudnn_dsa_packed_metadata validates the rest of the document metadata: it raises
TypeError unless cu_seqlens is int32, and ValueError unless cu_seqlens is one-dimensional,
starts at zero and ends at tokens and max_seqlen equals tokens. With the metadata of
cudnn_dsa_layout the row is packed, the keys are all of its tokens and the queries are all of them or one
context-parallel shard.
All tensors must be on one CUDA device. The two kernels accept BF16 inputs with latent + rope equal to 512
or 576, index_dim equal to 128, 32 or 64 indexer heads and remote_topk of at most 2048, and raise
otherwise.
Parameters:
BF16 tensor of shape [tokens, heads, latent + rope].
BF16 tensor of shape [keys, latent + rope], the key shared by all heads. Its first latent
channels are also the value. keys equals tokens unless metadata describes a shard.
BF16 indexer query of shape [tokens, index_heads, index_dim].
BF16 indexer key of shape [keys, index_dim].
FP32 or BF16 indexer head weights of shape [tokens, index_heads], as Ernie5_5Indexer
returns them. They are multiplied by sqrt(index_dim) in their own dtype before the kernel.
FP32 per-head sink logits of shape [heads].
Value up-projection of shape [heads, value_dim, latent].
Tensor of shape [tokens, heads * value_dim].
Virtual-head input factor of shape [heads, rank].
Virtual-head output factor of shape [heads, rank].
Int32 cumulative document lengths of the row: [0, tokens] of shape [2] without
metadata, of shape [documents + 1] with it.
Length of the document, equal to tokens. Not read with metadata.
Positive number of local causal keys per query.
Positive number of remote keys the indexer selects per query.
Factor applied to the query-key products.
Layout of a packed row and of the queries in it, as cudnn_dsa_layout returns it.
Returns: torch.Tensor
Core output of shape [tokens, heads * value_dim] in the dtype of query.
Raises:
ValueError: Iftokensis zero, the shapes of the query, the latent key/value, the value projection or the gate logits disagree,latentexceedslatent + rope, orwindoworremote_topkis not positive.NotImplementedError: Ifcu_seqlenshas more than two entries withoutmetadata.
Describe the documents of a packed row, and the queries in it, for the cuDNN DSA core.
Parameters:
Integer tensor of shape [row] on a CUDA device with the document of every token of the row. The ids increase along the row, a negative id marks padding, the row starts with a document, and the padding of a document follows its tokens.
Row offset of the first query.
Number of queries: the whole row, or one context-parallel shard of it.
Returns: tuple[torch.Tensor, CudnnDsaPackedMetadata]
The int32 cumulative document lengths of shape [documents + 1], and the metadata of the queries.
Score every key for every query with the ERNIE scaling order.
The score of a query and a key is the sum over heads of relu(<q_head, k>) * weight_head. The products and
the sum are computed in FP32 with autocast disabled. The weights are multiplied by sqrt(channels) in their
own dtype before the sum, and the sum is divided by sqrt(channels) afterwards; with BF16 weights this
rounding order affects which keys a top-k selects.
The scores are not masked. The caller applies the causal and document limits and selects the top-k, as
select_window_and_remote does.
Parameters:
Indexer query of shape [batch, queries, heads, channels].
Indexer key of shape [batch, keys, channels], shared by all heads.
Per-head weights of shape [batch, queries, heads].
Returns: torch.Tensor
FP32 scores of shape [batch, queries, keys].
Raises:
ValueError: Ifqis not rank 4,kis not rank 3,weightsis not [batch, queries, heads], or the batch or channel dimensions ofqandkdiffer.
Set requires_grad to False on every parameter of every Ernie5_5Indexer in a model.
The indexer only selects keys and receives no gradient from the attention output, so its parameters are kept
out of the trainable set. The call changes the flags once and installs no hook: code that enables gradients
afterwards, such as a trainability policy or model.requires_grad_(True), must call this function again.
Parameters:
Model, wrapper or pipeline stage that may contain ERNIE 5.5 DSA indexers. Parameters outside an
Ernie5_5Indexer are left unchanged.
Run the eager DSA core: latent multi-query attention with a sink, value up-projection, head mix and gate.
Every head attends with its own query to one latent key/value per token, over the keys that
select_window_and_remote selects. A per-head sink logit takes part in the softmax and its probability is
discarded, so a query without a selected key (padding) produces zeros. The attended latent is projected to the
value channels of each head, mixed across heads with virtual_head_mix and multiplied by the sigmoid of the
gate logits.
The logits, the softmax and the value up-projection are computed in FP32 with autocast disabled, whatever the
input dtypes; the head mix and the gate run in the dtype of query. The result is not guaranteed to equal the
result of cudnn_dsa_core bit for bit.
Parameters:
Tensor of shape [batch, sequence, heads, latent + rope]: the query of each head in the key space,
latent content channels followed by rope rotary channels.
Tensor of shape [batch, keys, latent + rope], the key shared by all heads. Its first
latent channels are also the value. keys equals sequence unless the queries are one
context-parallel shard of the row.
Finite index scores of shape [batch, sequence, keys], indexed [batch, query, key].
Integer tensor of shape [batch, sequence]; see select_window_and_remote.
Per-head sink logits of shape [heads].
Value up-projection of shape [heads, value_dim, latent].
Tensor of shape [batch, sequence, heads * value_dim].
Virtual-head input factor of shape [heads, rank].
Virtual-head output factor of shape [heads, rank].
Positive number of local keys per query.
Non-negative number of remote keys per query.
Factor applied to the query-key products.
Optional integer tensor of shape [batch, keys]; see select_window_and_remote.
Index of the first query among the keys.
Returns: torch.Tensor
Tensor of shape [batch, sequence, heads * value_dim] in the dtype of query, with the value channels of
Raises:
ValueError: If a rank or a dimension disagrees with the shapes above, orselect_window_and_remoterejects the scores, the document ids,windoworremote_topk.
Combine the remote indices of every query with its local causal window.
Keys are the tokens of one row, queries are all of them or one context-parallel shard, and an index is the
offset of a token in that row. The remote part of the result keeps the input order and replaces with -1 every
index that is absent, belongs to another document, or lies inside or after the local window. The local part
lists the last window offsets up to the query in ascending order, with -1 for positions before the start of
the document.
The real tokens of a document must be contiguous from its start. The values of the inputs are not checked for consistency.
Parameters:
Int32 tensor of shape [queries, remote_topk] with row offsets of remote keys, or -1 for absent entries.
Int32 tensor of shape [queries] with the row offset of the first token of each query’s document.
Int32 tensor of shape [queries] counting the tokens from the start of the document through the query.
Boolean tensor of shape [queries]. A False entry gives a row of -1.
Positive number of local causal positions per query.
Returns: torch.Tensor
The int32 indices of shape [queries, remote_topk + window], remote part first, with invalid entries set to
Raises:
ValueError: Ifwindowis not a positive integer,remote_indicesis not an int32 matrix, a metadata tensor does not have shape [queries], the starts or the lengths are not int32, the validity is not boolean, or the tensors are on different devices.
Apply the scaled Hadamard rotation of the DSA indexer.
Precision contract: the input, of any floating-point dtype, is rounded to BF16; the butterfly and the scaling by
channels ** -0.5 run in FP32; the result is rounded to BF16 once. On a CUDA device the
fast_hadamard_transform kernel computes this when the package is installed. Without the package, and on
every other device, the same arithmetic runs in PyTorch, so both paths return the same bits.
Parameters:
Floating-point tensor of shape […, channels], with arbitrary leading dimensions; channels is a
power of two.
Returns: torch.Tensor
BF16 tensor of shape […, channels] on the device of x.
Raises:
ValueError: Ifchannelsis not a power of two.
Select, for every query, a causal local window and the top-scoring remote keys of its document.
A key is eligible for a query when both are real tokens of the same document and the key is not after the
query. The local window is the query and the window - 1 eligible keys before it. The other eligible keys are
remote, and the remote_topk highest-scoring of them are added. The function runs without gradient tracking.
The queries are the keys, or, with key_document_ids, the consecutive keys that start at first_query:
one context-parallel shard of the row.
Parameters:
Finite floating tensor of shape [batch, queries, keys], indexed [batch, query, key].
Integer tensor of shape [batch, queries] on the device of scores. Tokens of one document
carry the same id, and a negative id marks padding. The caller must give every contiguous document of a
row its own id: tokens that share an id are treated as one document.
Positive number of local keys per query, the query included.
Non-negative number of remote keys per query. Zero selects the local window only.
Optional integer tensor of shape [batch, keys] with the document of every key. None means that the keys are the queries.
Index of the first query among the keys.
Returns: torch.Tensor
Boolean tensor of shape [batch, queries, keys], indexed [batch, query, key], that is True for the
Raises:
ValueError: Ifscoresis not [batch, queries, keys],document_idsis not [batch, queries], the key ids are not [batch, keys],windowis not positive,remote_topkis negative or a score is not finite.