Skip to navigation

nemo_automodel.components.models.ernie5_5.dsa

View as MarkdownOpen in Claude

ERNIE 5.5 dynamic sparse attention (DSA).

A DSA layer attends, for every query, to a causal local window of the query’s document and to the remote keys of the same document that an indexer selects. This module holds the layer and its parts:

  • apply_indexer_rope and apply_main_rope: the two rotary channel layouts of a DSA layer.
  • rotate_indexer_activation: the scaled Hadamard rotation of the indexer query and key.
  • Ernie5_5Indexer: the projections that produce the indexer query, key and per-head weights.
  • ernie5_5_index_scores and select_window_and_remote: the index scores and the dense key selection of the eager path.
  • latent_mqa_core: the eager attention core over a dense [sequence, sequence] selection.
  • merge_remote_and_window and cudnn_dsa_core: the sparse key indices and the attention core of the cuDNN path.
  • cudnn_dsa_layout: the document layout of a packed row, or of a context-parallel shard of it, for the cuDNN path.
  • Ernie5_5DSAFP32Parameters and Ernie5_5DSASparseFP32Parameters: the FP32 sink and virtual-head parameters, each with the core that consumes them.
  • Ernie5_5DSAAttention: the layer. backend.attn selects the eager path for short sequences or the cuDNN path; both paths share all parameters, and both attend within each document of a packed row and across the context-parallel shards of a row.
  • freeze_ernie5_5_indexer_params: disables gradients for the indexer parameters.

Provenance: the projection layout, the indexer precision contract and the operation order follow two sources.

Module Contents

Classes

NameDescription
Ernie5_5DSAAttentionERNIE 5.5 dynamic sparse attention (DSA) layer.
Ernie5_5DSAFP32ParametersHold the FP32 sink and virtual-head parameters of a DSA layer and run the eager core with them.
Ernie5_5DSASparseFP32ParametersHold the same FP32 parameters as the parent class and run the cuDNN core with them.
Ernie5_5IndexerProject hidden states and the query latent to the indexer query, key and per-head weights.

Functions

NameDescription
_selection_of_this_callReturn the key selection of select, or the one that the forward of this checkpointed layer call recorded.
apply_indexer_ropeRotate the leading channels of every head in the half-split rotary layout.
apply_main_ropeRotate adjacent channel pairs and return them in the half-split layout.
cudnn_dsa_coreRun the cuDNN DSA core on the tokens of one row.
cudnn_dsa_layoutDescribe the documents of a packed row, and the queries in it, for the cuDNN DSA core.
ernie5_5_index_scoresScore every key for every query with the ERNIE scaling order.
freeze_ernie5_5_indexer_paramsSet requires_grad to False on every parameter of every Ernie5_5Indexer in a model.
latent_mqa_coreRun the eager DSA core: latent multi-query attention with a sink, value up-projection, head mix and gate.
merge_remote_and_windowCombine the remote indices of every query with its local causal window.
rotate_indexer_activationApply the scaled Hadamard rotation of the DSA indexer.
select_window_and_remoteSelect, for every query, a causal local window and the top-scoring remote keys of its document.

Data

dsa_selection_replay

API

class nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAAttention(
hidden_size: int,
num_heads: int,
q_rank: int,
kv_rank: int,
nope_dim: int,
rope_dim: int,
value_dim: int,
index_heads: int,
index_dim: int,
vha_rank: int,
window: int,
remote_topk: int,
dtype: torch.dtype,
rope_theta: float = 8000000.0,
norm_eps: float = 1e-06,
max_eager_sequence: int = 512,
)

Bases: Module

ERNIE 5.5 dynamic sparse attention (DSA) layer.

Latent multi-query attention over a causal local window plus the remote keys an indexer selects, followed by the value up-projection, virtual-head mixing, the query gate and the output projection.

The attention backend selects one of two paths that share all parameters:

  • backend.attn == "cudnn": cuDNN top-k selection and sparse attention for one row: an unpadded document, or a packed row with the metadata of cudnn_dsa_layout. The forward runs with autocast disabled, and the kernels require BF16 inputs and the released latent and indexer dimensions.
  • backend.attn in ("eager", "torch", "sdpa"), or no backend: eager PyTorch attention over a dense [sequence, sequence] mask, limited to max_eager_sequence tokens. The three names select this same path; no scaled-dot-product kernel is called.

Every other value of backend.attn is rejected.

Under context parallelism a rank holds one contiguous shard of the row as queries and gathers the latent and indexer keys of all ranks. Neither path supports a KV cache.

The indexer only selects keys. Its projections receive no gradient from the attention output, and this layer computes no indexer loss, so the indexer is not trained here; freeze_ernie5_5_indexer_params keeps it frozen. This differs from the reference training recipe, which trains the indexer with a KL term against the attention distribution.

cp_group
ProcessGroup | None = None
fp32_params
= holder(num_heads, vha_rank)
gate_proj
indexer
k_b_proj
kv_a_layernorm
= nn.RMSNorm(kv_rank, eps=norm_eps, dtype=dtype)
kv_a_proj_with_mqa
o_proj
q_a_layernorm
= nn.RMSNorm(q_rank, eps=norm_eps, dtype=dtype)
q_a_proj
q_b_proj
softmax_offset
Tensor

FP32 per-head sink logits of shape [heads], owned by fp32_params.

softmax_scale
= (nope_dim + rope_dim) ** -0.5
uses_cudnn
= attn == 'cudnn'
v_b_proj
vha_postmix_U
Tensor

FP32 virtual-head input factor of shape [heads, vha_rank], owned by fp32_params.

vha_postmix_V
Tensor

FP32 virtual-head output factor of shape [heads, vha_rank], owned by fp32_params.

nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAAttention._forward_cudnn(
hidden: torch.Tensor,
position_ids: torch.Tensor,
cu_seqlens: torch.Tensor,
max_seqlen: int,
) -> torch.Tensor

Run the cuDNN path on one row, or on this rank’s shard of it.

Parameters:

hidden
torch.Tensor

Tensor of shape [1, sequence, hidden].

position_ids
torch.Tensor

Integer tensor of shape [1, sequence] with the position of each token in its document.

cu_seqlens
torch.Tensor

Int32 cumulative document lengths of the row; see cudnn_dsa_core.

max_seqlen
int

Length of the document, equal to sequence. Not read with metadata.

metadata
CudnnDsaPackedMetadata | None

Layout of a packed row and of the queries in it, or None for an unpadded row that holds one document.

Returns: torch.Tensor

Output of shape [1, sequence, hidden].

Raises:

  • ValueError: If hidden is not [1, sequence, hidden] or position_ids is not [1, sequence].
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAAttention._forward_eager(
hidden: torch.Tensor,
document_ids: torch.Tensor,
position_ids: torch.Tensor
) -> torch.Tensor

Run the eager path over a dense [sequence, sequence] selection.

Parameters:

hidden
torch.Tensor

Tensor of shape [batch, sequence, hidden], with at most max_eager_sequence tokens per row.

document_ids
torch.Tensor

Integer tensor of shape [batch, sequence] that identifies each contiguous document of a row; negative ids are padding.

position_ids
torch.Tensor

Integer tensor of shape [batch, sequence]; positions restart for each document.

Returns: torch.Tensor

Output of shape [batch, sequence, hidden].

Raises:

  • ValueError: If hidden is not rank 3, the ids do not have shape [batch, sequence], or the sequence is empty or longer than max_eager_sequence.
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAAttention._project(
hidden: torch.Tensor,
position_ids: torch.Tensor
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]

Project hidden states to the attention and indexer inputs.

Parameters:

hidden
torch.Tensor

Tensor of shape [batch, sequence, hidden].

position_ids
torch.Tensor

Integer tensor of shape [batch, sequence].

Returns: torch.Tensor

The query latent [batch, sequence, q_rank], the attention query

nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAAttention.forward(
hidden: torch.Tensor,
position_ids: torch.Tensor,
document_ids: torch.Tensor | None = None,
cu_seqlens: torch.Tensor | None = None,
max_seqlen: int | None = None,
) -> tuple[torch.Tensor, None]

Return the attention output.

The eager path takes document_ids; the cuDNN path takes cu_seqlens and max_seqlen, and metadata for a packed row. Passing the metadata of the other path raises. On the cuDNN path a row that holds several documents without metadata is rejected here, before the projections run; cudnn_dsa_core describes the remaining checks of cu_seqlens and max_seqlen.

Parameters:

hidden
torch.Tensor

Tensor of shape [batch, sequence, hidden]. The cuDNN path requires batch 1. Under context parallelism, sequence is this rank’s shard of the row.

position_ids
torch.Tensor

Integer tensor of shape [batch, sequence]; positions restart for each document.

document_ids
torch.Tensor | NoneDefaults to None

Eager path only. Integer tensor of shape [batch, sequence] that identifies each contiguous document of a row; negative ids are padding.

cu_seqlens
torch.Tensor | NoneDefaults to None

cuDNN path only. Int32 cumulative document lengths of the row: [0, sequence] of shape [2] for an unpadded row that holds one document, and of shape [documents + 1] with metadata.

max_seqlen
int | NoneDefaults to None

cuDNN path only. Length of the document, equal to sequence. Not read with metadata.

metadata
CudnnDsaPackedMetadata | NoneDefaults to None

cuDNN path only. Layout of a packed row and of the queries in it, as cudnn_dsa_layout returns it.

Returns: torch.Tensor

Output of shape [batch, sequence, hidden], and None in the place where

Raises:

  • ValueError: If shapes disagree, the metadata does not match the path, or the eager path receives no token or more than max_eager_sequence tokens.
  • NotImplementedError: If cu_seqlens has more than two entries without metadata.
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAAttention.init_weights(
init_std: float = 0.02
) -> None

Initialize all parameters, including after meta-device materialization.

The projections, the two separate K/V parameters and vha_postmix_U are drawn from a normal distribution, the norms are reset, and vha_postmix_V and the sink logits are set to zero.

Parameters:

init_std
floatDefaults to 0.02

Positive finite standard deviation of the normal draws.

Raises:

  • ValueError: If init_std is not a positive finite number. No parameter is changed in that case.
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAAttention.setup_cp_attention(
cp_mesh: torch.distributed.device_mesh.DeviceMesh
) -> None

Record the context-parallel group whose ranks hold consecutive shards of a row.

Parameters:

cp_mesh
DeviceMesh

Context-parallel device mesh offered by the parallelizer.

class nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAFP32Parameters(
num_heads: int,
vha_rank: int
)

Bases: Module

Hold the FP32 sink and virtual-head parameters of a DSA layer and run the eager core with them.

The three parameters form one module so that a dtype-aware FSDP wrap can treat them as one FP32 unit.

Parameters, all FP32: softmax_offset of shape [heads], the per-head sink logits, initialized to zeros; vha_postmix_U of shape [heads, rank], allocated without initialization and filled by the owning layer; vha_postmix_V of shape [heads, rank], initialized to zeros.

softmax_offset
vha_postmix_U
vha_postmix_V
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAFP32Parameters.forward(
query: torch.Tensor,
latent_kv: torch.Tensor,
scores: torch.Tensor,
document_ids: torch.Tensor,
value_up: torch.Tensor,
gate_logits: torch.Tensor,
window: int,
remote_topk: int,
softmax_scale: float,
key_document_ids: torch.Tensor | None = None,
first_query: int = 0
) -> torch.Tensor

Run latent_mqa_core with the parameters of this module.

The core runs inside this module’s forward because returning the parameters themselves would allow an FSDP post-forward reshard to free their storage before the caller uses them. latent_mqa_core validates the inputs and raises ValueError for shapes that disagree.

Parameters:

query
torch.Tensor

Tensor of shape [batch, sequence, heads, latent + rope], the queries of this rank.

latent_kv
torch.Tensor

Tensor of shape [batch, keys, latent + rope] shared by all heads: the keys of the whole row in row order. keys equals sequence without context parallelism.

scores
torch.Tensor

Finite index scores of shape [batch, sequence, keys], indexed [batch, query, key].

document_ids
torch.Tensor

Integer tensor of shape [batch, sequence]; negative ids are padding.

value_up
torch.Tensor

Value up-projection of shape [heads, value_dim, latent].

gate_logits
torch.Tensor

Tensor of shape [batch, sequence, heads * value_dim].

window
int

Positive number of local causal keys per query.

remote_topk
int

Non-negative number of remote keys per query.

softmax_scale
float

Factor applied to the query-key products.

key_document_ids
torch.Tensor | NoneDefaults to None

Optional integer tensor of shape [batch, keys]; see select_window_and_remote.

first_query
intDefaults to 0

Index of the first query among the keys.

Returns: torch.Tensor

Core output of shape [batch, sequence, heads * value_dim] in the dtype of query.

Raises:

  • RuntimeError: If one of the three parameters of this module is not FP32.
class nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSASparseFP32Parameters()

Bases: Ernie5_5DSAFP32Parameters

Hold the same FP32 parameters as the parent class and run the cuDNN core with them.

The core runs inside this module for the reason given in Ernie5_5DSAFP32Parameters.forward.

nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSASparseFP32Parameters.forward(
query: torch.Tensor,
latent_kv: torch.Tensor,
index_q: torch.Tensor,
index_k: torch.Tensor,
index_weights: torch.Tensor,
value_up: torch.Tensor,
gate_logits: torch.Tensor,
cu_seqlens: torch.Tensor,
max_seqlen: int,
window: int,
remote_topk: int,
softmax_scale: float,
) -> torch.Tensor

Run cudnn_dsa_core with the parameters of this module.

cudnn_dsa_core validates the inputs: it raises ValueError for shapes that disagree and NotImplementedError for a row that holds several documents without metadata.

Parameters:

query
torch.Tensor

BF16 tensor of shape [tokens, heads, latent + rope], the queries of this rank.

latent_kv
torch.Tensor

BF16 tensor of shape [keys, latent + rope] shared by all heads: the keys of the whole row in row order. keys equals tokens without context parallelism.

index_q
torch.Tensor

BF16 indexer query of shape [tokens, index_heads, index_dim].

index_k
torch.Tensor

BF16 indexer key of shape [keys, index_dim], in the order of latent_kv.

index_weights
torch.Tensor

Indexer head weights of shape [tokens, index_heads].

value_up
torch.Tensor

Value up-projection of shape [heads, value_dim, latent].

gate_logits
torch.Tensor

Tensor of shape [tokens, heads * value_dim].

cu_seqlens
torch.Tensor

Int32 cumulative document lengths of the row; see cudnn_dsa_core.

max_seqlen
int

Length of the document, equal to tokens. Not read with metadata.

window
int

Positive number of local causal keys per query.

remote_topk
int

Positive number of remote keys the indexer selects per query.

softmax_scale
float

Factor applied to the query-key products.

metadata
CudnnDsaPackedMetadata | NoneDefaults to None

Layout of a packed row and of the queries in it, as cudnn_dsa_layout returns it.

Returns: torch.Tensor

Core output of shape [tokens, heads * value_dim] in the dtype of query.

Raises:

  • RuntimeError: If one of the three parameters of this module is not FP32.
class nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5Indexer(
hidden_size: int,
q_rank: int,
num_heads: int,
head_dim: int,
dtype: torch.dtype,
norm_eps: float = 1e-06
)

Bases: Module

Project hidden states and the query latent to the indexer query, key and per-head weights.

The outputs feed the index scores, which only decide which remote keys the attention reads. Both inputs are detached, so no gradient reaches the rest of the model through the indexer.

Parameters: wq_b.weight of shape [heads * head_dim, q_rank], wk.weight of shape [head_dim, hidden] and weights_proj.weight of shape [heads, hidden] in the configured dtype, and k_norm.weight and k_norm.bias of shape [head_dim], which must stay FP32.

k_norm
weights_proj
wk
wq_b
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5Indexer.forward(
hidden: torch.Tensor,
q_latent: torch.Tensor,
rope: collections.abc.Callable[[torch.Tensor], torch.Tensor]
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]

Project and rotate the indexer inputs without propagating gradients into the backbone.

Parameters:

hidden
torch.Tensor

Tensor of shape [batch, sequence, hidden]. It is detached before the projections.

q_latent
torch.Tensor

Tensor of shape [batch, sequence, q_rank]. It is detached before the projection.

rope
Callable[[torch.Tensor], torch.Tensor]

Position rotation. It is called with the query of shape [batch, sequence, heads, head_dim] and with the key of shape [batch, sequence, 1, head_dim], and must return a tensor of the same shape and dtype. Each result is then rotated by rotate_indexer_activation.

Returns: torch.Tensor

A tuple of the rotated query of shape [batch, sequence, heads, head_dim], the rotated key of shape

Raises:

  • RuntimeError: If the weight or the bias of the key LayerNorm is not FP32.
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5Indexer.init_weights(
init_std: float = 0.02
) -> None

Initialize the projections and the key norm, including after meta-device materialization.

Parameters:

init_std
floatDefaults to 0.02

Positive finite standard deviation of the projection weights.

Raises:

  • ValueError: If init_std is not a positive finite number.
nemo_automodel.components.models.ernie5_5.dsa._selection_of_this_call(
select: collections.abc.Callable[[], torch.Tensor]
) -> torch.Tensor

Return the key selection of select, or the one that the forward of this checkpointed layer call recorded.

Two calls can select different keys among equal index scores, so the recomputation of a checkpointed decoder layer reuses the selection of its forward. Ernie5_5Block.nemo_checkpoint_context_fn binds the record of one call; outside a checkpointed call select runs unchanged.

Parameters:

select
Callable[[], torch.Tensor]

Computes the selection without gradient tracking: a boolean tensor of shape [batch, queries, keys] on the eager path, or int32 row offsets of shape [queries, remote_topk] on the cuDNN path.

Returns: torch.Tensor

The selection of the forward of this call.

nemo_automodel.components.models.ernie5_5.dsa.apply_indexer_rope(
x: torch.Tensor,
frequencies: torch.Tensor
) -> torch.Tensor

Rotate the leading channels of every head in the half-split rotary layout.

The first rope channels of a head are rotated, pairing channel i with channel i + rope / 2; the remaining channels pass through. The cosine and sine are computed in the dtype of frequencies and then cast to the dtype of x.

Parameters:

x
torch.Tensor

Tensor of shape [batch, sequence, heads, channels].

frequencies
torch.Tensor

Rotation angles in radians, of shape [batch, sequence, rope], on the device of x. rope is the number of rotated channels: positive, even and at most channels. Entries i and i + rope / 2 are the angle of one channel pair and are expected to be equal; this is not checked.

Returns: torch.Tensor

Newly allocated tensor with the shape and dtype of x: the first rope channels rotated, the other

Raises:

  • ValueError: If x is not rank 4, frequencies is not rank 3, their batch and sequence dimensions differ, or rope is zero, odd or larger than channels.
nemo_automodel.components.models.ernie5_5.dsa.apply_main_rope(
x: torch.Tensor,
frequencies: torch.Tensor
) -> torch.Tensor

Rotate adjacent channel pairs and return them in the half-split layout.

The input pairs channel 2i with channel 2i + 1. The output holds the rotated even channels in its first half and the rotated odd channels in its second half, which is the layout of apply_indexer_rope.

Parameters:

x
torch.Tensor

Tensor of shape [batch, sequence, heads, rope], with rope even. Every channel is rotated.

frequencies
torch.Tensor

Rotation angles in radians, of shape [batch, sequence, rope], on the device of x; see apply_indexer_rope.

Returns: torch.Tensor

Newly allocated tensor with the shape and dtype of x, in the half-split layout.

Raises:

  • ValueError: If x is not rank 4, frequencies is not rank 3, their final dimensions differ or are odd or zero, or their batch and sequence dimensions differ.
nemo_automodel.components.models.ernie5_5.dsa.cudnn_dsa_core(
query: torch.Tensor,
latent_kv: torch.Tensor,
index_q: torch.Tensor,
index_k: torch.Tensor,
index_weights: torch.Tensor,
sink: torch.Tensor,
value_up: torch.Tensor,
gate_logits: torch.Tensor,
vha_u: torch.Tensor,
vha_v: torch.Tensor,
cu_seqlens: torch.Tensor,
max_seqlen: int,
window: int,
remote_topk: int,
softmax_scale: float,
) -> torch.Tensor

Run the cuDNN DSA core on the tokens of one row.

cudnn_indexer_topk selects the remote keys of every query, merge_remote_and_window adds the local window, and cudnn_sparse_attention attends over the union with the per-head sink. The attended latent is then projected to the value channels of each head in FP32, mixed across heads and gated in the dtype of query, as in latent_mqa_core. The indexer inputs are detached, so the selection receives no gradient.

Without metadata the row is unpadded and holds one document, and queries and keys are the same tokens rows: cu_seqlens with more than two entries raises NotImplementedError, and before the first kernel is called prepare_cudnn_dsa_packed_metadata validates the rest of the document metadata: it raises TypeError unless cu_seqlens is int32, and ValueError unless cu_seqlens is one-dimensional, starts at zero and ends at tokens and max_seqlen equals tokens. With the metadata of cudnn_dsa_layout the row is packed, the keys are all of its tokens and the queries are all of them or one context-parallel shard.

All tensors must be on one CUDA device. The two kernels accept BF16 inputs with latent + rope equal to 512 or 576, index_dim equal to 128, 32 or 64 indexer heads and remote_topk of at most 2048, and raise otherwise.

Parameters:

query
torch.Tensor

BF16 tensor of shape [tokens, heads, latent + rope].

latent_kv
torch.Tensor

BF16 tensor of shape [keys, latent + rope], the key shared by all heads. Its first latent channels are also the value. keys equals tokens unless metadata describes a shard.

index_q
torch.Tensor

BF16 indexer query of shape [tokens, index_heads, index_dim].

index_k
torch.Tensor

BF16 indexer key of shape [keys, index_dim].

index_weights
torch.Tensor

FP32 or BF16 indexer head weights of shape [tokens, index_heads], as Ernie5_5Indexer returns them. They are multiplied by sqrt(index_dim) in their own dtype before the kernel.

sink
torch.Tensor

FP32 per-head sink logits of shape [heads].

value_up
torch.Tensor

Value up-projection of shape [heads, value_dim, latent].

gate_logits
torch.Tensor

Tensor of shape [tokens, heads * value_dim].

vha_u
torch.Tensor

Virtual-head input factor of shape [heads, rank].

vha_v
torch.Tensor

Virtual-head output factor of shape [heads, rank].

cu_seqlens
torch.Tensor

Int32 cumulative document lengths of the row: [0, tokens] of shape [2] without metadata, of shape [documents + 1] with it.

max_seqlen
int

Length of the document, equal to tokens. Not read with metadata.

window
int

Positive number of local causal keys per query.

remote_topk
int

Positive number of remote keys the indexer selects per query.

softmax_scale
float

Factor applied to the query-key products.

metadata
CudnnDsaPackedMetadata | NoneDefaults to None

Layout of a packed row and of the queries in it, as cudnn_dsa_layout returns it.

Returns: torch.Tensor

Core output of shape [tokens, heads * value_dim] in the dtype of query.

Raises:

  • ValueError: If tokens is zero, the shapes of the query, the latent key/value, the value projection or the gate logits disagree, latent exceeds latent + rope, or window or remote_topk is not positive.
  • NotImplementedError: If cu_seqlens has more than two entries without metadata.
nemo_automodel.components.models.ernie5_5.dsa.cudnn_dsa_layout(
document_ids: torch.Tensor,
first_query: int,
queries: int

Describe the documents of a packed row, and the queries in it, for the cuDNN DSA core.

Parameters:

document_ids
torch.Tensor

Integer tensor of shape [row] on a CUDA device with the document of every token of the row. The ids increase along the row, a negative id marks padding, the row starts with a document, and the padding of a document follows its tokens.

first_query
int

Row offset of the first query.

queries
int

Number of queries: the whole row, or one context-parallel shard of it.

Returns: tuple[torch.Tensor, CudnnDsaPackedMetadata]

The int32 cumulative document lengths of shape [documents + 1], and the metadata of the queries.

nemo_automodel.components.models.ernie5_5.dsa.ernie5_5_index_scores(
q: torch.Tensor,
k: torch.Tensor,
weights: torch.Tensor
) -> torch.Tensor

Score every key for every query with the ERNIE scaling order.

The score of a query and a key is the sum over heads of relu(<q_head, k>) * weight_head. The products and the sum are computed in FP32 with autocast disabled. The weights are multiplied by sqrt(channels) in their own dtype before the sum, and the sum is divided by sqrt(channels) afterwards; with BF16 weights this rounding order affects which keys a top-k selects.

The scores are not masked. The caller applies the causal and document limits and selects the top-k, as select_window_and_remote does.

Parameters:

q
torch.Tensor

Indexer query of shape [batch, queries, heads, channels].

k
torch.Tensor

Indexer key of shape [batch, keys, channels], shared by all heads.

weights
torch.Tensor

Per-head weights of shape [batch, queries, heads].

Returns: torch.Tensor

FP32 scores of shape [batch, queries, keys].

Raises:

  • ValueError: If q is not rank 4, k is not rank 3, weights is not [batch, queries, heads], or the batch or channel dimensions of q and k differ.
nemo_automodel.components.models.ernie5_5.dsa.freeze_ernie5_5_indexer_params(
model: torch.nn.Module
) -> None

Set requires_grad to False on every parameter of every Ernie5_5Indexer in a model.

The indexer only selects keys and receives no gradient from the attention output, so its parameters are kept out of the trainable set. The call changes the flags once and installs no hook: code that enables gradients afterwards, such as a trainability policy or model.requires_grad_(True), must call this function again.

Parameters:

model
nn.Module

Model, wrapper or pipeline stage that may contain ERNIE 5.5 DSA indexers. Parameters outside an Ernie5_5Indexer are left unchanged.

nemo_automodel.components.models.ernie5_5.dsa.latent_mqa_core(
query: torch.Tensor,
latent_kv: torch.Tensor,
scores: torch.Tensor,
document_ids: torch.Tensor,
sink: torch.Tensor,
value_up: torch.Tensor,
gate_logits: torch.Tensor,
vha_u: torch.Tensor,
vha_v: torch.Tensor,
window: int,
remote_topk: int,
softmax_scale: float,
key_document_ids: torch.Tensor | None = None,
first_query: int = 0
) -> torch.Tensor

Run the eager DSA core: latent multi-query attention with a sink, value up-projection, head mix and gate.

Every head attends with its own query to one latent key/value per token, over the keys that select_window_and_remote selects. A per-head sink logit takes part in the softmax and its probability is discarded, so a query without a selected key (padding) produces zeros. The attended latent is projected to the value channels of each head, mixed across heads with virtual_head_mix and multiplied by the sigmoid of the gate logits.

The logits, the softmax and the value up-projection are computed in FP32 with autocast disabled, whatever the input dtypes; the head mix and the gate run in the dtype of query. The result is not guaranteed to equal the result of cudnn_dsa_core bit for bit.

Parameters:

query
torch.Tensor

Tensor of shape [batch, sequence, heads, latent + rope]: the query of each head in the key space, latent content channels followed by rope rotary channels.

latent_kv
torch.Tensor

Tensor of shape [batch, keys, latent + rope], the key shared by all heads. Its first latent channels are also the value. keys equals sequence unless the queries are one context-parallel shard of the row.

scores
torch.Tensor

Finite index scores of shape [batch, sequence, keys], indexed [batch, query, key].

document_ids
torch.Tensor

Integer tensor of shape [batch, sequence]; see select_window_and_remote.

sink
torch.Tensor

Per-head sink logits of shape [heads].

value_up
torch.Tensor

Value up-projection of shape [heads, value_dim, latent].

gate_logits
torch.Tensor

Tensor of shape [batch, sequence, heads * value_dim].

vha_u
torch.Tensor

Virtual-head input factor of shape [heads, rank].

vha_v
torch.Tensor

Virtual-head output factor of shape [heads, rank].

window
int

Positive number of local keys per query.

remote_topk
int

Non-negative number of remote keys per query.

softmax_scale
float

Factor applied to the query-key products.

key_document_ids
torch.Tensor | NoneDefaults to None

Optional integer tensor of shape [batch, keys]; see select_window_and_remote.

first_query
intDefaults to 0

Index of the first query among the keys.

Returns: torch.Tensor

Tensor of shape [batch, sequence, heads * value_dim] in the dtype of query, with the value channels of

Raises:

  • ValueError: If a rank or a dimension disagrees with the shapes above, or select_window_and_remote rejects the scores, the document ids, window or remote_topk.
nemo_automodel.components.models.ernie5_5.dsa.merge_remote_and_window(
remote_indices: torch.Tensor,
document_starts: torch.Tensor,
causal_lengths: torch.Tensor,
query_valid: torch.Tensor,
window: int
) -> tuple[torch.Tensor, torch.Tensor]

Combine the remote indices of every query with its local causal window.

Keys are the tokens of one row, queries are all of them or one context-parallel shard, and an index is the offset of a token in that row. The remote part of the result keeps the input order and replaces with -1 every index that is absent, belongs to another document, or lies inside or after the local window. The local part lists the last window offsets up to the query in ascending order, with -1 for positions before the start of the document.

The real tokens of a document must be contiguous from its start. The values of the inputs are not checked for consistency.

Parameters:

remote_indices
torch.Tensor

Int32 tensor of shape [queries, remote_topk] with row offsets of remote keys, or -1 for absent entries.

document_starts
torch.Tensor

Int32 tensor of shape [queries] with the row offset of the first token of each query’s document.

causal_lengths
torch.Tensor

Int32 tensor of shape [queries] counting the tokens from the start of the document through the query.

query_valid
torch.Tensor

Boolean tensor of shape [queries]. A False entry gives a row of -1.

window
int

Positive number of local causal positions per query.

Returns: torch.Tensor

The int32 indices of shape [queries, remote_topk + window], remote part first, with invalid entries set to

Raises:

  • ValueError: If window is not a positive integer, remote_indices is not an int32 matrix, a metadata tensor does not have shape [queries], the starts or the lengths are not int32, the validity is not boolean, or the tensors are on different devices.
nemo_automodel.components.models.ernie5_5.dsa.rotate_indexer_activation(
x: torch.Tensor
) -> torch.Tensor

Apply the scaled Hadamard rotation of the DSA indexer.

Precision contract: the input, of any floating-point dtype, is rounded to BF16; the butterfly and the scaling by channels ** -0.5 run in FP32; the result is rounded to BF16 once. On a CUDA device the fast_hadamard_transform kernel computes this when the package is installed. Without the package, and on every other device, the same arithmetic runs in PyTorch, so both paths return the same bits.

Parameters:

x
torch.Tensor

Floating-point tensor of shape […, channels], with arbitrary leading dimensions; channels is a power of two.

Returns: torch.Tensor

BF16 tensor of shape […, channels] on the device of x.

Raises:

  • ValueError: If channels is not a power of two.
nemo_automodel.components.models.ernie5_5.dsa.select_window_and_remote(
scores: torch.Tensor,
document_ids: torch.Tensor,
window: int,
remote_topk: int,
key_document_ids: torch.Tensor | None = None,
first_query: int = 0
) -> torch.Tensor

Select, for every query, a causal local window and the top-scoring remote keys of its document.

A key is eligible for a query when both are real tokens of the same document and the key is not after the query. The local window is the query and the window - 1 eligible keys before it. The other eligible keys are remote, and the remote_topk highest-scoring of them are added. The function runs without gradient tracking.

The queries are the keys, or, with key_document_ids, the consecutive keys that start at first_query: one context-parallel shard of the row.

Parameters:

scores
torch.Tensor

Finite floating tensor of shape [batch, queries, keys], indexed [batch, query, key].

document_ids
torch.Tensor

Integer tensor of shape [batch, queries] on the device of scores. Tokens of one document carry the same id, and a negative id marks padding. The caller must give every contiguous document of a row its own id: tokens that share an id are treated as one document.

window
int

Positive number of local keys per query, the query included.

remote_topk
int

Non-negative number of remote keys per query. Zero selects the local window only.

key_document_ids
torch.Tensor | NoneDefaults to None

Optional integer tensor of shape [batch, keys] with the document of every key. None means that the keys are the queries.

first_query
intDefaults to 0

Index of the first query among the keys.

Returns: torch.Tensor

Boolean tensor of shape [batch, queries, keys], indexed [batch, query, key], that is True for the

Raises:

  • ValueError: If scores is not [batch, queries, keys], document_ids is not [batch, queries], the key ids are not [batch, keys], window is not positive, remote_topk is negative or a score is not finite.
nemo_automodel.components.models.ernie5_5.dsa.dsa_selection_replay: RecomputeReplay[Tensor] = RecomputeReplay('ERNIE 5.5 DSA key selection')