> This page is for version 0.4.0 · 26.04.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.models.ernie5_5.dsa

ERNIE 5.5 dynamic sparse attention (DSA).

A DSA layer attends, for every query, to a causal local window of the query's document and to the remote keys
of the same document that an indexer selects. This module holds the layer and its parts:

* `apply_indexer_rope` and `apply_main_rope`: the two rotary channel layouts of a DSA layer.
* `rotate_indexer_activation`: the scaled Hadamard rotation of the indexer query and key.
* `Ernie5_5Indexer`: the projections that produce the indexer query, key and per-head weights.
* `ernie5_5_index_scores` and `select_window_and_remote`: the index scores and the dense key selection of the
  eager path.
* `latent_mqa_core`: the eager attention core over a dense \[sequence, sequence] selection.
* `merge_remote_and_window` and `cudnn_dsa_core`: the sparse key indices and the attention core of the cuDNN path.
* `cudnn_dsa_layout`: the document layout of a packed row, or of a context-parallel shard of it, for the cuDNN path.
* `Ernie5_5DSAFP32Parameters` and `Ernie5_5DSASparseFP32Parameters`: the FP32 sink and virtual-head parameters,
  each with the core that consumes them.
* `Ernie5_5DSAAttention`: the layer. `backend.attn` selects the eager path for short sequences or the cuDNN
  path; both paths share all parameters, and both attend within each document of a packed row and across the
  context-parallel shards of a row.
* `freeze_ernie5_5_indexer_params`: disables gradients for the indexer parameters.

Provenance: the projection layout, the indexer precision contract and the operation order follow two sources.

* The ERNIE 5.5 DSA-layer attention of Megatron-Bridge (Apache License 2.0) at revision
  `7c55987a027084e87e36277c9711e87bc52a2116`:
  [https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/7c55987a027084e87e36277c9711e87bc52a2116/src/megatron/bridge/models/ernie/modeling_ernie55/latent_mqa_attention.py](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/7c55987a027084e87e36277c9711e87bc52a2116/src/megatron/bridge/models/ernie/modeling_ernie55/latent_mqa_attention.py)
* The DSA of Megatron-LM (3-clause BSD terms of the repository `LICENSE` file; its copyright notices, conditions
  and disclaimer are retained in the comment block above) at revision
  `f78809f6d9c88f8638ea7822966f35057a63fd5c`:
  [https://github.com/NVIDIA/Megatron-LM/blob/f78809f6d9c88f8638ea7822966f35057a63fd5c/megatron/core/transformer/experimental_attention_variant/dsa.py](https://github.com/NVIDIA/Megatron-LM/blob/f78809f6d9c88f8638ea7822966f35057a63fd5c/megatron/core/transformer/experimental_attention_variant/dsa.py)

## Module Contents

### Classes

| Name                                                                                                                | Description                                                                                     |
| ------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| [`Ernie5_5DSAAttention`](#nemo_automodel-components-models-ernie5_5-dsa-Ernie5_5DSAAttention)                       | ERNIE 5.5 dynamic sparse attention (DSA) layer.                                                 |
| [`Ernie5_5DSAFP32Parameters`](#nemo_automodel-components-models-ernie5_5-dsa-Ernie5_5DSAFP32Parameters)             | Hold the FP32 sink and virtual-head parameters of a DSA layer and run the eager core with them. |
| [`Ernie5_5DSASparseFP32Parameters`](#nemo_automodel-components-models-ernie5_5-dsa-Ernie5_5DSASparseFP32Parameters) | Hold the same FP32 parameters as the parent class and run the cuDNN core with them.             |
| [`Ernie5_5Indexer`](#nemo_automodel-components-models-ernie5_5-dsa-Ernie5_5Indexer)                                 | Project hidden states and the query latent to the indexer query, key and per-head weights.      |

### Functions

| Name                                                                                                              | Description                                                                                                 |
| ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| [`_selection_of_this_call`](#nemo_automodel-components-models-ernie5_5-dsa-_selection_of_this_call)               | Return the key selection of `select`, or the one that the forward of this checkpointed layer call recorded. |
| [`apply_indexer_rope`](#nemo_automodel-components-models-ernie5_5-dsa-apply_indexer_rope)                         | Rotate the leading channels of every head in the half-split rotary layout.                                  |
| [`apply_main_rope`](#nemo_automodel-components-models-ernie5_5-dsa-apply_main_rope)                               | Rotate adjacent channel pairs and return them in the half-split layout.                                     |
| [`cudnn_dsa_core`](#nemo_automodel-components-models-ernie5_5-dsa-cudnn_dsa_core)                                 | Run the cuDNN DSA core on the tokens of one row.                                                            |
| [`cudnn_dsa_layout`](#nemo_automodel-components-models-ernie5_5-dsa-cudnn_dsa_layout)                             | Describe the documents of a packed row, and the queries in it, for the cuDNN DSA core.                      |
| [`ernie5_5_index_scores`](#nemo_automodel-components-models-ernie5_5-dsa-ernie5_5_index_scores)                   | Score every key for every query with the ERNIE scaling order.                                               |
| [`freeze_ernie5_5_indexer_params`](#nemo_automodel-components-models-ernie5_5-dsa-freeze_ernie5_5_indexer_params) | Set `requires_grad` to False on every parameter of every `Ernie5_5Indexer` in a model.                      |
| [`latent_mqa_core`](#nemo_automodel-components-models-ernie5_5-dsa-latent_mqa_core)                               | Run the eager DSA core: latent multi-query attention with a sink, value up-projection, head mix and gate.   |
| [`merge_remote_and_window`](#nemo_automodel-components-models-ernie5_5-dsa-merge_remote_and_window)               | Combine the remote indices of every query with its local causal window.                                     |
| [`rotate_indexer_activation`](#nemo_automodel-components-models-ernie5_5-dsa-rotate_indexer_activation)           | Apply the scaled Hadamard rotation of the DSA indexer.                                                      |
| [`select_window_and_remote`](#nemo_automodel-components-models-ernie5_5-dsa-select_window_and_remote)             | Select, for every query, a causal local window and the top-scoring remote keys of its document.             |

### Data

[`dsa_selection_replay`](#nemo_automodel-components-models-ernie5_5-dsa-dsa_selection_replay)

### API

```python
class nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAAttention(
    hidden_size: int,
    num_heads: int,
    q_rank: int,
    kv_rank: int,
    nope_dim: int,
    rope_dim: int,
    value_dim: int,
    index_heads: int,
    index_dim: int,
    vha_rank: int,
    window: int,
    remote_topk: int,
    dtype: torch.dtype,
    rope_theta: float = 8000000.0,
    norm_eps: float = 1e-06,
    max_eager_sequence: int = 512,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None
)
```

**Bases:** `Module`

ERNIE 5.5 dynamic sparse attention (DSA) layer.

Latent multi-query attention over a causal local window plus the remote keys
an indexer selects, followed by the value up-projection, virtual-head mixing,
the query gate and the output projection.

The attention backend selects one of two paths that share all parameters:

* `backend.attn == "cudnn"`: cuDNN top-k selection and sparse attention for
  one row: an unpadded document, or a packed row with the metadata of
  `cudnn_dsa_layout`. The forward runs with autocast disabled, and the
  kernels require BF16 inputs and the released latent and indexer dimensions.
* `backend.attn` in `("eager", "torch", "sdpa")`, or no backend: eager
  PyTorch attention over a dense \[sequence, sequence] mask, limited to
  `max_eager_sequence` tokens. The three names select this same path; no
  scaled-dot-product kernel is called.

Every other value of `backend.attn` is rejected.

Under context parallelism a rank holds one contiguous shard of the row as
queries and gathers the latent and indexer keys of all ranks. Neither path
supports a KV cache.

The indexer only selects keys. Its projections receive no gradient from the
attention output, and this layer computes no indexer loss, so the indexer is
not trained here; `freeze_ernie5_5_indexer_params` keeps it frozen. This
differs from the reference training recipe, which trains the indexer with a
KL term against the attention distribution.

**`cp_group`** `ProcessGroup | None = None`

---

**`fp32_params`** `= holder(num_heads, vha_rank)`

---

**`gate_proj`**

---

**`indexer`**

---

**`k_b_proj`**

---

**`kv_a_layernorm`** `= nn.RMSNorm(kv_rank, eps=norm_eps, dtype=dtype)`

---

**`kv_a_proj_with_mqa`**

---

**`o_proj`**

---

**`q_a_layernorm`** `= nn.RMSNorm(q_rank, eps=norm_eps, dtype=dtype)`

---

**`q_a_proj`**

---

**`q_b_proj`**

---

**`softmax_offset`** `Tensor`

FP32 per-head sink logits of shape \[heads], owned by `fp32_params`.

---

**`softmax_scale`** `= (nope_dim + rope_dim) ** -0.5`

---

**`uses_cudnn`** `= attn == 'cudnn'`

---

**`v_b_proj`**

---

**`vha_postmix_U`** `Tensor`

FP32 virtual-head input factor of shape \[heads, vha_rank], owned by `fp32_params`.

---

**`vha_postmix_V`** `Tensor`

FP32 virtual-head output factor of shape \[heads, vha_rank], owned by `fp32_params`.

---

```python
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAAttention._forward_cudnn(
    hidden: torch.Tensor,
    position_ids: torch.Tensor,
    cu_seqlens: torch.Tensor,
    max_seqlen: int,
    metadata: nemo_automodel.components.models.glm_moe_dsa.kernels.cudnn_dsa.CudnnDsaPackedMetadata | None
) -> torch.Tensor
```

Run the cuDNN path on one row, or on this rank's shard of it.

**Parameters:**

**`hidden`** `torch.Tensor`

Tensor of shape \[1, sequence, hidden].

---

**`position_ids`** `torch.Tensor`

Integer tensor of shape \[1, sequence] with the position of each token in its document.

---

**`cu_seqlens`** `torch.Tensor`

Int32 cumulative document lengths of the row; see `cudnn_dsa_core`.

---

**`max_seqlen`** `int`

Length of the document, equal to `sequence`. Not read with `metadata`.

---

**`metadata`** `CudnnDsaPackedMetadata | None`

Layout of a packed row and of the queries in it, or None for an unpadded row that holds one
document.

---

**Returns:** `torch.Tensor`

Output of shape \[1, sequence, hidden].

**Raises:**

* `ValueError`: If `hidden` is not \[1, sequence, hidden] or `position_ids` is not \[1, sequence].

```python
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAAttention._forward_eager(
    hidden: torch.Tensor,
    document_ids: torch.Tensor,
    position_ids: torch.Tensor
) -> torch.Tensor
```

Run the eager path over a dense \[sequence, sequence] selection.

**Parameters:**

**`hidden`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden], with at most `max_eager_sequence` tokens per row.

---

**`document_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence] that identifies each contiguous document of a
row; negative ids are padding.

---

**`position_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence]; positions restart for each document.

---

**Returns:** `torch.Tensor`

Output of shape \[batch, sequence, hidden].

**Raises:**

* `ValueError`: If `hidden` is not rank 3, the ids do not have shape \[batch, sequence], or the sequence
  is empty or longer than `max_eager_sequence`.

```python
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAAttention._project(
    hidden: torch.Tensor,
    position_ids: torch.Tensor
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]
```

Project hidden states to the attention and indexer inputs.

**Parameters:**

**`hidden`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden].

---

**`position_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence].

---

**Returns:** `torch.Tensor`

The query latent \[batch, sequence, q_rank], the attention query

```python
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAAttention.forward(
    hidden: torch.Tensor,
    position_ids: torch.Tensor,
    document_ids: torch.Tensor | None = None,
    cu_seqlens: torch.Tensor | None = None,
    max_seqlen: int | None = None,
    metadata: nemo_automodel.components.models.glm_moe_dsa.kernels.cudnn_dsa.CudnnDsaPackedMetadata | None = None
) -> tuple[torch.Tensor, None]
```

Return the attention output.

The eager path takes `document_ids`; the cuDNN path takes `cu_seqlens`
and `max_seqlen`, and `metadata` for a packed row. Passing the metadata
of the other path raises. On the cuDNN path a row that holds several
documents without `metadata` is rejected here, before the projections run;
`cudnn_dsa_core` describes the remaining checks of `cu_seqlens` and
`max_seqlen`.

**Parameters:**

**`hidden`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden]. The cuDNN path requires batch 1. Under
context parallelism, `sequence` is this rank's shard of the row.

---

**`position_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence]; positions restart for each document.

---

**`document_ids`** `torch.Tensor | None` — default: None

Eager path only. Integer tensor of shape \[batch, sequence] that
identifies each contiguous document of a row; negative ids are padding.

---

**`cu_seqlens`** `torch.Tensor | None` — default: None

cuDNN path only. Int32 cumulative document lengths of the row: `[0, sequence]`
of shape \[2] for an unpadded row that holds one document, and of shape
\[documents + 1] with `metadata`.

---

**`max_seqlen`** `int | None` — default: None

cuDNN path only. Length of the document, equal to `sequence`. Not read with
`metadata`.

---

**`metadata`** `CudnnDsaPackedMetadata | None` — default: None

cuDNN path only. Layout of a packed row and of the queries in it, as
`cudnn_dsa_layout` returns it.

---

**Returns:** `torch.Tensor`

Output of shape \[batch, sequence, hidden], and None in the place where

**Raises:**

* `ValueError`: If shapes disagree, the metadata does not match the path, or
  the eager path receives no token or more than `max_eager_sequence` tokens.
* `NotImplementedError`: If `cu_seqlens` has more than two entries without `metadata`.

```python
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAAttention.init_weights(
    init_std: float = 0.02
) -> None
```

Initialize all parameters, including after meta-device materialization.

The projections, the two separate K/V parameters and `vha_postmix_U` are drawn from a normal
distribution, the norms are reset, and `vha_postmix_V` and the sink logits are set to zero.

**Parameters:**

**`init_std`** `float` — default: 0.02

Positive finite standard deviation of the normal draws.

---

**Raises:**

* `ValueError`: If `init_std` is not a positive finite number. No parameter is changed in that case.

```python
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAAttention.setup_cp_attention(
    cp_mesh: torch.distributed.device_mesh.DeviceMesh
) -> None
```

Record the context-parallel group whose ranks hold consecutive shards of a row.

**Parameters:**

**`cp_mesh`** `DeviceMesh`

Context-parallel device mesh offered by the parallelizer.

---

```python
class nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAFP32Parameters(
    num_heads: int,
    vha_rank: int
)
```

**Bases:** `Module`

Hold the FP32 sink and virtual-head parameters of a DSA layer and run the eager core with them.

The three parameters form one module so that a dtype-aware FSDP wrap can treat them as one FP32 unit.

Parameters, all FP32: `softmax_offset` of shape \[heads], the per-head sink logits, initialized to zeros;
`vha_postmix_U` of shape \[heads, rank], allocated without initialization and filled by the owning layer;
`vha_postmix_V` of shape \[heads, rank], initialized to zeros.

**`softmax_offset`**

---

**`vha_postmix_U`**

---

**`vha_postmix_V`**

---

```python
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSAFP32Parameters.forward(
    query: torch.Tensor,
    latent_kv: torch.Tensor,
    scores: torch.Tensor,
    document_ids: torch.Tensor,
    value_up: torch.Tensor,
    gate_logits: torch.Tensor,
    window: int,
    remote_topk: int,
    softmax_scale: float,
    key_document_ids: torch.Tensor | None = None,
    first_query: int = 0
) -> torch.Tensor
```

Run `latent_mqa_core` with the parameters of this module.

The core runs inside this module's forward because returning the parameters themselves would allow an FSDP
post-forward reshard to free their storage before the caller uses them. `latent_mqa_core` validates the
inputs and raises `ValueError` for shapes that disagree.

**Parameters:**

**`query`** `torch.Tensor`

Tensor of shape \[batch, sequence, heads, latent + rope], the queries of this rank.

---

**`latent_kv`** `torch.Tensor`

Tensor of shape \[batch, keys, latent + rope] shared by all heads: the keys of the
whole row in row order. `keys` equals `sequence` without context parallelism.

---

**`scores`** `torch.Tensor`

Finite index scores of shape \[batch, sequence, keys], indexed \[batch, query, key].

---

**`document_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence]; negative ids are padding.

---

**`value_up`** `torch.Tensor`

Value up-projection of shape \[heads, value_dim, latent].

---

**`gate_logits`** `torch.Tensor`

Tensor of shape \[batch, sequence, heads \* value_dim].

---

**`window`** `int`

Positive number of local causal keys per query.

---

**`remote_topk`** `int`

Non-negative number of remote keys per query.

---

**`softmax_scale`** `float`

Factor applied to the query-key products.

---

**`key_document_ids`** `torch.Tensor | None` — default: None

Optional integer tensor of shape \[batch, keys]; see `select_window_and_remote`.

---

**`first_query`** `int` — default: 0

Index of the first query among the keys.

---

**Returns:** `torch.Tensor`

Core output of shape \[batch, sequence, heads \* value_dim] in the dtype of `query`.

**Raises:**

* `RuntimeError`: If one of the three parameters of this module is not FP32.

```python
class nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSASparseFP32Parameters()
```

**Bases:** [Ernie5_5DSAFP32Parameters](#nemo_automodel-components-models-ernie5_5-dsa-Ernie5_5DSAFP32Parameters)

Hold the same FP32 parameters as the parent class and run the cuDNN core with them.

The core runs inside this module for the reason given in `Ernie5_5DSAFP32Parameters.forward`.

```python
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5DSASparseFP32Parameters.forward(
    query: torch.Tensor,
    latent_kv: torch.Tensor,
    index_q: torch.Tensor,
    index_k: torch.Tensor,
    index_weights: torch.Tensor,
    value_up: torch.Tensor,
    gate_logits: torch.Tensor,
    cu_seqlens: torch.Tensor,
    max_seqlen: int,
    window: int,
    remote_topk: int,
    softmax_scale: float,
    metadata: nemo_automodel.components.models.glm_moe_dsa.kernels.cudnn_dsa.CudnnDsaPackedMetadata | None = None
) -> torch.Tensor
```

Run `cudnn_dsa_core` with the parameters of this module.

`cudnn_dsa_core` validates the inputs: it raises `ValueError` for shapes that disagree and
`NotImplementedError` for a row that holds several documents without `metadata`.

**Parameters:**

**`query`** `torch.Tensor`

BF16 tensor of shape \[tokens, heads, latent + rope], the queries of this rank.

---

**`latent_kv`** `torch.Tensor`

BF16 tensor of shape \[keys, latent + rope] shared by all heads: the keys of the whole
row in row order. `keys` equals `tokens` without context parallelism.

---

**`index_q`** `torch.Tensor`

BF16 indexer query of shape \[tokens, index_heads, index_dim].

---

**`index_k`** `torch.Tensor`

BF16 indexer key of shape \[keys, index_dim], in the order of `latent_kv`.

---

**`index_weights`** `torch.Tensor`

Indexer head weights of shape \[tokens, index_heads].

---

**`value_up`** `torch.Tensor`

Value up-projection of shape \[heads, value_dim, latent].

---

**`gate_logits`** `torch.Tensor`

Tensor of shape \[tokens, heads \* value_dim].

---

**`cu_seqlens`** `torch.Tensor`

Int32 cumulative document lengths of the row; see `cudnn_dsa_core`.

---

**`max_seqlen`** `int`

Length of the document, equal to `tokens`. Not read with `metadata`.

---

**`window`** `int`

Positive number of local causal keys per query.

---

**`remote_topk`** `int`

Positive number of remote keys the indexer selects per query.

---

**`softmax_scale`** `float`

Factor applied to the query-key products.

---

**`metadata`** `CudnnDsaPackedMetadata | None` — default: None

Layout of a packed row and of the queries in it, as `cudnn_dsa_layout` returns it.

---

**Returns:** `torch.Tensor`

Core output of shape \[tokens, heads \* value_dim] in the dtype of `query`.

**Raises:**

* `RuntimeError`: If one of the three parameters of this module is not FP32.

```python
class nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5Indexer(
    hidden_size: int,
    q_rank: int,
    num_heads: int,
    head_dim: int,
    dtype: torch.dtype,
    norm_eps: float = 1e-06
)
```

**Bases:** `Module`

Project hidden states and the query latent to the indexer query, key and per-head weights.

The outputs feed the index scores, which only decide which remote keys the attention reads. Both inputs are
detached, so no gradient reaches the rest of the model through the indexer.

Parameters: `wq_b.weight` of shape \[heads \* head_dim, q_rank], `wk.weight` of shape \[head_dim, hidden] and
`weights_proj.weight` of shape \[heads, hidden] in the configured dtype, and `k_norm.weight` and
`k_norm.bias` of shape \[head_dim], which must stay FP32.

**`k_norm`**

---

**`weights_proj`**

---

**`wk`**

---

**`wq_b`**

---

```python
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5Indexer.forward(
    hidden: torch.Tensor,
    q_latent: torch.Tensor,
    rope: collections.abc.Callable[[torch.Tensor], torch.Tensor]
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]
```

Project and rotate the indexer inputs without propagating gradients into the backbone.

**Parameters:**

**`hidden`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden]. It is detached before the projections.

---

**`q_latent`** `torch.Tensor`

Tensor of shape \[batch, sequence, q_rank]. It is detached before the projection.

---

**`rope`** `Callable[[torch.Tensor], torch.Tensor]`

Position rotation. It is called with the query of shape \[batch, sequence, heads, head_dim] and with
the key of shape \[batch, sequence, 1, head_dim], and must return a tensor of the same shape and dtype.
Each result is then rotated by `rotate_indexer_activation`.

---

**Returns:** `torch.Tensor`

A tuple of the rotated query of shape \[batch, sequence, heads, head_dim], the rotated key of shape

**Raises:**

* `RuntimeError`: If the weight or the bias of the key LayerNorm is not FP32.

```python
nemo_automodel.components.models.ernie5_5.dsa.Ernie5_5Indexer.init_weights(
    init_std: float = 0.02
) -> None
```

Initialize the projections and the key norm, including after meta-device materialization.

**Parameters:**

**`init_std`** `float` — default: 0.02

Positive finite standard deviation of the projection weights.

---

**Raises:**

* `ValueError`: If `init_std` is not a positive finite number.

```python
nemo_automodel.components.models.ernie5_5.dsa._selection_of_this_call(
    select: collections.abc.Callable[[], torch.Tensor]
) -> torch.Tensor
```

Return the key selection of `select`, or the one that the forward of this checkpointed layer call recorded.

Two calls can select different keys among equal index scores, so the recomputation of a checkpointed decoder
layer reuses the selection of its forward. `Ernie5_5Block.nemo_checkpoint_context_fn` binds the record of one
call; outside a checkpointed call `select` runs unchanged.

**Parameters:**

**`select`** `Callable[[], torch.Tensor]`

Computes the selection without gradient tracking: a boolean tensor of shape \[batch, queries, keys]
on the eager path, or int32 row offsets of shape \[queries, remote_topk] on the cuDNN path.

---

**Returns:** `torch.Tensor`

The selection of the forward of this call.

```python
nemo_automodel.components.models.ernie5_5.dsa.apply_indexer_rope(
    x: torch.Tensor,
    frequencies: torch.Tensor
) -> torch.Tensor
```

Rotate the leading channels of every head in the half-split rotary layout.

The first `rope` channels of a head are rotated, pairing channel `i` with channel `i + rope / 2`; the
remaining channels pass through. The cosine and sine are computed in the dtype of `frequencies` and then cast
to the dtype of `x`.

**Parameters:**

**`x`** `torch.Tensor`

Tensor of shape \[batch, sequence, heads, channels].

---

**`frequencies`** `torch.Tensor`

Rotation angles in radians, of shape \[batch, sequence, rope], on the device of `x`. `rope` is
the number of rotated channels: positive, even and at most `channels`. Entries `i` and
`i + rope / 2` are the angle of one channel pair and are expected to be equal; this is not checked.

---

**Returns:** `torch.Tensor`

Newly allocated tensor with the shape and dtype of `x`: the first `rope` channels rotated, the other

**Raises:**

* `ValueError`: If `x` is not rank 4, `frequencies` is not rank 3, their batch and sequence dimensions differ,
  or `rope` is zero, odd or larger than `channels`.

```python
nemo_automodel.components.models.ernie5_5.dsa.apply_main_rope(
    x: torch.Tensor,
    frequencies: torch.Tensor
) -> torch.Tensor
```

Rotate adjacent channel pairs and return them in the half-split layout.

The input pairs channel `2i` with channel `2i + 1`. The output holds the rotated even channels in its first
half and the rotated odd channels in its second half, which is the layout of `apply_indexer_rope`.

**Parameters:**

**`x`** `torch.Tensor`

Tensor of shape \[batch, sequence, heads, rope], with `rope` even. Every channel is rotated.

---

**`frequencies`** `torch.Tensor`

Rotation angles in radians, of shape \[batch, sequence, rope], on the device of `x`; see
`apply_indexer_rope`.

---

**Returns:** `torch.Tensor`

Newly allocated tensor with the shape and dtype of `x`, in the half-split layout.

**Raises:**

* `ValueError`: If `x` is not rank 4, `frequencies` is not rank 3, their final dimensions differ or are
  odd or zero, or their batch and sequence dimensions differ.

```python
nemo_automodel.components.models.ernie5_5.dsa.cudnn_dsa_core(
    query: torch.Tensor,
    latent_kv: torch.Tensor,
    index_q: torch.Tensor,
    index_k: torch.Tensor,
    index_weights: torch.Tensor,
    sink: torch.Tensor,
    value_up: torch.Tensor,
    gate_logits: torch.Tensor,
    vha_u: torch.Tensor,
    vha_v: torch.Tensor,
    cu_seqlens: torch.Tensor,
    max_seqlen: int,
    window: int,
    remote_topk: int,
    softmax_scale: float,
    metadata: nemo_automodel.components.models.glm_moe_dsa.kernels.cudnn_dsa.CudnnDsaPackedMetadata | None = None
) -> torch.Tensor
```

Run the cuDNN DSA core on the tokens of one row.

`cudnn_indexer_topk` selects the remote keys of every query, `merge_remote_and_window` adds the local
window, and `cudnn_sparse_attention` attends over the union with the per-head sink. The attended latent is
then projected to the value channels of each head in FP32, mixed across heads and gated in the dtype of
`query`, as in `latent_mqa_core`. The indexer inputs are detached, so the selection receives no gradient.

Without `metadata` the row is unpadded and holds one document, and queries and keys are the same `tokens`
rows: `cu_seqlens` with more than two entries raises `NotImplementedError`, and before the first kernel is
called `prepare_cudnn_dsa_packed_metadata` validates the rest of the document metadata: it raises
`TypeError` unless `cu_seqlens` is int32, and `ValueError` unless `cu_seqlens` is one-dimensional,
starts at zero and ends at `tokens` and `max_seqlen` equals `tokens`. With the `metadata` of
`cudnn_dsa_layout` the row is packed, the keys are all of its tokens and the queries are all of them or one
context-parallel shard.

All tensors must be on one CUDA device. The two kernels accept BF16 inputs with `latent + rope` equal to 512
or 576, `index_dim` equal to 128, 32 or 64 indexer heads and `remote_topk` of at most 2048, and raise
otherwise.

**Parameters:**

**`query`** `torch.Tensor`

BF16 tensor of shape \[tokens, heads, latent + rope].

---

**`latent_kv`** `torch.Tensor`

BF16 tensor of shape \[keys, latent + rope], the key shared by all heads. Its first `latent`
channels are also the value. `keys` equals `tokens` unless `metadata` describes a shard.

---

**`index_q`** `torch.Tensor`

BF16 indexer query of shape \[tokens, index_heads, index_dim].

---

**`index_k`** `torch.Tensor`

BF16 indexer key of shape \[keys, index_dim].

---

**`index_weights`** `torch.Tensor`

FP32 or BF16 indexer head weights of shape \[tokens, index_heads], as `Ernie5_5Indexer`
returns them. They are multiplied by `sqrt(index_dim)` in their own dtype before the kernel.

---

**`sink`** `torch.Tensor`

FP32 per-head sink logits of shape \[heads].

---

**`value_up`** `torch.Tensor`

Value up-projection of shape \[heads, value_dim, latent].

---

**`gate_logits`** `torch.Tensor`

Tensor of shape \[tokens, heads \* value_dim].

---

**`vha_u`** `torch.Tensor`

Virtual-head input factor of shape \[heads, rank].

---

**`vha_v`** `torch.Tensor`

Virtual-head output factor of shape \[heads, rank].

---

**`cu_seqlens`** `torch.Tensor`

Int32 cumulative document lengths of the row: `[0, tokens]` of shape \[2] without
`metadata`, of shape \[documents + 1] with it.

---

**`max_seqlen`** `int`

Length of the document, equal to `tokens`. Not read with `metadata`.

---

**`window`** `int`

Positive number of local causal keys per query.

---

**`remote_topk`** `int`

Positive number of remote keys the indexer selects per query.

---

**`softmax_scale`** `float`

Factor applied to the query-key products.

---

**`metadata`** `CudnnDsaPackedMetadata | None` — default: None

Layout of a packed row and of the queries in it, as `cudnn_dsa_layout` returns it.

---

**Returns:** `torch.Tensor`

Core output of shape \[tokens, heads \* value_dim] in the dtype of `query`.

**Raises:**

* `ValueError`: If `tokens` is zero, the shapes of the query, the latent key/value, the value projection or
  the gate logits disagree, `latent` exceeds `latent + rope`, or `window` or `remote_topk` is not
  positive.
* `NotImplementedError`: If `cu_seqlens` has more than two entries without `metadata`.

```python
nemo_automodel.components.models.ernie5_5.dsa.cudnn_dsa_layout(
    document_ids: torch.Tensor,
    first_query: int,
    queries: int
) -> tuple[torch.Tensor, nemo_automodel.components.models.glm_moe_dsa.kernels.cudnn_dsa.CudnnDsaPackedMetadata]
```

Describe the documents of a packed row, and the queries in it, for the cuDNN DSA core.

**Parameters:**

**`document_ids`** `torch.Tensor`

Integer tensor of shape \[row] on a CUDA device with the document of every token of the row.
The ids increase along the row, a negative id marks padding, the row starts with a document, and the
padding of a document follows its tokens.

---

**`first_query`** `int`

Row offset of the first query.

---

**`queries`** `int`

Number of queries: the whole row, or one context-parallel shard of it.

---

**Returns:** `tuple[torch.Tensor, CudnnDsaPackedMetadata]`

The int32 cumulative document lengths of shape \[documents + 1], and the metadata of the queries.

```python
nemo_automodel.components.models.ernie5_5.dsa.ernie5_5_index_scores(
    q: torch.Tensor,
    k: torch.Tensor,
    weights: torch.Tensor
) -> torch.Tensor
```

Score every key for every query with the ERNIE scaling order.

The score of a query and a key is the sum over heads of `relu(&lt;q_head, k&gt;) * weight_head`. The products and
the sum are computed in FP32 with autocast disabled. The weights are multiplied by `sqrt(channels)` in their
own dtype before the sum, and the sum is divided by `sqrt(channels)` afterwards; with BF16 weights this
rounding order affects which keys a top-k selects.

The scores are not masked. The caller applies the causal and document limits and selects the top-k, as
`select_window_and_remote` does.

**Parameters:**

**`q`** `torch.Tensor`

Indexer query of shape \[batch, queries, heads, channels].

---

**`k`** `torch.Tensor`

Indexer key of shape \[batch, keys, channels], shared by all heads.

---

**`weights`** `torch.Tensor`

Per-head weights of shape \[batch, queries, heads].

---

**Returns:** `torch.Tensor`

FP32 scores of shape \[batch, queries, keys].

**Raises:**

* `ValueError`: If `q` is not rank 4, `k` is not rank 3, `weights` is not \[batch, queries, heads], or the
  batch or channel dimensions of `q` and `k` differ.

```python
nemo_automodel.components.models.ernie5_5.dsa.freeze_ernie5_5_indexer_params(
    model: torch.nn.Module
) -> None
```

Set `requires_grad` to False on every parameter of every `Ernie5_5Indexer` in a model.

The indexer only selects keys and receives no gradient from the attention output, so its parameters are kept
out of the trainable set. The call changes the flags once and installs no hook: code that enables gradients
afterwards, such as a trainability policy or `model.requires_grad_(True)`, must call this function again.

**Parameters:**

**`model`** `nn.Module`

Model, wrapper or pipeline stage that may contain ERNIE 5.5 DSA indexers. Parameters outside an
`Ernie5_5Indexer` are left unchanged.

---

```python
nemo_automodel.components.models.ernie5_5.dsa.latent_mqa_core(
    query: torch.Tensor,
    latent_kv: torch.Tensor,
    scores: torch.Tensor,
    document_ids: torch.Tensor,
    sink: torch.Tensor,
    value_up: torch.Tensor,
    gate_logits: torch.Tensor,
    vha_u: torch.Tensor,
    vha_v: torch.Tensor,
    window: int,
    remote_topk: int,
    softmax_scale: float,
    key_document_ids: torch.Tensor | None = None,
    first_query: int = 0
) -> torch.Tensor
```

Run the eager DSA core: latent multi-query attention with a sink, value up-projection, head mix and gate.

Every head attends with its own query to one latent key/value per token, over the keys that
`select_window_and_remote` selects. A per-head sink logit takes part in the softmax and its probability is
discarded, so a query without a selected key (padding) produces zeros. The attended latent is projected to the
value channels of each head, mixed across heads with `virtual_head_mix` and multiplied by the sigmoid of the
gate logits.

The logits, the softmax and the value up-projection are computed in FP32 with autocast disabled, whatever the
input dtypes; the head mix and the gate run in the dtype of `query`. The result is not guaranteed to equal the
result of `cudnn_dsa_core` bit for bit.

**Parameters:**

**`query`** `torch.Tensor`

Tensor of shape \[batch, sequence, heads, latent + rope]: the query of each head in the key space,
`latent` content channels followed by `rope` rotary channels.

---

**`latent_kv`** `torch.Tensor`

Tensor of shape \[batch, keys, latent + rope], the key shared by all heads. Its first
`latent` channels are also the value. `keys` equals `sequence` unless the queries are one
context-parallel shard of the row.

---

**`scores`** `torch.Tensor`

Finite index scores of shape \[batch, sequence, keys], indexed \[batch, query, key].

---

**`document_ids`** `torch.Tensor`

Integer tensor of shape \[batch, sequence]; see `select_window_and_remote`.

---

**`sink`** `torch.Tensor`

Per-head sink logits of shape \[heads].

---

**`value_up`** `torch.Tensor`

Value up-projection of shape \[heads, value_dim, latent].

---

**`gate_logits`** `torch.Tensor`

Tensor of shape \[batch, sequence, heads \* value_dim].

---

**`vha_u`** `torch.Tensor`

Virtual-head input factor of shape \[heads, rank].

---

**`vha_v`** `torch.Tensor`

Virtual-head output factor of shape \[heads, rank].

---

**`window`** `int`

Positive number of local keys per query.

---

**`remote_topk`** `int`

Non-negative number of remote keys per query.

---

**`softmax_scale`** `float`

Factor applied to the query-key products.

---

**`key_document_ids`** `torch.Tensor | None` — default: None

Optional integer tensor of shape \[batch, keys]; see `select_window_and_remote`.

---

**`first_query`** `int` — default: 0

Index of the first query among the keys.

---

**Returns:** `torch.Tensor`

Tensor of shape \[batch, sequence, heads \* value_dim] in the dtype of `query`, with the value channels of

**Raises:**

* `ValueError`: If a rank or a dimension disagrees with the shapes above, or `select_window_and_remote`
  rejects the scores, the document ids, `window` or `remote_topk`.

```python
nemo_automodel.components.models.ernie5_5.dsa.merge_remote_and_window(
    remote_indices: torch.Tensor,
    document_starts: torch.Tensor,
    causal_lengths: torch.Tensor,
    query_valid: torch.Tensor,
    window: int
) -> tuple[torch.Tensor, torch.Tensor]
```

Combine the remote indices of every query with its local causal window.

Keys are the tokens of one row, queries are all of them or one context-parallel shard, and an index is the
offset of a token in that row. The remote part of the result keeps the input order and replaces with -1 every
index that is absent, belongs to another document, or lies inside or after the local window. The local part
lists the last `window` offsets up to the query in ascending order, with -1 for positions before the start of
the document.

The real tokens of a document must be contiguous from its start. The values of the inputs are not checked for
consistency.

**Parameters:**

**`remote_indices`** `torch.Tensor`

Int32 tensor of shape \[queries, remote_topk] with row offsets of remote keys, or -1 for
absent entries.

---

**`document_starts`** `torch.Tensor`

Int32 tensor of shape \[queries] with the row offset of the first token of each query's
document.

---

**`causal_lengths`** `torch.Tensor`

Int32 tensor of shape \[queries] counting the tokens from the start of the document through
the query.

---

**`query_valid`** `torch.Tensor`

Boolean tensor of shape \[queries]. A False entry gives a row of -1.

---

**`window`** `int`

Positive number of local causal positions per query.

---

**Returns:** `torch.Tensor`

The int32 indices of shape \[queries, remote_topk + window], remote part first, with invalid entries set to

**Raises:**

* `ValueError`: If `window` is not a positive integer, `remote_indices` is not an int32 matrix, a metadata
  tensor does not have shape \[queries], the starts or the lengths are not int32, the validity is not
  boolean, or the tensors are on different devices.

```python
nemo_automodel.components.models.ernie5_5.dsa.rotate_indexer_activation(
    x: torch.Tensor
) -> torch.Tensor
```

Apply the scaled Hadamard rotation of the DSA indexer.

Precision contract: the input, of any floating-point dtype, is rounded to BF16; the butterfly and the scaling by
`channels ** -0.5` run in FP32; the result is rounded to BF16 once. On a CUDA device the
`fast_hadamard_transform` kernel computes this when the package is installed. Without the package, and on
every other device, the same arithmetic runs in PyTorch, so both paths return the same bits.

**Parameters:**

**`x`** `torch.Tensor`

Floating-point tensor of shape \[..., channels], with arbitrary leading dimensions; `channels` is a
power of two.

---

**Returns:** `torch.Tensor`

BF16 tensor of shape \[..., channels] on the device of `x`.

**Raises:**

* `ValueError`: If `channels` is not a power of two.

```python
nemo_automodel.components.models.ernie5_5.dsa.select_window_and_remote(
    scores: torch.Tensor,
    document_ids: torch.Tensor,
    window: int,
    remote_topk: int,
    key_document_ids: torch.Tensor | None = None,
    first_query: int = 0
) -> torch.Tensor
```

Select, for every query, a causal local window and the top-scoring remote keys of its document.

A key is eligible for a query when both are real tokens of the same document and the key is not after the
query. The local window is the query and the `window - 1` eligible keys before it. The other eligible keys are
remote, and the `remote_topk` highest-scoring of them are added. The function runs without gradient tracking.

The queries are the keys, or, with `key_document_ids`, the consecutive keys that start at `first_query`:
one context-parallel shard of the row.

**Parameters:**

**`scores`** `torch.Tensor`

Finite floating tensor of shape \[batch, queries, keys], indexed \[batch, query, key].

---

**`document_ids`** `torch.Tensor`

Integer tensor of shape \[batch, queries] on the device of `scores`. Tokens of one document
carry the same id, and a negative id marks padding. The caller must give every contiguous document of a
row its own id: tokens that share an id are treated as one document.

---

**`window`** `int`

Positive number of local keys per query, the query included.

---

**`remote_topk`** `int`

Non-negative number of remote keys per query. Zero selects the local window only.

---

**`key_document_ids`** `torch.Tensor | None` — default: None

Optional integer tensor of shape \[batch, keys] with the document of every key. None
means that the keys are the queries.

---

**`first_query`** `int` — default: 0

Index of the first query among the keys.

---

**Returns:** `torch.Tensor`

Boolean tensor of shape \[batch, queries, keys], indexed \[batch, query, key], that is True for the

**Raises:**

* `ValueError`: If `scores` is not \[batch, queries, keys], `document_ids` is not \[batch, queries], the key
  ids are not \[batch, keys], `window` is not positive, `remote_topk` is negative or a score is not
  finite.

```python
nemo_automodel.components.models.ernie5_5.dsa.dsa_selection_replay: RecomputeReplay[Tensor] = RecomputeReplay('ERNIE 5.5 DSA key selection')
```