Skip to navigation

DeepSeek Sparse Attention (DSA)

View as MarkdownOpen in Claude

This is an experimental API and subject to change.

Stream parameter types are operation-specific. Sparse-attention forward uses the current PyTorch stream when its stream argument is None; an explicit stream must be a concrete cuda.bindings.driver.CUstream. Convert a torch.cuda.Stream with cuda.bindings.driver.CUstream(stream.cuda_stream). The special CUstream(2) per-thread default sentinel is rejected by Torch interop: it cannot safely represent an allocator stream identity across host threads. Use a concrete stream instead.

Overview

The DeepSeek Sparse Attention (DSA) module integrates a set of CuTe-DSL kernels that support the sparse-attention path used by DeepSeek-style models. Most kernels target Hopper (SM90) and Blackwell (SM100+) GPUs. Sparse-attention forward supports SM90 and the mapped SM100-family capabilities 10.0, 10.3, and 10.7; the combined compressed-logits + Top-K path serves a validated packed-THD configuration on SM90 and broader configurations on SM100+. Dense Indexer Forward and standalone Indexer Top-K support both architectures. The kernels are delivered as Python classes / wrappers that follow the same APIBase pattern as other cuDNN Frontend operations.

Scope: this module ships CuTe-DSL kernels for sparse-attention forward and backward, indexer scores/top-K, sparse/dense score recompute, and sparse/dense indexer backward. On SM90, sparse forward covers H16/H32/H64/H96/H128 Prefill; on SM100 it covers regular H64/H96/H128 and small-top-k H128 Prefill. It does not include a decode or split-KV forward mode.

The module packages the following operations:

  1. Sparse Attention Forward – sparse Prefill MQA on SM90 and SM100.
  2. Sparse Attention Backward – DSA backward for flat MQA tensors on SM90/SM100.
  3. Indexer Forward – CuTe-DSL score kernel (Q @ K^T, ReLU, head reduce, ratio causal mask) that materializes dense scores.
  4. Combined Indexer Forward + Top-K – SM90/SM100 compact score generation, Top-K selection, and optional Top-K softmax in one public API call.
  5. Indexer Top-K – SM90+ CuTe-DSL radix top-K kernel with per-row seq_lens.
  6. Sparse Indexer / Attention Score Recompute – sparse (top-K) recompute of indexer and attention scores for training loss.
  7. Dense Indexer / Attention Score Recompute – dense (full-KV) analogues of the above.
  8. Indexer Backward – three-stage pipeline (score-grad, three GEMMs, dtype cast) for sparse top-K score tensors.
  9. Dense Indexer Backward – full-KV counterpart of Indexer Backward.

Architecture

Q, K, W ──┬─► IndexerForward ──► scores ──► IndexerTopK ──► topk_idxs
└─► IndexerForwardTopK ──► topk_idxs, logits, predict
│
v
SparseAttentionForward ──► out, lse
│
dout ──┤
v
SparseAttentionBackward
│
v
dq, dkv, d_sink
Training-score loss path:
attn_score, index_score ──► IndexerBackward ──► d_index_q, d_weights, d_index_k
(SparseIndexer/AttnScoreRecompute and DenseIndexer/AttnScoreRecompute
produce these score tensors; DenseIndexerBackward consumes dense raw scores.)

Installation

pip install nvidia-cudnn-frontend

API Usage

JAX ops (SM100 DSv4)

import jax.numpy as jnp
from cudnn.ops import dsa
scores = dsa.indexer_forward_jax(q, compressed_k, weights, ratio=4)
topk_indices = dsa.top_k_forward_jax(scores, seq_lens, top_k=512)
out, _, lse, _ = dsa.sparse_attention_forward_jax(
q_main, kv, global_indices, topk_lengths, attention_sinks,
indexer_topk=512, softmax_scale=1.0,
)
dq, dkv, dsink = dsa.sparse_attention_backward_jax(
q_main, kv, out, dout, lse, attention_sinks, global_indices, topk_lengths, softmax_scale=1.0
)

cudnn.ops.dsa exports these JAX ops. They accept JAX arrays, eagerly or under jax.jit, require no PyTorch, and run on SM100-family GPUs. Forward and backward are separate calls; the caller owns the VJP. Each has a PyTorch counterpart in cudnn.ops.dsa, the same name without _jax, which is the listed DSA wrapper: compressor_forward and compressor_backward are the CSA compressor wrappers in csa.md.

OpPyTorch counterpartContract
indexer_forward_jaxindexer_forward (DSA.indexer_forward_wrapper)BF16 q (B, S_q, 64, 128), k (B, S_k, 1, 128), weights (B, S_q, 64); ratio=4; S_k a multiple of 4. Packed THD: q (T_q, 64, 128), k (T_k, 1, 128), weights (T_q, 64) with int32 cu_seqlens_q / cu_seqlens_k and static max_seqlen_q / max_seqlen_k; scores are (T_q, ceil4(max_seqlen_k)), indexed by sequence-local key position. Returns FP32 ratio-causal scores, plus the row LSE with return_lse=True
top_k_forward_jaxtop_k_forward (DSA.indexer_top_k_wrapper)scores (n_rows, num_cols) FP32/FP16/BF16, int32 seq_lens (n_rows,); top_k \<= 2048. Returns unsorted int32 column ids, -1-padded
indexer_backward_jaxindexer_backward (DSA.indexer_backward_wrapper)Top-k KL indexer loss backward; topk=512 only. d_k is FP32
dense_indexer_backward_jaxdense_indexer_backward (DSA.dense_indexer_backward_wrapper)Dense KL indexer loss backward from indexer_forward_jax(..., return_lse=True). d_k is FP32
sparse_attention_forward_jaxsparse_attention_forward (DSA.sparse_attention_forward_wrapper)BF16 q (T_q, 64, 512), kv (T_kv, 512); int32 global indices (T_q, K) with K a multiple of 64
sparse_attention_backward_jaxsparse_attention_backward (DSA.sparse_attention_backward_wrapper)Gradients of q, kv and the FP32 attention sinks

DSA Namespace

from cudnn import DSA
DSA.SparseAttentionBackward
DSA.sparse_attention_backward_wrapper
DSA.SparseAttentionForward
DSA.sparse_attention_forward_wrapper
DSA.IndexerForward
DSA.indexer_forward_wrapper
DSA.indexer_forward_top_k_wrapper
DSA.IndexerForwardTopKPlan
DSA.prepare_indexer_forward_top_k
DSA.compress_topk_cand_buffer_size
DSA.compress_topk_cand_buffer_size_thd
DSA.IndexerTopK
DSA.indexer_top_k_wrapper
DSA.local_to_global_wrapper
DSA.compactify_wrapper
DSA.SparseIndexerScoreRecompute
DSA.sparse_indexer_score_recompute_wrapper
DSA.SparseAttnScoreRecompute
DSA.sparse_attn_score_recompute_wrapper
DSA.DenseIndexerScoreRecompute
DSA.dense_indexer_score_recompute_wrapper
DSA.DenseAttnScoreRecompute
DSA.dense_attn_score_recompute_wrapper
DSA.IndexerBackward
DSA.indexer_backward_wrapper
DSA.DenseIndexerBackward
DSA.dense_indexer_backward_wrapper

Components

1. Sparse Attention Forward

Sparse Prefill forward for flat MQA tensors. For query row t, head h, and valid top-k slot j, the score is softmax_scale * dot(q[t,h,:], kv[topk_idxs[t,j],:]). Invalid and OOB indices have score -inf; duplicate indices remain distinct softmax slots. The attention sink participates only in the output denominator, not in any stats.

  • Inputs
    • q: (total_S_q, H, D_qk) FP16 or BF16
    • kv: (total_S_kv, D_qk), with the same dtype as q (K = V latent; MQA)
    • topk_idxs: (total_S_q, K) INT32 global token ids; K may be any nonnegative value. SM100 pads a tail internally; SM90 masks it in place.
    • attn_sink (optional): (H,) FP32
    • topk_length (optional): (total_S_q,) INT32, clamped to [0, K]
    • softmax_scale (optional runtime float): defaults to 1 / sqrt(D_qk)
    • indexer_topk: compile-time prefix length
  • Outputs — stable TupleDict keys:
    • out: (total_S_q, H, 512), with the same dtype as q
    • max_logits: (total_S_q, H) FP32
    • lse: (total_S_q, H) FP32, KV-only and excluding the sink
    • lse_indexer: (total_S_q, H) FP32, or None for indexer_topk=0
  • Variants
    • SM90: H in {16, 32, 64, 96, 128}, D_qk in {512, 576}, and indexer_topk in {0, 512, 1024, 2048}
    • SM100 H64: D_qk in {512, 576}, indexer_topk in {0, 512, 1024, 2048}
    • SM100 H96: as H64. It launches the H64 kernel twice, on heads [0, 64) and [32, 96) of the same tensors, so it does the work of 128 heads; both launches write identical bytes to heads [32, 64).
    • SM100 H128 small-top-k Prefill: D_qk=512, indexer_topk in {0, 512, 1024}
    • SM100 H128 otherwise (D_qk=576, or D_qk=512 with indexer_topk=2048): the H64 kernel runs twice, on the disjoint heads [0, 64) and [64, 128), and each head’s bytes equal an H64 run on that window. For D_qk=512, indexer_topk=2048 therefore uses a different kernel than {0, 512, 1024}, and results can differ in rounding between the two.
  • Indexer prefix constraint — indexer_topk <= K. When topk_length is present, a meaningful lse_indexer additionally requires topk_length >= indexer_topk for that row. Shorter rows retain the SM100 source kernel’s +inf sentinel; their out, max_logits, and full lse outputs remain valid.
  • Empty rows — out=0, max_logits=-inf, lse=+inf; an enabled lse_indexer is also +inf.
  • Compilation lifecycle — on SM90, SparseAttentionForward.compile() compiles a storage-free symbolic plan and execute() only launches it with caller-owned outputs. The convenience wrapper allocates outputs and stages layouts before calling the same plan. The existing SM100 direct path retains its concrete-tensor cache and first-execute compilation.
  • CUDA Graph and workspace — the prepared SM90 execute() path is allocation-free, performs no host read or JIT, and requires no workspace. Compile before capture and provide preallocated outputs. Runtime total_S_q, total_S_kv, and logical K may change per execute(). H, Dqk, dtypes, and optional-operand presence must match the declaration; a plan for another supported H reuses the same compiled kernel without a new specialization.
  • Aliasing — prepared SM90 outputs must be byte-range-disjoint from every input and from one another. Empty tensors occupy no byte range.
  • Layout and stream — the convenience wrapper materializes non-contiguous or under-aligned inputs on the selected stream. Prepared SM90 execute() instead requires contiguous, adequately aligned tensors and raises before launch if the contract is not met. An explicit current_stream for prepared execution (or stream for the wrapper) must be a cuda.bindings.driver.CUstream belonging to q.device.

For D_qk=576, QK uses all 576 dimensions while V/output uses only the first 512 KV dimensions. attn_sink=None is equivalent to an all--inf sink.

Credits and cuDNN Frontend extensions

The SM90 kernel schedule and pipeline are adapted primarily from FlashMLA’s MIT-licensed Hopper sparse-prefill implementation. The cuDNN Frontend port extends that foundation with H16/H32/H64/H96/H128 coverage, arbitrary logical K tails and boundary handling, a bounded symbolic compile cache, and an allocation-free prepared-execution contract for CUDA Graph capture and replay.

import math
result = DSA.sparse_attention_forward_wrapper(
q,
kv,
topk_idxs,
attn_sink=attn_sink,
topk_length=topk_length,
softmax_scale=1.0 / math.sqrt(q.shape[-1]),
indexer_topk=512,
)
out = result["out"]
max_logits = result["max_logits"]
lse = result["lse"]
lse_indexer = result["lse_indexer"]

2. Sparse Attention Backward

Backward pass for DeepSeek Sparse Attention. Expects the forward wrapper’s out and KV-only lse (or equivalent tensors).

  • Inputs
    • q: (total_S_q, H, D) BF16/FP16
    • kv: (total_S_kv, D) (K = V; MQA)
    • out, dout: (total_S_q, H, D_v)
    • lse: (total_S_q, H) FP32
    • attn_sink: (H,) FP32
    • topk_idxs: (total_S_q, topk_max) INT32 (global)
    • topk_length (optional): (total_S_q,) INT32 — per-query valid count. The H128 two-CTA backends (D512 and D576) clamp it to [0, topk_max] and ignore every slot after that prefix, including valid indices pointing to nonfinite KV rows; the other backends expect topk_length \<= topk_max. Omitting this tensor uses all topk_max slots.

On Blackwell SM100/SM103, the public backward entry point automatically selects the tuned kernel from the device, dtype, and tensor shape. On SM100 (10, 0) and SM103 (10, 3) devices, BF16 H128 with head_dim = head_dim_v = 512 and topk_max ∈ {128, 512, 1024, 1152, 2048} uses the two-CTA specialization. Contiguous BF16 H128 with head_dim = 576, head_dim_v = 512, and the same topk_max set uses the H128/D576 two-CTA specialization. H16 with head_dim=576 uses the dedicated M128 sparse-row pipeline. FP16, other head counts and dimensions, and every other topk_max retain the existing generic/H16/H32 selection. Other compute capabilities, including SM107, do not select the two-CTA paths. No backend or tile-size argument is required. SM90 continues to use its Hopper-specific implementation. Every input must be contiguous on every route: check_support() declines a strided tensor with NotImplementedError naming it and its strides, and execute() rejects one with ValueError; the plan never repacks an input.

The SM100 H16/H32/H96 D576 specializations compile the physical row stride of topk_idxs from topk_max, while the query count and per-query topk_length remain dynamic. H64 and the generic/H128 D512 paths retain their existing layout; H128 D576 already uses a static topk_max stride.

For contiguous BF16 H16/H32 D576 inputs, q_cluster_mode="adaptive_pair" enables adjacent-query KV sharing. A device-side transform partitions each pair into one tile-aligned intersection and two unique segments, then evaluates the intersection with twice the head count. Its bounded hash storage scales with topk_max rather than total_S_kv; invalid, duplicate, low-overlap, and an odd final query retain the ordinary per-query work. The mode is opt-in, non-deterministic, CUDA Graph compatible, and supports topk_max \<= 2048. It is intended for high-overlap workloads; the device-side partition still runs when a pair falls back to ordinary work. H16 topk_max \<= 256 and H32 topk_max \<= 128 accept the mode but compile the ordinary backend directly.

With the default deterministic=False, the H128 specialization keeps the five tensor-core products in one two-CTA main kernel. It publishes FP32 O-dot-dO and folded-LSE statistics to the caller-provided scratch workspace, converts the FP32 dKV workspace to the public BF16 output, and completes dSink with a separate FP32 reduction kernel. The helper launches do not change the two-CTA topology of the core computation.

The H128/D576 specialization applies the same two-CTA topology to the MLA layout (head_dim = 576, head_dim_v = 512). Its compiled sequence with deterministic=False initializes the FP32 dKV workspace and the dSink output on the launch stream, accumulates dKV in that workspace, and converts it to the public BF16 output with a helper kernel. With the default deterministic=False, short query batches split each query’s top-k range across otherwise idle two-SM clusters; the split count is fixed when the plan is compiled.

On SM90, deterministic=True and the default deterministic_dkv_mode='records' use conflict-rank dKV storage for H16/H32/H64/H96/H128. A setup kernel builds an immutable 128-bit query-selection bitmap and four zero-initialized ticket words for every referenced (query wave, M64 head tile, KV row). Each query computes its stable conflict rank by popcounting lower query ordinals. Ranks zero through three accumulate into four separate dense FP32 planes. Higher ranks reuse plane zero after all lower plane-zero writers for that row publish monotonically increasing release tickets. On a resident 128-CTA wave, D512 uses four independent column groups and D576 uses two, so adjacent overflow ranks can pipeline disjoint columns instead of serializing the complete row update. A multi-launch schedule keeps one ticket per row because its phases are already stream ordered. Ranks one through three require no ticket because their planes are disjoint. A prepared plan reads the device SM count once and divides each wave into cooperative power-of-two phases with at most one CTA per SM. Cooperative launch guarantees that every writer a CTA can wait for is resident; same-stream phase order carries completed tickets across launches. The selected phase count is compiled into the plan and is not queried during execution or CUDA Graph replay. Head tiles and later waves remain stream ordered, and the postprocess kernel combines the four planes in fixed order. One warp per head computes d_sink with a fixed strided-lane/tree reduction order. Duplicate valid top-k indices within one query row remain outside this deterministic contract.

The conflict-rank dKV allocation is 4 * (K64 + mask_rows) * D32 * sizeof(float), where K64 = round_up(total_S_kv, 64), D32 = round_up(D, 32), and mask_rows = ceil(2 * ceil(total_S_q / 128) * ceil(H / 64) * K64 / D32). The factor two stores the immutable bitmap and four 32-bit column-phase ticket words; both groups are striped across the four planes.

An explicit deterministic_relay_cluster_size=C, for C in {4, 8, 16, 32, 64}, selects the previous 'shards' cluster relay even when the requested dKV mode is 'records'. Each 128-query wave then gives one FP32 dKV plane to every logical relay group. The plan queries CUDA cluster occupancy and uses the widest launchable physical C16, C8, or C4 cluster no larger than the logical relay; wider relays use ordered same-stream phases. The setup kernel builds packed C-bit ownership masks, relay CTAs wait only for lower ranks that update the same row, and the postprocess combines 128 / C planes. Its allocation is (128 / C) * (K64 + mask_rows) * D32 * sizeof(float), with mask_rows = ceil(ceil(total_S_q / 128) * ceil(H / 64) * (K64 / (mask_bits / C)) * (mask_bits / 32) / D32) and mask_bits equal to 64 for C64 and 32 otherwise. The caller-owned workspace is cleared and the masks are rebuilt on every execution, so changing topk_idxs or topk_length between launches is safe. After one eager execution populates the SM90 concrete-tensor compile caches, the caller-owned output/workspace path is CUDA Graph capturable. Replay keeps the declared tensor shapes and addresses fixed while allowing tensor contents, including topk_idxs and topk_length, to change. On H20-3e with BF16 H64/D512, Q=16384, KV=262144, and top-k 2048, conflict-rank records measured 119.223 ms and 3.008 GiB versus 484.915 ms and 4.508 GiB for the C16 logical relay baseline split into physical C8 launches: 4.07x faster with 33.3% less workspace. Historical explicit-relay H200 measurements and the full environments are recorded in benchmark/dsa/README.md.

On the AIStudio H200 allocation (runtime name NVIDIA L20X) with BF16 H64, Q=16384, KV=262144, and top-k 2048, the single-wave records schedule measured 69.147—69.217 ms for D512 and 83.164—83.285 ms for D576. The existing C16 relay measured 84.643 and 100.126 ms respectively; the H200-tuned C32 relay measured 75.936—75.970 and 94.546 ms. Each result uses five warmups and 20 allocation-free calls. Records are 1.224x faster than C16 and 1.098x faster than C32 for D512, and 1.204x faster than C16 and 1.137x faster than C32 for D576. When every query shares the same 2048 KV rows, column-phase tickets reduce D512 from 151.586 to 87.678 ms (1.73x) and D576 from 195.635 to 139.779 ms (1.40x) relative to the previous single-ticket records path.

On SM100/SM103, deterministic=True retains the two-CTA Tensor Core kernels for the supported BF16 H128/D512 and H128/D576 envelopes above. Queries run in same-stream waves with one two-CTA pair per query: D512 spreads the queries evenly over waves of at most 74, and D576 waves hold one query per two-SM cluster (at most 76) per wave round, at most 74 per round in three-round plans (devices with 152 or more SMs leave the remaining SMs idle there). Each wave accumulates dKV either in private FP32 slots or in conflict-rank storage (below); ordered duplicate-index writers and stream ordering between waves give each update a fixed accumulation order, and fixed-order reductions produce dKV and d_sink. With either H128 two-CTA layout, duplicate indices within a query row are inside the deterministic contract, as with the generic M64 conflict-rank layouts (below) and unlike its shard and relay layouts and the SM90 deterministic path. The plan option deterministic_dkv_mode (below) picks the layout of H128 two-CTA plans: under 'records' (the default) the plan chooses between the two layouts from the shapes, 'shards' keeps the private-plane layout for every shape, and 'chain' is rejected. The private-plane layouts hold 128 FP32 dKV planes for D512 and 96 for D576; H96 D576 two-CTA plans always use their 96-plane layout. 'shards' restores the private-plane layout but, for D512, not the bits of releases without conflict-rank storage: D512 private-plane plans stream the dK/dV round operands in four 32-head stages, which sums dKV in a different fixed order (dQ and d_sink are unchanged).

Conflict-rank dKV storage (deterministic H128 D512/D576). Within a wave, the queries that select a KV row are ranked by their position in the wave. The query of rank k \< P accumulates into dense FP32 plane k (the first write of a row above plane 0 stores, later ones add); higher ranks write compact overflow records, so every plane row and every record has a single writer per wave and the main kernel needs no writer ordering. A row that one query selects more than once takes per-slot records in every slot of that wave that selects it. A per-wave tail adds the overflow records to plane 0 in a fixed order (rank order, with long chains summed in fixed segments) and the per-slot records in (query, slot) order, with sorted slots for D576 top-k 1024 and wider; the final fold adds each row’s used planes in plane order. Small setup kernels chained to the main kernels with programmatic dependent launch compute each wave’s ranks and destination records before it runs. D512 plans with top-k 512 or less that keep at least three dense planes next to a doubled overflow area run these steps in a setup warp in each CTA of the preceding main kernel instead, and each wave’s tail in the next main kernel’s reducer warps, which shift a row’s ranks off plane 0 while that tail is pending.

  • When it is selected — in 'records' plans (the default; 'shards' plans keep the private-plane layout), automatically, from the KV extent and the top-k width, whenever the conflict-rank workspace fits the private-plane layout (D576 at top-k 128 also requires the layout sized for its expected conflict load to need at most three quarters of it). Short KV extents relative to the top-k width, where the private planes are already small, keep the private-plane layout. The choice depends only on the batch’s shapes and the mode, so one problem always takes the same path.
  • Workspace — scratch_workspace_bytes() reports the selected layout; it never exceeds the private-plane size of the same problem. The dense planes fill a budget of about 1 GiB, but at least one plane (and the per-row state) is kept, so past a few hundred thousand KV rows the workspace grows with the KV extent: at total_S_q=4096 about 2.1-2.4 GiB per million KV rows for D512 and 2.5-2.7 GiB for D576, about 1% of the private-plane layout (256 and 216 GiB per million rows). The D576 size also depends on the plan device’s SM count, through the cluster pairs each wave round launches: query scratch_workspace_bytes() for the device that executes the plan, since a size from a device with fewer SMs can be too small (execute() then rejects the workspace). Every execution re-initializes the conflict-rank state it reads, so one workspace can be reused across executions with different topk_idxs and topk_length, and a captured plan replays new input contents like the private-plane plans.
  • Determinism — outputs are bitwise reproducible from run to run on one GPU and software runtime. Duplicate indices within a query row, -1 and out-of-range indices, and any topk_length are inside this contract. The FP32 summation order is a fixed function of the plan and the indices: D512 waves follow the query count, D576 waves also the device’s SM count, and D576 one-round plans wider than top-k 128 that would stride their waves first run a probe launch that counts one wave’s writers and picks strided or consecutive waves from the index contents. Results are not claimed to be bitwise equal across GPUs or software versions. Conflict-rank and private-plane plans sum dKV and d_sink in different orders; their results agree within FP32 rounding, not bitwise. dQ is bitwise equal in both layouts except for D576 conflict-rank plans with top-k 1024 and wider, which sort each query’s rows by ranges of the KV extent. A sequence’s dKV is not batch invariant: packing it with other sequences changes the waves, the ranks and the plane count that order its sums, and the sorted D576 plans also change its dQ. d_sink follows the query positions in both layouts (see Determinism Support).

Other deterministic envelopes with H16/H32/H64/H96/H128 run the generic M64 kernel. Its dKV storage is a plan-time choice, deterministic_dkv_mode (below): by default on SM100/SM103, conflict-rank records wherever their workspace does not exceed the shard or relay layout of the same plan, and that layout elsewhere and on other devices. H16/H32 use a masked M64 head tile. Deterministic H96 D576 outside the two-CTA envelope (FP16, other top-k widths, or other devices) runs the generic M64 kernel only with conflict-rank storage, so only on SM100/SM103; every other such plan composes the H64 kernel with the tuned H32 tail, keeps 128 FP32 dKV planes, and rejects an explicit deterministic_relay_cluster_size.

Deterministic dKV storage (deterministic_dkv_mode). The option of SparseAttentionBackward and sparse_attention_backward_wrapper selects the dKV storage of deterministic SM100/SM103 plans of the generic M64 kernel as follows, and of the H128 two-CTA plans as described above:

  • 'records' (default) — conflict-rank storage. Each query wave runs 148 query CTAs per head block (74 for 64 < H <= 128 when a full wave’s records would exceed 0.4 GiB, and for split pairs, below). Within a wave, the queries that select a KV row are ranked by their wave position: rank k below the plan’s plane count P accumulates into dense FP32 plane k of the row, higher ranks into slot-private overflow records, so every plane row and every record has one writer per wave. Setup kernels before each main kernel assign the wave’s destinations and add the previous wave’s overflow records to plane 0 in rank order (long runs in fixed segments); a row that one query selects more than once takes per-slot records, added in (query, slot) order. The conversion folds each row’s used planes in plane order. P follows the expected number of writers per row and wave (W * top-k / total_S_kv for W query CTAs per wave) plus a margin, within about 1 GiB of planes. H96/H128 run both 64-head blocks of a query as one two-CTA cluster: head block 1 adds a tile after head block 0’s reducer threads published their adds of that tile through per-thread GPU-scope release/acquire completion words. H <= 64 plans whose full-wave workspace exceeds 1.65 times the larger of the half-wave workspace and 1 GiB run each query as a two-CTA pair that splits its top-k tiles; the second CTA hands its FP32 dQ partial to the first through the workspace. Plans with at least eight waves whose KV rows fit in 48 MiB interleave the waves’ queries (wave w of n holds queries w, w + n, ...).
  • 'chain' — the chain relay. Every head block of a wave runs in one launch of 296 query positions. CTAs claim (position, head block) tickets in start order and add into a shared plane row only after the row’s previous chain member signalled it, so each plane row accumulates in a fixed order without overflow records. It applies only when the top-k width is at most 2048, the plan’s dense planes span at most 2^19 rows (about 174K KV rows at its floor of three planes), and its workspace does not exceed the 'shards' layout (for H96 D576, the 128-plane layout of the H64 + H32 composition); every other plan follows the 'records' rule. In the table below, chain plans need less workspace than records plans; which mode is faster depends on how much the queries’ top-k rows overlap, so profile both on the target workload.
  • 'shards' — the shard layout or the cluster relay described below, chosen from the query and KV extents alone, with the kernels, workspace and bitwise results of releases without this option.

Further properties of the option:

  • Validation — values other than the default require deterministic=True and unknown values are rejected. 'chain' cannot be combined with deterministic_relay_cluster_size and is rejected on SM90 and on the two-CTA backends. On the H128 two-CTA backends 'records' chooses between conflict-rank and private-plane storage from the shapes and 'shards' selects the private-plane layout; H96 D576 two-CTA plans keep their 96-plane layout under both. Under 'records' and 'shards', an explicit relay size selects that cluster relay. The option is part of the compiled plan and of the wrapper’s plan cache key.
  • Devices — conflict-rank storage (the 'records' and 'chain' layouts and the H128 two-CTA conflict-rank layout) is provided on SM100 and SM103 (compute capability 10.0 and 10.3). On other SM100-family devices (compute capability 10.7), a 'records' plan runs as a 'shards' plan, with the same kernels, workspace and results, and 'chain' is rejected with a ValueError that names the compute capability. flash_attn_bwd_sm100_workspace_size() takes the keyword-only max_topk, launch_pairs, dkv_mode and device_capability: without max_topk it returns the shard, relay or private-plane size, an upper bound for every conflict-rank plan of the shape, and the default device_capability=None sizes for an SM100/SM103 device.
  • Workspace — scratch_workspace_bytes() reports the selected layout and is authoritative. It never exceeds the 'shards' layout of the same plan (for generic H96 D576 plans, the 128-plane layout of the H64 + H32 composition that their 'shards' plans run). A records plan sizes a full wave even when total_S_q is smaller, so its workspace does not shrink further with the query count. Every execution re-initializes the state it reads, so one workspace serves executions and CUDA Graph replays with different topk_idxs and topk_length.
  • Determinism — each mode is bitwise reproducible from run to run on one GPU and software runtime. -1 and out-of-range indices and topk_length values up to topk_max are inside the contract of every layout; records and chain plans also clamp larger lengths to topk_max. Duplicate indices within a query row are inside the contract of the records and chain layouts but not of the shard and relay layouts, which the other modes also select for some shapes; callers that need bitwise repeatability for every shape should pass unique valid indices within each query’s active top-k prefix. The layouts sum dKV in different orders, so their dKV agrees within FP32 rounding, not bitwise. d_sink has the same bits under every layout of the generic M64 kernel: one warp per head sums the queries in the same order (the H64 + H32 composition of H96 D576 is a separate kernel). dQ does not depend on the dKV layout either, except for chain plans, which gather a query’s rows in sorted order, and records plans with split pairs, which add two dQ partials. dKV is not batch invariant once a batch spans several waves, with any layout: the layout, the waves and the plane or shard count follow the batch’s query and KV extents. d_sink is: a uniform shift of the queries only rotates the lanes of that warp (see Determinism Support).
  • Compilation — the wave size and the split and interleave choices are compile-time constants of a records plan. For given heads, head dimension, dtype, top-k width and KV extent, the wave order gives records plans two compiled variants over the query count (interleaved or not), and 64-bit addressing, which large query extents need, adds another; the storage a plan selects depends on its query and KV extents. Each plan still compiles only in compile().
  • Launches — conflict-rank plans chain their kernels with programmatic dependent launch. A records wave launches two setup kernels and its main kernel; a chain wave does the same, with one more scan for the first wave, and computes O.dO in its setup. Each execution adds an initialization, the last wave’s setup, the fold and the d_sink reduction (records plans also launch a first-wave marking kernel and the O.dO kernel). The last grid of an execution completes only after the fold, so the next launch on the stream observes the final dKV. The 148-CTA wave matches the SM count of B200. The wave and setup-grid sizes are plan constants that do not depend on the device’s SM count, and correctness relies only on the two CTAs of a cluster being co-resident.

Scratch (scratch_workspace_bytes()) of representative BF16 plans:

HD_qkQKVtop-k'shards''records''chain'
645761283276820482.250 GiB1.382 GiB1.003 GiB
64576163843276820489.008 GiB1.389 GiB1.011 GiB
64512163843276820488.008 GiB1.239 GiB0.963 GiB
645764096327685129.002 GiB0.526 GiB0.505 GiB
6457640966553620482.283 GiB1.387 GiB1.008 GiB
645761638413107220489.258 GiB1.540 GiB0.879 GiB
6457616384524288204810.008 GiB1.859 GiB1.859 GiB (records)
64576163841048576204820.008 GiB3.035 GiB3.035 GiB (records)

Shard and relay layouts. deterministic_dkv_mode='shards', an explicit relay size, and shapes whose conflict-rank workspace would be larger use the bounded-wave shard layout. Ordinary extents select 32 dKV shards for total_S_q \<= 128, 64 for 129..256, and 128 for larger query batches. Queries run in same-stream waves with one CTA per shard; CTA lane i is the sole writer of FP32 dKV shard i in each wave. H96/H128 split each query wave into ordered M64 head-block launches, preserving single-writer ownership without multiplying the number of shards. Kernel launch ordering serializes both head blocks and shard reuse across waves. The 32-shard specialization converts with one fixed-order reduction; the 64- and 128-shard specializations first fold groups of eight and then combine those groups in fixed order.

If the selected shard layout would exceed 16 GiB, the generic M64 plan keeps each query wave at 128 CTAs and assigns one FP32 dKV plane to each cluster. Queries below 8192 rows prefer sixteen eight-CTA clusters; larger queries prefer thirty-two four-CTA clusters for additional reducer parallelism. If that preferred layout’s complete FP32-plane and packed-mask allocation would exceed 32 GiB, the plan selects an eight-plane, sixteen-CTA memory-first relay. The long-query policy deliberately skips cluster-8: GB200 sweeps at 393K, 512K, and 768K KV found it slower than cluster-16 while retaining twice as many FP32 planes. A setup kernel builds packed 4-, 8-, 16-, 32-, or 64-bit rank masks for every (query wave, head block, KV row). Reducer threads wait only for lower cluster ranks that actually update the same KV row, then publish that row with GPU-scope release/acquire ordering. The participating reducer-thread CTA barrier carries the release/acquire causality, so no additional GPU-wide fence is needed. Rank zero skips the predecessor handoff, the last rank skips the successor publication, and the final release needs no second rendezvous. Disjoint sparse rows continue in parallel; the fixed rank order handles both low- and high-overlap top-k rows without a global semaphore dependency on non-resident CTAs. The selected 2, 4, 8, 16, or 32 planes are combined directly in the fixed-order conversion kernel, without an intermediate FP32 prefold. On SM100/SM103 under the default 'records' mode, the automatic relay runs only where records do not apply: top-k widths above 14169 (a records wave counts its 148 * top-k slots in a 21-bit field) and shapes whose records would need more workspace than the relay. Duplicate top-k indices within one query row remain outside the deterministic contract of the shard and relay layouts. This condition is not checked at runtime; callers that require bitwise repeatability must provide unique valid indices within each query’s active top-k prefix.

Callers that have profiled their exact SM100 workload may set deterministic_relay_cluster_size to 4, 8, 16, 32, or 64 to select that many CTAs per logical relay group instead of using the automatic policy. The corresponding number of FP32 relay planes is 32, 16, 8, 4, or 2. Because SM100 hardware clusters contain at most 16 CTAs, the logical 32- and 64-CTA groups run as two and four same-stream 16-CTA phases, respectively. Each kernel boundary completes the preceding physical phase before the next ranks reuse the same plane, avoiding cross-cluster residency dependencies. Cluster-64 uses a 64-bit packed row mask and is intended as an explicit emergency memory-first choice; the automatic policy never selects it because its lower per-phase parallelism regresses profiled workloads. This is a plan-time specialization: it is part of the compile/cache key, fixes the workspace size, and remains unchanged during execute() and CUDA Graph replay. The default None preserves automatic selection. Requests on non-deterministic plans, non-generic backends (including deterministic H96 D576 plans that keep the H64 + H32 composition), shapes that do not require the large-context relay, and deterministic_dkv_mode='chain' plans are rejected rather than silently ignored. An explicit choice may exceed the automatic policy’s 32 GiB memory-first budget, so callers should always allocate the exact value returned by scratch_workspace_bytes().

For the shard layout, additional dKV workspace is selected_shards * K8 * D8 * sizeof(float), where K8 = round_up(total_S_kv, 8) and D8 = round_up(D, 8). The cluster relay uses relay_planes * (K8 + mask_rows) * D8 * sizeof(float), where relay_planes = 128 / relay_width. The packed mask uses a 32-bit word for relay widths up to 32 and a 64-bit word for relay width 64. With mask_word_bits equal to 32 or 64, its FP32 storage rows are mask_rows = ceil(ceil(total_S_q / 128) * ceil(H / 64) * ceil(K8 / (mask_word_bits / relay_width)) * (mask_word_bits / 32) / D8). The conflict-rank layouts depend on the plane count, the wave and the top-k width. One CTA per head reduces d_sink. Use scratch_workspace_bytes() as the authoritative full scratch size; a larger reusable workspace remains valid. Treat this as a reproducibility requirement rather than a performance-tuning knob: keep the default False when bitwise run-to-run stability is not needed.

The measurements below compare the shard and relay layouts (deterministic_dkv_mode='shards') with the fixed 128-plane layout. The H128/D576 two-CTA row measures its private-plane layout, which 'shards' plans keep; a default plan of that shape stores dKV in conflict-rank storage (0.916 GiB of scratch on a 148-SM B200; the size follows the SM count).

On GB200 with BF16, total_S_q=16384, top_k=2048, H=64, D_qk=576, and D_v=512, the memory-first cluster-16 route reduces full scratch from 288.008 GiB to 20.008 GiB at total_S_kv=1048576. The upstream allocation does not fit on the 185.0 GiB test GPU. At total_S_kv=524288, where both implementations fit, scratch falls from 144.008 GiB to 10.008 GiB and warmed execute() latency changes from 99.973 ms to 98.999 ms.

At total_S_kv=1048576 with CP64 (total_S_q=16384 per launch), BF16, top-k 2048, D576/Dv512, and contiguous inputs, the official upstream baseline and this change require:

HeadsUpstream scratchThis change scratchScratch reduction
16288.002 GiB20.002 GiB268.000 GiB (93.1%)
32288.004 GiB20.004 GiB268.000 GiB (93.1%)
64288.008 GiB20.008 GiB268.000 GiB (93.1%)

The upstream 128-plane baseline does not fit in the test system’s 185.0 GiB usable memory, so its 1M-KV latency cannot be measured. The following direct A/B covers every deterministic dispatch selected by the adaptive policy, plus the H128/D576 two-CTA private-plane control. All rows use BF16, D576/Dv512, top-k 2048, and contiguous inputs; H is 64 except for the H128 control.

DispatchQKVUpstream scratchThis change scratchScratch changeUpstream latencyThis change latencyLatency change
ordinary 32-plane128327682.250 GiB2.250 GiB0.0%1.737 ms1.737 ms+0.01%
ordinary 64-plane256327684.500 GiB4.500 GiB0.0%2.845 ms2.844 ms-0.03%
ordinary 128-plane16384327689.008 GiB9.008 GiB0.0%38.566 ms38.551 ms-0.04%
cluster-8 / 16-plane40966553618.002 GiB2.283 GiB-87.3%16.967 ms23.618 ms+39.2%
cluster-4 / 32-plane1638413107236.008 GiB9.258 GiB-74.3%49.858 ms57.753 ms+15.8%
cluster-16 / 8-plane16384524288144.008 GiB10.008 GiB-93.1%99.973 ms98.999 ms-1.0%
H128/D576 two-CTA16384327686.766 GiB6.766 GiB0.0%58.482 ms58.651 ms+0.3%

Latency values are medians of three alternating-order A/B rounds, each with three warm-ups and 20 timed, allocation-free execute() calls. The unchanged ordinary and two-CTA routes stay within 0.3%. Cluster-8 and cluster-4 trade 39.2% and 15.8% latency, respectively, for substantial workspace reduction; at 512K KV, cluster-16 is 1.0% faster than upstream while using 93.1% less scratch.

The follow-up relay scheduling pass was measured against merged commit 1dea4471a6e6c05e29d3bf281281fd156d345a84 on the same SM100 environment with PyTorch 2.14.1+cu130, CuTe DSL 4.8.1, cuDNN 9.24.0, BF16 H64, top-k 2048, five warm-ups, and 40 allocation-free calls. The automatic policy preserves the conservative eight-CTA layout below Q=8K because an earlier cluster-4 crossover did not generalize across SM100 devices.

DqkQKVMerged latencyOptimized latencySpeedupMerged scratchOptimized scratch
5761024655366.957 ms4.466 ms1.56x2.258 GiB2.258 GiB
576819213107231.677 ms24.142 ms1.31x9.129 GiB9.129 GiB
5761638426214464.390 ms48.320 ms1.33x18.508 GiB18.508 GiB
57616384524288101.409 ms64.426 ms1.57x10.008 GiB10.008 GiB
512819213107229.207 ms21.055 ms1.39x8.129 GiB8.129 GiB
5121638426214460.060 ms42.305 ms1.42x16.508 GiB16.508 GiB
5121638452428896.221 ms60.170 ms1.60x9.008 GiB9.008 GiB

For H64, a three-round, 20-call, reversed-order A/B measured prepared-plan eager execution at 104.579 ms and CUDA Graph replay at 104.225 ms (0.997x). Capture completed with PyTorch synchronization-debug error mode enabled. The cluster-16 regression also passed 100 repeated bitwise executions, graph capture/replay with poisoned outputs and workspace, and changed device-side topk_length values. Compilation and allocation remain plan-time operations; capture uses caller-owned outputs and workspace, and replay requires the same declared tensor shapes and storage addresses while their contents may change.

SparseAttentionBackward.scratch_workspace_bytes() reports the full scratch requirement on every backend: the SM100 FP32 statistics and dKV-accumulator planes, and on SM90 the FP32 dPsum and log2-scaled LSE vectors plus the dKV accumulator. The ordinary SM90 dKV term is align128(round64(total_S_kv) * round32(D) * 4) bytes; deterministic SM90 uses the selected relay term above. Both add 2 * align128(round64(total_S_q) * H * 4) bytes of statistics. Pass a contiguous CUDA uint8 tensor of at least this size to execute(..., workspace=workspace) and reuse it across calls; the launch sequence re-initializes the accumulators on every execution (in-kernel on SM100, and with one stream-ordered memset for the SM90 dKV accumulator and for d_sink on the routes that accumulate it atomically). execute(q, kv, out, dout, lse, attn_sink, topk_idxs, dq, dkv, ..., d_sink=...) requires caller-provided dq, dkv, and the keyword-only d_sink on every backend and never allocates or copies during execution. Every SM100 plan compiles its selected ordinary, deterministic, two-CTA, or adaptive-pair artifact in compile() from storage-free tensor metadata, so its first execute() performs no CuTe JIT and may be captured directly. The SM90 implementation retains its concrete-tensor compile caches and compiles on its first execution. The high-level wrapper allocates the outputs and the scratch on the launch stream when they are omitted. Inputs, outputs, and the live workspace must not alias; concurrent executions need distinct output and workspace storage.

  • Outputs — tuple (dq, dkv, d_sink)
  • Constraints — SM90 or Blackwell SM100/SM103; SM90 supports flat MQA tensors with head_dim ∈ {512, 576}
result = DSA.sparse_attention_backward_wrapper(
q, kv, out, dout, lse, attn_sink, topk_idxs,
softmax_scale=1.0 / math.sqrt(D),
topk_length=topk_length,
deterministic=True, # optional; SM90/SM100 H16/H32/H64/H96/H128
deterministic_relay_cluster_size=None, # optional: force 4, 8, 16, 32, or 64 CTAs/group
deterministic_dkv_mode="records", # optional: "chain" or "shards" (deterministic SM100/SM103 dKV storage)
q_cluster_mode="off", # optional: "adaptive_pair" for SM100 H16/H32 D576
)
dq, dkv, d_sink = result["dq"], result["dkv"], result["d_sink"]

3. Indexer Forward (score-only)

Computes dense indexer scores: S[b, q, k] = sum_h ReLU(Q_h · K_h^T) · W_h with a ratio-causal mask. For local query row q_local, valid KV columns satisfy k_local \< clamp((q_causal_offsets[b] + q_local + 1) // ratio, 0, seqlen_k_b). When q_causal_offsets is omitted, all offsets are zero.

q_causal_offsets[b] is the global uncompressed token index corresponding to local q[0] for batch or THD segment b. It is not the packed storage offset from cu_seqlens_q: cu_seqlens_q locates where a local Q segment is stored, while q_causal_offsets locates that segment in the global causal timeline. The K columns are assumed to be a compressed-KV prefix starting at global compressed column 0.

  • Inputs
    • q: (B, S_q, H_q, D) BF16, or the architecture-specific FP8 or FP4 format described below (packed FP4 stores D // 2 per row).
    • k: (B, S_k, H_kv, D) BF16, or the architecture-specific FP8 or FP4 format described below.
    • w: (B, S_q, H_q) BF16 or FP32 with BF16 Q/K, on SM90 and SM100. FP32 head weights retain their precision through score generation, without changing the Q/K dtype; FP32 W requires unit stride in its last dimension. Unsupported strides are rejected instead of copied. The SM90 FP8 path also accepts FP32 when weights have already been pre-scaled by q_scale * sm_scale; SM100 MXFP8, MXFP4 and NVFP4 require BF16 weights.
    • q_causal_offsets (optional): CUDA INT32 tensor with one entry per batch/THD segment, on the same device as q.
    • block_table + kv_lens (optional, SM100 BF16): read k straight from a paged key cache instead of a dense tensor, with the block-table contract of the QSA compressed-key cache. k is the page pool (num_pages, page_size, H_kv, D) or (num_pages, page_size, D); page_size is a multiple of 8 that divides 128 or is a multiple of it (16, 32, 64, 128, 256 all work). block_table[b, i] ((B, max_pages) INT32, unit column stride) is the pool row of sequence b’s i-th page and kv_lens[b] ((B,) INT32) its compressed-key count; the ratio-causal limit still applies on top. cu_seqlens_k must be None (THD queries keep cu_seqlens_q), and the score width is max_seqlen_k, which defaults to max_pages * page_size; keys at or past it are never scored, nor are pool rows past kv_lens[b] inside the last live page.
  • Output — scores: (B, S_q, S_k) FP32. On SM100 this is the [..., :S_k] view over a (B, S_q, ceil4(S_k)) (THD: (total_q, ceil4(max_seqlen_k))) allocation: the kernel binds out through a 16-byte-aligned layout, so the fp32 row stride must be a multiple of 4 elements and at least ceil4(S_k). When S_k % 4 != 0 the returned tensor is therefore a strided view, not a fresh contiguous copy. A caller-provided out must be such a view (ValueError otherwise); nothing is staged or copied back. IndexerForward takes the contiguous padded (B, S_q, S_k_padded) buffer both at construction and in execute(), binds out[..., :S_k] directly, and never writes columns [S_k, S_k_padded).
  • Strided inputs — IndexerForward.check_support() declines q/k/w whose innermost stride is not 1 with NotImplementedError (the kernel addresses them natively; nothing is repacked), and execute() raises ValueError for such live tensors. indexer_forward_wrapper and the SM100 combined Top-K path make strided inputs contiguous on the launch stream as a torch-op convenience, at the cost of one copy kernel per strided tensor. The SM90 combined Top-K path is capture-oriented and declines non-contiguous packed Q/K/W/scales.
  • Scope — FP32 weights are supported by Indexer Forward and Combined Indexer Forward + Top-K. The separate indexer score-recompute and backward APIs still require BF16 weights.
  • Precision paths — selected by precision on indexer_forward_wrapper and indexer_forward_top_k_wrapper; the IndexerForward class is BF16-only.
    • SM90 precision="fp8": Q/K use E4M3 and q_scale/k_scale are FP32 descales with one value per token/head. Set return_lse=True (or provide lse_out) to compute LSE in the same kernel invocation.

    • SM100 precision="mxfp8": Q/K use E4M3 with block-scaled, packed E8M0 scale tensors; sf_vec_size (default None, the format’s size) must be 32, and qhead_per_kv_head ∈ {4, 8, 16, 32, 64} (the tile is always 128 packed rows — the scale atom — so a 4-head group packs 32 query tokens per tile; all-ones scale blobs make it a plain E4M3 x E4M3 scorer). THD inputs additionally require cu_seqlens_q_scale_padded/cu_seqlens_k_scale_padded: contiguous CUDA INT32 prefix tensors of shape (B + 1,) on the Q/K device. The interface deliberately performs no device-to-host copy to inspect their values. The caller guarantees that each device-side prefix starts at zero, is monotonic, covers the corresponding logical sequence, satisfies the packed-row alignment (each Q span times qhead_per_kv_head is a multiple of 128 rows; each K span is a multiple of 128 tokens), and stays within the packed scale storage.

    • SM100 precision="mxfp4": Q/K are torch.float4_e2m1fn_x2, two E2M1 values per byte with element 2i in the low nibble, so their last dim is D // 2 (64); a packed uint8 buffer passes zero-copy through .view(torch.float4_e2m1fn_x2). q_scale/k_scale are the MXFP8 E8M0 scales unchanged: one per 32 head_dim values, so D // 32 (4) per row although the last dim is D // 2, in the same packed layout, built from logical (..., D // 32) scales by the packers in cudnn.deepseek_sparse_attention.utils.sm100.mxfp8_scale_utils: pack_q_scale_bshd/pack_k_scale_bshd for BSHD, and pack_q_scale_thd/pack_k_scale_thd into a caller-provided out with prefixes from make_scale_cu_seqlens_padded for THD. sf_vec_size must be 32 (its default), and THD inputs take the same cu_seqlens_q_scale_padded/cu_seqlens_k_scale_padded scale-prefix contract. Its kernel differs from the MXFP8 one only in the MMA: tcgen05 kind::mxf4, K = 64 per MMA. Requires a PyTorch build with torch.float4_e2m1fn_x2; otherwise precision="mxfp4" raises NotImplementedError.

    • SM100 precision="nvfp4": Q/K are packed E2M1 as for MXFP4 (torch.float4_e2m1fn_x2, last dim D // 2). q_scale/k_scale are non-negative torch.float8_e4m3fn scales, one per 16 head_dim values, so D // 16 (8) per row, packed by the same packers (they move any one-byte scale) and with the same THD scale-prefix contract. sf_vec_size must be 16, its default. The per-tensor FP32 global scales of NVFP4 are not inputs: fold the product of the Q and K global decode scales into sm_scale, since a score is sm_scale * sum_h relu(q_h . k) * w_h and the ReLU commutes with a positive scale. A decode scale multiplies an E2M1 value times its E4M3 block scale back to the real value: for a global scale stored as the encoding factor g = 448 * 6 / amax, fold 1 / g. sm_scale is a launch argument, so a new global scale per step compiles nothing. Its kernel differs from the MXFP4 one only in the scales: tcgen05 kind::mxf4nvf4 with 16-value blocks (scale_vec::4X), K = 64 per MMA, reading four E4M3 scales per row per MMA. Requires a PyTorch build with torch.float4_e2m1fn_x2; otherwise precision="nvfp4" raises NotImplementedError.

      result = DSA.indexer_forward_top_k_wrapper(
      q_fp4, k_fp4, w, top_k=2048, ratio=4, precision="nvfp4",
      q_scale=pack_q_scale_bshd(q_scale_e4m3, qhead_per_kv_head=64),
      k_scale=pack_k_scale_bshd(k_scale_e4m3),
      sm_scale=sm_scale * q_decode_scale * k_decode_scale,
      )
  • Constraints — head_dim == 128. The SM90 direct path supports qhead_per_kv_head ∈ {16, 32, 64} and currently requires H_kv == 1; SM100 BF16, MXFP8, MXFP4 and NVFP4 dense and combined Top-K paths support qhead_per_kv_head ∈ {4, 8, 16, 32, 64} at a fixed 128-row MMA tile (4 heads x 32 tokens, …, 64 heads x 2 tokens), so m_block_size must be 128. All currently require H_kv == 1 (MQA). A query may be up to ratio - 1 tokens longer than ratio * seqlen_k (its trailing tokens have not completed a compressed block and score the complete blocks only); a longer one is refused.
result = DSA.indexer_forward_wrapper(
q, k, w, ratio=4, q_causal_offsets=q_causal_offsets,
)
scores = result["scores"]

4. Combined Indexer Forward + Top-K

DSA.indexer_forward_top_k_wrapper provides a one-call alternative to the two-call dense indexer_forward_wrapper + indexer_top_k_wrapper sequence when compact score generation is desired. It returns aligned indices INT32 and logits FP32, plus softmax FP32 by default, without materializing dense scores. Pass return_softmax=False to omit softmax. BSHD output shape is (B, S_q, top_k); THD output shape is (total_q, top_k). Padded slots are -1/-inf, with zero softmax. The selected set is Top-K, but its slot order is not guaranteed to be descending. Set deterministic=True to break exact-value ties at the K-th boundary toward the smallest local KV indices, making the selected set reproducible across launches. This does not sort the output slots; the default False path retains the faster scheduling-dependent tie-break.

The combined compressed path supports different validated rows on SM90 and SM100. SM90 initially supports packed THD only with ratio=4, head_dim=128, H_kv=1, qhead_per_kv_head=64, and contiguous inputs. Q/K may be BF16 with BF16 or FP32 weights, or E4M3 FP8 with FP32 per-token/head descales and BF16 weights. SM90 does not support BSHD, MXFP8/MXFP4/NVFP4, paged K, or explicit query microbatching in this path. SM100 retains both BSHD and THD with BF16 Q/K and BF16 or FP32 weights, plus MXFP8, MXFP4 or NVFP4 Q/K with BF16 weights. topk_indices_global=True is the default. Optional caller-owned candidate/output/softmax/LSE buffers avoid per-call allocations; size the candidate buffer with compress_topk_cand_buffer_size for BSHD or compress_topk_cand_buffer_size_thd for THD. SM90 LSE is supported for BF16 and FP8; SM100 LSE is supported for BSHD and THD with BF16, MXFP8, MXFP4, and NVFP4. Explicit microbatching is BF16 BSHD-only and cannot be combined with LSE or explicit q_causal_offsets. BSHD with explicit q_causal_offsets currently computes its per-batch candidate offsets eagerly and is not CUDA-graph-capturable; THD capture requires the caller-provided offsets and buffers returned by compress_topk_cand_buffer_size_thd. The same preallocation rule applies to SM90 capture; SM90 capture additionally requires topk_indices_global=False, because its optional local-to-global conversion is outside the prepared two-kernel path. Compile both kernels with an eager warmup before capture. Compact addressing requires 0 \<= q_causal_offsets[b]; rows extending beyond the KV prefix are clamped to seqlen_k_b. THD MXFP8, MXFP4 and NVFP4 use the same caller-guaranteed device-side scale-prefix contract described in §3; the interface does not validate prefix values with a device-to-host copy.

For an explicit CUDA Graph lifecycle on SM90, use DSA.prepare_indexer_forward_top_k. It compiles both stages without launching them and returns an IndexerForwardTopKPlan with a caller-owned byte-workspace contract:

cand_batch_offsets, candidate_capacity = DSA.compress_topk_cand_buffer_size_thd(
cu_seqlens_q, cu_seqlens_k, ratio=4, q_causal_offsets=q_causal_offsets
) # preparation only; keep this outside capture
plan = DSA.prepare_indexer_forward_top_k(
q, k, w,
cu_seqlens_q, cu_seqlens_k, cand_batch_offsets,
top_k=512,
max_seqlen_q=max_q,
max_seqlen_k=max_k,
max_candidate_count=candidate_capacity,
sm_scale=128**-0.5,
sample_q_causal_offsets=q_causal_offsets,
deterministic=True,
return_softmax=True,
)
workspace = torch.empty(plan.get_workspace_size(), dtype=torch.uint8, device=q.device)
plan.execute(
q, k, w,
cu_seqlens_q, cu_seqlens_k, cand_batch_offsets,
topk_indices, topk_logits,
workspace=workspace,
q_causal_offsets=q_causal_offsets,
softmax_out=topk_softmax,
)

The plan deliberately fixes only the CUDA Graph capacities: Q/K storage extents, prefix tensor lengths (segment capacity), maximum per-segment Q/K lengths, output shape, and candidate-workspace capacity. Prefix, q_causal_offsets, and cand_batch_offsets contents remain device runtime data and may be updated in place between replays. Their addresses and shapes must remain fixed; Q/K prefix totals must continue to equal the fixed Q/K storage extents, every segment must stay within the declared maxima, and the final candidate offset must not exceed max_candidate_count. The plan never reads these values to the host. It returns local per-segment KV ids. Preparation owns compiled artifacts but no device allocation; execution uses only the caller’s inputs, outputs, and workspace and does not consult the process-global compile cache. The same plan can serve concurrent calls when each call has independent outputs and workspace. The caller must order input and metadata updates before the selected stream and retain every binding until that stream finishes.

sm_scale is a plan-time host scalar applied in FP32 by the scorer. Pass raw BF16 w and place any uniform softmax/head-normalization factors in this scalar; pre-multiplying them into BF16 w adds a BF16 rounding and passing the same factor in both places scales twice. The scalar is not a compile-cache specialization, so another plan value can reuse the compiled artifact, but a CUDA Graph launch captures its value and must be recaptured when it changes.

This first plan is intentionally not a general scheduler: it accepts the same contiguous packed-THD segments as the SM90 combined kernel and does not model CP ranks, zigzag order, arbitrary Q/K storage windows, or index maps. A caller such as Megatron owns those policies and lowers them into packed prefixes and causal offsets. Use a separate graph/capacity bucket when storage extents, segment capacity, maximum lengths, or output addresses change. A workspace may be shared by serialized same-stream users only when no graph can replay them concurrently and no later consumer reads its candidate scratch.

Query-window microbatching remains a caller policy. A caller may prepare one fixed-Q-capacity plan and invoke it once per window while reusing the full K tensor. For each active window, set its q_causal_offsets to the global query start. Represent a short tail window as two packed segments: the active Q rows paired with all K rows, followed by the unused Q-capacity rows paired with zero K rows. The unused rows then receive the documented padding sentinels (indices=-1, logits=-inf, softmax=0, and lse=-inf) and may be sliced away by the caller. Serialized windows on one stream may share a workspace, but their output buffers must be distinct until all consumers finish. The same plan accepts bindings with matching descriptors and different addresses; a captured graph node retains the addresses used during that capture.

On SM90 the first kernel reuses the Hopper WGMMA scorer and changes only its output epilogue: causal-valid scores are written to the 64-bit compact candidate layout. The second kernel is the common SM90+ deterministic radix Top-K implementation. Candidate memory is still quadratic in sequence length (about half the dense ratio-4 causal score storage), so long-context callers must budget capacity explicitly. The plan does not provide an internal SM90 THD microbatch scheduler; callers compose windows as described above.

Leave microbatch_rows=-1 to use the BSHD long-sequence memory policy; it enables windowing only for compatible BF16 shapes. Pass 0 to force one launch, or a positive value only when deliberately controlling the scratch footprint. Size the buffer with the launch’s microbatch_rows and precision: compress_topk_cand_buffer_size sizes MXFP8, MXFP4 and NVFP4 launches, which never microbatch, for one full launch.

Paged K (block_table + kv_lens, the §3 contract) serves BSHD and THD queries with BF16 Q/K in one launch and returns local key ids (topk_indices_global=False), which is what serving engines consume before mapping ids to physical cache rows. Caller-owned candidate buffers are sized with compress_topk_cand_buffer_size(..., paged=True) for BSHD and compress_topk_cand_buffer_size_thd(cu_seqlens_q, None, ratio, q_causal_offsets) for THD: both return the single-launch size, and the THD call takes no key prefix because a paged pool has none. No explicit microbatching and no MXFP8 yet.

Global versus local ids is primarily an interoperability and allocation choice, not an Indexer Backward tuning choice. Keep the default global ids when a downstream consumer requires flat KV ids. If every consumer accepts local ids, prefer topk_indices_global=False. Avoiding the local-to-global conversion can slightly improve wrapper performance, particularly for short sequences. Local ids also avoid conversion temporaries and enable a strictly zero-extra-allocation captured path when all documented buffers are supplied. The global-id conversion is CUDA-graph-safe; during capture, its temporaries are recorded in the graph’s private pool. Include any downstream conversion when comparing end-to-end performance.

cand_floats = DSA.compress_topk_cand_buffer_size(
B, S_q, S_k, ratio=4, return_lse=True,
precision="bf16", # must match the launch
)
cand = torch.empty(cand_floats, dtype=torch.float32, device=q.device)
result = DSA.indexer_forward_top_k_wrapper(
q, k, w, top_k=512,
ratio=4,
cand_buffer=cand,
return_lse=True,
)
topk_indices, topk_logits = result["indices"], result["logits"]
predict = result["softmax"]
lse = result["lse"]

5. Indexer Top-K

Radix top-K kernel for selecting candidate KV indices from indexer scores, with variable per-row effective length.

  • Inputs
    • input_values: (n_rows, num_cols) FP32/FP16/BF16
    • seq_lens: (batch_size,) INT32 (per-batch effective column count)
  • Outputs — tuple (indices, values) (values is None when return_val=False). Use return_val=False when only the indices are consumed, so no values output buffer is allocated or written.
  • Tie policy — tie_break=0 (default) permits arbitrary cutoff ties; 1 prefers smaller source-column indices and 2 prefers larger indices. The selected set follows this policy. When deterministic=False, output order remains unspecified. Ties use equal radix keys: +0 ranks before -0, as in the existing kernel; NaN ordering is unspecified. The policy applies only to this standalone API.
  • Deterministic mode — deterministic=True makes the selected set and output slots repeatable. With the default tie_break=0, smaller source indices win a cutoff tie. The returned local indices are canonicalized in ascending order, carrying values with them. Selection and canonicalization are fused into this CuTe DSL kernel rather than launching a separate finalize kernel.
  • Cost — nonzero tie policies scan source columns inside the selection kernel only when equal cutoff keys compete for the remaining slots. The scan stops once enough ties are found. Deterministic mode additionally sorts the selected indices in CTA shared memory. The compiled specialization chooses between an index-bitmap scan and an in-place bitonic sort according to the target architecture and shape; on SM100, top_k < 256 uses bitonic ordering directly, and on SM103 bitonic ordering is used from 128K bucketed columns at top_k <= 128 (<= 512 with more than 148 rows). Neither path adds a kernel, device allocation, or host synchronization; the default specialization retains the existing selection path.
  • Constraints — SM90+, 0 < top_k ≤ 2048, nonzero row/column/batch extents, and next_n > 0. seq_lens is contiguous INT32 on the input CUDA device, and every entry must be in [0, num_cols]; the device-resident values are not copied to the host for validation. On SM120, deterministic=True supports at most 16,384 columns for FP32 and 32,768 columns for FP16/BF16 because of the 99 KiB per-block shared-memory limit; check_support() rejects larger deterministic shapes before compilation.
result = DSA.indexer_top_k_wrapper(
scores.reshape(-1, scores.shape[-1]),
seq_lens,
top_k=512,
deterministic=True,
)
indices, values = result["indices"], result["values"]

IndexerTopK owns no device memory. execute(input_values, seq_lens, out_indices, out_values=None, current_stream=None, workspace=None) writes into caller-owned outputs: out_indices is (n_rows, top_k) INT32 and out_values is (n_rows, top_k) in the input dtype, both contiguous and on input_values’ device. out_values is required when the object was built with return_val=True and must be None otherwise. The radix scratch is carved from workspace, a CUDA uint8 tensor of at least scratch_workspace_bytes() bytes (n_rows * (2 for FP32, else 1) * num_cols * 4, rounded up to 128) that is 32-byte aligned; an absent or undersized workspace raises before launch. The writable out_indices, optional out_values, and carved scratch byte ranges must not overlap one another or the read-only inputs. Disjoint views of the same backing allocation are valid. The class API also requires a 32-byte aligned input base; the eager wrapper copies a contiguous storage-offset view when needed to satisfy this alignment. compile() builds the kernel from fake tensors and allocates nothing. indexer_top_k_wrapper allocates the outputs and the workspace per call on the launch stream. Rows the next_n stagger leaves empty are canonicalized to -1 indices and -inf values.

Compact vs. non-compact sparse indices

Providing topk_length declares a compact input layout for Sparse Attention Backward and the sparse score-recompute kernels: in each row, the first topk_length[row] slots must contain valid ids and every later slot must be filled with -1. The padding value is required even when topk_length is provided, because the SM100 attention score-recompute wrapper may select a non-compact kernel that identifies invalid slots by their index values. Use compactify_wrapper first when invalid entries are interspersed. A 3-D input is returned flattened as (B * S_q, topk) with a (B * S_q,) length tensor; reshape both when a BSHD score-recompute wrapper expects batched shapes. The operation preserves the input id convention; it does not convert local ids to global ids.

There is no universal faster representation. Compact execution can skip sparse tiles when valid prefixes are substantially shorter than the physical top-k width, while non-compact execution can benefit from static loop bounds when rows are nearly full. The SM100 sparse attention score-recompute wrapper uses separately tuned tile policies and can internally select its non-compact path when that has better code generation. If conversion is required solely for performance, benchmark the entire compactify_wrapper + consumer sequence for the target length distribution. Indexer Backward has no topk_length argument and always processes its fixed slot width; its score tensors and ids must remain aligned slot-for-slot.

For BF16 workloads with causal sparse indices, the following are useful starting points; the best choice still depends on the shape and valid-length distribution:

  • On SM100, prefer trying the non-compact path for Sparse Attention Score Recompute when optimizing this consumer in isolation; it can be slightly faster. For Sparse Indexer Score Recompute, preserve the producer’s representation, since the performance difference is generally small.
  • On SM90, prefer compact inputs for both sparse score-recompute kernels when the valid-prefix length is already available; compact execution can be slightly faster.
  • For Sparse Attention Backward on either architecture, prefer compact inputs when the valid-prefix length is already available. The benefit may be small when most rows are full.

Retain local ids when the producer and every downstream consumer accept them. If compactify_wrapper or an id conversion must be added, time that operation together with the consumer before choosing a representation.

6. Sparse Indexer Score Recompute

Computes softmax over top-K entries of the indexer score: predict[b, q, i] = softmax_i(sum_h ReLU(Q_h · K_{topk[i]}^T) · W_h).

  • Inputs: q_indexer, k_indexer, weights, topk_indices (optional topk_length). topk_indices are per-batch local KV ids by default; pass topk_indices_global=True when using ids encoded as batch_idx * S_k + local_idx.
  • Output — predict: (B, S_q, topk) FP32.
  • Layout contract (class vs wrapper) — SparseIndexerScoreRecompute.execute() takes out as a required, contiguous FP32 tensor matching sample_out; q_indexer / k_indexer / weights need a unit innermost stride and topk_indices / topk_length must be contiguous int32. check_support() declines anything else with NotImplementedError naming the tensor and its strides, and execute() re-validates the live tensors with ValueError; nothing is copied, cast, or allocated on the execute path. A prepared SM90 plan also rejects an out whose byte range overlaps any input. The wrapper keeps the convenience: it copies strided inputs contiguous, casts the index tensors to int32, allocates out when omitted, and stages an overlapping SM90 output before copying it back, all on stream.
  • Prepared SM90 plans — compile() builds the CuTeDSL artifact from symbolic batch and sequence extents. Exact B, S_q, S_k, runtime scale, pointers, and stream do not specialize the artifact. Physical top-K widths in [1, 2176] use one of 17 bounded 128-slot capacity buckets. execute() reads the public (B, S_q, H) weights layout directly, performs no layout copy or allocation, and is CUDA Graph capturable with caller-owned out. The kernel has a fixed reduction tree and one writer per output element, so identical inputs and one compiled plan produce bitwise-identical output bytes.

7. Sparse Attn Score Recompute

L1-normalised head-summed softmax over top-K entries: target[b, q, i] = sum_h exp(Q_h · K_{topk[i]}^T · scale - LSE_h) / Z.

  • Inputs: q_attn, k_attn, lse, topk_indices, softmax_scale (optional topk_length). topk_indices are per-batch local KV ids by default; pass topk_indices_global=True when using ids encoded as batch_idx * S_k + local_idx.
  • Output — target: (B, S_q, topk) FP32.
  • Note: the wrapper handles the -log2(e) * lse preprocessing internally.
  • Layout contract — as in §6: the class requires a contiguous out at execute(), declines a non-unit innermost stride on q_attn / k_attn / lse and non-contiguous or non-int32 topk_indices / topk_length in check_support(), and never copies; the wrapper keeps the convenience copies and allocates out when omitted. On SM100 a plan built without topk_length compiles the mTopkLength operand out entirely (no placeholder tensor).
  • Prepared SM90 plans — the same symbolic compile and bounded Top-K buckets as §6 apply. softmax_scale is a runtime scalar rather than a compile key, and the kernel reads (B, S_q, H) LSE directly. A warmed class plan with its caller-owned, disjoint out is allocation-free, CUDA Graph capturable on an explicit non-default stream, and bitwise deterministic for identical inputs.

8. Dense Indexer / Dense Attn Score Recompute

Full-KV (no top-K) analogues of §6 and §7. Each returns {'out', 'denom'}. They apply the same ratio-causal mask as Indexer Forward; masked positions are written as -inf and excluded from denom. Pass the same q_causal_offsets to all dense score tensors that feed the same loss path.

On SM100, Indexer Forward and Dense Indexer Score Recompute use the same unified kernel implementation: forward runs it with compute_lse=False and passes denom=None, so the LSE slot is compiled out (no placeholder buffer), while dense indexer score recompute runs it with compute_lse=True and a live denom. The shared implementation lives in score_recompute; indexer_forward only imports it. Dense Attention Score Recompute has separate BF16 and MXFP8 kernels because its score and normalization semantics differ from the indexer path; the MXFP8 one shares the indexer kernels’ warp-specialized scaffolding and block-scale plumbing.

  • Precision — on SM100, dense_indexer_score_recompute_wrapper takes precision="bf16", "mxfp8", "mxfp4", or "nvfp4" with the Q/K and scale contract of Indexer Forward (§3); its block-scaled precisions require qhead_per_kv_head ∈ {32, 64}. dense_attn_score_recompute_wrapper takes "bf16" or "mxfp8" and rejects "mxfp4" and "nvfp4". The DenseIndexerScoreRecompute and DenseAttnScoreRecompute classes, and SM90, are BF16-only.
  • Layouts — BSHD on SM90 and SM100+. THD packed (cu_seqlens_q/cu_seqlens_k with q: (total_q, H_q, D), out: (total_q, max_seqlen_k)) on SM100+ only: SM90 declines THD in check_support() with NotImplementedError because its kernel is BSHD-native and serving THD would need a host copy of cu_seqlens per call.
  • THD launch envelope — DenseIndexerScoreRecompute / DenseAttnScoreRecompute take max_seqlen_q and max_seqlen_k as plan-time ints in __init__: max_seqlen_q is required and max_seqlen_k defaults to (and must equal) sample_out.shape[1]. execute() never derives them from cu_seqlens, and caller-passed values must match the plan’s. The wrappers derive them from cu_seqlens when omitted, at the cost of one blocking device-to-host read per call (not CUDA-graph capturable); pass both.
  • Layout contract (class vs wrapper) — execute() takes out and denom_out as required, contiguous FP32 tensors matching the samples; q / k / weights (or lse) need a unit innermost stride, and cu_seqlens_q / cu_seqlens_k / q_causal_offsets must be contiguous 1-D int32. check_support() declines strided samples with NotImplementedError naming the tensor and its strides; execute() re-validates the live tensors with ValueError. Nothing is copied, cast, or allocated on the execute path (the SM90 kernel’s per-head (B,S,H)->(B,H,S) operand transpose is the one remaining copy, tracked as a deferred kernel change). The wrappers keep the convenience: strided inputs are copied contiguous, cu_seqlens_* are cast to int32, and out / denom_out are allocated when omitted, all on stream.

9. Indexer Backward

Three-stage sparse top-K pipeline that produces the training gradients for the indexer tower:

  1. ScoreGradSm90 / ScoreGradSm100 (kernel 1) — in-place score-grad precompute that overwrites attn_score (target) and reads index_score (predict) without modifying it.
  2. IndexerBackwardSm90 / IndexerBackwardSm100 (kernel 2) — three warp-specialised GEMMs produce d_index_q, d_weights, and a dIndexK_f32 accumulator.
  3. In ordinary execution with a BF16 d_index_k, a pure-torch copy (kernel 3) casts dIndexK_f32 to BF16; an fp32 d_index_k is the accumulator itself and needs no kernel 3. On SM100-family GPUs that private BF16-path accumulator is column-permuted, so kernel 2 can issue 16-byte reductions straight from registers without SMEM staging; the same copy restores the natural order. Deterministic execution instead uses the fixed-order reduction below to write the requested output dtype directly.

IndexerBackward / indexer_backward_wrapper run on SM90 and on SM100/SM103/SM107 (SM107 needs a CuTe DSL with the sm_107a target). check_support admits D == 128, block_I == 128, a positive top-k that is a multiple of block_I (callers with other logical extents pad the top-k tensors with -1 indices and zero scores), and a finite sm_scale > 0; SM100/SM103/SM107 additionally require H in {32, 64} and SM90 supports H == 64 plus H == 32 in deterministic mode. d_index_q and d_weights are BF16; d_index_k is BF16 or fp32.

With deterministic=True on SM90, SM100, SM103 or SM107, kernel 2 uses fixed producer-plane ownership and accumulates dK into a CTA-private FP32 workspace. The compile-time deterministic_dk_planes specialization selects 96, 128, or 148 producer planes (default 96). One CuTe DSL reduction kernel folds the planes in fixed CTA order, converting only once to the requested d_index_k dtype. The dQ and dW stores remain single-writer.

The plan owns no device memory. For non-deterministic execution, an fp32 d_index_k is the accumulator itself (zeroed on the launch stream inside execute: fused into the score-grad kernel on SM100, a memset on SM90; the BF16 scratch is cleared the same way, so callers never pre-zero); a BF16 d_index_k accumulates into a B * S_k * D fp32 view carved from the caller’s workspace. IndexerBackward.scratch_workspace_bytes() reports that ordinary scratch size (0 for fp32 d_index_k). execute(..., workspace=) takes the uint8 buffer on index_q.device, and a missing or undersized buffer raises ValueError before attn_score is touched. indexer_backward_wrapper allocates it per call on the launch stream and accepts the same optional workspace= keyword for reuse; executions that share one buffer must not overlap on the device.

deterministic=True is request-or-fail and supports H in {32, 64} on SM90/SM100/SM103/SM107. All supported devices require D == 128, block_I == 128, and topk % 128 == 0 for 128 <= topk <= 2048. Each producer CTA owns one plane. SM100-family kernels process fixed grid-stride row stripes; SM90 submits bounded waves of at most deterministic_dk_planes one-row CTAs on the same stream, so a later wave cannot reuse a plane until its predecessor completes. The main GEMM therefore retains its existing Tensor Core math while dK has a fixed association order across launches. Scratch is exactly ws_align(deterministic_dk_planes * B * S_k * D * sizeof(float)). It is cleared on the launch stream at every execute, then reduced once in fixed plane order. The live scratch byte range must not overlap any input or output; execute validates this before clearing the planes, while allowing disjoint views of the same allocation. A warmed plan with caller-owned outputs and workspace is CUDA Graph compatible. To capture the convenience wrapper, warm it first and pass every output plus workspace= explicitly; omitted buffers are allocated per call. A fixed plane count is bitwise reproducible across runs; changing the plane count changes the FP32 accumulation grouping, so results from two plane counts are not required to be bitwise identical. As with the existing deterministic sparse-attention backward, valid IDs in a semantic top-k row are expected to be distinct.

The TileLang fallback present in the upstream repo is dropped here (CuTe-DSL only). If the CuTe-DSL path fails the wrapper raises RuntimeError rather than silently falling back.

grad_loss = torch.ones((), dtype=torch.float32, device=index_q.device)
result = DSA.indexer_backward_wrapper(
index_q, weights, index_k,
attn_score, index_score, topk_indices,
grad_loss=grad_loss, sm_scale=1.0, loss_coeff=1.0, block_I=128,
deterministic=True, deterministic_dk_planes=128,
)
d_index_q, d_weights, d_index_k = (
result["d_index_q"], result["d_weights"], result["d_index_k"],
)

When compressed forward returns its fused softmax, backward can skip both indexer Q@K score recompute and the separate logits softmax. Pass softmax directly as index_score; backward treats this buffer as read-only. attn_score must use the same valid-slot mask. Because compressed forward returns global indices by default, also pass topk_indices_global=True unless forward used topk_indices_global=False. The public sparse indexer_backward_wrapper has a BSHD-shaped interface; BF16 THD tensors can use zero-copy B=1 views (squeeze the singleton K head and add a batch dimension) together with global Top-K indices. FP8, MXFP8, MXFP4, and NVFP4 indexer backward are not currently supported because the backward wrapper requires BF16 Q/K/W inputs.

On SM100, topk_indices_global describes the input encoding; it is not a performance-tuning switch. Local ids are range-checked and converted to flat ids in registers before the same optimized gather path. Do not add a separate local_to_global_wrapper launch only for Indexer Backward; preserve the producer’s id convention and set topk_indices_global to match it.

On SM100, the kernel accepts compact H32 index_q, weights, d_index_q, and d_weights tensors directly. TCGEN05 still executes its minimum M64 tile internally: the upper Q rows and weights are zero-filled, and the dQ/dW stores are clipped to the logical H32 extent. Callers must not pad these tensors to H64 or slice H64 outputs. This compact path also applies to deterministic=True: fixed producer ownership changes only the dK destination and row schedule, while the same H32 input padding and output predicates remain active. Other sub-M64 head counts remain unsupported; H64 and larger head counts retain their existing behavior.

On SM90, also prefer preserving the producer’s id convention. The performance difference between local and global ids is generally small, so a separate conversion solely for Indexer Backward is unlikely to help. Deterministic SM90 uses the same fixed producer-plane ownership and fixed-order reducer described above. Its stream-ordered bounded waves also form the synchronization boundary: every one-row CTA in wave N completes before wave N+1 can reuse the same producer planes, and the fixed reducer runs only after the last wave.

10. Dense Indexer Backward

Full-KV counterpart to Indexer Backward. It consumes raw dense score tensors and denominators produced by Dense Indexer / Dense Attn Score Recompute.

  • Inputs
    • index_q: (B, S_q, H, D) BF16
    • weights: (B, S_q, H) BF16
    • index_k: (B, S_k, D) BF16
    • attn_score, index_score: (B, S_q, S_k) FP32 raw dense scores
    • attn_l1norm, index_lse: (B, S_q) FP32 denominators
    • q_causal_offsets (optional): same offsets used for the corresponding Dense Indexer / Dense Attn Score Recompute outputs.
  • Outputs — d_index_q, d_weights, d_index_k
  • Constraints — SM90, SM100, SM103 or SM107, H == 64, ratio >= 1; SM100-family GPUs accept D in {64, 128} and block_I in {64, 128}, SM90 D == 128 and block_I == 128
  • Scratch — the dK reduction targets a pre-zeroed fp32 buffer: an fp32 d_index_k in place, a BF16 d_index_k through an fp32 accumulator carved from the caller’s workspace (DenseIndexerBackward.scratch_workspace_bytes() bytes on index_q.device, 0 for fp32 d_index_k; execute(..., workspace=)). dense_indexer_backward_wrapper allocates it per call on the launch stream unless workspace= is passed; calls sharing one buffer must not overlap.
grad_loss = torch.ones((), dtype=torch.float32, device=index_q.device)
dense_index = DSA.dense_indexer_score_recompute_wrapper(
index_q, index_k.unsqueeze(2), weights,
q_causal_offsets=q_causal_offsets,
)
dense_attn = DSA.dense_attn_score_recompute_wrapper(
attn_q, attn_k, lse, softmax_scale,
q_causal_offsets=q_causal_offsets,
)
result = DSA.dense_indexer_backward_wrapper(
index_q, weights, index_k,
dense_attn["out"], dense_attn["denom"],
dense_index["out"], dense_index["denom"],
grad_loss=grad_loss, sm_scale=1.0, loss_coeff=1.0, block_I=128, ratio=1,
q_causal_offsets=q_causal_offsets,
)

11. Pooled DSA Training Backward

The pooled variant of the indexer scores pools of pool_size consecutive keys instead of single keys. Each pool key is the softmax-weighted (gates + ape) mean of its tokens; the top-K pools are selected per query, and the attention itself runs on the tokens of the selected pools plus the incomplete tail of tokens that precede the query position. The training APIs live in cudnn.deepseek_sparse_attention (lazily exported) and compose with the token-level kernels above.

APIComputesReturns
pooled_keys_forward(keys, gates, ape, pool_size)pooled keys (B, S_k // P, D) from bf16 keys (B, S_k, D), bf16 gates (B, S_k), fp32 ape (P, D); D is 128{'pooled'}
unpool_backward(keys, gates, ape, d_pooled, pool_size)gradients of pooled_keys_forward{'d_keys', 'd_gates', 'd_ape'}
pooled_indexer_select(index_q, pooled, weights, k_pools, pool_size, sm_scale)top-k_pools pools per query under the ratio-causal rule and their softmax{'pool_ids' (B, S, K) INT32, 'predict' (B, S, K) FP32, 'logits'}
pooled_expand_compact(pool_ids, positions, pool_size, seqlen_k)compact token expansion: valid pools’ tokens, then the tail, then -1{'token_ids' (B, S, W), 'topk_length' (B, S)} with W = K*P + P - 1 rounded up to a multiple of 64
pooled_sparse_attention_backward(q, kv, out, dout, lse, pool_ids, positions, pool_size)sparse attention backward over the expanded tokens{'dq', 'dkv', 'd_sink', 'token_ids', 'topk_length'}
pooled_sparse_attention_backward_with_teacher(..., k_pools, ...)the same backward, plus the pool-level teacher summed inside the kernel{'dq', 'dkv', 'd_sink', 'target' (B, S, K) FP32, 'token_ids', 'topk_length'}
pooled_teacher_recompute(q_attn, k_attn, lse, token_ids, topk_length, k_pools, pool_size)pool-level teacher from the sparse attention score recompute{'target' (B, S, K)}
pooled_indexer_target(token_target, pool_ids, pool_size)pool-level teacher from a token-level teacher (tail dropped, renormalized){'target'}
pooled_indexer_backward(index_q, weights, keys, gates, ape, pool_ids, target, grad_loss, pool_size, sm_scale, loss_coeff, pooled, predict)indexer-loss backward on pooled keys through the unpooling{'d_index_q', 'd_weights', 'd_keys', 'd_gates', 'd_ape', 'd_pooled'}

Layouts: q/out/dout are (B, S, H, D), kv is (B, S_k, D), lse is (B, S, H) FP32, pool_ids are local pool indices with -1 for unselected slots (expected last), positions (B, S) are the query positions. The teacher target is the attention probability mass of each selected pool summed over heads and renormalized over the valid pools of the query; tail tokens are excluded. pooled_indexer_backward consumes target in place and recomputes predict when it is not supplied.

The underlying kernels take two options used by this composition: sparse_attn_score_recompute_wrapper(..., pool_size=P) emits the per-pool sums directly, and SparseAttentionBackward on SM100 accepts pool_target/pool_size to emit them from the backward’s own probabilities, accumulating fp32 partials across the attention head blocks (bit-reproducible up to two blocks, i.e. 128 heads); the option is not available with deterministic=True. Supported on SM100-class GPUs with the generic single-CTA backward backend; pool_size 4 and 16 are supported.

Limitations

  • CuTe DSL requirement — install nvidia-cutlass-dsl[cu13]>=4.8.0. SM90 forward requires the loaded CUDA 13 DSL compiler flavor and rejects a CUDA 12 compiler before JIT; the package version alone is not sufficient. Sparse Attention Forward emits the few SM100 instructions that are not yet exposed by the public CuTe facade through the MLIR LLVM inline-assembly interface available in the supported CuTe DSL builds.
  • Architecture support — Sparse Attention Forward supports SM90 and the mapped SM100-family capabilities 10.0, 10.3, and 10.7. Sparse Attention Backward, Score Recompute, Indexer Forward, Indexer Top-K, and Indexer Backward support SM90 and SM100. Dense score recompute in the THD (cu_seqlens) layout is SM100+ only; SM90 serves BSHD. The combined compressed-logits + Top-K forward serves SM90 for packed THD, ratio 4, D128, and 64Q/1KV; SM100+ retains its broader configurations. The standalone Indexer Top-K remains SM90+.
  • Forward scope — Prefill only; no FP8 cache, decode, or split-KV forward path. SM90 uses one CTA per query and 64-head tile, padding the last head tile internally when H is not a multiple of 64.
  • Sparse gather path — SM90 uses predicated 16-byte cp.async gathers, matching the Hopper C++ kernel. On SM100, the public CuTe facade does not yet expose the TMA gather4 operation, so the forward uses a feature-local LLVM inline-assembly bridge with the compiler-owned descriptor; the issued data movement remains hardware TMA gather4.
  • Indexer Forward only supports head_dim = 128. SM90 supports qhead_per_kv_head ∈ {16, 32, 64} with H_kv = 1; SM100 BF16, MXFP8, MXFP4, and NVFP4 support qhead_per_kv_head ∈ {4, 8, 16, 32, 64}. Both the dense and combined Top-K paths require H_kv = 1. MXFP4 and NVFP4 run on the SM100-family capabilities 10.0, 10.3, and 10.7 (no SM90 path).
  • Standalone Top-K only up to 2048; top_k > 2048 is not supported by its radix Top-K kernel. The combined compressed path uses a separate stage-2 implementation.
  • Compressed-path limits — the stage-1 compact score kernel is MQA-only (H_kv = 1); every precision keeps its 128-row tile for every head group; explicit microbatching cannot be combined with MXFP8, MXFP4, NVFP4, LSE, or explicit per-batch causal offsets.

Native PyTorch training composition

DSA.sparse_attention composes the existing native forward and backward wrappers, without FlashMLA. It accepts contiguous BF16 q[S_q,H,D], kv[S_kv,D], INT32 indices[S_q,K], optional FP32 attn_sink[H], and optional INT32 topk_length[S_q], on one SM100 device. Supported heads are H16/H32/H64/H96/H128, each with D512 or D576; output V dimension is 512. H16/H32 forward explicitly pads heads to 64 and slices the result. This cost belongs to the semantic adapter and is included when benchmarking the call. Physical K and sequence extents must be positive. K is padded to 128 for launch.

from cudnn import DSA
result = DSA.sparse_attention(q, kv, indices, attn_sink, topk_length=lengths)
result["out"].float().square().mean().backward()
score = DSA.sparse_attention_score_recompute(
q.detach(), kv.detach(), result["lse"], indices, topk_length=lengths,
)

Only out is differentiable (Q, KV and sink, first order). lse is KV-only natural-log LSE, excluding sink; max_logits and LSE are non-differentiable. No sink means an effective all--inf sink vector. Launches and adapter tensor operations use the current PyTorch stream. First use compiles the kernels; warm before CUDA graph capture. This allocating convenience composition is not a new allocation-free APIBase execute implementation.

Safe mode maps all out-of-range indices to -1, bounds lengths to [0,K], masks inactive suffixes, and compacts valid entries when lengths are supplied. Both passes use the same normalized metadata. Duplicate indices remain distinct slots. trusted_compact_metadata=True is an explicit producer contract: bounded valid active prefixes, negative inactive suffixes, bounded lengths; without lengths every nonnegative index is in range. It skips normalization, not launch padding. Violating that contract is unsafe.

Score recompute preserves original slots, including holes, and returns both target and effective indices. It sums per-head probabilities, masks invalid slots, then normalizes across the retained slots; all-masked rows return zeros. Targets are detached. Indexer training needs its own loss; discrete Top-K selection is not differentiated. This contract uses the existing native out return key, not the old unmerged bridge’s output key.

Optional SM90 teacher-target output

sparse_attention_backward_wrapper(..., return_cli_target=True, cli_target_head_count=real_heads) appends cli_target to the returned TupleDict. The default remains the original dq, dkv, d_sink result and preserves existing positional arguments. The new settings are keyword-only.

This output is an FP32 contiguous tensor [total_S_q, topk_max], in the input Top-K slot order. It contains the mean KV attention probability over real heads, reconstructed from the existing KV-only natural-log LSE, with any attention sink included in the softmax denominator by the existing preprocess. It is independent of dout. It is not the final normalized distillation target: consumers must normalize over their valid selected positions. Invalid slots, ignored suffixes and empty rows are zero. Duplicate indices retain the existing attention semantics; they are not deduplicated by this API.

The initial implementation supports SM90 and one physical 64-head block, with 1 \<= real_heads \<= 64. A caller padding smaller head counts must supply zero padded queries/upstream gradients and positive-infinity padded LSE values. It must use the matching real-head count. SM100-family and other architectures decline this optional output; their existing backward paths are unchanged.

The wrapper allocates the output on the launch stream. Direct SparseAttentionBackward plans take return_cli_target and cli_target_head_count at construction, then require a matching preallocated cli_target keyword output in execute(). It must be contiguous FP32 on Q’s device with the Top-K shape. Plans never allocate this output or change their output contract during execution. Inputs, outputs and workspace must not alias. Warm SM90 once before graph capture, as for ordinary sparse backward.

CLI consumes probabilities already reconstructed by the backward kernel; it does not change sparse attention forward or the backward gradient equations. Target reference, gradient parity, empty-row/replay and plan-contract tests live in test/python/deepseek_sparse_attention/cutedsl/test_DSA_cli_target.py. GPU validation of this rebased revision is pending; downstream results on older source revisions are not evidence for this revision.