DeepSeek Sparse Attention (DSA)
This is an experimental API and subject to change.
Stream parameter types are operation-specific. Sparse-attention forward uses
the current PyTorch stream when its stream argument is None; an explicit
stream must be a concrete cuda.bindings.driver.CUstream. Convert a
torch.cuda.Stream with cuda.bindings.driver.CUstream(stream.cuda_stream).
The special CUstream(2) per-thread default sentinel is rejected by Torch
interop: it cannot safely represent an allocator stream identity across host
threads. Use a concrete stream instead.
Overview
The DeepSeek Sparse Attention (DSA) module integrates a set of CuTe-DSL
kernels that support the sparse-attention path used by DeepSeek-style models.
Most kernels target Hopper (SM90) and Blackwell (SM100+) GPUs. Sparse-attention
forward supports SM90 and the mapped SM100-family capabilities 10.0, 10.3,
and 10.7; the combined compressed-logits + Top-K path serves a validated
packed-THD configuration on SM90 and broader configurations on SM100+.
Dense Indexer Forward and standalone Indexer Top-K support both architectures.
The kernels are delivered as Python classes /
wrappers that follow the same APIBase pattern as other cuDNN Frontend
operations.
Scope: this module ships CuTe-DSL kernels for sparse-attention forward and backward, indexer scores/top-K, sparse/dense score recompute, and sparse/dense indexer backward. On SM90, sparse forward covers H16/H32/H64/H96/H128 Prefill; on SM100 it covers regular H64/H96/H128 and small-top-k H128 Prefill. It does not include a decode or split-KV forward mode.
The module packages the following operations:
- Sparse Attention Forward – sparse Prefill MQA on SM90 and SM100.
- Sparse Attention Backward – DSA backward for flat MQA tensors on SM90/SM100.
- Indexer Forward – CuTe-DSL score kernel (Q @ K^T, ReLU, head reduce, ratio causal mask) that materializes dense scores.
- Combined Indexer Forward + Top-K – SM90/SM100 compact score generation, Top-K selection, and optional Top-K softmax in one public API call.
- Indexer Top-K – SM90+ CuTe-DSL radix top-K kernel with per-row
seq_lens. - Sparse Indexer / Attention Score Recompute – sparse (top-K) recompute of indexer and attention scores for training loss.
- Dense Indexer / Attention Score Recompute – dense (full-KV) analogues of the above.
- Indexer Backward – three-stage pipeline (score-grad, three GEMMs, dtype cast) for sparse top-K score tensors.
- Dense Indexer Backward – full-KV counterpart of Indexer Backward.
Architecture
Installation
API Usage
JAX ops (SM100 DSv4)
cudnn.ops.dsa exports these JAX ops. They accept JAX arrays, eagerly or under
jax.jit, require no PyTorch, and run on SM100-family GPUs. Forward and backward
are separate calls; the caller owns the VJP. Each has a PyTorch counterpart in
cudnn.ops.dsa, the same name without _jax, which is the listed DSA wrapper:
compressor_forward and compressor_backward are the CSA compressor wrappers in
csa.md.
DSA Namespace
Components
1. Sparse Attention Forward
Sparse Prefill forward for flat MQA tensors. For query row t, head h, and
valid top-k slot j, the score is
softmax_scale * dot(q[t,h,:], kv[topk_idxs[t,j],:]). Invalid and OOB indices
have score -inf; duplicate indices remain distinct softmax slots. The
attention sink participates only in the output denominator, not in any stats.
- Inputs
q:(total_S_q, H, D_qk)FP16 or BF16kv:(total_S_kv, D_qk), with the same dtype asq(K = V latent; MQA)topk_idxs:(total_S_q, K)INT32 global token ids; K may be any nonnegative value. SM100 pads a tail internally; SM90 masks it in place.attn_sink(optional):(H,)FP32topk_length(optional):(total_S_q,)INT32, clamped to[0, K]softmax_scale(optional runtime float): defaults to1 / sqrt(D_qk)indexer_topk: compile-time prefix length
- Outputs — stable
TupleDictkeys:out:(total_S_q, H, 512), with the same dtype asqmax_logits:(total_S_q, H)FP32lse:(total_S_q, H)FP32, KV-only and excluding the sinklse_indexer:(total_S_q, H)FP32, orNoneforindexer_topk=0
- Variants
- SM90:
H in {16, 32, 64, 96, 128},D_qk in {512, 576}, andindexer_topk in {0, 512, 1024, 2048} - SM100 H64:
D_qk in {512, 576},indexer_topk in {0, 512, 1024, 2048} - SM100 H96: as H64. It launches the H64 kernel twice, on heads
[0, 64)and[32, 96)of the same tensors, so it does the work of 128 heads; both launches write identical bytes to heads[32, 64). - SM100 H128 small-top-k Prefill:
D_qk=512,indexer_topk in {0, 512, 1024} - SM100 H128 otherwise (
D_qk=576, orD_qk=512withindexer_topk=2048): the H64 kernel runs twice, on the disjoint heads[0, 64)and[64, 128), and each head’s bytes equal an H64 run on that window. ForD_qk=512,indexer_topk=2048therefore uses a different kernel than{0, 512, 1024}, and results can differ in rounding between the two.
- SM90:
- Indexer prefix constraint —
indexer_topk <= K. Whentopk_lengthis present, a meaningfullse_indexeradditionally requirestopk_length >= indexer_topkfor that row. Shorter rows retain the SM100 source kernel’s+infsentinel; theirout,max_logits, and fulllseoutputs remain valid. - Empty rows —
out=0,max_logits=-inf,lse=+inf; an enabledlse_indexeris also+inf. - Compilation lifecycle — on SM90,
SparseAttentionForward.compile()compiles a storage-free symbolic plan andexecute()only launches it with caller-owned outputs. The convenience wrapper allocates outputs and stages layouts before calling the same plan. The existing SM100 direct path retains its concrete-tensor cache and first-execute compilation. - CUDA Graph and workspace — the prepared SM90
execute()path is allocation-free, performs no host read or JIT, and requires no workspace. Compile before capture and provide preallocated outputs. Runtimetotal_S_q,total_S_kv, and logical K may change perexecute(). H, Dqk, dtypes, and optional-operand presence must match the declaration; a plan for another supported H reuses the same compiled kernel without a new specialization. - Aliasing — prepared SM90 outputs must be byte-range-disjoint from every input and from one another. Empty tensors occupy no byte range.
- Layout and stream — the convenience wrapper materializes non-contiguous
or under-aligned inputs on the selected stream. Prepared SM90
execute()instead requires contiguous, adequately aligned tensors and raises before launch if the contract is not met. An explicitcurrent_streamfor prepared execution (orstreamfor the wrapper) must be acuda.bindings.driver.CUstreambelonging toq.device.
For D_qk=576, QK uses all 576 dimensions while V/output uses only the first
512 KV dimensions. attn_sink=None is equivalent to an all--inf sink.
Credits and cuDNN Frontend extensions
The SM90 kernel schedule and pipeline are adapted primarily from FlashMLA’s MIT-licensed Hopper sparse-prefill implementation. The cuDNN Frontend port extends that foundation with H16/H32/H64/H96/H128 coverage, arbitrary logical K tails and boundary handling, a bounded symbolic compile cache, and an allocation-free prepared-execution contract for CUDA Graph capture and replay.
2. Sparse Attention Backward
Backward pass for DeepSeek Sparse Attention. Expects the forward wrapper’s
out and KV-only lse (or equivalent tensors).
- Inputs
q:(total_S_q, H, D)BF16/FP16kv:(total_S_kv, D)(K = V; MQA)out,dout:(total_S_q, H, D_v)lse:(total_S_q, H)FP32attn_sink:(H,)FP32topk_idxs:(total_S_q, topk_max)INT32 (global)topk_length(optional):(total_S_q,)INT32 — per-query valid count. The H128 two-CTA backends (D512 and D576) clamp it to[0, topk_max]and ignore every slot after that prefix, including valid indices pointing to nonfinite KV rows; the other backends expecttopk_length \<= topk_max. Omitting this tensor uses alltopk_maxslots.
On Blackwell SM100/SM103, the public backward entry point automatically selects
the tuned kernel from the device, dtype, and tensor shape. On SM100 (10, 0) and
SM103 (10, 3) devices, BF16 H128 with head_dim = head_dim_v = 512 and
topk_max ∈ {128, 512, 1024, 1152, 2048} uses the two-CTA specialization.
Contiguous BF16 H128 with head_dim = 576, head_dim_v = 512, and the same
topk_max set uses the H128/D576 two-CTA specialization. H16 with
head_dim=576 uses the dedicated M128 sparse-row pipeline. FP16, other head
counts and dimensions, and every other topk_max retain the existing
generic/H16/H32 selection. Other compute capabilities, including SM107, do not
select the two-CTA paths. No backend or tile-size argument is required. SM90
continues to use its Hopper-specific implementation. Every input must be
contiguous on every route: check_support() declines a strided tensor with
NotImplementedError naming it and its strides, and execute() rejects one
with ValueError; the plan never repacks an input.
The SM100 H16/H32/H96 D576 specializations compile the physical row stride of
topk_idxs from topk_max, while the query count and per-query topk_length
remain dynamic. H64 and the generic/H128 D512 paths retain their existing
layout; H128 D576 already uses a static topk_max stride.
For contiguous BF16 H16/H32 D576 inputs, q_cluster_mode="adaptive_pair"
enables adjacent-query KV sharing. A device-side transform partitions each
pair into one tile-aligned intersection and two unique segments, then evaluates
the intersection with twice the head count. Its bounded hash storage scales
with topk_max rather than total_S_kv; invalid, duplicate, low-overlap, and
an odd final query retain the ordinary per-query work. The mode is opt-in,
non-deterministic, CUDA Graph compatible, and supports topk_max \<= 2048.
It is intended for high-overlap workloads; the device-side partition still
runs when a pair falls back to ordinary work. H16 topk_max \<= 256 and H32
topk_max \<= 128 accept the mode but compile the ordinary backend directly.
With the default deterministic=False, the H128 specialization keeps the five tensor-core products in one
two-CTA main kernel. It publishes FP32 O-dot-dO and folded-LSE statistics to the
caller-provided scratch workspace, converts the FP32 dKV workspace to the public BF16
output, and completes dSink with a separate FP32 reduction kernel. The helper
launches do not change the two-CTA topology of the core computation.
The H128/D576 specialization applies the same two-CTA topology to the MLA
layout (head_dim = 576, head_dim_v = 512). Its compiled sequence
with deterministic=False initializes the FP32 dKV workspace and the dSink output on the launch stream,
accumulates dKV in that workspace, and converts it to the public BF16 output
with a helper kernel. With the default deterministic=False, short query
batches split each query’s top-k range across otherwise idle two-SM clusters;
the split count is fixed when the plan is compiled.
On SM90, deterministic=True and the default
deterministic_dkv_mode='records' use conflict-rank dKV storage for
H16/H32/H64/H96/H128. A setup kernel builds an immutable 128-bit
query-selection bitmap and four zero-initialized ticket words for every
referenced (query wave, M64 head tile, KV row). Each query computes its stable
conflict rank by popcounting lower query ordinals. Ranks zero through three
accumulate into four separate dense FP32 planes. Higher ranks reuse plane zero
after all lower plane-zero writers for that row publish monotonically increasing
release tickets. On a resident 128-CTA wave, D512 uses four independent column groups
and D576 uses two, so adjacent overflow ranks can pipeline disjoint columns
instead of serializing the complete row update. A multi-launch schedule keeps
one ticket per row because its phases are already stream ordered. Ranks one
through three require no ticket because their planes are disjoint. A prepared
plan reads the device SM count once and divides each wave into cooperative
power-of-two phases with at most one CTA per SM. Cooperative launch guarantees
that every writer a CTA can wait for is resident; same-stream phase order
carries completed tickets across launches. The selected phase count is compiled
into the plan and is not queried during execution or CUDA Graph replay.
Head tiles and later waves remain stream ordered, and
the postprocess kernel combines the four planes in fixed order. One warp per
head computes d_sink with a fixed strided-lane/tree reduction order.
Duplicate valid top-k indices within one query row remain outside this
deterministic contract.
The conflict-rank dKV allocation is
4 * (K64 + mask_rows) * D32 * sizeof(float), where
K64 = round_up(total_S_kv, 64), D32 = round_up(D, 32), and
mask_rows = ceil(2 * ceil(total_S_q / 128) * ceil(H / 64) * K64 / D32).
The factor two stores the immutable bitmap and four 32-bit column-phase ticket
words; both groups are striped across the four planes.
An explicit deterministic_relay_cluster_size=C, for
C in {4, 8, 16, 32, 64}, selects the previous 'shards' cluster relay even
when the requested dKV mode is 'records'. Each 128-query wave then gives one
FP32 dKV plane to every logical relay group. The plan queries CUDA cluster
occupancy and uses the widest launchable physical C16, C8, or C4 cluster no
larger than the logical relay; wider relays use ordered same-stream phases.
The setup kernel builds packed
C-bit ownership masks, relay CTAs wait only for lower ranks that update the
same row, and the postprocess combines 128 / C planes. Its allocation is
(128 / C) * (K64 + mask_rows) * D32 * sizeof(float), with
mask_rows = ceil(ceil(total_S_q / 128) * ceil(H / 64) * (K64 / (mask_bits / C)) * (mask_bits / 32) / D32) and mask_bits equal to
64 for C64 and 32 otherwise.
The caller-owned workspace is cleared and the masks are rebuilt on every
execution, so changing topk_idxs or topk_length between launches is safe.
After one eager execution populates the SM90 concrete-tensor compile caches,
the caller-owned output/workspace path is CUDA Graph capturable. Replay keeps
the declared tensor shapes and addresses fixed while allowing tensor contents,
including topk_idxs and topk_length, to change.
On H20-3e with BF16 H64/D512, Q=16384, KV=262144, and top-k 2048, conflict-rank
records measured 119.223 ms and 3.008 GiB versus 484.915 ms and 4.508 GiB for
the C16 logical relay baseline split into physical C8 launches: 4.07x faster
with 33.3% less workspace. Historical explicit-relay H200 measurements and the
full environments are recorded in benchmark/dsa/README.md.
On the AIStudio H200 allocation (runtime name NVIDIA L20X) with BF16 H64,
Q=16384, KV=262144, and top-k 2048, the single-wave records schedule measured
69.147—69.217 ms for D512 and 83.164—83.285 ms for D576. The existing C16
relay measured 84.643 and 100.126 ms respectively; the H200-tuned C32 relay
measured 75.936—75.970 and 94.546 ms. Each result uses five warmups and 20
allocation-free calls. Records are 1.224x faster than C16 and 1.098x faster
than C32 for D512, and 1.204x faster than C16 and 1.137x faster than C32 for
D576. When every query shares the same 2048 KV rows, column-phase tickets
reduce D512 from 151.586 to 87.678 ms (1.73x) and D576 from 195.635 to
139.779 ms (1.40x) relative to the previous single-ticket records path.
On SM100/SM103, deterministic=True retains the two-CTA Tensor Core kernels
for the supported BF16 H128/D512 and H128/D576 envelopes above. Queries run in
same-stream waves with one two-CTA pair per query: D512 spreads the queries
evenly over waves of at most 74, and D576 waves hold one query per two-SM
cluster (at most 76) per wave round, at most 74 per round in three-round plans
(devices with 152 or more SMs leave the remaining SMs idle there). Each wave
accumulates dKV either in private FP32 slots or in conflict-rank storage
(below); ordered duplicate-index writers and stream ordering between waves give
each update a fixed accumulation order, and fixed-order reductions produce
dKV and d_sink. With either H128 two-CTA layout, duplicate indices within a
query row are inside the deterministic contract, as with the generic M64
conflict-rank layouts (below) and unlike its shard and relay layouts and the
SM90 deterministic path. The plan option deterministic_dkv_mode (below)
picks the layout of H128 two-CTA plans: under 'records' (the default) the
plan chooses between the two layouts from the shapes, 'shards' keeps the
private-plane layout for every shape, and 'chain' is rejected. The
private-plane layouts hold 128 FP32 dKV planes for D512 and 96 for D576; H96
D576 two-CTA plans always use their 96-plane layout. 'shards' restores the
private-plane layout but, for D512, not the bits of releases without
conflict-rank storage: D512 private-plane plans stream the dK/dV round operands
in four 32-head stages, which sums dKV in a different fixed order (dQ and
d_sink are unchanged).
Conflict-rank dKV storage (deterministic H128 D512/D576). Within a wave,
the queries that select a KV row are ranked by their position in the wave. The
query of rank k \< P accumulates into dense FP32 plane k (the first write of
a row above plane 0 stores, later ones add); higher ranks write compact
overflow records, so every plane row and every record has a single writer per
wave and the main kernel needs no writer ordering. A row that one query selects
more than once takes per-slot records in every slot of that wave that selects
it.
A per-wave tail adds the overflow records to plane 0 in a fixed order (rank
order, with long chains summed in fixed segments) and the per-slot records in
(query, slot) order, with sorted slots for D576 top-k 1024 and wider; the final
fold adds each row’s used planes in plane order. Small setup kernels chained
to the main kernels with programmatic dependent launch compute each wave’s
ranks and destination records before it runs. D512 plans with top-k 512 or
less that keep at least three dense planes next to a doubled overflow area run
these steps in a setup warp in each CTA of the preceding main kernel instead,
and each wave’s tail in the next main kernel’s reducer warps, which shift a
row’s ranks off plane 0 while that tail is pending.
- When it is selected — in
'records'plans (the default;'shards'plans keep the private-plane layout), automatically, from the KV extent and the top-k width, whenever the conflict-rank workspace fits the private-plane layout (D576 at top-k 128 also requires the layout sized for its expected conflict load to need at most three quarters of it). Short KV extents relative to the top-k width, where the private planes are already small, keep the private-plane layout. The choice depends only on the batch’s shapes and the mode, so one problem always takes the same path. - Workspace —
scratch_workspace_bytes()reports the selected layout; it never exceeds the private-plane size of the same problem. The dense planes fill a budget of about 1 GiB, but at least one plane (and the per-row state) is kept, so past a few hundred thousand KV rows the workspace grows with the KV extent: attotal_S_q=4096about 2.1-2.4 GiB per million KV rows for D512 and 2.5-2.7 GiB for D576, about 1% of the private-plane layout (256 and 216 GiB per million rows). The D576 size also depends on the plan device’s SM count, through the cluster pairs each wave round launches: queryscratch_workspace_bytes()for the device that executes the plan, since a size from a device with fewer SMs can be too small (execute()then rejects the workspace). Every execution re-initializes the conflict-rank state it reads, so one workspace can be reused across executions with differenttopk_idxsandtopk_length, and a captured plan replays new input contents like the private-plane plans. - Determinism — outputs are bitwise reproducible from run to run on one
GPU and software runtime. Duplicate indices within a query row,
-1and out-of-range indices, and anytopk_lengthare inside this contract. The FP32 summation order is a fixed function of the plan and the indices: D512 waves follow the query count, D576 waves also the device’s SM count, and D576 one-round plans wider than top-k 128 that would stride their waves first run a probe launch that counts one wave’s writers and picks strided or consecutive waves from the index contents. Results are not claimed to be bitwise equal across GPUs or software versions. Conflict-rank and private-plane plans sum dKV andd_sinkin different orders; their results agree within FP32 rounding, not bitwise. dQ is bitwise equal in both layouts except for D576 conflict-rank plans with top-k 1024 and wider, which sort each query’s rows by ranges of the KV extent. A sequence’s dKV is not batch invariant: packing it with other sequences changes the waves, the ranks and the plane count that order its sums, and the sorted D576 plans also change its dQ.d_sinkfollows the query positions in both layouts (see Determinism Support).
Other deterministic envelopes with H16/H32/H64/H96/H128 run the generic M64
kernel. Its dKV storage is a plan-time choice, deterministic_dkv_mode
(below): by default on SM100/SM103, conflict-rank records wherever their
workspace does not exceed the shard or relay layout of the same plan, and that
layout elsewhere and on other devices. H16/H32 use a masked M64 head tile.
Deterministic H96 D576 outside the two-CTA envelope (FP16, other top-k widths,
or other devices) runs the generic M64 kernel only with conflict-rank storage,
so only on SM100/SM103; every other such plan composes the H64 kernel with the
tuned H32 tail, keeps 128 FP32 dKV planes, and rejects an explicit
deterministic_relay_cluster_size.
Deterministic dKV storage (deterministic_dkv_mode). The option of
SparseAttentionBackward and sparse_attention_backward_wrapper selects the
dKV storage of deterministic SM100/SM103 plans of the generic M64 kernel as
follows, and of the H128 two-CTA plans as described above:
'records'(default) — conflict-rank storage. Each query wave runs 148 query CTAs per head block (74 for 64 < H <= 128 when a full wave’s records would exceed 0.4 GiB, and for split pairs, below). Within a wave, the queries that select a KV row are ranked by their wave position: rankkbelow the plan’s plane countPaccumulates into dense FP32 planekof the row, higher ranks into slot-private overflow records, so every plane row and every record has one writer per wave. Setup kernels before each main kernel assign the wave’s destinations and add the previous wave’s overflow records to plane 0 in rank order (long runs in fixed segments); a row that one query selects more than once takes per-slot records, added in (query, slot) order. The conversion folds each row’s used planes in plane order.Pfollows the expected number of writers per row and wave (W * top-k / total_S_kvforWquery CTAs per wave) plus a margin, within about 1 GiB of planes. H96/H128 run both 64-head blocks of a query as one two-CTA cluster: head block 1 adds a tile after head block 0’s reducer threads published their adds of that tile through per-thread GPU-scope release/acquire completion words. H <= 64 plans whose full-wave workspace exceeds 1.65 times the larger of the half-wave workspace and 1 GiB run each query as a two-CTA pair that splits its top-k tiles; the second CTA hands its FP32 dQ partial to the first through the workspace. Plans with at least eight waves whose KV rows fit in 48 MiB interleave the waves’ queries (wavewofnholds queriesw, w + n, ...).'chain'— the chain relay. Every head block of a wave runs in one launch of 296 query positions. CTAs claim (position, head block) tickets in start order and add into a shared plane row only after the row’s previous chain member signalled it, so each plane row accumulates in a fixed order without overflow records. It applies only when the top-k width is at most 2048, the plan’s dense planes span at most 2^19 rows (about 174K KV rows at its floor of three planes), and its workspace does not exceed the'shards'layout (for H96 D576, the 128-plane layout of the H64 + H32 composition); every other plan follows the'records'rule. In the table below, chain plans need less workspace than records plans; which mode is faster depends on how much the queries’ top-k rows overlap, so profile both on the target workload.'shards'— the shard layout or the cluster relay described below, chosen from the query and KV extents alone, with the kernels, workspace and bitwise results of releases without this option.
Further properties of the option:
- Validation — values other than the default require
deterministic=Trueand unknown values are rejected.'chain'cannot be combined withdeterministic_relay_cluster_sizeand is rejected on SM90 and on the two-CTA backends. On the H128 two-CTA backends'records'chooses between conflict-rank and private-plane storage from the shapes and'shards'selects the private-plane layout; H96 D576 two-CTA plans keep their 96-plane layout under both. Under'records'and'shards', an explicit relay size selects that cluster relay. The option is part of the compiled plan and of the wrapper’s plan cache key. - Devices — conflict-rank storage (the
'records'and'chain'layouts and the H128 two-CTA conflict-rank layout) is provided on SM100 and SM103 (compute capability 10.0 and 10.3). On other SM100-family devices (compute capability 10.7), a'records'plan runs as a'shards'plan, with the same kernels, workspace and results, and'chain'is rejected with aValueErrorthat names the compute capability.flash_attn_bwd_sm100_workspace_size()takes the keyword-onlymax_topk,launch_pairs,dkv_modeanddevice_capability: withoutmax_topkit returns the shard, relay or private-plane size, an upper bound for every conflict-rank plan of the shape, and the defaultdevice_capability=Nonesizes for an SM100/SM103 device. - Workspace —
scratch_workspace_bytes()reports the selected layout and is authoritative. It never exceeds the'shards'layout of the same plan (for generic H96 D576 plans, the 128-plane layout of the H64 + H32 composition that their'shards'plans run). A records plan sizes a full wave even whentotal_S_qis smaller, so its workspace does not shrink further with the query count. Every execution re-initializes the state it reads, so one workspace serves executions and CUDA Graph replays with differenttopk_idxsandtopk_length. - Determinism — each mode is bitwise reproducible from run to run on one
GPU and software runtime.
-1and out-of-range indices andtopk_lengthvalues up totopk_maxare inside the contract of every layout; records and chain plans also clamp larger lengths totopk_max. Duplicate indices within a query row are inside the contract of the records and chain layouts but not of the shard and relay layouts, which the other modes also select for some shapes; callers that need bitwise repeatability for every shape should pass unique valid indices within each query’s active top-k prefix. The layouts sum dKV in different orders, so their dKV agrees within FP32 rounding, not bitwise.d_sinkhas the same bits under every layout of the generic M64 kernel: one warp per head sums the queries in the same order (the H64 + H32 composition of H96 D576 is a separate kernel). dQ does not depend on the dKV layout either, except for chain plans, which gather a query’s rows in sorted order, and records plans with split pairs, which add two dQ partials. dKV is not batch invariant once a batch spans several waves, with any layout: the layout, the waves and the plane or shard count follow the batch’s query and KV extents.d_sinkis: a uniform shift of the queries only rotates the lanes of that warp (see Determinism Support). - Compilation — the wave size and the split and interleave choices are
compile-time constants of a records plan. For given heads, head dimension,
dtype, top-k width and KV extent, the wave order gives records plans two
compiled variants over the query count (interleaved or not), and 64-bit
addressing, which large query extents need, adds another; the storage a plan
selects depends on its query and KV extents. Each plan still compiles only
in
compile(). - Launches — conflict-rank plans chain their kernels with programmatic
dependent launch. A records wave launches two setup kernels and its main
kernel; a chain wave does the same, with one more scan for the first wave,
and computes O.dO in its setup. Each execution adds an initialization, the
last wave’s setup, the fold and the
d_sinkreduction (records plans also launch a first-wave marking kernel and the O.dO kernel). The last grid of an execution completes only after the fold, so the next launch on the stream observes the finaldKV. The 148-CTA wave matches the SM count of B200. The wave and setup-grid sizes are plan constants that do not depend on the device’s SM count, and correctness relies only on the two CTAs of a cluster being co-resident.
Scratch (scratch_workspace_bytes()) of representative BF16 plans:
Shard and relay layouts. deterministic_dkv_mode='shards', an explicit
relay size, and shapes whose conflict-rank workspace would be larger use the
bounded-wave shard layout. Ordinary extents select 32 dKV shards for
total_S_q \<= 128, 64 for 129..256, and 128 for larger query batches.
Queries run in same-stream waves with one CTA per shard; CTA lane i is the
sole writer of FP32 dKV shard i in each wave. H96/H128 split each query wave
into ordered M64 head-block launches, preserving single-writer ownership
without multiplying the number of shards. Kernel launch ordering serializes
both head blocks and shard reuse across waves. The 32-shard specialization
converts with one fixed-order reduction; the 64- and 128-shard
specializations first fold groups of eight and then combine those groups in
fixed order.
If the selected shard layout would exceed 16 GiB, the generic M64 plan
keeps each query wave at 128 CTAs and assigns one FP32 dKV plane to each
cluster. Queries below 8192 rows prefer sixteen eight-CTA clusters; larger
queries prefer thirty-two four-CTA clusters for additional reducer parallelism.
If that preferred layout’s complete FP32-plane and packed-mask allocation would
exceed 32 GiB, the plan selects an eight-plane, sixteen-CTA memory-first relay.
The long-query policy deliberately skips cluster-8: GB200 sweeps at 393K, 512K,
and 768K KV found it slower than cluster-16 while retaining twice as many FP32
planes. A setup kernel builds packed 4-, 8-, 16-, 32-, or 64-bit rank masks for
every (query wave, head block, KV row). Reducer threads wait only for lower cluster
ranks that actually update the same KV row, then publish that row with
GPU-scope release/acquire ordering. The participating reducer-thread CTA
barrier carries the release/acquire causality, so no additional GPU-wide fence
is needed. Rank zero skips the predecessor handoff, the last rank skips the
successor publication, and the final release needs no second rendezvous.
Disjoint sparse rows continue in parallel; the fixed rank order handles both
low- and high-overlap top-k rows without a global semaphore dependency on
non-resident CTAs. The selected 2, 4, 8, 16, or 32 planes are combined directly in
the fixed-order conversion kernel, without an intermediate FP32 prefold.
On SM100/SM103 under the default 'records' mode, the automatic relay runs
only where records do not apply: top-k widths above 14169 (a records wave
counts its 148 * top-k slots in a 21-bit field) and shapes whose records
would need more workspace than the relay.
Duplicate top-k indices within one query row remain outside the deterministic
contract of the shard and relay layouts. This condition is not checked at
runtime; callers that require bitwise repeatability must provide unique valid
indices within each query’s active top-k prefix.
Callers that have profiled their exact SM100 workload may set
deterministic_relay_cluster_size to 4, 8, 16, 32, or 64 to select
that many CTAs per logical relay group instead of using the automatic policy.
The corresponding number of FP32 relay planes is 32, 16, 8, 4, or
2. Because SM100 hardware clusters contain at most 16 CTAs, the logical
32- and 64-CTA groups run as two and four same-stream 16-CTA phases,
respectively. Each kernel boundary completes the preceding physical phase
before the next ranks reuse the same plane, avoiding cross-cluster residency
dependencies. Cluster-64 uses a 64-bit packed row mask and is intended as an
explicit emergency memory-first choice; the automatic policy never selects
it because its lower per-phase parallelism regresses profiled workloads. This
is a plan-time specialization: it is part of the compile/cache key, fixes the workspace size,
and remains unchanged during execute() and CUDA Graph replay. The default
None preserves automatic selection. Requests on non-deterministic plans,
non-generic backends (including deterministic H96 D576 plans that keep the
H64 + H32 composition), shapes that do not require the large-context relay,
and deterministic_dkv_mode='chain' plans are rejected rather than silently
ignored. An explicit choice may exceed the
automatic policy’s 32 GiB memory-first budget, so callers should always allocate
the exact value returned by scratch_workspace_bytes().
For the shard layout, additional dKV workspace is
selected_shards * K8 * D8 * sizeof(float), where
K8 = round_up(total_S_kv, 8) and D8 = round_up(D, 8). The cluster relay
uses relay_planes * (K8 + mask_rows) * D8 * sizeof(float), where
relay_planes = 128 / relay_width. The packed mask uses a 32-bit word for
relay widths up to 32 and a 64-bit word for relay width 64. With
mask_word_bits equal to 32 or 64, its FP32 storage rows are
mask_rows = ceil(ceil(total_S_q / 128) * ceil(H / 64) * ceil(K8 / (mask_word_bits / relay_width)) * (mask_word_bits / 32) / D8).
The conflict-rank layouts depend on the plane count, the wave and the top-k
width. One CTA per head reduces d_sink. Use scratch_workspace_bytes() as the
authoritative full scratch size; a larger reusable workspace remains valid.
Treat this as a reproducibility requirement rather than a performance-tuning
knob: keep the default False when bitwise run-to-run stability is not needed.
The measurements below compare the shard and relay layouts
(deterministic_dkv_mode='shards') with the fixed 128-plane layout. The
H128/D576 two-CTA row measures its private-plane layout, which 'shards'
plans keep; a default plan of that shape stores dKV in conflict-rank storage
(0.916 GiB of scratch on a 148-SM B200; the size follows the SM count).
On GB200 with BF16, total_S_q=16384, top_k=2048, H=64, D_qk=576, and
D_v=512, the memory-first cluster-16 route reduces full scratch from
288.008 GiB to 20.008 GiB at total_S_kv=1048576. The upstream allocation
does not fit on the 185.0 GiB test GPU. At total_S_kv=524288, where both
implementations fit, scratch falls from 144.008 GiB to 10.008 GiB and warmed
execute() latency changes from 99.973 ms to 98.999 ms.
At total_S_kv=1048576 with CP64 (total_S_q=16384 per launch), BF16,
top-k 2048, D576/Dv512, and contiguous inputs, the official upstream baseline
and this change require:
The upstream 128-plane baseline does not fit in the test system’s 185.0 GiB usable memory, so its 1M-KV latency cannot be measured. The following direct A/B covers every deterministic dispatch selected by the adaptive policy, plus the H128/D576 two-CTA private-plane control. All rows use BF16, D576/Dv512, top-k 2048, and contiguous inputs; H is 64 except for the H128 control.
Latency values are medians of three alternating-order A/B rounds, each with
three warm-ups and 20 timed, allocation-free execute() calls. The unchanged
ordinary and two-CTA routes stay within 0.3%. Cluster-8 and cluster-4 trade
39.2% and 15.8% latency, respectively, for substantial workspace reduction;
at 512K KV, cluster-16 is 1.0% faster than upstream while using 93.1% less
scratch.
The follow-up relay scheduling pass was measured against merged commit
1dea4471a6e6c05e29d3bf281281fd156d345a84 on the same SM100 environment with
PyTorch 2.14.1+cu130, CuTe DSL 4.8.1, cuDNN 9.24.0, BF16 H64, top-k 2048,
five warm-ups, and 40 allocation-free calls. The automatic policy preserves
the conservative eight-CTA layout below Q=8K because an earlier cluster-4
crossover did not generalize across SM100 devices.
For H64, a three-round, 20-call, reversed-order A/B measured prepared-plan
eager execution at 104.579 ms and CUDA Graph replay at 104.225 ms (0.997x).
Capture completed with PyTorch synchronization-debug error mode enabled. The
cluster-16 regression also passed 100 repeated bitwise executions, graph
capture/replay with poisoned outputs and workspace, and changed device-side
topk_length values. Compilation and allocation remain plan-time operations;
capture uses caller-owned outputs and workspace, and replay requires the same
declared tensor shapes and storage addresses while their contents may change.
SparseAttentionBackward.scratch_workspace_bytes() reports the full scratch
requirement on every backend: the SM100 FP32 statistics and dKV-accumulator
planes, and on SM90 the FP32 dPsum and log2-scaled LSE vectors plus the dKV
accumulator. The ordinary SM90 dKV term is
align128(round64(total_S_kv) * round32(D) * 4) bytes; deterministic SM90 uses
the selected relay term above. Both add
2 * align128(round64(total_S_q) * H * 4) bytes of statistics. Pass a contiguous CUDA
uint8 tensor of at least this size to execute(..., workspace=workspace) and
reuse it across calls; the launch sequence re-initializes the accumulators on
every execution (in-kernel on SM100, and with one stream-ordered memset for the
SM90 dKV accumulator and for d_sink on the routes that accumulate it
atomically). execute(q, kv, out, dout, lse, attn_sink, topk_idxs, dq, dkv, ..., d_sink=...) requires caller-provided dq, dkv, and the keyword-only
d_sink on every backend and never allocates or copies during execution.
Every SM100 plan compiles its selected ordinary, deterministic, two-CTA, or
adaptive-pair artifact in compile() from storage-free tensor metadata, so
its first execute() performs no CuTe JIT and may be captured directly. The
SM90 implementation retains its concrete-tensor compile caches and compiles on
its first execution. The high-level wrapper allocates the outputs and the
scratch on the launch stream when they are omitted. Inputs, outputs, and the
live workspace must not alias; concurrent executions need distinct output and
workspace storage.
- Outputs — tuple
(dq, dkv, d_sink) - Constraints — SM90 or Blackwell SM100/SM103; SM90 supports flat MQA tensors with
head_dim ∈ {512, 576}
3. Indexer Forward (score-only)
Computes dense indexer scores:
S[b, q, k] = sum_h ReLU(Q_h · K_h^T) · W_h with a ratio-causal mask.
For local query row q_local, valid KV columns satisfy
k_local \< clamp((q_causal_offsets[b] + q_local + 1) // ratio, 0, seqlen_k_b).
When q_causal_offsets is omitted, all offsets are zero.
q_causal_offsets[b] is the global uncompressed token index corresponding to
local q[0] for batch or THD segment b. It is not the packed storage offset
from cu_seqlens_q: cu_seqlens_q locates where a local Q segment is stored,
while q_causal_offsets locates that segment in the global causal timeline.
The K columns are assumed to be a compressed-KV prefix starting at global
compressed column 0.
- Inputs
q:(B, S_q, H_q, D)BF16, or the architecture-specific FP8 or FP4 format described below (packed FP4 storesD // 2per row).k:(B, S_k, H_kv, D)BF16, or the architecture-specific FP8 or FP4 format described below.w:(B, S_q, H_q)BF16 or FP32 with BF16 Q/K, on SM90 and SM100. FP32 head weights retain their precision through score generation, without changing the Q/K dtype; FP32 W requires unit stride in its last dimension. Unsupported strides are rejected instead of copied. The SM90 FP8 path also accepts FP32 when weights have already been pre-scaled byq_scale * sm_scale; SM100 MXFP8, MXFP4 and NVFP4 require BF16 weights.q_causal_offsets(optional): CUDA INT32 tensor with one entry per batch/THD segment, on the same device asq.block_table+kv_lens(optional, SM100 BF16): readkstraight from a paged key cache instead of a dense tensor, with the block-table contract of the QSA compressed-key cache.kis the page pool(num_pages, page_size, H_kv, D)or(num_pages, page_size, D);page_sizeis a multiple of 8 that divides 128 or is a multiple of it (16, 32, 64, 128, 256 all work).block_table[b, i]((B, max_pages)INT32, unit column stride) is the pool row of sequenceb’si-th page andkv_lens[b]((B,)INT32) its compressed-key count; the ratio-causal limit still applies on top.cu_seqlens_kmust beNone(THD queries keepcu_seqlens_q), and the score width ismax_seqlen_k, which defaults tomax_pages * page_size; keys at or past it are never scored, nor are pool rows pastkv_lens[b]inside the last live page.
- Output —
scores:(B, S_q, S_k)FP32. On SM100 this is the[..., :S_k]view over a(B, S_q, ceil4(S_k))(THD:(total_q, ceil4(max_seqlen_k))) allocation: the kernel bindsoutthrough a 16-byte-aligned layout, so the fp32 row stride must be a multiple of 4 elements and at leastceil4(S_k). WhenS_k % 4 != 0the returned tensor is therefore a strided view, not a fresh contiguous copy. A caller-providedoutmust be such a view (ValueErrorotherwise); nothing is staged or copied back.IndexerForwardtakes the contiguous padded(B, S_q, S_k_padded)buffer both at construction and inexecute(), bindsout[..., :S_k]directly, and never writes columns[S_k, S_k_padded). - Strided inputs —
IndexerForward.check_support()declinesq/k/wwhose innermost stride is not 1 withNotImplementedError(the kernel addresses them natively; nothing is repacked), andexecute()raisesValueErrorfor such live tensors.indexer_forward_wrapperand the SM100 combined Top-K path make strided inputs contiguous on the launch stream as a torch-op convenience, at the cost of one copy kernel per strided tensor. The SM90 combined Top-K path is capture-oriented and declines non-contiguous packed Q/K/W/scales. - Scope — FP32 weights are supported by Indexer Forward and Combined Indexer Forward + Top-K. The separate indexer score-recompute and backward APIs still require BF16 weights.
- Precision paths — selected by
precisiononindexer_forward_wrapperandindexer_forward_top_k_wrapper; theIndexerForwardclass is BF16-only.-
SM90
precision="fp8": Q/K use E4M3 andq_scale/k_scaleare FP32 descales with one value per token/head. Setreturn_lse=True(or providelse_out) to compute LSE in the same kernel invocation. -
SM100
precision="mxfp8": Q/K use E4M3 with block-scaled, packed E8M0 scale tensors;sf_vec_size(defaultNone, the format’s size) must be 32, andqhead_per_kv_head ∈ {4, 8, 16, 32, 64}(the tile is always 128 packed rows — the scale atom — so a 4-head group packs 32 query tokens per tile; all-ones scale blobs make it a plain E4M3 x E4M3 scorer). THD inputs additionally requirecu_seqlens_q_scale_padded/cu_seqlens_k_scale_padded: contiguous CUDA INT32 prefix tensors of shape(B + 1,)on the Q/K device. The interface deliberately performs no device-to-host copy to inspect their values. The caller guarantees that each device-side prefix starts at zero, is monotonic, covers the corresponding logical sequence, satisfies the packed-row alignment (each Q span timesqhead_per_kv_headis a multiple of 128 rows; each K span is a multiple of 128 tokens), and stays within the packed scale storage. -
SM100
precision="mxfp4": Q/K aretorch.float4_e2m1fn_x2, two E2M1 values per byte with element2iin the low nibble, so their last dim isD // 2(64); a packeduint8buffer passes zero-copy through.view(torch.float4_e2m1fn_x2).q_scale/k_scaleare the MXFP8 E8M0 scales unchanged: one per 32 head_dim values, soD // 32(4) per row although the last dim isD // 2, in the same packed layout, built from logical(..., D // 32)scales by the packers incudnn.deepseek_sparse_attention.utils.sm100.mxfp8_scale_utils:pack_q_scale_bshd/pack_k_scale_bshdfor BSHD, andpack_q_scale_thd/pack_k_scale_thdinto a caller-providedoutwith prefixes frommake_scale_cu_seqlens_paddedfor THD.sf_vec_sizemust be 32 (its default), and THD inputs take the samecu_seqlens_q_scale_padded/cu_seqlens_k_scale_paddedscale-prefix contract. Its kernel differs from the MXFP8 one only in the MMA: tcgen05kind::mxf4, K = 64 per MMA. Requires a PyTorch build withtorch.float4_e2m1fn_x2; otherwiseprecision="mxfp4"raisesNotImplementedError. -
SM100
precision="nvfp4": Q/K are packed E2M1 as for MXFP4 (torch.float4_e2m1fn_x2, last dimD // 2).q_scale/k_scaleare non-negativetorch.float8_e4m3fnscales, one per 16 head_dim values, soD // 16(8) per row, packed by the same packers (they move any one-byte scale) and with the same THD scale-prefix contract.sf_vec_sizemust be 16, its default. The per-tensor FP32 global scales of NVFP4 are not inputs: fold the product of the Q and K global decode scales intosm_scale, since a score issm_scale * sum_h relu(q_h . k) * w_hand the ReLU commutes with a positive scale. A decode scale multiplies an E2M1 value times its E4M3 block scale back to the real value: for a global scale stored as the encoding factorg = 448 * 6 / amax, fold1 / g.sm_scaleis a launch argument, so a new global scale per step compiles nothing. Its kernel differs from the MXFP4 one only in the scales: tcgen05kind::mxf4nvf4with 16-value blocks (scale_vec::4X), K = 64 per MMA, reading four E4M3 scales per row per MMA. Requires a PyTorch build withtorch.float4_e2m1fn_x2; otherwiseprecision="nvfp4"raisesNotImplementedError.
-
- Constraints —
head_dim == 128. The SM90 direct path supportsqhead_per_kv_head ∈ {16, 32, 64}and currently requiresH_kv == 1; SM100 BF16, MXFP8, MXFP4 and NVFP4 dense and combined Top-K paths supportqhead_per_kv_head ∈ {4, 8, 16, 32, 64}at a fixed 128-row MMA tile (4 heads x 32 tokens, …, 64 heads x 2 tokens), som_block_sizemust be 128. All currently requireH_kv == 1(MQA). A query may be up toratio - 1tokens longer thanratio * seqlen_k(its trailing tokens have not completed a compressed block and score the complete blocks only); a longer one is refused.
4. Combined Indexer Forward + Top-K
DSA.indexer_forward_top_k_wrapper provides a one-call alternative to the
two-call dense indexer_forward_wrapper + indexer_top_k_wrapper sequence
when compact score generation is desired. It returns aligned indices INT32
and logits FP32, plus softmax FP32 by default, without materializing dense
scores. Pass
return_softmax=False to omit softmax. BSHD output shape is
(B, S_q, top_k); THD output shape is (total_q, top_k). Padded slots are
-1/-inf, with zero softmax. The selected set is Top-K, but its slot order
is not guaranteed to be descending. Set deterministic=True to break
exact-value ties at the K-th boundary toward the smallest local KV indices, making the selected set
reproducible across launches. This does not sort the output slots; the default
False path retains the faster scheduling-dependent tie-break.
The combined compressed path supports different validated rows on SM90 and
SM100. SM90 initially supports packed THD only with ratio=4,
head_dim=128, H_kv=1, qhead_per_kv_head=64, and contiguous inputs. Q/K
may be BF16 with BF16 or FP32 weights, or E4M3 FP8 with FP32 per-token/head
descales and BF16 weights. SM90 does not support BSHD, MXFP8/MXFP4/NVFP4, paged K,
or explicit query microbatching in this path. SM100 retains both BSHD and THD
with BF16 Q/K and BF16 or FP32 weights, plus MXFP8, MXFP4 or NVFP4 Q/K with
BF16 weights.
topk_indices_global=True is the default. Optional caller-owned
candidate/output/softmax/LSE buffers avoid per-call allocations; size the
candidate buffer with compress_topk_cand_buffer_size for BSHD or
compress_topk_cand_buffer_size_thd for THD. SM90 LSE is supported for BF16
and FP8; SM100 LSE is supported for BSHD and THD with BF16, MXFP8, MXFP4, and
NVFP4.
Explicit microbatching is BF16 BSHD-only and cannot
be combined with LSE or explicit q_causal_offsets. BSHD with explicit
q_causal_offsets currently computes its per-batch candidate offsets eagerly
and is not CUDA-graph-capturable; THD capture requires the caller-provided
offsets and buffers returned by compress_topk_cand_buffer_size_thd. The same
preallocation rule applies to SM90 capture; SM90 capture additionally requires
topk_indices_global=False, because its optional local-to-global conversion is
outside the prepared two-kernel path. Compile both kernels with an eager
warmup before capture. Compact
addressing requires 0 \<= q_causal_offsets[b]; rows extending beyond the KV
prefix are clamped to seqlen_k_b. THD MXFP8, MXFP4 and NVFP4 use the same
caller-guaranteed device-side scale-prefix contract described in §3; the
interface does not validate prefix values with a device-to-host copy.
For an explicit CUDA Graph lifecycle on SM90, use
DSA.prepare_indexer_forward_top_k. It compiles both stages without launching
them and returns an IndexerForwardTopKPlan with a caller-owned byte-workspace
contract:
The plan deliberately fixes only the CUDA Graph capacities: Q/K storage
extents, prefix tensor lengths (segment capacity), maximum per-segment Q/K
lengths, output shape, and candidate-workspace capacity. Prefix,
q_causal_offsets, and cand_batch_offsets contents remain device runtime
data and may be updated in place between replays. Their addresses and shapes
must remain fixed; Q/K prefix totals must continue to equal the fixed Q/K
storage extents, every segment must stay within the declared maxima, and the
final candidate offset must not exceed max_candidate_count. The plan never
reads these values to the host. It returns local per-segment KV ids.
Preparation owns compiled artifacts but no device allocation; execution uses
only the caller’s inputs, outputs, and workspace and does not consult the
process-global compile cache. The same plan can serve concurrent calls when
each call has independent outputs and workspace. The caller must order input
and metadata updates before the selected stream and retain every binding until
that stream finishes.
sm_scale is a plan-time host scalar applied in FP32 by the scorer. Pass raw
BF16 w and place any uniform softmax/head-normalization factors in this
scalar; pre-multiplying them into BF16 w adds a BF16 rounding and passing the
same factor in both places scales twice. The scalar is not a compile-cache
specialization, so another plan value can reuse the compiled artifact, but a
CUDA Graph launch captures its value and must be recaptured when it changes.
This first plan is intentionally not a general scheduler: it accepts the same contiguous packed-THD segments as the SM90 combined kernel and does not model CP ranks, zigzag order, arbitrary Q/K storage windows, or index maps. A caller such as Megatron owns those policies and lowers them into packed prefixes and causal offsets. Use a separate graph/capacity bucket when storage extents, segment capacity, maximum lengths, or output addresses change. A workspace may be shared by serialized same-stream users only when no graph can replay them concurrently and no later consumer reads its candidate scratch.
Query-window microbatching remains a caller policy. A caller may prepare one
fixed-Q-capacity plan and invoke it once per window while reusing the full K
tensor. For each active window, set its q_causal_offsets to the global query
start. Represent a short tail window as two packed segments: the active Q rows
paired with all K rows, followed by the unused Q-capacity rows paired with zero
K rows. The unused rows then receive the documented padding sentinels
(indices=-1, logits=-inf, softmax=0, and lse=-inf) and may be sliced
away by the caller. Serialized windows on one stream may share a workspace,
but their output buffers must be distinct until all consumers finish. The same
plan accepts bindings with matching descriptors and different addresses; a
captured graph node retains the addresses used during that capture.
On SM90 the first kernel reuses the Hopper WGMMA scorer and changes only its output epilogue: causal-valid scores are written to the 64-bit compact candidate layout. The second kernel is the common SM90+ deterministic radix Top-K implementation. Candidate memory is still quadratic in sequence length (about half the dense ratio-4 causal score storage), so long-context callers must budget capacity explicitly. The plan does not provide an internal SM90 THD microbatch scheduler; callers compose windows as described above.
Leave microbatch_rows=-1 to use the BSHD long-sequence memory policy; it
enables windowing only for compatible BF16 shapes. Pass 0 to force one
launch, or a positive value only when deliberately controlling the scratch
footprint. Size the buffer with the launch’s microbatch_rows and
precision: compress_topk_cand_buffer_size sizes MXFP8, MXFP4 and NVFP4
launches, which never microbatch, for one full launch.
Paged K (block_table + kv_lens, the §3 contract) serves BSHD and THD
queries with BF16 Q/K in one launch and returns local key ids
(topk_indices_global=False), which is what serving engines consume before
mapping ids to physical cache rows. Caller-owned candidate buffers are sized
with compress_topk_cand_buffer_size(..., paged=True) for BSHD and
compress_topk_cand_buffer_size_thd(cu_seqlens_q, None, ratio, q_causal_offsets)
for THD: both return the single-launch size, and the THD call takes no key
prefix because a paged pool has none. No explicit microbatching and no MXFP8
yet.
Global versus local ids is primarily an interoperability and allocation
choice, not an Indexer Backward tuning choice. Keep the default global ids
when a downstream consumer requires flat KV ids. If every consumer accepts
local ids, prefer topk_indices_global=False. Avoiding the local-to-global
conversion can slightly improve wrapper performance, particularly for short
sequences. Local ids also avoid conversion temporaries and enable a
strictly zero-extra-allocation captured path when all documented buffers are
supplied. The global-id conversion is CUDA-graph-safe; during capture, its
temporaries are recorded in the graph’s private pool. Include any downstream
conversion when comparing end-to-end performance.
5. Indexer Top-K
Radix top-K kernel for selecting candidate KV indices from indexer scores, with variable per-row effective length.
- Inputs
input_values:(n_rows, num_cols)FP32/FP16/BF16seq_lens:(batch_size,)INT32 (per-batch effective column count)
- Outputs — tuple
(indices, values)(values isNonewhenreturn_val=False). Usereturn_val=Falsewhen only the indices are consumed, so no values output buffer is allocated or written. - Tie policy —
tie_break=0(default) permits arbitrary cutoff ties;1prefers smaller source-column indices and2prefers larger indices. The selected set follows this policy. Whendeterministic=False, output order remains unspecified. Ties use equal radix keys:+0ranks before-0, as in the existing kernel; NaN ordering is unspecified. The policy applies only to this standalone API. - Deterministic mode —
deterministic=Truemakes the selected set and output slots repeatable. With the defaulttie_break=0, smaller source indices win a cutoff tie. The returned local indices are canonicalized in ascending order, carrying values with them. Selection and canonicalization are fused into this CuTe DSL kernel rather than launching a separate finalize kernel. - Cost — nonzero tie policies scan source columns inside the selection
kernel only when equal cutoff keys compete for the remaining slots. The
scan stops once enough ties are found. Deterministic mode additionally sorts
the selected indices in CTA shared memory. The compiled specialization
chooses between an index-bitmap scan and an in-place bitonic sort according
to the target architecture and shape; on SM100,
top_k < 256uses bitonic ordering directly, and on SM103 bitonic ordering is used from 128K bucketed columns attop_k <= 128(<= 512with more than 148 rows). Neither path adds a kernel, device allocation, or host synchronization; the default specialization retains the existing selection path. - Constraints — SM90+,
0 < top_k ≤ 2048, nonzero row/column/batch extents, andnext_n > 0.seq_lensis contiguous INT32 on the input CUDA device, and every entry must be in[0, num_cols]; the device-resident values are not copied to the host for validation. On SM120,deterministic=Truesupports at most 16,384 columns for FP32 and 32,768 columns for FP16/BF16 because of the 99 KiB per-block shared-memory limit;check_support()rejects larger deterministic shapes before compilation.
IndexerTopK owns no device memory. execute(input_values, seq_lens, out_indices, out_values=None, current_stream=None, workspace=None) writes into
caller-owned outputs: out_indices is (n_rows, top_k) INT32 and out_values
is (n_rows, top_k) in the input dtype, both contiguous and on input_values’
device. out_values is required when the object was built with
return_val=True and must be None otherwise. The radix scratch is carved from
workspace, a CUDA uint8 tensor of at least scratch_workspace_bytes() bytes
(n_rows * (2 for FP32, else 1) * num_cols * 4, rounded up to 128) that is
32-byte aligned; an absent or undersized workspace raises before launch.
The writable out_indices, optional out_values, and carved scratch byte
ranges must not overlap one another or the read-only inputs. Disjoint views of
the same backing allocation are valid. The class API also requires a 32-byte
aligned input base; the eager wrapper copies a contiguous storage-offset view
when needed to satisfy this alignment.
compile() builds the kernel from fake tensors and allocates nothing.
indexer_top_k_wrapper allocates the outputs and the workspace per call on the
launch stream. Rows the next_n stagger leaves empty are canonicalized to
-1 indices and -inf values.
Compact vs. non-compact sparse indices
Providing topk_length declares a compact input layout for Sparse Attention
Backward and the sparse score-recompute kernels: in each row, the first
topk_length[row] slots must contain valid ids and every later slot must be
filled with -1. The padding value is required even when topk_length is
provided, because the SM100 attention score-recompute wrapper may select a
non-compact kernel that identifies invalid slots by their index values. Use
compactify_wrapper first when invalid entries are interspersed. A 3-D input
is returned flattened as (B * S_q, topk) with a (B * S_q,) length tensor;
reshape both when a BSHD score-recompute wrapper expects batched shapes. The
operation preserves the input id convention; it does not convert local ids to
global ids.
There is no universal faster representation. Compact execution can skip
sparse tiles when valid prefixes are substantially shorter than the physical
top-k width, while non-compact execution can benefit from static loop bounds
when rows are nearly full. The SM100 sparse attention score-recompute wrapper
uses separately tuned tile policies and can internally select its non-compact
path when that has better code generation. If conversion is required solely
for performance, benchmark the entire compactify_wrapper + consumer sequence
for the target length distribution. Indexer Backward has no topk_length
argument and always processes its fixed slot width; its score tensors and ids
must remain aligned slot-for-slot.
For BF16 workloads with causal sparse indices, the following are useful starting points; the best choice still depends on the shape and valid-length distribution:
- On SM100, prefer trying the non-compact path for Sparse Attention Score Recompute when optimizing this consumer in isolation; it can be slightly faster. For Sparse Indexer Score Recompute, preserve the producer’s representation, since the performance difference is generally small.
- On SM90, prefer compact inputs for both sparse score-recompute kernels when the valid-prefix length is already available; compact execution can be slightly faster.
- For Sparse Attention Backward on either architecture, prefer compact inputs when the valid-prefix length is already available. The benefit may be small when most rows are full.
Retain local ids when the producer and every downstream consumer accept them.
If compactify_wrapper or an id conversion must be added, time that operation
together with the consumer before choosing a representation.
6. Sparse Indexer Score Recompute
Computes softmax over top-K entries of the indexer score:
predict[b, q, i] = softmax_i(sum_h ReLU(Q_h · K_{topk[i]}^T) · W_h).
- Inputs:
q_indexer,k_indexer,weights,topk_indices(optionaltopk_length).topk_indicesare per-batch local KV ids by default; passtopk_indices_global=Truewhen using ids encoded asbatch_idx * S_k + local_idx. - Output —
predict:(B, S_q, topk)FP32. - Layout contract (class vs wrapper) —
SparseIndexerScoreRecompute.execute()takesoutas a required, contiguous FP32 tensor matchingsample_out;q_indexer/k_indexer/weightsneed a unit innermost stride andtopk_indices/topk_lengthmust be contiguous int32.check_support()declines anything else withNotImplementedErrornaming the tensor and its strides, andexecute()re-validates the live tensors withValueError; nothing is copied, cast, or allocated on the execute path. A prepared SM90 plan also rejects anoutwhose byte range overlaps any input. The wrapper keeps the convenience: it copies strided inputs contiguous, casts the index tensors to int32, allocatesoutwhen omitted, and stages an overlapping SM90 output before copying it back, all onstream. - Prepared SM90 plans —
compile()builds the CuTeDSL artifact from symbolic batch and sequence extents. ExactB,S_q,S_k, runtime scale, pointers, and stream do not specialize the artifact. Physical top-K widths in[1, 2176]use one of 17 bounded 128-slot capacity buckets.execute()reads the public(B, S_q, H)weights layout directly, performs no layout copy or allocation, and is CUDA Graph capturable with caller-ownedout. The kernel has a fixed reduction tree and one writer per output element, so identical inputs and one compiled plan produce bitwise-identical output bytes.
7. Sparse Attn Score Recompute
L1-normalised head-summed softmax over top-K entries:
target[b, q, i] = sum_h exp(Q_h · K_{topk[i]}^T · scale - LSE_h) / Z.
- Inputs:
q_attn,k_attn,lse,topk_indices,softmax_scale(optionaltopk_length).topk_indicesare per-batch local KV ids by default; passtopk_indices_global=Truewhen using ids encoded asbatch_idx * S_k + local_idx. - Output —
target:(B, S_q, topk)FP32. - Note: the wrapper handles the
-log2(e) * lsepreprocessing internally. - Layout contract — as in §6: the class requires a contiguous
outatexecute(), declines a non-unit innermost stride onq_attn/k_attn/lseand non-contiguous or non-int32topk_indices/topk_lengthincheck_support(), and never copies; the wrapper keeps the convenience copies and allocatesoutwhen omitted. On SM100 a plan built withouttopk_lengthcompiles themTopkLengthoperand out entirely (no placeholder tensor). - Prepared SM90 plans — the same symbolic compile and bounded Top-K buckets
as §6 apply.
softmax_scaleis a runtime scalar rather than a compile key, and the kernel reads(B, S_q, H)LSE directly. A warmed class plan with its caller-owned, disjointoutis allocation-free, CUDA Graph capturable on an explicit non-default stream, and bitwise deterministic for identical inputs.
8. Dense Indexer / Dense Attn Score Recompute
Full-KV (no top-K) analogues of §6 and §7. Each returns {'out', 'denom'}.
They apply the same ratio-causal mask as Indexer Forward; masked positions are
written as -inf and excluded from denom. Pass the same q_causal_offsets to
all dense score tensors that feed the same loss path.
On SM100, Indexer Forward and Dense Indexer Score Recompute use the same
unified kernel implementation: forward runs it with compute_lse=False and
passes denom=None, so the LSE slot is compiled out (no placeholder buffer),
while dense indexer score recompute runs it with compute_lse=True and a live
denom. The shared implementation lives in score_recompute;
indexer_forward only imports it.
Dense Attention Score Recompute has separate BF16 and MXFP8 kernels because its
score and normalization semantics differ from the indexer path; the MXFP8 one
shares the indexer kernels’ warp-specialized scaffolding and block-scale
plumbing.
- Precision — on SM100,
dense_indexer_score_recompute_wrappertakesprecision="bf16","mxfp8","mxfp4", or"nvfp4"with the Q/K and scale contract of Indexer Forward (§3); its block-scaled precisions requireqhead_per_kv_head ∈ {32, 64}.dense_attn_score_recompute_wrappertakes"bf16"or"mxfp8"and rejects"mxfp4"and"nvfp4". TheDenseIndexerScoreRecomputeandDenseAttnScoreRecomputeclasses, and SM90, are BF16-only. - Layouts — BSHD on SM90 and SM100+. THD packed (
cu_seqlens_q/cu_seqlens_kwithq:(total_q, H_q, D),out:(total_q, max_seqlen_k)) on SM100+ only: SM90 declines THD incheck_support()withNotImplementedErrorbecause its kernel is BSHD-native and serving THD would need a host copy ofcu_seqlensper call. - THD launch envelope —
DenseIndexerScoreRecompute/DenseAttnScoreRecomputetakemax_seqlen_qandmax_seqlen_kas plan-time ints in__init__:max_seqlen_qis required andmax_seqlen_kdefaults to (and must equal)sample_out.shape[1].execute()never derives them fromcu_seqlens, and caller-passed values must match the plan’s. The wrappers derive them fromcu_seqlenswhen omitted, at the cost of one blocking device-to-host read per call (not CUDA-graph capturable); pass both. - Layout contract (class vs wrapper) —
execute()takesoutanddenom_outas required, contiguous FP32 tensors matching the samples;q/k/weights(orlse) need a unit innermost stride, andcu_seqlens_q/cu_seqlens_k/q_causal_offsetsmust be contiguous 1-D int32.check_support()declines strided samples withNotImplementedErrornaming the tensor and its strides;execute()re-validates the live tensors withValueError. Nothing is copied, cast, or allocated on the execute path (the SM90 kernel’s per-head(B,S,H)->(B,H,S)operand transpose is the one remaining copy, tracked as a deferred kernel change). The wrappers keep the convenience: strided inputs are copied contiguous,cu_seqlens_*are cast to int32, andout/denom_outare allocated when omitted, all onstream.
9. Indexer Backward
Three-stage sparse top-K pipeline that produces the training gradients for the indexer tower:
ScoreGradSm90/ScoreGradSm100(kernel 1) — in-place score-grad precompute that overwritesattn_score(target) and readsindex_score(predict) without modifying it.IndexerBackwardSm90/IndexerBackwardSm100(kernel 2) — three warp-specialised GEMMs produced_index_q,d_weights, and adIndexK_f32accumulator.- In ordinary execution with a BF16
d_index_k, a pure-torch copy (kernel 3) castsdIndexK_f32to BF16; an fp32d_index_kis the accumulator itself and needs no kernel 3. On SM100-family GPUs that private BF16-path accumulator is column-permuted, so kernel 2 can issue 16-byte reductions straight from registers without SMEM staging; the same copy restores the natural order. Deterministic execution instead uses the fixed-order reduction below to write the requested output dtype directly.
IndexerBackward / indexer_backward_wrapper run on SM90 and on SM100/SM103/SM107
(SM107 needs a CuTe DSL with the sm_107a target).
check_support admits D == 128, block_I == 128, a positive top-k that is a
multiple of block_I (callers with other logical extents pad the top-k tensors
with -1 indices and zero scores), and a finite sm_scale > 0; SM100/SM103/SM107
additionally require H in {32, 64} and SM90 supports H == 64 plus H == 32 in deterministic mode. d_index_q
and d_weights are BF16; d_index_k is BF16 or fp32.
With deterministic=True on SM90, SM100, SM103 or SM107, kernel 2 uses fixed
producer-plane ownership and accumulates dK into a CTA-private FP32 workspace. The compile-time
deterministic_dk_planes specialization selects 96, 128, or 148 producer
planes (default 96). One CuTe DSL reduction kernel folds the planes in fixed
CTA order, converting only once to the requested d_index_k dtype. The dQ
and dW stores remain single-writer.
The plan owns no device memory. For non-deterministic execution, an fp32
d_index_k is the accumulator itself (zeroed on the launch stream inside
execute: fused into the score-grad kernel on SM100, a memset on SM90; the
BF16 scratch is cleared the same way, so callers never pre-zero); a BF16 d_index_k accumulates into a B * S_k * D fp32 view carved
from the caller’s workspace. IndexerBackward.scratch_workspace_bytes() reports
that ordinary scratch size (0 for fp32 d_index_k).
execute(..., workspace=) takes the uint8 buffer on index_q.device, and a
missing or undersized buffer raises ValueError before attn_score is touched.
indexer_backward_wrapper allocates it per call on the launch stream and accepts
the same optional workspace= keyword for reuse; executions that share one
buffer must not overlap on the device.
deterministic=True is request-or-fail and supports H in {32, 64} on
SM90/SM100/SM103/SM107. All supported devices require D == 128,
block_I == 128, and topk % 128 == 0 for
128 <= topk <= 2048. Each producer CTA owns one plane. SM100-family kernels
process fixed grid-stride row stripes; SM90 submits bounded waves of at most
deterministic_dk_planes one-row CTAs on the same stream, so a later wave cannot
reuse a plane until its predecessor completes. The main GEMM therefore retains
its existing Tensor Core math while dK has a fixed association order across
launches. Scratch is exactly
ws_align(deterministic_dk_planes * B * S_k * D * sizeof(float)). It is cleared on the launch stream at every execute, then
reduced once in fixed plane order. The live scratch byte range must not overlap
any input or output; execute validates this before clearing the planes, while
allowing disjoint views of the same allocation. A warmed plan with caller-owned
outputs and workspace is CUDA Graph compatible. To capture the convenience
wrapper, warm it first and pass every output plus workspace= explicitly;
omitted buffers are allocated per call. A fixed plane count is bitwise
reproducible across runs; changing the plane count changes the FP32
accumulation grouping, so results from two plane counts are not required to be
bitwise identical. As with the existing deterministic sparse-attention
backward, valid IDs in a semantic top-k row are expected to be distinct.
The TileLang fallback present in the upstream repo is dropped here
(CuTe-DSL only). If the CuTe-DSL path fails the wrapper raises
RuntimeError rather than silently falling back.
When compressed forward returns its fused softmax, backward can skip both
indexer Q@K score recompute and the separate logits softmax. Pass softmax
directly as index_score; backward treats this buffer as read-only.
attn_score must use the same valid-slot mask. Because compressed forward
returns global indices by default, also pass topk_indices_global=True unless
forward used topk_indices_global=False. The public sparse
indexer_backward_wrapper has a
BSHD-shaped interface; BF16 THD tensors can use zero-copy B=1 views (squeeze
the singleton K head and add a batch dimension) together with global Top-K
indices. FP8, MXFP8, MXFP4, and NVFP4 indexer backward are not currently supported
because the backward wrapper requires BF16 Q/K/W inputs.
On SM100, topk_indices_global describes the input
encoding; it is not a performance-tuning switch. Local ids are range-checked and converted to flat ids in registers before the
same optimized gather path. Do not add a separate local_to_global_wrapper
launch only for Indexer Backward; preserve the producer’s id convention and
set topk_indices_global to match it.
On SM100, the kernel accepts compact H32 index_q, weights,
d_index_q, and d_weights tensors directly. TCGEN05 still executes its
minimum M64 tile internally: the upper Q rows and weights are zero-filled, and
the dQ/dW stores are clipped to the logical H32 extent. Callers must not pad
these tensors to H64 or slice H64 outputs. This compact path also applies to
deterministic=True: fixed producer ownership changes only the dK destination
and row schedule, while the same H32 input padding and output predicates remain
active. Other sub-M64 head counts remain unsupported; H64 and larger head
counts retain their existing behavior.
On SM90, also prefer preserving the producer’s id convention. The performance difference between local and global ids is generally small, so a separate conversion solely for Indexer Backward is unlikely to help. Deterministic SM90 uses the same fixed producer-plane ownership and fixed-order reducer described above. Its stream-ordered bounded waves also form the synchronization boundary: every one-row CTA in wave N completes before wave N+1 can reuse the same producer planes, and the fixed reducer runs only after the last wave.
10. Dense Indexer Backward
Full-KV counterpart to Indexer Backward. It consumes raw dense score tensors and denominators produced by Dense Indexer / Dense Attn Score Recompute.
- Inputs
index_q:(B, S_q, H, D)BF16weights:(B, S_q, H)BF16index_k:(B, S_k, D)BF16attn_score,index_score:(B, S_q, S_k)FP32 raw dense scoresattn_l1norm,index_lse:(B, S_q)FP32 denominatorsq_causal_offsets(optional): same offsets used for the corresponding Dense Indexer / Dense Attn Score Recompute outputs.
- Outputs —
d_index_q,d_weights,d_index_k - Constraints — SM90, SM100, SM103 or SM107,
H == 64,ratio >= 1; SM100-family GPUs acceptD in {64, 128}andblock_I in {64, 128}, SM90D == 128andblock_I == 128 - Scratch — the dK reduction targets a pre-zeroed fp32 buffer: an fp32
d_index_kin place, a BF16d_index_kthrough an fp32 accumulator carved from the caller’s workspace (DenseIndexerBackward.scratch_workspace_bytes()bytes onindex_q.device, 0 for fp32d_index_k;execute(..., workspace=)).dense_indexer_backward_wrapperallocates it per call on the launch stream unlessworkspace=is passed; calls sharing one buffer must not overlap.
11. Pooled DSA Training Backward
The pooled variant of the indexer scores pools of pool_size consecutive keys instead of single keys. Each pool key is the
softmax-weighted (gates + ape) mean of its tokens; the top-K pools are selected per query, and the attention itself runs
on the tokens of the selected pools plus the incomplete tail of tokens that precede the query position. The training APIs live
in cudnn.deepseek_sparse_attention (lazily exported) and compose with the token-level kernels above.
Layouts: q/out/dout are (B, S, H, D), kv is (B, S_k, D), lse is (B, S, H) FP32, pool_ids are local pool
indices with -1 for unselected slots (expected last), positions (B, S) are the query positions. The teacher target is the
attention probability mass of each selected pool summed over heads and renormalized over the valid pools of the query; tail tokens
are excluded. pooled_indexer_backward consumes target in place and recomputes predict when it is not supplied.
The underlying kernels take two options used by this composition: sparse_attn_score_recompute_wrapper(..., pool_size=P) emits the
per-pool sums directly, and SparseAttentionBackward on SM100 accepts pool_target/pool_size to emit them from the backward’s
own probabilities, accumulating fp32 partials across the attention head blocks (bit-reproducible up to two blocks, i.e. 128 heads);
the option is not available with deterministic=True. Supported on SM100-class GPUs with the generic single-CTA backward backend; pool_size 4 and 16 are supported.
Limitations
- CuTe DSL requirement — install
nvidia-cutlass-dsl[cu13]>=4.8.0. SM90 forward requires the loaded CUDA 13 DSL compiler flavor and rejects a CUDA 12 compiler before JIT; the package version alone is not sufficient. Sparse Attention Forward emits the few SM100 instructions that are not yet exposed by the public CuTe facade through the MLIR LLVM inline-assembly interface available in the supported CuTe DSL builds. - Architecture support — Sparse Attention Forward supports SM90 and the
mapped SM100-family capabilities 10.0, 10.3, and 10.7.
Sparse Attention Backward, Score Recompute, Indexer Forward, Indexer Top-K,
and Indexer Backward support SM90 and SM100. Dense score recompute in the THD
(
cu_seqlens) layout is SM100+ only; SM90 serves BSHD. The combined compressed-logits + Top-K forward serves SM90 for packed THD, ratio 4, D128, and 64Q/1KV; SM100+ retains its broader configurations. The standalone Indexer Top-K remains SM90+. - Forward scope — Prefill only; no FP8 cache, decode, or split-KV forward path. SM90 uses one CTA per query and 64-head tile, padding the last head tile internally when H is not a multiple of 64.
- Sparse gather path — SM90 uses predicated 16-byte
cp.asyncgathers, matching the Hopper C++ kernel. On SM100, the public CuTe facade does not yet expose the TMAgather4operation, so the forward uses a feature-local LLVM inline-assembly bridge with the compiler-owned descriptor; the issued data movement remains hardware TMAgather4. - Indexer Forward only supports
head_dim = 128. SM90 supportsqhead_per_kv_head ∈ {16, 32, 64}withH_kv = 1; SM100 BF16, MXFP8, MXFP4, and NVFP4 supportqhead_per_kv_head ∈ {4, 8, 16, 32, 64}. Both the dense and combined Top-K paths requireH_kv = 1. MXFP4 and NVFP4 run on the SM100-family capabilities 10.0, 10.3, and 10.7 (no SM90 path). - Standalone Top-K only up to 2048;
top_k > 2048is not supported by its radix Top-K kernel. The combined compressed path uses a separate stage-2 implementation. - Compressed-path limits — the stage-1 compact score kernel is MQA-only
(
H_kv = 1); every precision keeps its 128-row tile for every head group; explicit microbatching cannot be combined with MXFP8, MXFP4, NVFP4, LSE, or explicit per-batch causal offsets.
Native PyTorch training composition
DSA.sparse_attention composes the existing native forward and backward
wrappers, without FlashMLA. It accepts contiguous BF16 q[S_q,H,D],
kv[S_kv,D], INT32 indices[S_q,K], optional FP32 attn_sink[H], and
optional INT32 topk_length[S_q], on one SM100 device. Supported heads are
H16/H32/H64/H96/H128, each with D512 or D576; output V dimension is 512.
H16/H32 forward explicitly pads heads to 64 and slices the result. This cost
belongs to the semantic adapter and is included when benchmarking the call.
Physical K and sequence extents must be positive. K is padded to 128 for launch.
Only out is differentiable (Q, KV and sink, first order). lse is KV-only
natural-log LSE, excluding sink; max_logits and LSE are non-differentiable.
No sink means an effective all--inf sink vector. Launches and adapter tensor
operations use the current PyTorch stream. First use compiles the kernels;
warm before CUDA graph capture. This allocating convenience composition is
not a new allocation-free APIBase execute implementation.
Safe mode maps all out-of-range indices to -1, bounds lengths to [0,K], masks
inactive suffixes, and compacts valid entries when lengths are supplied. Both
passes use the same normalized metadata. Duplicate indices remain distinct
slots. trusted_compact_metadata=True is an explicit producer contract:
bounded valid active prefixes, negative inactive suffixes, bounded lengths;
without lengths every nonnegative index is in range. It skips normalization,
not launch padding. Violating that contract is unsafe.
Score recompute preserves original slots, including holes, and returns both
target and effective indices. It sums per-head probabilities, masks invalid
slots, then normalizes across the retained slots; all-masked rows return zeros.
Targets are detached. Indexer training needs its own loss; discrete Top-K
selection is not differentiated. This contract uses the existing native out
return key, not the old unmerged bridge’s output key.
Optional SM90 teacher-target output
sparse_attention_backward_wrapper(..., return_cli_target=True, cli_target_head_count=real_heads) appends cli_target to the returned TupleDict.
The default remains the original dq, dkv, d_sink result and preserves existing
positional arguments. The new settings are keyword-only.
This output is an FP32 contiguous tensor [total_S_q, topk_max], in the input
Top-K slot order. It contains the mean KV attention probability over real heads,
reconstructed from the existing KV-only natural-log LSE, with any attention
sink included in the softmax denominator by the existing preprocess. It is independent
of dout. It is not the final normalized distillation target: consumers must
normalize over their valid selected positions. Invalid slots, ignored suffixes
and empty rows are zero. Duplicate indices retain the existing attention
semantics; they are not deduplicated by this API.
The initial implementation supports SM90 and one physical 64-head block, with
1 \<= real_heads \<= 64. A caller padding smaller head counts must supply zero
padded queries/upstream gradients and positive-infinity padded LSE values. It
must use the matching real-head count. SM100-family and other architectures
decline this optional output; their existing backward paths are unchanged.
The wrapper allocates the output on the launch stream. Direct
SparseAttentionBackward plans take return_cli_target and
cli_target_head_count at construction, then require a matching preallocated
cli_target keyword output in execute(). It must be contiguous FP32 on Q’s
device with the Top-K shape. Plans never allocate this output or change their
output contract during execution. Inputs, outputs and workspace must not alias.
Warm SM90 once before graph capture, as for ordinary sparse backward.
CLI consumes probabilities already reconstructed by the backward kernel; it
does not change sparse attention forward or the backward gradient equations.
Target reference, gradient parity, empty-row/replay and plan-contract tests live
in test/python/deepseek_sparse_attention/cutedsl/test_DSA_cli_target.py.
GPU validation of this rebased revision is pending; downstream results on older
source revisions are not evidence for this revision.