Simplified sparse attention
Experimental PyTorch and JAX attention with a trainable, single-head indexer. The
kernels ship in the nvidia-cudnn-frontend wheel under Apache-2.0; no separate
attention wheel is required. Install PyTorch separately, and install JAX with
CUDA support to use the JAX API. CuTe DSL 4.8.0 or newer is required.
The PyTorch path for long rectangular inputs (at least 131,072 keys) needs CUDA conditional
graph support, tested with PyTorch 2.13. An unavailable conditional-graph API
is rejected before filtered-state allocation or compilation.
The indexer scores each permitted key with q_idx @ k_idx.T / sqrt(128) and
selects the largest topk scores. Attention computes softmax over those keys
with scale 1/16, then multiplies by V. The auxiliary objective is the KL
divergence from the mean attention distribution across query heads to the
indexer’s selected-key distribution. The teacher distribution is detached.
API
output has Q’s shape and BF16 dtype. indexer_loss is an FP32 scalar that
already includes loss_coeff (default 0.1) and division by
loss_denominator. Do not apply the coefficient again. Zero disables the
indexer training signal. Autograd returns gradients for Q, K, V, indexer Q,
and indexer K. The index selection and teacher distribution are not
differentiated. Higher-order derivatives are unsupported.
All tensors must reside on one CUDA device. Supply contiguous metadata in the
types above. Each valid row must have 0 <= key_start < key_end <= Tk;
invalid rows must be marked in query_valid. Bounds describe causal and
document masks in physical key coordinates. triangular_scores=True requires
Tq == Tk and no bounds above the causal diagonal. Use
all_query_rows_valid=True only when every row is valid;
mask_invalid_rows=False likewise requires no padding. Indexer inputs must
be finite. The optional SSA_DEBUG_CHECKS=1 validates finiteness and selected
bounds, but synchronizes the host and is for debugging only.
The caller owns projections, packed layout construction, and distributed
communication. Supply all query heads when forming the teacher distribution.
The optional kv_ready_event is joined after index selection and before
attention reads K/V. Q, indexer Q/K, and metadata must already be ready on
the input device’s current stream. The caller must keep K/V storage alive
on both producer and consumer streams.
Phase 1: dense indexer warmup
dense_indexer_kl_forward trains the indexer against the frozen dense teacher over
every permitted key. It is available from both cudnn.ops and
cudnn.simplified_sparse_attention. It returns a scalar tensor, with gradients
only for indexer Q and K; teacher Q/K and teacher_lse receive no gradients.
use_kf=True selects the fused CuTe kernel on SM100/SM103: BF16 Q/K and
indexer inputs, Hq 16/32/96, teacher dimension 256, and indexer dimension 128.
The scales must match the example. Supply contiguous FP32 natural-log
teacher_lse of shape [Tq, Hq] from dense attention with the same Q/K,
scale, and key bounds. All teacher heads must be present. The kernel
normalizes their summed probabilities before computing KL against the
indexer softmax. No topk selection occurs in Phase 1.
The default use_kf=False selects the chunked PyTorch implementation and
does not require teacher_lse. Its optional query_chunk_size and
key_chunk_size bound temporary score tiles; defaults are 256 and 1024.
These options do not tune the fused kernel. Both paths accept packed THD
and batch-one SBHD tensors. Metadata uses the bounds and padding contract
above; loss_denominator is a positive scalar or one-element tensor.
Both paths allocate scratch and compute indexer gradients during forward.
The returned loss already includes loss_coeff / loss_denominator.
Higher-order derivatives and external CUDA graph capture are unsupported,
including after warmup. The fused path uses atomic reductions and has no
deterministic mode. Phase 1 currently exposes a PyTorch API; the JAX API
below covers Phase 2 sparse attention.
Workspace and CUDA graphs
The convenience API allocates outputs, saved autograd state, and scratch. The long-sequence filtered indexer also keeps repair storage per device and stream. To reserve its maximum rectangular extents before training:
Reserve on the stream that will run attention. Calls above the reservation can grow storage and rebuild the internal repair graph. Short-key paths do not need persistent repair storage.
PyTorch host threads that share a stream serialize access to cached scratch, including reserve and H16 backward calls. Separate streams keep separate state. These locks do not synchronize the GPU.
External CUDA graph capture is unsupported, including after reservation or warmup. The public API rejects capture before launching work. Internal conditional CUDA graphs repair filtered rows; they do not provide capture support for the public training call. There is no cuDNN Graph API node.
JAX
cudnn.ops.simplified_sparse_attention_forward_jax returns a plain
(output, indexer_loss) tuple. It supports jax.jit, jax.grad, and
jax.vjp for the five activation inputs. Pass metadata as array arguments:
Shapes, dtypes, key bounds, padding, and loss scaling match the PyTorch API.
topk, loss_coeff, and triangular_scores are static options. Metadata,
index selection, and teacher targets are not differentiated. Use one process,
one GPU per call, and only one supported GPU architecture visible to that
process. Distributed execution, vmap, and higher-order derivatives are
unsupported. PyTorch remains an import dependency of the shared kernel modules;
execution uses native JAX FFI calls, without PyTorch tensor conversion or a
Python callback.
XLA owns scratch and schedules kernels on its stream. There is no JAX reserve API or K/V producer event. Long rectangular inputs use the shared filtered selector and an XLA loop that repairs failed rows in groups of 128. They do not need PyTorch’s conditional-graph API. Square triangular input currently uses a dense score workspace in the JAX path.
External CUDA graph capture is unsupported for JAX as well. These FFI calls opt out of XLA CUDA command-buffer capture; JIT remains supported.
Neither API has a deterministic mode. FP32 atomic gradient reductions can vary with scheduling. Moving packed rows can also change scalar-loss rounding. See the determinism support page (GitHub) for measured limits.
Validation and benchmarks
The suite covers numerical forward/backward results, auxiliary loss scaling, invalid inputs, upstream-gradient ownership, top-k edge cases, capture rejection, bytewise replay, packed batch changes, and golden hashes.
The benchmark guide measures PyTorch eager and JAX JIT forward/training latency after warmup.