Skip to navigation

Simplified sparse attention

View as MarkdownOpen in Claude

Experimental PyTorch and JAX attention with a trainable, single-head indexer. The kernels ship in the nvidia-cudnn-frontend wheel under Apache-2.0; no separate attention wheel is required. Install PyTorch separately, and install JAX with CUDA support to use the JAX API. CuTe DSL 4.8.0 or newer is required. The PyTorch path for long rectangular inputs (at least 131,072 keys) needs CUDA conditional graph support, tested with PyTorch 2.13. An unavailable conditional-graph API is rejected before filtered-state allocation or compilation.

The indexer scores each permitted key with q_idx @ k_idx.T / sqrt(128) and selects the largest topk scores. Attention computes softmax over those keys with scale 1/16, then multiplies by V. The auxiliary objective is the KL divergence from the mean attention distribution across query heads to the indexer’s selected-key distribution. The teacher distribution is detached.

API

import torch
from cudnn.ops import SimplifiedSparseAttentionMetadata, simplified_sparse_attention_forward
# Q: [Tq, Hq, 256]; K/V: [Tk, 1, 256]; indexer Q/K: [Tq, 128]/[Tk, 128].
metadata = SimplifiedSparseAttentionMetadata(
key_starts=key_starts, # Inclusive key bounds, int32 [Tq].
key_ends=key_ends, # Exclusive key bounds, int32 [Tq].
query_valid=query_valid, # bool [Tq]; False for padding.
loss_denominator=denominator, # FP32 [1], positive global query count.
triangular_scores=False,
mask_invalid_rows=True,
)
result = simplified_sparse_attention_forward(q, k, v, q_idx, k_idx, metadata, topk=512)
output, indexer_loss = result # Also result["output"], result["indexer_loss"].
loss = output.float().square().sum() + indexer_loss
loss.backward()

output has Q’s shape and BF16 dtype. indexer_loss is an FP32 scalar that already includes loss_coeff (default 0.1) and division by loss_denominator. Do not apply the coefficient again. Zero disables the indexer training signal. Autograd returns gradients for Q, K, V, indexer Q, and indexer K. The index selection and teacher distribution are not differentiated. Higher-order derivatives are unsupported.

PropertySupported values
GPUSM100, SM103
Input dtypeBF16
Query heads16, 32, 96
KV heads / head dimension1 / 256
Indexer dimension128
topk512, 1024, 2048
Token layoutPacked THD; rectangular Q/K extents allowed
Q/K/V storageCompact head/channel dimensions; 16-byte-aligned pointers and token strides
Empty streamsUnsupported; padding rows are supported

All tensors must reside on one CUDA device. Supply contiguous metadata in the types above. Each valid row must have 0 <= key_start < key_end <= Tk; invalid rows must be marked in query_valid. Bounds describe causal and document masks in physical key coordinates. triangular_scores=True requires Tq == Tk and no bounds above the causal diagonal. Use all_query_rows_valid=True only when every row is valid; mask_invalid_rows=False likewise requires no padding. Indexer inputs must be finite. The optional SSA_DEBUG_CHECKS=1 validates finiteness and selected bounds, but synchronizes the host and is for debugging only.

The caller owns projections, packed layout construction, and distributed communication. Supply all query heads when forming the teacher distribution. The optional kv_ready_event is joined after index selection and before attention reads K/V. Q, indexer Q/K, and metadata must already be ready on the input device’s current stream. The caller must keep K/V storage alive on both producer and consumer streams.

Phase 1: dense indexer warmup

dense_indexer_kl_forward trains the indexer against the frozen dense teacher over every permitted key. It is available from both cudnn.ops and cudnn.simplified_sparse_attention. It returns a scalar tensor, with gradients only for indexer Q and K; teacher Q/K and teacher_lse receive no gradients.

from cudnn.ops import dense_indexer_kl_forward
indexer_loss = dense_indexer_kl_forward(
q, k, q_idx, k_idx,
key_starts=key_starts, key_ends=key_ends, query_valid=query_valid,
loss_denominator=denominator,
teacher_softmax_scale=1 / 16,
student_softmax_scale=128 ** -0.5,
loss_coeff=0.1,
teacher_lse=teacher_lse,
use_kf=True,
)
indexer_loss.backward()

use_kf=True selects the fused CuTe kernel on SM100/SM103: BF16 Q/K and indexer inputs, Hq 16/32/96, teacher dimension 256, and indexer dimension 128. The scales must match the example. Supply contiguous FP32 natural-log teacher_lse of shape [Tq, Hq] from dense attention with the same Q/K, scale, and key bounds. All teacher heads must be present. The kernel normalizes their summed probabilities before computing KL against the indexer softmax. No topk selection occurs in Phase 1.

The default use_kf=False selects the chunked PyTorch implementation and does not require teacher_lse. Its optional query_chunk_size and key_chunk_size bound temporary score tiles; defaults are 256 and 1024. These options do not tune the fused kernel. Both paths accept packed THD and batch-one SBHD tensors. Metadata uses the bounds and padding contract above; loss_denominator is a positive scalar or one-element tensor.

Both paths allocate scratch and compute indexer gradients during forward. The returned loss already includes loss_coeff / loss_denominator. Higher-order derivatives and external CUDA graph capture are unsupported, including after warmup. The fused path uses atomic reductions and has no deterministic mode. Phase 1 currently exposes a PyTorch API; the JAX API below covers Phase 2 sparse attention.

Workspace and CUDA graphs

The convenience API allocates outputs, saved autograd state, and scratch. The long-sequence filtered indexer also keeps repair storage per device and stream. To reserve its maximum rectangular extents before training:

from cudnn.ops import simplified_sparse_attention_reserve
simplified_sparse_attention_reserve(max_query_len, max_key_len, topk=2048)

Reserve on the stream that will run attention. Calls above the reservation can grow storage and rebuild the internal repair graph. Short-key paths do not need persistent repair storage.

PyTorch host threads that share a stream serialize access to cached scratch, including reserve and H16 backward calls. Separate streams keep separate state. These locks do not synchronize the GPU.

External CUDA graph capture is unsupported, including after reservation or warmup. The public API rejects capture before launching work. Internal conditional CUDA graphs repair filtered rows; they do not provide capture support for the public training call. There is no cuDNN Graph API node.

JAX

cudnn.ops.simplified_sparse_attention_forward_jax returns a plain (output, indexer_loss) tuple. It supports jax.jit, jax.grad, and jax.vjp for the five activation inputs. Pass metadata as array arguments:

import jax
from cudnn.ops import simplified_sparse_attention_forward_jax
@jax.jit
def training_loss(q, k, v, q_idx, k_idx, starts, ends, valid, denominator):
output, indexer_loss = simplified_sparse_attention_forward_jax(
q, k, v, q_idx, k_idx, starts, ends, valid, denominator,
topk=512, loss_coeff=0.1,
)
return (output.astype("float32") ** 2).sum() + indexer_loss
gradients = jax.grad(training_loss, argnums=(0, 1, 2, 3, 4))

Shapes, dtypes, key bounds, padding, and loss scaling match the PyTorch API. topk, loss_coeff, and triangular_scores are static options. Metadata, index selection, and teacher targets are not differentiated. Use one process, one GPU per call, and only one supported GPU architecture visible to that process. Distributed execution, vmap, and higher-order derivatives are unsupported. PyTorch remains an import dependency of the shared kernel modules; execution uses native JAX FFI calls, without PyTorch tensor conversion or a Python callback.

XLA owns scratch and schedules kernels on its stream. There is no JAX reserve API or K/V producer event. Long rectangular inputs use the shared filtered selector and an XLA loop that repairs failed rows in groups of 128. They do not need PyTorch’s conditional-graph API. Square triangular input currently uses a dense score workspace in the JAX path.

External CUDA graph capture is unsupported for JAX as well. These FFI calls opt out of XLA CUDA command-buffer capture; JIT remains supported.

Neither API has a deterministic mode. FP32 atomic gradient reductions can vary with scheduling. Moving packed rows can also change scalar-loss rounding. See the determinism support page (GitHub) for measured limits.

Validation and benchmarks

The suite covers numerical forward/backward results, auxiliary loss scaling, invalid inputs, upstream-gradient ownership, top-k edge cases, capture rejection, bytewise replay, packed batch changes, and golden hashes.

cd test/python
pytest simplified_sparse_attention/cutedsl -m L0
# Longer filtered-indexer and repair checks:
pytest simplified_sparse_attention/cutedsl -m L1

The benchmark guide measures PyTorch eager and JAX JIT forward/training latency after warmup.