Skip to navigation

Sparse Attention

View as MarkdownOpen in Claude

These pages describe attention APIs with selected tokens, blocks, or masks, and related compression operations. The APIs are experimental and subject to change.

APIUse
Native Sparse Attention (NSA)Selection, compression, top-k, and sliding-window attention.
DeepSeek Sparse Attention (DSA)Sparse attention and its indexer, score, and top-k operations.
Simplified Sparse AttentionPacked attention with a trainable top-k indexer.
Query Sparse Attention (QSA)Attention over selected four-token blocks, with bitmap and indexed training APIs.
HSTU AttentionMasked attention for recommender models, with SiLU scores instead of softmax.
HSTU LMSDFused layer normalization, multiplication, SiLU, and dropout.
Block Sparse Attention (BSA)Attention over a caller-supplied block-sparse mask.
CSA Fused CompressorCompression for compressed sparse attention, with aligned HCA backward.
Flex AttentionAttention with reusable static-mask plans.

See Attention for scaled dot-product attention and Linear Attention for recurrent attention APIs.