Block Sparse Attention (BSA)
This is an experimental API and subject to change.
Overview
Block Sparse Attention computes non-causal scaled dot-product attention over a
block-level sparse pattern. For query token i, let m = floor(i / block_size)
be its query-block index and let K_m be the union of the key/value blocks
listed for that query block. The operation is
The softmax is normalized once over all valid tokens in the selected blocks.
block_sizes can shorten individual key/value blocks so that padded tokens do
not participate in the softmax.
Unlike NSA Selection Attention, BSA supplies one list of key/value blocks per query block. NSA Selection supplies routing metadata at query-token granularity and is one component of the larger NSA pipeline.
BSA is implemented with Python CuTe DSL/JIT kernels.
Installation
Except on Windows ARM64, the CuTe DSL runtime is a required dependency, so the base install is enough:
On Windows ARM64, the CuTe DSL and TVM-FFI runtime dependencies are excluded from the base installation, so the base install does not enable BSA there.
The package-wide nvidia-cutlass-dsl[cu13]>=4.8.0 dependency floor applies to
BSA. Sage FP8 relies on functionality introduced in CuTe DSL 4.6.1, which the
package floor already satisfies. A defensive runtime check remains for
environments that force an older DSL; the KPage path has the same check at
the package floor and was verified on CuTe DSL 4.6.2, 4.7.0, and 4.7.1.
Forward
The default layout is BHSD. Pass layout="bshd" for tensors in BSHD
layout. The output follows the input layout, while lse is always the FP32
natural-log log-sum-exp with shape (B, H_q, S_q).
For explicit Blackwell blk64 calls, TMA-aligned BSHD inputs are read through strided views. Both unsplit outputs and split-KV combined outputs are written directly in the requested layout. Misaligned inputs retain their staging fallback.
Sparse metadata
Here N_q = ceil(S_q / sparse_block_size) and
N_kv = ceil(S_kv / sparse_block_size). Except for packed GQA described below,
metadata is per query head. The metadata tensors must reside on the same CUDA
device as q, k, and v. The value ranges below are a hard caller contract.
Value ranges (caller contract)
Let K_max = q2k_block_index.shape[-1]. Only the active prefix of each
q2k_block_index row — the first q2k_block_nums[b, h, m] entries with
variable counts, or the first block_sparse_num entries with a fixed count —
is consumed. Values in the inactive suffix are ignored.
- Within each row, active
q2k_block_indexvalues must be unique integers in[0, N_kv). - For forward with variable counts, every
q2k_block_numsvalue must be in[0, K_max]whenallow_empty_block_nums=True, and in[1, K_max]otherwise. Backward variable counts may be in[0, K_max]. - With fixed counts,
block_sparse_nummust be in[1, K_max]. The FP16/BF16 SM100/SM103 blk128 path additionally requires an even value, i.e. an evenblock_sparse_numin[2, K_max]. - The
block_sizesentry for every physical KV block referenced by an activeq2k_block_indexvalue must be in[1, sparse_block_size]. Entries for unreferenced physical KV blocks are ignored. A zero-sized referenced block is not supported by regular FP16/BF16 attention; useq2k_block_nums(or the sparse index prefix) to drop the block instead. FP8 SM100/SM103 blk128 also accepts zero-sized blocks, as specified in its architecture contract below.
Tensor value ranges and per-row uniqueness are not validated at runtime. Violating the contract is unsupported and may produce invalid results or invalid device memory accesses.
On the SM100/SM103 blk128 path, pack_gqa=None automatically packs GQA when
the GQA ratio r = H_q / H_kv divides 128. Packed metadata has shape
(B, H_kv, ceil(S_q * r / 128), K_max). Pass pack_gqa=False to use the
unpacked (B, H_q, ceil(S_q / 128), K_max) contract on every architecture.
Explicit pack_gqa=True requires r to divide 128.
When block_sizes=None, each referenced physical KV block is treated as full.
Provide block_sizes whenever a referenced final block is only partially
valid.
sparse_block_size=None chooses blk64 on SM90/SM120 and blk128 on
SM100/SM103. On SM90 or SM120, passing sparse_block_size=128 selects a native
KV128 kernel that consumes blk128 metadata directly; it does not
expand the metadata or invoke the blk64 kernel. Passing sparse_block_size=64
explicitly selects the SM100/SM103 blk64 CuTe DSL path, whose shape support is
narrower. kv_splits is available on SM90 and the explicit Blackwell blk64
path; use_clc applies only to the explicit Blackwell blk64 path.
For SM120 blk128 with fixed block_sparse_num, full physical KV blocks
(block_sizes=None), and a KV sequence length divisible by 128, the dispatcher
uses an FA4-style native specialization with a dedicated K/V load warp,
register-resident Q, and a four-fold-unrolled sparse loop. Most CTAs process
128 Q rows; an underfilled final scheduling wave may use 64 Q rows per CTA
while still loading full KV128 blocks and using the parent Q block’s original
metadata. This is Q-work scheduling, not KV128-to-KV64 lowering. Variable
per-row block counts, explicit block sizes, and partial final KV blocks use
the general native blk128 kernel. The FA4-style specialization requires
nvidia-cutlass-dsl >= 4.7.0; an older public DSL version is rejected with
a version-specific error before the specialized kernel is imported. The
package-wide downstream dependency floor is unchanged.
kv_splits=2..256 computes FP32 partial outputs and combines them, with
workspace growing linearly in the split count. SM90 accepts an explicit integer
split count or kv_splits="auto" (the default stays 1). On SM90, once a head’s K/V
exceeds L2 the unsplit kernel turns DRAM-bound, and a split grid orders every Q tile of one KV
range before the next. The automatic policy splits only when a head’s K/V clearly exceeds L2,
with a threshold that rises as heads fall (above 1x L2 with 16+ heads, 1.5x with 4-15, 4x with
fewer), and then keeps about L2 / 2 of K/V per split while each split still holds at least 32
blocks of a row. The BF16/FP16 MHA policy applies at one head per wave (ceil(S_q / 64) at least
3 * SMs), at most 25% density and fixed counts, and declines (one split) otherwise. It was
measured on H100 with independent random indices and video-generation top-k masks (for example 128 MiB K/V per head with 4-8
heads: 4-8 splits, +27-67%; 256 MiB: 8-16 splits, up to +86%); 32 MiB per head never gained.
A caller-provided output also resolves to one split, and so does "auto" on SM90 blk128, where the
policy has not been measured. The SM100/SM103 blk64 path also accepts kv_splits="auto"; CLC is
compatible with split execution. Pass use_clc=True to use persistent CLC
scheduling with kv_splits>1, or use_clc=False to use one tile per CTA. The
default use_clc=None keeps CLC disabled for split execution because the
automatic scheduler policy has not been tuned for that combination.
The BF16 automatic policy is tuned assuming approximately uniform random KV
indices with little overlap between the neighborhoods of different Q blocks.
It uses host-visible tensor metadata and device properties, without reading
indices or counts back to the host. Highly overlapping, shared or local-window
indices can favor a different split count even at the same shape and density;
benchmark explicit kv_splits values for those masks. This assumption affects
performance only, not attention semantics or supported mask patterns.
For fewer than 512 total Q tiles (B * H * ceil(S_q / 64)), the existing policy
uses the fixed active block count, or index capacity for variable counts:
256—449 blocks select two splits, 450—899 select four, and 900 or more select
eight. Larger Q grids keep one split except in the measured long-Q region:
BF16 MHA with Q/K/V head dimensions 128, S_q >= 262144, a fixed count of at
least 512 active KV blocks, and density at most 1/8 (active blocks divided by
ceil(S_k / 64)). In that region, K+V bytes per batch and KV head above twice
the device L2 size select eight splits for at least 900 active blocks and four
or more heads; a single head requires at least 1024 active blocks. Otherwise,
four or more heads select four splits when per-head K+V bytes exceed L2.
All other cases, including an unavailable L2 size, keep one split. Variable
per-row counts also keep one split on larger Q grids because index capacity
does not describe actual row work. The Sage FP8 policy is separate.
Automatic selection
falls back to a smaller split count when the estimated live workspace does not
fit the available CUDA allocator budget; an explicit split count that exceeds
that budget raises RuntimeError.
Sage FP8 forward
Sage FP8 is forward-only and supports blk64 on SM90, SM100/SM103, and SM120,
plus native blk128 on SM90, SM100/SM103, and SM120. Its public wrapper accepts contiguous
BF16 Q, K, and V tensors in BHSD layout and performs FP8 quantization internally:
Q, K, and V must be BF16 MHA tensors in contiguous BHSD layout, have matching
batch and head counts, and use D=128. The wrapper accepts the same sparse
index metadata used by the regular forward API for the selected block size.
It lazily loads the quantizer, creates the E4M3 tensors and FP32 scales needed
by the kernel, and does not expose those implementation details as public
inputs or outputs.
sparse_block_size defaults to 64. On SM120, pass sparse_block_size=128 with
q2k_block_index shaped (B, H, ceil(S_q / 128), K_max) and, when supplied,
q2k_block_nums shaped (B, H, ceil(S_q / 128)). Active indices address
physical 128-token KV blocks, so they must be in [0, ceil(S_kv / 128)).
For block_sizes, use N_kv = ceil(S_kv / 128) and valid-token counts in
[1, 128] for referenced blocks. The native kernel consumes this metadata
directly. SM100/SM103 also support this blk128 metadata layout, with the
additional empty-block and split-KV semantics below.
On SM90, pass sparse_block_size=128 with indices shaped
(B, H, S_q // 128, capacity) and KV block IDs measured in units of 128 tokens.
The result is a one-key TupleDict containing o_tensor, a contiguous BF16
tensor of shape (B, H, S_q, 128). This API does not return LSE and has no
backward implementation.
V quantization keeps the reciprocal and multiplication in FP32 until the final E4M3 conversion, including the internal BHDS and blocked V layouts. Q retains the Sage BF16 intermediate-rounding recipe; K uses FP32 centering. FP32 rounding near an FP8 midpoint can select an adjacent E4M3 value across implementations.
SM120 per-token Q/K scaling
On SM120 blk128, set k_quant_block_size=1 to use one FP32 scale for every
(batch, head, token) of both Q and K, covering all 128 head features.
The default k_quant_block_size=16 preserves the existing recipe. This
argument controls numerics, not the sparse block size or an autotuning knob.
K is still centered by its FP32 sequence mean before quantization. Its
per-token descale is max(amax(abs(K_centered[token, :])), 1e-3) / 448,
and the quotient is rounded to E4M3. Q retains its existing BF16 intermediate
rounding, and V keeps one FP32 scale per (head, channel), reduced over
batch and the entire key sequence. This does not use MXFP8 block-scale
instructions, Q centering, or a Q-mean/delta correction.
The kernel applies each K descale to its own QK accumulator column before the maximum reduction and exponentiation. P uses the unchanged factor of 256 before E4M3 conversion, with an unrounded FP32 denominator. V descales are applied to output channels after PV accumulation. A transposed V storage tile of 128 tokens is only a layout choice, not a V quantization block.
For blocked V storage, the per-token kernel establishes the maximum over the complete 128-key score tile, then converts and consumes P in 32- or 64-key chunks. This interleaves probability conversion with PV while retaining the original ordered FP32 denominator and K32 MMA accumulation order. The chunk width is an internal scheduling choice with a separate compile-cache key; it does not change the quantization granularity or normalize each chunk independently. The coarse-K and unblocked-V paths retain their existing schedule.
Per-token K requires CuTe DSL >= 4.8.0, SM120, sparse_block_size=128, MHA,
and D=128. Tails, variable sparse counts, empty rows, and partial KV blocks
follow the existing SM120 contract. Other architectures and blk64 reject
this option rather than silently falling back to a coarser recipe. Finer
scales can reduce quantization error, but are not a guarantee of better
model quality on every input; kernel and quantization-inclusive latency
should both be measured for the intended workload.
The SM100/SM103 blk128 kernel retains the upstream rounded-probability normalization: P uses a factor of 128, and its denominator is accumulated from the same rounded FP8 probabilities by the widened PV Tensor Core operation. Other FP8 kernels use their existing normalization. Outputs are approximate and are not expected to be bitwise equal across block sizes or architectures.
The architecture-specific FP8 contracts are:
- SM90 accepts any positive batch and head counts and requires both sequence
lengths to be multiples of the selected
sparse_block_size(64 or 128). It uses fixedblock_sparse_numwith full KV blocks of that size. Native blk128 shares each K/V tile between two compute warpgroups. Sparse indices may be non-contiguous (including prefix slices, stepped slices, permuted views, and broadcasts); their strides are consumed directly without an index copy.q2k_block_numsandblock_sizesare not supported. It does not use split-KV. V quantization writes sequence-contiguous BHDS storage internally, so the attention kernel directly loads V for Hopper FP8 WGMMA without an in-kernel transpose. Both block sizes accumulate PV directly into FP32 output registers using FP8 WGMMA, accepting its reduced internal accumulation precision instead of promoting each KV tile with a separate FP32 addition. - SM100/SM103 blk64 accepts any positive batch and head counts,
and requires both sequence lengths to be multiples of 64. It uses fixed
block_sparse_numwith full 64-token KV blocks;q2k_block_numsandblock_sizesare not supported. Split-KV is selected internally, and the public FP8 API does not exposekv_splitsoruse_clc. - SM100/SM103 native blk128 accepts positive arbitrary Q/KV lengths, including
physical tails, with MHA and D=128. Fixed counts may be odd; variable
q2k_block_numsmay be zero, producing zero output.block_sizesaccepts rank 1, 2, or 3 and values in[0, 128]; zero masks a whole block, and the physical final-block length always limits the supplied extent. Inactive index capacity is ignored. Sparse metadata views are materialized as needed. The optionalkv_splits=1..256partitions the live KV list using FP32 partials and the existing combine kernel; the default is one. Other FP8 paths require the new keyword to remain at its default.softmax_scale, when supplied, must be a positive finite host scalar. This implementation is limited to compute capabilities 10.0 and 10.3 and requires CuTe DSL >= 4.6.2. - SM120 supports blk64 and blk128, accepts any positive batch and head counts,
non-aligned Q/KV sequence tails, fixed or per-query-block counts, and
block_sizesshaped(N_kv,),(B, N_kv), or(B, H, N_kv). It does not use split-KV.
block_sparse_attention_fp8_forward relies on functionality introduced in
CuTe DSL 4.6.1; the SM90 blk128 path requires 4.6.2. Package-supported
installations provide CuTe DSL 4.8.0 or newer.
Its internal Q/K/V quantizer is also implemented in CuTe DSL and is
loaded lazily, so importing cudnn does not eagerly import it.
Backward
Backward is an explicit API rather than a registered PyTorch autograd operation. It recomputes probabilities from the forward output and LSE:
The result keys are dq_tensor, dk_tensor, and dv_tensor. Optional
preallocated dq_tensor, dk_tensor, and dv_tensor arguments are supported.
The backward implementation builds a bucketed K-to-Q CSR task layout on the
GPU. bucket_size_blocks is an optional tuning override; leaving it unset uses
the backend default.
For SM100/SM103 blk128, long-query automatic buckets retain at least one SM
wave of logical (batch, head, KV block, query group) tasks, or the previous
default’s task count when it is smaller. Otherwise, the previous bucket
default is used. This shape-based guard does not inspect device-side mask
contents; explicit bucket_size_blocks overrides remain authoritative.
For blk64, with contiguous per-row count metadata, at least two KV blocks and at least
1024 (batch, head, KV block) tasks, backward pairs KV neighborhoods to reuse
common Q/dO loads and dQ accumulation; smaller problems do not recover the
per-call pairing cost and use regular tasks.
SM90 pairs KV neighborhoods the same way and runs each matched pair in one
blk128-shaped kernel: a CTA walks the pair’s Q blocks once, ordered
first-only / common / second-only, and each of its two compute warpgroups
computes only the Q blocks of its own KV block, so Q/dO loads and the dQ
reduction are shared and Q/dO are read in a single sweep.
Pairing metadata is limited to 64 MiB; unsupported metadata layouts use regular tasks, and so do
fixed counts on SM100. On SM90, fixed counts pair as well (the wrapper supplies the full per-row
count), which measured faster on long video-generation masks; the query bucket stays 384, which
measured fastest against 96 through a single group. Duplicate sparse edges retain their multiplicity.
Paired tasks retain FP32 dK/dV accumulation even with a single query bucket;
on SM90 and SM100, regular tasks store dK/dV directly in the output dtype, without the FP32
workspace and its conversion pass, when all query blocks fall in one query bucket and
seqlen_k is a multiple of 64.
The initial eight-KV matching is refined within 32-KV windows, with refinement
skipped above 1024 bitset words. Metadata construction is included in the
wrapper’s work and is rebuilt for each call.
Backward interleaves its tasks across attention heads only for queries shorter
than 98,304 tokens, where it was measured to help; longer queries keep the
head-major task order.
These optimizations keep the public block size at 64 and preserve the existing
wrapper-owned workspace model. They do not add an allocation-free graph engine.
Use benchmark/bsa/benchmark_sm100_blk64.py for a synthetic complete-API
comparison; it records the imported source, input hashes and individual CUDA
Event samples. Numerical tests, sanitizer checks and measurements on each
target GPU are required before drawing performance conclusions.
Prepared backward metadata (SM90 blk64)
On SM90 with sparse_block_size=64 the backward’s metadata kernels (the
bucketed K-to-Q CSR build and, on the paired-KV path, the pair bitsets,
matching, refinement and partitioned CSR; about 0.33 ms and ten launches on
a 255K-token production mask) depend on the sparse metadata and the sequence
lengths only, never on q, k, v or dout. block_sparse_attention_backward_metadata
runs them once and returns a BsaBackwardMetadata; a backward that takes it
through metadata= launches only the preprocess, the main kernel and the
dK/dV/dQ conversions (plus, on the paired-KV path, a copy of the object’s small
per-call dK/dV arrival bookkeeping), so a mask that serves several backward passes (every
layer of a training step, or the backward of the forward that has just been
launched) pays for its metadata once, off the backward’s critical path.
The positional arguments of block_sparse_attention_backward_metadata have the
backward’s meaning; bucket_size_blocks is fixed when the object is built
(None selects the SM90 default) and a backward may pass the same value or
None. The object keeps the metadata tensors alive and remembers their
storage address, shape, strides, dtype, device and in-place version counter:
a backward that passes it with other metadata tensors, with the same tensors
after an in-place change, with another batch, head count or sequence length,
or with another bucket_size_blocks raises ValueError instead of walking a
stale pattern. Rebuild the object after changing the metadata. The prepare
call issues kernels on the current stream of the metadata’s device and does
not synchronize the host, so it can run on a side stream (for example right
after the forward launch) with the usual event ordering before the backward;
the backward records the object’s tensors on its own stream when it consumes
them, so the object may be released as soon as the backward call returns,
even when it was built on another stream. Both the default backward and the
prepared one can be captured into a CUDA graph. metadata= is exclusive with
kpage= (a KPageLayout already holds its metadata) and is not accepted by
the SM100/SM110 backward or by sparse_block_size=128.
Backward defaults to blk64 on SM90 and blk128 on SM100/SM103. Pass the same
explicit sparse_block_size used by forward when selecting the Blackwell blk64
path. SM100/SM103 blk128 backward accepts one-dimensional block_sizes
(int32, shape (N_kv,)) and masks each KV block’s padded suffix consistently
with forward. Valid tokens occupy the prefix of each physical block; callers
zero dout for padded queries and discard their outputs. SM110 blk128 backward
still requires block_sizes=None.
KPage (blk128 on SM100/SM103 and SM90, partially occupied blocks)
kpage=True on block_sparse_attention_forward and
block_sparse_attention_backward selects a second set of blk128 kernels for
sparse patterns in which many selected blocks are only partially valid, as
happens when a 3D video latent is flattened and block_sizes marks the rows
past each block’s content. The base kernels load and compute every selected
block as a full 128-row tile and mask the padding; KPage splits each selected
K/V block into 16-row pages and gathers only the valid pages, from any blocks,
into the same 128-row MMA tile during the TMA load. A block with v valid
rows then costs 16 * ceil(v / 16) rows instead of 128. Query blocks with at
most 64 valid rows (see q_block_sizes below) run on an M64 x N256 kernel,
the others on M128 x N128. Backward packs K/V pages into owner tiles the same
way and walks the union of their Q pages with a per-step page-pair mask, so
block_sizes is supported in backward on this path.
On SM90 the same calls run without a separate blk128 KPage kernel. The layout
rewrites the metadata as 64-row halves: KV block j becomes halves 2j
(min(block_sizes[j], 64) rows) and 2j + 1 (the rest), and Q block i
becomes halves 2i and 2i + 1 with the same list. Halves without valid
rows are dropped: a KV half leaves every list, and a Q half (from
q_block_sizes) gets an empty list. The forward runs the SM90 blk64 KPage
kernel on these halves (described below), where each pair of halves is one
128-row query block. The backward runs the SM90 blk64 backward on them, with
LSE = +inf on padding query rows so they take no part. The outputs follow
the same contract as on SM100/SM103.
kpage=True computes the same function as the base path with the same
arguments (block_sizes masks keys, every query row is valid) and builds
the page layout on the call. kpage_prepare_layout builds a KPageLayout
once from the metadata so repeated calls with the same sparse pattern (every
layer of a diffusion step, every training step with a fixed packing) skip the
metadata kernels. The layout depends on the metadata only, never on q, k,
or v. Backward tables are built on the first backward call with that layout.
Prefer a prebuilt layout. The walk tables are sized from counts computed on
the device, so building a layout synchronizes the host with the current
stream: the host waits until all previously queued GPU work has finished,
kernels queued after the call start late, and the call cannot be captured into
a CUDA graph. kpage=True builds a layout on every call (the forward and the
backward each build their own), so it pays this on every layer. Calls that
take a KPageLayout do not synchronize. Build one layout per sparse pattern
outside the layer loop and pass it to the forward and backward of every layer
that uses the pattern; on SM100/SM103 also pass with_backward=True so the
backward tables are not built (with one more synchronization) on the first
backward call. Keep kpage=True for one-off calls and experiments.
q_block_sizes (int32, one entry per query block) is specific to this path:
query rows at or past q_block_sizes[i] in block i are padding. They are
not computed, which is where the M64 x N256 route and part of the speedup on
video layouts come from, and their rows of o, lse, and dq are written
as zero. Inputs whose query side carries the same per-block padding as the
key side pass q_block_sizes=block_sizes.
Contract, in addition to the metadata ranges above:
- SM100/SM103,
sparse_block_size=128, BF16,head_dim=128; MHA, GQA, and MQA with metadata per query head (pack_gqa=Nonebehaves asFalse,pack_gqa=Trueis not taken);seqlen_qandseqlen_kindependent and arbitrary (rows past a sequence end are padding of its final block, as on the base path);block_sizesof shape(N_kv,). BHSD or BSHD tensors (q/k/v/o/do and the gradients) that are compact in their layout. A batch is run asbatch * headsindependent heads, soblock_sizesis shared across the batch (as on the base SM100 path) and a BSHD batch must be 1.kv_splits,use_clc, andbucket_size_blocksare not taken; a fixed count may be odd. - SM90 differs: MHA only,
seqlen_qandseqlen_kmust be multiples of 64 (a sequence may end 64 rows into its final block), andbwd_kv_groupsis not taken. allow_empty_block_numshas the base meaning: withTrue, a query block withq2k_block_nums == 0getsO = 0,LSE = -inf, anddQ = 0; withFalse(default) such a block is a metadata error. Backward accepts empty blocks as on the base path.- Masked key rows (at or past
block_sizes[j]in blockj) ofkandv, and padding query rows ofqanddowhenq_block_sizesis given, take no part in the computation and may hold any finite value. Backward writes zeros to the masked rows ofdkanddv. Values on valid rows are exact. kpage_prepare_layout(validate=True)(default) checks the metadata on the device and raisesValueErrornaming the offending entry. The check is read back together with the walk-table sizes, so a layout build synchronizes the host once with or without it;with_backward=Truealso builds the backward tables (one more synchronization on SM100/SM103), which otherwise happens on the first backward call.seqlen_q/seqlen_kare needed only when a sequence is not a multiple of 128 or whenblock_sizesis omitted andseqlen_k != seqlen_q(kpage=Truetakes them from the tensors).- A
KPageLayoutremembers the metadata tensors it was built from; passing it together with other metadata, or with tensors of another shape, raisesValueError.
The forward output of the KPage path can be fed to the KPage backward or to a
base backward with block_sizes=None when every block is full; the two paths
compute the same function and differ only in the execution layout.
When to use it: the gain comes from padding (block_sizes < 128, and
q_block_sizes for the query side) and grows with the padding fraction
(1.2-1.3x at 10%, 3.2-3.8x at 59% in benchmark/bsa; on SM90, forward
1.2-1.4x at 10% and 3.6-3.8x at 59%, forward + backward 1.1-1.2x and
2.5-2.7x). Without
padding the page gather costs more than the base kernel’s whole-block load,
so with block_sizes=None the base path is faster (measured 0.6x at four
blocks per row, 0.96x at 128; SM90 forward 0.84-0.91x, backward
0.98-1.12x). GQA/MQA run with per-query-head metadata; the
base path’s packed GQA is not available here. BSHD inputs are supported but
the strided page gathers make the forward slower than on BHSD (measured 1.3x
on a 16-head layout); prefer BHSD. A seqlen_k that is not a multiple of 16
selects forward kernels whose K/V descriptors carry the row extent instead of
whole 16-row pages (measured 3-5% slower forward; the backward is
unaffected), so pad K/V to a multiple of 16 rows when that matters. A call
carries about 0.1 ms of host work (metadata checks, tensor conversion), which
only matters for sub-millisecond problems.
KPage forward on blk64 metadata
On SM100/SM103 kpage=True (or a layout built with
kpage_prepare_layout(..., sparse_block_size=64)) also takes
sparse_block_size=64 metadata, for the forward only. Every 64-row query
block is one tile of the M64 x N256 kernel, whose 256-row K/V stage is
gathered from sixteen 16-row pages, so a selected 64-row block with v valid
rows costs 16 * ceil(v / 16) rows instead of 64.
On SM90 the same call runs a kernel that pairs adjacent query blocks
(2p, 2p + 1): one CTA walks the union of their KV block lists as 16-row
pages gathered into 128-row K/V tiles, ordered first-only, common,
second-only, and its two compute warpgroups (one per query block) compute only
their own blocks’ tiles. K/V of the blocks both query blocks select are thus
loaded once, on top of the page compaction. The layout holds these pair walks
when it is built on an SM90 device.
The contract is that of the base blk64 forward with these differences:
- BF16,
head_dim=128, MHA;seqlen_qandseqlen_kmultiples of 64; a BSHD batch must be 1, as above. - Every query row is computed, as on the base blk64 path: there is no
q_block_sizes, and the output rows of query padding are not zeroed. - Forward only.
block_sparse_attention_backward(..., kpage=...)withsparse_block_size=64raises; run the base backward on the KPage forward output. A layout is bound to thesparse_block_sizeit was built for.
When to use it: as on blk128, the gain comes from block_sizes < 64 and
grows with the padding fraction, and without padding the page gather costs
more than the base kernel’s whole-block load. On random top-k patterns
(B300, BHSD, layout reused) it measured 1.11-1.36x at 30% padded tokens,
0.91-1.08x at 10%, and 0.85-0.96x without padding. Building the layout costs
about 0.3-1 ms per call on those shapes, so reuse a layout from
kpage_prepare_layout whenever the metadata repeats; with kpage=True the
build is paid on every call. Benchmark your own patterns before switching.
On SM90 the gain also grows with the overlap of adjacent query blocks’ KV
lists (local or video-like patterns); full blocks with little overlap, such as
independent top-k selections, run slower than the base kernel.
Current support
Forward
For FP16/BF16, SM90 blk64 requires S_q to be a multiple of 64; native blk128 supports
positive arbitrary Q/KV lengths, including partial final blocks. Its fixed count may be
any positive value. The SM100/SM103 blk128 fixed count must be even and at
least two (the KPage path accepts any positive count); SM120 and the explicit
Blackwell blk64 path accept any positive fixed count. Variable counts use q2k_block_nums. allow_empty_block_nums
defaults to False; when it is True, empty rows (q2k_block_nums == 0)
produce O = 0 and LSE = -inf. SM90 blk64 selects the empty-row handling as a
compile-time specialization, so the default non-empty configuration keeps its
branch-free fast path. Its split-KV execution excludes empty rows. Native SM90
blk128 supports empty rows and empty splits with kv_splits=1..256.
Sage FP8 uses the architecture-specific contracts above. SM100/SM103 native
blk128 supports empty rows and explicit split-KV; its default remains unsplit.
Backward
On SM90 with sparse_block_size=64 the backward also accepts a prepared BsaBackwardMetadata
through metadata= (see Prepared backward metadata);
every other path rejects the argument.
Backward is not implemented for SM120. It requires equal QK/V dimensions and, except on the SM100/SM103 KPage path, does not currently support GQA/MQA.
SM90 native blk128 requires CuTe DSL >= 4.6.2. Q/K/V, O/dO and gradient
buffers must have a contiguous head dimension, 16-byte aligned base pointers,
and row strides divisible by 16 bytes. BHSD/BSHD layouts and aligned noncompact
rows are addressed directly. Sparse metadata can be strided and remains on the
GPU; backward makes noncontiguous sparse indices and counts contiguous before
building CSR, as in blk64. Forward supports block_sizes of rank 1, 2, or 3; backward supports ranks
1 and 2. Both paths predicate sequence tails even without block_sizes.
Forward uses Q128 tiles and shares each KV128 load across two compute
warpgroups. Backward uses Q64xKV128 CTAs: both compute warpgroups execute all
five GEMMs, splitting KV rows for QK/dP/dK/dV and head dimensions for dQ.
Each native Q128/KV128 CSR edge streams two Q64 subtiles without expanding
the metadata. Backward accumulates gradients with FP32 atomics
and is not deterministic. Benchmark the explicit path for your shapes; blk64
remains the default, and native blk128 is not faster for every workload.
Limitations
The current sparse kernels do not implement causal or local masking, dropout,
mask_mod, score_mod, paged KV cache, softcap, or variable-length packed
sequences. Regular forward/backward inputs must be rank four, use FP16/BF16 as
allowed above, and have a contiguous last (head-dimension) axis. Sage FP8 uses
the stricter BF16 BHSD contract described above and quantizes to E4M3
internally. Regular forward outputs are contiguous in the requested BHSD or
BSHD layout, including split-KV execution; FP8 output is contiguous BF16 BHSD.
Compilation is lazy. The first call for a new static configuration JIT-compiles the relevant kernel; subsequent calls reuse an in-process cache.
The Torch public surface consists of allocating function wrappers under
cudnn.ops, also available under cudnn and cudnn.BSA; there is no separate APIBase class or explicit compile()
lifecycle for BSA.
Correctness tests and FP32 references are under
test/python/block_sparse_attention/cutedsl.
SM100/SM103 FP8 blk128 source
The native producer and its private transfer/epilogue helpers are adapted from
NVIDIA-JerryChen/Block-Sparse-Attention, f953680:
bsa_fwd_sm100_fp8_fused_pv.py and bsa_fwd_sm100_fp8.py map to the same
filenames under python/cudnn/block_sparse_attention/csrc/fwd/sm100_blk128/.
The 16-token TMEM row-max helper is incorporated into the private blk128 helper
module. The integration uses cuDNN’s existing quantizer, allocating Torch API,
device-specific JIT cache, current-stream execution, and split combine.
Standalone upstream package entry points and native AOT deployment are not
part of this API.
Acknowledgements
We would like to express our gratitude to huangyitong.hyt@alibaba-inc.com and wenting.swt@alibaba-inc.com for providing testing and optimization feedback throughout the deployment process, which has continuously advanced the BSA kernel toward Speed of Light.
Experimental JAX support
cudnn.ops.block_sparse_attention_forward_jax and
cudnn.ops.block_sparse_attention_backward_jax provide explicit forward/backward;
cudnn.block_sparse_attention.jax_api.block_sparse_attention adds first-order reverse-mode differentiation.
All three accept JAX arrays, eagerly or under jax.jit, and require no PyTorch.
The cudnn.jax namespace has been removed; use the cudnn.ops names.
The JAX implementation reuses the SM100/SM103 blk128 forward, bucketed CSR, backward preprocess,
backward, and gradient conversion kernels. Install jax[cuda13] alongside the
frontend. The runtime imports no PyTorch; torch parity tests are separate.
The JAX signatures omit Torch-specific launch options and caller-provided output buffers. XLA owns the outputs and workspaces.
Initial contract
Variable counts must be in [1,C], or [0,C] with
allow_empty_block_nums=True. Only each row’s active prefix is read. Active
indices must be unique and in [0,Sk/128). An empty row returns zero output,
negative-infinity LSE, and zero query gradient. Unselected K/V receive zero
contributions. Metadata values are the caller’s responsibility: the runtime
validates shapes, dtypes, and static options, without copying metadata to the
host or synchronizing to inspect it. Invalid values are outside the contract.
SM103/SM110/SM120, blk64, FP16/FP8, GQA, different V dimensions, partial blocks,
variable sequence lengths, causal masks, dropout, sharding, vmap, JVP, and
higher derivatives are unsupported. KDA is outside this change. Static options
(including scale and fixed count) specialize compilation; sparse arrays remain
runtime operands. Do not mutate saved forward inputs or metadata before backward.
Usage
Explicit forward returns a JAX pytree with o_tensor, lse_tensor; backward returns dq_tensor,
dk_tensor, dv_tensor. Both support tuple unpacking and dictionary access. Explicit helpers
do not register autodiff rules; use attention for jax.grad. LSE and metadata
are not differentiable public outputs. Backward must receive the exact matching
forward Q/K/V/O/LSE, sparse pattern, scale, layout, and empty-row option.
Execution and ownership
Every launch receives XLA’s CUDA stream. Forward is one custom call; backward composes CSR construction, preprocessing, attention backward, and gradient conversion inside one custom call. XLA owns all outputs and workspaces. CSR counts and multi-bucket dK/dV accumulators are initialized for each invocation; preprocessing clears dQ accumulation. No mutable global scratch or public input buffer is donated. Atomic accumulation does not promise bitwise determinism.
TensorSpec.mode presents compact BHSD/BSHD memory in the order expected by each
kernel. It does not rewrite DLPack capsules or insert a Python transpose. XLA may
still insert layout conversions when surrounding operations use incompatible
physical layouts; this is not a universal zero-copy guarantee.
Two cached call builders retain static configuration and kernel objects, never
input arrays or pointers. XLA handles compilation and execution; there is no
separate plan lifecycle. One custom_vjp registration connects forward and
backward. bucket_size_blocks optionally controls backward query buckets;
the default reuses the torch path’s heuristic. Multi-device placement and cache
portability across architectures require further qualification.
The JAX tests live in test/python/block_sparse_attention/jax and are collected by the OSS
suite. Torch-free import and gradient checks run in fresh subprocesses. To run
the suite in a torch-free container, pass --confcutdir=test/python/block_sparse_attention/jax
so pytest does not load the torch-based parent conftest.