Skip to navigation

Block Sparse Attention (BSA)

View as MarkdownOpen in Claude

This is an experimental API and subject to change.

Overview

Block Sparse Attention computes non-causal scaled dot-product attention over a block-level sparse pattern. For query token i, let m = floor(i / block_size) be its query-block index and let K_m be the union of the key/value blocks listed for that query block. The operation is

Oi=softmaxj∈Km(QiKjTD)Vj.O_i = \text{softmax}_{j \in K_m} \left(\frac{Q_i K_j^T}{\sqrt{D}}\right)V_j.

The softmax is normalized once over all valid tokens in the selected blocks. block_sizes can shorten individual key/value blocks so that padded tokens do not participate in the softmax.

Unlike NSA Selection Attention, BSA supplies one list of key/value blocks per query block. NSA Selection supplies routing metadata at query-token granularity and is one component of the larger NSA pipeline.

BSA is implemented with Python CuTe DSL/JIT kernels.

Installation

Except on Windows ARM64, the CuTe DSL runtime is a required dependency, so the base install is enough:

pip install nvidia-cudnn-frontend

On Windows ARM64, the CuTe DSL and TVM-FFI runtime dependencies are excluded from the base installation, so the base install does not enable BSA there.

The package-wide nvidia-cutlass-dsl[cu13]>=4.8.0 dependency floor applies to BSA. Sage FP8 relies on functionality introduced in CuTe DSL 4.6.1, which the package floor already satisfies. A defensive runtime check remains for environments that force an older DSL; the KPage path has the same check at the package floor and was verified on CuTe DSL 4.6.2, 4.7.0, and 4.7.1.

Forward

import torch
from cudnn.ops import (
block_sparse_attention_forward,
block_sparse_attention_backward,
block_sparse_attention_fp8_forward,
)
q = torch.randn(1, 8, 1024, 128, device="cuda", dtype=torch.bfloat16)
k = torch.randn(1, 8, 2048, 128, device="cuda", dtype=torch.bfloat16)
v = torch.randn_like(k)
# For the SM90 blk64 path, this example has 16 Q blocks and 32 KV blocks.
# Every Q block attends to the first four KV blocks.
q2k_block_index = torch.arange(4, device="cuda", dtype=torch.int32)
q2k_block_index = q2k_block_index.view(1, 1, 1, 4).expand(1, 8, 16, 4).contiguous()
block_sizes = torch.full((32,), 64, device="cuda", dtype=torch.int32)
result = block_sparse_attention_forward(
q,
k,
v,
q2k_block_index,
block_sparse_num=4,
block_sizes=block_sizes,
sparse_block_size=64,
)
o, lse = result
# The same values are available as result["o_tensor"] and result["lse_tensor"].

The default layout is BHSD. Pass layout="bshd" for tensors in BSHD layout. The output follows the input layout, while lse is always the FP32 natural-log log-sum-exp with shape (B, H_q, S_q).

For explicit Blackwell blk64 calls, TMA-aligned BSHD inputs are read through strided views. Both unsplit outputs and split-KV combined outputs are written directly in the requested layout. Misaligned inputs retain their staging fallback.

Sparse metadata

ArgumentShapeDtypeMeaning
q2k_block_index(B, H_q, N_q, K_max)int32KV-block IDs; only the valid prefix of each last dimension is read
block_sparse_numscalarPython intFixed valid-prefix length for every query block
q2k_block_nums(B, H_q, N_q)int32Optional per-query-block valid-prefix lengths; overrides block_sparse_num
block_sizes(N_kv,) or backend-supported batched formint32Number of valid tokens in each KV block

Here N_q = ceil(S_q / sparse_block_size) and N_kv = ceil(S_kv / sparse_block_size). Except for packed GQA described below, metadata is per query head. The metadata tensors must reside on the same CUDA device as q, k, and v. The value ranges below are a hard caller contract.

Value ranges (caller contract)

Let K_max = q2k_block_index.shape[-1]. Only the active prefix of each q2k_block_index row — the first q2k_block_nums[b, h, m] entries with variable counts, or the first block_sparse_num entries with a fixed count — is consumed. Values in the inactive suffix are ignored.

  • Within each row, active q2k_block_index values must be unique integers in [0, N_kv).
  • For forward with variable counts, every q2k_block_nums value must be in [0, K_max] when allow_empty_block_nums=True, and in [1, K_max] otherwise. Backward variable counts may be in [0, K_max].
  • With fixed counts, block_sparse_num must be in [1, K_max]. The FP16/BF16 SM100/SM103 blk128 path additionally requires an even value, i.e. an even block_sparse_num in [2, K_max].
  • The block_sizes entry for every physical KV block referenced by an active q2k_block_index value must be in [1, sparse_block_size]. Entries for unreferenced physical KV blocks are ignored. A zero-sized referenced block is not supported by regular FP16/BF16 attention; use q2k_block_nums (or the sparse index prefix) to drop the block instead. FP8 SM100/SM103 blk128 also accepts zero-sized blocks, as specified in its architecture contract below.

Tensor value ranges and per-row uniqueness are not validated at runtime. Violating the contract is unsupported and may produce invalid results or invalid device memory accesses.

On the SM100/SM103 blk128 path, pack_gqa=None automatically packs GQA when the GQA ratio r = H_q / H_kv divides 128. Packed metadata has shape (B, H_kv, ceil(S_q * r / 128), K_max). Pass pack_gqa=False to use the unpacked (B, H_q, ceil(S_q / 128), K_max) contract on every architecture. Explicit pack_gqa=True requires r to divide 128.

When block_sizes=None, each referenced physical KV block is treated as full. Provide block_sizes whenever a referenced final block is only partially valid.

sparse_block_size=None chooses blk64 on SM90/SM120 and blk128 on SM100/SM103. On SM90 or SM120, passing sparse_block_size=128 selects a native KV128 kernel that consumes blk128 metadata directly; it does not expand the metadata or invoke the blk64 kernel. Passing sparse_block_size=64 explicitly selects the SM100/SM103 blk64 CuTe DSL path, whose shape support is narrower. kv_splits is available on SM90 and the explicit Blackwell blk64 path; use_clc applies only to the explicit Blackwell blk64 path.

For SM120 blk128 with fixed block_sparse_num, full physical KV blocks (block_sizes=None), and a KV sequence length divisible by 128, the dispatcher uses an FA4-style native specialization with a dedicated K/V load warp, register-resident Q, and a four-fold-unrolled sparse loop. Most CTAs process 128 Q rows; an underfilled final scheduling wave may use 64 Q rows per CTA while still loading full KV128 blocks and using the parent Q block’s original metadata. This is Q-work scheduling, not KV128-to-KV64 lowering. Variable per-row block counts, explicit block sizes, and partial final KV blocks use the general native blk128 kernel. The FA4-style specialization requires nvidia-cutlass-dsl >= 4.7.0; an older public DSL version is rejected with a version-specific error before the specialized kernel is imported. The package-wide downstream dependency floor is unchanged.

kv_splits=2..256 computes FP32 partial outputs and combines them, with workspace growing linearly in the split count. SM90 accepts an explicit integer split count or kv_splits="auto" (the default stays 1). On SM90, once a head’s K/V exceeds L2 the unsplit kernel turns DRAM-bound, and a split grid orders every Q tile of one KV range before the next. The automatic policy splits only when a head’s K/V clearly exceeds L2, with a threshold that rises as heads fall (above 1x L2 with 16+ heads, 1.5x with 4-15, 4x with fewer), and then keeps about L2 / 2 of K/V per split while each split still holds at least 32 blocks of a row. The BF16/FP16 MHA policy applies at one head per wave (ceil(S_q / 64) at least 3 * SMs), at most 25% density and fixed counts, and declines (one split) otherwise. It was measured on H100 with independent random indices and video-generation top-k masks (for example 128 MiB K/V per head with 4-8 heads: 4-8 splits, +27-67%; 256 MiB: 8-16 splits, up to +86%); 32 MiB per head never gained. A caller-provided output also resolves to one split, and so does "auto" on SM90 blk128, where the policy has not been measured. The SM100/SM103 blk64 path also accepts kv_splits="auto"; CLC is compatible with split execution. Pass use_clc=True to use persistent CLC scheduling with kv_splits>1, or use_clc=False to use one tile per CTA. The default use_clc=None keeps CLC disabled for split execution because the automatic scheduler policy has not been tuned for that combination.

The BF16 automatic policy is tuned assuming approximately uniform random KV indices with little overlap between the neighborhoods of different Q blocks. It uses host-visible tensor metadata and device properties, without reading indices or counts back to the host. Highly overlapping, shared or local-window indices can favor a different split count even at the same shape and density; benchmark explicit kv_splits values for those masks. This assumption affects performance only, not attention semantics or supported mask patterns.

For fewer than 512 total Q tiles (B * H * ceil(S_q / 64)), the existing policy uses the fixed active block count, or index capacity for variable counts: 256—449 blocks select two splits, 450—899 select four, and 900 or more select eight. Larger Q grids keep one split except in the measured long-Q region: BF16 MHA with Q/K/V head dimensions 128, S_q >= 262144, a fixed count of at least 512 active KV blocks, and density at most 1/8 (active blocks divided by ceil(S_k / 64)). In that region, K+V bytes per batch and KV head above twice the device L2 size select eight splits for at least 900 active blocks and four or more heads; a single head requires at least 1024 active blocks. Otherwise, four or more heads select four splits when per-head K+V bytes exceed L2. All other cases, including an unavailable L2 size, keep one split. Variable per-row counts also keep one split on larger Q grids because index capacity does not describe actual row work. The Sage FP8 policy is separate.

Automatic selection falls back to a smaller split count when the estimated live workspace does not fit the available CUDA allocator budget; an explicit split count that exceeds that budget raises RuntimeError.

Sage FP8 forward

Sage FP8 is forward-only and supports blk64 on SM90, SM100/SM103, and SM120, plus native blk128 on SM90, SM100/SM103, and SM120. Its public wrapper accepts contiguous BF16 Q, K, and V tensors in BHSD layout and performs FP8 quantization internally:

fp8_result = block_sparse_attention_fp8_forward(
q,
k,
v,
q2k_block_index,
block_sparse_num=4,
sparse_block_size=64,
)
o_fp8 = fp8_result["o_tensor"]

Q, K, and V must be BF16 MHA tensors in contiguous BHSD layout, have matching batch and head counts, and use D=128. The wrapper accepts the same sparse index metadata used by the regular forward API for the selected block size. It lazily loads the quantizer, creates the E4M3 tensors and FP32 scales needed by the kernel, and does not expose those implementation details as public inputs or outputs.

sparse_block_size defaults to 64. On SM120, pass sparse_block_size=128 with q2k_block_index shaped (B, H, ceil(S_q / 128), K_max) and, when supplied, q2k_block_nums shaped (B, H, ceil(S_q / 128)). Active indices address physical 128-token KV blocks, so they must be in [0, ceil(S_kv / 128)). For block_sizes, use N_kv = ceil(S_kv / 128) and valid-token counts in [1, 128] for referenced blocks. The native kernel consumes this metadata directly. SM100/SM103 also support this blk128 metadata layout, with the additional empty-block and split-KV semantics below.

On SM90, pass sparse_block_size=128 with indices shaped (B, H, S_q // 128, capacity) and KV block IDs measured in units of 128 tokens.

The result is a one-key TupleDict containing o_tensor, a contiguous BF16 tensor of shape (B, H, S_q, 128). This API does not return LSE and has no backward implementation.

V quantization keeps the reciprocal and multiplication in FP32 until the final E4M3 conversion, including the internal BHDS and blocked V layouts. Q retains the Sage BF16 intermediate-rounding recipe; K uses FP32 centering. FP32 rounding near an FP8 midpoint can select an adjacent E4M3 value across implementations.

SM120 per-token Q/K scaling

On SM120 blk128, set k_quant_block_size=1 to use one FP32 scale for every (batch, head, token) of both Q and K, covering all 128 head features. The default k_quant_block_size=16 preserves the existing recipe. This argument controls numerics, not the sparse block size or an autotuning knob.

out = block_sparse_attention_fp8_forward(
q, k, v, q2k_block_index,
sparse_block_size=128,
k_quant_block_size=1,
)["o_tensor"]

K is still centered by its FP32 sequence mean before quantization. Its per-token descale is max(amax(abs(K_centered[token, :])), 1e-3) / 448, and the quotient is rounded to E4M3. Q retains its existing BF16 intermediate rounding, and V keeps one FP32 scale per (head, channel), reduced over batch and the entire key sequence. This does not use MXFP8 block-scale instructions, Q centering, or a Q-mean/delta correction.

The kernel applies each K descale to its own QK accumulator column before the maximum reduction and exponentiation. P uses the unchanged factor of 256 before E4M3 conversion, with an unrounded FP32 denominator. V descales are applied to output channels after PV accumulation. A transposed V storage tile of 128 tokens is only a layout choice, not a V quantization block.

For blocked V storage, the per-token kernel establishes the maximum over the complete 128-key score tile, then converts and consumes P in 32- or 64-key chunks. This interleaves probability conversion with PV while retaining the original ordered FP32 denominator and K32 MMA accumulation order. The chunk width is an internal scheduling choice with a separate compile-cache key; it does not change the quantization granularity or normalize each chunk independently. The coarse-K and unblocked-V paths retain their existing schedule.

Per-token K requires CuTe DSL >= 4.8.0, SM120, sparse_block_size=128, MHA, and D=128. Tails, variable sparse counts, empty rows, and partial KV blocks follow the existing SM120 contract. Other architectures and blk64 reject this option rather than silently falling back to a coarser recipe. Finer scales can reduce quantization error, but are not a guarantee of better model quality on every input; kernel and quantization-inclusive latency should both be measured for the intended workload.

The SM100/SM103 blk128 kernel retains the upstream rounded-probability normalization: P uses a factor of 128, and its denominator is accumulated from the same rounded FP8 probabilities by the widened PV Tensor Core operation. Other FP8 kernels use their existing normalization. Outputs are approximate and are not expected to be bitwise equal across block sizes or architectures.

The architecture-specific FP8 contracts are:

  • SM90 accepts any positive batch and head counts and requires both sequence lengths to be multiples of the selected sparse_block_size (64 or 128). It uses fixed block_sparse_num with full KV blocks of that size. Native blk128 shares each K/V tile between two compute warpgroups. Sparse indices may be non-contiguous (including prefix slices, stepped slices, permuted views, and broadcasts); their strides are consumed directly without an index copy. q2k_block_nums and block_sizes are not supported. It does not use split-KV. V quantization writes sequence-contiguous BHDS storage internally, so the attention kernel directly loads V for Hopper FP8 WGMMA without an in-kernel transpose. Both block sizes accumulate PV directly into FP32 output registers using FP8 WGMMA, accepting its reduced internal accumulation precision instead of promoting each KV tile with a separate FP32 addition.
  • SM100/SM103 blk64 accepts any positive batch and head counts, and requires both sequence lengths to be multiples of 64. It uses fixed block_sparse_num with full 64-token KV blocks; q2k_block_nums and block_sizes are not supported. Split-KV is selected internally, and the public FP8 API does not expose kv_splits or use_clc.
  • SM100/SM103 native blk128 accepts positive arbitrary Q/KV lengths, including physical tails, with MHA and D=128. Fixed counts may be odd; variable q2k_block_nums may be zero, producing zero output. block_sizes accepts rank 1, 2, or 3 and values in [0, 128]; zero masks a whole block, and the physical final-block length always limits the supplied extent. Inactive index capacity is ignored. Sparse metadata views are materialized as needed. The optional kv_splits=1..256 partitions the live KV list using FP32 partials and the existing combine kernel; the default is one. Other FP8 paths require the new keyword to remain at its default. softmax_scale, when supplied, must be a positive finite host scalar. This implementation is limited to compute capabilities 10.0 and 10.3 and requires CuTe DSL >= 4.6.2.
  • SM120 supports blk64 and blk128, accepts any positive batch and head counts, non-aligned Q/KV sequence tails, fixed or per-query-block counts, and block_sizes shaped (N_kv,), (B, N_kv), or (B, H, N_kv). It does not use split-KV.

block_sparse_attention_fp8_forward relies on functionality introduced in CuTe DSL 4.6.1; the SM90 blk128 path requires 4.6.2. Package-supported installations provide CuTe DSL 4.8.0 or newer. Its internal Q/K/V quantizer is also implemented in CuTe DSL and is loaded lazily, so importing cudnn does not eagerly import it.

Backward

Backward is an explicit API rather than a registered PyTorch autograd operation. It recomputes probabilities from the forward output and LSE:

dout = torch.randn_like(o)
grads = block_sparse_attention_backward(
dout,
q,
k,
v,
o,
lse,
q2k_block_index,
block_sparse_num=4,
block_sizes=block_sizes,
sparse_block_size=64,
)
dq, dk, dv = grads

The result keys are dq_tensor, dk_tensor, and dv_tensor. Optional preallocated dq_tensor, dk_tensor, and dv_tensor arguments are supported. The backward implementation builds a bucketed K-to-Q CSR task layout on the GPU. bucket_size_blocks is an optional tuning override; leaving it unset uses the backend default.

For SM100/SM103 blk128, long-query automatic buckets retain at least one SM wave of logical (batch, head, KV block, query group) tasks, or the previous default’s task count when it is smaller. Otherwise, the previous bucket default is used. This shape-based guard does not inspect device-side mask contents; explicit bucket_size_blocks overrides remain authoritative.

For blk64, with contiguous per-row count metadata, at least two KV blocks and at least 1024 (batch, head, KV block) tasks, backward pairs KV neighborhoods to reuse common Q/dO loads and dQ accumulation; smaller problems do not recover the per-call pairing cost and use regular tasks. SM90 pairs KV neighborhoods the same way and runs each matched pair in one blk128-shaped kernel: a CTA walks the pair’s Q blocks once, ordered first-only / common / second-only, and each of its two compute warpgroups computes only the Q blocks of its own KV block, so Q/dO loads and the dQ reduction are shared and Q/dO are read in a single sweep. Pairing metadata is limited to 64 MiB; unsupported metadata layouts use regular tasks, and so do fixed counts on SM100. On SM90, fixed counts pair as well (the wrapper supplies the full per-row count), which measured faster on long video-generation masks; the query bucket stays 384, which measured fastest against 96 through a single group. Duplicate sparse edges retain their multiplicity. Paired tasks retain FP32 dK/dV accumulation even with a single query bucket; on SM90 and SM100, regular tasks store dK/dV directly in the output dtype, without the FP32 workspace and its conversion pass, when all query blocks fall in one query bucket and seqlen_k is a multiple of 64. The initial eight-KV matching is refined within 32-KV windows, with refinement skipped above 1024 bitset words. Metadata construction is included in the wrapper’s work and is rebuilt for each call. Backward interleaves its tasks across attention heads only for queries shorter than 98,304 tokens, where it was measured to help; longer queries keep the head-major task order.

These optimizations keep the public block size at 64 and preserve the existing wrapper-owned workspace model. They do not add an allocation-free graph engine. Use benchmark/bsa/benchmark_sm100_blk64.py for a synthetic complete-API comparison; it records the imported source, input hashes and individual CUDA Event samples. Numerical tests, sanitizer checks and measurements on each target GPU are required before drawing performance conclusions.

Prepared backward metadata (SM90 blk64)

On SM90 with sparse_block_size=64 the backward’s metadata kernels (the bucketed K-to-Q CSR build and, on the paired-KV path, the pair bitsets, matching, refinement and partitioned CSR; about 0.33 ms and ten launches on a 255K-token production mask) depend on the sparse metadata and the sequence lengths only, never on q, k, v or dout. block_sparse_attention_backward_metadata runs them once and returns a BsaBackwardMetadata; a backward that takes it through metadata= launches only the preprocess, the main kernel and the dK/dV/dQ conversions (plus, on the paired-KV path, a copy of the object’s small per-call dK/dV arrival bookkeeping), so a mask that serves several backward passes (every layer of a training step, or the backward of the forward that has just been launched) pays for its metadata once, off the backward’s critical path.

from cudnn.ops import block_sparse_attention_backward, block_sparse_attention_backward_metadata
# Metadata as for the backward; the sequence lengths size the CSR. Runs on the current stream, never synchronizes.
prepared = block_sparse_attention_backward_metadata(
q2k_block_index, block_sparse_num, block_sizes, q2k_block_nums, seqlen_q=q.shape[2], seqlen_k=k.shape[2], sparse_block_size=64
)
for layer in layers: # every backward over this mask skips the metadata kernels
dq, dk, dv = block_sparse_attention_backward(do, q, k, v, o, lse, q2k_block_index, block_sparse_num, block_sizes, q2k_block_nums, sparse_block_size=64, metadata=prepared)

The positional arguments of block_sparse_attention_backward_metadata have the backward’s meaning; bucket_size_blocks is fixed when the object is built (None selects the SM90 default) and a backward may pass the same value or None. The object keeps the metadata tensors alive and remembers their storage address, shape, strides, dtype, device and in-place version counter: a backward that passes it with other metadata tensors, with the same tensors after an in-place change, with another batch, head count or sequence length, or with another bucket_size_blocks raises ValueError instead of walking a stale pattern. Rebuild the object after changing the metadata. The prepare call issues kernels on the current stream of the metadata’s device and does not synchronize the host, so it can run on a side stream (for example right after the forward launch) with the usual event ordering before the backward; the backward records the object’s tensors on its own stream when it consumes them, so the object may be released as soon as the backward call returns, even when it was built on another stream. Both the default backward and the prepared one can be captured into a CUDA graph. metadata= is exclusive with kpage= (a KPageLayout already holds its metadata) and is not accepted by the SM100/SM110 backward or by sparse_block_size=128.

Backward defaults to blk64 on SM90 and blk128 on SM100/SM103. Pass the same explicit sparse_block_size used by forward when selecting the Blackwell blk64 path. SM100/SM103 blk128 backward accepts one-dimensional block_sizes (int32, shape (N_kv,)) and masks each KV block’s padded suffix consistently with forward. Valid tokens occupy the prefix of each physical block; callers zero dout for padded queries and discard their outputs. SM110 blk128 backward still requires block_sizes=None.

KPage (blk128 on SM100/SM103 and SM90, partially occupied blocks)

kpage=True on block_sparse_attention_forward and block_sparse_attention_backward selects a second set of blk128 kernels for sparse patterns in which many selected blocks are only partially valid, as happens when a 3D video latent is flattened and block_sizes marks the rows past each block’s content. The base kernels load and compute every selected block as a full 128-row tile and mask the padding; KPage splits each selected K/V block into 16-row pages and gathers only the valid pages, from any blocks, into the same 128-row MMA tile during the TMA load. A block with v valid rows then costs 16 * ceil(v / 16) rows instead of 128. Query blocks with at most 64 valid rows (see q_block_sizes below) run on an M64 x N256 kernel, the others on M128 x N128. Backward packs K/V pages into owner tiles the same way and walks the union of their Q pages with a per-step page-pair mask, so block_sizes is supported in backward on this path.

On SM90 the same calls run without a separate blk128 KPage kernel. The layout rewrites the metadata as 64-row halves: KV block j becomes halves 2j (min(block_sizes[j], 64) rows) and 2j + 1 (the rest), and Q block i becomes halves 2i and 2i + 1 with the same list. Halves without valid rows are dropped: a KV half leaves every list, and a Q half (from q_block_sizes) gets an empty list. The forward runs the SM90 blk64 KPage kernel on these halves (described below), where each pair of halves is one 128-row query block. The backward runs the SM90 blk64 backward on them, with LSE = +inf on padding query rows so they take no part. The outputs follow the same contract as on SM100/SM103.

from cudnn.ops import block_sparse_attention_forward, block_sparse_attention_backward, kpage_prepare_layout
# Metadata as for the base path: block_sizes[j] is the number of valid keys of KV block j.
o, lse = block_sparse_attention_forward(q, k, v, q2k_block_index, block_sparse_num, block_sizes, kpage=True)
dq, dk, dv = block_sparse_attention_backward(do, q, k, v, o, lse, q2k_block_index, block_sparse_num, block_sizes, kpage=True)
# Reuse one layout across calls; q_block_sizes marks padding on the query side as well.
layout = kpage_prepare_layout(q2k_block_index, block_sparse_num, block_sizes, q_block_sizes=block_sizes)
o, lse = block_sparse_attention_forward(q, k, v, q2k_block_index, block_sparse_num, block_sizes, kpage=layout)
dq, dk, dv = block_sparse_attention_backward(do, q, k, v, o, lse, q2k_block_index, block_sparse_num, block_sizes, kpage=layout)

kpage=True computes the same function as the base path with the same arguments (block_sizes masks keys, every query row is valid) and builds the page layout on the call. kpage_prepare_layout builds a KPageLayout once from the metadata so repeated calls with the same sparse pattern (every layer of a diffusion step, every training step with a fixed packing) skip the metadata kernels. The layout depends on the metadata only, never on q, k, or v. Backward tables are built on the first backward call with that layout.

Prefer a prebuilt layout. The walk tables are sized from counts computed on the device, so building a layout synchronizes the host with the current stream: the host waits until all previously queued GPU work has finished, kernels queued after the call start late, and the call cannot be captured into a CUDA graph. kpage=True builds a layout on every call (the forward and the backward each build their own), so it pays this on every layer. Calls that take a KPageLayout do not synchronize. Build one layout per sparse pattern outside the layer loop and pass it to the forward and backward of every layer that uses the pattern; on SM100/SM103 also pass with_backward=True so the backward tables are not built (with one more synchronization) on the first backward call. Keep kpage=True for one-off calls and experiments.

q_block_sizes (int32, one entry per query block) is specific to this path: query rows at or past q_block_sizes[i] in block i are padding. They are not computed, which is where the M64 x N256 route and part of the speedup on video layouts come from, and their rows of o, lse, and dq are written as zero. Inputs whose query side carries the same per-block padding as the key side pass q_block_sizes=block_sizes.

Contract, in addition to the metadata ranges above:

  • SM100/SM103, sparse_block_size=128, BF16, head_dim=128; MHA, GQA, and MQA with metadata per query head (pack_gqa=None behaves as False, pack_gqa=True is not taken); seqlen_q and seqlen_k independent and arbitrary (rows past a sequence end are padding of its final block, as on the base path); block_sizes of shape (N_kv,). BHSD or BSHD tensors (q/k/v/o/do and the gradients) that are compact in their layout. A batch is run as batch * heads independent heads, so block_sizes is shared across the batch (as on the base SM100 path) and a BSHD batch must be 1. kv_splits, use_clc, and bucket_size_blocks are not taken; a fixed count may be odd.
  • SM90 differs: MHA only, seqlen_q and seqlen_k must be multiples of 64 (a sequence may end 64 rows into its final block), and bwd_kv_groups is not taken.
  • allow_empty_block_nums has the base meaning: with True, a query block with q2k_block_nums == 0 gets O = 0, LSE = -inf, and dQ = 0; with False (default) such a block is a metadata error. Backward accepts empty blocks as on the base path.
  • Masked key rows (at or past block_sizes[j] in block j) of k and v, and padding query rows of q and do when q_block_sizes is given, take no part in the computation and may hold any finite value. Backward writes zeros to the masked rows of dk and dv. Values on valid rows are exact.
  • kpage_prepare_layout(validate=True) (default) checks the metadata on the device and raises ValueError naming the offending entry. The check is read back together with the walk-table sizes, so a layout build synchronizes the host once with or without it; with_backward=True also builds the backward tables (one more synchronization on SM100/SM103), which otherwise happens on the first backward call. seqlen_q / seqlen_k are needed only when a sequence is not a multiple of 128 or when block_sizes is omitted and seqlen_k != seqlen_q (kpage=True takes them from the tensors).
  • A KPageLayout remembers the metadata tensors it was built from; passing it together with other metadata, or with tensors of another shape, raises ValueError.

The forward output of the KPage path can be fed to the KPage backward or to a base backward with block_sizes=None when every block is full; the two paths compute the same function and differ only in the execution layout.

When to use it: the gain comes from padding (block_sizes < 128, and q_block_sizes for the query side) and grows with the padding fraction (1.2-1.3x at 10%, 3.2-3.8x at 59% in benchmark/bsa; on SM90, forward 1.2-1.4x at 10% and 3.6-3.8x at 59%, forward + backward 1.1-1.2x and 2.5-2.7x). Without padding the page gather costs more than the base kernel’s whole-block load, so with block_sizes=None the base path is faster (measured 0.6x at four blocks per row, 0.96x at 128; SM90 forward 0.84-0.91x, backward 0.98-1.12x). GQA/MQA run with per-query-head metadata; the base path’s packed GQA is not available here. BSHD inputs are supported but the strided page gathers make the forward slower than on BHSD (measured 1.3x on a 16-head layout); prefer BHSD. A seqlen_k that is not a multiple of 16 selects forward kernels whose K/V descriptors carry the row extent instead of whole 16-row pages (measured 3-5% slower forward; the backward is unaffected), so pad K/V to a multiple of 16 rows when that matters. A call carries about 0.1 ms of host work (metadata checks, tensor conversion), which only matters for sub-millisecond problems.

KPage forward on blk64 metadata

On SM100/SM103 kpage=True (or a layout built with kpage_prepare_layout(..., sparse_block_size=64)) also takes sparse_block_size=64 metadata, for the forward only. Every 64-row query block is one tile of the M64 x N256 kernel, whose 256-row K/V stage is gathered from sixteen 16-row pages, so a selected 64-row block with v valid rows costs 16 * ceil(v / 16) rows instead of 64.

On SM90 the same call runs a kernel that pairs adjacent query blocks (2p, 2p + 1): one CTA walks the union of their KV block lists as 16-row pages gathered into 128-row K/V tiles, ordered first-only, common, second-only, and its two compute warpgroups (one per query block) compute only their own blocks’ tiles. K/V of the blocks both query blocks select are thus loaded once, on top of the page compaction. The layout holds these pair walks when it is built on an SM90 device.

o, lse = block_sparse_attention_forward(q, k, v, q2k_block_index, block_sparse_num, block_sizes, sparse_block_size=64, kpage=True)
layout = kpage_prepare_layout(q2k_block_index, block_sparse_num, block_sizes, q2k_block_nums, sparse_block_size=64)
o, lse = block_sparse_attention_forward(q, k, v, q2k_block_index, block_sparse_num, block_sizes, q2k_block_nums, sparse_block_size=64, kpage=layout)

The contract is that of the base blk64 forward with these differences:

  • BF16, head_dim=128, MHA; seqlen_q and seqlen_k multiples of 64; a BSHD batch must be 1, as above.
  • Every query row is computed, as on the base blk64 path: there is no q_block_sizes, and the output rows of query padding are not zeroed.
  • Forward only. block_sparse_attention_backward(..., kpage=...) with sparse_block_size=64 raises; run the base backward on the KPage forward output. A layout is bound to the sparse_block_size it was built for.

When to use it: as on blk128, the gain comes from block_sizes < 64 and grows with the padding fraction, and without padding the page gather costs more than the base kernel’s whole-block load. On random top-k patterns (B300, BHSD, layout reused) it measured 1.11-1.36x at 30% padded tokens, 0.91-1.08x at 10%, and 0.85-0.96x without padding. Building the layout costs about 0.3-1 ms per call on those shapes, so reuse a layout from kpage_prepare_layout whenever the metadata repeats; with kpage=True the build is paid on every call. Benchmark your own patterns before switching. On SM90 the gain also grows with the overlap of adjacent query blocks’ KV lists (local or video-like patterns); full blocks with little overlap, such as independent top-k selections, run slower than the base kernel.

Current support

Forward

ArchitectureSparse blockPublic input / kernel dtypeQK / V dimensionsAttention
SM9064 or 128 (explicit)FP16, BF16each of 64, 96, 128MHA, GQA, MQA
SM9064BF16 / FP8 E4M3QK=128, V=128MHA
SM90128 (explicit)BF16 / FP8 E4M3QK=128, V=128MHA
SM100/SM103128FP16, BF16QK=V=64, 96, or 128MHA, GQA, MQA
SM100/SM10364 (explicit)BF16QK=128, V=128MHA
SM100/SM10364BF16 / FP8 E4M3QK=128, V=128MHA
SM100/SM103128 (explicit)BF16 / FP8 E4M3QK=128, V=128MHA
SM100/SM103128, kpage=TrueBF16QK=128, V=128MHA, GQA, MQA
SM100/SM10364, kpage=TrueBF16QK=128, V=128MHA
SM9064, kpage=TrueBF16QK=128, V=128MHA
SM90128, kpage=TrueBF16QK=128, V=128MHA
SM12064FP16, BF16QK=128, V=128MHA, GQA, MQA
SM120128 (explicit)FP16, BF16QK=128, V=128MHA, GQA, MQA
SM12064BF16 / FP8 E4M3QK=128, V=128MHA
SM120128 (explicit)BF16 / FP8 E4M3QK=128, V=128MHA

For FP16/BF16, SM90 blk64 requires S_q to be a multiple of 64; native blk128 supports positive arbitrary Q/KV lengths, including partial final blocks. Its fixed count may be any positive value. The SM100/SM103 blk128 fixed count must be even and at least two (the KPage path accepts any positive count); SM120 and the explicit Blackwell blk64 path accept any positive fixed count. Variable counts use q2k_block_nums. allow_empty_block_nums defaults to False; when it is True, empty rows (q2k_block_nums == 0) produce O = 0 and LSE = -inf. SM90 blk64 selects the empty-row handling as a compile-time specialization, so the default non-empty configuration keeps its branch-free fast path. Its split-KV execution excludes empty rows. Native SM90 blk128 supports empty rows and empty splits with kv_splits=1..256. Sage FP8 uses the architecture-specific contracts above. SM100/SM103 native blk128 supports empty rows and explicit split-KV; its default remains unsplit.

Backward

ArchitectureSparse blockDtypeHead dimensionAttention
SM9064 or 128 (explicit)BF16128MHA
SM100/SM10364BF16128MHA
SM100/SM103128BF1664 or 128MHA
SM100/SM103128, kpage=TrueBF16128MHA, GQA, MQA; block_sizes supported
SM90128, kpage=TrueBF16128MHA; block_sizes supported

On SM90 with sparse_block_size=64 the backward also accepts a prepared BsaBackwardMetadata through metadata= (see Prepared backward metadata); every other path rejects the argument.

Backward is not implemented for SM120. It requires equal QK/V dimensions and, except on the SM100/SM103 KPage path, does not currently support GQA/MQA.

SM90 native blk128 requires CuTe DSL >= 4.6.2. Q/K/V, O/dO and gradient buffers must have a contiguous head dimension, 16-byte aligned base pointers, and row strides divisible by 16 bytes. BHSD/BSHD layouts and aligned noncompact rows are addressed directly. Sparse metadata can be strided and remains on the GPU; backward makes noncontiguous sparse indices and counts contiguous before building CSR, as in blk64. Forward supports block_sizes of rank 1, 2, or 3; backward supports ranks 1 and 2. Both paths predicate sequence tails even without block_sizes. Forward uses Q128 tiles and shares each KV128 load across two compute warpgroups. Backward uses Q64xKV128 CTAs: both compute warpgroups execute all five GEMMs, splitting KV rows for QK/dP/dK/dV and head dimensions for dQ. Each native Q128/KV128 CSR edge streams two Q64 subtiles without expanding the metadata. Backward accumulates gradients with FP32 atomics and is not deterministic. Benchmark the explicit path for your shapes; blk64 remains the default, and native blk128 is not faster for every workload.

Limitations

The current sparse kernels do not implement causal or local masking, dropout, mask_mod, score_mod, paged KV cache, softcap, or variable-length packed sequences. Regular forward/backward inputs must be rank four, use FP16/BF16 as allowed above, and have a contiguous last (head-dimension) axis. Sage FP8 uses the stricter BF16 BHSD contract described above and quantizes to E4M3 internally. Regular forward outputs are contiguous in the requested BHSD or BSHD layout, including split-KV execution; FP8 output is contiguous BF16 BHSD.

Compilation is lazy. The first call for a new static configuration JIT-compiles the relevant kernel; subsequent calls reuse an in-process cache.

The Torch public surface consists of allocating function wrappers under cudnn.ops, also available under cudnn and cudnn.BSA; there is no separate APIBase class or explicit compile() lifecycle for BSA.

Correctness tests and FP32 references are under test/python/block_sparse_attention/cutedsl.

SM100/SM103 FP8 blk128 source

The native producer and its private transfer/epilogue helpers are adapted from NVIDIA-JerryChen/Block-Sparse-Attention, f953680: bsa_fwd_sm100_fp8_fused_pv.py and bsa_fwd_sm100_fp8.py map to the same filenames under python/cudnn/block_sparse_attention/csrc/fwd/sm100_blk128/. The 16-token TMEM row-max helper is incorporated into the private blk128 helper module. The integration uses cuDNN’s existing quantizer, allocating Torch API, device-specific JIT cache, current-stream execution, and split combine. Standalone upstream package entry points and native AOT deployment are not part of this API.

Acknowledgements

We would like to express our gratitude to huangyitong.hyt@alibaba-inc.com and wenting.swt@alibaba-inc.com for providing testing and optimization feedback throughout the deployment process, which has continuously advanced the BSA kernel toward Speed of Light.

Experimental JAX support

cudnn.ops.block_sparse_attention_forward_jax and cudnn.ops.block_sparse_attention_backward_jax provide explicit forward/backward; cudnn.block_sparse_attention.jax_api.block_sparse_attention adds first-order reverse-mode differentiation. All three accept JAX arrays, eagerly or under jax.jit, and require no PyTorch. The cudnn.jax namespace has been removed; use the cudnn.ops names.

The JAX implementation reuses the SM100/SM103 blk128 forward, bucketed CSR, backward preprocess, backward, and gradient conversion kernels. Install jax[cuda13] alongside the frontend. The runtime imports no PyTorch; torch parity tests are separate.

The JAX signatures omit Torch-specific launch options and caller-provided output buffers. XLA owns the outputs and workspaces.

Initial contract

PropertySupported
DeviceAll arrays on one CUDA GPU of compute capability 10.0 or 10.3 (SM100 or SM103); other GPUs may be visible. Under jax.jit (including a zero-argument closure that captures every array), or with no committed JAX array among the inputs, the target is read off the visible GPUs, so a host exposing both SM100 and SM103 GPUs is refused there (eager calls on committed arrays are not). Single process
DataBF16 Q/K/V/O/dO/dQ/dK/dV; FP32 LSE and accumulation
DimensionsMHA with equal head counts, D=64 or 128; positive sequence lengths divisible by 128
LayoutCompact BHSD or BSHD, independently specialized
Sparsity128-token blocks, int32 indices [B,H,Sq/128,C], no block_sizes
CountsFixed even block_sparse_num in [2,C], or runtime int32 q2k_block_nums[B,H,Sq/128]
DifferentiationFirst-order reverse-mode Q/K/V gradients through block_sparse_attention

Variable counts must be in [1,C], or [0,C] with allow_empty_block_nums=True. Only each row’s active prefix is read. Active indices must be unique and in [0,Sk/128). An empty row returns zero output, negative-infinity LSE, and zero query gradient. Unselected K/V receive zero contributions. Metadata values are the caller’s responsibility: the runtime validates shapes, dtypes, and static options, without copying metadata to the host or synchronizing to inspect it. Invalid values are outside the contract.

SM103/SM110/SM120, blk64, FP16/FP8, GQA, different V dimensions, partial blocks, variable sequence lengths, causal masks, dropout, sharding, vmap, JVP, and higher derivatives are unsupported. KDA is outside this change. Static options (including scale and fixed count) specialize compilation; sparse arrays remain runtime operands. Do not mutate saved forward inputs or metadata before backward.

Usage

import jax
import jax.numpy as jnp
from functools import partial
from cudnn.block_sparse_attention.jax_api import block_sparse_attention as attention
from cudnn.ops import (
block_sparse_attention_forward_jax as forward,
block_sparse_attention_backward_jax as backward,
)
q = jnp.ones((1, 2, 256, 64), dtype=jnp.bfloat16)
k = jnp.ones((1, 2, 512, 64), dtype=jnp.bfloat16)
v = jnp.ones_like(k)
indices = jnp.broadcast_to(jnp.array([0, 2], jnp.int32), (1, 2, 2, 2))
o, lse = forward(q, k, v, indices, 2) # eager; asynchronous GPU execution
run = jax.jit(partial(forward, block_sparse_num=2))
result = run(q, k, v, indices)
o, lse = result["o_tensor"], result["lse_tensor"]
dq, dk, dv = backward(jnp.ones_like(o), q, k, v, o, lse, indices, 2)
def loss(q, k, v, indices):
return attention(q, k, v, indices, 2).astype(jnp.float32).sum()
grad = jax.jit(jax.grad(loss, argnums=(0, 1, 2)))
dq, dk, dv = grad(q, k, v, indices)
dq.block_until_ready() # needed for host timing, not between GPU operations

Explicit forward returns a JAX pytree with o_tensor, lse_tensor; backward returns dq_tensor, dk_tensor, dv_tensor. Both support tuple unpacking and dictionary access. Explicit helpers do not register autodiff rules; use attention for jax.grad. LSE and metadata are not differentiable public outputs. Backward must receive the exact matching forward Q/K/V/O/LSE, sparse pattern, scale, layout, and empty-row option.

Execution and ownership

Every launch receives XLA’s CUDA stream. Forward is one custom call; backward composes CSR construction, preprocessing, attention backward, and gradient conversion inside one custom call. XLA owns all outputs and workspaces. CSR counts and multi-bucket dK/dV accumulators are initialized for each invocation; preprocessing clears dQ accumulation. No mutable global scratch or public input buffer is donated. Atomic accumulation does not promise bitwise determinism.

TensorSpec.mode presents compact BHSD/BSHD memory in the order expected by each kernel. It does not rewrite DLPack capsules or insert a Python transpose. XLA may still insert layout conversions when surrounding operations use incompatible physical layouts; this is not a universal zero-copy guarantee.

Two cached call builders retain static configuration and kernel objects, never input arrays or pointers. XLA handles compilation and execution; there is no separate plan lifecycle. One custom_vjp registration connects forward and backward. bucket_size_blocks optionally controls backward query buckets; the default reuses the torch path’s heuristic. Multi-device placement and cache portability across architectures require further qualification.

The JAX tests live in test/python/block_sparse_attention/jax and are collected by the OSS suite. Torch-free import and gradient checks run in fresh subprocesses. To run the suite in a torch-free container, pass --confcutdir=test/python/block_sparse_attention/jax so pytest does not load the torch-based parent conftest.