> This page is for version Latest (default).
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/cudnn/latest/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/cudnn/llms.txt.

# Block Sparse Attention (BSA)

**This is an experimental API and subject to change.**

## Overview

Block Sparse Attention computes non-causal scaled dot-product attention over a
block-level sparse pattern. For query token `i`, let `m = floor(i / block_size)`
be its query-block index and let `K_m` be the union of the key/value blocks
listed for that query block. The operation is

$$
O_i = \text{softmax}_{j \in K_m}
\left(\frac{Q_i K_j^T}{\sqrt{D}}\right)V_j.
$$

The softmax is normalized once over all valid tokens in the selected blocks.
`block_sizes` can shorten individual key/value blocks so that padded tokens do
not participate in the softmax.

Unlike [NSA Selection Attention](/cudnn/operations/sparse_attention/nsa#1-selection-attention), BSA supplies one
list of key/value blocks per **query block**. NSA Selection supplies routing
metadata at query-token granularity and is one component of the larger NSA
pipeline.

BSA is implemented with Python CuTe DSL/JIT kernels.

## Installation

Except on Windows ARM64, the CuTe DSL runtime is a required dependency, so the base install is enough:

```bash
pip install nvidia-cudnn-frontend
```

On Windows ARM64, the CuTe DSL and TVM-FFI runtime dependencies are excluded from the base installation, so the base install does not enable BSA there.

The package-wide `nvidia-cutlass-dsl[cu13]>=4.8.0` dependency floor applies to
BSA. Sage FP8 relies on functionality introduced in CuTe DSL 4.6.1, which the
package floor already satisfies. A defensive runtime check remains for
environments that force an older DSL; the KPage path has the same check at
the package floor and was verified on CuTe DSL 4.6.2, 4.7.0, and 4.7.1.

## Forward

```python
import torch
from cudnn.ops import (
    block_sparse_attention_forward,
    block_sparse_attention_backward,
    block_sparse_attention_fp8_forward,
)

q = torch.randn(1, 8, 1024, 128, device="cuda", dtype=torch.bfloat16)
k = torch.randn(1, 8, 2048, 128, device="cuda", dtype=torch.bfloat16)
v = torch.randn_like(k)

# For the SM90 blk64 path, this example has 16 Q blocks and 32 KV blocks.
# Every Q block attends to the first four KV blocks.
q2k_block_index = torch.arange(4, device="cuda", dtype=torch.int32)
q2k_block_index = q2k_block_index.view(1, 1, 1, 4).expand(1, 8, 16, 4).contiguous()
block_sizes = torch.full((32,), 64, device="cuda", dtype=torch.int32)

result = block_sparse_attention_forward(
    q,
    k,
    v,
    q2k_block_index,
    block_sparse_num=4,
    block_sizes=block_sizes,
    sparse_block_size=64,
)
o, lse = result
# The same values are available as result["o_tensor"] and result["lse_tensor"].
```

The default layout is `BHSD`. Pass `layout="bshd"` for tensors in `BSHD`
layout. The output follows the input layout, while `lse` is always the FP32
natural-log log-sum-exp with shape `(B, H_q, S_q)`.

For explicit Blackwell blk64 calls, TMA-aligned BSHD inputs are read through
strided views. Both unsplit outputs and split-KV combined outputs are written
directly in the requested layout. Misaligned inputs retain their staging fallback.

### Sparse metadata

| Argument | Shape | Dtype | Meaning |
| --- | --- | --- | --- |
| `q2k_block_index` | `(B, H_q, N_q, K_max)` | `int32` | KV-block IDs; only the valid prefix of each last dimension is read |
| `block_sparse_num` | scalar | Python `int` | Fixed valid-prefix length for every query block |
| `q2k_block_nums` | `(B, H_q, N_q)` | `int32` | Optional per-query-block valid-prefix lengths; overrides `block_sparse_num` |
| `block_sizes` | `(N_kv,)` or backend-supported batched form | `int32` | Number of valid tokens in each KV block |

Here `N_q = ceil(S_q / sparse_block_size)` and
`N_kv = ceil(S_kv / sparse_block_size)`. Except for packed GQA described below,
metadata is per query head. The metadata tensors must reside on the same CUDA
device as `q`, `k`, and `v`. The value ranges below are a hard caller contract.

#### Value ranges (caller contract)

Let `K_max = q2k_block_index.shape[-1]`. Only the active prefix of each
`q2k_block_index` row — the first `q2k_block_nums[b, h, m]` entries with
variable counts, or the first `block_sparse_num` entries with a fixed count —
is consumed. Values in the inactive suffix are ignored.

- Within each row, active `q2k_block_index` values must be unique integers in
  `[0, N_kv)`.
- For forward with variable counts, every `q2k_block_nums` value must be in
  `[0, K_max]` when `allow_empty_block_nums=True`, and in `[1, K_max]`
  otherwise. Backward variable counts may be in `[0, K_max]`.
- With fixed counts, `block_sparse_num` must be in `[1, K_max]`. The
  FP16/BF16 SM100/SM103 blk128 path additionally requires an even value, i.e. an even
  `block_sparse_num` in `[2, K_max]`.
- The `block_sizes` entry for every physical KV block referenced by an active
  `q2k_block_index` value must be in `[1, sparse_block_size]`. Entries for
  unreferenced physical KV blocks are ignored. A zero-sized referenced block is
  not supported by regular FP16/BF16 attention; use `q2k_block_nums` (or the
  sparse index prefix) to drop the block instead. FP8 SM100/SM103 blk128 also
  accepts zero-sized blocks, as specified in its architecture contract below.

Tensor value ranges and per-row uniqueness are not validated at runtime.
Violating the contract is unsupported and may produce invalid results or
invalid device memory accesses.

On the SM100/SM103 blk128 path, `pack_gqa=None` automatically packs GQA when
the GQA ratio `r = H_q / H_kv` divides 128. Packed metadata has shape
`(B, H_kv, ceil(S_q * r / 128), K_max)`. Pass `pack_gqa=False` to use the
unpacked `(B, H_q, ceil(S_q / 128), K_max)` contract on every architecture.
Explicit `pack_gqa=True` requires `r` to divide 128.

When `block_sizes=None`, each referenced physical KV block is treated as full.
Provide `block_sizes` whenever a referenced final block is only partially
valid.

`sparse_block_size=None` chooses blk64 on SM90/SM120 and blk128 on
SM100/SM103. On SM90 or SM120, passing `sparse_block_size=128` selects a native
KV128 kernel that consumes blk128 metadata directly; it does not
expand the metadata or invoke the blk64 kernel. Passing `sparse_block_size=64`
explicitly selects the SM100/SM103 blk64 CuTe DSL path, whose shape support is
narrower. `kv_splits` is available on SM90 and the explicit Blackwell blk64
path; `use_clc` applies only to the explicit Blackwell blk64 path.

For SM120 blk128 with fixed `block_sparse_num`, full physical KV blocks
(`block_sizes=None`), and a KV sequence length divisible by 128, the dispatcher
uses an FA4-style native specialization with a dedicated K/V load warp,
register-resident Q, and a four-fold-unrolled sparse loop. Most CTAs process
128 Q rows; an underfilled final scheduling wave may use 64 Q rows per CTA
while still loading full KV128 blocks and using the parent Q block's original
metadata. This is Q-work scheduling, not KV128-to-KV64 lowering. Variable
per-row block counts, explicit block sizes, and partial final KV blocks use
the general native blk128 kernel. The FA4-style specialization requires
`nvidia-cutlass-dsl >= 4.7.0`; an older public DSL version is rejected with
a version-specific error before the specialized kernel is imported. The
package-wide downstream dependency floor is unchanged.

`kv_splits=2..256` computes FP32 partial outputs and combines them, with
workspace growing linearly in the split count. SM90 accepts an explicit integer
split count or `kv_splits="auto"` (the default stays 1). On SM90, once a head's K/V
exceeds L2 the unsplit kernel turns DRAM-bound, and a split grid orders every Q tile of one KV
range before the next. The automatic policy splits only when a head's K/V clearly exceeds L2,
with a threshold that rises as heads fall (above 1x L2 with 16+ heads, 1.5x with 4-15, 4x with
fewer), and then keeps about L2 / 2 of K/V per split while each split still holds at least 32
blocks of a row. The BF16/FP16 MHA policy applies at one head per wave (`ceil(S_q / 64)` at least
`3 * SMs`), at most 25% density and fixed counts, and declines (one split) otherwise. It was
measured on H100 with independent random indices and video-generation top-k masks (for example 128 MiB K/V per head with 4-8
heads: 4-8 splits, +27-67%; 256 MiB: 8-16 splits, up to +86%); 32 MiB per head never gained.
A caller-provided output also resolves to one split, and so does `"auto"` on SM90 blk128, where the
policy has not been measured. The SM100/SM103 blk64 path also accepts `kv_splits="auto"`; CLC is
compatible with split execution. Pass `use_clc=True` to use persistent CLC
scheduling with `kv_splits>1`, or `use_clc=False` to use one tile per CTA. The
default `use_clc=None` keeps CLC disabled for split execution because the
automatic scheduler policy has not been tuned for that combination.

The BF16 automatic policy is tuned assuming approximately uniform random KV
indices with little overlap between the neighborhoods of different Q blocks.
It uses host-visible tensor metadata and device properties, without reading
indices or counts back to the host. Highly overlapping, shared or local-window
indices can favor a different split count even at the same shape and density;
benchmark explicit `kv_splits` values for those masks. This assumption affects
performance only, not attention semantics or supported mask patterns.

For fewer than 512 total Q tiles (`B * H * ceil(S_q / 64)`), the existing policy
uses the fixed active block count, or index capacity for variable counts:
256--449 blocks select two splits, 450--899 select four, and 900 or more select
eight. Larger Q grids keep one split except in the measured long-Q region:
BF16 MHA with Q/K/V head dimensions 128, `S_q >= 262144`, a fixed count of at
least 512 active KV blocks, and density at most 1/8 (active blocks divided by
`ceil(S_k / 64)`). In that region, K+V bytes **per batch and KV head** above twice
the device L2 size select eight splits for at least 900 active blocks and four
or more heads; a single head requires at least 1024 active blocks. Otherwise,
four or more heads select four splits when per-head K+V bytes exceed L2.
All other cases, including an unavailable L2 size, keep one split. Variable
per-row counts also keep one split on larger Q grids because index capacity
does not describe actual row work. The Sage FP8 policy is separate.

Automatic selection
falls back to a smaller split count when the estimated live workspace does not
fit the available CUDA allocator budget; an explicit split count that exceeds
that budget raises `RuntimeError`.

## Sage FP8 forward

Sage FP8 is forward-only and supports blk64 on SM90, SM100/SM103, and SM120,
plus native blk128 on SM90, SM100/SM103, and SM120. Its public wrapper accepts contiguous
BF16 Q, K, and V tensors in `BHSD` layout and performs FP8 quantization internally:

```python
fp8_result = block_sparse_attention_fp8_forward(
    q,
    k,
    v,
    q2k_block_index,
    block_sparse_num=4,
    sparse_block_size=64,
)
o_fp8 = fp8_result["o_tensor"]
```

Q, K, and V must be BF16 MHA tensors in contiguous BHSD layout, have matching
batch and head counts, and use `D=128`. The wrapper accepts the same sparse
index metadata used by the regular forward API for the selected block size.
It lazily loads the quantizer, creates the E4M3 tensors and FP32 scales needed
by the kernel, and does not expose those implementation details as public
inputs or outputs.

`sparse_block_size` defaults to 64. On SM120, pass `sparse_block_size=128` with
`q2k_block_index` shaped `(B, H, ceil(S_q / 128), K_max)` and, when supplied,
`q2k_block_nums` shaped `(B, H, ceil(S_q / 128))`. Active indices address
physical 128-token KV blocks, so they must be in `[0, ceil(S_kv / 128))`.
For `block_sizes`, use `N_kv = ceil(S_kv / 128)` and valid-token counts in
`[1, 128]` for referenced blocks. The native kernel consumes this metadata
directly. SM100/SM103 also support this blk128 metadata layout, with the
additional empty-block and split-KV semantics below.

On SM90, pass `sparse_block_size=128` with indices shaped
`(B, H, S_q // 128, capacity)` and KV block IDs measured in units of 128 tokens.

The result is a one-key `TupleDict` containing `o_tensor`, a contiguous BF16
tensor of shape `(B, H, S_q, 128)`. This API does not return LSE and has no
backward implementation.

V quantization keeps the reciprocal and multiplication in FP32 until the
final E4M3 conversion, including the internal BHDS and blocked V layouts.
Q retains the Sage BF16 intermediate-rounding recipe; K uses FP32 centering.
FP32 rounding near an FP8 midpoint can select an adjacent E4M3 value across
implementations.

### SM120 per-token Q/K scaling

On SM120 blk128, set `k_quant_block_size=1` to use one FP32 scale for every
`(batch, head, token)` of **both Q and K**, covering all 128 head features.
The default `k_quant_block_size=16` preserves the existing recipe. This
argument controls numerics, not the sparse block size or an autotuning knob.

```python
out = block_sparse_attention_fp8_forward(
    q, k, v, q2k_block_index,
    sparse_block_size=128,
    k_quant_block_size=1,
)["o_tensor"]
```

K is still centered by its FP32 sequence mean before quantization. Its
per-token descale is `max(amax(abs(K_centered[token, :])), 1e-3) / 448`,
and the quotient is rounded to E4M3. Q retains its existing BF16 intermediate
rounding, and V keeps one FP32 scale per `(head, channel)`, reduced over
**batch and the entire key sequence**. This does not use MXFP8 block-scale
instructions, Q centering, or a Q-mean/delta correction.

The kernel applies each K descale to its own QK accumulator column before
the maximum reduction and exponentiation. P uses the unchanged factor of
256 before E4M3 conversion, with an unrounded FP32 denominator. V descales
are applied to output channels after PV accumulation. A transposed V storage
tile of 128 tokens is only a layout choice, **not** a V quantization block.

For blocked V storage, the per-token kernel establishes the maximum over the
complete 128-key score tile, then converts and consumes P in 32- or 64-key
chunks. This interleaves probability conversion with PV while retaining the
original ordered FP32 denominator and K32 MMA accumulation order. The chunk
width is an internal scheduling choice with a separate compile-cache key;
it does not change the quantization granularity or normalize each chunk
independently. The coarse-K and unblocked-V paths retain their existing schedule.

Per-token K requires CuTe DSL >= 4.8.0, SM120, `sparse_block_size=128`, MHA,
and D=128. Tails, variable sparse counts, empty rows, and partial KV blocks
follow the existing SM120 contract. Other architectures and blk64 reject
this option rather than silently falling back to a coarser recipe. Finer
scales can reduce quantization error, but are not a guarantee of better
model quality on every input; kernel and quantization-inclusive latency
should both be measured for the intended workload.

The SM100/SM103 blk128 kernel retains the upstream rounded-probability
normalization: P uses a factor of 128, and its denominator is accumulated from
the same rounded FP8 probabilities by the widened PV Tensor Core operation.
Other FP8 kernels use their existing normalization. Outputs are approximate
and are not expected to be bitwise equal across block sizes or architectures.

The architecture-specific FP8 contracts are:

- SM90 accepts any positive batch and head counts and requires both sequence
  lengths to be multiples of the selected `sparse_block_size` (64 or 128).
  It uses fixed `block_sparse_num` with full KV blocks of that size.
  Native blk128 shares each K/V tile between two compute warpgroups.
  Sparse indices may be non-contiguous (including prefix
  slices, stepped slices, permuted views, and broadcasts); their strides are
  consumed directly without an index copy. `q2k_block_nums` and `block_sizes`
  are not supported. It does not use split-KV. V quantization
  writes sequence-contiguous BHDS storage internally, so the attention kernel
  directly loads V for Hopper FP8 WGMMA without an in-kernel transpose.
  Both block sizes accumulate PV directly into FP32 output registers using
  FP8 WGMMA, accepting its reduced internal accumulation precision instead
  of promoting each KV tile with a separate FP32 addition.
- SM100/SM103 blk64 accepts any positive batch and head counts,
  and requires both sequence lengths to be multiples of 64. It uses fixed
  `block_sparse_num` with full 64-token KV blocks; `q2k_block_nums` and
  `block_sizes` are not supported. Split-KV is selected internally, and the
  public FP8 API does not expose `kv_splits` or `use_clc`.
- SM100/SM103 native blk128 accepts positive arbitrary Q/KV lengths, including
  physical tails, with MHA and D=128. Fixed counts may be odd; variable
  `q2k_block_nums` may be zero, producing zero output. `block_sizes` accepts
  rank 1, 2, or 3 and values in `[0, 128]`; zero masks a whole block, and the
  physical final-block length always limits the supplied extent. Inactive
  index capacity is ignored. Sparse metadata views are materialized as needed.
  The optional `kv_splits=1..256` partitions the live KV list using FP32 partials
  and the existing combine kernel; the default is one. Other FP8 paths require
  the new keyword to remain at its default. `softmax_scale`, when supplied,
  must be a positive finite host scalar. This implementation is limited to
  compute capabilities 10.0 and 10.3 and requires CuTe DSL >= 4.6.2.
- SM120 supports blk64 and blk128, accepts any positive batch and head counts,
  non-aligned Q/KV sequence tails, fixed or per-query-block counts, and
  `block_sizes` shaped `(N_kv,)`, `(B, N_kv)`, or `(B, H, N_kv)`. It does not
  use split-KV.

`block_sparse_attention_fp8_forward` relies on functionality introduced in
CuTe DSL 4.6.1; the SM90 blk128 path requires 4.6.2. Package-supported
installations provide CuTe DSL 4.8.0 or newer.
Its internal Q/K/V quantizer is also implemented in CuTe DSL and is
loaded lazily, so importing `cudnn` does not eagerly import it.

## Backward

Backward is an explicit API rather than a registered PyTorch autograd
operation. It recomputes probabilities from the forward output and LSE:

```python
dout = torch.randn_like(o)
grads = block_sparse_attention_backward(
    dout,
    q,
    k,
    v,
    o,
    lse,
    q2k_block_index,
    block_sparse_num=4,
    block_sizes=block_sizes,
    sparse_block_size=64,
)
dq, dk, dv = grads
```

The result keys are `dq_tensor`, `dk_tensor`, and `dv_tensor`. Optional
preallocated `dq_tensor`, `dk_tensor`, and `dv_tensor` arguments are supported.
The backward implementation builds a bucketed K-to-Q CSR task layout on the
GPU. `bucket_size_blocks` is an optional tuning override; leaving it unset uses
the backend default.

For SM100/SM103 blk128, long-query automatic buckets retain at least one SM
wave of logical `(batch, head, KV block, query group)` tasks, or the previous
default's task count when it is smaller. Otherwise, the previous bucket
default is used. This shape-based guard does not inspect device-side mask
contents; explicit `bucket_size_blocks` overrides remain authoritative.

For blk64, with contiguous per-row count metadata, at least two KV blocks and at least
1024 (batch, head, KV block) tasks, backward pairs KV neighborhoods to reuse
common Q/dO loads and dQ accumulation; smaller problems do not recover the
per-call pairing cost and use regular tasks.
SM90 pairs KV neighborhoods the same way and runs each matched pair in one
blk128-shaped kernel: a CTA walks the pair's Q blocks once, ordered
first-only / common / second-only, and each of its two compute warpgroups
computes only the Q blocks of its own KV block, so Q/dO loads and the dQ
reduction are shared and Q/dO are read in a single sweep.
Pairing metadata is limited to 64 MiB; unsupported metadata layouts use regular tasks, and so do
fixed counts on SM100. On SM90, fixed counts pair as well (the wrapper supplies the full per-row
count), which measured faster on long video-generation masks; the query bucket stays 384, which
measured fastest against 96 through a single group. Duplicate sparse edges retain their multiplicity.
Paired tasks retain FP32 dK/dV accumulation even with a single query bucket;
on SM90 and SM100, regular tasks store dK/dV directly in the output dtype, without the FP32
workspace and its conversion pass, when all query blocks fall in one query bucket and
`seqlen_k` is a multiple of 64.
The initial eight-KV matching is refined within 32-KV windows, with refinement
skipped above 1024 bitset words. Metadata construction is included in the
wrapper's work and is rebuilt for each call.
Backward interleaves its tasks across attention heads only for queries shorter
than 98,304 tokens, where it was measured to help; longer queries keep the
head-major task order.

These optimizations keep the public block size at 64 and preserve the existing
wrapper-owned workspace model. They do not add an allocation-free graph engine.
Use `benchmark/bsa/benchmark_sm100_blk64.py` for a synthetic complete-API
comparison; it records the imported source, input hashes and individual CUDA
Event samples. Numerical tests, sanitizer checks and measurements on each
target GPU are required before drawing performance conclusions.

### Prepared backward metadata (SM90 blk64)

On SM90 with `sparse_block_size=64` the backward's metadata kernels (the
bucketed K-to-Q CSR build and, on the paired-KV path, the pair bitsets,
matching, refinement and partitioned CSR; about 0.33 ms and ten launches on
a 255K-token production mask) depend on the sparse metadata and the sequence
lengths only, never on `q`, `k`, `v` or `dout`. `block_sparse_attention_backward_metadata`
runs them once and returns a `BsaBackwardMetadata`; a backward that takes it
through `metadata=` launches only the preprocess, the main kernel and the
dK/dV/dQ conversions (plus, on the paired-KV path, a copy of the object's small
per-call dK/dV arrival bookkeeping), so a mask that serves several backward passes (every
layer of a training step, or the backward of the forward that has just been
launched) pays for its metadata once, off the backward's critical path.

```python
from cudnn.ops import block_sparse_attention_backward, block_sparse_attention_backward_metadata

# Metadata as for the backward; the sequence lengths size the CSR. Runs on the current stream, never synchronizes.
prepared = block_sparse_attention_backward_metadata(
    q2k_block_index, block_sparse_num, block_sizes, q2k_block_nums, seqlen_q=q.shape[2], seqlen_k=k.shape[2], sparse_block_size=64
)
for layer in layers:  # every backward over this mask skips the metadata kernels
    dq, dk, dv = block_sparse_attention_backward(do, q, k, v, o, lse, q2k_block_index, block_sparse_num, block_sizes, q2k_block_nums, sparse_block_size=64, metadata=prepared)
```

The positional arguments of `block_sparse_attention_backward_metadata` have the
backward's meaning; `bucket_size_blocks` is fixed when the object is built
(`None` selects the SM90 default) and a backward may pass the same value or
`None`. The object keeps the metadata tensors alive and remembers their
storage address, shape, strides, dtype, device and in-place version counter:
a backward that passes it with other metadata tensors, with the same tensors
after an in-place change, with another batch, head count or sequence length,
or with another `bucket_size_blocks` raises `ValueError` instead of walking a
stale pattern. Rebuild the object after changing the metadata. The prepare
call issues kernels on the current stream of the metadata's device and does
not synchronize the host, so it can run on a side stream (for example right
after the forward launch) with the usual event ordering before the backward;
the backward records the object's tensors on its own stream when it consumes
them, so the object may be released as soon as the backward call returns,
even when it was built on another stream. Both the default backward and the
prepared one can be captured into a CUDA graph. `metadata=` is exclusive with
`kpage=` (a `KPageLayout` already holds its metadata) and is not accepted by
the SM100/SM110 backward or by `sparse_block_size=128`.

Backward defaults to blk64 on SM90 and blk128 on SM100/SM103. Pass the same
explicit `sparse_block_size` used by forward when selecting the Blackwell blk64
path. SM100/SM103 blk128 backward accepts one-dimensional `block_sizes`
(`int32`, shape `(N_kv,)`) and masks each KV block's padded suffix consistently
with forward. Valid tokens occupy the prefix of each physical block; callers
zero `dout` for padded queries and discard their outputs. SM110 blk128 backward
still requires `block_sizes=None`.

## KPage (blk128 on SM100/SM103 and SM90, partially occupied blocks)

`kpage=True` on `block_sparse_attention_forward` and
`block_sparse_attention_backward` selects a second set of blk128 kernels for
sparse patterns in which many selected blocks are only partially valid, as
happens when a 3D video latent is flattened and `block_sizes` marks the rows
past each block's content. The base kernels load and compute every selected
block as a full 128-row tile and mask the padding; KPage splits each selected
K/V block into 16-row pages and gathers only the valid pages, from any blocks,
into the same 128-row MMA tile during the TMA load. A block with `v` valid
rows then costs `16 * ceil(v / 16)` rows instead of 128. Query blocks with at
most 64 valid rows (see `q_block_sizes` below) run on an M64 x N256 kernel,
the others on M128 x N128. Backward packs K/V pages into owner tiles the same
way and walks the union of their Q pages with a per-step page-pair mask, so
`block_sizes` is supported in backward on this path.

On SM90 the same calls run without a separate blk128 KPage kernel. The layout
rewrites the metadata as 64-row halves: KV block `j` becomes halves `2j`
(`min(block_sizes[j], 64)` rows) and `2j + 1` (the rest), and Q block `i`
becomes halves `2i` and `2i + 1` with the same list. Halves without valid
rows are dropped: a KV half leaves every list, and a Q half (from
`q_block_sizes`) gets an empty list. The forward runs the SM90 blk64 KPage
kernel on these halves (described below), where each pair of halves is one
128-row query block. The backward runs the SM90 blk64 backward on them, with
`LSE = +inf` on padding query rows so they take no part. The outputs follow
the same contract as on SM100/SM103.

```python
from cudnn.ops import block_sparse_attention_forward, block_sparse_attention_backward, kpage_prepare_layout

# Metadata as for the base path: block_sizes[j] is the number of valid keys of KV block j.
o, lse = block_sparse_attention_forward(q, k, v, q2k_block_index, block_sparse_num, block_sizes, kpage=True)
dq, dk, dv = block_sparse_attention_backward(do, q, k, v, o, lse, q2k_block_index, block_sparse_num, block_sizes, kpage=True)

# Reuse one layout across calls; q_block_sizes marks padding on the query side as well.
layout = kpage_prepare_layout(q2k_block_index, block_sparse_num, block_sizes, q_block_sizes=block_sizes)
o, lse = block_sparse_attention_forward(q, k, v, q2k_block_index, block_sparse_num, block_sizes, kpage=layout)
dq, dk, dv = block_sparse_attention_backward(do, q, k, v, o, lse, q2k_block_index, block_sparse_num, block_sizes, kpage=layout)
```

`kpage=True` computes the same function as the base path with the same
arguments (`block_sizes` masks keys, every query row is valid) and builds
the page layout on the call. `kpage_prepare_layout` builds a `KPageLayout`
once from the metadata so repeated calls with the same sparse pattern (every
layer of a diffusion step, every training step with a fixed packing) skip the
metadata kernels. The layout depends on the metadata only, never on `q`, `k`,
or `v`. Backward tables are built on the first backward call with that layout.

Prefer a prebuilt layout. The walk tables are sized from counts computed on
the device, so building a layout synchronizes the host with the current
stream: the host waits until all previously queued GPU work has finished,
kernels queued after the call start late, and the call cannot be captured into
a CUDA graph. `kpage=True` builds a layout on every call (the forward and the
backward each build their own), so it pays this on every layer. Calls that
take a `KPageLayout` do not synchronize. Build one layout per sparse pattern
outside the layer loop and pass it to the forward and backward of every layer
that uses the pattern; on SM100/SM103 also pass `with_backward=True` so the
backward tables are not built (with one more synchronization) on the first
backward call. Keep `kpage=True` for one-off calls and experiments.

`q_block_sizes` (int32, one entry per query block) is specific to this path:
query rows at or past `q_block_sizes[i]` in block `i` are padding. They are
not computed, which is where the M64 x N256 route and part of the speedup on
video layouts come from, and their rows of `o`, `lse`, and `dq` are written
as zero. Inputs whose query side carries the same per-block padding as the
key side pass `q_block_sizes=block_sizes`.

Contract, in addition to the metadata ranges above:

- SM100/SM103, `sparse_block_size=128`, BF16, `head_dim=128`; MHA, GQA, and
  MQA with metadata per query head (`pack_gqa=None` behaves as `False`,
  `pack_gqa=True` is not taken); `seqlen_q` and `seqlen_k` independent and
  arbitrary (rows past a sequence end are padding of its final block, as on
  the base path); `block_sizes` of shape `(N_kv,)`.
  BHSD or BSHD tensors (q/k/v/o/do and the gradients) that are compact in
  their layout. A batch is run as `batch * heads` independent heads, so
  `block_sizes` is shared across the batch (as on the base SM100 path) and a
  BSHD batch must be 1. `kv_splits`, `use_clc`, and `bucket_size_blocks`
  are not taken; a fixed count may be odd.
- SM90 differs: MHA only, `seqlen_q` and `seqlen_k` must be multiples of 64
  (a sequence may end 64 rows into its final block), and `bwd_kv_groups` is
  not taken.
- `allow_empty_block_nums` has the base meaning: with `True`, a query block
  with `q2k_block_nums == 0` gets `O = 0`, `LSE = -inf`, and `dQ = 0`; with
  `False` (default) such a block is a metadata error. Backward accepts empty
  blocks as on the base path.
- Masked key rows (at or past `block_sizes[j]` in block `j`) of `k` and `v`,
  and padding query rows of `q` and `do` when `q_block_sizes` is given, take
  no part in the computation and may hold any finite value. Backward writes
  zeros to the masked rows of `dk` and `dv`. Values on valid rows are exact.
- `kpage_prepare_layout(validate=True)` (default) checks the metadata on the
  device and raises `ValueError` naming the offending entry. The check is read
  back together with the walk-table sizes, so a layout build synchronizes the
  host once with or without it; `with_backward=True` also builds the backward
  tables (one more synchronization on SM100/SM103), which otherwise happens on
  the first backward call. `seqlen_q` / `seqlen_k` are needed only when a
  sequence is not a multiple of 128 or when `block_sizes` is omitted and
  `seqlen_k != seqlen_q` (`kpage=True` takes them from the tensors).
- A `KPageLayout` remembers the metadata tensors it was built from; passing it
  together with other metadata, or with tensors of another shape, raises
  `ValueError`.

The forward output of the KPage path can be fed to the KPage backward or to a
base backward with `block_sizes=None` when every block is full; the two paths
compute the same function and differ only in the execution layout.

When to use it: the gain comes from padding (`block_sizes < 128`, and
`q_block_sizes` for the query side) and grows with the padding fraction
(1.2-1.3x at 10%, 3.2-3.8x at 59% in `benchmark/bsa`; on SM90, forward
1.2-1.4x at 10% and 3.6-3.8x at 59%, forward + backward 1.1-1.2x and
2.5-2.7x). Without
padding the page gather costs more than the base kernel's whole-block load,
so with `block_sizes=None` the base path is faster (measured 0.6x at four
blocks per row, 0.96x at 128; SM90 forward 0.84-0.91x, backward
0.98-1.12x). GQA/MQA run with per-query-head metadata; the
base path's packed GQA is not available here. BSHD inputs are supported but
the strided page gathers make the forward slower than on BHSD (measured 1.3x
on a 16-head layout); prefer BHSD. A `seqlen_k` that is not a multiple of 16
selects forward kernels whose K/V descriptors carry the row extent instead of
whole 16-row pages (measured 3-5% slower forward; the backward is
unaffected), so pad K/V to a multiple of 16 rows when that matters. A call
carries about 0.1 ms of host work (metadata checks, tensor conversion), which
only matters for sub-millisecond problems.

### KPage forward on blk64 metadata

On SM100/SM103 `kpage=True` (or a layout built with
`kpage_prepare_layout(..., sparse_block_size=64)`) also takes
`sparse_block_size=64` metadata, for the forward only. Every 64-row query
block is one tile of the M64 x N256 kernel, whose 256-row K/V stage is
gathered from sixteen 16-row pages, so a selected 64-row block with `v` valid
rows costs `16 * ceil(v / 16)` rows instead of 64.

On SM90 the same call runs a kernel that pairs adjacent query blocks
(`2p`, `2p + 1`): one CTA walks the union of their KV block lists as 16-row
pages gathered into 128-row K/V tiles, ordered first-only, common,
second-only, and its two compute warpgroups (one per query block) compute only
their own blocks' tiles. K/V of the blocks both query blocks select are thus
loaded once, on top of the page compaction. The layout holds these pair walks
when it is built on an SM90 device.

```python
o, lse = block_sparse_attention_forward(q, k, v, q2k_block_index, block_sparse_num, block_sizes, sparse_block_size=64, kpage=True)

layout = kpage_prepare_layout(q2k_block_index, block_sparse_num, block_sizes, q2k_block_nums, sparse_block_size=64)
o, lse = block_sparse_attention_forward(q, k, v, q2k_block_index, block_sparse_num, block_sizes, q2k_block_nums, sparse_block_size=64, kpage=layout)
```

The contract is that of the base blk64 forward with these differences:

- BF16, `head_dim=128`, MHA; `seqlen_q` and `seqlen_k` multiples of 64; a
  BSHD batch must be 1, as above.
- Every query row is computed, as on the base blk64 path: there is no
  `q_block_sizes`, and the output rows of query padding are not zeroed.
- Forward only. `block_sparse_attention_backward(..., kpage=...)` with
  `sparse_block_size=64` raises; run the base backward on the KPage forward
  output. A layout is bound to the `sparse_block_size` it was built for.

When to use it: as on blk128, the gain comes from `block_sizes < 64` and
grows with the padding fraction, and without padding the page gather costs
more than the base kernel's whole-block load. On random top-k patterns
(B300, BHSD, layout reused) it measured 1.11-1.36x at 30% padded tokens,
0.91-1.08x at 10%, and 0.85-0.96x without padding. Building the layout costs
about 0.3-1 ms per call on those shapes, so reuse a layout from
`kpage_prepare_layout` whenever the metadata repeats; with `kpage=True` the
build is paid on every call. Benchmark your own patterns before switching.
On SM90 the gain also grows with the overlap of adjacent query blocks' KV
lists (local or video-like patterns); full blocks with little overlap, such as
independent top-k selections, run slower than the base kernel.

## Current support

### Forward

| Architecture | Sparse block | Public input / kernel dtype | QK / V dimensions | Attention |
| --- | ---: | --- | --- | --- |
| SM90 | 64 or 128 (explicit) | FP16, BF16 | each of 64, 96, 128 | MHA, GQA, MQA |
| SM90 | 64 | BF16 / FP8 E4M3 | QK=128, V=128 | MHA |
| SM90 | 128 (explicit) | BF16 / FP8 E4M3 | QK=128, V=128 | MHA |
| SM100/SM103 | 128 | FP16, BF16 | QK=V=64, 96, or 128 | MHA, GQA, MQA |
| SM100/SM103 | 64 (explicit) | BF16 | QK=128, V=128 | MHA |
| SM100/SM103 | 64 | BF16 / FP8 E4M3 | QK=128, V=128 | MHA |
| SM100/SM103 | 128 (explicit) | BF16 / FP8 E4M3 | QK=128, V=128 | MHA |
| SM100/SM103 | 128, `kpage=True` | BF16 | QK=128, V=128 | MHA, GQA, MQA |
| SM100/SM103 | 64, `kpage=True` | BF16 | QK=128, V=128 | MHA |
| SM90 | 64, `kpage=True` | BF16 | QK=128, V=128 | MHA |
| SM90 | 128, `kpage=True` | BF16 | QK=128, V=128 | MHA |
| SM120 | 64 | FP16, BF16 | QK=128, V=128 | MHA, GQA, MQA |
| SM120 | 128 (explicit) | FP16, BF16 | QK=128, V=128 | MHA, GQA, MQA |
| SM120 | 64 | BF16 / FP8 E4M3 | QK=128, V=128 | MHA |
| SM120 | 128 (explicit) | BF16 / FP8 E4M3 | QK=128, V=128 | MHA |

For FP16/BF16, SM90 blk64 requires `S_q` to be a multiple of 64; native blk128 supports
positive arbitrary Q/KV lengths, including partial final blocks. Its fixed count may be
any positive value. The SM100/SM103 blk128 fixed count must be even and at
least two (the KPage path accepts any positive count); SM120 and the explicit
Blackwell blk64 path accept any positive fixed count. Variable counts use `q2k_block_nums`. `allow_empty_block_nums`
defaults to `False`; when it is `True`, empty rows (`q2k_block_nums == 0`)
produce `O = 0` and `LSE = -inf`. SM90 blk64 selects the empty-row handling as a
compile-time specialization, so the default non-empty configuration keeps its
branch-free fast path. Its split-KV execution excludes empty rows. Native SM90
blk128 supports empty rows and empty splits with `kv_splits=1..256`.
Sage FP8 uses the architecture-specific contracts above. SM100/SM103 native
blk128 supports empty rows and explicit split-KV; its default remains unsplit.

### Backward

| Architecture | Sparse block | Dtype | Head dimension | Attention |
| --- | ---: | --- | ---: | --- |
| SM90 | 64 or 128 (explicit) | BF16 | 128 | MHA |
| SM100/SM103 | 64 | BF16 | 128 | MHA |
| SM100/SM103 | 128 | BF16 | 64 or 128 | MHA |
| SM100/SM103 | 128, `kpage=True` | BF16 | 128 | MHA, GQA, MQA; `block_sizes` supported |
| SM90 | 128, `kpage=True` | BF16 | 128 | MHA; `block_sizes` supported |

On SM90 with `sparse_block_size=64` the backward also accepts a prepared `BsaBackwardMetadata`
through `metadata=` (see [Prepared backward metadata](#prepared-backward-metadata-sm90-blk64));
every other path rejects the argument.

Backward is not implemented for SM120. It requires equal QK/V dimensions and,
except on the SM100/SM103 KPage path, does not currently support GQA/MQA.

SM90 native blk128 requires CuTe DSL >= 4.6.2. Q/K/V, O/dO and gradient
buffers must have a contiguous head dimension, 16-byte aligned base pointers,
and row strides divisible by 16 bytes. BHSD/BSHD layouts and aligned noncompact
rows are addressed directly. Sparse metadata can be strided and remains on the
GPU; backward makes noncontiguous sparse indices and counts contiguous before
building CSR, as in blk64. Forward supports `block_sizes` of rank 1, 2, or 3; backward supports ranks
1 and 2. Both paths predicate sequence tails even without `block_sizes`.
Forward uses Q128 tiles and shares each KV128 load across two compute
warpgroups. Backward uses Q64xKV128 CTAs: both compute warpgroups execute all
five GEMMs, splitting KV rows for QK/dP/dK/dV and head dimensions for dQ.
Each native Q128/KV128 CSR edge streams two Q64 subtiles without expanding
the metadata. Backward accumulates gradients with FP32 atomics
and is not deterministic. Benchmark the explicit path for your shapes; blk64
remains the default, and native blk128 is not faster for every workload.

## Limitations

The current sparse kernels do not implement causal or local masking, dropout,
`mask_mod`, `score_mod`, paged KV cache, softcap, or variable-length packed
sequences. Regular forward/backward inputs must be rank four, use FP16/BF16 as
allowed above, and have a contiguous last (head-dimension) axis. Sage FP8 uses
the stricter BF16 BHSD contract described above and quantizes to E4M3
internally. Regular forward outputs are contiguous in the requested BHSD or
BSHD layout, including split-KV execution; FP8 output is contiguous BF16 BHSD.

Compilation is lazy. The first call for a new static configuration JIT-compiles
the relevant kernel; subsequent calls reuse an in-process cache.

The Torch public surface consists of allocating function wrappers under
`cudnn.ops`, also available under `cudnn` and `cudnn.BSA`; there is no separate `APIBase` class or explicit `compile()`
lifecycle for BSA.

Correctness tests and FP32 references are under
`test/python/block_sparse_attention/cutedsl`.

## SM100/SM103 FP8 blk128 source

The native producer and its private transfer/epilogue helpers are adapted from
[NVIDIA-JerryChen/Block-Sparse-Attention, f953680](https://github.com/NVIDIA-JerryChen/Block-Sparse-Attention/tree/f953680ff93a2ef2321ad7dc4291451eea4a9c3c/csrc/fwd/sm100_blk128):
`bsa_fwd_sm100_fp8_fused_pv.py` and `bsa_fwd_sm100_fp8.py` map to the same
filenames under `python/cudnn/block_sparse_attention/csrc/fwd/sm100_blk128/`.
The 16-token TMEM row-max helper is incorporated into the private blk128 helper
module. The integration uses cuDNN's existing quantizer, allocating Torch API,
device-specific JIT cache, current-stream execution, and split combine.
Standalone upstream package entry points and native AOT deployment are not
part of this API.

## Acknowledgements

We would like to express our gratitude to [huangyitong.hyt@alibaba-inc.com](mailto:huangyitong.hyt@alibaba-inc.com) and
[wenting.swt@alibaba-inc.com](mailto:wenting.swt@alibaba-inc.com) for providing testing and optimization feedback
throughout the deployment process, which has continuously advanced the BSA kernel
toward Speed of Light.

## Experimental JAX support

`cudnn.ops.block_sparse_attention_forward_jax` and
`cudnn.ops.block_sparse_attention_backward_jax` provide explicit forward/backward;
`cudnn.block_sparse_attention.jax_api.block_sparse_attention` adds first-order reverse-mode differentiation.
All three accept JAX arrays, eagerly or under `jax.jit`, and require no PyTorch.
The `cudnn.jax` namespace has been removed; use the `cudnn.ops` names.

The JAX implementation reuses the SM100/SM103 blk128 forward, bucketed CSR, backward preprocess,
backward, and gradient conversion kernels. Install `jax[cuda13]` alongside the
frontend. The runtime imports no PyTorch; torch parity tests are separate.

The JAX signatures omit Torch-specific launch options and caller-provided
output buffers. XLA owns the outputs and workspaces.

### Initial contract

| Property | Supported |
| --- | --- |
| Device | All arrays on one CUDA GPU of compute capability 10.0 or 10.3 (SM100 or SM103); other GPUs may be visible. Under `jax.jit` (including a zero-argument closure that captures every array), or with no committed JAX array among the inputs, the target is read off the visible GPUs, so a host exposing both SM100 and SM103 GPUs is refused there (eager calls on committed arrays are not). Single process |
| Data | BF16 Q/K/V/O/dO/dQ/dK/dV; FP32 LSE and accumulation |
| Dimensions | MHA with equal head counts, D=64 or 128; positive sequence lengths divisible by 128 |
| Layout | Compact BHSD or BSHD, independently specialized |
| Sparsity | 128-token blocks, int32 indices `[B,H,Sq/128,C]`, no `block_sizes` |
| Counts | Fixed even `block_sparse_num` in `[2,C]`, or runtime int32 `q2k_block_nums[B,H,Sq/128]` |
| Differentiation | First-order reverse-mode Q/K/V gradients through `block_sparse_attention` |

Variable counts must be in `[1,C]`, or `[0,C]` with
`allow_empty_block_nums=True`. Only each row's active prefix is read. Active
indices must be unique and in `[0,Sk/128)`. An empty row returns zero output,
negative-infinity LSE, and zero query gradient. Unselected K/V receive zero
contributions. Metadata **values are the caller's responsibility**: the runtime
validates shapes, dtypes, and static options, without copying metadata to the
host or synchronizing to inspect it. Invalid values are outside the contract.

SM103/SM110/SM120, blk64, FP16/FP8, GQA, different V dimensions, partial blocks,
variable sequence lengths, causal masks, dropout, sharding, `vmap`, JVP, and
higher derivatives are unsupported. KDA is outside this change. Static options
(including scale and fixed count) specialize compilation; sparse arrays remain
runtime operands. Do not mutate saved forward inputs or metadata before backward.

### Usage

```python
import jax
import jax.numpy as jnp
from functools import partial
from cudnn.block_sparse_attention.jax_api import block_sparse_attention as attention
from cudnn.ops import (
    block_sparse_attention_forward_jax as forward,
    block_sparse_attention_backward_jax as backward,
)

q = jnp.ones((1, 2, 256, 64), dtype=jnp.bfloat16)
k = jnp.ones((1, 2, 512, 64), dtype=jnp.bfloat16)
v = jnp.ones_like(k)
indices = jnp.broadcast_to(jnp.array([0, 2], jnp.int32), (1, 2, 2, 2))

o, lse = forward(q, k, v, indices, 2)  # eager; asynchronous GPU execution
run = jax.jit(partial(forward, block_sparse_num=2))
result = run(q, k, v, indices)
o, lse = result["o_tensor"], result["lse_tensor"]

dq, dk, dv = backward(jnp.ones_like(o), q, k, v, o, lse, indices, 2)

def loss(q, k, v, indices):
    return attention(q, k, v, indices, 2).astype(jnp.float32).sum()

grad = jax.jit(jax.grad(loss, argnums=(0, 1, 2)))
dq, dk, dv = grad(q, k, v, indices)
dq.block_until_ready()  # needed for host timing, not between GPU operations
```

Explicit forward returns a JAX pytree with `o_tensor`, `lse_tensor`; backward returns `dq_tensor`,
`dk_tensor`, `dv_tensor`. Both support tuple unpacking and dictionary access. Explicit helpers
do not register autodiff rules; use `attention` for `jax.grad`. LSE and metadata
are not differentiable public outputs. Backward must receive the exact matching
forward Q/K/V/O/LSE, sparse pattern, scale, layout, and empty-row option.

### Execution and ownership

Every launch receives XLA's CUDA stream. Forward is one custom call; backward
composes CSR construction, preprocessing, attention backward, and gradient
conversion inside one custom call. XLA owns all outputs and workspaces. CSR
counts and multi-bucket dK/dV accumulators are initialized for each invocation;
preprocessing clears dQ accumulation. No mutable global scratch or public input
buffer is donated. Atomic accumulation does not promise bitwise determinism.

`TensorSpec.mode` presents compact BHSD/BSHD memory in the order expected by each
kernel. It does not rewrite DLPack capsules or insert a Python transpose. XLA may
still insert layout conversions when surrounding operations use incompatible
physical layouts; this is not a universal zero-copy guarantee.

Two cached call builders retain static configuration and kernel objects, never
input arrays or pointers. XLA handles compilation and execution; there is no
separate plan lifecycle. One `custom_vjp` registration connects forward and
backward. `bucket_size_blocks` optionally controls backward query buckets;
the default reuses the torch path's heuristic. Multi-device placement and cache
portability across architectures require further qualification.

The JAX tests live in `test/python/block_sparse_attention/jax` and are collected by the OSS
suite. Torch-free import and gradient checks run in fresh subprocesses. To run
the suite in a torch-free container, pass `--confcutdir=test/python/block_sparse_attention/jax`
so pytest does not load the torch-based parent conftest.