Indexed QSA training
These attention calls consume existing indices; they do not compute the indexer score or top-k. See QSA entry points and the indexer (docs, GitHub) for block selection, bitmap training and the backward engine.
cudnn.ops.qsa_indexed_forward and qsa_indexed_backward accept packed token
indices directly. The implementation lives in cudnn.QSA.indexed and shares
QSA’s attention kernels. It supports BF16
12Q/1KV and 6Q/1KV (one KV head shared by twelve or six query heads) with head
dimension 256 on SM100/SM103 with CuTeDSL >= 4.7 and on SM107 with
CuTeDSL >= 4.8 (the backward needs the CUDA 13 build of the DSL).
These explicit calls return TupleDict objects in the order shown. They do not
register an autograd function. Save the original FP32 output and natural-log
LSE; casting the BF16 output back to FP32 is insufficient. Packed documents
reach the backward through cu_seqlens (int32 [B + 1] prefix sums) and
max_seq_len with selection_mode="causal_topk" (below). Without cu_seqlens
the rows are one sequence, which stays exact for packed documents but serves
only the first document on the dense prefix path.
Hq is 12 or 6. All tensors are contiguous on one CUDA device. Data tensors
require 16-byte alignment; indices and LSE require four-byte alignment. T > 0
and T * Hq * 256 <= INT32_MAX. Scale is a finite host scalar.
W is a positive multiple of four, at most 2052. Each four-slot group describes
one distinct complete block: [start, start+1, start+2, start+3], or four -1
padding slots. A partial causal tail is a consecutive prefix followed by -1.
Valid tokens are absolute packed token IDs, belong to the query’s document,
and never exceed the query’s absolute token ID. A tail begins at the document’s
own block origin; packed document starts need not be divisible by four. Groups
may appear in any order and padding groups may occur between selected groups.
Groups must not overlap or repeat within a row. These device-value relationships
are caller contracts; execution does not read indices back to the host. A row’s
blocks accumulate in the order of its groups, so the result bytes follow that
order.
The Relax layout with 512 complete four-token groups plus a four-slot causal tail fits directly, including invalid tail slots. A 2051-wide carrier must be padded explicitly by the caller; this API does not silently convert it. Empty rows produce zero output/gradients and negative-infinity LSE. Selection is not differentiated. Pass the same selection and scale to forward and backward.
A GPU hash table compacts the selected starts. Backward runs the engine
described in QSA (docs, GitHub)
over them. A nonfinite K/V value of a masked token adds exact zeros, and one of
a selected token poisons its queries. Every call rebuilds metadata, including
during CUDA Graph replay. The backward compiles different kernels below and
above the 22528-token dense crossover, from 196608 tokens on, and for each
deterministic value. Before capture, warm up with the query head count,
token counts, max_seq_len and deterministic the capture will use.
The public wrappers allocate outputs and one workspace. The internal APIBase
plan validates current storage and uses only caller-owned buffers in execute;
its cache retains immutable geometry and compiled kernels. Indexed metadata
storage is proportional to T * W, independent of the full key-block count.
At 128K with W=2052, compact IDs occupy 256.5 MiB in either direction, compared
with 16 GiB for the bitmap entry’s [T, NB] IDs. The 4 GiB bitmap is also
unnecessary. These are metadata sizes, not total process
memory; saved state, outputs and FP32 gradient accumulators are additional.
Complete blocks and causal tails
Packed documents and the top-512 selection contract are declared through the same public entry point:
selection_mode="general" remains the default. The causal_topk mode declares
the following stricter input contract; it is never inferred from tensor shapes:
- Q/K/V, grad_out and saved FP32 output are finite. LSE is finite and comes from forward with the same indices and scale.
W=2052andscale=1/16. For query positionpwithin its document, select exactlymin((p + 1) // 4, 512)distinct, complete four-token blocks from that document’s causal prefix, followed by its incomplete causal tail. Complete blocks fill consecutive groups starting at column zero, in any order. No padding holes or empty selections are allowed. Before 512 blocks are available, pad the unused groups with-1; the tail may follow the complete blocks or occupy the last four slots, as in Relax. After 512 blocks are available, the tail occupies the last four slots. Invalid tail slots and all unused slots are-1.- Packed inputs require
cu_seqlens: contiguous device int32 prefix sums[B + 1], starting at zero, ending atT, and nondecreasing. Empty documents and document starts not divisible by four are supported. Indices use absolute packed token IDs; block origins restart at each document boundary. Withoutcu_seqlens, allTtokens are one document. - Optional
max_seq_lenis a host integer in[1, T]bounding every document length. It defaults toT. A bound below the 22528-token dense crossover plans no sparse path, so the workspace is smaller and no sparse launches run. Every document of every replay must respect this bound. The kernel rereads devicecu_seqlenson replay; boundaries are not cached.
These device-value promises are caller contracts, not host-side checks.
Use general for arbitrary selections. Passing cu_seqlens or max_seq_len
with general is rejected because that mode obtains its bounds from indices.
The keyword-only arguments preserve existing positional calls.
deterministic (False by default) applies to both modes.
deterministic=True repeats the bytes when every
document’s groups lie on one four-token grid, which causal_topk requires and
general does not check. Both modes run the
backward engine described in QSA (docs, GitHub),
which also describes deterministic and gives the record
and workspace sizing.
The 12Q/1KV and 6Q/1KV backward requires the CUDA 13 CuTeDSL compiler in addition to the CuTeDSL versions and architectures above. A CUDA 13 PyTorch build alone does not establish the DSL compiler version; unsupported compiler builds are rejected before kernel compilation.
The indexed entry is useful when a producer already has token IDs. Use the unchanged Relax
comparison in benchmark/qsa/benchmark_relax.py for the intended workload;
Miles ratios do not establish a speedup over Relax, and attention microbenchmarks
do not measure selector, TE, tensor-parallel communication or model throughput.