Skip to navigation

Gated Attention Block (SM107 / SM100)

View as MarkdownOpen in Claude

This is an experimental API and subject to change.

Overview

The gated attention block is the first model-level FE-OSS API: a set of FROST CuTe-DSL kernels behind one Python class, one workspace and one execute() call. It implements the gated attention sub-layer used by Qwen3.5- and Qwen3.8-style models (the API is named by op geometry, the model is provenance only; the Qwen3.8 family’s geometries, the Qwen3.5-397B tensor-parallel shards and the bound up to which the block is Qwen3.8-Flash-Next’s sparse layer are under Qwen3.8 family / Qwen3.8-Flash-Next):

h [B, S, d_model] (post input-layernorm)
|
| (1) QKV + GATE projection h @ W_qkvg^T FROST GEMM
+----------------------------------------------------------------------------+
| Q [B,S,H_q,D] | GATE [B,S,H_q,D] | K [B,S,H_kv,D] | V [B,S,H_kv,D]
|
| (2)+(3) QK-RMSNorm per head (optional) + partial RoPE on the first rope_dim ONE kernel (Q and K only)
|
| (4) SDPA O = softmax(Q K^T * scale + mask) V, GQA H_q / H_kv FROST SDPA (the Rubin d256 rows; the SM100 d256 kernels on cc 10.0)
|
| (5) gate O_gated = O * sigmoid(GATE) elementwise, after the SDPA epilogue
|
| (6) out projection O_gated @ W_o^T FROST GEMM
v
out [B, S, d_model]

Every stage is a FROST kernel: the two projections drive the shipped FROST GEMM engine (pinned by name in the graph’s ranked plan list), stage (4) drives the shipped FROST SDPA through its standalone adapter, and stages (2)+(3) and (5) are this block’s own kernels. There are no cuBLAS or cuDNN-backend call-outs.

Target parts. Every other architecture is declined with a typed error (NotImplementedError), never served slowly; the served parts are ONE tuple, cudnn.gated_attention_block.DENSE_SERVING_CCS = ((10, 7), (10, 0)), which the SDPA stage’s part gate, serving_envelope() and the test suite’s markers read:

partforms servedstatus
NVIDIA Rubin (SM107, cc 10.7)every form: the three precisions, the fusion knobs, the serving forms (write-through, EXTEND, DECODE), the block-sparse / indexer modes, the e4m3 cache, the backwardmeasured (the tables below)
cc 10.0 (SM100: B200 / GB200; 2026-10-29)the DENSE bf16 / f16 INFERENCE forms only — the dense prefill form (kv_source="self"), the EXTEND form and the DECODE form over the pools — through the shared SM100 d256 kernels (the paged prefill tile with PackGQA, the SM100 d256 decode tile at 16 columns and its split combine in the combine module’s own forms, unmeasured on this line), the sigmoid gate as its own launchFUNCTIONAL ONLY, unmeasured (the cells ran 2026-10-29 on the GB200 class, cc 10.0, 152 SMs: 351 passed / 0 failed over the suites’ cc 10.0 rows; no performance number); declined by name there: fuse_gate=True (the sm100 row claims no gate epilogue), the fused projection forks (fuse_norm_rope=True outside the DECODE form’s lowering, fuse_norm_rope_small_m=True, an explicit fuse_cache_write=True: sm_107a renderings), quant (FP8 / MXFP8), save_for_backward=True (the backward binds the Rubin SDPA backward), thd=True over the block’s own K / V, every QsaSpec form (the sparse core, the lean tile through the block, the indexer, the e4m3 cache), decode_q_tile_max, the 32-column decode arm (17 .. 32 packed rows on a wide launch, an unpacked step of 17 .. 32 tokens: DECODE_WIDE_TILE_CCS, empty until its cells run); use_pdl auto reads False
cc 10.3 (B300)none — the same SM100 kernels, unproven through the blockdeclined BY NAME until its cells run

Three precisions share the signature, and two fp4 modes ride the MXFP8 one (both are fields on MxQuantSpec, so they are unspellable on the bf16 and per-tensor FP8 pipelines rather than declined):

modehow it is selectedpipeline
bf16 / fp16h, weights and cos/sin in bf16 or fp16, quant=None5 stages, 5 launches (4 with inplace_qkv)
FP8, per-tensor static scalese4m3 h / W_qkvg / W_o and a QuantSpecunfused: FP8 projections with the descale folded into the epilogue, bf16 norm+RoPE, two quantize passes (Q/K/V, gated O), the Rubin per-tensor FP8 SDPA and an FP8 out projection (9 launches); fully fused: 3 launches
FP8 projections + bf16 attention + e4m3 gated O — the PAGED serving forms (2026-10-30)e4m3 h / W_qkvg / W_o and QuantSpec(attention_dtype=torch.bfloat16) (QuantSpec.bf16_attention(descale_h, descale_w_qkvg, descale_w_o, scale_o)), with kv_source="paged"the two FP8 projections with the descale folded into their bf16 epilogue, bf16 norm+RoPE and pool write-through, the bf16 decode / prefill tiles over bf16 pools (the bf16 serving forms’ own kernels and routes: the gate in the combine / the paged body’s epilogue on cc 10.7, its own launch otherwise), ONE quantize pass (the gated O at scale_o) and the FP8 out projection — launch counts MEASURED by the CUPTI census cell at decode row counts (B 2 x S_q 1, every kernel counted): cc 10.7 with the gate in the combine 6 (write-through folded) / 7 (separate cache write), 7 / 8 with the separate gate; cc 10.0 (the gate its own launch) 7 / 8 — the bf16 serving chain’s own counts for the same declaration: both FP8 forced-tile GEMMs are one kernel each where the bf16 chain’s small-M out projection is a split-K kernel plus a reduce, so the quantize launch takes the reduce’s place; the DECODE and EXTEND forms only (“Serving: FP8 projections with bf16 attention” below)
MXFP8, per-32-element E8M0 block scalese4m3 h / W_qkvg codes, the two scale-factor blobs (h_sf, w_qkvg_sf) and an MxQuantSpecunfused: block-scale projection GEMM, bf16 norm+RoPE, MXFP8 quantize (Q/K rowwise, V columnwise), the Rubin d256 MXFP8 SDPA, per-tensor quantize of O, FP8 out projection (9 stages); fully fused: 3 launches
MXFP8 with MXFP4 weightsMxQuantSpec(w_qkvg_dtype=torch.float4_e2m1fn_x2) and an e2m1 W_qkvg [N, d_model // 2]; h, h_sf and w_qkvg_sf unchangedunfused: the MXFP8 pipeline with stage (1) on the catalog’s mixed MXFP8 x MXFP4 block-scale row (E8M0 scales per 32 on both sides) — the same 9 launches; fully fused: the same 3 launches, the fused projection fork’s e2m1-B arm reading the packed codes and the unchanged blob inside its norm+RoPE+quant epilogue (dense only, like the MXFP8 fused pipeline)
MXFP8 with fp4 O (NVFP4 or MXFP4)MxQuantSpec(o_fp4=Fp4Format.NVFP4 | Fp4Format.MXFP4), an e2m1 W_o [d_model, H_q * D // 2] of the SAME format and its scale blob (sample_w_o_sf / w_o_sf)the per-tensor tail is replaced: one quantize_fp4 launch writes the gated O as e2m1 codes plus the out projection’s scale blob, and stage (6) becomes the fp4 x fp4 block-scale GEMM (no per-tensor scale on either side). Unfused: 9 launches (quantize_fp4 takes the per-tensor quantize’s place); fully fused: 4 launches (the gated MXFP8 SDPA writes bf16 O, then quantize_fp4, then the fp4 out projection). Composes with the MXFP4 weights on both pipelines (fully fused: the e2m1-B projection fork + the fp4 tail, 4 launches)

Fusion knobs

Two optional forward fusions are constructor flags; each is a different compiled specialization behind the same execute() signature, and both default to off. The backward has two knobs of its own, both default off and both bitwise the unfused block: fuse_gate_bwd (the gate backward emits the SDPA backward’s delta, one launch fewer) and fuse_wgrad_overlap (the two weight-gradient GEMMs run on a block-owned side stream, forked from and joined back to the launch stream through events, overlapping the SDPA backward chain) — see Backward. On cc 10.0 (2026-10-29) both forward fusions are typed declines by name: the fused projection fork is an sm_107a rendering (“rendered for sm_107a (Rubin); this device is SM100” — except on the DECODE form at decode row counts, where fuse_norm_rope=True is LOWERED to the unfused chain and therefore served), and fuse_gate=True is refused by the device’s engine row (“fuse_gate=True is not served by the f16/bf16 SM100 SDPA engine row”: no gate epilogue on the SM100 d256 kernels, no gate in the SM100 line’s split combine) — stage (5) runs as its own launch under fuse_gate=False.

  • fuse_norm_rope=True folds stages (2)+(3) into stage (1)‘s epilogue: Q/K tiles are normed and rotated on the fp32 accumulator and written once (needs inplace_qkv; inference only, no pre-norm Q/K is kept). Under FP8 / MXFP8 the same epilogue also quantizes: compact e4m3 q8 / k8 / v8 (and the MXFP8 scale factors) come straight out of the GEMM, so the quantize passes disappear.
  • fuse_gate=True folds stage (5) into the SDPA epilogue: O *= sigmoid(GATE) after the dead-row select, with the gate tile TMA-staged by the load warp (inference only: no pre-gate O for the backward). Dense and packed (thd=True): under THD the gate rides the packed d256 prefill’s epilogue through the SDPA’s THD gate descriptor (G mapped like Q / O, at the slab’s gate column stride), so the packed block is one launch fewer too.
  • fuse_prepare=True (serving, with paged_kv_page_size > 0) folds stages (2)+(3) and the paged-cache write-through (4w) into ONE launch: Q / K are normed and rotated in place and the post-RoPE K, the V and the raw indexer key land in the pools from the same registers — every output bitwise the two-stage form, one launch fewer (two with the indexer band). In-place bf16 / f16 only, on the dense declaration with the write-through alone or on the block-sparse DECODE mode over the pools (kv_source="paged" on a QsaSpec geometry, 2026-10-21: the prepare kernel’s paged arms — the raw-key pool under caller lists, the indexer’s query + compress arms under index_source="indexer"; fuse_prepare=None is the MEASURED default there: True on cc 10.7 (2026-10-21: the fused step +9.6 .. +14.5 % at B <= 4 and +4.4 .. +10.0 % at B 32 over 19 decode cells, bitwise), False on every other part until measured, and False out of place (inplace_qkv=False) on every part — the AUTO reading is never True where an explicit True is declined); fuse_norm_rope together with it is a typed decline (pick one), as are thd=True and kv_source="paged" on a DENSE geometry (the prepare kernel’s packed arm and the dense DECODE form’s are follow-ups) and kv_cache_dtype=torch.float8_e4m3fn (the e4m3 cache’s write-through quantizes in its own stage); a geometry with rope_dim=0 (RMSNorm only) is a ValueError at check_support naming the zero-width RoPE table operands the fused launch would bind. use_pdl=True launches the fused stage with the programmatic-dependent-launch attribute, as it does the two stages it replaces. See “Serving: write-through”.

With both on, the block is three launches: proj(+norm+RoPE[+quant]) -> sdpa(+gate) -> out_proj.

geometry.qk_norm=False runs RoPE-only Q/K on every path (norm weights are passed as None, no rstd is produced, the backward drops the norm-weight gradients).

API

import cudnn
from cudnn.gated_attention_block import (
GatedAttentionBlockFwd, GatedAttentionBlockGeometry, QuantSpec, MxQuantSpec, Fp4Format,
build_fused_qkvg_weight, qkvg_from_hf, qkvg_from_vllm, GatedAttentionBlockBwd, RecomputePolicy, SavedForBackward,
)

Geometry

GatedAttentionBlockGeometry (frozen dataclass; validate() raises ValueError):

fieldmeaning
d_modelmodel width
h_q, h_kvquery heads and KV heads (GQA); h_q % h_kv == 0
d_headhead dim of Q, K, V and O (d_qk == d_v); selects the SDPA flavor (d256 today)
rope_dimthe LEADING dims of each head that rotate; [rope_dim, d_head) pass through
qk_norm (True)apply QK-RMSNorm before RoPE; False = RoPE only, no norm weights, no rstd
qk_norm_eps (1e-6)RMSNorm epsilon
attn_scale (None)softmax scale; None = d_head ** -0.5
is_causal (True), causal_bottom_right (False)causal mask and its diagonal alignment
window_left, window_right (-1)sliding window bounds; -1 = unbounded

Qwen3.8 family / Qwen3.8-Flash-Next

The block’s math is also the gated attention sub-layer of the Qwen3.8 family (provenance only — the API stays named by op geometry). The five geometries below are pinned in the layout contract (test_layout_contract.py: the stage-(1) column map, the GQA ratio, the tile plans, the norm kernel’s fitted tile) and run end to end against the fp32 oracle on Rubin (test_block_end_to_end.py, the qwen38 cells: the unfused bf16 pipeline at all five and the fully fused pipeline at 24 / 2, B = 2 for the Flash-Next geometries and B = 1 for the two large siblings, S in 2051, the weights loaded through qkvg_from_hf):

modeld_modelh_q / h_kvN (= n_qkvg)GQA groupstages (2)+(3) kernel
Qwen3.8-Flash-Next, TP 1256024 / 21331212TMA, 12-row tile
Qwen3.8-Flash-Next, TP 2256012 / 1665612TMA, 12-row tile
Qwen3.8-Flash-Next, TP 425606 / 135846LDG (no TMA tile fits 6 / 1)
Qwen3.8-27B512024 / 4143366TMA, 12-row tile
Qwen3.8-2.4T-A95B819264 / 43481616TMA, 16-row tile

All five share d_head = 256, rope_dim = 64, QK-RMSNorm with zero-centered weights and the per-head [q_h | gate_h] split of q_proj — load them through qkvg_from_hf (next section). The 27B and 2.4T members are dense gated attention: the block IS their attention layer at every sequence length it serves (a prefill block: S = 1 is decode, out of the prefill bodies’ scope).

Qwen3.5-397B tensor-parallel shards. The 397B’s full-attention layer (32 / 2 at TP 1, d_model 4096; the same d_head 256, rope_dim 64, QK-RMSNorm and per-head [q_h | gate_h] split, loaded through qkvg_from_hf or qkvg_from_vllm) as the serving stacks shard it — the query heads divided across the ranks, the two KV heads divided at TP 2 and REPLICATED from TP 4 on (one KV head per rank; the TP 4 shard is the serving deployment’s) — is pinned the same way (test_layout_contract.py, the qwen35_shard cells: the N-axis map, the tile plans at 64 / 128 / 256, the fitted norm tile) and runs end to end on Rubin in all three forms, one cell per shard within the dense suite’s budget: dense (test_block_end_to_end.py: B = 1, S = 512, HF-form weights against both oracles), EXTEND (test_block_serving_extend.py: two sequences, a 100-token step over a 60-row prefix and a 156-token step over a 100-row prefix) and DECODE (test_block_serving_decode.py: S_q = 1 over kv_lens 100 / 384 / 500 and a padded entry), each shard on the kernel block size its deployment allocates:

shardd_modelh_q / h_kvN (= n_qkvg)GQA groupstages (2)+(3) kernelend-to-end cells (bf16)
Qwen3.5-397B, TP 1409632 / 21740816TMA, 16-row tilethe perf tables below (the kernel cell); the serving suites’ 32 / 2 matrix at a shrunk width
Qwen3.5-397B, TP 2409616 / 1870416TMA, 16-row tiledense; EXTEND and DECODE at page 16
Qwen3.5-397B, TP 440968 / 146088TMA, 8-row tiledense; EXTEND and DECODE at page 32
Qwen3.5-397B, TP 840964 / 125604TMA, 4-row tile (one row per warp)dense; EXTEND and DECODE at page 64

Every N is a whole number of 256-column tiles, so the forced 256-wide GEMM tile of the serving route holds at every shard and qkvg_tile_plan(64 / 128 / 256) is well defined; the replicated single KV head lets every power-of-two head count fit a TMA norm tile (the 3.8 family’s 6 / 1 is the one pinned geometry that takes the LDG kernel). A shard’s decode route is its GROUP’s (the rule reads h_q / h_kv and d_head only, never d_model or the KV-head count): 16 / 1 takes the 32 / 2 entries of the table under “Serving: the decode form”, 8 / 1 its own, 4 / 1 the group-4 entries — every shard’s group divides the prefill tile, so the KV is streamed once per sequence and KV head at every S_q <= 8 once the step fills the part (the decode tile in one unit, then the packed prefill tile at B x H_kv >= DECODE_PREFILL_ROUTE_MIN_CTAS = 64 CTAs; below that fill the steps past the single-unit table ride two units of the 64-column decode tile, split over the part); past a group’s packed routes (32 / 2 from S_q 9) the step rides the decode tile unpacked, one head per CTA, up to S_q = 32; a typed decline past 32 tokens.

Flash-Next: exact for <= 2051 visible tokens. Flash-Next’s attention layer is Qwen Sparse Attention (QSA): the same gated core restricted, per query, to indexer-selected 4-token blocks under a 2048-token budget — the top min(512, n_blocks) complete blocks plus the open tail block, at most 2048 + 4 - 1 = 2051 keys per query. A query that sees n tokens has floor(n / 4) complete blocks, and every one of them is selected when floor(n / 4) <= 512, i.e. n <= 2051: the selection is the identity and causal attention over the visible tokens IS the QSA layer. So for every query with at most 2051 visible tokens — every sequence of up to 2051 tokens — the dense block computes Flash-Next’s attention layer exactly: the same function (a sparse evaluation sums the same terms in another order, so the identity holds within the suite’s budget, not bitwise). Beyond 2051 visible tokens QSA drops the blocks its indexer did not select while this block attends densely — a different function: declare the geometry with qsa=QsaSpec(...) and hand the selection in as execute(block_ids=) (“Sparse attention (QSA)” below) and the block runs the sparse core as its attention stage — or declare QsaSpec(index_source="indexer", index_band=True) and the block derives the selection itself (the in-block indexer: the prefill form over the prompt, and since 2026-10-21 its DECODE step over the paged compressed-key cache).

Weights and tables

  • W_qkvg [N, d_model] with N = (2*H_q + 2*H_kv) * D, column blocks Q | GATE | K | V (a fifth block INDEX under a block-sparse geometry’s indexer band, below); the block’s tile alignment is QKVG_TILE_ALIGN = 64 columns. Build it ONCE at load time, never per call:

    • from a HF Qwen checkpoint (Qwen3-Next, Qwen3.5, the Qwen3.8 family / Flash-Next) with the documented entry point, which also prepares the two QK-norm weights:

      w_qkvg, w_q_norm, w_k_norm = qkvg_from_hf(
      attn.q_proj.weight, attn.k_proj.weight, attn.v_proj.weight, attn.q_norm.weight, attn.k_norm.weight, geometry,
      act_dtype=torch.bfloat16, # the block's activation dtype; all three results come back in it
      )

      It applies the two conventions those checkpoints share: the double-width q_proj is split PER HEAD (q_proj(x).view(..., H_q, 2*D).chunk(2, dim=-1), i.e. weight rows [q_0 | gate_0 | q_1 | gate_1 | ...]), and the QK-norm weights are zero-centered (x_normed * (1 + w), w initialised to zeros), so the block — which multiplies by the vector it is given — receives (1 + w) in the activation dtype (rounding cost below).

    • from three matrices with an explicit layout: build_fused_qkvg_weight(w_q_gate, w_k, w_v, geometry, q_gate_layout="per_head" | "flat"). The layout is a property of the checkpoint and is not inferable from the tensor ("flat" = all Q heads, then all GATE heads); getting it wrong applies every gate to the wrong head with no error anywhere. Omitting q_gate_layout still means "flat" but emits a FutureWarning (the deprecation category Python shows under its default filters, so a loader in a library module sees it too) — pass it explicitly.

    • from a serving stack’s merged qkv_proj.weight at a tensor-parallel rank (vLLM’s QKVParallelLinear, SGLang’s likewise): qkvg_from_vllm(qkv_proj_weight, q_norm_weight, k_norm_weight, geometry, act_dtype=torch.bfloat16). The merged weight’s rows are [q_proj | k_proj | v_proj] at the RANK’s head counts — 2*H_q*D rows of q_proj in the checkpoint’s per-head [q_h | gate_h] order, then H_kv*D of k_proj and H_kv*D of v_proj — and geometry describes the shard (h_q, h_kv = the rank’s heads: 8 / 1 for a 32 / 2 model at TP 4, the KV head replicated across the ranks that share it). The twin cuts the three row ranges and hands them to qkvg_from_hf, so all three results are bitwise qkvg_from_hf’s on the same HF tensors: the per-head split is fixed (there is no layout argument), and the norm weights — replicated on every rank — take the (1 + w) form derived from the geometry. w_qkvg is a row permutation of the input with the same byte count, so a layer may replace its merged weight by it in place and keep its o_proj.weight as W_o. Under an indexer band the indexer projection arrives as index_qk_proj_weight=, or appended to the merged weight’s rows as a stack that packs the (replicated) indexer projection into the same GEMM holds it — one of the two, never both.

  • W_o [d_model, H_q * D].

  • cos, sin [B, S, rope_dim] rotary tables in the activation dtype (rotate-half convention on the first rope_dim dims of every head).

  • w_q_norm, w_k_norm [D] (both None iff geometry.qk_norm is False). The block multiplies the normalised row by this vector as given, so a zero-centered checkpoint weight is handed in as (1 + w) (qkvg_from_hf does this). Forming 1 + w in fp32 and rounding it once into the activation dtype costs at most half an ulp at 1.0 per channel — 2^-8 = 0.39 % relative in bf16, 2^-11 = 0.049 % in f16 — a systematic per-channel scale error inside the block’s accuracy budget (cos >= 0.999 on out) but not bit-faithful to the checkpoint — measured end to end at the five Qwen3.8 family geometries (the qwen38 cells of test_block_end_to_end.py, Rubin, w ~ N(0, 0.1)): the block on the rounded weights reads cos >= 0.99998 against an oracle fed the fp32 (1 + w) at every cell (and 0.999995-0.999996 unfused / 0.999986-0.999987 fully fused against the oracle fed the same rounded weights). Per channel, for w ~ N(0, sigma) (2^18 samples, seed 0, the table test_rounding_of_a_zero_centered_norm_weight_handed_in_as_one_plus_w prints; rel err = |rnd(1 + w) - (1 + w)| / (1 + w)):

    sigma of wbf16 max rel errbf16 mean rel errbf16 channels rounded to exactly 1.0f16 max rel errf16 mean rel errf16 channels rounded to exactly 1.0
    1e-30.389 %0.078 %97.5 %0.049 %0.018 %28.5 %
    1e-20.389 %0.146 %22.9 %0.049 %0.018 %3.0 %
    1e-10.389 %0.144 %2.4 %0.049 %0.018 %0.3 %

    A trained -2^-9 < w < 2^-8 is lost entirely in bf16 — an asymmetric interval, because bf16’s spacing is 2^-8 just below 1.0 and 2^-7 just above (f16 loses -2^-12 < w < 2^-11); it is the 97.5 % at sigma = 1e-3 above. The faithful form — the weight stored as w, the 1 added in fp32 inside the norm kernels behind a geometry field norm_weight_offset = 1.0 — is a follow-up; qkvg_from_hf derives which form to hand over from that same geometry field (absent or 0.0 today), so the two can never compose into 1 + (1 + w).

  • MXFP8 only: h_sf and w_qkvg_sf, the E8M0 scale factors of h and W_qkvg in cuDNN’s F8_128x4 order (uint8 or float8_e8m0fnu; byte counts from cudnn.gated_attention_block.kernels.proj_gemm.sf_blob_bytes).

  • fp4 weights are packed e2m1, dtype torch.float4_e2m1fn_x2, stored [N, K // 2] — two codes per byte along the contraction axis, LOW nibble = even k. The block checks the STORAGE shape ([N, d_model // 2] for W_qkvg, [d_model, H_q * D // 2] for W_o); a logical [N, K] fp4 tensor, or uint8 storage, is a typed ValueError (torch can .view(torch.float4_e2m1fn_x2) packed bytes but cannot cast to fp4).

    • MXFP4 W_qkvg (MxQuantSpec.w_qkvg_dtype=torch.float4_e2m1fn_x2): w_qkvg_sf is UNCHANGED — the same E8M0 / 32 F8_128x4 blob over n_qkvg x d_model as for e4m3 codes. The fused projection fork reads the same [N, d_model // 2] storage and the same blob through the packed sub-byte TMA format, whose rules the fused stage checks before any launch (typed ValueError): the fused pipeline’s d_model % 128 == 0, a 32-byte-aligned weight base address and a row stride that is a multiple of 32 bytes (a contiguous [N, d_model // 2] weight satisfies both; a 16-byte-aligned slice of a larger buffer or a padded-row view does not), and — on both arms — a unit stride along K.
    • fp4 W_o (MxQuantSpec.o_fp4): its scale blob w_o_sf is in the SAME format as O — float8_e4m3fn scales per 16 for Fp4Format.NVFP4, float8_e8m0fnu per 32 for Fp4Format.MXFP4 (either as uint8 or as that dtype), sf_blob_bytes(d_model, H_q * D, block) bytes in F8_128x4 order (padded to whole 128-row x 4-block atoms; pad bytes, if any, 0x00).

Sparse attention (QSA) — declaration, layout, loader and the sparse attention core

QsaSpec (frozen dataclass; attached as GatedAttentionBlockGeometry(qsa=...); validate() raises ValueError) declares block-sparse attention: every query attends to the keys of its SELECTED 4-token blocks and to the open tail block of its visible range, under the causal mask (Qwen Sparse Attention: a 2048-token budget at block size 4 = the defaults block_size=4, top_k=512). It is a DECLARATION ATTRIBUTE — it changes the function — never a knob.

fieldmeaning
block_size (4)tokens per selectable block; 4 only (one gather of four rows per block)
top_k (512)blocks per query in the caller’s list: a multiple of 4 in [4, 512] (it sizes the kernel’s index staging)
index_source ("caller")"caller": execute(block_ids=) carries the selection; "indexer": the block runs the indexer over its own band and derives the selection (bf16, index_band=True; “The in-block indexer” below)
index_band (False)W_qkvg carries a fifth band ProjBlock.INDEX of (index_heads + index_kv_heads) * index_head_dim columns below V
index_cache_dtype (None)appended: the indexer cache’s dtype under index_source="indexer" — None / torch.bfloat16 (the compressed keys and the queries as the compress kernel writes them) or torch.float8_e4m3fn (the fp8 indexer cache: both as plain e4m3 of the same values, no scales; on the prefill form scored through the scorer’s MXFP8 arm with unit scale factors, index_k_compressed then an e4m3 buffer; AT DECODE over the pools (2026-10-21) the paged index_kv_cache is e4m3 — the prepare / compress kernels store the e4m3 cast of the bf16 row directly and the paged scorer reads it on its fp8 mma.sync arm, no cast kernel). Numerics-changing (3 mantissa bits move the weakest selected blocks at the k-th boundary: 99.0 / 97.4 / 96.5 % set agreement with the bf16 selection at S = 4096 / 32768 / 131072 on synthetic activations, every flip a rank-boundary flip), so an attribute, never a knob; bf16 stays the default and the reference. Refused under index_source="caller" (nothing of the block reads or writes an indexer cache there); e5m2 named
index_list_form (None)appended: the block-sparse DECODE mode’s list form (kv_source="paged" on a dense [B, S_q] block) — "shared": ONE list per sequence, [B, top_k], selected at the step-0 position kv_lens[b] - S_q and shared by the step’s rows (the MTP form; S_q <= 4, the sparse core’s shared-list cap); "per_row": one list per row, [B, S_q, top_k] / [T, top_k], at the row’s own position (the serving stacks’ per-row selection) at ANY S_q; None: "shared" at S_q <= 4 and "per_row" past it under caller lists (the former derivation), "per_row" under the in-block indexer over the pools. A declaration attribute (numerics-changing: which keys the speculative rows see), never a knob. Typed declines: "shared" past four rows (shared_list_rows), "shared" under the indexer over the pools; "shared" on the block’s own K / V (kv_source="self") is refused by name. The envelope publishes decode.index_list_forms / decode.index_list_form_default
index_compress_rounding ("single")appended (2026-10-29): the rounding points of the indexer’s two compress chains — the pool-4 compressed KEY and the pool-1 indexer QUERY — at every writer (the prefill form’s two recipes, the chunk / decode step form, the fused prepare’s arms: a compile key of each, so one declaration’s chunked prefill and its decode steps are one chain). "single" (the default, the reference): fp32 through the mean, the RMSNorm and the RoPE, one rounding at the store (under the e4m3 cache the e4m3 cast of that bf16 row). "hf": the HF / vLLM-prepare dtype flow — the pooled key rounded to the io dtype before its norm, the normed row (query and key, all dims) rounded before its rotation, the rotation’s three operations each rounded, the store exact — the same ROUNDING POINTS as a vLLM-written cache row, not its bits (the fp32 reduction orders differ). The attention’s own Q / K norm + RoPE is not governed by it. Numerics-changing, so an attribute, never a knob; refused under index_source="caller". The envelope publishes decode.index_compress_roundings
index_heads, index_kv_heads, index_head_dim, index_norm_eps (4, 1, 128, 1e-6)the indexer’s geometry; they size the band (and index_kv_heads x index_head_dim the raw indexer key’s write-through pool); under index_source="indexer" the scorer reads them (index_heads one of its packed head groups, index_head_dim its head dim)

QsaSpec.identity_bound (top_k * block_size + block_size - 1 = 2051 at the defaults) is the visible-token count up to which every query’s complete blocks fit the list, so a full list reproduces dense causal attention exactly. is_causal=False or a sliding window together with qsa is a ValueError (the selection IS the sparsity).

Served today:

  • the layout: qkvg_blocks / qkvg_block_widths / qkvg_offsets / qkvg_heads / qkvg_head_dims grow a fifth entry iff index_band (at d_model=2560, h_q=24, h_kv=2: N = 13312 + 640 = 13952, offsets 0, 6144, 12288, 12800, 13312; a 256-column tile plan is not defined there, a 64- or 128-column one is — at decode row counts, B x S_q <= 128, the slab GEMM takes the small-M route’s 128-row swap-AB tile, 109 CTAs, so the band declaration over the pools compiles with the opt-in flag unset and is capture-safe like the four-band one; past 128 rows the band slab takes the GEMM engine’s OWN baseline tile for its shape — the kernel the graph route ran there (select_config at the part’s SM count: 128x224 in a 2x1 cluster at 2048 rows on a 212-SM part, 128x256 past 4096 rows), named by the driver so the slab rides the same JIT-only route: the prefill / extend / chunked forms of a band declaration compile flag-free and are capture-safe too. Both rules read the part’s SM count: the small-M route’s 128-row tile is its overflow branch, taken where the slab’s 218 64-row tiles exceed one wave — every served part (<= 212 SMs); a part with 218 or more SMs would keep the band slab on the graph route at decode row counts — and the baseline pick’s tile and cluster follow the SM count, so a recorded plan name replays within one SM-count class); a geometry without the band keeps exactly its four-band tuples. Enumerate geometry.qkvg_blocks, never ProjBlock itself.
  • the loader: qkvg_from_hf(..., index_qk_proj_weight=attn.index_qk_proj.weight) and build_fused_qkvg_weight(..., index_qk_proj_weight=) append the band (required iff declared, refused otherwise); qkvg_from_vllm takes it the same way, or appended to the merged weight’s rows (a serving stack that packs the indexer projection into its QKV GEMM).
  • the UNFUSED projection writes the wider slab; index_k_raw_view(proj_slab, geometry, B, S) and GatedAttentionBlockFwd.index_k_raw(workspace) expose the raw indexer key (the band’s last index_kv_heads * index_head_dim columns, pre-norm, un-rotated — what a serving cache keeps) as a zero-copy [B, S, index_kv_heads, index_head_dim] view.
  • execute(..., block_ids=, block_lens=) (appended): block_ids [T, top_k] int32 (or [B, S, top_k]), per query the ids of its selected complete blocks (block b = tokens [4b, 4b + 4) of its own sequence), the valid prefix then -1; block_lens [T] int32 optional. Checked for form only (dtype, rank, shape, contiguity, device — never read on the host; an int64 list is refused, not converted); refused on a block declared without qsa.
  • the sparse attention core: stage (4) of a block declared with qsa is the index-list sparse d256 forward (the Rubin kernel behind the standalone adapter cudnn.sdpa.fwd.sparse_gqa_sm107), at the same operands and strides as the dense stage — the slab’s Q / K / V column slices in place, or the compact buffers with inplace_qkv=False — and the same launch count; get_workspace_size() is the dense block’s (the core needs no scratch). What a query attends to, on device: the keys of its listed complete blocks that lie inside its causal range, plus the open tail block of its visible range (always), under the per-batch seq_lens when given. The list’s count per query is min(top_k, floor((pos + 1) / 4)), clamped by block_lens[t] when given; a -1 at an index below the count contributes no key (a masked block — its bytes are still fetched, so short lists should pass block_lens); entries at or beyond the count are never read. A row with no visible key (seq_lens[b] == 0) is out = 0, LSE = -inf exactly. With the full list — every complete block, which fits up to identity_bound visible tokens — the block computes dense causal attention: the same function within the suite’s budget, summed in another order (not bitwise). Served: bf16 / f16, d_head == 256, h_q // h_kv <= 16, dense [B, S, d_model] with or without seq_lens, return_lse on or off, fuse_norm_rope (without the band), inplace_qkv on or off, the indexer band on the unfused projection, paged_kv_page_size write-through; validated on Rubin (SM107) at the 24/2, 12/1, 6/1 and 32/2 geometries against the fp32 QSA oracle.
  • packed sequences (thd=True, num_sequences=B, max_seq_len=S_max, cu_seqlens=, “Packed sequences (THD)” below): the sparse core’s packed arm over the same [1, T, H, D] views. execute(seq_lens=) (REQUIRED: the [B] int32 lengths or the [B+1] int32 prefix sums) serves the Q and the KV side alike, and block_ids [T, top_k] carries each token’s blocks numbered FROM ITS OWN SEQUENCE’S FIRST TOKEN (block j of sequence b = tokens [4j, 4j + 4) of that sequence; the count rule and the open tail are per sequence, positions restarting at every sequence like the RoPE tables). get_workspace_size() grows by the packed SDPA’s small metadata scratch. A one-sequence packing is bitwise the dense B = 1, S = T sparse block; an empty sequence owns no rows. Validated on Rubin per sequence against the per-sequence fp32 QSA oracle (bf16 and f16). fuse_gate=True composes with thd=True (the sparse core’s epilogue gate on the packed [1, T, H_q, D] GATE slice, read in place like O; validated on Rubin: the gated packed block within one rounding of the activation dtype of the unfused packed block on O, the LSE bitwise). Not under thd: the in-block indexer (the scorer reads dense [B, S] prompts) — a typed decline naming the feature. The paged write-through composes with the packed sparse block (the stage addresses tokens; the packed write-through cell of the end-to-end module), and so does the paged READ since 2026-10-30: kv_source="paged" on a packed QsaSpec declaration is the packed block-sparse EXTEND form — one execute for a mixed serving step of decode rows and prefill chunks behind their cached prefixes (“The packed block-sparse EXTEND form” under “Serving: the extend form” below).
  • the in-block indexer: a block declared QsaSpec(index_source="indexer", index_band=True) derives the selection itself from the band — execute(..., w_iq_norm=, w_ik_norm=) instead of block_ids= — and hands it back through the optional block_ids_out / block_lens_out / index_k_compressed; the contract is “The in-block indexer” in the serving section below — and, over the pools (kv_source="paged", the block-sparse DECODE mode), its DECODE step: “The in-block indexer’s DECODE step over the pools” there (2026-10-21). bf16 only (the scorer’s dtype); validated on Rubin against the two-stage oracle (the selection under the margin rule, attention exact given the kernel’s selection).

Every claim of the sparse core — the dtypes, the head dim, the block size, the top_k range, the GQA group, the arms it does not carry — is read off the adapter’s capabilities record (cudnn.sdpa.fwd.sparse_gqa_sm107.SPARSE_CAPABILITIES), which the adapter’s own check_support enforces; the block transcribes none of it. Typed declines at declaration, naming the feature: save_for_backward (sparse training is out of scope; GatedAttentionBlockBwd refuses a qsa geometry), quant, fuse_norm_rope together with index_band (the fused projection renders 256-column tiles; 640 is not a whole number of them — the unfused projection serves the band), causal_bottom_right, an fp32 activation, h_q // h_kv > 16, d_head != 256 (thd together with paged_kv_page_size is served: the write-through addresses tokens, so the packed sparse block builds the same stage); under index_source="indexer" also an f16 activation (with or without the fp8 indexer cache), an index_head_dim other than the scorer’s 128, an index_heads outside the scorer’s packed head groups (4, 8, 16, 32, 64) and thd; index_cache_dtype other than None / bf16 / e4m3 (e5m2 named), or e4m3 under index_source="caller", from QsaSpec.validate(); at execute an index_k_compressed of the other dtype (the message names the declared one); and at check_support, a CuTe DSL below the floor the Rubin target needs (the public 4.8.0 wheel), named with the installed version, before the kernel module is imported. Under index_source="indexer" the floor is checked at declaration as well — the scorer’s head groups are read off the DSA tree, whose package imports its kernels, so a too-old DSL declines from GatedAttentionBlockFwd(...) before that import (and again from the stage’s check_support). fuse_gate=True is served: the sparse core stages each query’s GATE slice (the slab’s GATE columns, read in place) into its epilogue and writes O * sigmoid(GATE) in place of O after its dead-row select, so stage (5) disappears exactly as on the dense block; the gated O differs from the unfused block’s by at most one rounding of the activation dtype (the unfused path rounds O to bf16 / f16 before the gate), the LSE is bitwise the same.

Serving: index lists and paged KV

The contract a serving stack writes its shim against. It is ONE contract at the block’s two entry points — the block itself (GatedAttentionBlockFwd(..., geometry_with_qsa, paged_kv_page_size=P).execute(..., block_ids=, block_lens=, k_cache=, v_cache=, slot_mapping=, ...): the whole sub-layer from h to out) and the standalone sparse attention core behind its stage (4) (cudnn.sdpa.fwd.sparse_gqa_sm107.SparseGqaFwdDslSm107(q=, k=, v=, o=, lse=, block_ids=, block_lens=, seq_kv_lens=, top_k=): the attention call alone, BSHD operands at the strides its docstring admits) — the block hands the index tensors to the core unchanged. Everything below is checked for FORM on the host (dtype, rank, shape, contiguity, device); every VALUE (an id, a count, a slot, a length) is device data the kernels consume, so a stack’s tensors are handed over as they are: never read back, never converted, never reallocated, graph-capturable.

the stack hands informmeaning
block_idsint32 [T, top_k] with T = B x S_q (row b x S_q + s is query s of sequence b), or [B, S_q, top_k]; contiguous; int32 ONLY (an int64 list is refused, never converted)per query, the ids of its SELECTED COMPLETE 4-token blocks, RELATIVE to the query’s own sequence: block k is that sequence’s tokens [4k, 4k + 4) — never a page, a slot or a global token index; the valid prefix then -1; order unspecified (a top-k output)
block_lensint32 [T] / [B, S_q], contiguous; optionalthe valid-prefix length per query, consumed on device by the count rule below
QsaSpec.index_list_form (a declaration attribute)None / "shared" / "per_row"the decode mode’s list form: one step-0 list per sequence ([B, top_k], S_q <= 4) or one list per row ([B, S_q, top_k], any S_q); None = shared up to four rows, per row past them (caller lists) and per row under the in-block indexer over the pools
top_kQsaSpec.top_k: any multiple of 4 in [4, 512] (512 = the 2048-token budget over 4-token blocks, the default)the list width; it sizes the kernel’s per-query index staging, so a list is exactly top_k wide
seq_lens (block) / seq_kv_lens (core)int32 [B], optional and declared (seq_lens_present=True)the KV length of every sequence; a query sees min(pos + 1, seq_lens[b]) tokens; 0 = a dead sequence
paged_kv_page_size (P, a declaration attribute)a positive multiple of 16 — so 4 divides P and a 4-token block never straddles a pagethe write-through: pools k_cache / v_cache [num_pages, H_kv, P, D] (HND compact or NHD by strides) and slot_mapping [T] int32 / int64 (flat slot page x P + offset; a negative slot writes nothing) — the next section
block_table, kv_lensint32 (B, max_pages); int32 [B] = the logical KV length INCLUDING the step’s new tokensthe paged-READ inputs (decode, and chunked prefill over a cached prefix): REQUIRED under kv_source="paged" — the block-sparse DECODE / extend-with-prefix mode (below, and “Serving: the decode form”)
kv_cache_dtype (a declaration attribute)None (= the activation dtype: bf16 / f16 pools) or torch.float8_e4m3fn with per-tensor fp32 k_scale / v_scale at execute (1-element CUDA tensors, value = code x scale)the pool’s dtype: a bf16 Q over a bf16 pool, or a bf16 Q over an e4m3 pool with the attribute (numerics-changing, hence an attribute) — the quantizing write-through on the prefill forms, the gathered cast + the d256 decode tile on the block-sparse decode mode at S_q <= 4 (“Serving: the e4m3 KV cache” below); an fp8 Q is not a served form
return_lse -> lsefp32 [B, H_q, S_q], optionalmax + log(sum) in natural log over the query’s visible keys; -inf on a row with no visible key

The count rule — which entries of a row are read. For a query at position pos of its sequence (dense: the row index s; decode mode: kv_lens[b] - S_q + j, below) the kernel derives n_sel_default = min(top_k, floor((pos + 1) / 4)) — the complete blocks the query can see — and reads count = min(n_sel_default, clamp(block_lens[t], 0, n_sel_default)) entries when block_lens is given, n_sel_default entries otherwise. A -1 at an index BELOW the count is “no key” (nothing is attended, the bytes of a zero-filled block still cross the fabric — a stack whose lists are shorter than the derived default SHOULD pass block_lens); an entry at or beyond the count is NEVER read, valid-looking or not. A negative or oversized block_lens value is clamped, never faulted.

The open tail block is the kernel’s — never list it. The incomplete block that closes the query’s visible range n_vis = min(pos + 1, seq_lens[b]) — the n_vis % 4 tokens from 4 x floor(n_vis / 4) to n_vis - 1; for a query inside its sequence, the (pos + 1) % 4 tokens from 4 x floor((pos + 1) / 4) to pos — is always visible and derived on device from the position and the KV length; a list carries complete blocks only. A listed block lying entirely past the visible range (strictly past the open one) contributes nothing: its keys are masked one by one against the position and the KV length, never a fault. The open block itself, listed inside the count, is the violation below — attended twice.

A list is a SET of distinct complete-block ids; what happens on a violation. The contract: distinct ids, each a complete block of the query’s visible range (4k + 3 <= pos), the valid prefix then -1. The kernel does not check it (the ids are device data; a check would be a read-back) and does not deduplicate: an id listed twice, or the open block listed inside the count, is attended TWICE — the softmax over the multiset, finite and bounded, a wrong number, never a hang or a fault. The selections both serving stacks produce are top-k outputs over complete blocks, so no served list violates it; the test tree’s detector (block_ids_contract_violations in test/python/gated_attention_block/cutedsl/gated_block_qsa_reference.py) rejects both forms, and the kernel suite pins the twice-attended reading so a change of it is visible.

Dead rows. A query with no visible key (seq_lens[b] == 0; a decode row of an empty sequence) is out = 0, LSE = -inf exactly — a select on the emptiness predicate, never a scaled residue.

The identity below 2051 visible tokens, and why there is no dense route under caller lists. A query with n <= 2051 visible tokens has at most 512 complete blocks, so the FULL list makes the sparse core compute dense causal attention (the same function, summed in another order). The block still ALWAYS runs the sparse core under index_source="caller": it never reads a list, so it cannot know that a list is the identity, and a dense route would ignore the list — a different function, never a knob. Under index_source="indexer" the block DOES know: at S <= 2051 it launches no scorer and hands the sparse core the identity list as a plan-time constant (below); routing that case to the dense SDPA instead of the sparse core would be a legal performance knob there, and stays a follow-up.

The decode mode and the speculative (MTP) rows — served (2026-10-14): a dense [B, S_q <= 4] declaration on the QsaSpec geometry with kv_source="paged" (“Serving: the decode form” below). S_q <= 4 query rows per sequence (one new token, or 1 + drafts), attention over the pools through block_table / kv_lens. ONE list per sequence, block_ids [B, top_k] (+ block_lens [B]), selected at the STEP-0 position pos_0 = kv_lens[b] - S_q — never replicated per draft row. The visible set of row j at pos_j = pos_0 + j is DEFINED as (i) the listed blocks, complete at step 0, inside its causal range, plus (ii) EVERY token from the step-0 tail start 4 x floor((pos_0 + 1) / 4) to pos_j — so the block completed between pos_0 and pos_j, which the step-0 list cannot hold, stays visible, the row’s own token included. vLLM’s index reuse applies the step-0 list with each ROW’s own tail (tail_start = 4 x floor((pos_j + 1) / 4)), which at pos_j = 4k + 3 has an empty tail and hides block k from that row: a strict subset of this set; both are approximations of a per-row re-selection. The reference oracle takes the step-0 position explicitly (pos0); passing None yields the per-row-tail reading, so a shim that needs that parity can pin it. Chunked prefill over a cached prefix is the same paged-READ mode with one list per new row ([T, top_k]) at positions prefix_len + s: a dense [B, S_q > 4] declaration on the QsaSpec geometry with kv_source="paged" takes block_ids [B, S_q, top_k] / [T, top_k] (the list form is QsaSpec.index_list_form: None derives it from S_q at declaration — one per sequence up to four rows, one per row past it —, "per_row" declares one list per row at ANY S_q, the step’s rows at 1 .. 4 included — the serving stacks’ per-row selection, and the form the in-block indexer’s decode step over the pools binds —, "shared" the step-0 list at S_q <= 4 by name), at any S_q.

The indexer fields. QsaSpec.index_heads, index_kv_heads, index_head_dim, index_norm_eps size the fifth band of W_qkvg — and index_kv_heads x index_head_dim the raw indexer key’s write-through pool index_k_raw (next section); under index_source="caller" no kernel scores with them, under index_source="indexer" they are the scorer’s.

The in-block indexer (QsaSpec(index_source="indexer", index_band=True), bf16). The block derives the selection from its own fifth band — five launches at T <= 4096 (two compress launches, the scorer GEMM, the radix top-k, the row sort; one more sort window per further 4096 rows; two casts more under the fp8 indexer cache, below), no kernel of its own: (1) the band’s indexer QUERIES, RMSNormed with w_iq_norm (index_norm_eps) and partially rotated at the token’s position — the block-compress kernel at pool 1 over the slab’s query columns, bitwise the block’s own norm + RoPE chain on those rows (one fp32 pass, one rounding); (2) the COMPRESSED KEYS — the band’s raw key head mean-pooled in fp32 per complete 4-token block, RMSNormed with w_ik_norm and rotated at the block START position 4j, one rounding — the same kernel over the slab’s raw-key slice; (3) the SELECTION — the indexer scorer sum_h relu(q_h . kbar_j) / sqrt(index_head_dim) over the blocks below floor((pos + 1) / 4) and the fused radix top-k (ties to the smaller id) — cudnn.ops.qsa_select_forward, the DSA compressed-logits path, so the dense [T, n_blocks] score tensor is never materialized; (4) the CANONICAL ORDER — the top-k’s set is reproducible but its slot order is not, and the core accumulates in list order, so each row is put in the block’s ONE canonical order (the selected ids DESCENDING, the -1 padding last — since 2026-10-29 by the FROST row canonicalizer cudnn_block_qsa_canonical_list in place on the list, no torch.sort and no allocation; the identity list below the bound is descending too) before the core: two executes of the block are bitwise, and the list a caller gets back is canonical on EVERY form of the block (the prefill form, the decode step, the extend step) — the standalone paged selector’s ASCENDING order is that function’s own contract. The list is [T, top_k] int32 with exactly the derived count min(top_k, floor((pos + 1) / 4)) of valid ids then -1, so the sparse core takes it without block_lens.

the stack hands in / gets backformmeaning
w_iq_norm, w_ik_norm[index_head_dim] activation dtype, contiguous, 16-byte-aligned base; REQUIREDthe indexer’s two RMSNorm weights in the pre-folded (1 + w) form the attention’s w_q_norm / w_k_norm use (the loader’s convention)
block_ids / block_lens—REFUSED: the block derives the selection
block_ids_outint32 [T, top_k] or [B, S_q, top_k], contiguous, 16-byte-aligned base; optionalthe block’s OWN selection in canonical order (ids descending, then -1), written in place (the step-0 list a speculative-decoding caller reuses for the drafted rows)
block_lens_outint32 [T] or [B, S_q], contiguous; optionalthe valid-prefix length of every row of that list, min(top_k, floor((pos + 1) / 4)) — the position rule, padding rows included
index_k_compressedthe indexer cache dtype (QsaSpec.index_cache_dtype: the activation dtype by default, torch.float8_e4m3fn under the fp8 indexer cache) [B, >= floor(S / 4), index_head_dim], contiguous, 16-byte-aligned base; optionalthe compressed-key cache: the compress writes rows [0, floor(S_b / 4)) of every sequence (its complete blocks) and never touches the rest, so a cache sized for a longer context keeps its later rows

A storage-offset view of the two norm weights, the cache or block_ids_out that starts a few bytes off a 16-byte boundary is refused by name at execute, before any stage runs (the compress and cast kernels move those buffers in 16-byte vectors; the list is the sparse core’s TMA / bulk-copy base) — the rule the caller-list form applies to block_ids.

The RoPE tables are the attention’s own. The compress reads cos / sin at row 4j for block j (every block start is a position of the prefill) and the query norm at row pos: no appended block-start table and no position ids. A decode step’s per-token tables do not reach a block start; its compress (the step form of the same kernel) states its own table-extent rule.

The identity below 2051 visible tokens. At S <= QsaSpec.identity_bound every query’s complete blocks fit the list, so the selection is the identity: the block launches NO scorer and NO query norm, hands the sparse core a plan-time constant list (ids[t, j] = j below the row’s count, -1 past it; copied into block_ids_out on request) and reserves no indexer workspace; the key compress runs only when index_k_compressed is given (a serving cache wants the prompt’s compressed keys whatever its length). Past the bound the block is BITWISE the caller-list block fed its own block_ids_out.

Padding rows and seq_lens. A row at or past its sequence’s KV length is scored like every other row (the position rule); the compressed-key rows past the sequence’s complete blocks are zero-filled in the block’s workspace per execute, so such a row’s extra candidates score 0 — a positive visible block always wins, a zero-score tie goes to the smaller (visible) id — and the sparse core masks every key past the length anyway. The caller’s index_k_compressed is not zero-filled (its rows past a sequence’s blocks are the caller’s); a padding row’s list may name blocks past its range, which attention never reads.

The fp8 indexer cache (QsaSpec(index_source="indexer", index_band=True, index_cache_dtype=torch.float8_e4m3fn)). The compressed keys and the indexer queries as plain e4m3 of the RMSNormed + rotated values (no scales — the serving stacks’ fp8 indexer cache recipe), scored through cudnn.ops.qsa_select_forward’s e4m3 arm: the DSA MXFP8 unified scorer with all-ones E8M0 scale blobs, i.e. sum_h relu(e4m3(q_h) . e4m3(kbar_j)) / sqrt(128). No new kernel: the two compress launches keep writing bf16 (the one-rounding chain) and the FP8 pipeline’s per-tensor cast (sat_e4m3(row x 1.0) — round-to-nearest-even, saturating at 448, never a NaN) writes the e4m3 operands into two workspace slots appended after the indexer’s seven (every layout without the attribute is byte-identical); the caller’s index_k_compressed is then an e4m3 buffer written by one more cast over the floor(S / 4) rows of every sequence — under seq_lens the rows [floor(S_b / 4), floor(S / 4)) of a padded sequence receive ZEROS (the bf16 form leaves them untouched); rows at or past floor(S / 4) stay the caller’s in both forms. Below the identity bound only the cache launches run (the bf16 compress into a staging slot, the cast into the cache: two launches where the bf16 form runs one); past it the census is the bf16 indexer’s plus two casts (plus the cache cast). The selection is a different FUNCTION of the inputs (3 mantissa bits: on synthetic activations the e4m3 selection agrees with the bf16 one on 99.0 % of the blocks at S = 4096, 97.4 % at 32768 and 96.5 % at 131072, every flipped block a rank-boundary flip within ~5 % of the row’s top score; bf16 vs fp32 agrees at 99.8 %), so the dtype is a declaration attribute, never a knob, bf16 the default and the reference; the model-quality question of serving it is the caller’s. The e4m3 block’s selection is pinned against the top-k of the fp64 scores of its own e4m3 operands (exact up to ties), against the fp32-operand oracle under the margin rule, and against the bf16 block on the same inputs (the agreement reported, every flip within 8 % of the row’s top score); its first execute captures and replays bitwise (test_block_qsa_end_to_end.py). The compress kernel’s e4m3 store arm would fold the prefill form’s casts away (a lever, not built: the padded-row zero-fill contract decides).

At decode over the pools (2026-10-21). The in-block indexer’s DECODE step (“The in-block indexer’s DECODE step over the pools” below) serves the attribute WITHOUT casts: the fused prepare’s query + compress arms, the step form and the pool-1 compress STORE e4m3 directly — each row is the bf16 form’s packed words rounded once more by cvt.rn.satfinite.e4m3x2.f32 (round-to-nearest-even, saturating at +-448, never a NaN), i.e. the e4m3 cache is the e4m3 cast of the bf16 row — plain e4m3 at unit scale, no scale field anywhere; the ring stays bf16 (a bit copy). That is the serving stacks’ indexer-cache chain and the prefill form’s own convention, so a paged cache a stack primes from the prefill form’s e4m3 index_k_compressed rows and the step appends is ONE chain, bitwise (the multi-step cell). The paged scorer reads the e4m3 pair on its fp8 mma.sync.m16n8k32 arm (128-B rows, 4 k-steps, half the shared-memory ring; every e4m3 x e4m3 product exact in fp32). sample_index_kv_cache / index_kv_cache are REQUIRED in the declared dtype (refused both ways, at declaration and at execute); the ix_q workspace slot is carved at 1 B per element (every bf16 carve byte-identical); the launch census is the bf16 form’s (8 fused / 11 unfused; no cast kernel). The oracle is EXACT: the step’s set == the fp64 top-k of the kernel’s own e4m3 (ix_q, kbar) up to ties, the cache rows bitwise the bf16 twin’s cast, fused == unfused, two launches / the first-execute capture bitwise, the multi-step uniform cache; the bf16-vs-e4m3 agreement is a statistic (every flipped block at the rank boundary). The envelope publishes decode.index_cache_dtypes == (bf16, e4m3); the e4m3 MAIN pool together with the indexer stays declined (indexer_e4m3_pool). Performance (measured 2026-10-22/23: one exclusive Rubin cc 10.7 GPU, 212 SMs at the locked clock, every arm in one process per cell, A/B/A CUDA-graph replay; speed-ups positive = faster): the lever of the e4m3 cache over the bf16 cache is the scorer’s bytes — +10.8 % (7.4 us) of the decode step at B 32 x 128K x S_q 1 (the paged scorer 23.8 -> 16.8 us), +1.6 % geomean over 13 page-16 cells, +4.6 % over the B 32 cells, within +-1 % at B <= 4 except the two B 1 x 8K cells (-1.1 / -1.0 %: the e4m3 path’s fixed cost in the small prepare / score / top-k kernels at a 0.26 MB cache, where halving the bytes buys nothing); the block over the e4m3 cache against the serving baseline’s own fp8 indexer path +7.5 % geomean over those 13 cells; the selection exact up to ties and the cache rows bitwise the bf16 twin’s cast on every measured cell. The e4m3 column is a different function in every table; bf16 stays the default and the reference.

What it costs. Past the bound the workspace grows by the indexer’s queries (T x index_heads x index_head_dim x 2 B), the compressed keys (B x S / 4 x index_head_dim x 2 B), the list, its sorted copy and its scores (T x top_k x 12 B), the row sort’s index scratch and the scorer’s compact-logits scratch, ~ B x S^2 / 8 fp32 (512 MiB per 32K-token sequence): the price of one launch pair that never materializes the dense score tensor; a row-windowed selection bounds it and is the lever for longer prompts. compile() warms the scorer and the top-k at the real shape (their compile keys carry it), so execute compiles nothing and allocates nothing of its own on either indexer-cache dtype (the e4m3 scorer’s padded scale prefix is a plan constant); the one allocator transient on the path is torch’s: the row sort’s 4 KB index template per 4096-row window (torch.sort builds it even into preallocated outputs; an in-kernel row sort would remove it). The plan constants (the query prefix, the scratch offsets, the counts, the identity list, the casts’ scale word) are built on the host at compile() and uploaded on the launch stream by the first execute() after it, never written at compile() on the ambient stream; a CUDA graph that captures a block’s FIRST execute records those uploads with their values (constants: a replay re-writes the same ones), so warm the block up on the capture stream before capturing, as every capture of the block does. qsa_select_forward itself creates its cached constants (the constant head weights, the packed-K row offsets, the e4m3 arm’s unit scale blobs) on the launch stream of the first call that needs them and orders a later call on another stream behind that fill through an event recorded at the creation (not inside a capture, where no such event can be waited on — the warm-up rule above covers it); the head weights and the unit scale blobs are bucketed by power-of-two capacity (one flat allocation per bucket, the scorer handed a view in the exact packed shape), so a serving process’s distinct geometries keep O(log n) such constants resident, never one per shape.

What the suite asserts (test_block_qsa_end_to_end.py, the two-stage oracle): the selection as a SET against the top-k of the scores recomputed in fp64 from the kernel’s own bf16 queries and compressed keys up to ties; against the fp32-operand oracle under the MARGIN RULE — a block in exactly one of the two sets is accepted iff its oracle score lies within tol of the row’s k-th oracle score, tol the measured perturbation of the scores by the bf16 rounding of the operands (not a tuned number); the same against the HF-rounding oracle (the pooled key cast to bf16 before its norm); the queries and compressed keys within one bf16 ulp of the once-rounded fp32 chain; the count rule on every row; and attention EXACT given the kernel’s selection — the block oracle on the kernel’s list within the dense suites’ budgets.

The paged decode scorer (standalone). The first piece of that decode-step indexer ships on its own: cudnn.ops.qsa_select_decode_paged_forward(qi, kbar, block_table, kv_lens, seq_q=1, top_k=512) scores a decode step’s seq_q indexer query rows per sequence (qi [B x seq_q, index_heads, 128] bf16) against the compressed keys of the complete 4-token blocks each row can see, read from the PAGED compressed-key cache the serving stacks keep on the main block table (kbar [num_pages, page_size / 4, 1, 128] bf16: block j of sequence b is row j % (page_size / 4) of page block_table[b, j // (page_size / 4)]; a power-of-two page size of at least 8), and selects the top_k per row with the deterministic radix top-k. Row s of sequence b sits at position kv_lens[b] - seq_q + s and sees counts = clamp(floor((pos + 1) / 4), 0, n_blocks_max) complete blocks; the scorer kernel writes logits[t, 0:counts[t]] (scale x sum_h relu(qi[t, h] . kbar[b, j]) in fp32: the ReLU-summed, head-reduced indexer score) and the counts on device — no torch prologue — and the top-k returns block_ids [T, top_k] int32 in ASCENDING order with -1 past block_lens = min(top_k, counts) (ties to the smaller id), so two calls are bitwise — the standalone selector’s own contract (the gated attention BLOCK’s lists are DESCENDING on every form: its one canonical order). A -1 table entry under a visible block reads as a zero key (logit 0); a padding sequence (kv_lens == 0) writes nothing but its counts (0) and never reads the cache. The scorer kernel alone (kernels.qsa_score_paged.compile_qsa_score_paged / run_qsa_score_paged: one warp-level mma.sync.m16n8k16 per 8 keys x 16 D over the seq_q x index_heads query rows padded to 16 — the query tile staged once per CTA in shared memory behind one CTA barrier, the keys streamed by cp.async through a per-warp ring into the B fragments — with an L2 evict-first hint on those copies where the installed DSL’s bundled CUDA compiler is 13.4 or later, i.e. the cu13 toolchain of the 4.8.0 wheel, which the DSL binds only under a CUDA 13 driver (l2_hint, a compile key of the kernel-level entry: None = that rule; the 13.3 PTX assembler of the public 4.7.0 cu13 wheel mis-encodes the hinted copy on sm_90 / sm_100 and the 12.9 assembler of the cu12 flavour — what any wheel, 4.8.0 included, binds under a CUDA 12 driver — emits the same encoding, so below 13.4 the plain copy is traced — the same bytes, bitwise the same logits; True forced on such a DSL is a typed decline naming the compiler it found, the wheel and the remedy, the cu13 flavour of 4.8.0 or later under a CUDA 13 driver; False the plain copy on any DSL); the programmatic-dependent-launch pair under use_pdl) runs on any CuTe DSL device from sm_80; the paired selector needs the radix top-k’s cc 9.0. The e4m3 pair (qi and kbar float8_e4m3fn: the fp8 indexer cache) is served since 2026-10-21 by the scorer’s e4m3 READ arm — mma.sync.m16n8k32 .e4m3 on 128-B rows, sm_89+, a different selection function chosen by the operands’ dtype; a mixed pair is a form error; units_per_warp (1 / 2 / 4 / 8 units of 32 compressed rows per warp, default 1) and use_pdl are performance knobs of the entry, ring_depth (2 / 4 / 8 n-tiles of keys in flight per warp, default 4) and l2_hint of the kernel-level compile — each the same function. The entry’s module and its helpers import no CuTe DSL: the DSL floor, the sm_107a target on cc 10.7 and the radix top-k’s cc 9.0 are typed NotImplementedErrors raised BEFORE the kernel module is imported, so the import path resolves on the public 4.6.2 wheel too; stream takes a torch.cuda.Stream, a raw stream handle or None (torch’s current stream), and every torch operation of the call runs on it. The in-block binding of this scorer is the decode indexer’s step (“The in-block indexer’s DECODE step over the pools” below, 2026-10-21); the dense-cache form of this selection is a follow-up.

The CuTe DSL floor. The Rubin sparse core needs the public nvidia-cutlass-dsl 4.8.0 wheel (the sm_107a target); below it check_support() raises a typed NotImplementedError that names the installed version, BEFORE any kernel module is imported — never a KeyError from inside the DSL. An index_source="indexer" block raises the same decline already at declaration (the declines paragraph above).

A prefill step with write-through into the stack’s pools, the stack’s own lists:

import torch
from cudnn.gated_attention_block import GatedAttentionBlockFwd, GatedAttentionBlockGeometry, QsaSpec
geometry = GatedAttentionBlockGeometry(d_model=2560, h_q=24, h_kv=2, d_head=256, rope_dim=64, qsa=QsaSpec(top_k=512))
blk = GatedAttentionBlockFwd(h, w_qkvg, w_q_norm, w_k_norm, cos, sin, w_o, out, geometry,
return_lse=True, seq_lens_present=True, paged_kv_page_size=16)
blk.check_support()
blk.compile()
workspace = torch.empty(blk.get_workspace_size(), dtype=torch.uint8, device=h.device)
blk.execute(h, w_qkvg, w_q_norm, w_k_norm, cos, sin, w_o, out, workspace,
seq_lens=seq_lens, # [B] int32: the KV length of every sequence (padding below S)
lse=lse, # [B, H_q, S] fp32, natural log
block_ids=block_ids, block_lens=block_lens, # [B * S, 512] int32 sequence-relative block ids, -1 padded; [B * S] int32
k_cache=k_cache, v_cache=v_cache, # [num_pages, 2, 16, 256] pools in h's dtype (HND; NHD by strides)
slot_mapping=slot_mapping) # [B * S] int32 / int64 flat slots (page * 16 + offset); negative = no write

How a stack’s vocabulary maps onto it (the shim is a renaming, not a conversion):

the stack hasthe block / core takes
the top-k output over compressed blocks — vLLM’s block_indices [num_tokens, 512] BEFORE its block-to-token expansion, SGLang’s topk_indices [rows, 512] when its backend keeps block indicesblock_ids, as is (int32, contiguous); no expansion to token ids
the stack’s own indexer chain (the index_qk_proj GEMM, the compress, the scorer, the top-k)QsaSpec(index_source="indexer"): the block’s — the band in the one projection GEMM, two compress launches, the scorer pair, the row sort; block_ids_out / block_lens_out hand the list back, index_k_compressed fills the stack’s compressed-key cache
the per-row count — vLLM’s min(visible_blocks, 512) (the expand kernel’s complete_blocks), SGLang’s visible-block count of its row rangesblock_lens (optional; pass it whenever a list is shorter than the derived default)
vLLM’s packed [num_tokens, 2052] buffer of token ids with the trailing count columnnot a served form (the token-id arm is a follow-up) — keep the block-id output of the top-k
slot_mapping / out_cache_locslot_mapping (flat slot = page x P + offset; negative = no write)
the page table at the stack’s page granularity, seq_lensblock_table, kv_lens (the paged-READ mode: kv_source="paged", the pools and the table as declaration samples — the table CONTIGUOUS)
k_scale / v_scale (the per-layer DEscales of an fp8 KV cache; device tensors)kv_cache_dtype=torch.float8_e4m3fn at declaration, execute(k_scale=, v_scale=) as 1-element fp32 CUDA tensors (read on device; a host float is refused)
the open tail, the causal mask, the per-row visible_blocks clampnothing — the kernel derives all three from the position and the KV length
the output gatenothing — stage (5) is the block’s (sigmoid(gate) after the dead-row select); a split merge is internal

The decode top-k kernel (standalone today). cudnn.gated_attention_block.qsa_topk.qsa_topk_decode(logits, counts, block_ids, *, cluster_ctas=None, threads_per_cta=None, use_pdl=False, stream=None) — the DSL-free host entry: the kernel module kernels/qsa_topk_decode.py behind it imports the CuTe DSL at module scope, so the entry gates the DSL version FIRST and imports the kernel module lazily (a DSL below the floor reads as a typed NotImplementedError naming the version, never as an error from inside the DSL; qsa_topk_decode_kernel_module() hands the gated module, with its plan-time / per-execute split, to an engine) — selects, per query row, the top-k block ids of the decode scorer’s fp32 logits: logits [T, n_blocks_max] fp32 (row t scored in columns [0, counts[t]); the columns past them never influence the result — the kernel may read them, so they must be addressable, which they are inside the tensor), counts [T] int32 (the VISIBLE-block count per row, clamped in-kernel to [0, n_blocks_max]; a row with more than top_k visible blocks gets the score-based top-k, a row with at most top_k the identity list of every visible block — so never pass the list length min(top_k, n_vis) as the count: every row with more visible blocks than top_k would then read as exactly top_k, take the identity path and return its oldest top_k blocks instead of a selection, with no error), block_ids [T, top_k] int32 — the CALLER’s buffer, nothing is allocated — written in CANONICAL order: the min(top_k, counts[t]) selected ids strictly descending, then -1. The set is the exact top-k of the fp32 values (every selected logit at or above every unselected one; a tie at the k-th value keeps the smaller ids), so two launches are bitwise and the list is the form the sparse core’s list stage consumes without a sort. One thread block per row (threads_per_cta 512 / 1024), or the row split over a thread-block cluster of cluster_ctas 4 / 8 / 16 CTAs (1024 threads each; the histogram merged cluster-wide by st.async + mbarrier, no release-ordered cluster barrier): the row’s keys stay in registers, ONE 4096-bin histogram over the eight octaves below the row maximum finds the k-th bin, its candidates are ranked exactly (an all-pairs rank up to 128 of them, narrowing passes beyond), and a per-warp ballot compaction writes the canonical list. Both knobs are performance-only (every served value is the same function, pinned bitwise by the tests; no performance number is quoted until one is measured on an exclusive GPU). Typed declines: a CuTe DSL below the floor or without the device’s target (the version / target named), use_pdl=True below cc 9.0 (griddepcontrol exists from Hopper), a per-CTA shared-memory need above the device’s opt-in limit (the bytes named: smem_bytes_for(cols_cap, cluster_ctas, threads_per_cta, top_k) — the 32 KB histogram plus a pool sized by the fast path’s staging regions (threads / 32 x (top_k + 128) words, at most 96 KB) or the fallback’s list and ballot words, about 114 KB at 32768 columns x 1024 threads x top_k 512, above the 99 KB opt-in limit of some parts, which serve 512 threads); cluster_ctas > 1 below cc 9.0 (clusters and DSMEM exist from Hopper) or on a public CuTe DSL wheel below 4.8.0 (the cluster exchanges’ vector st.async stores fail to compile on the released 4.7.0; the single-CTA form serves and the convenience entry picks it there). The declines are read in this order: the DSL floor, the geometry, the device’s target, the device gates (clusters / DSMEM and griddepcontrol, cc 9.0), the cluster form’s DSL floor, the shared-memory limit — a device gate before a form’s DSL gate, so a part below cc 9.0 is told what it lacks whatever wheel it runs; ValueError on any malformed form — a cluster size outside 1 / 4 / 8 / 16, a thread count outside the domain (1024 for the cluster form), more than 64 keys per thread, a row block of more than 65535 rows. The in-block decode indexer binds it over the paged compressed-key cache (“The in-block indexer’s DECODE step over the pools” below, 2026-10-21); the dense standalone selector keeps the radix top-k it binds today.

Serving: write-through into a paged KV cache (paged_kv_page_size)

A serving stack keeps every token’s post-norm, post-RoPE K and its V in a paged cache. GatedAttentionBlockFwd(..., paged_kv_page_size=P) (appended; 0 = off, the default) declares that write-through: after norm + RoPE the block writes each token’s K and V row into the caller’s pools at execute(slot_mapping=) — ONE extra launch (two with the indexer band’s raw-key pool), or NONE with the knob fuse_prepare=True, which runs norm + RoPE and the write-through as a single launch that is bitwise the two-stage form — and then attends over its own K / V exactly as before, so out is bitwise the block’s without it and get_workspace_size() does not change. It is a declaration ATTRIBUTE (it changes the input contract), never a knob (fuse_prepare is the knob: it changes no input and no output, only the launch count). Dense [B, S, d_model] and packed (thd=True) blocks alike: the stage addresses tokens, so the packed block (B = 1, S = T inside) runs the same launch over the same [T, H_kv, D] views.

input (appended to execute)shape / dtypecontract
k_cache, v_cache[num_pages, H_kv, P, D] views in the activation dtype (bf16 / f16), D contiguous, the other three strides FREE: HND compact (H*P*D, P*D, D, 1); NHD storage by strides (P*H*D, D, H*D, 1); or a stack’s UNIFIED content row [num_pages, H_kv, P, 2D] with K at +0 and V at +D (token stride 2D) — kv_cache[..., :D] and kv_cache[..., D:] are the two views. Every non-innermost stride a whole number of 16-byte vectors and a 16-byte-aligned base (a typed ValueError otherwise: each row is written in 16-byte vectors)REQUIRED under paged_kv_page_size > 0; post-RoPE K lands in k_cache, V in v_cache; nothing is repacked
slot_mapping[T] int32 or int64, contiguous; the flat slot page * P + offset of every token (T = B*S, or the packed token count)REQUIRED; a NEGATIVE slot (a padded token) writes nothing; a slot past the pool’s capacity writes nothing either — never a write past the pool; slot VALUES are device data, never read on the host
index_k_raw[num_pages, P, index_head_dim] activation dtype, D contiguous: the raw indexer key’s pool, written at the same slotsREQUIRED iff the geometry declares the indexer band (QsaSpec.index_band), refused otherwise
block_table, kv_lensthe paged-READ modes’ page table and logical KV lengths (“Serving: the extend form” / “Serving: the decode form” below)REQUIRED under kv_source="paged"; refused (ValueError) on a block declared kv_source="self", which attends over its own K / V

A serving stack’s packed K|V cache — ONE tensor per layer, [num_pages, H_kv, P, 2D], holding K in columns [0, D) and V in [D, 2D) of every row (the HND cache of the 24 / 4 model family at d_head 256: [pages, 4, 16 or 32, 512] bf16) — IS the unified layout above and is bound without a copy as its two half VIEWS, k_cache = kv[..., :D] and v_cache = kv[..., D:]: both views carry the strides (H_kv*P*2D, P*2D, 2D, 1) — D contiguous, every other stride a whole number of 16-byte vectors (2D elements = 1024 B at d256), the V view’s base D elements past the K view’s — so the pool contract admits them and the two samples share one layout kind. The write-through lands K in the [0, D) half and V in the [D, 2D) half of a token’s row and touches no other byte of the tensor (the other half of a written row is a row of the OTHER pool), and the paged forms read the halves back. Pinned by one cell per form at the stack’s own shape — 24 / 4, pages 16 (one token per sequence) and 32 (a four-token speculative step), kv to 2048 with a padded and a short entry, int64 slots, bf16 and f16; out bitwise the HND-compact pools’, every other byte of the tensor untouched (the packed_kv_rows cells of test_block_serving_decode.py / test_block_serving_extend.py, L0 on every served part). Validated per form: the DECODE cells passed on cc 10.0 and on cc 10.7; the EXTEND cells passed on cc 10.0, and on cc 10.7 (a Rubin development node).

P is a positive multiple of 16 (the serving stacks’ page granularity; a 4-token sparse block then never straddles a page); a block declared without the attribute REFUSES every one of these inputs (ValueError) rather than ignoring a pool it would never write. fuse_cache_write (appended; None = auto): on the bf16 / f16 fused norm + RoPE pipeline without an indexer band the write-through is FOLDED into the fused projection’s epilogue — every K / V tile’s post-norm, post-RoPE row is stored to the pools from the same registers the slab store reads (the pool row is the slab row, bitwise the separate kernel; a negative slot or one past the pools writes nothing; no new barrier, no new SMEM), one launch fewer per step (EXTEND 3 stages with fuse_gate=True, DECODE 4); False keeps the separate cache-write launch (the control; on the DECODE form’s unfused chain it also keeps the AUTO small-M fold of fuse_cache_write_small_m=None off — an explicit fuse_cache_write_small_m names that arm itself and is followed either way); True on the unfused chain, a quantized pipeline or an indexer band is a typed NotImplementedError; blk.cache_write_fused says what runs. The slot dtype (int32 / int64) is a compile key of the fused arm as of the separate kernel: both are compiled. Served on the bf16 / f16 inference pipelines (inplace_qkv on or off, fuse_norm_rope on or off, fuse_gate on or off, dense or packed); validated on Rubin at pages 16 / 32 / 64 / 128 on all three pool layouts, dense and packed, bf16 and f16 (the live pool rows bitwise the K / V rows attention consumed; padded tokens unwritten). quant (the pools of a quantized cache arrive with a cache-dtype attribute and its cast kernel) and save_for_backward with paged_kv_page_size are typed NotImplementedErrors naming the feature — with ONE exception since 2026-10-30: the FP8-projection + bf16-attention mode (QuantSpec(attention_dtype=torch.bfloat16), “Serving: FP8 projections with bf16 attention” below) writes bf16 post-RoPE K / V into bf16 pools exactly as the bf16 block does and is served on the paged forms (the decline’s text names it as the way in); a CuTe DSL below FROST’s floor is declined at check_support, named with the installed version, before the kernel module is imported.

fuse_prepare=True needs the attribute (a ValueError without it), the in-place pipeline (inplace_qkv=True, a ValueError otherwise) and not fuse_norm_rope (a typed NotImplementedError: the norm + RoPE already live in the projection epilogue there); thd=True, kv_source="paged" on a DENSE geometry (the DECODE form) and kv_cache_dtype=torch.float8_e4m3fn together with it are typed NotImplementedErrors naming the knob (the prepare kernel’s packed arm and the dense DECODE form’s are follow-ups; the e4m3 cache’s write-through quantizes in its own stage; on the block-sparse DECODE mode over the pools the knob is SERVED — “The in-block indexer’s DECODE step over the pools” below); rope_dim=0 with the knob is a ValueError at check_support, with or without qk_norm, naming the zero-width RoPE table operands: the fused launch binds cos / sin as [T, rope_dim] operands and its artifact is traced with one-column tables, so a [T, 0] table is not an operand it takes. The fused stage takes use_pdl as the two stages it replaces do (the attribute and the kernel’s griddepcontrol pair). Its kernel (kernels/qsa_prepare.py) also carries the indexer’s decode-step arms — the queries’ norm + RoPE, the raw-key ring and the incremental compress of the compressed-key cache — which the block binds when its decode mode lands.

Forward

blk = GatedAttentionBlockFwd(
sample_h, sample_w_qkvg, sample_w_q_norm, sample_w_k_norm, sample_cos, sample_sin, sample_w_o, sample_out,
geometry,
return_lse=False, # also write the softmax stats [B, H_q, S] fp32
save_for_backward=False, # keep what GatedAttentionBlockBwd needs (implies return_lse)
seq_lens_present=False, # per-batch KV lengths (padding) at execute time
inplace_qkv=None, # None -> not save_for_backward: norm/RoPE Q,K in place in the projection slab
fuse_norm_rope=False, # stages (2)+(3) inside the projection epilogue
fuse_gate=False, # stage (5) inside the SDPA epilogue
fuse_cache_write=None, # serving: where the paged write-through runs -- None = folded into the fused projection's epilogue where that arm serves, False = its own launch (the AUTO small-M fold off too), True = the fused arm (at decode row counts the request keeps the fork) or a typed decline -- "Serving: write-through into a paged KV cache"
quant=None, # QuantSpec (FP8) | MxQuantSpec (MXFP8, + fp4 weights / fp4 O) | None (bf16 / fp16)
sample_h_sf=None, sample_w_qkvg_sf=None, # MXFP8 scale-factor blobs
sample_w_o_sf=None, # fp4 O only (MxQuantSpec.o_fp4): the e2m1 W_o's scale blob, in O's format
saved_gate_copy=False, # training save mode: True copies the GATE band into a compact saved.gate (see below)
thd=False, # PACKED sequences: h / cos / sin / out are [T, .] (or [1, T, .]) token matrices -- see "Packed sequences (THD)"
num_sequences=None, # THD only, REQUIRED there: B, the number of sequences in the packing
max_seq_len=None, # THD only, REQUIRED there: S_max, the longest sequence the plan admits
cu_seqlens=False, # THD only: execute(seq_lens=) is [B+1] int32 prefix sums instead of [B] int32 lengths
paged_kv_page_size=0, # serving: write the post-RoPE K / V through into paged pools at execute(slot_mapping=) -- see "Serving: write-through"
kv_source="self", # serving: "paged" = stage (4) attends over the POOLS via execute(block_table=, kv_lens=): the EXTEND form (packed) or the DECODE form (dense) -- "Serving: the extend form" / "Serving: the decode form"
token_capacity=None, # serving, THD only: the packed rows are a CAPACITY (lengths may sum to T_live <= T); must equal T when given
sample_k_cache=None, sample_v_cache=None, sample_block_table=None, # kv_source="paged" only: the pools' and the table's shape / dtype / strides (compiled in; any table row stride)
max_kv_len=None, # the DECODE form: the logical KV bound kv_lens[b] <= max_kv_len; None = max_pages * P
decode_split_kv=None, # the DECODE form: the decode tile's split-KV; None = the decode split model (fp32 partials in the workspace + one combine)
fuse_prepare=None, # serving: stages (2)+(3) and that write-through as ONE launch (needs paged_kv_page_size > 0; same function); None = False on the dense forms and the caller-list band form, the MEASURED default on the indexer's DECODE step over the pools (True on cc 10.7 since 2026-10-21 where the fused form is admissible, False out of place and on other parts) -- "The in-block indexer's DECODE step over the pools"
kv_cache_dtype=None, # serving: None = the activation dtype; torch.float8_e4m3fn = the e4m3 KV cache (per-tensor k_scale / v_scale at execute) -- "Serving: the e4m3 KV cache"
decode_route_policy=None, # the DECODE form: None = "single_unit" (the decode tile in one unit, else the packed prefill tile where the step fills the part); "token_units" = the adapter's route -- "Serving: the decode form"
decode_q_tile_max=None, # the DECODE form: None = the decode tile's whole width table (16, 32, 64); 32 = the route before the 64-column width landed (a performance-only cap)
use_pdl=None, # programmatic dependent launch over the chain: None = True on the DECODE form's decode-tile routes on cc 10.7 (the measured default), False everywhere else; True / False explicit -- "Serving: the decode form"
fuse_norm_rope_small_m=None, # the DECODE form: None = fuse_norm_rope=True LOWERS stage (1) to the unfused chain at B x S_q <= 128 (the measured faster step); True = the fused fork as declared
fuse_cache_write_small_m=None, # the DECODE form's unfused chain: None = the paged write-through folded into the norm + RoPE launch at B x S_q < 128 rows or S_q = 1 (the measured rule), its own launch past that and under an explicit fuse_cache_write=False; True / False explicit
sample_index_kv_cache=None, sample_index_state_cache=None, sample_index_state_block_table=None, # the in-block indexer's DECODE / EXTEND steps over the pools (QsaSpec(index_source="indexer") with kv_source="paged"), REQUIRED there, refused elsewhere: the paged compressed-key cache [n_pages, P / 4, 1, 128], the ring of raw indexer keys [blocks, cap, 1, 128] (cap >= S_q + 3 on the DECODE step, cap >= 3 on the EXTEND step), its [B, 1] int32 table -- "The in-block indexer's DECODE and EXTEND steps over the pools"
index_state_slot_mapping_present=False, # that step: execute(index_state_slot_mapping=) names the ring's write rows (block x cap + pos % cap, -1 dead; slot_mapping's dtype) instead of the rows the kernels derive from the ring table
index_commit_present=False, # that step: a per-sequence commit count at execute (a follow-up: True is a typed NotImplementedError)
rope_position_tables_only=False, # that step under fuse_prepare=True: cos / sin REFUSED at execute; the fused launch rotates Q / K and the indexer queries from cos_pos / sin_pos at the device-derived position (the unfused twin with it is a typed decline)
index_select_form="auto", # the indexer over the pools (2026-10-29): "auto" = the DECODE step inside the decode scorer's envelope (S_q x next_pow2(index_heads) <= 64) and the EXTEND step past it; "decode" / "extend" pin a form (performance-only) -- "The in-block indexer's DECODE and EXTEND steps over the pools"
)
workspace = torch.empty(blk.get_workspace_size(), dtype=torch.uint8, device=h.device)
blk.execute(h, w_qkvg, w_q_norm, w_k_norm, cos, sin, w_o, out, workspace,
seq_lens=None, # dense: the [B] int32 KV lengths iff seq_lens_present; THD: REQUIRED, the [B] lengths / [B+1] prefix sums; REFUSED on the DECODE form
lse=None, saved=None, current_stream=None, h_sf=None, w_qkvg_sf=None,
w_o_sf=None, # required iff MxQuantSpec.o_fp4, refused otherwise
k_cache=None, v_cache=None, slot_mapping=None, # REQUIRED iff paged_kv_page_size > 0, refused otherwise -- "Serving: write-through"
block_table=None, kv_lens=None, # REQUIRED iff kv_source="paged", refused otherwise -- "Serving: the extend form" / "Serving: the decode form"
num_tokens_live=None, # the bounded packed form's LIVE token count, a host int (None = the capacity) -- "Serving: the extend form"
k_scale=None, v_scale=None, # REQUIRED iff kv_cache_dtype=torch.float8_e4m3fn (1-element fp32 CUDA tensors), refused otherwise
w_iq_norm=None, w_ik_norm=None, # REQUIRED iff QsaSpec(index_source="indexer"), refused otherwise -- "The in-block indexer"
index_k_compressed=None, block_ids_out=None, block_lens_out=None, # the indexer's optional outputs: the PREFILL form's compressed-key rows (refused at the decode step: the cache is paged), the list and its counts
index_kv_cache=None, index_state_cache=None, index_state_block_table=None, # REQUIRED iff the indexer's DECODE / EXTEND step over the pools, refused otherwise (16-byte bases; the declared forms) -- "The in-block indexer's DECODE and EXTEND steps over the pools"
index_state_slot_mapping=None, # REQUIRED iff index_state_slot_mapping_present (slot_mapping's dtype; REQUIRED on the EXTEND step: its ring tail write takes the stack's slots), refused otherwise
cos_pos=None, sin_pos=None, # REQUIRED on that step: [1 or B, >= max_pages x P, rope_dim] position tables -- the table is the CEILING on positions; refused elsewhere
n_commit=None) # refused (index_commit_present=True is a follow-up)

Every appended argument (sample_w_o_sf, w_o_sf, saved_gate_copy, the four packing knobs thd / num_sequences / max_seq_len / cu_seqlens, the serving attributes paged_kv_page_size / kv_source / token_capacity / kv_cache_dtype with their samples, the DECODE form’s max_kv_len / decode_split_kv, the knob fuse_prepare, the execute inputs, num_tokens_live, k_scale / v_scale, the indexer’s five execute inputs and — last, 2026-10-21 — the decode indexer’s three samples + three flags and its seven execute inputs) sits at the end with a default, so positional callers of the bf16, FP8 and MXFP8 pipelines are unchanged; MxQuantSpec.o_fp4 and sample_w_o_sf must be given together (a typed ValueError names the missing half), and num_sequences / max_seq_len / cu_seqlens are refused without thd=True.

Training forward (save_for_backward=True)

The UNFUSED pipelines, out of place: bf16 / fp16, and the per-tensor FP8 (QuantSpec) and MXFP8 (MxQuantSpec) pipelines (inplace_qkv defaults to False there; fuse_norm_rope / fuse_gate are typed declines; the MXFP8 pipeline’s fp4 modes train too and write the MXFP8 record byte for byte — see “The fp4 weight modes’ backward”). The block writes through the caller-owned SavedForBackward record wherever the backward needs a tensor, with the same kernels as inference (out is bitwise the inference block’s): the projection GEMM writes saved.proj_slab, the SDPA writes the pre-gate saved.o and saved.lse, norm+RoPE writes saved.rstd_q / rstd_k, and the sigmoid gate lands out of place in the workspace. execute() still allocates nothing.

Under FP8 / MXFP8 the record is the SAME record the bf16 forward writes — a bf16 proj_slab (the dequantized stage-(1) product) whose Q/K bands are pre-norm, the bf16 pre-gate o, the exact fp32 lse, rstd_* — with h the caller’s e4m3 codes (h_sf is a forward input, never a record field). One routing differs from quantized inference: norm+RoPE writes the normed Q/K out of place into two compact bf16 workspace slots (+17 KiB/token at the 397B geometry) instead of back over the slab, and the quantize stages read those slots, so the slab keeps the pre-norm bands the backward differentiates; the launch count (9) and the bytes moved are unchanged, and out, o, lse and the GATE / V bands are bitwise the quantized inference forward’s. GatedAttentionBlockBwd (bf16 / fp16) consumes such a record given the dequantized bf16 h and weights (dataclasses.replace(saved, h=h_dequantized); a record handed through with its e4m3 h is a typed ValueError naming that contract and the quant=QuantSpec declaration); under quant=QuantSpec the backward differentiates the per-tensor fp8 record natively, as written, and under quant=MxQuantSpec the MXFP8 record (see Backward).

What that backward computes over a quantized record — the numerics contract. The record’s o and lse are the quantized SDPA’s: computed over the e4m3 q8 / k8 / v8 the forward quantized, with the kernel’s e4m3 P. The bf16 backward recomputes bf16 Q / K from the pre-norm slab bands (and reads the slab’s bf16 V), differentiates the bf16 chain through them, and recomputes P = exp(S - lse) from the bf16 scores against the quantized lse, so P no longer row-normalises exactly. Its gradients are therefore the bf16 chain’s gradients evaluated at the quantized forward’s o / lse — a straight-through-style approximation whose distance from the exact gradient of the dequantized bf16 model is of the order of the fp8 quantization error of Q / K / V / P — not a bf16-accurate gradient of the dequantized model; dW_o inherits the forward’s own o error on top (it contracts dy with the gated o the model actually produced). The test holds the result to the module’s bf16 bounds against an fp64 oracle seeded with the record’s o / lse (the exact function of the record) and reports its cosine against the unquantized fp64 chain.

from cudnn.gated_attention_block import SavedForBackward, saved_slab_views
B, S, T = h.shape[0], h.shape[1], h.shape[0] * h.shape[1]
g = geometry
blk = GatedAttentionBlockFwd(h, w_qkvg, w_q_norm, w_k_norm, cos, sin, w_o, out, g, save_for_backward=True,
seq_lens_present=seq_lens is not None) # saved_gate_copy=False: the proj_slab save mode
blk.check_support(); blk.compile()
workspace = torch.empty(blk.get_workspace_size(), dtype=torch.uint8, device=h.device) # no slab, no O: the record holds them
proj_slab = torch.empty(T, g.n_qkvg, dtype=h.dtype, device=h.device) # [B*S, n_qkvg] (or [B, S, n_qkvg]), contiguous
q_pre, gate, k_pre, v = saved_slab_views(proj_slab, g, B, S) # zero-copy [B, S, heads, D] views of its bands
saved = SavedForBackward(
h=h, # the SAME tensor execute() runs on (verified)
gate=gate, q_pre=q_pre, k_pre=k_pre, # the views above, or None (the backward derives them)
o=torch.empty(B, S, g.h_q, g.d_head, dtype=h.dtype, device=h.device), # PRE-gate O, compact
lse=torch.empty(B, g.h_q, S, dtype=torch.float32, device=h.device), # natural-log LSE; `lse=` at execute is optional
rstd_q=torch.empty(B, S, g.h_q, dtype=torch.float32, device=h.device), # both None iff geometry.qk_norm is False
rstd_k=torch.empty(B, S, g.h_kv, dtype=torch.float32, device=h.device),
proj_slab=proj_slab,
seq_lens=seq_lens, # the SAME tensor passed to execute(seq_lens=), or None
# seq_lens_form=None: a dense record (seq_lens is a padding mask or None); "lengths" / "prefix" under thd=True (below)
)
blk.execute(h, w_qkvg, w_q_norm, w_k_norm, cos, sin, w_o, out, workspace, seq_lens=seq_lens, saved=saved)

A runnable version of this recipe, self-checked against torch on the record alone (the slab is h @ W_qkvg^T, rstd_* are the RMSNorm statistics of the pre-norm bands, out is (o * sigmoid(gate)) @ W_o^T): samples/frost/gated_attention_block/00_training_forward.py.

Two save modes, chosen at declaration because the workspace carve differs (execute checks that the record agrees):

modedeclarationwhat is savedforward cost
proj_slab (default)saved_gate_copy=False; saved.proj_slab REQUIREDthe whole stage-(1) slab (34 KiB/token at the 397B geometry): gate, q_pre, k_pre and V are its column bands, nothing to recomputenone — the GEMM writes the slab in place of the workspace one (6 launches, the inference chain’s)
gate-copysaved_gate_copy=True; saved.proj_slab=None, saved.gate a compact [B, S, H_q, D] bufferthe GATE band (16 KiB/token); q_pre / k_pre only if you pass buffers for them (else the backward recomputes them from h, RecomputePolicy.RECOMPUTE_QK_PRE)one extra elementwise launch per copied band; the workspace keeps its slab

Recommendation: proj_slab up to S = 32K, gate-copy beyond (the whole slab, 34 KiB/token at the 397B geometry, is about twice the GATE band alone).

Contracts verified before any launch, each a ValueError naming the field: saved.h is h’s storage; saved.seq_lens is the seq_lens tensor passed to execute (or both None — the backward declines padding at declaration from this field, without a device read); lse (optional) is saved.lse’s storage; saved.o is compact [B, S, H_q, D] in the activation dtype; saved.rstd_q / rstd_k are [B, S, H] fp32 compact (present iff geometry.qk_norm); every caller buffer a kernel writes (proj_slab, o, lse, rstd_*, the gate-copy targets) is 16-byte aligned — they are TMA-store targets, and a slice at an odd element offset is refused here rather than failing untyped after stage (1) launched; in the proj_slab mode saved.gate / q_pre / k_pre, when given, alias saved.proj_slab exactly as saved_slab_views spells them (each may be None there: the backward derives it from the slab). The refusal runs the other way too: saved= on a block declared without save_for_backward is a ValueError naming the knob — an inference forward writes none of the record’s tensors, so a silently ignored record would reach the backward uninitialised. A padded forward (seq_lens) is served: a dead entry (seq_lens[b] == 0) leaves saved.o[b] == 0, saved.lse[b] == -inf and out[b] == 0 exactly.

execute() allocates nothing, reads nothing back to the host and converts nothing: every intermediate is a strided view of the caller’s workspace, sized honestly by get_workspace_size(), so the call is CUDA-graph friendly. All stages run on one launch stream (torch’s current stream, or current_stream), on both the FROST GEMM route and the DSL stages. Dtype, layout and shape mismatches, an unsupported architecture, or a feature the selected specialization cannot serve raise typed errors at construction (check_support) rather than at launch.

Quantization specs:

  • QuantSpec(descale_h, descale_w_qkvg, descale_w_o, scale_q, scale_k, scale_v, scale_o, dtype=torch.float8_e4m3fn, attention_dtype=None) — static per-tensor scales for the FP8 pipeline. attention_dtype (appended 2026-10-30; an op attribute, never a knob — it changes the numerics) selects what stage (4) computes on: None = the fp8 d256 SDPA pipeline above (Q / K / V quantized at scale_q / scale_k / scale_v), torch.bfloat16 = the FP8-PROJECTION + BF16-ATTENTION mode of the paged serving forms, under which scale_q / scale_k / scale_v MUST be None (there is no Q / K / V quantization point: each is refused by name otherwise) and the three descales and scale_o stay finite positive floats; QuantSpec.bf16_attention(descale_h, descale_w_qkvg, descale_w_o, scale_o) spells it, QuantSpec.attention_bf16 reads it back. validate() rows: the dtype (e4m3 only), attention_dtype None / bf16 only (torch.float16 by name: the quantized pipelines’ activation dtype is bf16), a None scale under attention_dtype=None a typed ValueError naming the scale and the way in, a Q / K / V scale set under the mode a ValueError naming it. attention_dtype is a plan fact of update_quant_scales (below).

  • MxQuantSpec(descale_w_o, scale_o=1.0, dtype=torch.float8_e4m3fn, block_size=32, w_qkvg_dtype=torch.float8_e4m3fn, o_fp4=None) — the MXFP8 pipeline; the MXFP8 SDPA writes e4m3 O unscaled, so the fully fused path needs scale_o == 1.0. The two appended fields select the fp4 modes:

    • w_qkvg_dtype: torch.float8_e4m3fn (default, MXFP8 x MXFP8) or torch.float4_e2m1fn_x2 (MXFP4 W_qkvg, the mixed row); any other dtype is a typed NotImplementedError.
    • o_fp4: None (default, per-tensor e4m3 O) or an Fp4Format member (anything else is a TypeError). Under o_fp4 neither side of the out projection has a per-tensor scale — the quantizer writes block scales only and W_o dequantizes through w_o_sf in the MMA — so scale_o and descale_w_o must both be 1.0 (typed ValueError otherwise; a non-unit value would be silently dropped by the block-scale GEMM).
  • Fp4Format (enum; one member = codes x scale dtype x block, so an illegal pairing cannot be spelled):

    membercodesscale dtypescale block along K
    Fp4Format.NVFP4e2m1torch.float8_e4m3fn16
    Fp4Format.MXFP4e2m1torch.float8_e8m0fnu32

    Properties: block_size, sf_torch_dtype, sf_cudnn_dtype (and fmt_name, the kernel’s format key). Neither format carries a global (per-tensor) scale: an NVFP4 block’s e4m3 scale is max(amax * fp32(1/6), 2^-9) (the e4m3 min-subnormal floor keeps an all-zero block’s scale NONZERO, so the encode x / scale stays finite), an MXFP4 block’s E8M0 scale is the power of two at or above amax * fp32(1/6) — fp32(1/6), not an exact / 6: the kernel and the reference multiply by the fp32 constant, and the two differ by one ulp; a dead (fully masked or zero-length) row quantizes to codes 0 exactly. The O codes are round-to-nearest-even on the e2m1 grid, saturating at 6.

Per-step recalibration — update_quant_scales. The scales of a QuantSpec / MxQuantSpec are plan-time constants, and a training step moves them: the weights change, so descale_w_qkvg / descale_w_o (and descale_h, the normed residual stream’s) are recomputed from the tensors quantized this step, and a per-tensor fp8 recipe recalibrates scale_q / scale_k / scale_v / scale_o from a previous step’s activation amax — the block itself runs no amax pass, the caller computes them from the record it already holds. GatedAttentionBlockFwd.update_quant_scales(spec, *, current_stream=None) re-points a COMPILED block at a new spec without recompiling: an in-place write of the block’s device scalars (the GEMM alphas, the quantize scales, the fp8 SDPA’s descales, and the fully fused fp8 fork’s scale vector) on the launch stream (current_stream, else torch’s current stream), then blk.quant = spec; every consumer binds those tensors at execute, so the next execute on that stream runs at the new scales, and a CUDA graph that captured an execute of a warmed-up block replays with the live values — unless the captured execute was the block’s first: its one-time scalar write (one fill kernel per value; the first-execute write below) is captured with the capture-time values, so every replay re-writes them and overrides a later eager update_quant_scales; warm up with one eager execute on the capture stream before capturing (as every capture of the block does), or treat such a graph as pinned to the scales it captured and re-capture after a recalibration. GatedAttentionBlockBwd.update_quant_scales(spec) is the backward half: the plan-time constants the next execute’s prologue launch stores from its kernel arguments are re-resolved on the host (no device write; a CUDA graph captured before the call keeps the old constants — re-capture after a recalibration). Typed refusals, before any write: a block declared without quant (ValueError), a spec of the other class (TypeError), a differing plan fact — dtype, block_size, w_qkvg_dtype, o_fp4 — (ValueError naming it: those select kernels and the carve), the spec class’s own validate exactly as the declaration applied it (the fully fused MXFP8 path keeps scale_o == 1.0; a zero / inf / NaN scale is its ValueError), and a block not yet compiled (RuntimeError); under o_fp4 there is nothing to write and the call validates and records it. The per-layer recipe of a training loop: one compiled block pair per dtype configuration, fwd.update_quant_scales(spec_l) right before layer l’s forward and bwd.update_quant_scales(spec_l) right before its backward (autograd runs the backward long after every layer’s forward, so the step’s spec travels with the layer), spec_l’s descale_* from the tensors quantized this step and its activation scales from the previous step’s record of that layer. compile() only allocates those scalars; their values are written on the launch stream — by the first execute of a compiled block and by every update_quant_scales — never on the stream ambient at compile time, and every such write first waits on an event compile() records behind its allocations (under torch.use_deterministic_algorithms(True) an allocation itself fills with NaN on that stream), so a block compiled on one stream and recalibrated or first executed on another reads what its execution stream wrote, and nothing enqueued at compile time can land later and undo an update. Inside a CUDA-graph capture that wait is skipped: warm the block up with one eager execute before capturing (the rule below).

Packed sequences (THD)

thd=True runs the block over ONE packed token matrix holding B sequences back to back — the layout a varlen caller already holds (TransformerEngine’s and FlashAttention’s “THD”): no padding, no per-batch axis, the per-sequence lengths as an int32 tensor.

lens = [300, 128, 200] # B = 3 sequences, T = 628 packed tokens
h = torch.empty(sum(lens), d_model, device=dev, dtype=torch.bfloat16) # [T, d_model] or [1, T, d_model]; out likewise
cos, sin = ... # [T, rope_dim] (or [1, T, rope_dim]): PER-TOKEN tables, positions restarting at every sequence
seq_lens = torch.tensor(lens, dtype=torch.int32, device=dev) # [B] lengths ...
cu_seqlens = torch.tensor([0, 300, 428, 628], dtype=torch.int32, device=dev) # ... or [B+1] prefix sums (cu_seqlens=True; any base)
blk = GatedAttentionBlockFwd(h, w_qkvg, w_q_norm, w_k_norm, cos, sin, w_o, out, geometry,
thd=True, num_sequences=len(lens), max_seq_len=max(lens)) # + save_for_backward=True for training
blk.check_support(); blk.compile()
workspace = torch.empty(blk.get_workspace_size(), dtype=torch.uint8, device=dev)
blk.execute(h, w_qkvg, w_q_norm, w_k_norm, cos, sin, w_o, out, workspace, seq_lens=seq_lens) # REQUIRED under thd
  • Shapes. sample_h / h, cos / sin, out are [T, .] or [1, T, .]. Internally the block is a B = 1, S = T block: every token-wise stage (the two projections, norm + RoPE, the gate, the quantize passes) is the SAME launch as the dense block’s — at B = 1 the packed block is bitwise the dense B=1, S=T one — and only the SDPA runs its packed specialization (the FROST d256 THD arm, SCHED_NATURAL). The workspace carve is the dense B=1, S=T block’s plus the packed SDPA’s own scratch (metadata and per-sequence descriptors; get_workspace_size() reports it).
  • Lengths. execute(seq_lens=) is REQUIRED: a contiguous 1-D int32 CUDA tensor on h’s device with exactly B entries (cu_seqlens=False, lengths) or B+1 (cu_seqlens=True, prefix sums — normalized to their first entry on the device, so a tensor sliced from a larger prefix works). It is handed to the SDPA as BOTH the Q-side and the KV-side lengths (self-attention) and is never read on the host, so the block stays CUDA-graph capturable: a replay may carry NEW lengths written through the captured tensor, as long as B is unchanged, every length is <= max_seq_len and the lengths still sum to T (rewrite cos / sin for the new packing too).
  • RoPE tables are per token. Build cos / sin per sequence (positions 0 .. len_i - 1) and concatenate along the token axis. A dense [B, S, rope_dim] table, or positions running across sequence boundaries, is a plausible-but-wrong RoPE after the first sequence.
  • Declaration bounds (typed ValueError): num_sequences >= 1, 2 <= max_seq_len <= T (S = 1 is decode, out of the prefill bodies’ scope; on cc 10.7 the standalone SDPA forward serves decode-shaped d = 256 graphs on its decode tile — docs/operations/Attention.md, “Paged attention” — and the block serves dense paged declarations through the DECODE form described under “Serving: the decode form” below) and num_sequences * max_seq_len >= T — a smaller product would silently cap the SDPA chain’s packed capacity below T (the tokens past it are not processed, with no message downstream). seq_lens_present is mutually exclusive with thd (there is no per-batch KV padding mask under THD); the three packing knobs on a dense block are refused.
  • Caller contract on the lengths (device data, never validated on the host): every length in [0, max_seq_len], non-decreasing prefix sums, and sum(lengths) == T. Zero-length sequences are served (a caller whose B varies pads with empty sequences; an empty sequence owns no rows). sum == T is a contract, not a capacity: the SDPA writes nothing past the live total (rows [cu[B], T) of saved.o / saved.lse stay untouched), but the token-wise stages run over all T rows and the backward’s weight-gradient GEMMs contract over all T rows, so lengths summing below T contaminate dW_qkvg / dW_o — finite or NaN. Declare T as the live total.
  • Training record. The record is the dense record at (1, T) — proj_slab [T, n_qkvg], o [1, T, H_q, D], lse [1, H_q, T] (head-major, head stride exactly T: the ONE packed Stats layout the backward reads), rstd_* [1, T, H] — plus saved.seq_lens (REQUIRED: the lengths tensor execute ran with, identity-verified) and saved.seq_lens_form ("lengths" / "prefix", matching cu_seqlens; None is the padded dense record’s value and is refused under thd). Both save modes serve.
  • Served / declined. Served: bf16 / fp16 (inference and training, in place and out of place), the per-tensor FP8 unfused pipeline (QuantSpec; inference and training — a packed FP8 training forward writes the same bf16 record at (1, T) as the dense quantized training forward, with saved.h the e4m3 h; the packed per-tensor fp8 backward (quant=QuantSpec) differentiates it as written, the packed bf16 backward given the dequantized bf16 h and weights, exactly as on the dense side), the MXFP8 unfused pipeline (MxQuantSpec, the fp4 modes w_qkvg_dtype / o_fp4 included; inference and training, the same bf16 record at (1, T)): its three quantize_mxfp8 stages run their PACKED arm and write the Rubin d256 MXFP8 SDPA’s per-sequence-tile-padded scale-factor layout — per head the tiles of every sequence in cu_seqlens order, V’s two D-planes adjacent inside a tile, the slot sized at the capacity H * ((T + 127 * B) // 128) * 1024 bytes (the only carve difference to the dense B=1, S=T block; slack tiles and pad bytes 0x00), each tile’s sequence resolved on device from the lengths tensor (no cap on B; at B = 1 the packed block is bitwise the dense one on its outputs, record and e4m3 payload, and V’s scale-factor slot holds the dense atoms in plane-adjacent order); fuse_norm_rope (bf16 / fp16 inference in place: the projection fork norms and rotates per token with the per-token tables); fuse_gate (bf16 / fp16 inference: stage (5) folds into the packed SDPA’s epilogue through the SDPA’s THD gate descriptor — the packed block at B = 1 is bitwise the dense gated block, the fused gate within one output rounding of the separate stage, one launch fewer: proj(+norm+RoPE) -> sdpa(+gate) -> out_proj plus the packed SDPA’s setup launch); the paged KV-cache write-through (paged_kv_page_size: slot_mapping is [T] over the packed tokens, a pad token’s slot -1 — “Serving: write-through into a paged KV cache” below) and, on top of it, the EXTEND form (kv_source="paged": the SDPA reads every sequence’s K / V from the pools; token_capacity makes T a capacity — “Serving: the extend form” below; fuse_gate rides the paged body’s epilogue there too, one launch fewer), block-sparse attention (QsaSpec with caller lists: the sparse core’s packed arm, block_ids numbered per sequence, fuse_gate composing with it — “Sparse attention (QSA)” above). Declined, typed: the fully fused quantized pipelines (the quantized THD SDPA bodies have no THD gate descriptor, and the fused MXFP8 projection fork decodes (b, s_tile) once per 128-row GEMM tile, which a packed tile may straddle), the MXFP8 block BACKWARD over a packed record (its SDPA-layout MX quantize stages run the quantizer’s dense arm only; the packed record and the packed delta exist, the packed MXFP8 backward is a follow-up) and the in-block indexer under QsaSpec (the paged write-through composes with the packed sparse block: the stage addresses tokens, so slot_mapping is [T] there too).
  • Declare max_seq_len tight. The SDPA’s unit grid is the plan-time envelope B * ceil(max_seq_len / tile) * H_q with dead units past the live total, and the backward’s dS workspace scales with ceil128(max_seq_len).

Serving: the e4m3 KV cache (kv_cache_dtype)

The e4m3 KV cache (kv_cache_dtype=torch.float8_e4m3fn, 2026-10-14; bf16 activations). The serving stacks’ --kv-cache-dtype fp8 recipe: the pools hold e4m3 codes under per-tensor fp32 DEscales execute(k_scale=, v_scale=) (value = code x scale; 1-element fp32 CUDA tensors read on device — vLLM’s per-layer k_scale / v_scale), a bf16 Q attends over the dequantized rows. A declaration ATTRIBUTE beside paged_kv_page_size (it changes the write-through’s numerics), never a knob; kv_cache_dtype=None (or the activation dtype) is the bf16 / f16 pool as before. Two served forms, one attribute:

  • The quantizing WRITE-THROUGH on every form that writes through over the block’s own K / V (the dense prefill, QsaSpec or dense; the packed prefill is a typed decline until its accept cell lands): after norm + RoPE the post-RoPE K / V rows are quantized per tensor — code = sat_e4m3(row x (1 / scale)), one fp32 product and one rounding, the FP8 pipeline’s own cast kernel over the slab band — into two compact e4m3 workspace slots and bit-copied into the e4m3 pools through their bf16 views (three launches where the bf16 write-through runs one; no new kernel). The raw indexer key keeps its bf16 pool. out is bitwise the bf16-pool block’s (the write-through never changes attention); the pool rows are bitwise (row.float() * (1 / scale)).clamp(-448, 448).to(torch.float8_e4m3fn).
  • The block-sparse DECODE mode over e4m3 pools (kv_source="paged" on a dense [B, S_q <= 4] QsaSpec declaration — the shared list): stage (4) is the gathered cast + the Rubin d256 DECODE tile — kernels/qsa_kv_upcast.py reads, per (token, KV head) item, the sequence’s listed 4-token blocks and the open tail over [0, kv_lens[b]) through block_table[b], dequantizes each e4m3 row (bf16(fp32(code) x scale)) and writes the visible rows COMPACTED in list order into the workspace scratch [B x S_q x H_kv, 2, 2176, 256] bf16 plus a per-item length (0 = a dead item); the decode tile then runs over that scratch as a plain padded dense SDPA (no causal mask — the gathered rows ARE the visible set — the per-item length on the KV side only, the group packed, unsplit). The sparse core is never taught fp8. The mode norms + rotates OUT of place (the tile’s per-item Q is a view of the compact normed Q alone: inplace_qkv=None reads as False there, fuse_norm_rope is declined) and gates in its own launch (blk.gate_fused False whatever fuse_gate says); get_workspace_size() grows by the scratch (2.2 MB per item) and holds it.

Typed declines (each pinned by test_block_qsa_fp8_kv.py): torch.float8_e5m2 (by name) and any other dtype; the attribute without paged_kv_page_size; quant / save_for_backward with it (the write-through’s own rows; the FP8-projection + bf16-attention mode too — its pools STAY bf16, an e4m3 cache under it is a follow-up: the attribute’s own row names it); f16 activations; over the pools the EXTEND form and the dense DECODE form (the paged THD body and the decode tile read bf16 / f16 pools: a dense e4m3 pool needs the tile’s in-kernel dequant arm — a follow-up), S_q > 4 (the per-token list form: 2.2 MB of scratch per item), return_lse (the tile over the scratch writes item-major Stats), decode_split_kv > 1 (unsplit over at most 17 KV tiles per item), fuse_norm_rope and an explicit inplace_qkv=True; pools of the wrong dtype at declaration and at execute (both ways); k_scale / v_scale missing under the attribute or given without it. serving_envelope(...).decode.kv_cache_dtypes names the pool dtypes served at a geometry / dtype ((bf16, e4m3) on a QsaSpec geometry with bf16 activations, the activation dtype alone otherwise). Validated on Rubin against the fp32 oracle ON THE DEQUANTIZED POOL (the prefix rows quantized by torch with the per-tensor scales, the step rows the block’s own write; the gated O within atol 2e-2, cos(out) > 0.999, dead rows exact, the pools’ step rows bitwise the torch cast, two launches bitwise; the distance to the pre-quantization bf16 oracle reported, no gate): S_q 1 .. 4, B 1 / 3 / 4, kv_lens up to 20000 with dead and shorter-than-the-step entries, pages 16 / 32 / 64 on HND / unified / NHD pools, block_lens given / absent, the indexer band’s raw-key write-through, the 24 / 2 Flash-Next geometry, the MTP residues, the write-through-only prefill form, the workspace sentinel, the CUDA-graph replay over new scales / lengths / tables / lists / slots / inputs. Performance: none claimed (functional cells only); the unfused e4m3 decode step runs 9 launches against the bf16 mode’s 5 — the quantizing write inside the projection epilogue, the reciprocal inside the write kernel and the in-kernel dequant are the levers a later change measures.

Serving: the extend form (kv_source="paged", token_capacity)

Served for bf16 / f16 activations (quant=None) and, since 2026-10-30, under the FP8-projection + bf16-attention mode (QuantSpec(attention_dtype=torch.bfloat16) with e4m3 h / weights over the same bf16 pools: the section below; cc 10.7 and cc 10.0). Served on cc 10.7 (the cells below) and, since 2026-10-29, on cc 10.0 through the same binding — the shared SM100 d256 paged THD body (sm100/d256_f16_lookahead/kernel.py: CGA2, unsplit, unpacked, NATURAL, no gate, no LSE), the write-through before it, the sigmoid gate as its own launch; functional only, unmeasured there, fuse_gate=True and fuse_norm_rope=True typed declines by name (the part table at the top; test_block_serving_extend.py’s requires_blackwell cells ran 2026-10-29 on a cc 10.0 part of the 152-SM class: 82 passed / 0 failed in the module’s cc 10.0 selection, within the suites’ 351 / 0 record).

Every serving prefill of a sequence with a cached prefix — a chunked prefill, a prefix-cache hit, a speculative verify — is an EXTEND step: the step’s new tokens are written into the pools, and attention runs over each sequence’s WHOLE logical KV range (its prefix and the new tokens) from the pools, on the bottom-right diagonal. kv_source="paged" (appended; "self" = the block as before) declares exactly that on a PACKED declaration:

P, B_max, T_max, max_pages = 16, 128, 2048, 2048 # the kernel block size, the step's sequence / token capacities, the table width
geometry = GatedAttentionBlockGeometry(d_model=4096, h_q=32, h_kv=2, d_head=256, rope_dim=64, causal_bottom_right=True)
blk = GatedAttentionBlockFwd(h, w_qkvg, w_q_norm, w_k_norm, cos, sin, w_o, out, geometry, # h / cos / sin / out: [T_max, .] token matrices
thd=True, num_sequences=B_max, max_seq_len=T_max, cu_seqlens=True, token_capacity=T_max,
paged_kv_page_size=P, kv_source="paged",
sample_k_cache=k_cache, sample_v_cache=v_cache, # the pools' [num_pages, H_kv, P, D] shape / dtype / strides
sample_block_table=block_table) # [B_max, max_pages] int32 (a row slice of a wider table binds)
blk.check_support(); blk.compile()
workspace = torch.zeros(blk.get_workspace_size(), dtype=torch.uint8, device=h.device) # ZEROED once (the bounded-token contract below)
blk.execute(h, w_qkvg, w_q_norm, w_k_norm, cos, sin, w_o, out, workspace,
seq_lens=query_start_loc, # [B_max + 1] int32 prefix sums of the step's NEW tokens per sequence (zero-length pads)
k_cache=k_cache, v_cache=v_cache, # the pools the samples described, EXACTLY (shape and strides; the layout is compiled in)
slot_mapping=slot_mapping, # [T_max] int32 / int64: the new tokens' slots; -1 on every pad row
block_table=block_table, kv_lens=kv_lens, # [B_max, >= max_pages] int32 page ids; [B_max] int32 LOGICAL lengths incl. the new tokens
num_tokens_live=None) # or the step's live token count T_live: the token-indexed stages over T_live rows, no recompile

Five launches on ONE stream — four with fuse_gate=True: (1)+(2)+(3) the projection with norm + RoPE over all T_max rows — or over the T_live rows of num_tokens_live — (or (1) then (2+3)), (4w) the write-through of the new tokens’ K / V at slot_mapping, (4) the SDPA over the pools — query row i of a sequence with q_len new tokens and logical length kv_len attends keys k <= i + kv_len - q_len of block_table[b]’s pages; under fuse_gate=True its epilogue applies O *= sigmoid(GATE) (the paged body carries the Rubin d256 gate seams, so stage (5) is not launched) — then (5) the gate (unfused only) and (6) the out projection. On the plain chain the write is ordered before the read by STREAM order alone: no kernel on the chain launches as a programmatic dependent launch (pinned by test_block_serving_extend.py), so a kernel that adopts PDL after a pool write must wait on its dependent grid before its first K / V load. The EXTEND form’s use_pdl default is False (the measured default is the DECODE form’s, “Serving: the decode form”); under use_pdl=True the switch is PER STAGE: the write-through (and the norm + RoPE pass, the separate gate) launch with the programmatic-stream-serialization attribute and their griddepcontrol pair, while the paged prefill tile — which carries no sites — launches plain, so that link serializes and the read still follows the write; blk.pdl_stages names the stages that took the attribute (never "sdpa" on this form), blk.use_pdl_effective is False, and serving_envelope().extend.pdl_stages says so before anything is declared.

appendedcontract
kv_source"self" (default) or "paged"; "paged" requires paged_kv_page_size > 0, thd=True, geometry.is_causal=True and geometry.causal_bottom_right=True, bf16 / f16 activations (quant=None) or the FP8-projection + bf16-attention mode (QuantSpec(attention_dtype=torch.bfloat16), e4m3 h / weights, bf16 pools), inference (save_for_backward=False), return_lse=False, fuse_gate as declared (True folds stage (5) into the paged SDPA’s epilogue), window_left=-1, window_right unbounded or 0, d_head=256; a bottom-right QsaSpec(index_source="caller") geometry declares the packed block-sparse EXTEND form instead (its own paragraph below: bf16 / f16, return_lse served, decode_split_kv None, use_pdl None / False); every other combination is a typed decline naming the feature (GatedAttentionBlockFwd.serving_envelope lists them)
token_capacityTHD only; None = the tight contract (lengths sum to T), an int = the BOUNDED form and must equal the samples’ packed row count T: a step’s lengths may sum to T_live <= T; rows [T_live, T) are pad rows — finite h / cos / sin, slot -1 (never written to the cache), never attended; their out rows are UNSPECIFIED but finite given finite inputs. Zero the workspace once and slice out[:T_live]: an unwritten o row then reads 0, so pad rows of out are exactly 0 until a later, longer step overwrites those o rows (the o slot is aliased by no other stage’s scratch — pinned). Refused with save_for_backward (the backward’s weight-gradient GEMMs contract over every row)
sample_k_cache, sample_v_cache, sample_block_tableREQUIRED under kv_source="paged", refused otherwise: the pools’ [num_pages, H_kv, P, D] shape / dtype / strides (the three stride patterns of the write-through, K and V of one HND / NHD kind, every non-innermost stride a 16-byte multiple) and the [num_sequences, max_pages] int32 block table (its row stride is free — a [:, :max_pages] slice of a wider table — and compiled in); the logical S_kv the tables may address is max_pages * P
block_table (execute)[num_sequences, >= max_pages] int32 on h’s device, the page axis contiguous, the row stride as declared; row b lists sequence b’s pages in logical order (the serving stacks’ NULL block 0 for a padded entry); read on device
kv_lens (execute)[num_sequences] int32 contiguous: each sequence’s LOGICAL KV length INCLUDING this step’s new tokens (prefix + q_len); 0 = a padded entry (no keys: its out rows, if it has any, are exactly 0); read on device
k_cache, v_cache (execute)the pools, matching the samples EXACTLY (shape and strides; another page count or layout is another plan). Rows past kv_len inside a sequence’s last live page may hold stale values but must be FINITE (a serving pool is zero-initialised); pages past the live page count are never read
num_tokens_live (execute, appended, the bounded form only)the step’s LIVE token count T_live as a HOST int; None (default) = the declared capacity, every stage over all T_max rows as before. With an int the projection (+ norm + RoPE), the cache write, the gate and the out projection run over T_live <= T_max rows with NO recompile (their kernels take the token count symbolically; the SDPA keeps the declared geometry and reads the live total from seq_lens on device, so the lengths must sum to T_live); h / cos / sin / slot_mapping / out may carry the declared T_max rows (the first T_live are read / written) or EXACTLY T_live rows — a serving stack hands hidden_states[rows], output[rows] and slot_mapping[rows] directly, with no staging copy — and rows [T_live, T_max) of out are left UNTOUCHED (not written: slice nothing, or keep them as they were). Typed before any launch: a block declared without token_capacity refuses it by name, T_live outside [1, T_max] names the bound (T_live > T_max is the decline: more tokens are another declaration), a quantized pipeline and a dense projection on the graph route (a bounded block declared without kv_source="paged") decline. A CUDA-graph capture bakes the host int into the captured launches: a replay serves the live count it was captured with, so a serving stack captures ONE graph per live-count bucket and pads a shorter step to the bucket (finite h / cos / sin pad rows, slot -1); the uncaptured EXTEND steps are where the count removes the staging copies

GatedAttentionBlockFwd.serving_envelope(geometry, dtype, device) -> ServingEnvelope is the static envelope a serving stack’s init-time dispatch predicate reads instead of catching exceptions: the device / DSL facts, and per form (extend and decode, described under “Serving: the decode form” below) whether it is served for that geometry and dtype (served / reason), the declaration it requires, the validated page sizes and pool layouts, the dtypes, and declines — every typed decline the form raises at declaration with its exact message, read off the same table the constructor raises from, so no declaration-time decline escapes the envelope.

Validated on Rubin (cc 10.7) at pages 16 / 32 / 64 / 128 on all three pool layouts, bf16 and f16, with vLLM-shaped buffers: padded entries (kv_len = 0, the NULL block 0, slot -1), prefixes ending mid-page, tail tiles, one / three / four / eight KV tiles, pads at the end and in the middle of the packing, a row-sliced block table, the 2-chunk extend and the prefix-hit step against the single-shot oracle within the dense suite’s budget (cos >= 0.999 on out, atol 2e-2 on the gated O), live rows bitwise between a bounded and a tight declaration, and — with the projection GEMM driver’s capture-safe route in the tree — a CUDA-graph capture of the first execute replayed over new lengths / tables / slots; num_tokens_live at T_live 412 and 300 of T_max 512: the live rows bitwise the capacity launch and the rows past them untouched, the token tensors at exactly T_live rows bitwise too, a capture at T_live 412 replaying bitwise and serving the padded next chunk; the three Qwen3.5-397B tensor-parallel shards (16 / 1, 8 / 1, 4 / 1 over d_model 4096, one replicated KV head — the table under “Qwen3.8 family” above) each with one EXTEND step at pages 16 / 32 / 64 and one DECODE step at the same pages — the rows of the support-matrix tracker’s Rubin d256 gated attention block -- serving forms section; the standalone adapter’s own paged-prefill contract applies to the pools (docs/operations/Attention.md, “Paged attention”).

The packed block-sparse EXTEND form (QsaSpec with caller lists; 2026-10-30). kv_source="paged" on a PACKED declaration of a bottom-right QsaSpec(index_source="caller") geometry is the sparse core’s THD x PAGED x BOTTOM_RIGHT composition: ONE execute serves a mixed serving step — decode rows beside prefill chunks of any length, every sequence behind its cached prefix, in ANY sequence order. Stage (4) is the index-list sparse core reading the pools: row j of sequence b at position kv_lens[b] - q_len_b + j, the listed 4-token blocks of its OWN sequence (block_ids [T, top_k] / [1, T, top_k] int32, sequence-relative ids, one list per packed row; block_lens optional) plus the open tail over [0, kv_lens[b]) through block_table[b], unsplit (the packed form’s claim-counter scheduler has no split axis: decode_split_kv stays None), the gate in the core’s epilogue (fuse_gate=True) or its own launch, the LSE served (return_lse=True: [1, H_q, T] fp32, head-major packed). execute(seq_lens=) is the Q side ([B_max] lengths or [B_max + 1] prefix sums per cu_seqlens), kv_lens the KV side ([B_max]: each sequence’s LOGICAL length INCLUDING this step’s tokens; 0 = a dead entry whose rows, if any, are out = 0 / LSE = -inf; a sequence shorter than its chunk leaves its rows at a negative position dead), block_table EXACTLY the declared table (sample_block_table CONTIGUOUS, one row per sequence of the packing — never Q’s batch extent of 1), token_capacity optional (None = the tight contract; the BOUNDED form is the serving contract: the CAPACITY launch over the T rows with the pad rows’ slot -1, their out rows exactly 0 on the once-zeroed workspace; the live-count launch num_tokens_live where the dense projections resolve a JIT tile). Device-value contract (reported, never raised): kv_lens[b] in {0} or [q_len_b, max_pages x P]. Typed declines, each raised with EXACTLY the text of its extend.declines row: thd_paged_select_form (index_select_form other than "auto"), thd_paged_split (decode_split_kv), thd_paged_lean (sparse_tile_form="lean": the lean tile has no claim-counter form), thd_paged_shared_list (index_list_form="shared": no meaning over ragged packed rows), thd_paged_pdl (use_pdl=True), thd_paged_fuse_prepare, thd_paged_e4m3_pool (kv_cache_dtype=torch.float8_e4m3fn), thd_paged_indexer (index_source="indexer" on the packed form is a follow-up: hand the selection in under caller lists, or run the dense extend step per chunk-length group); the table’s sparse row stays reachable through a sparse-adapter record without the composition claim (SparseCapabilities.thd_paged_kv). The envelope reads extend.sparse_core, extend.index_sources == ("caller",), extend.index_list_forms == ("per_row",), extend.sparse_tiles == ("sparse_d256_f16",), extend.writes_lse and extend.pdl_stages == () (the appended tail of ExtendEnvelope; a dense geometry reads the defaults), and extend.requires spells the form. Where the geometry alone refuses the packed form (index_list_form="shared", index_source="indexer") the envelope’s verdict is the declaration’s: extend.served is False with that row’s text as extend.reason, "extend" is absent from forms and extend.sparse_tiles is empty (the geometry’s DECODE record is its own). A CUDA-graph capture takes the block after ONE eager warm-up on the capture stream (the first execute may create the projection route’s per-stream state); the replay then serves new lengths / tables / lists / slots / pools through the captured tensors. TWO-RUN EQUIVALENCE: the step computes the same function as a dense [B_dec, S_q] decode block plus one dense [1, T] paged QsaSpec block per chunk over the same pools — per row BITWISE where the dense control runs the sparse core unsplit (decode_split_kv == 1, read back from the control) and its slab GEMM takes the same projection route (both past the small-M bound of 128 rows), within the sparse modules’ bars (cos(out) > 0.999, the gated O inside the bf16 / f16 OUTPUT bound, the LSE within atol 2e-2) otherwise — the class follows from the control’s resolution, never from a measurement. No kernel body changed for the form: the composition is the three const_expr arms’ disjoint facts (the item decode, the gather row, the position); the barrier bundle, the SMEM / TMEM maps and every arrive count are the dense arm’s. The cells: test/python/gated_attention_block/cutedsl/test_block_qsa_packed_extend.py (the mixed step vs the two-run form at the 8 / 2 and 24 / 2 geometries, pages 16 / 64 on HND / NHD / unified pools, bf16 and f16; the fill-rule boundary; any order; the degenerate rows at every residue; the capacity and live-count launches; the workspace / allocation / census pins; the capture) and the adapter’s own in test/python/sdpa/frost/test_sdpa_fwd_sparse_sm107.py (every sequence of a packing bitwise the dense [1, q_len_b] paged bottom-right per-row run). Validated on a Rubin development node (cc 10.7, 204 SMs; functional cells, no performance claim): the block module’s 23 cells — the 300-row chunk of the mixed step BITWISE its dense control (the control resolved the core at split 1 on the forced-tile route), the 128- and 160-row chunks and every decode group within the bars (their controls resolved splits 3 / 6 / 9 / 16, or the lean tile at one row per sequence), the gated O within 0.31 of the output bound and the LSE within 3.7e-4 of the oracle on every cell — and the adapter’s 113 cells of the composition (every sequence bitwise its dense control; max|dO| 0.011 bf16 / 0.0012 f16, max|dLSE| 2e-6 vs the oracle). Functional only: the one-execute measurement against the two-run form follows.

Serving: the decode form (kv_source="paged" on a dense [B, S_q] block)

A decode step hands the attention sub-layer B sequences’ NEW tokens — one each, or 1 + num_spec with speculative decoding — whose keys and values live in the paged cache. GatedAttentionBlockFwd(sample_h [B, S_q, d_model], ..., paged_kv_page_size=P, kv_source="paged", sample_k_cache=, sample_v_cache=, sample_block_table=) declares that form: the step’s tokens are written through (4w) and stage (4) attends, per sequence, over the first kv_lens[b] keys of the pages block_table[b] lists — the prefix AND the step’s own tokens — on the bottom-right causal diagonal (the step’s rows are the LAST rows of a kv_lens[b]-key sequence; geometry.is_causal=True therefore needs geometry.causal_bottom_right=True; geometry.window_* are not served). The batch is UNIFORM (S_q rows per sequence; a serving stack’s padded entry is kv_lens = 0 + block table row 0 + slot -1 and yields out rows exactly 0, nothing written to block 0). The pipeline is (1)+(2)+(3) -> (4w) -> (4) [+ combine] -> (5) -> (6) — past decode row counts. At decode row counts (B x S_q <= 128, the small-M projection route’s bound) a fuse_norm_rope=True declaration LOWERS stage (1) to the unfused chain, (1) -> (2)+(3) -> (4w) -> (4) [+ combine] -> (5) -> (6): the small-M route’s slab GEMM and the norm + RoPE pass with the paged write-through folded in — fuse_cache_write_small_m (appended; None = auto): the post-RoPE K / V rows go to the pools from the norm + RoPE kernel’s own tiles (the V band rides the launch, the pools bitwise the separate write) at B x S_q < 128 rows or at S_q = 1 (measured to B 128; past it the rule is reachable only under an explicit fuse_norm_rope=False, the C0 lowering handing rows > 128 to the fused fork), the separate cache-write launch at B x S_q >= 128 with S_q > 1 and wherever an explicit fuse_cache_write=False asks for it by name (the request outranks the AUTO fold) — six launches with the split combine (seven with the separate write) instead of the 256-wide fused fork, which pads the step’s handful of rows to one token tile and streams the slab weight from a quarter of the SMs (the measured B 1 .. 32 decode step is faster on the unfused chain). The write-through rule is the measurement on a 212-SM cc 10.7 part: the arm wins on every B 1 .. 4 cell (+1.1 .. +3.2 % per step on the plain chain, +2.4 .. +4.9 % with PDL on) and at B 32 / 128 x S_q = 1, and loses 0.7-2.7 us per step at B 32 .. 128 x S_q = 4 (128-512 rows); True / False pin either arm wherever the arm is served (True is a typed decline on the fused fork, a quantized pipeline, the e4m3 cache, an indexer band, the EXTEND form), blk.cache_write_in_norm_rope reads back what runs, CACHE_WRITE_SMALL_M_DEFAULT_MAX_ROWS is the bound. fuse_norm_rope stays what was declared, blk.fuse_norm_rope_effective reads back what runs (cache_write_fused follows it: the write-through arm lives in the fork), and fuse_norm_rope_small_m=True (a declaration knob, appended; performance-only, the same function either way within the fp32 roundings of two kernels) keeps the fused fork as declared — the A/B’s control arm; so does an explicit fuse_cache_write=True (the write-through arm lives in the fork, and the request outranks the AUTO lowering: fuse_norm_rope_effective reads True, cache_write_fused True); only the explicit contradiction fuse_norm_rope_small_m=False + fuse_cache_write=True is a typed decline, naming both knobs. bf16 / f16, no indexer band, 16-bit pools, the dense DECODE form only: the EXTEND form, the block-sparse mode and the quantized pipelines run fuse_norm_rope as declared.

Programmatic dependent launch is the form’s default on its decode-tile routes on cc 10.7 (use_pdl=None -> True; DECODE_PDL_DEFAULT_CCS names the measured parts and every other part reads False until measured there; use_pdl=False is the plain chain anywhere, True the switched chain where it is served): the norm + RoPE pass, the separate cache write, the separate gate, the decode tile and its split combine launch with the programmatic-stream-serialization attribute and carry their griddepcontrol pair (a wait before the first read of a predecessor’s output, a release after the prologue), bitwise the plain chain eagerly and under capture; the FROST GEMM stages — the slab GEMM, the out projection and its split-K reducer — launch with the attribute whatever the switch says (their templates’ own constant), so a use_pdl=False chain still holds three programmatic launches. blk.use_pdl is the resolved switch, blk.use_pdl_requested the declaration (None = auto), blk.pdl_stages / blk.use_pdl_effective count the switched stages only, and serving_envelope().decode.pdl_default says what auto resolves to on a part before anything is declared (False wherever the form is not served — another part, an unserved dtype; on the dense form the per-step reading is decode.pdl_default and decode.route(s_q, batch).pdl_capable, a prefill-route step launching plain; under the block-sparse mode it is decode.pdl_default itself for a served step — the sparse core carries the sites at every served step width, and the dense route table does not apply there). The block-sparse decode mode takes the switch too (a QsaSpec geometry over bf16 / f16 pools): under use_pdl=True the sparse core launches with the attribute and carries its griddepcontrol sites (every role’s wait before its first read of a predecessor’s output — the slab, the pools, the block list, the lengths, the table — and no explicit release: the CTA exit triggers the dependent grid at the core’s completion, the placement measured on the cc 10.7 class, where a release at the core’s prologue left the split combine’s CTAs resident through the core’s whole run and cost 3.2-4.4 us per step at B <= 4) and its split combine takes the pair, bitwise the plain chain eagerly and under capture (the first execute of a fresh switched declaration, captured with no warm-up, replays bitwise the eager plain chain); blk.pdl_stages then names sdpa_sparse (the sdpa kind) beside the norm + RoPE pass, the write-through and the separate gate. Its AUTO reading is True on cc 10.7 (SPARSE_DECODE_PDL_DEFAULT_CCS = ((10, 7),) since 2026-10-27: the switched caller-list block measured against the plain one in one process on a 212-SM cc 10.7 part — CUDA-graph replay, the served body on each route, the lean tile at B 1 and the tcgen05 core at B 4 and B 32 — read +0.6 .. +2.5 % per step at B 1 x {8K, 32K, 128K} x S_q 1 / 4, +2.4 .. +3.5 % at B 4 and +2.3 .. +2.7 % at B 32 x {8K, 32K, 128K} x S_q 1, +0.6 % / +3.8 % at B 4 / B 32 x 8K x S_q 4, bitwise on every cell and no cell slower than its control band; False on every other part until measured there, never under the in-block indexer, and use_pdl=False is the plain chain on cc 10.7), and use_pdl=True with thd=True on a QsaSpec geometry is a typed decline (the packed form reads the setup launch’s metadata above every role wait). Measured on a 212-SM cc 10.7 part: 1.1-1.5 us per decode step at B 1 (B 1 x KV 4K +5.0 %, 32K +4.0 %, 128K +2.5 %; +2.8 % geomean over the B 1 .. 4 cells), flat within the control pair where the tile dominates; with the write-through folded in, B 1 x KV 4K 32.3 -> 29.3 us (+10.2 %) against the plain seven-launch chain. The fused kernel the stage runs is a declaration fact:

On cc 10.0 (2026-10-29; DENSE_SERVING_CCS) the same form runs the SM100 d256 decode tile (sm100/d256_f16_swap_ab/kernel.py) under the SM100 LINE’s route — decode_route(geometry, S_q, device_cc=(10, 0), batch=, max_kv_len=), the SDPA adapter’s own rule off cc 10.7, input for input (config_sm100.decode_d256_q_tile with the wide-launch reading config_sm100.d256_wide_decode_launch(batch, groups, max_kv_len): at least 8 units streaming at least 4096 128-key tiles between them; groups = H_kv packed, H_q unpacked): packed rows S_q x group <= 16 ride the 16-column tile in ONE unit; 16 < S_q x group <= 32 is the adapter’s 32-column arm at a WIDE launch — a typed decline naming the lift until that arm’s cells run (DECODE_WIDE_TILE_CCS: cc 10.3 since 2026-11-01, functional only; no cc 10.0 cell of the block has run the 32-column tile, and its lift is the tuple gaining (10, 0)); else the paged PREFILL tile with the whole group packed wherever the group divides 128 (16 / 8 / 4 / 32), at ANY batch (the line has no two-unit arm, so there is no fill rule) under BOTH policies (they coincide; decode_route_policy is recorded); else the 16-column tile UNPACKED (group KV streams) to 16 tokens; an unpacked step of 17 .. 32 tokens needs the 32-column width — the same arm: routed by the adapter for a wide launch only and declined by the block at every launch until its cells run — a typed decline predicted by serving_envelope(...).decode.routes / .route(S_q, batch, max_kv_len) and raised at declaration; decode_q_tile_max (the Rubin table’s cap) is refused by name; DecodeRoute.claimed is False on every SM100 cell (the claimed envelope is the Rubin measurement), DecodeRoute.device_cc / .max_kv_len read the table’s part and the bound back (an unserved part — cc 10.3, an sm80 host — reads the Rubin table on every reading, device_cc / route_cc None: the home reading). The split count comes from the decode split model’s Blackwell fit, the split combine runs the combine module’s own forms, unmeasured on this line (the decode-grid pin is the cc 10.7 rule’s), fuse_gate=True is a typed decline (the gate is its own launch after the combine), use_pdl=None reads False (the explicit True is served: the SM100 tile and the combine carry the griddepcontrol pair). The small-launch table at the served groups (tile 16 = the SM100 decode tile packed, U16 unpacked with group streams, P = the paged prefill tile packed, [W] = at a wide launch this cell is the 32-column arm packed: a typed decline today, -- = a typed decline at every launch, the same arm unpacked):

group (H_q / H_kv)S_q = 1S_q = 2S_q = 3 .. 5S_q = 6 .. 8S_q = 9 .. 16S_q = 17 .. 32
6 (24 / 4)tile 16tile 16U16 (6 streams) [W]U16U16—
12 (24 / 2)tile 16U16 (12 streams) [W]U16U16U16—
16 (32 / 2, 16 / 1)tile 16P [W]PPU16 (16 streams)—
8 (8 / 1)tile 16tile 16P [W at 3 / 4]PP—
4 (4 / 1)tile 16tile 16tile 16 at 3 / 4, P at 5 [W at 5 .. 8]PPP
MHAU16U16U16U16U16—

(The wide launches the arm would take, declined today, at group 6: W at B 2 x kv >= 64K or B 32 x kv >= 4K; U32 at B 1 x kv >= 22K or B 4 x kv >= 5.5K.) Functional only — the cells of 2026-10-29 on a cc 10.0 part of the 152-SM class: the oracle matrix at every served group x S_q 1 .. 5, the 27B geometry, the pool matrix at every page size and dtype, two launches, capture / replay, the explicit PDL switch, the prefill route at the standard batch, all passed (the suites’ record: 351 passed / 0 failed over the cc 10.0 rows); nothing is measured on that part; the 32-column arm’s cells (the MTP T 3 .. 5 step at group 6 on a wide launch) are the first to run and measure there, and DECODE_WIDE_TILE_CCS gaining (10, 0) is their lift. The Rubin route of record follows.

The route is TABLE-DRIVEN (the default policy, decode_route_policy="single_unit"): ONE ordered table of the Rubin decode tile’s single-unit widths, config_sm107.D256_DECODE_SINGLE_UNIT_Q_TILES = (16, 32, 64) (the 64-column tile landed 2026-10-15: four softmax groups, one unit of up to 64 packed rows, admitted for every packed group of this form — paged under the bottom-right causal mask), and ONE rule — packed rows S_q x group within the widest width ride that width in ONE token unit, the KV streamed once per (sequence, KV head); past the table the step rides the paged d256 PREFILL tile with the whole group packed (one Q tile per sequence and KV head: the KV streamed once) wherever that tile packs the group — a group dividing its 128 rows — AND the step fills the part: that tile is one UNSPLIT CTA per (sequence, KV head), so it needs B x H_kv >= DECODE_PREFILL_ROUTE_MIN_CTAS = 64 CTAs (measured on the Rubin performance class at 32 / 2, S_q = 8, kv 32K: 330.6 us at B 1 against 58.7 on two 64-column units; 389.9 at B 32 against the two units’ 472.6); below the fill, and for a group it cannot pack (12 = 24 / 2), the step keeps the decode tile’s token-unit arm (up to two units of the 64-column tile, two streams, split over the part by the decode split model) and its unpacked arm past that, the fewest streams any kernel offers a step that keeps the part busy (the prefill tile stays wherever no two-unit route holds the step, whatever the batch). DecodeRoute.kv_streams names the KV streams per (sequence, KV head), DecodeRoute.tile / policy / q_tile_max / batch which kernel ran under which policy, width cap and batch (decode_route(geometry, S_q, batch=); the envelope’s routes are the reading without a batch, serving_envelope().decode.route(S_q, batch=) the one at a batch):

(group = H_q / H_kv, S_q)stage (4)decode_route(geometry, S_q)
S_q x group within a width of the table (16, 32, 64)the Rubin d256 decode tile (sm107/d256_f16_swap_ab/kernel.py, the swap-AB tile at that width), the whole group packed, ONE token unittile = "decode_d256_f16", n_q 16 / 32 / 64, token_units = 1, kv_streams = 1, pack_gqa = True
past the table, a group dividing 128 (16 = 32 / 2 and 16 / 1, 8 = 8 / 1, 4 = 4 / 1, 32 = 64 / 2, …) with S_q x group <= 128, the step filling the part (B x H_kv >= DECODE_PREFILL_ROUTE_MIN_CTAS = 64) or no two-unit route holding itthe paged d256 prefill tile (the shared sm100/d256_f16_lookahead/kernel.py body on its paged THD arm: the EXTEND form’s binding at the dense step’s (B, S_q) envelope, the whole group packed, unsplit, the gate in its epilogue): one Q tile per (sequence, KV head)tile = "prefill_d256_f16", n_q = 0, token_units = 0, kv_streams = 1, pack_gqa = True, batch = the declared batch
past the table, below that fill (32 / 2 at S_q 5 .. 8 under 32 sequences, 8 / 1 at 9 .. 16 and 4 / 1 at 17 .. 32 under 64), a group the prefill tile cannot pack (12), or a step past its 128 rows: up to two TOKEN UNITS of 64 // group tokens (24 / 2 at S_q 6 .. 10)the 64-column decode tile in token units, ceil(S_q / (64 // group)) CTAs per (KV head, sequence, split), each streaming the KV range once, split over the part by the decode split modeln_q = 64, token_units = 2, kv_streams = 2, pack_gqa = True
… and past two units, up to S_q = DECODE_MAX_ROWS = 32 (32 / 2 from S_q 9, 24 / 2 from 11, 8 / 1 from 17, 24 / 4 from 21; 64 / 2 from 5, 128 / 2 from 3)the decode tile UNPACKED (the SDPA adapter’s own rule, the arm every group had before the table landed): one Q head per CTA, one token unit, S_q live rows of the 16-column tile (to 16 tokens) or the 32-column tile (17 .. 32) — declared, unmeasuredn_q = 16 / 32, token_units = 1, kv_streams = group, pack_gqa = False
MHA (group = 1), S_q <= 32the 16-column tile unpacked (the bit-exact unpacked fold; the 32-column tile from 17 tokens)n_q = 16 / 32, kv_streams = 1, pack_gqa = False
S_q > 32 (DECODE_MAX_ROWS = DECODE_MAX_S_Q)NOT served — a typed NotImplementedError at declaration (the decode_rows row): a longer dense step over the pools is the EXTEND form’s PACKED (thd=True) business; DECODE_PREFILL_TILE_ROWS = 128 bounds the prefill route only (past it the token-unit and unpacked arms, never a decline)tile = None, reason set
decode_q_tile_max=16 at 17 .. 32 tokensNOT served — the cap holds the table below the 32-column tile the unpacked step needs: honoured, never widened (a typed decline naming the cap; the envelope’s routes predict it)tile = None, reason names the cap

Which S_q rides which tile at the served geometries (the default policy; S_q 1 .. 8; “prefill” = the prefill tile where the step fills the part, B x H_kv >= 64, and two units of the 64-column tile below it):

geometry (group)12345678
32 / 2, 16 / 1 (16)tile 16tile 32tile 64tile 64prefill / tile 64 x 2 unitsprefill / tile 64 x 2prefill / tile 64 x 2prefill (128 rows) / tile 64 x 2
24 / 2 (12)tile 16tile 32tile 64tile 64tile 64tile 64 x 2 unitstile 64 x 2 unitstile 64 x 2 units
8 / 1 (8)tile 16tile 16tile 32tile 32tile 64tile 64tile 64tile 64
24 / 4, 6 / 1 (6)tile 16tile 16tile 32tile 32tile 32tile 64tile 64tile 64
4 / 1 (4)tile 16tile 16tile 16tile 16tile 32tile 32tile 32tile 32
64 / 2 (32)tile 32tile 64prefill / tile 64 x 2prefill (128 rows) / tile 64 x 2unpacked (160 rows, 32 streams)unpackedunpackedunpacked

Under decode_q_tile_max=32 (the route before the 64-column tile landed, value for value — the perf A/B’s “develop behaviour” arm): 32 / 2 rides the prefill tile from S_q 3 (below the fill: two units of the 32-column tile at S_q 3 / 4, the prefill tile at 5 .. 8 where two units hold no more than 4 tokens); 24 / 2 two units of the 32-column tile at S_q 3 / 4 and the 16-column tile UNPACKED (12 streams) at S_q 5 .. 8; 8 / 1 the prefill tile from S_q 5 (below the fill two units of the 32-column tile to S_q 8); 24 / 4 two units of the 32-column tile at S_q 6 .. 8.

decode_route_policy="token_units" (a declaration knob, appended; None = "single_unit") is the SDPA adapter’s own route verbatim — the widest admitted width in up to two token units for every packed group past the table (32 / 2 at S_q 5 .. 8: two units of the 64-column tile, two streams), then the unpacked tile, whatever the batch — the other arm of the two routes’ A/B (below the prefill route’s fill the two policies coincide; at the fill the default policy takes the prefill tile and token_units keeps the two units); decode_q_tile_max (a declaration knob, appended; None = the whole table, 32 = the measured widths) caps the table under either policy; decode_route(geometry, S_q, policy=, max_q_tile=, batch=) and serving_envelope(..., decode_route_policy=, decode_q_tile_max=) read them. The CLAIMED envelope is 1 <= S_q <= DECODE_CLAIMED_MAX_S_Q = 4 on the PACKED decode tile at a width up to DECODE_CLAIMED_MAX_N_Q = 32 (DecodeRoute.claimed: the measured cells — 24 / 2 and 32 / 2 at S_q <= 2, and at S_q 3 / 4 under decode_q_tile_max=32); the served envelope runs to S_q = DECODE_MAX_S_Q = DECODE_MAX_ROWS = 32 (the unpacked arm’s widest step; an MTP verify step of up to 8 tokens rides the one- and two-stream routes), and the 64-column tile’s cells, the prefill route, the token-unit cells past S_q = 4 and the unpacked arm are routed and declared, unmeasured (the 64-column tile is bitwise the two-unit 32-column form at the same split: the SDPA suite’s cross-form pin, repeated at the block level). The tail tile READS the pool rows past kv_lens[b] — the last page’s stale rows, and block 0 for the pages a short row does not list — and masks them to P = 0: they contribute nothing as long as they are FINITE, as a zero-initialized cache’s are (a NaN there is 0 x NaN). On the decode tile the plan is a split-KV plan whenever the SDPA heuristics’ decode split model says so (decode_split_kv=None: an under-full launch splits until its K / V streams saturate HBM; decode_split_kv=<int> pins it) — fp32 partials in the block’s workspace (get_workspace_size() holds them) and ONE combine launch — and fuse_gate=True then rides the split COMBINE: the tile has no gate seams, the combine multiplies the merged fp32 value by sigmoid(G) before the single cast (FROST’s one-rounding convention; the split is floored at 2 under the gate). On an unsplit decode tile the gate is its own launch, stage (5): blk.gate_fused says which runs; fuse_gate stays what was declared. On the prefill route the paged body runs UNSPLIT (decode_split_kv must be None or 1: a pinned split there is a typed decline) and fuse_gate=True rides its epilogue, as on the EXTEND form (gate_fused, no separate launch).

The decode split model behind decode_split_kv=None (cudnn.sdpa.fwd.heuristics.choose_decode_tile_split_kv, the same function the SDPA graph API uses for a standalone decode graph; the block hands it the route’s token units, the declared max_kv_len, the part’s SM count, the tile’s width and the step’s (B, H_q, S_q)) is fitted on the Rubin performance class: it prices a CTA’s KV stream per tile WIDTH (the 64-column tile streams a KV range in about three quarters of the time of two 32-column units, which is why the route takes it wherever two units would be needed), charges a launch of more CTAs than SMs its tail WAVE in full (a CTA streams its chunk serially, so 256 CTAs on 212 SMs cost two waves, not 1.2) and therefore offers the splits that fill one wave — 3 / 5 / 6 / 7 beside the powers of two — and prices the combine by the kernel the launch takes: the decode-grid combine kernel (flat in the split count) on a small grid from split 8 on and on any grid inside its bound from split 32, the lean kernel otherwise, the form pinned at compile. The block reads the model’s eager-safe choice (the split path’s second host launch charged against its saving); decode_split_kv stays the one performance knob of the form — the same function under every value, measured or not — so a replaying stack whose optimum differs pins it.

inputshape / dtypecontract
sample_k_cache, sample_v_cache (declaration)[num_pages, H_kv, P, D] views in the activation dtype: HND compact, NHD by strides, or the two halves of a unified [num_pages, H_kv, P, 2D] row (K at +0, V at +D), both of ONE layout kindREQUIRED under kv_source="paged": the strides are compiled into the SDPA’s TMA descriptors; execute’s pools must be these views (shape AND strides)
sample_block_table (declaration)[B, max_pages] int32, the page axis contiguous, ANY row stride — a row slice of a stack’s [num_reqs, max_blocks_per_req] table binds as a viewREQUIRED; max_pages * P bounds the logical KV length
max_kv_len (declaration)int, kv_lens[b] <= max_kv_len <= max_pages * Poptional; the masks’ clamp and the split model’s extent; None = max_pages * P
decode_split_kv (declaration)None or an int >= 1 (> 1 on the decode tile only; the prefill route runs unsplit)the one performance knob of the form: the same function under every value
decode_route_policy (declaration)None (= "single_unit") or "token_units"the route policy (the table above / the adapter’s token-unit route): a performance-only knob, the same function under either; DecodeRoute.policy reads it back, DecodeRoute.batch the batch the default policy’s prefill-route fill (DECODE_PREFILL_ROUTE_MIN_CTAS) was read at
fuse_norm_rope_small_m (declaration)None (= the lowering at B x S_q <= 128, unless fuse_cache_write=True asks for the fork’s write-through arm), True (the fused fork as declared) or False (the lowering, explicit; with fuse_cache_write=True a typed decline)whether the fused norm + RoPE projection runs at decode row counts when fuse_norm_rope=True: a performance-only knob; blk.fuse_norm_rope_effective reads back what runs; refused with fuse_norm_rope=False and off the dense DECODE form
decode_q_tile_max (declaration)None (= the whole table (16, 32, 64)) or a width of it (32 = the route before the 64-column tile landed)the width cap on the decode tile’s single-unit table: a performance-only knob, the same function under every width; DecodeRoute.q_tile_max reads it back and the compiled width is pinned against the route at compile(); a cap held below the width an unpacked step needs is a typed decline naming it
use_pdl (declaration)None (default = auto: True on the DECODE form’s decode-tile routes for the bf16 / f16 block on the parts of DECODE_PDL_DEFAULT_CCS — cc 10.7, the measured part — and on the block-sparse decode mode’s caller-list block on the parts of SPARSE_DECODE_PDL_DEFAULT_CCS — cc 10.7 since 2026-10-27 —, False everywhere else), True or False (explicit, honoured on every part)programmatic dependent launch over the chain: a performance-only knob (bitwise the plain chain, eagerly and under capture), bf16 / f16 only (an explicit True is a typed decline with quant, the e4m3 KV cache (kv_cache_dtype=torch.float8_e4m3fn), and with thd=True on a QsaSpec geometry; auto reads False there and never True where the explicit True is refused — pools in the activation dtype, kv_cache_dtype=None or that dtype, are the bf16 / f16 block). PER STAGE: the norm + RoPE pass, the separate cache write, the separate sigmoid gate and the decode tile with its split combine take it — or, on a QsaSpec geometry, the block-sparse core with its split combine (sdpa_sparse; served since 2026-10-21; auto True there on cc 10.7 since 2026-10-27 — SPARSE_DECODE_PDL_DEFAULT_CCS, the measured part — and False on every other part until measured; never under the in-block indexer, which declines the switch); the paged prefill tile of the prefill route carries no sites and launches plain (the link serializes, correct by PDL’s semantics), so auto stays False on that route; the FROST GEMM stages launch with the attribute regardless. blk.use_pdl is the resolved switch, blk.use_pdl_requested the declaration, blk.pdl_stages / blk.use_pdl_effective read back what launched with the attribute, DecodeRoute.pdl_capable says it per route, serving_envelope().decode.pdl_stages / .pdl_default (the latter for a served form: DECODE_PDL_DEFAULT_CCS on the dense form, SPARSE_DECODE_PDL_DEFAULT_CCS — cc 10.7, measured 2026-10-27 — under the block-sparse mode; the per-step reading is decode.pdl_default and decode.route(s_q, batch).pdl_capable on the dense form and decode.pdl_default itself under the block-sparse mode, whose core carries the sites at every served step width) and .extend.pdl_stages per form
fuse_cache_write_small_m (declaration)None (default = auto: the paged write-through folded into the norm + RoPE launch at B x S_q < CACHE_WRITE_SMALL_M_DEFAULT_MAX_ROWS = 128 rows or at S_q = 1, the separate launch at 128+ rows with S_q > 1 — the measured rule — and under an explicit fuse_cache_write=False, the separate launch asked for by name), True (folded in; a typed decline where the arm is not served; the strongest request, followed beside fuse_cache_write=False too) or False (the separate launch)where the paged write-through runs on the DECODE form’s UNFUSED chain (the lowered fuse_norm_rope=True declaration or fuse_norm_rope=False): bf16 / f16, no quant, no indexer band, 16-bit pools, the TMA norm + RoPE kernel; a performance-only knob (the pools and the slab bitwise either way, one launch fewer); blk.cache_write_in_norm_rope reads back what runs, the stage list loses "cache_write" under the arm
sparse_tile_form (declaration; a QsaSpec geometry only)"auto" (default), "lean" or "tcgen05"WHICH BODY of the index-list sparse core serves stage (4) on the block-sparse DECODE mode — the tcgen05 core (sparse_d256_f16) or the LEAN decode tile (sparse_lean_d256_f16: mma.sync + cp.async, 128 threads per m16 tile, no TMEM / TMA / CLC): a performance-only knob (the same O / LSE under either body within the fixed-order-sum budget; never bitwise across the bodies, bitwise within each). "auto" consults the placement rule (cc 10.7, B x H_kv <= 4 CTAs per chunk — the lean grid’s own unit: the step’s S_q rows ride inside the CTA as m16 tiles, so S_q changes no grid and is not in the bound —, top_k >= 8, the lean body admissible) only on the parts of heuristics.SPARSE_LEAN_TILE_DEFAULT_CCS — cc 10.7 since the end-to-end gate measured the lean route against the tcgen05 route there (2026-10-26, a 212-SM part at the clock lock, one process per cell, the control pair within 1 %: B 1 x 8K x S_q 1 34.8 -> 32.5 us per decode step (+6.8 %, 2.22 us; +5.9 % / 1.93 us symmetric with the arms built in the other order) and B 1 x 32K 36.6 -> 34.6 us (+5.7 %, 1.98 us; +5.6 % symmetric); LEAN_SPARSE_MAX_UNITS = 4 refit to the last class that wins inside its band; the unit is the CTAs per chunk since the same table’s S_q 4 rows: B 1 x S_q 4 wins +6.4 / +6.6 / +6.8 % at 2 per chunk, B 4 x S_q 4 loses -8.8 .. -9.7 % at 8), so "auto" is the lean tile on cc 10.7 inside the bound and the tcgen05 core on every other part and past the bound —; "lean" is honoured where the body serves the declaration (the DECODE mode at S_q <= 4 under the shared list — one list per sequence, every row of the step in one CTA — or the per-row list at one row per sequence, bf16 / f16, G <= 16, top_k >= 8; it ALWAYS splits: decode_split_kv None = the lean split of the default k, 65 at top_k 512, or one of heuristics.sparse_lean_splits(top_k) = 65 / 33 / 22 / 17 / 13 / 11 / 10 / 9, which SELECTS k) and a typed decline naming the arm elsewhere (per-row lists past one row — the lean CTA gathers one list per sequence —, S_q past 4, THD, the prefill form, a split that is not a lean split, top_k below 8); a pinned decode_split_kv that is not a lean split keeps "auto" on the core (the pin is the stronger knob). blk.sparse_tile_form_requested keeps the declaration, blk.sparse_tile_form the resolved body, blk.sparse_tile its kernel_template (pinned against the compiled kernel); serving_envelope().decode.sparse_tiles / .sparse_tile(s_q, batch, ...) / .sparse_tile_lean_default say so before anything is declared. Over e4m3 pools (kv_cache_dtype=torch.float8_e4m3fn: the gathered cast + the dense decode tile, no body of the sparse core) an explicit body is a typed decline naming the arm and blk.sparse_tile reads None (decode.sparse_tile(..., kv_cache_dtype=torch.float8_e4m3fn) too). Refused (ValueError) off a QsaSpec geometry unless "auto"
sparse_sub_tiles_per_cta (declaration; with sparse_tile_form="lean" only)None (default) or an int in 1 .. min(8, n_sub - 1)the lean tile’s k — its 32-key sub-tiles per CTA (the split is ceil(n_sub / k), n_sub = ceil((top_k + 2) x 4 / 32) = 65 at top_k 512): None = the default (heuristics.sparse_lean_sub_tiles_per_cta: 1 today — the k 2 / 3 / 4 renderings trace at 104-164 registers, a measured lever of the gate, not a default), an int pins it (with a pinned decode_split_kv the two must agree); a lean-only pin, refused by name (ValueError) under "auto" / "tcgen05" so it is never silently ignored. Performance-only: the same function under every k
index_select_form (declaration; the in-block indexer over the pools only)"auto" (default), "decode" or "extend"WHICH STEP FORM derives a QsaSpec(index_source="indexer") block’s selection over the paged side caches (2026-10-29): the DECODE step (the paged scorer stages the step’s query rows once per CTA, S_q x next_pow2(index_heads) <= 64) or the EXTEND step (a T-row chunk behind a cached prefix at any S_q <= max_kv_len: the chunk compress with the ring read-only, the dense kbar gather, the compact-logits scorer per row window, the canonical-order kernel; “The in-block indexer’s DECODE and EXTEND steps over the pools” below). "auto" takes the decode step inside its envelope and the extend step past it — never on the ring’s cap: a ring too short for the decode step at a small S_q is that step’s typed refusal, a stack pins "extend" there (cap >= 3); "decode" past the envelope keeps the typed indexer_query_rows decline. A performance-only knob (the sparse_tile_form precedent): the same selection function under either form up to the fp32 accumulation order of two scorers — bitwise WITHIN a form, never ACROSS. blk.index_select_form keeps the declaration, blk.index_select_form_resolved the form; serving_envelope().decode.index_select_forms lists the vocabulary. Refused (ValueError) off the indexer over the pools unless "auto"
block_table (execute)[B, max_pages] int32 at the declared strides; 0 = the NULL block of a padded entryREQUIRED; page ids are device data
kv_lens (execute)[B] int32 contiguous: the LOGICAL KV length of every sequence INCLUDING the step’s tokens; 0 = a padded entryREQUIRED; bound to the SDPA’s KV side ONLY (seq_kv_lens = kv_lens, seq_q_lens = None: every Q row of the step is live)
seq_lens (execute)—REFUSED under kv_source="paged" (the dense block’s self-attention length is a Q-side length; the KV length is kv_lens)
k_cache, v_cache, slot_mapping (execute)as under “Serving: write-through” (slot_mapping is [B * S_q])REQUIRED (the write-through is part of the form)

GatedAttentionBlockFwd.serving_envelope(geometry, dtype, device) -> ServingEnvelope is the STATIC record a serving stack reads at construction instead of catching exceptions: the forms served (forms, drawn from extend / decode) and one record per form (extend: ExtendEnvelope, decode: DecodeEnvelope — served, the reason it would decline with, every typed decline of the form in declines; the EXTEND record’s appended tail on a QsaSpec geometry — sparse_core, index_sources, index_list_forms, sparse_tiles, 2026-10-30), page_sizes (16 / 32 / 64 / 128), pool_layouts (the three stride patterns), the decode route per (group, S_q) (decode.routes for S_q 1 .. 33 — the served envelope and its first decline —, decode.route(s_q) for any step, decode.max_s_q = decode.max_rows = 32 / decode.claimed_max_s_q = 4, decode.policy / decode.q_tile_max; the policy and the width cap the record reads are serving_envelope(..., decode_route_policy=, decode_q_tile_max=)’s), the pool dtypes (dtypes: bf16 / f16 — an e4m3 pool is not served), the part and the CuTe DSL facts (device_cc, cutedsl_ok / cutedsl_reason) — derived from the same helpers the declaration and check_support run, so no decline is reachable at execute that the record did not predict (the execute-time refusals are the caller’s malformed tensors, form-checked by name). It never raises for a device the block does not serve. Typed declines of the form, at declaration: kv_source="paged" without paged_kv_page_size, a packed (thd=True) declaration (the EXTEND form is its own binding), a quantized quant other than the FP8-projection + bf16-attention mode (QuantSpec(attention_dtype=torch.bfloat16): SERVED since 2026-10-30, the section below), save_for_backward, a head dim other than 256, a sparse geometry on the PACKED form (a dense QsaSpec declaration is the block-sparse decode mode below), is_causal without causal_bottom_right, a pool or table sample off its contract, max_kv_len past the table’s reach, a step past DECODE_MAX_ROWS = DECODE_MAX_S_Q = 32 tokens (dense geometries; the decode_rows row), a decode_q_tile_max held below the width an unpacked step needs, a pinned decode_split_kv > 1 on the prefill route; kv_source="self" refuses every paged-read input rather than ignoring it.

On a QsaSpec geometry the same declaration is the block-sparse DECODE / extend-with-prefix mode (2026-10-14). GatedAttentionBlockFwd(sample_h [B, S_q, d_model], ..., geometry_with_qsa (causal_bottom_right=True), paged_kv_page_size=P, kv_source="paged", sample_k_cache=, sample_v_cache=, sample_block_table=): the step’s tokens are written through (4w) and stage (4) is the index-list sparse core READING THE POOLS — per sequence the keys of the listed 4-token blocks and the open tail over the first kv_lens[b] tokens of its block_table row, on the bottom-right diagonal (row j at pos_j = kv_lens[b] - S_q + j; a negative position — a sequence shorter than the step — is a dead row, kv_lens = 0 a dead entry: out = 0, LSE = -inf exactly). The list form is a DECLARATION fact — QsaSpec.index_list_form (appended; None = derived from S_q): "shared" — at S_q <= 4 ONE list per sequence, execute(block_ids=) [B, top_k] int32 (+ block_lens [B]), selected at the step-0 position kv_lens[b] - S_q and shared by the step’s rows (the visible set of “The decode mode and the speculative (MTP) rows” above: the listed blocks plus every token from the step-0 tail start to the row’s own position; the default under caller lists up to four rows, a typed decline past them); "per_row" — one list per row, [B, S_q, top_k] / [B x S_q, top_k] (+ block_lens [B, S_q] / [B x S_q]) at positions kv_lens[b] - S_q + j — the serving stacks’ per-row selection, a chunked prefill over a cached prefix — at ANY S_q (the default past four rows and under the in-block indexer over the pools; the sparse core runs one work item per query row: no 32-row cap, no decode-tile route). What differs from the dense form: sample_block_table must be CONTIGUOUS (the sparse core binds the whole [B, max_pages] table, and execute(block_table=) is exactly that table — a row slice of a wider table is a typed ValueError); return_lse is served ([B, H_q, S_q] fp32, natural log); fuse_gate is served at every split (the core’s epilogue unsplit, the split combine otherwise); decode_split_kv is None (the fill rule: ceil(SMs / (B x H_kv x S_q)) chunks, at most min(16, ceil((top_k + 1) / 32)) — the cap from the B 1 / B 4 x 32K x S_q 1 split sweeps on a 212-SM part with the pinned decode-grid combine; where that fill leaves a second wave of SMs (B x H_kv x S_q x split > SMs: a CTA streams its chunk serially, so 256 equal items on 212 SMs cost two item durations) the wave rule takes the split in [1, that cap] with the fewest waves x (tiles per chunk + 1), ties to the smaller — on a 212-SM part the T 128 step 3 chunks instead of none, B 32 x S_q 1 3 instead of 4, B 4 x S_q 4 6 instead of 7, every one-wave launch its fill-rule split) or an int in [1, ceil((top_k + 1) / 32)] — the one performance knob (fp32 partials inside get_workspace_size(), one combine launch), the same function under every value; max_kv_len keeps its contract (at most the table’s reach). Typed declines: a split past the item’s tile count, causal_bottom_right on the block’s OWN K / V (kv_source="self"); the packed (thd=True) form over the pools is the packed block-sparse EXTEND form since 2026-10-30 (“Serving: the extend form” above; the serving table’s sparse row stays reachable through a sparse-adapter record without the composition claim); the in-block indexer over the pools (index_source="indexer") and fuse_prepare on this mode are SERVED since 2026-10-21 (the next paragraph). The envelope’s decode.sparse_core names the stage, decode.requires spells both list forms, decode.routes is empty (the dense tile’s facts do not apply). Two bodies, one function (sparse_tile_form, the knobs table above): the sparse core is the tcgen05 tile (sparse_d256_f16) or, on sparse_tile_form="lean", the LEAN decode tile (sparse_lean_d256_f16: one mma.sync + cp.async CTA of 128 x N_MT threads per (sequence, KV head, chunk of k 32-key sub-tiles; N_MT = ceil(S_q x G / 16) m16 tiles of four warps each), no TMEM / TMA / CLC, the SAME fp32 split-major partials into the workspace, the same combine with the gate riding it) — at S_q <= 4 under the shared list (one list per sequence: every row of the step in one CTA, n_mt = ceil(S_q x G / 16) m16 tiles over one gathered sub-tile; the per-row list at one row per sequence is the same list), bf16 / f16, G <= 16, top_k >= 8, decode_split_kv None (the lean split 65 at the default k 1) or a lean split; "auto" takes the lean tile on cc 10.7 inside the placement bound (heuristics.SPARSE_LEAN_TILE_DEFAULT_CCS = ((10, 7),) since the gate measured the route end to end; LEAN_SPARSE_MAX_UNITS = 4 CTAs per chunk, B x H_kv: B 1 and B 2 on two KV heads at every served S_q 1 .. 4) and the tcgen05 core on every other part and past the bound; blk.sparse_tile reads the body back. Over e4m3 pools (kv_cache_dtype=torch.float8_e4m3fn) stage (4) is the gathered cast + the dense decode tile, no body of the sparse core: an explicit body is a typed decline and blk.sparse_tile reads None. On the standalone adapter an explicit split_kv=1 reads as unset (1 is its constructor default): under tile_form="auto" on a listed part it takes the lean split, so an unsplit launch there is pinned with tile_form="tcgen05" (the block’s decode_split_kv defaults to None and keeps the core for an explicit 1). The two bodies agree within the fixed-order-sum budget (the block suite: cos(out) > 0.999, the gated O within the suites’ dtype-specific output bound — per element |d| <= 2^-7 max|ref| + 2^-6 |ref| in bf16, |d| <= 2^-10 max|ref| + 2^-9 |ref| in f16, and cos >= 0.999 — and the LSE within 2e-2, the list bitwise), never bitwise across them. Validated on Rubin against the fp32 oracle over the whole sequence (test_block_qsa_decode.py: S_q 1 .. 4 shared and 8 / 64 per row, B 1 / 2 / 4, kv_lens up to 20000 with dead and shorter-than-the-step entries, pages 16 / 32 / 64 on all three layouts, bf16 / f16, splits 1 / 2 / 4 / 17 and the rule, the gate fused and separate, the LSE, the indexer band’s raw-key write-through, the 24 / 2 Flash-Next geometry; the MTP residues on a short sequence; the block’s form BITWISE the standalone adapter on the same views; the workspace sentinel; CUDA-graph capture of the first execute and the replay over new lengths / tables / lists / slots / inputs bitwise eager — on the four-band geometry and on the indexer-band one at N 13952, whose slab rides the small-M route’s 128-row tile; the band chain’s launch census one above the band-off chain’s, the raw-key write).

The in-block indexer’s DECODE and EXTEND steps over the pools (QsaSpec(index_source="indexer", index_band=True) with kv_source="paged"; bf16; the DECODE step 2026-10-21, the EXTEND step 2026-10-29). The block-sparse DECODE mode derives the step’s selection itself: declare the mode with the indexer geometry and THREE more samples — sample_index_kv_cache [n_pages, P / 4, 1, 128] (the PAGED compressed-key cache in the activation dtype: compressed row j of sequence b lives on page block_table[b, j // (P / 4)], row j % (P / 4); page 0 is the null page; a page holds its rows contiguously), sample_index_state_cache [blocks, cap, 1, 128] (the RING of raw indexer keys, position-indexed — the raw key of token position p of a sequence lives at row p % cap of its block; cap >= S_q + 3 on the DECODE step and cap >= 3 on the EXTEND step, the compress kernel’s rule min_ring_rows per form; block 0 is the null block) and sample_index_state_block_table [B, 1] int32 (the ring block per sequence; 0 for a dead entry) — and execute(..., w_iq_norm=, w_ik_norm=, index_kv_cache=, index_state_cache=, index_state_block_table=, cos_pos=, sin_pos=), the position tables [1 or B, >= max_pages x P, rope_dim] over the table’s whole reach (the table is the CEILING on positions: one row short is a ValueError naming the bound before any stage runs). block_ids / block_lens, index_k_raw (the raw key lives in the ring), index_k_compressed (the cache is paged) and n_commit are refused by name; the optional block_ids_out [B, S_q, top_k] / block_lens_out [B, S_q] hand the step’s selection back (a serving stack’s A/B against its own selector: the caller-list band block fed them is BITWISE this block). The list form is per row (index_list_form None or "per_row"; "shared" is the typed decline indexer_shared_list). The step: (1) the band’s indexer QUERIES, RMSNormed with w_iq_norm and rotated at the token’s position (bitwise the prefill form’s pool-1 compress); (2) every committed row’s raw key into the ring at pos % cap, and the row that completes a 4-token block pools the block’s four raw keys (the rows below the step from the ring, the rest from the step) through the compress chain — RMSNorm with w_ik_norm, partial RoPE at the block START position cos_pos[4k] — into the block’s cache row, BITWISE the prefill compress of the same keys (so a prefill-written row and a decode-written row are one chain); (3) the paged scorer (cudnn_block_qsa_score_paged) over each row’s visible complete blocks floor((pos + 1) / 4) through the table — relu(q_h . kbar_j) summed over the heads times 1 / sqrt(128); counts = the VISIBLE count, block_lens_out = min(top_k, counts); (4) the decode top-k (cudnn_qsa_topk_decode, the plan-time knobs from the row count and the table’s reach; ids strictly descending, -1 past min(top_k, counts)) into block_ids_out or the workspace; (5) the sparse core over the pools on the per-row list. A row at a negative position (kv_lens[b] < S_q) and a dead entry (kv_lens = 0, ring block 0, page 0) write nothing, score nothing (counts 0, every slot -1, out = 0, LSE = -inf). Launches: under fuse_prepare=True the prepare launch (frost_qsa_prepare) carries the norm + RoPE, the K / V write-through, the indexer-query norm and the ring / compress arm, and the stage adds the scorer and the top-k — eight kernels on the whole step with a split (the slab, the prepare, the scorer, the top-k, the sparse tile, the combine, the out projection and its split-K reducer); under False the norm + RoPE pass, the pool-1 query compress, the step compress (frost_qsa_compress_step) and the K / V write-through run as their own launches — eleven; None is the MEASURED default (serving_envelope(...).decode.fuse_prepare_default: True on cc 10.7 — the 2026-10-21 end-to-end gate on an exclusive 212-SM part measured the fused step +9.6 .. +14.5 % faster at B <= 4 and +4.4 .. +10.0 % at B 32 over 19 decode cells, bitwise —, False on every other part until measured, and False out of place — inplace_qkv=False — on every part: the AUTO reading is never True where an explicit True is declined); every output, the ring and the cache are BITWISE between the two readings. index_state_slot_mapping_present=True takes the ring’s write rows from execute(index_state_slot_mapping=) (block x cap + pos % cap, -1 on a dead row; int32 or int64 = slot_mapping’s dtype) instead of deriving them from the ring table — bitwise the derived form; rope_position_tables_only=True (fused only) refuses cos / sin and rotates Q / K and the indexer queries from cos_pos / sin_pos at the device-derived position — bitwise the per-token form; index_commit_present=True is a follow-up (a typed NotImplementedError). No plan constant, allocation or host read on the execute path: the FIRST execute of a fresh declaration captures into a CUDA graph without a warm-up, and the replay over a new step’s tensors is bitwise eager. The workspace carve is the caller-list band block’s plus five slots at the end (ix_q, the logits [T, max_pages x P / 4] fp32, the counts, the ids, the compressed-slot report). The envelope publishes the facts a serving stack reads (never re-spells): decode.index_sources (("caller", "indexer") at bf16 x index_head_dim 128), index_cache_dtypes ((bf16, e4m3): the e4m3 compressed-key cache served at decode since 2026-10-21, “At decode over the pools” above), index_head_dims ((128,)), index_kv_heads ((1,)), index_block_sizes ((4,)), index_top_k_range ((4, 512, 4)), index_query_rows_max (64: S_q x next_pow2(index_heads) rows per sequence, the DECODE step’s envelope), index_state_rows_extra (3: cap >= S_q + 3 on the DECODE step), index_select_forms / index_extend_ring_rows_min (3: cap >= 3 on the EXTEND step) / index_extend_select_budget_bytes / index_compress_roundings (the EXTEND step’s facts, 2026-10-29), index_page_sizes ((16, 32, 64, 128): P / 4 a power of two), fuse_prepare_default; decode.kv_cache_dtypes is (bf16,) under the indexer (the e4m3 pools wait for the sparse core’s in-kernel dequant arm: indexer_e4m3_pool). Typed declines by row name (decode.declines, each the text the declaration raises): indexer_query_rows (since 2026-10-29 under the explicit index_select_form="decode" pin only: "auto" takes the EXTEND step past the envelope), indexer_page_size, indexer_e4m3_pool, indexer_shared_list, fuse_prepare_rope_tables_unfused, indexer_f16, indexer_head_dim, indexer_thd (the indexer_e4m3_cache row retired 2026-10-21: the e4m3 cache is served), and the EXTEND step’s rows indexer_extend_fuse_prepare, indexer_extend_ring_slots, indexer_extend_rope_tables_only, indexer_extend_ring_rows, indexer_extend_chunk_rows (the last raised with the two numbers appended after the shared text); use_pdl=True under the indexer keeps a typed decline by name (the caller-list sparse chain serves the switch; the indexer’s step threads the switch it is handed and is handed False until its own switched chain is measured).

The EXTEND step (index_select_form="extend", or "auto" past the decode scorer’s envelope S_q x next_pow2(index_heads) <= 64; 2026-10-29). The same declaration at S_q = T is a T-row CHUNK of every sequence behind a cached prefix of P = kv_lens[b] - T tokens (P = 0: the first chunk, a prefill through the serving caches): the block selects for the chunk’s rows itself over the PREFIX’s compressed keys plus the chunk’s own complete blocks, and WRITES the serving caches — (1) the band’s indexer queries normed + rotated per token (the pool-1 compress; QsaSpec.index_compress_rounding names the chain’s rounding points); (2) the CHUNK COMPRESS, the step compress’s chunk form with the ring READ-ONLY (frost_qsa_compress_step(ring_write=False)): every chunk row that completes a 4-token block pools it from the ring (the straddling block’s rows below P) and the chunk’s rows, RMSNorm + partial RoPE at the block START from cos_pos / sin_pos, into the paged compressed-key cache through the main table — BITWISE the from-scratch prefill compress of the same keys; (3) the GATHER (cudnn_block_qsa_kbar_gather): every sequence’s min(floor(kv_lens[b] / 4), max_pages x P / 4) compressed rows through the table into a dense workspace kbar (zero rows up to ceil(T / 4) on a dead / short sequence), the per-row visible counts, the select pieces’ first query positions (clamped so no scorer write leaves its slab) and a per-sequence report; (4) the SELECT per row WINDOW — the DSA compact-logits scorer + the fused radix top-k (cudnn.ops.qsa_select_forward with the device q_pos0, deterministic) over at most R = min(T, max(16, QSA_EXTEND_SELECT_BUDGET_BYTES // (n_blocks_max x 4))) rows per launch pair (the 512 MiB module constant bounds ONE fp32 scratch slot reused window by window; ceil(B x T / R) windows: one at B 1 x T 2048 behind 128K, two at B 4); (5) the CANONICAL ORDER (cudnn_block_qsa_canonical_list, in place: the ids descending, -1 past min(top_k, counts), a dead row all -1) into block_ids_out or the workspace; then the K / V write-through + the RING arm — the chunk’s raw keys at positions >= kv_lens[b] - cap into the ring at execute(index_state_slot_mapping=) (the stack’s block x cap + pos % cap for its last cap tokens, -1 elsewhere; REQUIRED on this form), AFTER the chunk compress’s ring reads in stream order — and the sparse core per row. 12 launches at one select window (the slab, norm + RoPE, the four indexer launches + the scorer pair, two cache writes, the core, the out projection; + 1 split combine, + 2 per further window). Always unfused (fuse_prepare=None reads False; an explicit True is the typed indexer_extend_fuse_prepare: the chunk’s ring read and its tail write live in two launches); rope_position_tables_only=False (indexer_extend_rope_tables_only); index_state_slot_mapping_present=True (indexer_extend_ring_slots); the ring sample needs cap >= 3 (indexer_extend_ring_rows: the straddling block reads at most three positions below the chunk, distinct modulo any ring of that many rows — the kernel’s TRUE bound, not an asserted one; the DECODE step’s cap >= S_q + 3 does not apply; vLLM’s 4 satisfies it); S_q <= max_kv_len (indexer_extend_chunk_rows). THE RING INPUT CONTRACT: positions 4 floor(P / 4) .. P - 1 of every sequence already sit in the ring (at most three: the stack’s own rule token + cap >= end, or this block’s tail arm on the previous chunk / decode step) — a stack that primed fewer rows reads stale slots, which no device report can see. THE DEVICE-VALUE CONTRACT: kv_lens[b] in {0} u [S_q, max_pages x P] — 0 a dead / padding sequence (exact: every list slot -1, block_lens_out 0, out = 0 / LSE = -inf, no cache / ring / pool row touched); 0 < kv_lens[b] < S_q a violation REPORTED as -2 in the workspace’s per-sequence report (the live rows’ selection position-misaligned); kv_lens[b] > max_pages x P the other violation, -3 (the clamped query positions keep every scorer write inside its slab — memory-safe by construction — and the compress’s blocks past the table report -2); never raised, never read by the block. CACHE PARITY: a chunk-written row is BITWISE the from-scratch compress and bitwise a decode-written row of the same four keys — one chain under either rounding convention ("single" the reference; "hf" the HF / vLLM rounding points, not their bits, for a stack that mixes vLLM-written rows; the attribute is a compile key of every writer — the prefill form, the chunk form, the decode step unfused and fused — and the multi-step cell runs under both values). MIXED STEPS under the in-block indexer: a serving step with decodes AND prefills is TWO plans per layer — the decode step at S_q <= 16 and the extend step at S_q = T — executed back to back on one stream (disjoint sequences, the order free); ragged chunks are several extend executes grouped by T (front-padding a shorter chunk into a uniform T would pool garbage raw keys into prefix groups, which the block cannot detect). With CALLER lists the same step is ONE execute of the packed block-sparse EXTEND form (thd=True + kv_source="paged", 2026-10-30; “Serving: the extend form” above) — the indexer’s packed form is the follow-up that brings the one-execute step to the in-block selection. The workspace carve is the caller-list band block’s plus nine slots at the end (the queries, the slot report, the dense kbar [B, max_pages x P / 4, 128], the piece positions, the counts, the report, the ids, the scores, the window scratch R x max_pages x P / 4 fp32 — 256 MiB at B 1 x T 2048 behind 128K; get_workspace_size() holds all of it). The first execute of a fresh plan uploads the window plan’s constants, so a CUDA-graph capture takes the block after ONE eager warm-up (the prefill form’s rule; the decode step’s no-warm-up claim does not extend to this form). The envelope publishes decode.index_select_forms (("auto", "decode", "extend")), index_extend_ring_rows_min (3), index_extend_select_budget_bytes and index_compress_roundings (("single", "hf")).

Validated on Rubin against the two-stage oracle (the step’s set equals the fp64 top-k of the block’s own operands up to ties; the attention of every row within the budget of the fp32 oracle fed the kernel’s list; the ring rows and the completed cache rows bitwise the compress kernel; dead rows exact): S_q 1 .. 4 x B 1 / 2 / 4 x kv_lens 2 .. 32768 with dead and shorter-than-the-step entries, pages 16 / 32 / 64 on the three layouts, both knob readings bitwise with the launch census by kernel name, the identity region bitwise the caller-list block on the full list, the first-execute capture, the ring-slot and position-table forms, the MTP residues, the workspace sentinel, and eight carried decode steps after a prefill (a rejected draft re-stepped; the cache rows bitwise a from-scratch compress of the accepted stream after every step). Functional cells on a development node; the fusion’s default flipped to True on cc 10.7 after the exclusive-node end-to-end gate measured it (2026-10-21). The EXTEND step validated on a Rubin development node (cc 10.7; functional cells, no perf claim — the perf row is a later change’s; test_block_qsa_extend.py, test_block_qsa_end_to_end.py) against the same two-stage oracle for the chunk form: chunk starts at every residue of the block grid (incl. residue 2), a true shorter-than-the-remainder chunk (T < 4 - P % 4: no block completes, the tail alone enters the ring), T < cap chunks carrying the ring’s tail through the ring arm, the chunk-start pin (a row lists the straddling block iff it completes at or before that row), the identity rows + the first sparse row, two select windows at a lowered budget, the 128K-token reach, the e4m3 cache twins, the 24 / 2 geometry, the extend pin at decode widths against the decode form (the same function within the budget, never bitwise), cache parity under both roundings, the two device-value reports, the workspace honesty with no allocation and no host sync, the launch census by kernel name, the capture after one warm-up, and the 301-token prompt as chunks [125, 2, 1, 173] followed by eight carried decode steps.

Serving: FP8 projections with bf16 attention (QuantSpec(attention_dtype=torch.bfloat16))

A serving stack whose attention projections are static per-tensor FP8 W8A8 while its KV cache and attention math are bf16 — qkv_proj and o_proj each an FP8 linear with one input scale and one weight scale, the pools bf16, the o_proj INPUT an e4m3 tensor at the o_proj input scale — declares the paged serving forms with e4m3 h / W_qkvg / W_o and QuantSpec.bf16_attention(descale_h, descale_w_qkvg, descale_w_o, scale_o) (= QuantSpec(..., scale_q=None, scale_k=None, scale_v=None, ..., attention_dtype=torch.bfloat16); appended 2026-10-30; served on cc 10.7 and cc 10.0, FP8_PROJ_BF16_ATTENTION_CCS). The pipeline:

(1) proj h8 x W8_qkvg -> fp32 -> * alpha_qkvg -> bf16 slab the FP8 FROST GEMM (the forced 256-wide tile, the alpha epilogue): the unfused FP8 pipeline's stage (1)
(2+3) norm+RoPE in place on the slab's bf16 Q / K bands unchanged; the DECODE form's write-through fold (fuse_cache_write_small_m) admitted
(4w) write post-RoPE K / V rows -> the bf16 pools at slot_mapping unchanged (the pools STAY bf16)
(4) attention the bf16 d256 decode tile [+ split combine] / the paged d256 prefill tile over the pools; Q / GATE = the slab bands = the bf16 serving forms, route for route
(5) gate in the combine (cc 10.7, the decode tile at split >= 2), in the paged body's epilogue (cc 10.7, fuse_gate=True), its own launch otherwise
(5q) quantize_o o8 = cvt.rn.satfinite(bf16 O_gated * scale_o) the one quantize launch of the mode (kernels/quantize.py)
(6) out_proj o8 x W8_o -> fp32 -> * alpha_o -> bf16 out, alpha_o = descale_w_o / scale_o the FP8 FROST GEMM

Launch counts, MEASURED by the CUPTI census cell (test_fp8_proj_bf16_attention_launch_census_is_the_bf16_chain_plus_the_quantize_launch, the 24 / 2 geometry at d_model 2560, B 2 x KV 512, S_q 1, the split ladder pinned at decode_split_kv=4 — the decode tile + its combine on every part —, no numerical reference; every kernel the profiler sees is counted and the per-arm totals are pinned, so a tree that changes a stage’s kernel count turns the cell red and these numbers follow it): cc 10.7 — six with the gate in the combine and the write-through folded, seven with the separate cache write, seven / eight with the separate gate (fuse_gate=False); cc 10.0, where the gate is always its own launch — seven folded / eight with the separate cache write (blk.cache_write_in_norm_rope / blk.gate_fused read the arms back). These are the bf16 serving chain’s own counts for the same declaration: both FP8 forced-tile GEMMs run as one kernel each where the bf16 chain’s small-M out projection runs as a split-K kernel plus a reduce kernel, so the mode’s quantize launch takes the reduce’s place; no memset or memcpy on either chain. scale_q / scale_k / scale_v have no quantization point under the mode and are refused BY NAME if set; no descale reaches the SDPA; the workspace carves no compact e4m3 Q / K / V (o8 alone); MxQuantSpec has no such attribute.

The static-scale mapping (a stack’s per-tensor FP8 W8A8 attention projections): descale_h = the qkv projection’s INPUT scale (x = x8 * s_in); descale_w_qkvg = the merged qkv weight’s ONE per-tensor weight scale (a stack that merges three per-matrix scales into one requantized scale hands that one — the block has one alpha, three distinct scales are not its contract); descale_w_o = the o projection’s weight scale; scale_o = 1 / the o projection’s INPUT scale (the stored e4m3 byte decodes as byte * s_in_o). The merged weight’s codes go through qkvg_from_vllm / qkvg_from_hf as e4m3 (the permutation runs on the bytes, the per-tensor scale is unchanged by it; per-tensor scales only). Host / device contract: the scales are Python floats at declaration (finite positive, validate()), THREE 1-element fp32 device tensors (alpha_qkvg, alpha_o, scale_o) allocated at compile(), written on the launch stream at the first execute and by update_quant_scales (which refuses a differing attention_dtype as a plan fact); no host readback, no device read on the host path; a captured FIRST execute pins the captured values — warm up before capturing (the rule above).

Numerics. The block’s byte is cvt.rn.satfinite(bf16(O32 * sigmoid(G)) * scale_o) on the combine / epilogue routes (one bf16 rounding of the gated value) and cvt.rn.satfinite(bf16(bf16(O32) * sigmoid(G)) * scale_o) on the elementwise-gate routes; a serving stack’s gate-and-quantize convention, e4m3(bf16(bf16(O) * sigmoid(G)) * inv) with inv = scale_o, is the same class — within ONE e4m3 ulp at the larger magnitude wherever one e4m3 ulp exceeds the chain’s ABSOLUTE noise floor — the e4m3 value band |x| >= 4, where one ulp is >= 0.5: near zero the two chains differ not by a rounding of a shared value but by the kernels’ tanh-form sigmoid against torch’s for strongly negative gates and by the attention’s own accumulation order, an absolute floor that is many ulps of the 2^-9 subnormal spacing, so below the band the distances are reported, never claimed, and no tolerance is widened; never bitwise; the oracle of the cells replicates the mode’s one quantization point and the bound is the bf16 serving forms’ own (cos(out) > 0.999, never the fp8 suite’s 0.99). A padded entry (kv_lens = 0) stores the byte 0 and its out row is exactly 0; the satfinite store never produces NaN / inf. Validated on a Rubin development node (cc 10.7, 204 SMs, shared, functional only — no performance number is claimed for the mode: the FP8 forced-tile GEMM at decode row counts is a FUNCTIONAL claim; the tracker’s “FP8-projection + bf16-attention mode” paragraph holds the run record) and on the SM100 development part (cc 10.0, 148 SMs, shared, functional only: the same cells with the gate as its own launch).

Served (each an accept cell): kv_source="paged" on the dense DECODE form (every route the bf16 form takes) and the packed EXTEND form; pages 16 / 32 / 64 / 128, every pool layout, inplace_qkv None / True, fuse_norm_rope=False, fuse_gate as the bf16 form reads it, fuse_cache_write_small_m None / True / False on the DECODE form, decode_route_policy / decode_q_tile_max / max_kv_len / decode_split_kv as before, update_quant_scales, CUDA-graph capture of the first execute (both projections on the capture-safe jit-only route), serving_envelope(..., quant=spec) (each form’s quant_modes names the spelling served on the part). Refused by name (each a typed NotImplementedError / ValueError with its reject cell): kv_source="self" (dense or packed over the block’s own K / V, with or without the write-through), fuse_norm_rope=True (the fused fork writes compact e4m3 Q / K / V, no bf16 slab; fuse_norm_rope_small_m / an explicit fuse_cache_write=True fall with it), inplace_qkv=False, save_for_backward / return_lse (an inference form), kv_cache_dtype=torch.float8_e4m3fn (the pools stay bf16), a QsaSpec geometry, use_pdl=True (the quantize launch carries no griddepcontrol sites; AUTO reads False), num_tokens_live (the FP8 GEMMs’ live-row launch), MxQuantSpec and QuantSpec(attention_dtype=None) on the paged forms (no paged leg: the write-through’s decline names the mode as the way in), attention_dtype=torch.float16, 16-bit h beside the QuantSpec (FP8 needs both halves), and the mode on a served dense part off FP8_PROJ_BF16_ATTENTION_CCS (cc 10.7 and cc 10.0 today — the cc 10.0 admission is the FP8 forced-tile GEMM cell of test_block_fp8_bf16_attention.py plus the mode’s DECODE / EXTEND cells, run 2026-10-30 on the SM100 development part, functional only; a further part lifts in the same way: the tuple gains it together with the mode’s cells on it). Follow-ups (not in this change): the fused gated e4m3 store in the dense decode tile’s split combine (the sparse adapter carries it; the dense adapter has no out_scale yet), the FP8 small-M GEMM arm at decode row counts, the mode over the block’s own K / V (the dense and packed prefill forms), an e4m3 KV cache under the mode, use_pdl=True, num_tokens_live.

Backward

GatedAttentionBlockBwd(sample_dy, sample_saved, sample_w_qkvg, sample_w_q_norm, sample_w_k_norm, sample_cos, sample_sin, sample_w_o, geometry, *, recompute=RecomputePolicy.RECOMPUTE_QK_PRE, need_dh=True, need_dw_qkvg=True, need_dw_o=True, need_dw_norms=None, seq_lens_present=False, dw_norm_dtype=torch.float32, fuse_gate_bwd=False, fuse_wgrad_overlap=False, thd=False, num_sequences=None, max_seq_len=None, cu_seqlens=False, quant=None, grad_scaling="current", grad_scale_margin_log2=0) is the block backward: eight stages on ONE launch stream (the two weight-gradient GEMMs on a block-owned side stream under fuse_wgrad_overlap, joined back before execute returns), no allocation, against the forward’s SavedForBackward(h, gate, o, lse, rstd_q, rstd_k, q_pre=None, k_pre=None, proj_slab=None, seq_lens=None) record written in the proj_slab save mode (GatedAttentionBlockFwd(save_for_backward=True), the default saved_gate_copy=False; gate / q_pre / k_pre may be None there — they are bands of proj_slab). Recipe, the same lifecycle as the forward:

from cudnn.gated_attention_block import GatedAttentionBlockBwd
bwd = GatedAttentionBlockBwd(dy, saved, w_qkvg, w_q_norm, w_k_norm, cos, sin, w_o, geometry) # every need_* True
bwd.check_support()
bwd.compile() # the artifacts first: get_workspace_size() needs them
workspace = torch.empty(bwd.get_workspace_size(), dtype=torch.uint8, device=dy.device)
dh, dw_qkvg, dw_o = torch.empty_like(saved.h), torch.empty_like(w_qkvg), torch.empty_like(w_o)
dw_q_norm, dw_k_norm = (torch.empty(geometry.d_head, dtype=torch.float32, device=dy.device) for _ in range(2)) # fp32
bwd.execute(dy, saved, w_qkvg, w_q_norm, w_k_norm, cos, sin, w_o, dh=dh, dw_qkvg=dw_qkvg, dw_o=dw_o,
dw_q_norm=dw_q_norm, dw_k_norm=dw_k_norm, workspace=workspace) # + current_stream=, seq_lens=None

gated_attention_block_backward(dy, saved, w_qkvg, w_q_norm, w_k_norm, cos, sin, w_o, geometry, *, seq_lens=None, recompute=..., current_stream=None, fuse_gate_bwd=False, fuse_wgrad_overlap=False, thd=False, max_seq_len=None, quant=None, grad_scaling="current", scale_dp=None, scale_dy=None, scale_do=None, scale_dqkvg=None, h_t=None, h_t_sf=None, w_qkvg_t=None, w_qkvg_t_sf=None, w_o_t=None, w_o_t_sf=None, grad_scale_margin_log2=0) allocates the gradients and the workspace on the launch stream (current_stream, else torch’s current stream — the caching allocator orders a buffer’s reuse only against the stream it was allocated on), caches the compiled block per declaration and returns {"dh", "dw_qkvg", "dw_o", "dw_q_norm", "dw_k_norm"}; which entries exist follows requires_grad on saved.h / w_qkvg / w_o / w_q_norm / w_k_norm (the tensors are handed to the block detached). Under thd=True it derives num_sequences and cu_seqlens from the record (saved.seq_lens.numel(), saved.seq_lens_form) and requires max_seq_len (a ValueError naming it alone otherwise); both, with max_seq_len, are part of its cache key. quant / grad_scaling (the quantized backward below) join the key too, and scale_dp / scale_dy / scale_do / scale_dqkvg and the transposed artifacts h_t / h_t_sf / w_qkvg_t / w_qkvg_t_sf / w_o_t / w_o_t_sf pass through to execute; the gradients are allocated in dy’s dtype (bf16 under quant, where saved.h and the weights are e4m3 codes).

Packed sequences (THD). GatedAttentionBlockBwd(..., thd=True, num_sequences=B, max_seq_len=S_max, cu_seqlens=False) — the forward’s four knobs, appended last — differentiates the packed training record of the section above: dy is [T, d_model] or [1, T, d_model], cos / sin the per-token tables, dh comes out in saved.h’s shape (the forward’s h, [T, d_model] or [1, T, d_model]; the two ranks may differ between dy and h). saved.seq_lens is REQUIRED (the [B] / [B+1] int32 tensor the forward ran with; validated on the host: dtype, rank, element count per cu_seqlens, device, contiguity — never its values) and saved.seq_lens_form must match the block’s form: a padded DENSE record (seq_lens_form=None) is refused by a packed backward, a packed record by a dense one. execute(seq_lens=) is optional and, when given, must be saved.seq_lens itself. Every token-wise stage is the dense B=1, S=T block’s (the two weight-gradient GEMMs contract over K = T; the two data-gradient GEMMs produce T rows and contract over d_model and n_qkvg); the SDPA backward runs the packed d=256 chain (SdpaBwdDslSm107(thd=True): its own setup launches, per-sequence descriptors, the kv-blocked dS workspace, the GQA fold bounded by the live total on the device). fuse_wgrad_overlap and fuse_gate_bwd are both served, each bitwise the plain packed block: the gate backward’s delta at B = 1, S = T is byte for byte the packed head-major [1, H_q, ceil128(T)] delta the packed chain reads (tail zeroed), so under fuse_gate_bwd the chain’s own dot_do_o launch is gone and its delta region moves into the block’s (test_thd_fused_gate_bwd_is_bitwise_the_unfused_packed_block). The declaration bounds and the lengths contract are the forward’s. Workspace: the dense B=1, S=T carve plus the packed chain’s scratch — delta [1, H_q, ceil128(T)] (unless fuse_gate_bwd), ONE head chunk of the kv-blocked dS, [qh_chunk, ceil256(T + 256 B), ceil128(max_seq_len)] in the activation dtype, the metadata and descriptors, the GQA partials — so declare max_seq_len tight.

What runs (launch order, T = B*S): the out-projection dgrad dO_gated = dY @ W_o; the sigmoid-gate backward (dO, dG into the GATE band of the [T, N] dqkvg slab, O_gated for the wgrad); the out-projection wgrad dW_o = dY^T @ O_gated; the recompute of the post-norm / post-RoPE Q, K from the saved slab (the forward’s norm+RoPE kernel) and a compact copy of V; the Rubin d=256 SDPA backward (SdpaBwdDslSm107) into compact dQ / dK / dV; the fused RoPE-adjoint + RMSNorm backward writing the Q / K / V bands of dqkvg plus fp32 dW_norm partials and their fixed-order reduce; the projection wgrad dW_qkvg = dQKVG^T @ h and dgrad dh = dQKVG @ W_qkvg. 12 + c*(2+q) kernel launches (c = the SDPA backward’s head chunks, q = its dQ GEMM launches per chunk: 1 under its single-launch dQ rendering, the GQA ratio under the per-member twin; more when S % 128 != 0) — 15 / 22 at the test geometry (S=256 / S=1000), counted by CUPTI (test_launch_count_is_honest, on Rubin cc 10.7). Deterministic by construction: no atomics anywhere, and the four GEMMs run the block’s forced 256-wide N tile at one split-K slice (compile() refuses a heuristic fallback typed; check_support declines a d_model % 256 != 0 geometry) — two runs are bitwise equal, pinned on Rubin by test_two_runs_are_bitwise and test_a_caller_stream_orders_every_stage. need_* fixes at build time which GEMMs exist (an output given for a need_*=False, or missing for a need_*=True, is a typed error at execute); need_dw_norms=None follows geometry.qk_norm, and asking for norm-weight gradients under qk_norm=False is a typed decline. RecomputePolicy (SAVE_ALL / RECOMPUTE_QK_PRE) both read the slab’s bands today; RECOMPUTE_GATE is reserved.

Fusion knob — fuse_gate_bwd (default False). The SDPA backward’s first launch is delta = rowsum(dO * O) over the dO the gate backward just wrote and the O it just read. With the knob on, the gate-backward kernel emits delta as a fourth output (the bf16 / fp16-rounded dO it stores, summed in the chain’s own dot_do_o order, the pad rows zeroed) into a block-owned fp32 [B, H_q, S_pad] region ([1, H_q, ceil128(T)] under thd: the dense arm at B = 1, S = T is the packed chain’s own head-major layout), and the SDPA backward adapter is built with external_delta=True and reads that tensor: one launch and one read each of O and dO fewer (11 + c*(2+q) launches, 14 / 21 at the test geometry), the adapter’s own delta region gone from its scratch (the block’s region takes its place, same bytes). Performance-only in the strict sense: the gradients are bitwise the unfused block’s (test_fused_gate_bwd_is_bitwise_the_unfused_block, bf16 and fp16, dense and causal, B=1 and B=3 under GQA; packed: test_thd_fused_gate_bwd_is_bitwise_the_unfused_packed_block), and the two-run and stream-order pins run under both knob values. Measured whole-backward effect: see the performance section.

Scheduling knob — fuse_wgrad_overlap (default False). The two weight-gradient GEMMs are consumed by nothing inside the backward: dW_o = dY^T @ O_gated is ready after the gate backward, dW_qkvg = dQKVG^T @ h after the norm/RoPE backward. In order they sit on the launch stream between stages they do not feed, so their SM time is serialized with the SDPA backward chain (below full SM occupancy at small S, with launch gaps between its kernels) and with the dh dgrad. With the knob each of them is issued on a block-owned side stream instead: the side stream waits an event recorded on the launch stream right after the GEMM’s producer stage (fork), and the launch stream waits an event recorded on the side stream after the GEMM (join) at the end of execute — the latest legal point, because the side GEMMs read only buffers no later stage writes (dy, O_gated, the finished dqkvg slab, saved.h) and get their own appended scratch region (gemm_scratch_side, sized to the two wgrad plans; the forced tile at one split-K slice carves no scratch, so it is 256 B at the 397B geometry) rather than sharing the in-order GEMMs’ gemm_scratch with the dh dgrad they now run next to. Every write the caller can observe is therefore still ordered on the launch stream before execute returns: an ambient or an explicit launch stream sees the whole backward exactly as without the knob (the stream-order probe test_a_caller_stream_orders_every_stage runs under it, both arms), the same kernels launch (the CUPTI count is unchanged), and the gradients are bitwise the in-order block’s (test_fuse_wgrad_overlap_is_bitwise_the_in_order_block: bf16 and fp16, dense and causal, B=1 and B=3 under GQA, and composed with fuse_gate_bwd) — the GEMMs are deterministic, so moving them to another stream moves no bit. The side stream is the block’s own: one dedicated non-blocking stream at the device’s lowest priority (a filler below any launch stream), created through the driver at compile() together with the four events and released with the block — nothing per execute, and never a torch pool stream, so never a caller’s launch stream. One compiled block may be driven from several host threads on different launch streams (the fork, the side GEMM’s enqueue and the join record are one locked section of host-side enqueues, and the join’s check and wait take the same lock); the convenience wrapper’s per-call workspace, freed at return, is reused only behind the join. CUDA-graph capture of execute works under the knob: the fork and the join are recorded as graph edges (the side stream joins the capture and is joined back before it ends), and the replay is bitwise the eager run (test_cuda_graph_capture_replays_bitwise, both knob values). The two do not compose on one block: a CUDA-graph capture of execute must not overlap an execute of the same compiled block from another thread, and two concurrent captures of one block are likewise unsupported — the block has ONE side stream, which belongs to the capture from the capture’s first fork until its join, so an eager fork or join onto it in that window (or a second capture’s fork) is a typed RuntimeError naming the situation, raised before the capture is touched, instead of the capture’s invalidation or the eager stream silently joining the graph (test_fuse_wgrad_overlap_capture_and_eager_executes_do_not_overlap); the capturing thread’s own forks and joins pass. A caller that mixes the two on one block (the convenience wrapper caches one block per declaration for the process) serialises each capture against the block’s eager executes. The knob is a typed ValueError at check_support when neither need_dw_o nor need_dw_qkvg is set (nothing to overlap); the convenience wrapper, whose needs follow requires_grad, runs the in-order block instead when the weights are frozen (a frozen-weights training step must not fail over a scheduling knob). Measured whole-backward effect: see the performance section.

Quantized backward (per-tensor fp8) — quant (default None). GatedAttentionBlockBwd(..., quant=QuantSpec, grad_scaling="current"), with the per-tensor fp8 training forward’s own QuantSpec, differentiates the quantized training record as written — e4m3 saved.h, e4m3 w_qkvg / w_o, the bf16 slab / o / lse / rstd — from a bf16 dy, and returns bf16 dh / dw_qkvg / dw_o and fp32 dw_*_norm. What runs (launch order): the fused PROLOGUE launch — four independent jobs dispatched by block range: the scalar-block init (every slot zeroed, descale_dp, and the QuantSpec’s plan-time constants stored from the launch’s kernel arguments), the amax of dY as per-CTA partials, the recompute of the post-norm / post-RoPE Q, K with an e4m3 epilogue (Q8 / K8 at the forward’s static scale_q / scale_k — kernel arguments of that launch, the values of their slots —, bit-exactly the forward’s own SDPA operands: the bf16 rounding first, then the cast, so no bf16 rebuild is written), and V8 straight from the slab band at scale_v (no compaction launch); the e4m3 quantize of dY, which reduces the partials and publishes amax_dy; the e4m3 out-projection dgrad dO_gated = dY8 @ W_o8 * alpha; the gate backward’s fp8 arm (dO, dG, the e4m3 O_gated for the wgrad, delta = rowsum(dO * O) — always, it is the fp8 SDPA backward’s external delta —, and per-CTA partials of the stored dO’s and dG’s max |.|, one plain store each from a persistent grid); the e4m3 quantize of dO, which reduces the dO partials and publishes amax_do; the e4m3 out-projection wgrad dW_o = dY8^T @ O_gated8 * alpha; the Rubin d=256 per-tensor fp8 SDPA backward (SdpaBwdDslSm107Fp8, external delta, amax_dP requested) into bf16 dQ / dK / dV; the fused RoPE-adjoint + RMSNorm backward, storing per-CTA partials of the Q / K / V bands’ max |.|; the fused EPILOGUE launch — the fixed-order dW_norm reduce (one column per block: the same fp32 chain as the standalone reduce) and the e4m3 quantize of dQKVG, every block reducing amax_dqkvg from the dG and band partials and the first publishing it; the e4m3 projection wgrad dW_qkvg = dQKVG8^T @ h8 * alpha and dgrad dh = dQKVG8 @ W_qkvg8 * alpha. Every gradient amax is a max over per-CTA partials its producer stored (order-free, bitwise the standalone amax pass; the block’s own kernels run no atomic — the per-warp atomicMax into one slot the folds first shipped as serialised at the L2 and cost the gate backward 0.7 ms at S = 32K), and the two fused launches are bitwise the standalone chain they replaced. Every e4m3 GEMM runs the block’s forced tile at its 64-byte MMA K form with a bf16 output and an fp32 alpha = descale_A * descale_B epilogue read from a device slot. 12 + c*(2+q) kernel launches — 15 at the test geometry (Q/K RMSNorm on, GQA; 15 RoPE-only, the epilogue launch stays for the quantize; 15 MHA: the SDPA backward’s dK fold shares the dV fold’s launch under GQA, and its setup is one launch; +3 when S % 128 != 0, the q-side staging pads, and +2 when S % 256 != 0, the kv-side pads: 20 at a padded causal S such as 992 or 1000, 17 at S = 384; 17 before the SDPA backward merged its setup and fold launches, 24 before the launch fusion), counted by CUPTI in the quantized backward’s own suite. Gradient scales (grad_scaling, a declaration attribute — it moves the e4m3 rounding points, so it is never a knob): "current" derives every gradient’s per-tensor scale ON DEVICE from its own amax pass in this step (2**(floor(log2(448 / amax)) - FP8_GRAD_SCALE_MARGIN_LOG2), with FP8_GRAD_SCALE_MARGIN_LOG2 = 0); the margin is the appended declaration attribute grad_scale_margin_log2 (an int in [0, 8], the module constant by default): a compile-time constant of EVERY gradient quantize — dY, dO, dQKVG here, dY alone under MxQuantSpec — so a non-default value is a different compiled block (never a knob, never a scalar slot), refused without quant and under "delayed" (the caller’s scales carry their own headroom there); quant_scalars() publishes the scales that ran; "delayed" reads the previous step’s scale_dy / scale_do / scale_dqkvg from execute(...) instead (each a 1-element fp32 CUDA tensor, required there and refused under "current") while the amax passes still publish this step’s amax. The softmax scale scale_s = 2**FP8_SCALE_S_LOG2 (= 2**8) is a module constant; scale_dp — the fp8 SDPA backward’s dP scale, cuDNN’s contract — is the caller’s 1-element fp32 CUDA tensor at execute(scale_dp=), required under quant and refused without, and its descale is derived from it on device. bwd.quant_scalars(workspace) returns zero-copy 1-element fp32 views of the block’s scalar region, named amax_dy / amax_do / amax_dqkvg / amax_dp, scale_dy / descale_dy / scale_do / descale_do / scale_dqkvg / descale_dqkvg, alpha_b1 / alpha_b2 / alpha_b7 / alpha_b8, descale_dp, and the QuantSpec’s plan-time constants (scale_q / scale_k / scale_v / scale_o, descale_q / descale_k / descale_v / descale_o, descale_w_o / descale_h / descale_w_qkvg, scale_s / descale_s, scale_dqkv) — stored by the step’s FIRST launch (the prologue’s scalar-init job) from its kernel arguments, so nothing is written to the device at compile() and an execute on any stream reads only what that stream wrote (synchronise the stream after the step before reading them; they are rewritten by the next execute). Workspace: the bf16 O_gated, recomputed Q / K and compact V regions are replaced by the e4m3 dY8 / dO8 / O_gated8 / Q8 / K8 / V8 / dQKVG8 plus the 256-B scalar block and the fp32 dY amax partials (one word per CTA of the prologue’s amax job, at most SMs x 8) — about +12 KiB/token at the 397B geometry —, the delta region is always carved, and the SDPA scratch is the fp8 chain’s (its e4m3 dS chunk is half the bf16 one). Determinism: the block’s own kernels run no atomic (every gradient amax is a max over per-CTA partials); the one atomicMax left is the fp8 SDPA row’s amax_dP, an int32 fold of non-negative fp32 bit patterns and therefore order-free — so two executes are bitwise equal under every knob set (pinned by test_fp8_two_runs_are_bitwise). fuse_gate_bwd is accepted and inert under quant: the fused delta is mandatory there. fuse_wgrad_overlap is served (the side-stream GEMMs fork after the slots and operands they read are written). Packed sequences (thd=True, the THD section below) are served over the packed per-tensor fp8 training record as written: every stage but the SDPA row is token-wise at B = 1, S = T, and the fp8 SDPA row runs its THD chain with the gate backward’s delta as its external one — at B = 1, S = T that is the packed head-major [1, H_q, ceil128(T)] delta the packed chain reads — so no dot pre-pass runs over the e4m3 payloads (one rounding of dO, the dense arm’s delta contract). Declined (typed, naming the attribute): e5m2 codes, an fp16 dy (the quantized backward is bf16), a bf16 saved.h or bf16 weights with a QuantSpec and e4m3 codes without one (both ways), and a geometry whose Q / K rebuild only the LDG norm + RoPE kernel can tile: the fused prologue runs the TMA kernel, whose tile_rows must divide h_q, be a multiple of h_kv and of 4 (nothing in 1..16 does for h_q = 20 MHA or h_q = 6 over h_kv = 2; the bf16 backward serves such a geometry through the LDG rebuild). There is no B*S rule: the two weight-gradient GEMMs contract over the token axis with MN-major e4m3 operands, and the TMA 16-byte rule binds an operand’s contiguous axis only, so a ragged token count (S = 1000 at B = 1) is served with its weight gradients. Every bf16 decline (padding, window_left == 0, d_model % 256, Rubin only, …) is unchanged.

Quantized backward (MXFP8) — quant=MxQuantSpec. GatedAttentionBlockBwd(..., quant=MxQuantSpec, grad_scaling="current"), with the MXFP8 training forward’s own MxQuantSpec (descale_w_o, scale_o: the per-tensor pair of the out projection, the one per-tensor side of that pipeline), differentiates the MXFP8 training record as written — e4m3 saved.h and weights, the bf16 slab / o / lse / rstd — from a bf16 dy, and returns bf16 dh / dw_qkvg / dw_o and fp32 dw_*_norm. What runs (launch order, one stream; the per-tensor fp8 chain’s shape — two fused small-kernel launches and one dual-axis quantize, every byte bitwise the unfused chain’s): the fused PROLOGUE — the scalar-block init (every slot zeroed, the MxQuantSpec’s plan-time constants from the launch’s kernel arguments; no descale_dp — the MXFP8 SDPA backward has no dP scalar), the amax of dY as per-CTA partials, the post-norm / post-RoPE rebuild of Q and K from the slab on a one-head x 32-token tile with the MXFP8 quantizes of both written straight out of registers, rowwise AND columnwise (q8 / sf_q, q_T8 / sf_q_T, k8 / sf_k, k_T8 / sf_k_T; q8 / sf_q and k8 / sf_k bitwise the forward’s own; no bf16 rebuild buffer), and the rowwise quantize of V straight from the slab’s V band (the backward’s V operand is rowwise, so the forward’s columnwise v8 cannot serve) —; the per-tensor e4m3 quantize of dY — the ONE per-tensor gradient of this pipeline, at the grad_scaling recipe’s scale ("current" derived on device, "delayed" the caller’s execute(scale_dy=)); the e4m3 out-projection dgrad dO_gated = dY8 @ W_o8 * alpha; the gate backward’s fp8 arm (dO, dG, the e4m3 O_gated at scale_o for the wgrad, delta = rowsum(dO * O) — always, it is the MXFP8 SDPA backward’s external delta); ONE dual-axis MXFP8 quantize of dO — rowwise (the row’s dP operand) and columnwise (its dV operand) from one read, in the SDPA’s own scale-factor layouts —; the e4m3 out-projection wgrad dW_o = dY8^T @ O_gated8 * alpha; the Rubin d=256 MXFP8 SDPA backward (SdpaBwdDslSm107Mxfp8, external delta, its block-scaled dS chain) into bf16 dQ / dK / dV; the fused RoPE-adjoint + RMSNorm backward; the fused EPILOGUE — the dW_norm reduce and the dual-axis MXFP8 quantize of dQKVG from one read, in the GEMMs’ canonical F8_128x4 scale-factor order: rowwise [T, N] for the dgrad and TRANSPOSED (32-token blocks along T) as the contiguous e4m3 [N, T] for the wgrad —; the two block-scale projection GEMMs over transposed operands, the E8M0 dequant exact in the MMA (no alpha): dW_qkvg = dQKVG8^T . h^T against the CALLER’s h_t — h re-quantized along tokens, e4m3 [d_model, T] contiguous — with its blob h_t_sf, and dh = dQKVG8 . W_qkvg^T against the caller’s w_qkvg_t — W_qkvg re-quantized along N, e4m3 [d_model, N] — with w_qkvg_t_sf (quantized once per weight update). The four artifacts are execute keywords (h_t / h_t_sf required when need_dw_qkvg, w_qkvg_t / w_qkvg_t_sf when need_dh, each refused otherwise), validated on the host before any launch: dtype, shape, the contiguous K-major storage (a .t() view of the un-transposed codes is refused by name), 16-B alignment, the blobs’ padded byte count (kernels.proj_gemm.sf_blob_bytes(d_model, T) / (d_model, N)). The scale-factor blob of a transposed artifact is sized by sf_blob_bytes(rows, k) = ceil128(rows) x ceil128(k) / 32, which is the same number for (rows, k) and (k, rows): the byte count does not validate the blob’s orientation. A blob built over the un-transposed matrix (the forward’s h_sf handed as h_t_sf) passes every host check and produces a wrong weight gradient; build it over the transposed matrix exactly as the artifact it scales, and verify a new caller against the reference once. The small launches are fused as on the per-tensor fp8 chain: 10 block launches with every gradient (20 before — the fused PROLOGUE replaces the scalar init, the dY amax pass, the bf16 Q / K rebuild and the five SDPA-operand quantizes; the dual-axis dO launch the two dO quantizes; the fused EPILOGUE the dW_norm reduce and the two dQKVG quantizes; every payload, scale-factor blob, scalar slot and gradient bitwise the unfused chain’s), plus the SDPA row’s 1 + c*(2+q) + (g > 1) with q = 1 — the block-scale arm of the row launches its dQ GEMM once per head chunk, like the plain renderings (its dQ record takes b_head_group = the GQA group: B and its scale factors are indexed by h // group; bitwise the per-member launches it replaced) —: 15 launches at the test geometry (S = 512, B = 2, GQA 8/2, c = 1, Q/K RMSNorm on), 15 RoPE-only (the epilogue stays for the cast), 14 MHA, 15 at the 397B geometry (B = 1, S = 512, GQA 32/2, c = 1, g = 16: the suite’s own 397B census cell) — the per-tensor fp8 chain’s count at the GQA cells —, and more at a padded S (the row’s staging pads: 30 at S = 992 or S = 1008 under GQA with the weight gradients, 29 at the dgrad-only S = 1000, 28 at S = 992 MHA, 22 at S = 384), every figure counted by CUPTI on Rubin (cc 10.7; identical on a 204-SM and a 212-SM part — two datasets of one tree, since torch’s Philox draws follow the SM count, and the accept suite’s docstring carries both datasets’ margins) in the MXFP8 backward’s own suite (test_mxfp8_launch_count_is_honest, ten census cells: the launch records against an expectation computed from the block’s rows and the adapter’s facts, never typed; 0 memsets, 0 memcpys). The two changes arrived one at a time and each was counted the same way: the unfused chain over the row’s per-member dQ measured 28 / 27 / 24 / 40 and 43 / 41 / 38 / 35 on the same cells, the unfused chain over the single-launch dQ 25 / 24 / 24 / 25 and 40 / 38 / 38 / 32, the fused chain over the per-member dQ 18 / 18 / 14 / 30 and 33 / 32 / 28 / 25. Measured on Rubin cc 10.7 (212 SMs, locked clocks, CUPTI device time, the 397B geometry at S = 8K) the PROLOGUE runs in 0.099 ms against the eight launches it replaces at 0.184 (+85 %), the dual-axis dO launch in 0.042 against 0.116 (+178 %), the EPILOGUE in 0.088 against 0.122 (+39 %); the PROLOGUE reads its own bytes at 3.7 TB/s against the per-tensor fp8 prologue’s 6.3 TB/s (its two-pass 32-token tile is resident 5-6 CTAs per SM against the shipped tile’s 14), which is why the workspace carve is keyed on the prologue’s ARM (mx_prologue_arm): the alternative arm that keeps the bf16 TMA store and quantizes q_T / k_T from the bf16 buffers by a dual-axis launch carves the two bf16 rebuild regions again and changes nothing else. Under GQA the MXFP8 SDPA backward folds its per-Q-head dK partials in fp32 and rounds the sum once, like the reference, while its per-Q-head dV partials are bf16 (the kernel stores them from its epilogue; fp32 ones do not fit its 327 KiB shared-memory budget), so dV carries one bf16 rounding per group member where a once-rounded reference carries one in total (relative RMS about 3e-3 at a group of 4, the geometry the tests run, measured on the per-tensor fp8 row before it moved to fp32 partials); the modelled oracle folds dV the same way and the distance to a once-rounded fold is reported per cell. bwd.quant_scalars(workspace) returns the same 29 views; eight are live — amax_dy / scale_dy / descale_dy / alpha_b1 / alpha_b2 (the dY point) and the constants scale_o / descale_o / descale_w_o — and the other 21 read exactly 0.0 (no dO / dQKVG / dP scalar: those gradients are block-scaled; no per-tensor static scales of an fp8 record). scale_dp / scale_do / scale_dqkvg are refused at execute. Workspace: the bf16 O_gated and compact V regions are replaced by the per-tensor dY8 / O_gated8, every block-scaled payload with its scale-factor blob (dO8 rowwise and columnwise, Q8 / K8 rowwise and columnwise, V8 rowwise — D / 32 scale bytes per row —, dQKVG8 [T, N] and dQKVG8^T [N, T] with their padded canonical blobs, the transposed pair only when the projection weight gradient is requested), the 256-B scalar block and the dY amax partials; the bf16 recompute of Q / K is not carved either (the fused prologue quantizes it out of registers). Measured with get_workspace_size() at default knobs (Rubin cc 10.7): +64.6 KiB/token at the 397B geometry, B = 1, S = 512 (122,912,000 B against the bf16 block’s 89,031,168 B; +81.6 with the unfused chain’s bf16 rebuild regions), of which the block’s own carve is +47.75 KiB/token and the MXFP8 row’s scratch +16.9 (+17.9 at S = 1024, +19.9 at S = 2048: the row’s share grows with S, the carve’s is flat; the GEMM scratch is 0 on both); at the test geometry (B = 2, S = 512) +4.9 KiB/token (carve +12.65, row scratch +4.22, the bf16 GEMM plans’ 12 MiB split-K scratch gone on the MXFP8 K64 block-scale plans). The delta region is always carved, and the SDPA scratch is the MXFP8 row’s (its block-scaled dS: two e4m3 payloads plus their E8M0 atoms, 2 + 2/32 bytes per element; under GQA its bf16 dV and fp32 dK per-Q-head partials). Determinism: no atomic anywhere on the MXFP8 chain — the one amax (dY) is a max over per-CTA partials, the row’s GQA fold is a fixed-order reduce, the block-scale GEMMs are deterministic — so two executes are bitwise equal under every knob set. fuse_gate_bwd is accepted and inert (the fused delta is mandatory); fuse_wgrad_overlap is served (the side-stream GEMMs fork after the operands and blobs they read are written). Declined (typed, naming the attribute) on top of the per-tensor fp8 arm’s: thd=True with an MxQuantSpec (dense-only: the backward’s SDPA-layout MX quantize stages run the quantizer’s dense arm only; the packed MXFP8 record the forward writes and the packed head-major delta both exist, the packed MXFP8 backward is a follow-up — the per-tensor fp8 backward is served packed), B*S % 32 != 0 when a projection weight gradient is requested — the weight-gradient GEMM contracts over the token axis through one E8M0 scale per 32-element K block, and the transposed quantize writes whole 32-token blocks — (pass need_dw_qkvg=False, pad or batch the sequence to a multiple of 32, or run the per-tensor fp8 backward, whose weight gradients take no block scales; the data gradients dh / dW_o are served at any T), an artifact given without its need or a need without its artifact, a .t()-view artifact, a wrong blob byte count or dtype. The block binds the MXFP8 row’s dense plan with an external delta: nothing here changes the row’s capabilities.

The fp4 weight modes’ backward — quant=MxQuantSpec(w_qkvg_dtype=torch.float4_e2m1fn_x2) and / or MxQuantSpec(o_fp4=...). The MXFP8 pipeline’s two fp4 modes train on the unfused pipeline: their training forward (save_for_backward=True) writes the SAME record as the MXFP8 training forward, byte for byte — the fp4 tail replaces only the workspace’s per-tensor o8 by the e2m1 o4 and its blob, and the SDPA writes bf16 o — and GatedAttentionBlockBwd(quant=<that spec>) is the MXFP8 backward with the two DATA-gradient GEMMs on the FROST block-scale catalog’s fp4 rows (the forward’s own renderings at the dgrad’s shapes) over the caller’s TRANSPOSED e2m1 artifacts. The weight gradients stay 8-bit (h is e4m3 in every fp4 mode: dW_o per-tensor e4m3, dW_qkvg the MXFP8 block-scale GEMM). An MXFP4 W_qkvg puts dh = dQKVG8 . W_qkvg^T on the mixed e4m3 x e2m1 row, so w_qkvg_t is the packed e2m1 [d_model, N // 2] (torch.float4_e2m1fn_x2, two codes per byte along N, low nibble = even n) with the UNCHANGED E8M0 / 32 w_qkvg_t_sf — the same keyword, its dtype following w_qkvg_dtype; nothing else changes (the MXFP8 launch census — 15 at the test geometry, 15 RoPE-only, 14 MHA — and the same carve). An fp4 W_o (o_fp4; scale_o == descale_w_o == 1.0 by MxQuantSpec’s own rule) puts dO_gated = dY . W_o^T on a block-scale row over execute(w_o_t=, w_o_t_sf=) (appended; required iff o_fp4 whatever the need_* set, since the gate backward needs dO_gated; refused otherwise) — W_o re-quantized along d_model in o_fp4’s format, packed e2m1 [H_q * D, d_model // 2] with its blob (sf_blob_bytes(H_q * D, d_model, block): e4m3 scales per 16 for Fp4Format.NVFP4, E8M0 per 32 for Fp4Format.MXFP4; the other format’s blob is a byte-count decline) — and adds ONE launch, a block quantization of dY right after the per-tensor one: under MXFP4 the MX-rowwise e4m3 dY with its canonical E8M0 blob (the mixed row again; the gradient stays 8-bit), under NVFP4 dY itself cast to NVFP4 — packed e2m1 with e4m3 scales per 16, the NVFP4 x NVFP4 row, the only catalog row for an e2m1 side with e4m3 scales — as a two-level cast: an NVFP4 block’s e4m3 scale is max(amax / 6, 2^-9), so a 16-element block of a raw output gradient whose amax sits under the e2m1 midpoint 2^-11 would quantize to all zeros. The cast is therefore taken at the per-tensor power-of-two scale of the dY quantization (scale_dy, read from its slot in-kernel: the tensor’s amax lands in [224, 448] and a block is zeroed only when its amax sits more than about 19 octaves below the tensor’s) and undone in the gate backward, which multiplies the dgrad’s output by descale_dy before every use — exact for a power of two, so a power-of-two scaling of dY leaves every gradient bitwise equivariant (the suite’s 2^-13 pin, which the single-level cast fails on its first assertion: at 2^-13 it keeps about 0.004-0.006 % of the codes). Wherever the single-level scale byte was a normal e4m3 value the codes are identical (a power of two only shifts the exponent), so the pre-scale is purely a floor remedy. Launches: the MXFP8 count under an MXFP4 W_qkvg alone, + 1 under an fp4 W_o (16 at the test geometry with Q/K RMSNorm, 16 RoPE-only, 15 MHA, 31 at the padded GQA cell S = 992 with the weight gradients; the fp4 suite’s census counts the test-geometry, MHA and padded cells by CUPTI on Rubin against an expectation computed from the MXFP8 table plus the one dY block quantize, never typed). Workspace: dy_mx8 + its blob (MXFP4) or dy4 + its blob (NVFP4) appended last; every MXFP8 region is unchanged. The weight gradients are allocated at their logical shapes — (n_qkvg, d_model) and (d_model, H_q * D) in dy’s dtype — never with empty_like(<weight>): a packed e2m1 weight’s .shape is its storage [rows, K // 2] (the convenience wrapper sizes them from the geometry; a packed e2m1 weight carries requires_grad like any other tensor, so its gradient is requested the same way). bwd.quant_scalars() reads the same eight live slots, scale_o / descale_o / descale_w_o at 1.0 and alpha_b2 published but read by no GEMM (the block-scale dgrad has no alpha). The oracle dequantizes the transposed e2m1 artifacts through their blobs — two fake-quants of one master weight along its two axes, the fp4 training recipe’s straight-through estimator — and takes the same two-level dY point under NVFP4; the accept suite is test_block_backward_fp4.py (five configurations: MXFP4 W_qkvg; NVFP4 W_o; MXFP4 W_o; both with either W_o format). Declined (typed, naming the attribute): an artifact in the wrong dtype for its weight’s mode (an e4m3 w_qkvg_t under an e2m1 W_qkvg and the reverse; uint8 bytes of packed codes get the .view(torch.float4_e2m1fn_x2) hint), a logical [rows, K] fp4 artifact (twice the packed data), a .t() view, w_o_t / w_o_t_sf without o_fp4 or missing with it, the other format’s w_o_t_sf, an fp4 h (not served: h stays e4m3), the fp4 modes with a fused training forward (the forward’s own declines), B*S % 32 != 0 with a projection weight gradient (inherited).

Workspace (get_workspace_size(), after compile()): the block’s own regions — dO, the [T, N] dqkvg slab, O_gated, the recomputed Q / K / V, compact dQ / dK / dV — (6*H_q + 6*H_kv) * D * 2 bytes per token in bf16 (102 KiB/token at the 397B geometry), plus the SDPA backward’s scratch (delta — the block’s own region under fuse_gate_bwd — and the per-Q-head dK / dV partials, ~32 KiB/token, and ONE dS chunk of qh_chunk x S_q_pad x S_kv_pad x 2 bytes with qh_chunk a multiple of the GQA group: 4.25 / 8.50 / 33.0 GiB at S = 8K / 16K / 32K for the 397B geometry at B=1), plus the dW_norm partial planes ((n_ctas_q + n_ctas_k) x D x 4 bytes, at most 2 x SMs x 8 x 1 KiB), plus the GEMMs’ scratch (max(plan.workspace_bytes): 0 at 397B, 12 MiB at the test geometry — the backend heuristic’s split-K partials, never launched on the forced tile; under fuse_wgrad_overlap a second such region, gemm_scratch_side, for the two side-stream wgrad GEMMs, sized to their plans, appended last). At S=32K, B=1, 397B: ~36 GiB in total, dominated by the dS chunk.

Requirements and limits

  • CUDNN_FRONTEND_ENABLE_FROST_ENGINES: every FROST engine is opt-in, and the block drives them. The flag is read LIVE, when a GEMM plan’s ranked list is built at compile() — not once at import cudnn — and a host module must never set it for the process (it flips every opt-in engine into every FE graph the process builds). The block does not need it: a dense bf16 / f16 projection whose N the forced 256-wide GEMM tile divides (n_qkvg, d_model and h_q * d_head all % 256 == 0, as at every served geometry), or whose shape the small-M route names a tile for (B x S_q <= 128 on a dense bf16 / f16 projection whose N the route’s tile grid divides — the indexer band’s 13952-column slab included: 109 x 128-row tiles), or whose shape the large-M rule names the GEMM engine’s own baseline tile for (past 128 rows on a dense bf16 / f16 projection whose N 128 divides and 256 does not — the band slab at prefill / extend / chunked row counts; the kernel the graph route ran there, named so the route is the JIT-only one), compiles with the flag UNSET on FROST’s JIT-only route (plan.route == "jit-only": no backend plan, no cuDNN handle at execute; frost_gemm never enters a ranked list), and the block-scale (MXFP8 / fp4-weight) projections log a warning and take the same route. With the flag set a plain declaration’s backend admits the graph and pins the same JIT by name (graph+jit); a capture_safe=True declaration never enters the ranked list whatever the flag says. What still needs the flag: a dense shape WITHOUT a JIT tile (the graph route IS the plan there: an N off both tile grids, or a 128-grid shape the engine’s planner could re-plan) and the per-tensor-fp8 projections — a RuntimeError at compile() whose message names the flag (the GEMM plan is built there, not at check_support()). Nothing ever falls back to a cuDNN backend plan, so an unset flag is never a silently slower block. The SDPA stage binds its FROST class directly and does not consult the flag.
  • CUDA-graph capture of a serving stack (the FIRST execute of a fresh declaration captured on a NEW stream, with no eager warm-up): on the forced-tile JIT route the captured first execute launches nothing and replays bitwise an eager run; the GRAPH GEMM route (no forced tile) creates a cuDNN handle at its first execute per (device, stream), which cannot happen inside a capture — a typed RuntimeError naming the route and the remedy (one eager execute on the capture stream first, or the forced-tile route), and no handle is created inside the capture; compile() under capture is refused the same way (the GEMM plan builds a cuDNN graph and JIT-compiles a kernel there). The GEMM driver’s serving declaration, build_proj_gemm(capture_safe=True), never enters the ranked list — no backend plan, no FROST-engine compile, whether the flag is set or not — so the plan is on the JIT route by construction: a refused forced compile is a typed NotImplementedError naming the tile and the reason, never the warning + graph-heuristic fallback the plain declaration takes. The block sets capture_safe on each dense projection stage of a paged declaration iff the driver’s auto resolver names a JIT tile for the stage’s own shape — so the indexer band’s slab (N 13952) is capture-safe at every row count: at decode row counts through the small-M route (the band capture / replay cell of test_block_qsa_decode.py), past 128 rows through the engine’s own baseline tile (the large-M rule; the extend ladder’s cells past 128 rows of test_block_qsa_extend.py). A shape the resolver names no tile for runs eagerly, and a capture there is the driver’s typed refusal. The quantized block’s first-execute scale write is NOT refused under capture (the warm-up / re-capture contract above is the rule): test_block_serving_capture.py.
  • Rubin (SM107) for every form, or cc 10.0 (SM100) for the dense bf16 / f16 inference forms alone (the part table at the top; functional only); cuDNN 9.x, nvidia-cutlass-dsl >= 4.8.0.dev0 (the Rubin arch names; the 4.7.0 floor on cc 10.0), torch.
  • Backward: bf16 / fp16 (both against fp64 autograd on Rubin: test_block_backward.py) — and per-tensor fp8 over the fp8 training record (quant=QuantSpec: bf16 dy and gradients, dense or packed (thd=True), any B*S, head counts the TMA Q / K rebuild tiles), and MXFP8 over the MXFP8 training record (quant=MxQuantSpec: bf16 dy and gradients, dense only, B*S % 32 == 0 when a projection weight gradient is requested, the caller’s transposed artifacts h_t / h_t_sf / w_qkvg_t / w_qkvg_t_sf at execute; the fp4 weight modes on the same record — a packed e2m1 w_qkvg_t under an MXFP4 W_qkvg, w_o_t / w_o_t_sf under o_fp4); Rubin only — the block binds ONE FROST engine class per declaration (SdpaBwdDslSm107, the Rubin d=256 SDPA backward; under quant its per-tensor fp8 row, SdpaBwdDslSm107Fp8, or its MXFP8 row, SdpaBwdDslSm107Mxfp8) and never falls back to the cuDNN backend’s d=256 backward, exactly as the forward binds its FROST SDPA class (AGENTS.md Rule 9, a stated design decision: every other device is a typed decline); d_head = 256; seq_len >= 2 (S = 1 is decode, out of the prefill bodies’ scope); d_model % 256 == 0 (the forced GEMM tile behind the determinism contract; h_q * d_head satisfies it through d_head = 256); saved.proj_slab required (the gate-copy save set, saved_gate_copy=True, is served by a later PR); no seq_lens yet (typed — at declaration from seq_lens_present=True or a sample record whose seq_lens is a tensor, and at execute for the record handed there: a padded record contradicts a dense declaration and is refused before any launch; the sdpa_bwd_sm107 row declines padding); window_left > 0 only (or -1), window_right unbounded or 0 only; dw_norm_dtype=torch.float32 only; rope_dim > 0; get_workspace_size() after compile(). A dense S % 128 != 0 is differentiated like any other record: the forward’s SDPA row masks its KV tail in-kernel (#1520); causal covers the tail.
  • Packed sequences (thd=True), forward and backward: bf16 / fp16; the per-tensor FP8 and MXFP8 unfused forwards (the fp4 modes included), inference and training, and the per-tensor fp8 BACKWARD over its packed record as written (quant=QuantSpec with the packing knobs: the fp8 SDPA row’s THD chain reading the gate backward’s packed delta, every other stage token-wise at B = 1, S = T); the packed bf16 backward takes a packed quantized record with the dequantized h and weights (a record handed to it with its e4m3 h is the same typed decline as on the dense side, after the packed-length checks); fuse_norm_rope (bf16 / fp16 inference). num_sequences >= 1, 2 <= max_seq_len <= T, num_sequences * max_seq_len >= T; the lengths tensor contiguous 1-D int32 on h’s device with B (cu_seqlens=False) or B+1 (cu_seqlens=True) entries; every length <= max_seq_len, the lengths summing to T (the caller contract, not host-validated); the training record carries saved.seq_lens and saved.seq_lens_form. Declined (typed): seq_lens_present together with thd, fuse_gate, the fully fused quantized pipelines, the MXFP8 backward over a packed record, the packing knobs on a dense block.
  • d_head = 256 (the Rubin d256 SDPA flavor with the fused gate); d_model % 128 == 0 under MXFP8.
  • Paged KV-cache write-through (paged_kv_page_size, a positive multiple of 16): bf16 / f16 inference, dense [B, S, d_model] and packed (thd=True); the pools are the activation dtype, [num_pages, H_kv, P, D] views with D contiguous and every other stride a whole number of 16-byte vectors (HND compact, NHD by strides, or the two halves of a unified [num_pages, H_kv, P, 2D] row); quant and save_for_backward together with it are typed declines (“Serving: write-through into a paged KV cache” above) — except the FP8-projection + bf16-attention mode on the paged forms (the bullet below).
  • The EXTEND form (kv_source="paged"): bf16 / f16 inference on a PACKED declaration with paged_kv_page_size > 0, geometry.causal_bottom_right=True (is_causal=True), no window, no QsaSpec, d_head=256, return_lse=False, fuse_gate=False; the three samples at declaration, block_table / kv_lens at execute (refused under kv_source="self"); the pools match the samples exactly; token_capacity (THD only, equal to T, inference only) bounds the live tokens. Validated at pages 16 / 32 / 64 / 128 on all three pool layouts, bf16 and f16 (“Serving: the extend form” above); every other combination is a typed decline serving_envelope predicts.
  • The FP8-projection + bf16-attention mode (QuantSpec(attention_dtype=torch.bfloat16), 2026-10-30): the DECODE and EXTEND forms with e4m3 h / W_qkvg / W_o under static per-tensor scales, bf16 pools and attention, the gated O quantized to e4m3 at scale_o for the FP8 out projection; scale_q / scale_k / scale_v None; fuse_norm_rope=False, inplace_qkv None / True, inference only, kv_cache_dtype None, no QsaSpec, no use_pdl, no num_tokens_live; cc 10.7 and cc 10.0 (FP8_PROJ_BF16_ATTENTION_CCS; the gate its own launch on cc 10.0), declined by name on every other served dense part (“Serving: FP8 projections with bf16 attention” above); functional claim only, no performance number.
  • The serving DECODE form (kv_source="paged" on a dense [B, S_q] block over paged_kv_page_size > 0): bf16 / f16 inference, d_head = 256, is_causal with causal_bottom_right (no window), the batch uniform, block_table + kv_lens at execute (seq_lens refused), the pools and the table exactly the declared samples; the served envelope S_q <= 32 (the claimed envelope S_q <= 4 on the Rubin d256 decode tile; the MTP verify step to S_q = 8 on the one- and two-stream routes; the prefill route, the token-unit cells past S_q = 4 and the unpacked arm up to 32 tokens declared, unmeasured), a typed decline beyond; fuse_gate rides the decode tile’s split combine or the paged prefill body’s epilogue, else its own launch (“Serving: the decode form” above). On a QsaSpec geometry the same declaration is the block-sparse DECODE / extend-with-prefix mode: stage (4) the sparse core reading the pools, one list per sequence at S_q <= 4 (block_ids [B, top_k]) and one per row past it (any S_q), decode_split_kv in [1, ceil((top_k + 1) / 32)] or the fill rule, fuse_gate and return_lse served, sample_block_table contiguous, fuse_prepare served (None = the measured default), the in-block indexer’s DECODE and EXTEND steps over the pools served (index_source="indexer": bf16, the per-row list, the paged compressed-key cache + the ring + its table as samples; index_select_form the step form — the decode step inside the decode scorer’s envelope, the extend step past it: a T-row chunk behind a cached prefix that writes the caches itself; “The in-block indexer’s DECODE and EXTEND steps over the pools” above); the packed form is a typed decline.
  • The e4m3 KV cache (kv_cache_dtype=torch.float8_e4m3fn): bf16 activations with paged_kv_page_size > 0; the quantizing write-through on the dense prefill forms (QsaSpec or dense) and the block-sparse DECODE mode over e4m3 pools at S_q <= 4 (the gathered cast + the d256 decode tile, out-of-place norm, the gate as its own launch, unsplit, no LSE); execute(k_scale=, v_scale=) 1-element fp32 CUDA tensors; the packed prefill, the EXTEND form, the dense DECODE form, S_q > 4, return_lse, a split, fuse_norm_rope, f16, e5m2 and quant with it are typed declines (“Serving: the e4m3 KV cache” above).
  • FP8 / MXFP8: the UNFUSED pipelines train (save_for_backward=True writes the bf16 record described above), the fp4 modes of the MXFP8 pipeline included; the fully fused quantized pipelines are inference only. The backward is bf16 / fp16 — and per-tensor fp8 / MXFP8 over the quantized training records (quant=QuantSpec / quant=MxQuantSpec, the record as written, the fp4 weight modes through their transposed e2m1 artifacts); the bf16 backward takes either record with the dequantized bf16 h and weights.
  • FP8: a dense (no-mask) sequence length off a multiple of 128 is served; the SDPA masks the KV tail in-kernel (#1520); MXFP8: e4m3 codes only (e5m2 is a typed decline); the fully fused MXFP8 path needs scale_o == 1.0 and, at B > 1, S % 128 == 0 (a scale-factor atom is per sequence).
  • fuse_gate and fuse_norm_rope are inference-only specializations (no pre-gate O, no pre-norm Q/K).
  • Block-sparse attention (QsaSpec): bf16 / f16 inference, dense [B, S, d_model] (with or without seq_lens) or packed (thd=True, caller lists with sequence-relative ids), d_head == 256, h_q // h_kv <= 16, caller lists (block_ids, top_k a multiple of 4 in [4, 512]) or the in-block indexer (index_source="indexer": bf16, the band, index_head_dim 128, index_heads in the scorer’s packed groups; dense prompts, and the DECODE step over the pools); the indexer band on the UNFUSED projection; fuse_gate (the sparse core’s epilogue gate) composes with it, dense or packed; save_for_backward, quant, fuse_norm_rope together with the band, causal_bottom_right on the block’s own K / V (kv_source="self"; over the pools it is the block-sparse decode mode, where the in-block indexer’s DECODE step is served since 2026-10-21), and thd together with the in-block indexer are typed declines (“Sparse attention (QSA)” above; thd together with the paged write-through is served — the stage addresses tokens); what a serving stack hands in — the list form, the count rule, the decode-mode list, the pools, the indexer’s inputs and outputs — is the contract page “Serving: index lists and paged KV” above.
  • fp4 (MxQuantSpec.w_qkvg_dtype / o_fp4): MXFP8 pipeline only (unrepresentable on QuantSpec / bf16); trained on the unfused pipeline (save_for_backward=True writes the MXFP8 record; the fused forks stay inference-only) and differentiated by GatedAttentionBlockBwd(quant=MxQuantSpec) over the caller’s transposed e2m1 artifacts; no global per-tensor scale in either fp4 format (scale_o == descale_w_o == 1.0 under o_fp4); d_head % (4 * block) == 0 under o_fp4 (whole 4-block scale words per head: 64 for NVFP4, 128 for MXFP4; d_head = 256 passes both); the MXFP4 W_qkvg is served on both pipelines — fully fused through the MXFP8 projection fork’s e2m1-B arm (NormRopeFusionParams.weight_fp4, feature-detected: a checkout whose fork lacks the field declines typed) under the fused pipeline’s own rules (inference only, dense only, S % 128 == 0 at B > 1, scale_o == 1.0). h stays e4m3 (an fp4 h is not served), and an fp4 W_o with an e4m3 O is not a served pairing.

Performance

Whole block, B=1, h_q=32 h_kv=2 d=256 d_model=5120 (the 397B geometry), Rubin perf node (212 SMs, SM clock locked at 2376 MHz), speedup over the same bf16 torch chain (median of 5 launch-interleaved rounds x 30 launches; the bf16 FROST control pair stayed within 0.6 %). The fp4 modes (MXFP4 weights, NVFP4 / MXFP4 O) are measured separately below, at d_model = 4096, and are not columns of these d_model = 5120 tables.

Causal:

Sbf16 FROSTbf16 fully fusedFP8 unfusedFP8 fully fusedMXFP8 unfusedMXFP8 fully fused
20482.96x3.16x4.22x4.90x3.84x4.30x
40963.06x3.27x4.52x5.28x4.10x4.81x
81922.79x2.95x4.33x4.90x4.06x4.68x
163842.37x2.44x3.93x4.27x3.75x4.15x
327681.90x1.96x3.41x3.61x3.22x3.44x

Dense (no mask):

Sbf16 FROSTbf16 fully fusedFP8 unfusedFP8 fully fusedMXFP8 unfusedMXFP8 fully fused
20482.78x2.98x4.07x4.82x3.85x4.45x
40962.74x2.93x4.24x4.87x4.00x4.58x
81922.33x2.44x3.84x4.27x3.71x4.13x
163841.85x1.90x3.37x3.63x3.30x3.51x
327681.50x1.53x2.92x3.03x2.81x2.86x

fp4 modes, d_model = 4096 (not the 5120 of the tables above; h_q=32 h_kv=2 d=256, QK-norm on), B=1, Rubin perf node (212 SMs, SM clock locked at 2376 MHz and SAMPLED per row: 2364 MHz at 4K, 2340 / 2364 at 8K, 2184 / 2100 at 16K and 2052 / 1968 at 32K causal / dense — the lock power-caps at the long shapes), speedup over the same bf16 torch chain (median of 5 launch-interleaved rounds x 30 launches, every arm of a row in one process; the MXFP8 FROST control pair within 0.6 % at 4K-16K and 2.0-2.5 % at dense 32K under the cap). S = 2048 is not quoted: in a ten-arm process that row is a sub-millisecond window whose control pair read 30-44 %. The dense S = 32768 cells of BOTH fully fused NVFP4 O columns read 11-12 % slower than their unfused twins — a standing anomaly of the fused NVFP4 O pipeline, independent of the weight format.

Causal:

SMXFP8 unfusedMXFP8 fully fusedMXFP4 weights unfusedMXFP4 weights fully fusedNVFP4 O fully fusedMXFP4 weights + NVFP4 O fully fused
40964.32x4.75x4.42x4.84x4.80x4.87x
81924.24x4.71x4.35x4.76x4.71x4.76x
163843.87x4.15x3.96x4.17x4.17x4.18x
327683.24x3.39x3.32x3.41x3.47x3.47x

Dense (no mask):

SMXFP8 unfusedMXFP8 fully fusedMXFP4 weights unfusedMXFP4 weights fully fusedNVFP4 O fully fusedMXFP4 weights + NVFP4 O fully fused
40964.18x4.55x4.26x4.60x4.54x4.58x
81923.83x4.12x3.91x4.16x4.11x4.14x
163843.32x3.46x3.37x3.48x3.40x3.43x
327682.82x2.88x2.89x2.91x2.57x2.56x

Read across a row: the fully fused MXFP4-weight block is faster than the fully fused MXFP8 block by +1.9 / +1.0 / +0.6 / +0.6 % (causal, 4K .. 32K) and +1.1 / +1.0 / +0.5 / +1.0 % (dense) — the halved weight bytes of a projection that stays MMA-bound (55-81 % of the 8-bit K32 MMA cap, causal); the 4K / 8K cells clear their control pair by more than 3x, the 16K / 32K cells of both masks sit under 2x theirs (causal -0.32 / -0.38 %, dense -0.59 / +2.53 %) and are reported, not claimed — and faster than the unfused MXFP4-weight block by +9.6 / +9.5 / +5.3 / +2.6 % (causal) and +7.9 / +6.3 / +3.1 / +0.7 % (dense), the fusion itself. With both fp4 modes the fully fused block sits within +1.4 / +1.0 / +0.1 / +0.1 % (causal) of the fully fused NVFP4 O block, its 16K / 32K cells inside the control spread.

Backward, fuse_gate_bwd (the gate backward feeding the SDPA backward’s delta): whole-backward wall time of the bf16 block backward at the 397B geometry, B=1, causal, QK-norm on, Rubin perf node (212 SMs), knob off and on interleaved launch by launch in one process with the knob-off arm timed twice as the control, 3 rounds x 20 launches per process, 3 fresh processes per S; +X % = unfused ms / fused ms - 1, positive = the knob is faster. The gradients were bitwise equal between the arms in every process.

Sknob off (ms)knob on (ms)fuse_gate_bwd vs unfusedcontrol pair
20480.5280.516+2.3 %within 0.2 %
81922.4242.389+1.5 %within 0.6 %
3276825.2625.03+0.7 % (positive in 3 of 3 processes, but within 2x the control: a weak claim)within 0.5 %

The SM clock was locked at 2376 MHz but power-capped during the 8K and 32K runs (sampled 2150-2376 and 1550-1850 MHz), so only the interleaved ratios are quoted; the absolute 8K / 32K milliseconds are capped-clock numbers. The knob removes one launch and one read each of O and dO (32 KiB/token) per backward; it stays off by default.

Backward, fuse_wgrad_overlap (the two weight-gradient GEMMs on the block’s side stream): the same protocol — whole-backward wall time of the bf16 block backward at the 397B geometry, B=1, causal, QK-norm on, Rubin perf node class (212 SMs), knob off and on interleaved launch by launch in one process (shuffled slot order) with the knob-off arm timed twice as the control, 3 rounds x 20 launches per process, 3 fresh processes per S; +X % = the median over processes of the per-process interleaved ratio in-order / overlapped - 1 (the ms columns are medians of the capped-clock absolute times and need not reproduce it), positive = the knob is faster. The gradients were bitwise equal between the arms in every process and the kernel count of one backward was unchanged (the same launches on another stream). Two settings of the other knob, since the two compose:

With fuse_gate_bwd off in both arms:

Sknob off (ms)knob on (ms)fuse_wgrad_overlap vs in-ordercontrol pair
20480.5140.499+2.8 %within 0.2 %
81922.3262.305+0.5 % (positive in 3 of 3 processes, within 1.2x the control: a weak claim)within 0.4 %
3276822.8822.72+0.7 %within 0.3 %

With fuse_gate_bwd on in both arms (the fused configuration):

Sknob off (ms)knob on (ms)fuse_wgrad_overlap vs in-ordercontrol pair
20480.5080.495+2.7 % (positive in 3 of 3 processes, within 1.5x the control: a weak claim)within 1.8 %
81922.2862.259+1.0 %within 0.2 %
3276822.7222.69+0.7 % (positive in 3 of 3 processes, within 1.8x the control: a weak claim)within 0.4 %

What overlaps, from the CUPTI timeline of one backward at S=8K: dW_o runs alongside the Q/K recompute, the V compaction and the chain’s first kernel, and dW_qkvg alongside the dW_norm reduce and the tail of the dh dgrad; the persistent SDPA backward chain occupies every SM, so the side stream mostly removes the launch gaps around the two GEMMs rather than hiding them (the CUPTI-measured time two kernels of one backward were in flight together rose from about 6 us in order to about 65 us with the knob at S=8K, with fuse_gate_bwd off or on: dW_o co-resident for 32-34 us, dW_qkvg for 31-32 us). The gain is therefore largest at small S and shrinks as the chain’s share grows. The clock was locked but power-capped under the sustained 8K / 32K chain, so only the interleaved ratios are quoted. The knob stays off by default.

Packed sequences (THD), forward — the packed block against the dense block, Rubin (cc 10.7, 212 SMs, SM clock locked at 2376 MHz and sampled during every cell: 1872-2328 MHz, the lock power-caps under the long cells), both geometries 32/2 (h_q=32 h_kv=2) and 64/8 (h_q=64 h_kv=8) at d_model = 4096 (not the 5120 of the forward tables above), d_head = 256, RoPE 64, Q/K RMSNorm on, causal and dense (no mask), bf16, the unfused FP8 block and the unfused MXFP8 block (the three quantized / unquantized pipelines over the SAME bf16 draws). Every cell is one process holding the six FROST blocks (each dtype’s packed block and its dense twin: identical FLOPs, identical kernels except the SDPA’s packed specialization and, for MXFP8, the packed scale-factor layout of its three quantize_mxfp8 stages) and the bf16 torch chain, slots round-robin by launch with the slot order shuffled per iteration, 3 rounds of 150 / 60 / 20 launches per slot (2K / 8K / 32K; varlen 60), median per slot per round then the median of rounds (CUDA events around the whole block), the packed bf16 arm timed twice as the control pair (-1.97 .. +0.74 % over the cells: within 0.9 % in 15 of the 16, the 64/8 causal 32K x 1 cell at -1.97 % makes that cell’s numbers a weak claim). packed overhead = packed ms / dense ms - 1 on uniform packings (B sequences of S tokens each: the same FLOPs; positive = the packed block is slower); the varlen cell packs [2048, 4096, 6144, 8192] (B=4, max_seq_len = 8192, 20480 tokens) and reports TFLOP/s on its exact per-sequence FLOPs — the sum over the four sequences of the one FLOP model every row of these tables uses, analytic_flops of test/python/gated_attention_block/cutedsl/benchmark_baseline.py: both attention BMMs over S_i^2, halved under the causal mask, plus the dense qkv_gate and output projections over the S_i tokens, so a row’s bf16, FP8 and MXFP8 arms share one FLOP count — beside a FLOP-scaled estimate of the dense block’s time (dense ms at B x S_max x FLOPs_varlen / FLOPs_dense: the dense twin timed in the same process as the packed arm, not the uniform tables’ 8K row, and the same FLOP model at B x S_max; it assumes time scales linearly with work — an estimate, not a measured equal-work dense run), a dense torch run at B x S_max (which attends over the padding) and a per-sequence torch loop (exact FLOPs, one dense call per sequence); speed-ups are positive numbers, base ms / new ms - 1. The percentage of peak is against 8192 FLOP/clk/SM (bf16) x 212 SMs x the SM clock sampled during the cell; FP8 and MXFP8 against the K32 cap of 16384 FLOP/clk/SM (the part’s K64 peak is 32768, twice that, so halve those percentages for it). Every packed arm is gated per sequence (the fp32 oracle for bf16 and the fake-quant oracles of the two quantized pipelines at S <= 4096, else the dense block’s own rows or the per-sequence torch chain); the FP8 and MXFP8 varlen rows have no per-sequence quantized reference above S_max = 4096 and are reported ungated. The 32K cells pack B=2 (32/2) and B=1 (64/8) sequences.

Forward, uniform packings (geomean packed overhead bf16 +2.3 %, FP8 -2.1 %, MXFP8 +0.8 %):

heads Q/KVmaskSB (tokens)dense bf16 mspacked bf16 mspacked bf16 overheadpacked bf16 TFLOP/s (% of peak)dense FP8 mspacked FP8 mspacked FP8 overheadpacked FP8 TFLOP/s (% of the K32 cap)dense MXFP8 mspacked MXFP8 mspacked MXFP8 overheadpacked MXFP8 TFLOP/s (% of the K32 cap)
32/2causal20484 (8192)0.6640.702+5.7 %2838 (71 %)0.4530.491+8.3 %4062 (50 %)0.4490.461+2.7 %4321 (54 %)
32/2causal81924 (32768)3.6263.704+2.1 %3043 (81 %)2.1942.310+5.3 %4879 (65 %)2.1822.170-0.6 %5194 (69 %)
32/2causal327682 (65536)15.91415.898-0.1 %3078 (90 %)9.4638.944-5.5 %5470 (80 %)8.8308.928+1.1 %5480 (80 %)
32/2dense20484 (8192)0.7330.776+6.0 %2921 (73 %)0.4890.530+8.4 %4282 (54 %)0.4900.500+2.1 %4536 (57 %)
32/2dense81924 (32768)5.0435.049+0.1 %3103 (85 %)2.8983.054+5.4 %5130 (70 %)2.8822.891+0.3 %5420 (74 %)
32/2dense327682 (65536)27.19827.171-0.1 %3096 (90 %)17.43714.315-17.9 %5876 (85 %)14.30714.330+0.2 %5870 (85 %)
64/8causal20484 (8192)1.3751.430+4.1 %2882 (74 %)0.9130.971+6.4 %4246 (54 %)0.8950.907+1.3 %4545 (58 %)
64/8causal81924 (32768)7.9097.914+0.1 %2918 (83 %)5.1584.920-4.6 %4693 (67 %)4.6144.609-0.1 %5010 (72 %)
64/8causal327681 (32768)16.29316.830+3.3 %2940 (89 %)10.4859.417-10.2 %5254 (79 %)9.2979.571+2.9 %5170 (78 %)
64/8dense20484 (8192)1.5181.593+4.9 %2934 (77 %)0.9911.061+7.1 %4405 (57 %)0.9890.993+0.3 %4707 (61 %)
64/8dense81924 (32768)10.87110.882+0.1 %2930 (86 %)6.9886.475-7.3 %4924 (72 %)6.2286.236+0.1 %5114 (75 %)
64/8dense327681 (32768)27.94028.374+1.6 %2984 (92 %)17.94115.143-15.6 %5591 (86 %)15.41915.267-1.0 %5545 (85 %)

Forward, varlen packing [2048, 4096, 6144, 8192] (the FLOP-scaled column is packed ms / (dense ms at B x S_max x FLOPs_varlen / FLOPs_dense) - 1, an estimate that assumes time scales linearly with work; positive = the packed block is slower than that estimate):

heads Q/KVmaskdtypepacked msTFLOP/s (% of peak; FP8 / MXFP8: of the K32 cap)vs the FLOP-scaled dense-time estimatespeed-up vs dense torch at B x S_maxspeed-up vs the per-sequence torch loop
32/2causalbf162.0183151 (78 %)+5.9 %+378.0 %+204.4 %
32/2causalFP81.2884934 (61 %)+11.1 %+648.7 %+376.7 %
32/2causalMXFP81.2105252 (65 %)+6.6 %+696.8 %+407.4 %
32/2densebf162.5543297 (83 %)+3.3 %+324.3 %+164.3 %
32/2denseFP81.5805329 (67 %)+10.8 %+585.9 %+327.3 %
32/2denseMXFP81.5055594 (70 %)+7.2 %+620.0 %+348.5 %
64/8causalbf164.1733129 (81 %)+3.6 %+392.0 %+207.3 %
64/8causalFP82.6544920 (63 %)+1.7 %+673.6 %+383.2 %
64/8causalMXFP82.5085206 (67 %)+6.6 %+718.5 %+411.2 %
64/8densebf165.2853251 (83 %)+2.0 %+335.3 %+167.3 %
64/8denseFP83.2875227 (67 %)-3.5 %+599.9 %+329.8 %
64/8denseMXFP83.1235501 (70 %)+5.9 %+636.7 %+352.4 %

Packed sequences (THD), backward — the packed block backward against the dense one, Rubin (cc 10.7, 204 SMs, SM clock 2376 MHz, sampled 2052-2352 MHz: the lock holds on the short cells, the long ones power-cap below it), the same two geometries at d_model = 4096, bf16. Every cell is one process holding the packed block and its dense twin, slots round-robin by launch with the slot order shuffled per iteration, 3 rounds of at least 60 ms per slot, median per slot per round then the median of rounds (CUDA events around the whole block, plus the CUPTI device time of every launch over 30 iterations), the packed arm timed twice as the control pair (within 0.9 % in every cell); the same overhead / varlen conventions as the forward tables, the percentage of peak against 8192 FLOP/clk/SM x 204 SMs x the sampled clock. Every packed arm is gated per sequence (fp32 / fp64 oracles at S <= 4096, else the dense block’s own rows or the per-sequence torch chain). The 32K backward cells pack B=1: the backward protocol keeps four blocks resident (dense and packed, with and without fuse_wgrad_overlap), each with its dS head chunk (about 36 GiB at 32 heads x 32K), which does not fit the device at B=2.

Backward (the block backward alone; the training-step column is the packed overhead of one training forward + backward; geomean packed overhead +1.0 %):

heads Q/KVmaskSB (tokens)dense mspacked mspacked overheadwith fuse_wgrad_overlaptraining step (forward + backward)packed TFLOP/s (% of peak)packed speed-up vs torch
32/2causal20484 (8192)1.7161.739+1.3 %+1.2 %+2.5 %2371 (62 %)+194.7 %
32/2causal81924 (32768)10.10310.202+1.0 %+0.9 %+1.4 %2425 (68 %)+128.1 %
32/2causal327681 (32768)23.33223.558+1.0 %+1.2 %+1.1 %2450 (71 %)+69.8 %
32/2dense20484 (8192)1.9551.972+0.9 %+0.6 %+2.1 %2439 (64 %)+181.7 %
32/2dense81924 (32768)14.01914.029+0.1 %-0.3 %+0.1 %2547 (71 %)+113.1 %
32/2dense327681 (32768)39.63839.669+0.1 %-0.2 %+0.1 %2564 (75 %)+76.0 %
64/8causal20484 (8192)3.4543.471+0.5 %+0.7 %+1.8 %2455 (65 %)+211.7 %
64/8causal81924 (32768)20.56221.243+3.3 %+3.6 %+2.6 %2381 (67 %)+121.8 %
64/8causal327681 (32768)47.25647.738+1.0 %+0.7 %+0.6 %2441 (70 %)+66.3 %
64/8dense20484 (8192)3.9323.951+0.5 %+0.2 %+2.1 %2504 (68 %)+193.1 %
64/8dense81924 (32768)28.49129.182+2.4 %+2.7 %+1.8 %2487 (70 %)+107.7 %
64/8dense327681 (32768)80.66380.618-0.1 %+0.3 %+0.2 %2537 (73 %)+69.4 %

Per stage, backward, causal, S=8192, B=4 (32768 tokens), dense vs packed (unfused):

backward stage (CUPTI device ms per iteration)32/2 dense32/2 packeddelta64/8 dense64/8 packeddelta
B2 dO_gated dgrad GEMM0.6040.603-0.1 %1.1601.280+10.3 %
B3 sigmoid_gate_bwd0.3490.351+0.4 %0.6960.700+0.6 %
B1 dW_o wgrad GEMM0.7000.702+0.2 %1.2551.386+10.5 %
Q/K norm+RoPE recompute0.1450.145-0.1 %0.3150.321+1.8 %
compact V (elementwise)0.0110.011+0.4 %0.0340.035+3.5 %
fill / memset0.001—not launched (packed)0.001—not launched (packed)
SDPA dot_do_o0.1690.169+0.3 %0.3240.344+6.3 %
SDPA main kernel (dV + dS)2.2082.283+3.4 %4.3585.172+18.7 %
SDPA stage-3 dK GEMM1.3391.342+0.2 %2.6812.821+5.2 %
SDPA stage-3 dQ GEMM1.1221.121-0.1 %2.2542.396+6.3 %
SDPA dkv_reduce (GQA fold)0.1700.174+2.3 %0.3390.363+7.3 %
B5+B6 qk_norm_rope_bwd0.2210.223+0.9 %0.4860.502+3.1 %
dW_norm reduce0.0050.005+1.8 %0.0050.005+4.8 %
B7 dW_qkvg wgrad GEMM1.3281.346+1.3 %2.7722.916+5.2 %
B8 dh dgrad GEMM1.2901.333+3.3 %2.8943.023+4.4 %
B4 packed SDPA setup (metadata, per-sequence descriptors)—0.018packed only—0.060packed only
all launches (median of the per-iteration sums)9.5889.811+2.3 %19.54621.294+8.9 %

Both come from the same CUPTI launch records (30 profiled iterations, the first 3 dropped): a stage row is the mean over the 27 kept iterations of that stage’s device time, the all-launches row the median of the same iterations’ per-iteration sums, so the stage rows add up to the mean total (9.662 / 9.826 ms at 32/2, 19.574 / 21.324 ms at 64/8), 0.1-0.8 % above the median.

Backward, varlen packing [2048, 4096, 6144, 8192] (unfused; fuse_wgrad_overlap moves the packed block by -0.8..+1.3 %, positive = faster; the FLOP-scaled column is the same estimate as in the forward table):

heads Q/KVmaskviewpacked msTFLOP/s (% of peak)vs the FLOP-scaled dense-time estimatespeed-up vs dense torch at B x S_maxspeed-up vs the per-sequence torch loop
32/2causalbackward5.7572387 (63 %)+8.2 %+287.4 %+153.9 %
32/2causaltraining step7.8802551 (67 %)+7.4 %+312.8 %+166.6 %
32/2densebackward7.7352443 (64 %)+10.0 %+267.3 %+130.5 %
32/2densetraining step10.4652610 (69 %)+8.3 %+281.9 %+137.6 %
64/8causalbackward11.8422379 (63 %)+8.0 %+286.6 %+148.4 %
64/8causaltraining step16.2412539 (67 %)+5.9 %+313.7 %+162.7 %
64/8densebackward15.9922406 (65 %)+8.8 %+265.7 %+124.0 %
64/8densetraining step21.7572558 (69 %)+7.0 %+279.0 %+132.8 %

Scheduler policy: the packed SDPA pins SCHED_NATURAL. The same block built with SCHED_LPT on the packed SDPA (bitwise-identical O, LSE and block output) changed the whole-block forward time by 32/2: 2K -0.2 %, 8K -2.4 %, 32K -1.0 %; 64/8: 2K -1.7 %, 8K -2.8 %, 32K -1.4 % (2K / 8K / 32K causal, bf16; positive would be a gain; the FP8 packed decode ignores the policy: 32/2 -0.0 %, 64/8 -0.3 %), so NATURAL stays.

  • The SDPA epilogue gate is also reachable through the graph API: an sdpa node followed by sigmoid and mul pointwise nodes on O is served fused by the Rubin d256 FROST SDPA engines — see Attention, “Fused epilogue gate”.
  • How the block is composed (workspace, streams, fusion knobs, typed declines) and how to build the next one: Composing multi-kernel blocks in Python.
  • The MLA sibling of the fused projection epilogue: GEMM + RoPE + MXFP8 Projection.
  • Tests: test/python/gated_attention_block/cutedsl/ (layout contract, reference oracle, end to end, FP8, MXFP8, fp4 weights / fp4 O (test_block_fp4.py, test_proj_gemm_fp4.py, test_quantize_fp4.py) and their backward (test_block_backward_fp4.py), packed sequences (test_block_thd.py, test_block_thd_backward.py, test_block_thd_mxfp8.py: the per-sequence oracles, the typed declines, the dense-vs-packed pins), per-stage kernels, stream ordering).