Skip to navigation

SDPA Backward, d = 256 and d in (256, 512] (SM107 / Rubin; SM100 / SM103 via the 2x2-datapath body)

View as MarkdownOpen in Claude

This is an experimental API and subject to change.

Overview

SDPA backward pass for head dimension 256 on the NVIDIA Rubin line (SM107, cc 10.7 through 11.9), implemented with CuTe DSL primitives: tcgen05 MMAs with the accumulator resident in TMEM, TMA loads and stores, a 2-CTA cluster per KV block and a warp-specialized schedule. It consumes the forward activations (Q/K/V/O), the loss gradient dO and the forward Stats (natural-log LSE) and produces dQ/dK/dV.

Three FROST engines serve the d = 256 pass on cc 10.7 (cudnn.sdpa.bwd.engines), and two more rows share the chain (the SM100 / SM103 d = 256 row and the cc 10.7 d in (256, 512] row, both described further down). The FP8 and MXFP8 rows are opt_in. The SM100 / SM103 d = 256 row is a default candidate that the backward heuristics rank against the backend; sdpa_bwd_sm107 and sdpa_bwd_sm107_d512 are default candidates that rank behind it (they serve a graph the backend declines). For the opt-in rows, set CUDNN_FRONTEND_ENABLE_FROST_ENGINES=1 before import cudnn; for any of the five, pin the engine from the ranked plan list (graph.plans / graph.select_plan(i)) when validating or measuring, because the bf16 d256 graph also has a native backend plan:

  • sdpa_bwd_sm107 — bf16 / fp16 sdpa_backward() graphs;
  • sdpa_bwd_sm107_fp8 — per-tensor FP8 E4M3 sdpa_fp8_backward() graphs (cuDNN’s contract: scalar descales in, FP8 or half gradients plus the amax_dQ/dK/dV/dP outputs);
  • sdpa_bwd_sm107_mxfp8 — block-scale MXFP8 sdpa_mxfp8_backward() graphs (E4M3 payloads with F8_128x4 E8M0 scale factors, bf16 o_f16 / dO_f16, bf16 gradients, no amax_* outputs — a graph requesting them is declined; see MXFP8). The sole provider of that graph on Rubin: cuDNN 9.27 has no MXFP8 d = 256 backward kernel there.

A fourth engine, sdpa_bwd_sm100_d256 (bf16 / fp16, SM100 / SM103, cc 10.0-10.6; a default candidate, ranked against the backend in sdpa/bwd/heuristics.py), runs the SAME chain on the Blackwell line over the 2x2-datapath main kernel (see “The 2x2-datapath body” below); pin it by name without the opt-in flag (startswith("sdpa_bwd_sm100_d256")), because the bf16 d256 graph also has a native backend plan there (cuDNN engine 5 on B200 / 9.26).

A fifth engine, sdpa_bwd_sm107_d512 (bf16 / fp16, cc 10.7 - 11.9, default behind the backend, 2026-10-01), serves d in (256, 512] (multiples of 8, envelope-served on 512-wide tiles) on the same line — see “The d in (256, 512] row” below. It is the half-precision FROST d > 256 backward on cc 10.7. cuDNN 9.26 builds no d > 256 backward plan on cc 10.7, so a pin by startswith("sdpa_bwd_sm107_d512") is still the honest way to measure it.

The opt-in sdpa_bwd_d512_fp8 pair serves exact D512 per-tensor E4M3 sdpa_fp8_backward graphs on SM100, SM103 and SM107. Inputs use compact BSHD storage, with FP32 Stats and the standard tensor descales/scales; gradients may be E4M3, FP16 or BF16. It supports MHA/GQA/MQA, top-left causal or unmasked attention, and a left sliding window. The deterministic flag keeps the same fixed ownership and GQA reduction order. Scratch contains only FP32 delta rows, padded to 128 queries; no full score or dS matrix is stored. THD, padding masks, bottom-right alignment, right-band widening, bias, sinks and MXFP8 remain outside this pair’s contract. Numerical and bitwise tests have run on SM100 and SM107; SM103 execution remains unverified.

There is no separate high-level standalone wrapper for the whole pass; use the graph API or an adapter’s execute method. Standalone THD execution takes per-batch sequence lengths, and adapters may be constructed with external_delta=True. Graph execution supports ragged inputs on all three d = 256 rows when the bindings expose max_total_seq_len_q and max_total_seq_len_kv on the FP8 and MXFP8 nodes; older pybind extensions decline those graph plans. Graph plans compute delta internally. See Sequence lengths and THD below for the packed contract.

The kernels live in python/cudnn/sdpa/bwd/kernels/sm107/ (d256_f16_4x1/kernel.py, d256_fp8/kernel.py, d256_mxfp8/kernel.py). Shared configuration lives in python/cudnn/sdpa/bwd/config_sm107.py, the adapters in python/cudnn/sdpa/bwd/api_dsl_sm107.py, and the rest of the launch chain is shared with the other Blackwell-line backward engines (kernels/bprop_matmul_blackwell.py, kernels/bprop_chain_common.py).

Requirements

The CuTeDSL runtime (nvidia-cutlass-dsl >= 4.8.0, which knows sm_107a and the 576-column TMEM allocation Rubin needs, plus apache-tvm-ffi), a CUDA 13.4+ driver (the kernels use Rubin’s 327 KiB oversized shared-memory carveout), and an SM107-line device.

API usage

import cudnn, torch
# BSHD-physical storage, logical (B, H, S, D) as cuDNN declares it
def bshd(b, h, s, d):
return [s * h * d, d, h * d, 1]
g = cudnn.pygraph(io_data_type=cudnn.data_type.BFLOAT16,
intermediate_data_type=cudnn.data_type.FLOAT,
compute_data_type=cudnn.data_type.FLOAT)
q = g.tensor(name="q", dim=[b, h_q, s_q, 256], stride=bshd(h_q, s_q, 256))
k = g.tensor(name="k", dim=[b, h_kv, s_kv, 256], stride=bshd(h_kv, s_kv, 256))
v = g.tensor(name="v", dim=[b, h_kv, s_kv, 256], stride=bshd(h_kv, s_kv, 256))
o = g.tensor(name="o", dim=[b, h_q, s_q, 256], stride=bshd(h_q, s_q, 256))
do = g.tensor(name="dO", dim=[b, h_q, s_q, 256], stride=bshd(h_q, s_q, 256))
stats = g.tensor(name="stats", dim=[b, h_q, s_q, 1], stride=[h_q * s_q, s_q, 1, 1],
data_type=cudnn.data_type.FLOAT) # the forward's natural-log LSE
dq, dk, dv = g.sdpa_backward(name="bwd", q=q, k=k, v=v, o=o, dO=do, stats=stats,
attn_scale=1.0 / 16.0, use_causal_mask=True)
for t, (h, s) in ((dq, (h_q, s_q)), (dk, (h_kv, s_kv)), (dv, (h_kv, s_kv))):
t.set_output(True).set_data_type(cudnn.data_type.BFLOAT16).set_dim([b, h, s, 256]).set_stride(bshd(h, s, 256))
g.validate(); g.build_operation_graph(); g.create_execution_plans([cudnn.heur_mode.A])
# pin the FROST row (the bf16 d256 graph has a native backend plan too)
idx = next(i for i in range(len(g.plans)) if g.get_plan_name_at_index(i).startswith("sdpa_bwd_sm107"))
g.select_plan(idx); g.check_support(); g.build_plans()
ws = torch.empty(g.get_workspace_size(), dtype=torch.uint8, device="cuda")
g.execute({q: q_t, k: k_t, v: v_t, o: o_t, do: do_t, stats: stats_t, dq: dq_t, dk: dk_t, dv: dv_t}, ws)

The FP8 row is the same graph through g.sdpa_fp8_backward(...) with E4M3 q/k/v/o/dO, the twelve scalar descale_* / scale_* inputs as (1, 1, 1, 1) fp32 tensors, dQ/dK/dV declared E4M3 (or bf16 / fp16), and any of amax_dQ/dK/dV/dP marked set_output(True).

The kernel chain

One backward call is a fixed sequence of launches on the caller’s stream, every scratch buffer carved from the caller’s workspace (graph.get_workspace_size() is a build-time function of the shape). All three d = 256 rows (sdpa_bwd_sm107, sdpa_bwd_sm107_fp8, and sdpa_bwd_sm107_mxfp8) are prepared launches (bwd/prepared_sm107.py, kernels/sm107/prepared_host.py): the plan compiles ONE pointer-host artifact that runs the whole chain from device pointers, and every stage below — the padding copies, the seq_kv fill and the (rare) dS zero-fill included, plus the fp8 row’s fold + quantize passes — is a kernel of that artifact. No torch op runs on the execute path; the graph binds its variant pack straight into the artifact, follows the handle’s stream and captures into a CUDA graph.

dot delta = rowsum(dO ∘ O) (fp8: × descale_o · descale_dO)
main per (kv block, head, batch), one 2-CTA cluster: walks the q tiles that
attend the block; dV accumulates in TMEM and is stored per Q head;
dS = attn_scale · P ∘ (dP − delta) is written to a kv-major
[B, H_chunk, S_kv, S_q] GMEM workspace (fp8: e4m3 dS_q = e4m3(dS · scale_dP),
amax_dP over the fp32 dS before the scale)
mm_dk dK = dS · Q batched GEMM over the workspace (bprop_matmul_blackwell,
the d = 256 cluster tile: 2x1, 256 × 256 per pair, no N padding;
fp8: the K64 fp8 arm over the e4m3 dS and the e4m3 Q payload,
epilogue · descale_dP · descale_q — then · scale_dK → e4m3 dK +
amax_dK at MHA, or fp32 true-unit per-Q-head partials under GQA;
MXFP8: the block-scale arm, EPI_NONE -- the caller's bf16 dK at MHA,
fp32 true-unit per-Q-head partials under GQA)
mm_dq dQ = dSᵀ · K same GEMM, the other operand major (fp8: · descale_dP · descale_k,
amax_dQ, · scale_dQ → the gradient dtype, straight into dQ)
fold GQA only (half row): dK/dV = fixed-order sum of each KV head's group of
per-Q-head partials. fp8 row: dV always (fold + amax_dV + scale_dV + cast);
dK under GQA, in the same launch (the partials are fp32 under GQA and summed
BEFORE the amax, scale and cast, so the gradient is rounded once -- like the
reference; a bf16 partial would be rounded a second time). MXFP8 row: dK / dV
under GQA through dkv_reduce -- the dK partials fp32 (rounded once), the dV
partials bf16 (one rounding per group member: the main kernel stores them from
its epilogue and an fp32 staging does not fit its shared memory)

The workspace is head-chunked (and batch-chunked on the half row) to one 8 GiB budget shared by all three rows (at B=1 H=128 S=8K: 64-head chunks on the half row, one 128-head launch on the fp8 row, 32-head chunks on the MXFP8 row; at 16K every row still chunks); the artifact’s host loops over the chunks with head_base / batch_base, so one compiled artifact serves every launch of a plan. The budget is a chunking constant: get_workspace_size() reports the chunk’s whole carve and the caller allocates it — a caller that cannot hold it bounds the plan with deselect_workspace_greater_than(...), a typed decline before any launch. Measured on Rubin (cc 10.7, 212 SMs, SM clock 2376 MHz) at that shape, whole row: 8 GiB over 4 GiB is +3.7 % causal / +0.8 % dense on the bf16 row (4 -> 2 launches), +0.9 % dense on the fp8 row (2 -> 1), and +9.3 % causal / +1.4 % dense on the MXFP8 row (8 -> 4; the control twin’s value — the arm read +11.6 % against a slot whose control pair spread 2 %); every extra chunk launch ends in a scheduler tail the next launch cannot fill.

On the SM100 / SM103 dense BF16 d = 256 row, a chunk may also divide one GQA group when even one complete group at batch size one exceeds that budget. Existing whole-group choices that fit are preserved. Each smaller chunk reads the group’s K head and writes its own global per-query-head dK/dV partials; the final reduction still sums the complete group once, after every chunk. FP16, packed THD and other architecture rows retain their whole-group choices.

On the same dense BF16 D256 row, chunks that contain complete GQA groups reuse dK/dV partial buffers after reducing each chunk. With no query or KV padding, those buffers hold only (batch_chunk, S_kv, query_head_chunk, D) instead of all batches and query heads. Input batch/head origins stay global; partial-output origins are local, and the ordered FP32 group reduction writes the corresponding final dK/dV slice before reuse. The dS budget and chunk geometry are unchanged. Chunks smaller than one GQA group retain the global partials and final reduction described above. FP16, packed THD, padded shapes, and other architecture rows keep their existing partial-buffer allocation.

The 8 GiB target is not a hard workspace limit: one query head alone can exceed it, and gradients and other scratch regions also require storage. For example, dense BF16 B=4, S_q=S_kv=131072, H_q=8, H_kv=2, D=256 uses a one-head dS slab of 32 GiB rather than a four-head slab of 128 GiB, reducing the full required workspace from about 132 GiB to 36 GiB. Allocate the size reported by the selected plan; these sizes do not include model tensors or communication buffers.

The two GEMMs render the shared template at its d = 256 cluster tile (MatmulTemplateParams.cgrp_tile_mn = (256, 256): cluster 2x1, one 256-row × 256-column tile per 2-CTA pair, six 32 KiB operand stages, a double-buffered 256-column TMEM accumulator so one tile’s epilogue overlaps the next tile’s mainloop). The SM100 d512 chain’s (512, 512) tile would put the N-rank pair of every cluster on columns 256..511 that a d = 256 gradient does not have — half of every cluster’s MMA work as TMA-OOB zero loads and clipped stores. The selection is the sm107 adapter’s alone (Rubin line, d = 256); the SM100 chain’s renderings are unchanged, and a bitwise pin (test_stage3_d256_rendering_is_bitwise_the_padded_one) holds the two tiles’ dQ / dK / dV to identical bits (same k-tile walk, same 256x256x16 instruction, same fp32 accumulation order).

Main kernel

One cga2 pair (2 CTAs, 12 warps each) owns a 256-row KV block of one (batch, head) and iterates over the 128-row q tiles that attend it (lane = kv row, the transpose of the forward). Per q tile the MMA warp issues S = K·Qᵀ, dP = V·dOᵀ (both into TMEM) and dV += P·dO (TMEM accumulator, resident for the whole block); eight softmax warps recompute P = exp2(S·scale·log2e − LSE·log2e) from the forward’s LSE, form dS, and stream it through a SMEM ring to the workspace by TMA store. The bf16/fp16 body splits the K·Qᵀ contraction so half of K rides TMEM (a UTCCP per block); the fp8 body keeps a lookahead MMA order and a two-slot fp8 P ring in TMEM; the MXFP8 body picks its S issue order per mask arm at load time (the lookahead on masked graphs, K·Qᵀ at the top of the q iteration on dense ones — the same MMAs, bitwise identical, measured faster each way). Rubin’s 576 TMEM columns and 327 KiB SMEM carveout are what let dV stay resident at d = 256 — the SM100 d512 backward is a different, three-stage shape.

The 2x2-datapath body (kernels/sm100/d256_f16_2x2/kernel.py; the SM100 row, and the Rubin twin)

The same chain has a second main kernel on the 2x2 tcgen05 datapath: tcgen05.mma.cta_group::2 with the collective M = 128, i.e. 64 kv rows per CTA per sub-block (the 4x1 body above is M = 256, 128 rows per CTA). A 64 x N fp32 accumulator then lands as row m -> TMEM lane m + 64 * (n // (N/2)), column n % (N/2): S and dP take 64 columns each, dV 128, so S / dP are double-buffered and everything fits 512 non-exclusive TMEM columns, and every MMA operand is an SMEM SS operand (K / V / P as 64-row 128-B-swizzled K-major slabs, Q / dO N-split, dO_dv in the transposed BT form) — no UTCCP K-split, no TMEM P alias. P is lane-written into a 2-deep SMEM ring (each lane a 64-B half row at the swizzled address) and published to the leader CTA’s MMA with fence.proxy.async plus a .release.cta arrive; every slot reuse (S, dP, P) is an explicit mbarrier. The body has two profiles selected by TemplateParams.datapath_2x2_profile (bwd/config_d256_2x2.py): profile 1 (one sub-block per CTA, a 128-row kv block per pair, 210 KiB of SMEM, descriptor version 0) is the SM100 / SM103 row sdpa_bwd_sm100_d256 (bwd/api_dsl_sm100_d256.py), the footprint that fits 227 KiB / 512 columns; profile 2 (two sub-blocks per CTA, a 256-row block, 322 KiB, descriptor version 1) is the Rubin twin behind the module constants api_dsl_sm107.BWD_D256_2X2 (default False: the shipped 4x1 rendering is unchanged) and BWD_D256_2X2_PROFILE (2; 1 runs the SM100 body on Rubin, the A/B arm). The register split is per profile: 176 / 152 on profile 1, 224 / 56 on profile 2 (its 64-column compute lanes spill at 176). Both keep the 256-row kv WRITE PAIR: a 128-row block derives its q range from the pair it belongs to, so the stage-3 GEMMs’ K-trim and the no-zero-fill contract are exactly the 4x1 chain’s. The MMA issue order is a config constant (CfgBwdD256x2.MMA_LOOKAHEAD): profile 1 ships the NATURAL order (S(i), dP(i), BMM2(i) per q tile) — on B200 it measured stage 2 at 3781 us against 4525 us for the lookahead order (S(i+1) between dP(i) and BMM2(i)) on the dense B=1 H_q=32 H_kv=2 S=8192 bf16 shape, 2020 vs 1974 us causal (A/B/A x3 with a control pair: the causal leg stays 1.4 % slower under NATURAL, the dense leg 11.8 % faster); profile 2 keeps the lookahead. Whole backward on that shape: 5673 us dense / 3256 us causal against the backend’s engine 5 at 7265 / 3939 us (see the tracker footnote). On the Rubin board (2026-10-01) the twin traced and ran on both profiles — fp64-oracle accepts, poisoned-workspace cases, two-launch bitwise, and dQ / dK / dV BITWISE the 4x1 body’s — but measured slower than the 4x1 body (profile 2 at 1.20x / 1.09x (causal stage 2 / whole) and 1.22x / 1.12x (dense); CUDA events, the 4x1 body as the in-session control), so BWD_D256_2X2 stays False.

The d in (256, 512] row (sdpa_bwd_sm107_d512; bwd/api_dsl_sm107_d512.py)

Head dims above 256 are a different chain on this line too: the SM100 large-head-dim backward (bwd/api_dsl.py::SdpaBwdDslSm100; see the SM100/SM103 D512 BPROP entry in the FROST support tracker) — delta = rowsum(dO·O) -> a stage-2 kernel that writes S and dS to [B, H_chunk, S_q, S_kv] GMEM workspaces (heads chunked to a 4 GiB budget) -> dV = Sᵀ·dO, dK = dSᵀ·Q, dQ = dS·K as the bprop_matmul_blackwell GEMMs at the (512, 512) cluster tile -> the GQA fold — with stage 2 ALWAYS the 2x2-datapath body kernels/sm107/d512_f16_2x2/kernel.py at the cc 10.7 ring arm: two independent tcgen05.mma.cta_group::2 pairs per (4,1,1) cluster, 64 q rows per CTA, both BMMs (Q·Kᵀ, dO·Vᵀ) as SMEM SS operands with d streamed in 64-column chunks, an 8-stage K/V chunk ring (the SM100 body runs 4) and two cast stages (SM100: 1) filling 320 of the line’s 325 KiB usable SMEM, 256 TMEM columns, tcgen05 descriptor version 0 at zero margin. The file is the SM100 twin’s sibling (diff sm100/ sm107/ is the review surface; its rendering at these parameters is PTX-identical, pinned by a committed md5 record — a board-only pin: the public 4.7.0 DSL has no sm_107a target, so the md5 and SASS cases skip in public CI and the fork is held there by its code-diff allowlist, its source pins and its import-time _requires), and the row is its own EngineSpec rather than a widened sdpa_bwd_sm100 because the 4x1 role split that row renders by default never ran on this line. Served: d in (256, 512] in multiples of 8 (envelope-served on 512-wide tiles: d = 264 pays d = 512’s MMA; the floor is exclusive at 256, which the d256 rows above own), any S_q / S_kv (padded to 256 / 128 and masked), dense, top-left and bottom-right causal, right-band widening, sliding window (left), MHA / GQA / MQA, BSHD-physical io, contiguous fp32 Stats, bf16 / fp16. Declined on day one (each asserted by test_sdpa_bwd_d512_sm107.py and flipped only with a board-run accept + tracker line): THD / ragged, dense_flex layouts, dense padding masks, sink / dSink, bias / dBias, deterministic, decode shapes. Pin it by startswith("sdpa_bwd_sm107_d512"); there is no backend d > 256 backward plan on cc 10.7 to fall back to.

Masks

The main kernel bounds WHICH q tiles a KV block attends (causal: from the diagonal; sliding window: up to the window), rounds that range outward to a 256-row q pair, and zeroes P on masked cells, so dV, dS, dK and dQ inherit the mask. The stage-3 GEMMs render a TWO-SIDED K-trim over the same band (dK starts at the block’s first attended q pair and ends after the window; dQ starts at the window and ends after the last attended kv block), so each GEMM multiplies only the band’s tiles — under a sliding window that is ~(W + 256) / S of the dense K range instead of all of it. The pair rounding is what makes the trim self-contained: every tile a GEMM reads was written by the kernel (masked cells as stored zeros), so the workspace needs no zero-fill under any mask; the one exception the adapter still fills for is a top-left window with S_q > roundup(S_kv + W, 256) (q rows past every block’s window, written by no block). Poisoned-workspace tests pin this per mask, and a bitwise pin against the untrimmed rendering keeps the trim numerically inert.

Sequence lengths

The kernels compile with every extent concrete (S_q % 128 == 0, S_kv % 256 == 0). Any other length is served by padding: Q / dO and the LSE (with +inf, so P = 0) are staged into zero-filled padded copies, K / V likewise, and a padded S_kv selects the kernels’ padded-mask specialization at the uniform real length, so every padded kv row’s dS / dV is exactly zero. The GEMMs read real-extent slices and write the caller’s tensors directly.

Per-batch KV lengths are served on the standalone surface of all three rows: construct the adapter (SdpaBwdDslSm107, SdpaBwdDslSm107Fp8 or SdpaBwdDslSm107Mxfp8) with seq_kv_lens_present=True and pass seq_kv_lens (a contiguous [B] int32 device tensor) to execute. The same padded-mask specialization then reads seq_kv_lens[b] in place of the uniform length: every kv row at or past its batch’s length is select-dead (dS = dV = 0 exactly), a length of 0 is a dead batch whose dQ / dK / dV are exact zeros whatever its Stats rows hold (0 or -inf) — and on the fp8 row the amax row gate follows the same per-batch length, so a dead or shortened batch folds nothing into amax_dV / amax_dP — and under bottom-right causal the diagonal is per batch (seq_kv_lens[b] − S_q) while the GEMMs’ K-trim is computed from the uniform S_kv − S_q — so for that arm the chain keeps the dS workspace zero-fill (ahead of every batch / head chunk on the batch-chunked bf16 row, where a chunk’s workspace slot may hold the previous batch’s dS in tiles the next batch’s narrower band does not write; once per execute on the fp8 / MXFP8 rows, which walk the whole batch in-grid) and drops a sliding window from the stage-3 trim (a window edge anchored on the uniform diagonal would skip live tiles of a shorter batch): the GEMMs read the plain bottom-right band there. Two contract points are device data the host does not validate: every entry must satisfy 0 <= seq_kv_lens[b] <= S_kv, and the K / V rows at or past a batch’s length must hold finite data (on the MXFP8 row their scale-factor atoms too: an E8M0 NaN byte past the length is a NaN dP) — the kernels select P = 0 on them, but dS is (dP − delta) ∘ P and NaN × 0 = NaN; the dense rows get finite pads from the adapter’s zero-filled staging copies, the per-batch arm reads the caller’s buffers as they are (finite garbage past the length is fine, a NaN is not). The graph padding mask stays declined on every row: a padded sdpa_backward graph carries seq_len_q as well as seq_len_kv (the frontend requires both) and no body threads per-batch Q lengths, so serving the graph form would mean ignoring the q lengths.

An externally computed delta is the other plan fact of that standalone surface, on every row, dense and THD: external_delta=True at construction declares that the caller computes stage 1’s delta = rowsum(dO ∘ O) and hands it to execute(..., delta_tensor=) — an fp32 contiguous tensor of external_delta_shape, 16-byte aligned, on the plan’s device, holding the raw row dot (attn_scale is applied in the main kernel): on a dense plan (B, H_q, S_q_pad) (S_q_pad = S_q rounded up to the 128-row q tile, zeros past S_q); on a THD plan the PACKED head-major (1, H_q, ceil128(T_q)) the packed chain reads at the packed token index (T_q = the plan’s token capacity, the declared max_total_seq_len_q tightened to B · S_max; zeros past T_q) — the dense layout at B = 1, S = T_q, so one producer serves both forms. The chain then launches no dot and reads O once less, and the workspace carve has no delta region (scratch_workspace_bytes() shrinks by exactly it); the operand is checked before any bind — dtype, shape, strides, device, alignment, each a typed ValueError — and a plan built without the flag refuses a delta. The producer this exists for is the gated attention block’s sigmoid-gate backward kernel, which forms the delta over the dO it stores in dot’s own reduction order, so the fused and the unfused block backward are bitwise equal (fuse_gate_bwd in gated_attention_block.md). The flag is independent of seq_kv_lens_present: each decides its own appended operand (the lengths, then the delta) and a plan built with both takes both at execute. It is a plan fact, not an eligibility fact — Capabilities and the graph path are unchanged (no graph declares a delta, so a graph plan keeps the chain’s own dot launch and its region). The units are the row’s: the bf16 / fp16 and MXFP8 rows read the raw half-precision dot (the MXFP8 row’s own pre-pass is dot over its o_f16 / dO_f16 ports, so a producer forming it in that order is bitwise the chain’s own); the fp8 row reads delta in true units, unscaled — its own pre-pass is the row sum of the e4m3 payload codes times descale_o · descale_dO, nobody applies those descales to a caller’s delta, so a delta formed from the bf16 O / dO binds as is and is not bitwise the row’s own pre-pass (its tests compare against an oracle fed the same delta). The pad rows [S_q, S_q_pad) must be finite zeros on every row: the kernels read them, and under the MXFP8 row’s block-scaled dS a 32-element q block straddling the pad folds them into the real columns’ E8M0 scale. The o / descale_o (fp8) and o_f16 / dO_f16 (MXFP8) operands stay required under the flag and are read by nothing. Under THD the same contract holds in the packed form above: the chain reads the caller’s tensor where it read its own delta region, launches no dot and carves no region (the carve shrinks by exactly H_q · ceil128(T_q) fp32 values); a producer that forms the delta in dot’s order over the packed O / dO is bitwise the chain’s own on the bf16 / fp16 and MXFP8 rows, and the fp8 row again reads it in true units.

Bottom-right causal at a ragged S_q is served on all three rows: every body takes the real lengths (seqlen_q_real / seqlen_kv_real) and derives the diagonal S_kv − S_q and the q-tile trim from them, never from the padded compile extent (Capabilities.bottom_right_s_q_multiple = 1 on every row).

THD / ragged (packed varlen)

sdpa_bwd_sm107 (bf16 / fp16), sdpa_bwd_sm107_fp8 (per-tensor E4M3) and sdpa_bwd_sm107_mxfp8 (block-scale MXFP8) serve a ragged backward: Q/K/V/O/dO and dQ/dK/dV declared as the envelope (B, H, S_max, D) with a per-tensor ragged offset over PACKED storage ([1, T, H, D] rows: element stride 1, head stride D, token stride >= H·D and a multiple of 8 elements), use_padding_mask=True with the per-sequence seq_len_q / seq_len_kv as (B,) int32 tensors, and BOTH max_total_seq_len_q / max_total_seq_len_kv declared (the packed workspace is sized from them at build time; a graph without them is declined at plan creation — on the quantized rows the sdpa_fp8_backward / sdpa_mxfp8_backward nodes and their bindings carry the two attributes as trailing keywords, and a pybind extension built before them cannot declare them, so a ragged fp8 / MXFP8 graph through such an extension is that typed decline while the standalone surfaces serve). Stats is the forward’s packed Stats in either layout the forward emits — token-major (T, H) or head-major (1, H, head_stride) with head_stride >= T. The standalone surface is SdpaBwdDslSm107(..., thd=True, max_total_seq_len_q=.., max_total_seq_len_kv=.., thd_stats_token_major=.., thd_stats_head_stride=..) — or SdpaBwdDslSm107Fp8(...) with the same keywords plus amax_requested, or SdpaBwdDslSm107Mxfp8(...) with the same keywords plus its MXFP8 sample operands — with execute(seq_q_lens=.., seq_kv_lens=..) taking (B,) lengths or (B+1,) prefix sums per side (the fp8 row’s twelve scalars and requested amax tensors, the MXFP8 row’s payloads and scale-factor tensors, as on their dense surfaces); thd_stats_head_stride (head-major only) is required when the Stats buffer’s head stride is not exactly the packed capacity — the FROST forwards emit (1, H, ceil64(T)) — and must cover the packed total (the graph path infers it from the ragged strides).

How it runs: one setup launch builds the metadata on device (no host cumsum), the dS workspace is blocked over packed KV tokens (each sequence owns a 256-row-aligned block), the main kernel claims (sequence, kv block, head) units from a device counter, reads the packed operands through descriptors clamped to the live packed totals (a NaN in the declared-but-unused capacity tail cannot reach an MMA), masks each sequence’s kv tail and q pad columns from its own lengths (bottom-right’s diagonal is S_kv[b] − S_q[b] per sequence) and stores dV through per-sequence clipped descriptors; the gradient GEMMs run over the blocked rows with per-sequence output descriptors and the same two-sided K-trim as the dense path, PER SEQUENCE (every bound derived from the sequence’s own lengths and its own diagonal, so a GEMM reads only dS tiles the kernel wrote for that sequence and a tile whose band is empty — a kv block no query attends, a q pair with no key — is stored as exact zeros without a read), so the workspace needs no zero-fill under any mask; under GQA the dQ GEMM runs once per head chunk over the packed K heads, as on the dense path. Nothing past the packed totals is written into the caller’s gradients: dQ / dK / dV stop at the per-sequence clipped output descriptors and the GQA fold stops at the live kv total on device (the rows up to the declared capacity keep whatever the caller left there). A unit without query rows — an empty-Q sequence, a spare unit of the occupancy-sized grid — loads every operand past the clamped extent (zero-filled), so even an all-NaN Q / dO capacity with no live query row yields exact-zero dK / dV. Served under THD: none / causal / bottom-right / sliding window, GQA / MQA, empty sequences on either side (their gradients are exact zeros), an external delta in its packed head-major form (see Sequence lengths above). Declined under THD: right-band widening, bias.

On the fp8 row the same mechanism runs in e4m3: packed e4m3 payloads through the packed-total-clamped descriptors, a kv-blocked e4m3 dS workspace (dS_q = e4m3(dS · scale_dP); bf16 on the FP8_DS_DTYPE = DTYPE_BF16 twin, which needs compact packed rows — token stride exactly H·D — for its exact e4m3 → bf16 upcast of Q / K), the fp8 K64 gradient GEMMs trimmed per sequence with their descale / quantize epilogue, and the fold + quantize passes bounded on device at the live totals (cu_k[B] for dV / dK, cu_q[B] for the twin’s dQ), so nothing past the packed totals is written into the caller’s gradients. Every amax is the max over the packed live region — dead units, kv pad rows, q pad columns and the capacity tail are excluded at all four fold sites (the main kernel’s row gate and fold values, the GEMM epilogue’s per-row gate, the bounded fold passes) — and there is one scale_dP per packed batch, the forward’s one-scalar-per-operand convention over packed tokens.

On the MXFP8 row the same mechanism runs over the packed e4m3 payloads (q / k / v / dO and the transposed-quantization q_T / k_T / dO_T, all packed [1, T, H, D] rows) with one contract the other rows do not have: the seven scale-factor tensors travel PACKED per-sequence-TILE-padded, the forward’s convention. Per head, every sequence’s ceil(s_b / 128) F8_128x4 tiles follow each other in cu_seqlens order — sequence b’s tiles start at cu_sf[b] = Σ_{i<b} ceil(s_i / 128), not at cu[b] / 128 — 1024 bytes per (head, tile) at d = 256: rowwise (descale_q / k / v / dO) the tile’s 128 rows x 8 groups; columnwise (descale_q_T / k_T / dO_T) BOTH D planes of the (head, tile) contiguous (plane stride one 512-byte atom, tile stride the whole slab — the dense tensors are D-plane-major over the whole tensor, and reading a packed tensor through that layout fetches plane 1 from the wrong place by an S-dependent offset). The packed tile count is a per-call fact derived from the bound buffer’s byte size: whole H x 1024-byte tile rows, one count per side (descale_q / q_T / dO / dO_T and descale_k / k_T / v must each agree), at least Σ_b ceil(s_b / 128) tiles per head (the scale-factor maps’ tile extent is the live total; a shorter buffer is read past its end — device data the host cannot check) and at most the plan’s capacity, the larger of ceil(max_total_seq_len / 128) + B tiles per head and the declared sample’s own count — the whole-row, one-count-per-side and capacity rules each a typed ValueError at bind; so the graph may declare any dims with the right byte total, the dense capacity B × ceil(S_max / 128) included (the forward’s graph layout), and bind a buffer of exactly those bytes. The producer’s pad bytes may hold anything (a 0xFF is an E8M0 NaN): the chain re-stages the five scale tensors whose pad positions are read — descale_v / dO / dO_T by the main kernel’s dP / dV operands and, under P-b, descale_q_T / k_T by the block-scale gradient GEMMs (whole atoms) — into packed staging copies with every byte scaling a position at or past its sequence’s length zeroed, per execute, from the device prefixes; descale_q / k pads are harmless (an S NaN is select-dead) and bind as they are. Both dS policies serve THD: P-c runs the bf16 THD gradient GEMMs over the packed q_T / k_T dequantized exactly to bf16 per token (no pad byte is read), P-b the block-scale arm’s THD leg (the kv-blocked payloads + atoms, B’s scale factors through the per-sequence SF tile prefixes) with dQ once per head chunk under GQA, as on the dense P-b chain: the dQ record’s b_head_group is the GQA group, so B and its scale factors are indexed by h // group (the SF tile prefix is a token-side term) and one launch covers the whole head chunk, bitwise the per-member launches. No amax (the row’s contract); Stats comes from the caller — no Rubin MXFP8 THD forward row feeds it yet.

FP8 numerics (sdpa_bwd_sm107_fp8)

Every scalar is a 1-element fp32 device tensor read in-kernel, never a host fold. S = S_acc · descale_q · descale_k; P is recomputed from the forward’s exact fp32 Stats and quantized to E4M3 with scale_s for the dV MMA; dV = dV_acc · descale_s · descale_dO; dP = dP_acc · descale_v · descale_dO; dS = attn_scale · P ∘ (dP − delta) in fp32 (amax_dP is its amax before the scale and the cast, as the C++ node reduces it), then dS_q = e4m3(dS · scale_dP) into an E4M3 workspace — the cuDNN recipe (delayed scaling: feed the previous step’s amax_dP). The gradient GEMMs run Rubin’s dense-FP8 K64 MMA over the e4m3 dS and the e4m3 Q / K payloads and undo both scalings in their epilogue: acc · descale_dP · descale_k (dQ) / · descale_q (dK), then amax_dQ / amax_dK over that true-unit value, · scale_dQ / scale_dK and the cast to the graph’s gradient dtype — written straight into dQ, and into dK at MHA. Under GQA the dK partials leave the GEMM in fp32 (the true-unit value, EPI_DESCALE) and the main kernel stores its per-Q-head dV partials in fp32 too (dtype_o = FP32); one fold launch sums each KV head’s group in fixed order and only then folds amax_dK / amax_dV, applies scale_dK / scale_dV and casts — the gradient is rounded once, like the reference (a bf16 partial would round it a second time). dV always takes that fold launch (bf16 partials at MHA, where the fold is a copy + amax, scale and cast); dK joins it under GQA, so the e4m3 chain runs ONE fold + quantize launch and no fold at all for dQ (quantized in its GEMM epilogue), as the kernel chain above lists it. api_dsl_sm107.FP8_DS_DTYPE = DTYPE_BF16 selects the pre-quantized twin used for A/B and oracle work: bf16 dS, bf16 GEMMs over exact E4M3 → bf16 upcasts of Q / K, bf16 partials, its three gradients folded + quantized in two launches (dV + dK in one, then dQ), descale_dP / scale_dP bound and unused.

MXFP8 numerics (sdpa_bwd_sm107_mxfp8)

The block-scale row is the same graph through g.sdpa_mxfp8_backward(...) — E4M3 q / k / v / dO plus the transposed-quantization payloads q_T / k_T / dO_T (each contraction axis needs its own 1x32 quantization), bf16 o_f16 / dO_f16, fp32 stats, and the seven F8_128x4-reordered E8M0 scale tensors (descale_q / q_T / k / k_T / v / dO / dO_T; descale_v is the rowwise V scale, as in the C++ node’s own reference math) — with dQ / dK / dV declared bf16. One kernel plus scale-factor plumbing (sm107/d256_mxfp8/kernel.py, the fp8 body’s pipeline): the scale factors ride their operands’ TMA barriers into TMEM and dequantize inside every tcgen05 block-scale MMA, so S and dP are in TRUE units; P is recomputed from the exact fp32 Stats and quantized to E4M3 with the fixed 2^8 scale (byte 119, cuDNN’s MXFP8 convention; p_scale_log2 is pinned to 8) into the fp8 body’s two-slot TMEM P ring (the scale factors of each q iteration ride along in the slot that is dead that iteration) for dV += P · dO_T; dS = attn_scale · P ∘ (dP − delta) is formed from the fp32 P (never the e4m3 P — pinned by test_ds_is_computed_from_the_fp32_p_not_the_e4m3_p).

dS policy: P-b ships (config_sm107.DS_SF_POLICY_DEFAULT = DS_SF_P_B, read by the adapter constant api_dsl_sm107.MXFP8_DS_SF_POLICY when the adapter is built — a module constant, never a knob): dS is quantized to e4m3 per 32-element block in both orientations — the kernel writes two payloads (ds_dk scaled per 32-q block of a kv row, ds_dq per 32-kv block of a q column) plus their F8_128x4 E8M0 atoms, and the gradient GEMMs render MatmulTemplateParams.block_scale over them and the columnwise q_T / k_T scale factors, dequantizing in the MMA with no dequant pass (Rubin-line only). Accept cells run the oracle at quantize_ds=True (its own 1x32 quantization of the fp32 dS) under the fp8 recipe on dV, dK and dQ alike: both sides round dS to e4m3 per block from fp32 values that agree to ~1e-6, so a midpoint flip of one dS cell moves one gradient row by one e4m3 step at the block’s scale — the fp8 row’s dS-flip class. P-c — dS through a bf16 workspace into the bf16 renderings over the exactly dequantized bf16 q_T / k_T (prepared_host.dequant_mxfp8_to_bf16_host: e4m3 × 2^(e−127) is a bf16 value), so dQ / dK carry no dS quantization at all, more accurate than cuDNN’s 1x32-quantized dS and the oracle twin (sdpa.mxfp8_ref.compute_ref_backward(..., quantize_ds=False)) — stays built and selectable through api_dsl_sm107.MXFP8_DS_SF_POLICY = DS_SF_P_C, with the bf16 row’s recipe on dK / dQ. The flip was a numerics change (the accept matrix re-run on Rubin, the support-matrix cell re-written), decided on the measured A/B: on Rubin (cc 10.7, 212 SMs, SM clock 2376 MHz) at B=1 H=128/128 S=8192 the whole backward is +22.0 % (dense) / +17.1 % (causal) faster than the bf16-dS chain; the block-scaled chain chunks its stage-2 workspace against the rows’ shared 8 GiB budget (32-head chunks at 8K H=128) for another +1.4 % / +9.3 % over a 4 GiB budget’s 16-head chunks (the control twin’s value; the arm read +11.6 %).

Padding on this row has one obligation the other rows do not: the producer’s scale-factor tensors cover ceil128(S) rows / groups and their pad bytes are undefined, while the kernel reads the pad positions (a 0xFF there is an E8M0 NaN → 0 × NaN in S on the Q side, NaN in the dead rows’ dS on the kv side). So the artifact re-stages descale_q / dO / dO_T (when S_q % 128 != 0) and descale_k / v (when S_kv % 256 != 0, grown to the kernel’s 256-row pad) with every pad byte zeroed, next to the zero-padded payload copies. Poisoned-pad RED-then-green tests pin it. Under THD the same obligation is per SEQUENCE: the scale tensors are packed per-sequence-tile-padded, and the chain re-stages descale_v / dO / dO_T (and descale_q_T / k_T under P-b) with every byte past each sequence’s length zeroed from the device prefixes (see THD above).

Under GQA the MXFP8 SDPA backward folds its per-Q-head dK partials in fp32 and rounds the sum once, like the reference, while its per-Q-head dV partials are bf16 (the kernel stores them from its epilogue; fp32 ones do not fit its 327 KiB shared-memory budget), so dV carries one bf16 rounding per group member where a once-rounded reference carries one in total (relative RMS about 3e-3 at a group of 4, the geometry the tests run, measured on the per-tensor fp8 row before it moved to fp32 partials); the modelled oracle folds dV the same way and the distance to a once-rounded fold is reported per cell. The dK fold is pinned bitwise: the row’s dK torch.equals the fixed-order fp32 sum of its own fp32 partials rounded once (test_mxfp8_gqa_dk_is_the_once_rounded_fold_of_its_fp32_partials); MHA is untouched (the GEMM writes the caller’s bf16 dK, the same kernels and bits).

Not produced: the amax_dQ / dK / dV outputs — a graph that marks them real (the backend’s canonical MXFP8 backward shape) is declined, typed. This row is the sole provider of the d = 256 MXFP8 backward on Rubin (cuDNN 9.27 has no such kernel on smVersion 1070), so that graph gets cudnnGraphNotSupportedError at plan creation.

Support surface and constraints

  • SM107-line devices (cc 10.7 – 11.9); sdpa_bwd_sm100_d256 on SM100 / SM103 (cc 10.0 – 10.6, bf16 / fp16 only, the 2x2-datapath body)
  • Head dims: d_qk = d_v = 256 exactly (no envelope)
  • Dtypes: bf16 / fp16 (sdpa_bwd_sm107); E4M3 payloads with E4M3, bf16 or fp16 gradients (sdpa_bwd_sm107_fp8; E5M2 is declined); E4M3 payloads with F8_128x4 E8M0 scales and bf16 o_f16 / dO_f16 / dQ / dK / dV (sdpa_bwd_sm107_mxfp8; fp16 gradients are declined — the bf16 GEMM stores its io dtype, an fp16 cast pass is a follow-up). Stats fp32, contiguous (B, H_q, S_q, 1)
  • Layout: BSHD-physical Q/K/V/O/dO/dQ/dK/dV (and q_T / k_T / dO_T / dO_f16; stride order 3,1,2,0); packed BSHD rows under THD on every row (the MXFP8 row’s scale-factor tensors packed per-sequence-tile-padded)
  • Masks: none, causal (top-left or bottom-right), sliding window (left, with or without causal); any S_q / S_kv on every row
  • GQA/MQA: any H_kv dividing H_q
  • Declined (asserted by tests): graph padding masks (seq_len_q/kv — a padded graph carries both lengths and no body threads per-batch Q lengths; per-batch KV lengths are served on every row’s standalone adapter, see Sequence lengths; a RAGGED padded graph is THD and served on sdpa_bwd_sm107, sdpa_bwd_sm107_fp8 and sdpa_bwd_sm107_mxfp8), sink / dSink, bias / dBias, right-band widening, dense_flex layouts, decode shapes (S_q == 1), use_deterministic_algorithm (the chains have no atomics; the claim waits on the bring-up sweep), dropout / ALiBi / softcap; on the MXFP8 row also the amax_dQ / dK / dV outputs, fp16 gradients and any p_scale_log2 != 8
  • Workspace (carved from the caller’s buffer): fp32 delta (not carved under the standalone adapters’ external_delta), one head/batch chunk of the dS workspace (B_chunk · H_chunk · S_kv · S_q bytes at e4m3 on the fp8 row, · 2 on the half and MXFP8 rows), padded staging copies when S_q / S_kv are not tile multiples, per-Q-head dK/dV partials under GQA; the fp8 row adds the per-Q-head dV partials (fp32 under GQA, bf16 at MHA) and, under GQA, the fp32 dK partials, plus an amax scratch; the MXFP8 row adds the block-scaled dS chain’s second e4m3 payload and two scale-factor atom tensors (or, on its bf16-dS twin, the two dequantized bf16 q_T / k_T slabs) and the zero-filled scale-factor pad slabs, and carves its GQA dK partials fp32 (its dV partials bf16; under THD: the packed scale-factor staging copies at the plan’s tile capacity and the per-sequence SF tile prefixes). Use graph.get_workspace_size().

The separate opt-in sdpa_bwd_d512_mxfp8 row supplies D512 block-scale backward on SM100/SM103/SM107 with CuTeDSL4.8.0. It consumes canonical F8_128x4 E8M0 scales and separately quantized Q_T/K_T/dO_T payloads directly, produces FP16/BF16 gradients, and keeps only FP32 row-delta scratch. Its compact dense MHA/GQA contract includes unmasked/top-left causal attention and causal left windows whose query rows retain a key. Deterministic execution uses fixed gradient owners. It does not produce Amax outputs.