SDPA backward, d = 64 or 128 (SM100 line and Rubin)
The opt-in sdpa_bwd_sm100_d128 engine serves dense multi-head attention with
BF16 or FP16 inputs and gradients, compact BSHD physical storage, and
contiguous FP32 natural-log Stats. Dense MHA supports head dimensions 64 and 128,
unmasked or top-left causal attention, and a left sliding window, including
rectangular query and key/value lengths within the dense MHA declaration
bounds below.
Packed THD MHA supports both dtypes at D128. Dense and THD GQA/MQA support
BF16 at D128, without sliding windows. Packed
and grouped paths require SM100, SM103 or SM107 (cc 10.0, 10.3 or 10.7), CuTeDSL 4.8 or newer (a build with the
sm_107a target on cc 10.7) and positive FP32 attention scale. The same kernels serve every admitted part; the compile
target follows the device. Grouped dense lengths must be multiples of 128;
packed grouped lengths may have tails or empty actual sequences. Other masks,
bias, sinks and dropout are declined. Dense MHA, GQA and MQA support deterministic
requests; the packed path still declines them.
Use the ordinary cuDNN graph preparation and execution API. After building and selecting a plan, allocate the size reported by that plan:
Plan construction selects the implementation from declarations and the build
device. On SM100, SM103 or SM107 with public CuTeDSL 4.8 or newer and a positive attention
scale and no deterministic request, the D64 and D128 MHA three-kernel chain (D128 without a sliding window, D64 with or without one) computes the row delta while initializing FP32 dQ
scratch, accumulates dQ and writes dK/dV, then converts dQ to the output type.
Sequence lengths need not be multiples of 128: the tail tiles’ loads are zero-filled,
their stores and dQ reductions are clipped, and the tail Stats rows contribute nothing.
The output conversion applies no extra scale. Its scratch size is
4 * B * H * round_up(S_q, 128) * (D + 1) + 240 bytes. The final 240 bytes cover internal
alignment for any valid 16-byte-aligned caller workspace.
Dense MHA with use_deterministic_algorithm=True selects the kernel pair on SM100 and
SM103 under the same CuTeDSL and positive-scale gates as the accumulation chain: the row delta,
a key-owner kernel that accumulates dK/dV on chip and stores them, then a query-owner
kernel that recomputes the score and score-gradient tiles and stores dQ once from its
on-chip accumulator. Every gradient tile has one owner and a fixed summation order, so
repeated executions return identical bits. It serves D64 and D128 in both dtypes,
unmasked or top-left causal attention, a left sliding window, and GQA/MQA with an
integral head ratio (the key-owner kernel walks the group’s query heads). When the
key tiles are unmasked and their count would leave the last wave of SMs mostly idle,
each key tile’s query range is split in two or four: every range stores FP32 dK/dV
partials to the workspace and one fold launch per output sums them in a fixed order.
The workspace holds the FP32 delta rows, 4 * B * H_q * round_up(S_q, 128) bytes,
plus 8 * splits * B * round_up(S_kv, 128) * H_kv * D bytes of partials under a
split, plus 240 bytes; query the plan. Dense MHA requests outside those gates (other devices,
older CuTeDSL, negative scales) use the one-kernel recompute implementation with zero
workspace; grouped requests outside them are declined. The choice is made when the
plan is built.
A D128 left sliding window without a deterministic request keeps the recompute
implementation. D64 uses native 64-wide operand/MMA tiles and gradient stores on every implementation; score and dS
tiles retain their 128-column geometry. A left window bounds both Q and KV owners’ reduction
loops and masks boundary tiles; rows containing one legal key have zero score
gradients and contribute no dQ/dK.
The graph’s left bound counts keys, while standalone window_size_left counts
positions before the query, so a graph left bound of 128 corresponds to a
standalone radius of 127. Window declarations also require
max(S_q,S_kv) + window_size_left + 129 <= 2**31-1.
Dense MHA declarations that do not meet the three-kernel eligibility conditions, including negative finite nonzero scales and CuTeDSL releases older than 4.8, use the one-kernel recompute implementation with zero workspace. Always query the plan’s size rather than assuming either implementation. Zero and nonfinite scales are unsupported.
Dense MHA declaration bounds
Dense MHA declarations are bounded before either implementation is selected:
max(S_q,S_kv) <= 2**31-128, B*H*S_q <= 2**31-1,
B*H*(ceil(S_q/128)+ceil(S_kv/128)) <= 2**31-1, and
max(S_q,S_kv)*H*128*2 < 2**40. These conservative limits keep tile rounding,
Stats indexing, the recompute launch grid and tensor-map byte strides
representable. Exceeding a limit declines the engine before compilation.
The accumulation implementation requires both attention scale and its product with log2(e) to remain finite and positive in FP32. A scalar override on an already compiled accumulation plan must satisfy that requirement. Invalid overrides raise before any kernel launches; they do not change the plan.
Inputs, outputs and workspace must remain on the declared CUDA device. Gradient outputs must not overlap each other or any input, and workspace must not overlap any operand. Execution binds the current buffers and stream, initializes its scratch each time, and performs no allocation, layout copy or device-to-host read. The plan supports CUDA-graph capture and replay. FP32 dQ atomic accumulation does not promise bitwise repeatability; the deterministic kernel pair does.
The three-kernel implementation is selected for D64, and for D128 without a window, on SM100, SM103 and SM107 when determinism is not requested. For other supported devices in the row’s range, or declarations that do not meet those conditions, plan construction uses the one-kernel recompute implementation described above. The engine remains opt-in while performance qualification proceeds.
For packed THD, declare positive max_total_seq_len_q and
max_total_seq_len_kv capacities and device sequence lengths. Each port’s
ragged offsets retain their declared origins, including nonzero initial
offsets and gaps. Packed rows require compact head/token strides; the logical
batch stride is unused. FP32 Stats may be packed token-major (T,H) or
head-major (1,H,head_stride), with the latter covering the physical query
capacity. Actual sequences may be empty, and lengths need not divide 128.
The packed MHA plan launches metadata/tensor-map setup, Delta plus dQ initialization, the backward main kernel, then dQ conversion. Its caller workspace holds FP32 dQ and Delta, sequence metadata, per-port origins and seven tensor maps per batch; it has no dK/dV workspace. Query the plan’s workspace size. Execution reads lengths and offsets on the device and performs no allocation, tensor conversion or host read of metadata. Gradient outputs and workspace must also be disjoint from all metadata operands. Declared capacities must be positive even when every actual sequence is empty. This path requires cc 10.0, 10.3 or 10.7 and both scale and scale*log2(e) finite and positive in FP32.
GQA/MQA requires Hq > Hkv > 0 and an integral Hq/Hkv; ratios need not
be powers of two. Each query head uses KV head q_head // (Hq/Hkv) and writes
its own BF16 dK/dV partials. A final kernel sums them in fixed query-head order using FP32
and converts to the native gradient dtype. Thus partial storage rounds each
query head’s contribution before the group fold. Regular correctness tests
retain the same numerical tolerances as MHA.
Grouped plans append two caller-workspace regions: dense
[B,Skv,Hq,128] or packed [1,round128(Tkv_compact),Hq,128], each at two
bytes per element. Packed partials use compact KV prefixes; the final fold
writes each dK/dV port at its independent declared origin. Dense grouped
execution has four launches; packed grouped execution has five. Their dQ
scratch, device metadata, alias checks, rebinding and capture contract match
the corresponding MHA plan. Always query workspace size from the plan.
Grouped BF16 GQA/MQA at long sequence lengths on cc 10.0 / 10.3 / 10.7 (SM100 / SM103 / SM107)
may be served by a two-CTA pair implementation (one cluster of two CTAs per 256-row
KV block) instead of the one-CTA grouped body, on packed THD and on dense compact
BSHD declarations alike. The op-level field cta_pair on the adapter selects it:
auto (the default) uses the pair when both declared S_q and S_kv are at least
8192 and the device admits a cluster of two, off always uses the one-CTA body, and
on uses the pair at any sequence length and raises a typed error naming the
failing predicate (layout, compute capability, cluster admission, grouped heads)
otherwise. The pair implementations (accumulation_grouped_thd_2cta on packed THD,
accumulation_grouped_dense_2cta on dense compact BSHD) keep the grouped contract:
the same inputs, outputs and declaration requirements as the one-CTA grouped body of
the layout (dense: compact BSHD storage, contiguous FP32 Stats, lengths that are
multiples of 128, BF16), a workspace sized like the packed grouped chain (the dense
twin’s packed capacities are B * S_q and B * S_kv), five launches, and dQ accumulated
in FP32 without ordering guarantees. Always query the plan’s workspace size. The dense
twin’s dK/dV are bitwise the packed chain’s for the same per-sequence data.
Grouped support and packed support do not imply performance parity with another backend; that qualification is a separate step.
FP16 grouped attention is declined because per-query-head partials can overflow FP16 before cancellation in the final group sum. Wider partial storage is required before enabling that path. FP16 dense and packed MHA remain supported.