> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/cudnn/latest/operations/sparse_attention/csa/llms.txt. # CSA Fused Compressor **This is an experimental API and subject to change.** ## Overview The CSA module hosts CuTe-DSL kernels for the CSA/HCA experimental attention variants (the components that are not shared with the [DSA module](/operations/sparse_attention/dsa)). Its first operation is the **fused Compressor**: one forward and one backward kernel for the `Compressor` gated-softmax pooling region (THD packed layout) used by CSA/HCA in Megatron-LM. The kernels were ported from Megatron-LM at the maintainers' request ([Megatron-LM PR #5984](https://github.com/NVIDIA/Megatron-LM/pull/5984); measurements and numerics in [Megatron-LM issue #5968](https://github.com/NVIDIA/Megatron-LM/issues/5968)). The eager region they replace decomposes into ~39 forward and ~51 backward kernel launches per call (at `compress_ratio = 4`, `coff = 2`) and materializes `(total_comp, 2*ratio, 1, head_dim)` window intermediates. The default fused path is one forward and one backward kernel plus one fp32 `dAPE`-buffer zero-fill. Deterministic dAPE replaces that fill with a fixed-order reduction stage (two stages for ratio 4). Both backward modes write `dKV`/`dScore` in full, including exact zeros to never-consumed positions, so those buffers need no fills. ### Semantics For each THD segment `s` (`cu_seqlens[s]..cu_seqlens[s+1]`) and each output block `b` of `ratio` tokens, with the overlapping window form (`coff == 2`, window size `2 * ratio`): - `k in [0, ratio)`: previous block's token, first-half projection column, APE row `k` — invalid for the segment's first block (score `-inf`, kv `0`); - `k in [ratio, 2*ratio)`: own block's token, second-half projection column, APE row `k - ratio`. The own-block window form (`coff == 1`, window size `ratio`) drops the overlap: every `k in [0, ratio)` is the block's own token on projection column `j` with APE row `k`, and every window is fully valid (no first-block exception). ```text out[b, j] = sum_k kv[w(b,k), c(k,j)] * softmax_k(score[w(b,k), c(k,j)] + ape[k % ratio, c(k,j)]) ``` computed in fp32 with a single final bf16 rounding. Per-segment tail tokens (`seqlen % ratio`) are dropped, as in the eager code. Output rows beyond `cu_seqlens_comp[-1]` (a static CUDA-graph capacity) are exact zeros on SM90 and SM100+; the forward skips their APE/KV/score loads and softmax, and the backward ignores incoming gradients on those rows. Consequently a positive static capacity is valid even when every segment is shorter than `ratio`. ### Numerics All arithmetic is fp32 with one final bf16 rounding; `mul.rn.f32` / `fma.rn.f32` are pinned in PTX so results do not depend on compiler FMA contraction. The numerics contract is **per ratio family** (the two families intentionally differ — do not assume the ratio \{2, 4\} guarantees at ratio=128): - **`ratio in {2, 4}`** (generic kernels; the bitwise contract — ratio=4 as originally validated, ratio=2 added on the same terms): against an fp32-intermediate eager reference (same op order, fp32 throughout), `dKV`/`dScore` are **bit-identical** and the forward matches within one bf16 rounding step on a tiny fraction of elements. - **`ratio == 128`** (deterministic tolerance contract): the kernels are **deterministic and faithful to the fp32-intermediate eager reference** (the same eager region computed with fp32 intermediates and one final bf16 rounding — the comparator every number below is measured against): 1. **Determinism** — forward `out` and backward `dKV`/`dScore` are bitwise run-to-run deterministic on all supported inputs (see below; NaN-prefill replay tested, including bit-stable replay of the NaN pattern on the gate's overflow-intermediate case), 2. **Same values within final-bf16 rounding on the gate tolerances** — `out`/`dKV`/`dScore` match the eager reference within differing elements `<= max(1, 0.1%)` of the tensor and `max_abs <= 1.6e-2`, thresholds **calibrated on the gate's documented input distribution** (`kv`, `grad_out` ~ N(0, 1) bf16, `score` ~ N(0, 1.5²) bf16, `ape` ~ N(0, 0.25²) fp32). Absolute bf16 deviations are NOT magnitude-free — bf16's grid is relative, so scaling `kv`/`grad_out` by `2^k` scales every deviation exactly `2^k`, with differing-element counts and fp64 parity unchanged for as long as the scaled inputs keep every fp32 intermediate finite (see 3.). The gate's ×64 scaled-input case commits that as evidence (deviations ×64 forward/`dKV`, ×4096 `dScore`; counts and parity identical to its unit-scale twin), and 3. **The eager reference's non-finite propagation** — both sides compute the window math in fp32, so finite inputs that overflow the eager reference's fp32 intermediates (a `score + ape` sum or a backward `kv · grad_out` product beyond fp32 range — elementwise ops both sides evaluate identically) poison the kernels' outputs as they poison the reference's, instead of the kernels returning clean-looking finite values. The gate's overflow-intermediate case commits this: `score` = bf16 max with `ape` = fp32 max drives fp32 `score + ape` to +Inf, every fused output is NaN with the NaN mask equal to the eager reference's on all four outputs, and the pattern replays bit-stable; the fp64 comparator is explicitly skipped there because the fp64 oracle stays finite (a NaN-vs-finite distance gates nothing). The mirror caveat: the fused evaluation order saturates EARLIER than the eager one near fp32 max — its forward chunk partials are un-normalized (up to `2 · ratio ×` the reference's normalized weighted sums), so e.g. uniform-score inputs with `|kv|` within a factor `~2 · ratio` of fp32 max return ±Inf where the eager reference is still finite. Non-finite inputs follow the same rule wherever they fall in the forward's window chunks: a NaN or +Inf `score + ape` at one window position makes that output NaN, and a -Inf position (a masked one) gets weight 0 (`test_r128_forward_special_values_match_eager`). On inputs whose fp32 intermediates stay finite in BOTH evaluation orders — the eager reference's and the fused kernels' (which saturate earlier, see 3.) — `out`/`dKV`/`dScore` additionally carry **fp64-oracle parity**: per case and per tensor they are **at least as close to an fp64 oracle as the fp32-intermediate eager reference itself** (`err_fused <= err_eager * (1 + 1e-6) + 1e-4`, asserted in the tests and on every finite-intermediate case of the committed gate script): the fused reorders and fast-exp buckets do not lose accuracy against the eager reference on the tested envelope. Worst observed on the gate distribution (19 unit-scale cases = 15 union — the 128k-token d=512 packs select the large-bucket schedules — + 4 padding, B200; reproduce with `benchmark/csa/gate_csa_compressor_r128.py`, which also asserts the case list selects every shipped schedule): forward 0.0275% differing elements (9/32,768), `dKV` 0.0041% (2,770/67,108,864), `dScore` 0.0113% (104/921,856); worst `max_abs` 3.91e-3 for forward/`dKV` (the 131k-token d=512 cases) and 1.95e-3 for `dScore` (bf16-rounding magnitudes). The deviations come from measured-win reduction reorders (forward chunked-softmax merges; the backward's fused `den`/`S` partial merge with a hoisted `1/den`) and an `ex2.approx`-based fast exp enabled on specific schedule buckets in BOTH directions (forward: per-bucket table entries; backward: the default outside the d=128 vec=1 small-pack buckets). `dAPE` (fp32, atomic accumulation) is gated at `max_abs <= 1e-3` vs the fp32 eager reference on the gate distribution — observed worst 2.0e-6, run-to-run replay deltas of the same magnitude — and scales the same way (observed 2.0e-3 under the ×64 case). `dAPE`'s distance from the fp64 oracle fluctuates run to run around the eager reference's own (within ~1.5× either way — atomic reduction-order noise), so its parity check is asserted on the unit-scale cases, where the formula's absolute term dominates, and recorded on scaled inputs. In BOTH families the forward, `dKV` and `dScore` are **bitwise run-to-run deterministic** (fixed chunk boundaries and merge orders, no atomics in those outputs; NaN-prefill replay is part of the test suite). The backward has two dAPE policies: - `deterministic=False` (default) keeps one fp32 atomic per `(k, dim)` per CTA. This fastest path is not bitwise deterministic and raises under strict `torch.use_deterministic_algorithms(True)` (warn-only mode warns and runs). Its class plan requires `workspace=None`; workspace never switches a compiled atomic plan to deterministic execution. - `deterministic=True` writes each CTA's register-local dAPE sums to caller-owned fp32 partial planes and launches fixed-order reduction stages. Ratio 4 first reduces independent groups of 32 CTA planes: eight warps each sum four contiguous planes, then warp 0 merges those eight sums. A second kernel folds the resulting group sums with an eight-way contiguous tree. This hierarchical grid supplies thousands of CTAs on long packed sequences instead of assigning the whole partial axis to 16 CTAs. The ratio-4 producer accumulates 8 rows per CTA below 65,536 compressed rows and 24 rows per CTA at or above that boundary. The long-context schedule cuts producer partial storage by about two thirds while retaining enough CTA parallelism; the selected row count is part of the compiled/fast-launch cache key. Ratio 2 and ratio 128 use a serial ascending-CTA fold. Both forms contain no dAPE atomics, fully overwrite `grad_ape`, are bitwise replay-deterministic, and are accepted by strict torch deterministic mode. The class API exposes `scratch_workspace_bytes()` and requires a contiguous uint8 `workspace` of at least that size at `execute`; the high-level wrapper allocates it. Both launches use the caller stream and are CUDA-graph capture/replay safe after `compile()`. On SM90, production `coff=2`, ratio-2 backward adds one measured 16-row producer bucket while keeping 8 rows for smaller capacities. Atomic plans switch at 8,192 compressed rows for `head_dim=128`. For `head_dim=512`, the atomic 16-row bucket is limited to the measured `[4,096, 5,121)` safe interval; larger capacities retain 8 rows because additional grid-wave occupancy steps showed regressions. Deterministic plans switch at 4,096 and 2,048 rows for `head_dim=128` and `512`, respectively, and keep 16 rows thereafter. Other `coff`/head dimensions and architecture families retain the 8-row schedule. The deterministic 16-row bucket halves the partial workspace. The selected row count is bound by the prepared plan, so `execute()` does not query device capability or specialize on device-side sequence metadata. #### Deterministic dAPE performance CUDA-event medians over 100 iterations after 30 warmups on one GB200 (capability `(10, 0)`, CUDA 12.9, PyTorch 2.9.1, CUTLASS DSL 4.7) are below. Atomic and deterministic variants share all operand addresses, measurement order alternates AB/BA, and the whole run is repeated with plan construction order reversed. `operation` includes the atomic path's required dAPE zero-fill; deterministic dAPE overwrites its output and therefore needs no fill. JIT and allocations are outside the measurement. | configuration | workspace | atomic kernel | deterministic kernel | operation ratio, deterministic / atomic (both construction orders) | |---|---:|---:|---:|---:| | r4, c2, d128, 1x8192 | 1.031 MiB | 32.5–34.0 us | 39.8–42.9 us | 0.96–0.98x | | r4, c2, d128, 3x8192 | 3.094 MiB | 40.6–41.1 us | 45.7–47.1 us | 0.94–0.95x | | r4, c2, d512, 1x8192 | 4.125 MiB | 42.5–42.8 us | 50.8–51.2 us | 0.97–0.99x | | r4, c2, d512, 3x8192 | 12.375 MiB | 90.7–91.5 us | 89.6–91.0 us | 0.92–0.93x | | r128, c1, d128, 1x8192 | 4 MiB | 31.1–32.6 us | 38.2–41.8 us | 0.91–0.93x | | r128, c2, d128, 3x8192 | 8 MiB | 47.5–48.0 us | 53.1–54.5 us | 0.93–0.94x | | r128, c1, d512, 1x8192 | 16 MiB | 40.8–41.0 us | 42.2–44.0 us | 0.84–0.86x | | r128, c2, d512, 3x8192 | 16 MiB | 111.7–112.1 us | 115.9–116.9 us | 0.95–0.96x | For the production `coff=1`, `head_dim=128`, ratio-4 long-context path with local pooling disabled, the compressor runs after the raw CP shards are gathered and therefore still sees the global packed token count. CP32 and CP64 therefore present the same 256K compressor shape. On one L20D (capability `(10, 3)`, CUDA 13.0, PyTorch 2.12.1, CUTLASS DSL 4.8.1), the current unequal-document shapes measured as follows; each range covers both plan-construction orders: | configuration | workspace | atomic operation | deterministic operation | deterministic / atomic | |---|---:|---:|---:|---:| | r4, c1, d128, packed 256K, CP32/64 | 5.502 MiB | 116.6–118.0 us | 100.6–101.1 us | 0.85–0.86x | | r4, c1, d128, packed 1M, CP64 | 22.002 MiB | 372.0–373.9 us | 308.5–309.2 us | 0.83x | Compared with the 8-row deterministic producer on the same source and host, the 24-row schedule reduces workspace from 16.5 to 5.502 MiB at 256K and from 66 to 22.002 MiB at 1M. Deterministic operation time also falls from 105.0 to about 100.9 us and from 328.5 to about 309.0 us, respectively. dAPE remains within the benchmark's fp64 error gate (maximum absolute errors `1.80e-5` and `4.52e-5`). Reproduce the long-context rows with `benchmark/csa/bench_csa_dape_deterministic.py --long-context`. For the compact matrix above, omit that option; use `--construct-order deterministic-first` for the reversed build and `--atomic-only` against an unchanged-source checkout for the independent control. ### Support surface (`check_support`) | architecture | ratio | coff | head dimension | forward | backward | deterministic backward | |---|---:|---:|---:|---|---|---| | SM90 | 2 or 4 | 1 or 2 | generic launch bound below | yes | yes | yes | | SM100+ | 2 or 4 | 1 or 2 | generic launch bound below | yes | yes | yes | | SM100+ | 128 | 1 or 2 | 128 or 512 | yes | yes | yes | | SM90 | 128 | any | any | no | no | no | The primary SM90 training configuration is packed THD, `ratio=4`, `coff=1`, `head_dim=128`, BF16 KV/score/output, FP32 APE and int32 prefix metadata. Both dAPE policies use the same forward/backward math; `deterministic=True` selects the caller-workspace fixed-order reduction. - SM90 and compute-capability major **>= 10** for the generic ratio `{2, 4}` kernels. Ratio 128 remains SM100-or-newer only; its schedule and tolerance contract have not yet been validated on SM90. - `ratio in {2, 4}`, `coff in {1, 2}` (`coff == 2` is the production CSA/HCA configuration, `coff == 1` the own-block window form; `ratio == 2` is the configuration used in production training) — served by the generic kernels, which are generic over `(ratio, head_dim, coff in {1, 2})` but keep the whole pooling window in registers (register-bound beyond `ratio = 32`) - `ratio == 128`, `coff in {1, 2}`, `head_dim in {128, 512}` — served by dedicated kernels (see [ratio=128](#ratio128) below); the wrappers and class APIs route by `ratio` transparently, and the gate can be widened per configuration once validated. **The ratio=128 numerics contract differs** (see [Numerics](#numerics)) - BF16 `kv` / `score` / `out`, FP32 `ape`, int32 `cu_seqlens` / `cu_seqlens_comp` - int32 flat offsets: `total_tokens * coff * head_dim < 2**31` and `total_comp * head_dim < 2**31` - Capacity-only rows do not require a token window. Device prefix metadata still owns the logical contract: every valid compressed row must describe a complete `ratio` token block. - `head_dim <= 8388480` (forward launch `gridDim.y` bound) - contiguous tensors on one CUDA device, with 16-byte-aligned base pointers (4-byte for the int32 cu_seqlens) — contiguity does not imply base alignment for storage-offset views, so this is checked per call ## Installation ```bash pip install nvidia-cudnn-frontend ``` For a source checkout in a CUDA 13 environment, keep the CUDA extra when installing or pinning CuTe DSL: ```bash python -m pip install 'nvidia-cutlass-dsl[cu13]>=4.8.0' python -m pip install -e . ``` ## API Usage ### JAX ops (SM100 ratio-4 CSA / ratio-128 HCA) ```python from cudnn.ops import dsa out = dsa.compressor_forward_jax( kv, score, ape, cu_seqlens, cu_seqlens_comp, total_comp=static_output_rows ) # DSv4 HCA: non-overlapping ratio-128 pooling, kv/score width == head_dim. out = dsa.compressor_forward_jax( kv, score, ape, cu_seqlens, cu_seqlens_comp, total_comp=static_output_rows, ratio=128, coff=1 ) grad_kv, grad_score, grad_ape = dsa.compressor_backward_jax( kv, score, ape, cu_seqlens, cu_seqlens_comp, grad_out, ratio=128, coff=1 ) ``` `cudnn.ops.dsa.compressor_forward_jax` and `cudnn.ops.dsa.compressor_backward_jax` accept JAX arrays, eagerly or under `jax.jit`, and require no PyTorch. They support `ratio=4, coff=2` (CSA) and `ratio=128, coff in {1, 2}` with `head_dim in {128, 512}` (HCA), BF16 KV/scores, FP32 APE and int32 packed-sequence metadata, on SM100-family GPUs. Forward and backward are separate calls; the caller owns the VJP. The PyTorch counterparts are `cudnn.ops.dsa.compressor_forward` and `cudnn.ops.dsa.compressor_backward`, the same functions as `cudnn.csa.csa_compressor_forward_wrapper` and `csa_compressor_backward_wrapper`. ### High-level wrappers ```python from cudnn import CSA # forward: (total_tokens, coff*head_dim) BF16 kv/score, (ratio, coff*head_dim) FP32 ape, # (B+1,) int32 cu_seqlens / cu_seqlens_comp result = CSA.csa_compressor_forward_wrapper( kv, score, ape, cu_seqlens, cu_seqlens_comp, ratio=4, head_dim=128, coff=2, total_comp=None, # defaults to cu_seqlens_comp[-1] (synchronizes); pass a static # capacity explicitly to stay CUDA-graph capture-safe stream=None, ) out = result["out"] # (total_comp, head_dim) BF16 grads = CSA.csa_compressor_backward_wrapper( kv, score, ape, cu_seqlens, cu_seqlens_comp, grad_out, ratio=4, head_dim=128, coff=2, stream=None, ) grad_kv, grad_score, grad_ape = grads # BF16, BF16, FP32 ``` Set `coff=1` for the `ratio`-token own-block window (and use `kv` / `score` tensors whose packed width is `head_dim`); `coff=2` selects the `2 * ratio` overlapping window shown above. The wrappers keep a bounded LRU of prepared API instances. The generic forward artifact is shared per `(ratio, head_dim, coff, device)`; generic backward adds `deterministic` and the bounded `rows_per_cta` schedule bucket. In particular, the generic row-count domain is the finite set `{8, 16, 24}`: SM90 ratio-2 uses the documented capacity bucket, and ratio-4 deterministic backward has only its documented short/long capacity buckets. The SM90 and SM100-family artifacts remain device-scoped and are lowered for the active device; packed token totals and per-sequence length distributions within a schedule bucket are runtime data, not compile specializations. ### Class API ```python from cudnn import CSACompressorForward, CSACompressorBackward op = CSACompressorForward( sample_kv, sample_score, sample_ape, sample_cu_seqlens, sample_cu_seqlens_comp, sample_out, ratio=4, coff=2, ) op.check_support() op.compile() op.execute(kv, score, ape, cu_seqlens, cu_seqlens_comp, out, current_stream=None) ``` `CSACompressorBackward.execute` additionally takes `grad_out` and the `grad_kv` / `grad_score` / `grad_ape` buffers. `grad_kv` / `grad_score` may be **uninitialized**: the kernel writes every position (disjoint, atomic-free stores; never-consumed positions — segment tails, the last block's first-half columns (`coff == 2` only; `coff == 1` has no first-half columns), segments shorter than `ratio`, token-capacity padding beyond `cu_seqlens[-1]` — get exact zeros from their unique owning CTA, matching autograd). When `total_comp == 0` the kernel is not launched and the buffers are left untouched (zero them yourself if you need autograd's exact zeros; the high-level wrapper does). With the default `deterministic=False`, `grad_ape` must be **zero-initialized before every `execute` call and before every CUDA-graph replay that reuses the buffer**: the kernel only accumulates into it with fp32 atomics. The high-level wrapper supplies this zero-fill, including during graph replay. With `CSACompressorBackward(..., deterministic=True)`, call `scratch_workspace_bytes()` after `check_support()`, then provide `execute(..., workspace=...)` a same-device, contiguous CUDA uint8 buffer that is 16-byte aligned, does not overlap any input or gradient tensor, and has at least the reported capacity. The deterministic path fully overwrites `grad_ape` without a zero-fill. ### CUDA graphs The launch path is capture-compatible once the kernels for a `(ratio, head_dim, coff)` configuration are compiled: run one warmup call (or `compile()`) per configuration before capturing, and pass `total_comp` explicitly. A call that would JIT under capture raises a `RuntimeError` instead of corrupting the capture. For deterministic backward capture, construct and compile `CSACompressorBackward(..., deterministic=True)`, query `scratch_workspace_bytes()`, and allocate the returned workspace before capture. The captured `execute()` performs no JIT, allocation, host synchronization or device-prefix readback. Prefix contents may change at the captured device addresses on replay; tensor addresses, prepared capacities and launch topology remain fixed. ### Environment variables - `CUDNNFE_CSA_COMPRESSOR_FAST_LAUNCH=0` — disable the cached-launch host optimization (a per-config snapshot of the CuTe-DSL launch state, replayed with in-place argument mutation; it removes tens of microseconds of per-call host overhead for these microsecond-scale kernels). The snapshot construction introspects private-but-stable DSL internals; on any structural mismatch (e.g. a future `nvidia-cutlass-dsl` upgrade) it falls back to the regular launch path automatically. ## Performance Reproduce the primary SM90 configuration with: ```bash python benchmark/csa/bench_csa_compressor.py --ratio 4 --coff 1 --head-dim 128 python benchmark/csa/bench_csa_dape_deterministic.py --sm90-primary python benchmark/csa/bench_csa_dape_deterministic.py --sm90-primary --construct-order deterministic-first ``` The first command reports eager/fused forward, backward and CUDA Graph replay timing; the latter two compare atomic and deterministic dAPE with JIT and allocation excluded. Measured on one SM90 GPU (CC 9.0, driver 570.148.08, CUDA 13.0, cuDNN 9.20, PyTorch 2.12.1, CUTLASS DSL 4.8.1); BF16 `kv`/`score`, FP32 `ape`; `ratio = 4`, `coff = 1`, `head_dim = 128`; median of 50 iterations after 20 warmup: | THD pack | eager fwd | fused fwd | fwd | eager bwd | fused bwd | bwd | eager graph total | fused graph total | graph total | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 1 x 8192 | 205.7 us | 40.9 us | **5.0x** | 270.0 us | 54.4 us | **5.0x** | 202.2 us | 17.9 us | **11.3x** | | 3 x 8192 | 222.9 us | 39.8 us | **5.6x** | 261.6 us | 57.2 us | **4.6x** | 306.4 us | 28.4 us | **10.8x** | The opt-in deterministic backward used 0.52 MiB and 1.55 MiB of caller workspace for the two packs. Its operation time was 35.87 us and 35.49 us, respectively, versus 29.84 us and 35.04 us for atomic dAPE (1.20x and 1.01x). The deterministic dAPE maximum absolute error versus the fp64 oracle was `3.41e-6` and `5.46e-6`. Measured on 1x B200 (CC 10.0, driver 590.48.01); BF16 `kv`/`score`, FP32 `ape`; `ratio = 4`, `coff = 2`; THD packs of 8192-token sequences; eager baseline = the exact replaced region of Megatron-LM `Compressor._forward_thd` on identical inputs. *Isolated GPU kernel time* (nsys, sum of kernel durations per iteration, 50 iterations after 20 warmup; no launch/host overhead; backward includes its `dAPE` zero-fill): | THD pack | head_dim | eager fwd | fused fwd | fwd | eager bwd | fused bwd | bwd | |---|---|---|---|---|---|---|---| | 1 x 8192 | 128 | 117.8 us | 4.5 us | **26.5x** | 187.2 us | 12.8 us | **14.6x** | | 3 x 8192 | 128 | 229.8 us | 10.0 us | **23.0x** | 352.7 us | 22.2 us | **15.9x** | | 1 x 8192 | 512 | 263.3 us | 12.4 us | **21.2x** | 425.0 us | 22.8 us | **18.6x** | | 3 x 8192 | 512 | 664.3 us | 35.0 us | **19.0x** | 1155.8 us | 66.0 us | **17.5x** | *End-to-end wall clock of the same region* (CUDA events, median of 100 after 30 warmup; includes launch overhead; eager backward goes through torch autograd with the forward outside the timed region, fused backward is the explicit backward wrapper call — not comparable to the kernel-time numbers above): | THD pack | head_dim | eager fwd | fused fwd | fwd | eager bwd | fused bwd | bwd | |---|---|---|---|---|---|---|---| | 1 x 8192 | 128 | 343.7 us | 37.4 us | **9.2x** | 558.7 us | 51.9 us | **10.8x** | | 3 x 8192 | 128 | 389.3 us | 38.2 us | **10.2x** | 585.8 us | 62.4 us | **9.4x** | | 1 x 8192 | 512 | 423.3 us | 39.9 us | **10.6x** | 666.5 us | 62.8 us | **10.6x** | | 3 x 8192 | 512 | 831.7 us | 62.7 us | **13.3x** | 1503.6 us | 108.3 us | **13.9x** | The previously published per-call wall clock (commit `b3ceb7c`) was `333.7 / 37.8 / 506.7 / 47.2 us` at `1 x 8192 / 128` (and analogously for the other packs). The re-run above reproduces the forward columns within ~±4% and the fused/backward columns within +2-14%: the fused per-call wall clock carries the cached-launcher host optimization, whose snapshot replay jitters run-to-run (the published fused-forward is even non-monotonic in pack size, `37.8 -> 35.1 us`), and eager backward drifts with GPU boost-clock state. The CUDA-graph replay numbers below collapse that host jitter into a single replay. *CUDA-graph replay of the same region, both implementations captured symmetrically as a forward-only graph and a forward+backward graph* (median of 100 after 30 warmup, replay timing via `benchmark/csa/bench_csa_compressor.py`; capturing each side into a graph collapses its per-operation launches into a single replay, so this is the fairest wall-clock basis for launch-bound shapes): | THD pack | head_dim | eager fwd | fused fwd | fwd | eager total | fused total | total | |---|---|---|---|---|---|---|---| | 1 x 8192 | 128 | 119.2 us | 10.8 us | **11.0x** | 364.6 us | 22.8 us | **15.9x** | | 3 x 8192 | 128 | 235.3 us | 14.8 us | **15.8x** | 668.4 us | 38.6 us | **17.3x** | | 1 x 8192 | 512 | 269.8 us | 17.1 us | **15.7x** | 802.5 us | 40.0 us | **20.0x** | | 3 x 8192 | 512 | 673.1 us | 39.2 us | **17.1x** | 2057.6 us | 107.8 us | **19.0x** | `eager total` / `fused total` capture forward + backward together. The eager total graph captures the autograd backward of the captured forward against stable, pre-allocated zero `.grad` buffers (the captured region zeros them in place, then runs forward + backward, so every replay accumulates into a zeroed buffer — numerically identical to a single fresh backward, verified per shape in the harness); the fused total graph captures the forward wrapper immediately followed by the backward wrapper. The backward replay alone is not a separately captured quantity — it is approximately `total - fwd` and is reported only as that reference, never as a measured column. Graph speedups are `eager / fused` of the displayed µs, truncated to one decimal. Capturing each side into a graph narrows the absolute eager-vs-fused forward gap most on the smallest, launch-bound shape (about 2.8x at `1 x 8192 / 128` forward) and barely on the largest, less launch-bound shape (1.2x at `3 x 8192 / 512`); the speedup ratio is similar to or larger than per-call because a graph replay collapses each side's per-operation launches into one. Environment: driver 590.48.01, PyTorch 2.13.0 (CUDA 13.3), `nvidia-cutlass-dsl` 4.6.1. Measurement basis: identical inputs over exactly the replaced region for both implementations; per-call eager backward = the torch autograd backward with the forward outside the timed region; graph eager backward = the autograd backward of the captured forward, captured together with it against stable zero `.grad` buffers; fused backward = the backward wrapper (kernel + the fp32 `dAPE` zero-fill + host validation, no autograd engine). Both wall-clock tables above (per-call and graph) are from a single run of `benchmark/csa/bench_csa_compressor.py`; the kernel-time table is from nsys. An ncu hardware-ceiling audit of the **ported kernels (prior to the two optimization commits)** — cache-flushed, `--set full`, all four benchmark shapes — showed measured DRAM read volume matching the algorithmically necessary bytes within 1% (the THD gather adds no over-read; stores fully coalesced at 32/32 bytes per sector, loads 29-30/32), with neither L2 (<27% of peak) nor DRAM (<33%) close to saturation at these microsecond-scale sizes: the gap to a pure DRAM-floor time was a mix of sub-wave grid width / occupancy, memory latency, and (at the largest shape) issue pressure — not wasted traffic. The two optimizations that audit identified are folded in here: 32-bit vectorized forward accesses and backward kernel-side zero-writes replacing the two bf16 grad-buffer fills. They do not change the bytes the kernels must read, so the traffic-optimality conclusion carries over; the utilization percentages above predate them. ## ratio=128 `ratio = 128` is served by dedicated kernels (`compressor_sm100_r128.py`): the generic kernels keep the whole `coff * ratio` window in per-thread registers, which spills kilobytes of local memory per thread at `ratio = 128` (255-register cap). The dedicated forward streams the window with a chunked softmax (one CTA per output row, the window split over `threadIdx.y` chunk-rows, per-column `(max, denom, acc)` triples, one fixed-order smem merge); its launch schedule is **bucketed by the output-row count** (small / default / large, all precompiled for CUDA-graph safety) and per bucket selects the online-rescale or two-phase accumulation form and, where it measured faster, an `ex2.approx`-based fast exp. The dedicated backward stages each row's window into shared memory chunk-parallel, computes `exp` and per-chunk partial `den`/`S` sums in the same pass, merges the partials per column in a fixed chunk order, and stores the gradients with a hoisted `1/den` multiply — plus kernel-side zero-writes to never-consumed slots and, by default, fp32-atomic `dAPE`, exactly as at ratio=4. The opt-in deterministic policy shares the generic family's CTA-partial plus fixed-order reduction design. The backward defaults to the same fast exp outside the d=128 small-pack (vec=1) buckets. **The ratio=128 numerics contract is the deterministic tolerance contract described in [Numerics](#numerics), NOT the generic-family (ratio \{2, 4\}) bitwise-`dKV`/`dScore` contract.** The reduction orders (forward chunk merge, backward `den`/`S` partial merge), the backward's hoisted reciprocal, and the fast-exp buckets all differ from the eager op order by design, each adopted on a measured same-GPU win and gated on tolerance + fp64-oracle parity (worst observed deviations in [Numerics](#numerics)). Everything stays bitwise run-to-run: fixed chunk boundaries and merge orders, no atomics in forward/`dKV`/`dScore`. Both kernels are register-flat in the window length (ptxas sm_100a: forward 32-51 registers, backward 64-128 registers, 0 spill / 0 stack across every shipped (config, schedule) kernel — 16 kernels total; reproduce the per-kernel table with `benchmark/csa/reg_probe_csa_compressor_r128.py`) and JIT in ~0.4-1.0 s per configuration. The backward picks its `rows_per_cta` at launch time (a runtime argument — no recompile) so the grid fits one resident wave; the static row capacity fixes it under CUDA-graph capture. Small packs (`nb_total <= 192`, d=128) switch the backward to a vec=1 schedule bucket (also precompiled) for grid-fill. Every backward bucket except the c1d128 default additionally loads `grad_out` once per row and keeps the register vec live across the phase barriers (the `goreuse` schedule field): adopted per bucket on measured pure-kernel wins — up to ~26% on the vec=1 small buckets, whose small grids underfill the machine, so the +16/+1 register cost is not the binding constraint — while the c1d128 default measured ~1.3% slower with it at 131k tokens and keeps the phase-4 reload. `dKV`/`dScore` are bitwise-identical with and without the field (the same read-only values move into registers). Measured on 1x B200 (CC 10.0, driver 590.48.01), torch 2.13.0 / CUDA 13.3 / `nvidia-cutlass-dsl` 4.6.1; eager baseline = the fp32-intermediate reference region of the test suite on identical inputs (the numerics-contract reference; the upstream eager region differs only by a bf16 weight-rounding cast at ~equal cost). Packs are single-sequence THD unless noted. *Isolated GPU kernel time* (nsys, sum of kernel durations inside a `cudaProfilerApi`-gated 100-iteration unsynced loop / 100; forward and backward each measured in their own loop; the fwd+bwd total is one combined loop per training step — forward + backward + the ~1 us fp32 `dAPE` zero-fill — and is not the sum of the standalone columns, because the interleaved backward changes L2 residency for the forward at long context): | config | tokens | eager fwd | fused fwd | fwd | eager bwd | fused bwd | bwd | fused fwd+bwd total | |---|---|---|---|---|---|---|---|---| | coff 1, d 128 | 8192 | 84.7 us | 5.7 us | **15.0x** | 127.3 us | 6.3 us | **20.2x** | 13.0 us | | coff 1, d 128 | 131072 | 451.8 us | 14.9 us | **30.2x** | 471.9 us | 63.7 us | **7.4x** | 82.5 us | | coff 2, d 128 | 8192 | 187.1 us | 6.4 us | **29.1x** | 246.2 us | 10.5 us | **23.4x** | 18.1 us | | coff 2, d 128 | 65536 | 726.3 us | 16.4 us | **44.4x** | 960.0 us | 61.2 us | **15.7x** | 76.7 us | | coff 1, d 512 | 65536 | 641.1 us | 48.4 us | **13.2x** | 795.0 us | 104.2 us | **7.6x** | 150.0 us | | coff 2, d 512 | 65536 | 2315.2 us | 78.8 us | **29.4x** | 3261.9 us | 205.8 us | **15.9x** | 287.7 us | | coff 2, d 512 | 131072 | 4526.5 us | 117.6 us | **38.5x** | 6304.3 us | 457.6 us | **13.8x** | 580.7 us | *End-to-end wall clock per call* (CUDA events, median of 100 after 30 warmup; fused backward includes a `grad_ape.zero_()` per call; not comparable to the kernel-time numbers above — at small packs the ~9 us python launch path dominates the 5.7-11 us kernels): | config | tokens | fused fwd | fused bwd | |---|---|---|---| | coff 1, d 128 | 8192 | 14.1 us | 20.8 us | | coff 1, d 128 | 131072 | 24.4 us | 82.1 us | | coff 2, d 128 | 8192 | 15.2 us | 25.3 us | | coff 2, d 128 | 65536 | 25.5 us | 75.6 us | | coff 1, d 512 | 65536 | 56.3 us | 118.9 us | | coff 2, d 512 | 65536 | 88.1 us | 220.8 us | The ratio=128 numerics contract (21-case gate: 15 union + 1 scaled-input + 4 padding cases, each running the determinism, tolerance and fp64-parity gates, plus 1 overflow-intermediate case that gates bit-stable NaN replay and NaN-pattern equality vs the eager fp32 reference with the fp64 comparator explicitly skipped, with shipped schedule coverage asserted) and the per-kernel ptxas table are reproducible from the committed scripts `benchmark/csa/gate_csa_compressor_r128.py` and `benchmark/csa/reg_probe_csa_compressor_r128.py`. ## Testing ```bash (cd test/python && pytest compressed_sparse_attention/cutedsl/test_CSA_compressor.py) ``` The tests validate numerics against an fp32-intermediate eager reference (bitwise `dKV`/`dScore` at ratio \{2, 4\}; the deterministic tolerance contract at ratio=128, with fp64-oracle parity on finite-intermediate inputs), the upstream eager numerics, plus ragged packs, static-capacity padding, kernel-side zero-writes into uninitialized gradient buffers (NaN-canary with exact-zero assertions on every never-consumed slot class, and the `total_comp == 0` zeros fallback), run-to-run determinism for both dAPE policies, deterministic workspace validation and CUDA-graph overwrite semantics, atomic-path `grad_ape` zeroing ownership, the ratio=128 dispatch envelope (schedule selection at every `nb_total` bucket boundary at L0, and one execution of every shipped (config, schedule) kernel against the full contract — tolerance, fp64-oracle parity, determinism — at L1; run `pytest -m "L0 or L1"`), CUDA-graph capture/replay, and `check_support` boundaries. ## Docs - [Aligned HCA Backward](https://docs.nvidia.com/cudnn/latest/operations/sparse_attention/csa/aligned_hca_backward.md)