> This page is for version Latest (default).
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/cudnn/latest/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/cudnn/llms.txt.

# CSA Fused Compressor

**This is an experimental API and subject to change.**

## Overview

The CSA module hosts CuTe-DSL kernels for the CSA/HCA experimental attention variants
(the components that are not shared with the [DSA module](/cudnn/operations/sparse_attention/dsa)). Its first operation
is the **fused Compressor**: one forward and one backward kernel for the `Compressor`
gated-softmax pooling region (THD packed layout) used by CSA/HCA in Megatron-LM.

The kernels were ported from Megatron-LM at the maintainers' request
([Megatron-LM PR #5984](https://github.com/NVIDIA/Megatron-LM/pull/5984); measurements
and numerics in
[Megatron-LM issue #5968](https://github.com/NVIDIA/Megatron-LM/issues/5968)). The eager
region they replace decomposes into ~39 forward and ~51 backward kernel launches per
call (at `compress_ratio = 4`, `coff = 2`) and materializes `(total_comp, 2*ratio, 1, head_dim)`
window intermediates. The default fused path is one forward and one backward kernel
plus one fp32 `dAPE`-buffer zero-fill. Deterministic dAPE replaces that fill with a
fixed-order reduction stage (two stages for ratio 4). Both backward modes write
`dKV`/`dScore` in full, including
exact zeros to never-consumed positions, so those buffers need no fills.

### Semantics

For each THD segment `s` (`cu_seqlens[s]..cu_seqlens[s+1]`) and each output block `b` of
`ratio` tokens, with the overlapping window form (`coff == 2`, window size `2 * ratio`):

- `k in [0, ratio)`: previous block's token, first-half projection column, APE row `k`
  — invalid for the segment's first block (score `-inf`, kv `0`);
- `k in [ratio, 2*ratio)`: own block's token, second-half projection column, APE row
  `k - ratio`.

The own-block window form (`coff == 1`, window size `ratio`) drops the overlap: every
`k in [0, ratio)` is the block's own token on projection column `j` with APE row `k`, and
every window is fully valid (no first-block exception).

```text
out[b, j] = sum_k kv[w(b,k), c(k,j)] * softmax_k(score[w(b,k), c(k,j)] + ape[k % ratio, c(k,j)])
```

computed in fp32 with a single final bf16 rounding. Per-segment tail tokens
(`seqlen % ratio`) are dropped, as in the eager code. Output rows beyond
`cu_seqlens_comp[-1]` (a static CUDA-graph capacity) are exact zeros on SM90 and
SM100+; the forward skips their APE/KV/score loads and softmax, and the backward
ignores incoming gradients on those rows. Consequently a positive static capacity is
valid even when every segment is shorter than `ratio`.

### Numerics

All arithmetic is fp32 with one final bf16 rounding; `mul.rn.f32` / `fma.rn.f32` are
pinned in PTX so results do not depend on compiler FMA contraction. The numerics
contract is **per ratio family** (the two families intentionally differ — do not
assume the ratio \{2, 4\} guarantees at ratio=128):

- **`ratio in {2, 4}`** (generic kernels; the bitwise contract — ratio=4 as originally
  validated, ratio=2 added on the same terms): against an
  fp32-intermediate eager reference (same op order, fp32 throughout), `dKV`/`dScore`
  are **bit-identical** and the forward matches within one bf16 rounding step on a
  tiny fraction of elements.
- **`ratio == 128`** (deterministic tolerance contract): the kernels are
  **deterministic and faithful to the fp32-intermediate eager reference** (the
  same eager region computed with fp32 intermediates and one final bf16 rounding —
  the comparator every number below is measured against):

  1. **Determinism** — forward `out` and backward `dKV`/`dScore` are bitwise
     run-to-run deterministic on all supported inputs (see below; NaN-prefill
     replay tested, including bit-stable replay of the NaN pattern on the gate's
     overflow-intermediate case),
  2. **Same values within final-bf16 rounding on the gate tolerances** —
     `out`/`dKV`/`dScore` match the eager reference within differing elements
     `<= max(1, 0.1%)` of the tensor and `max_abs <= 1.6e-2`, thresholds
     **calibrated on the gate's documented input distribution** (`kv`, `grad_out`
     ~ N(0, 1) bf16, `score` ~ N(0, 1.5²) bf16, `ape` ~ N(0, 0.25²) fp32).
     Absolute bf16 deviations are NOT magnitude-free — bf16's grid is relative, so
     scaling `kv`/`grad_out` by `2^k` scales every deviation exactly `2^k`, with
     differing-element counts and fp64 parity unchanged for as long as the scaled
     inputs keep every fp32 intermediate finite (see 3.). The gate's ×64
     scaled-input case commits that as evidence (deviations ×64 forward/`dKV`,
     ×4096 `dScore`; counts and parity identical to its unit-scale twin), and
  3. **The eager reference's non-finite propagation** — both sides compute the
     window math in fp32, so finite inputs that overflow the eager reference's
     fp32 intermediates (a `score + ape` sum or a backward `kv · grad_out` product
     beyond fp32 range — elementwise ops both sides evaluate identically) poison
     the kernels' outputs as they poison the reference's, instead of the kernels
     returning clean-looking finite values. The gate's overflow-intermediate case
     commits this: `score` = bf16 max with `ape` = fp32 max drives fp32
     `score + ape` to +Inf, every fused output is NaN with the NaN mask equal to
     the eager reference's on all four outputs, and the pattern replays
     bit-stable; the fp64 comparator is explicitly skipped there because the fp64
     oracle stays finite (a NaN-vs-finite distance gates nothing). The mirror
     caveat: the fused evaluation order saturates EARLIER than the eager one near
     fp32 max — its forward chunk partials are un-normalized (up to `2 · ratio ×`
     the reference's normalized weighted sums), so e.g. uniform-score inputs with
     `|kv|` within a factor `~2 · ratio` of fp32 max return ±Inf where the eager
     reference is still finite. Non-finite inputs follow the same rule wherever
     they fall in the forward's window chunks: a NaN or +Inf `score + ape` at one
     window position makes that output NaN, and a -Inf position (a masked one)
     gets weight 0 (`test_r128_forward_special_values_match_eager`).

  On inputs whose fp32 intermediates stay finite in BOTH evaluation orders — the
  eager reference's and the fused kernels' (which saturate earlier, see 3.) —
  `out`/`dKV`/`dScore` additionally carry **fp64-oracle parity**: per case and per
  tensor they are **at least as close to an fp64 oracle as the fp32-intermediate
  eager reference itself** (`err_fused <= err_eager * (1 + 1e-6) + 1e-4`, asserted
  in the tests and on every finite-intermediate case of the committed gate
  script): the fused reorders and fast-exp buckets do not lose accuracy against
  the eager reference on the tested envelope.

  Worst observed on the gate distribution (19 unit-scale cases = 15 union — the
  128k-token d=512 packs select the large-bucket schedules — + 4 padding, B200;
  reproduce with `benchmark/csa/gate_csa_compressor_r128.py`, which also asserts
  the case list selects every shipped schedule): forward 0.0275% differing elements
  (9/32,768), `dKV` 0.0041% (2,770/67,108,864), `dScore` 0.0113% (104/921,856);
  worst `max_abs` 3.91e-3 for forward/`dKV` (the 131k-token d=512 cases) and
  1.95e-3 for `dScore` (bf16-rounding magnitudes). The deviations come from
  measured-win reduction reorders (forward chunked-softmax merges; the backward's
  fused `den`/`S` partial merge with a hoisted `1/den`) and an `ex2.approx`-based
  fast exp enabled on specific schedule buckets in BOTH directions (forward:
  per-bucket table entries; backward: the default outside the d=128 vec=1
  small-pack buckets). `dAPE` (fp32, atomic accumulation) is gated at
  `max_abs <= 1e-3` vs the fp32 eager reference on the gate distribution — observed
  worst 2.0e-6, run-to-run replay deltas of the same magnitude — and scales the
  same way (observed 2.0e-3 under the ×64 case). `dAPE`'s distance from the fp64
  oracle fluctuates run to run around the eager reference's own (within ~1.5×
  either way — atomic reduction-order noise), so its parity check is asserted on
  the unit-scale cases, where the formula's absolute term dominates, and recorded
  on scaled inputs.

In BOTH families the forward, `dKV` and `dScore` are **bitwise run-to-run
deterministic** (fixed chunk boundaries and merge orders, no atomics in those
outputs; NaN-prefill replay is part of the test suite). The backward has two dAPE
policies:

- `deterministic=False` (default) keeps one fp32 atomic per `(k, dim)` per CTA.
  This fastest path is not bitwise deterministic and raises under strict
  `torch.use_deterministic_algorithms(True)` (warn-only mode warns and runs).
  Its class plan requires `workspace=None`; workspace never switches a compiled
  atomic plan to deterministic execution.
- `deterministic=True` writes each CTA's register-local dAPE sums to caller-owned
  fp32 partial planes and launches fixed-order reduction stages. Ratio 4 first reduces
  independent groups of 32 CTA planes: eight warps each sum four contiguous planes,
  then warp 0 merges those eight sums. A second kernel folds the resulting group sums
  with an eight-way contiguous tree. This hierarchical grid supplies thousands of CTAs
  on long packed sequences instead of assigning the whole partial axis to 16 CTAs.
  The ratio-4 producer accumulates 8 rows per CTA below 65,536 compressed rows and
  24 rows per CTA at or above that boundary. The long-context schedule cuts producer
  partial storage by about two thirds while retaining enough CTA parallelism; the
  selected row count is part of the compiled/fast-launch cache key.
  Ratio 2 and ratio 128 use a serial ascending-CTA fold. Both forms contain no dAPE
  atomics, fully overwrite `grad_ape`, are bitwise replay-deterministic, and are
  accepted by strict torch deterministic mode.
  The class API exposes `scratch_workspace_bytes()` and requires a contiguous uint8
  `workspace` of at least that size at `execute`; the high-level wrapper allocates it.
  Both launches use the caller stream and are CUDA-graph capture/replay safe after
  `compile()`.

On SM90, production `coff=2`, ratio-2 backward adds one measured 16-row producer
bucket while keeping 8 rows for smaller capacities. Atomic plans switch at 8,192
compressed rows for `head_dim=128`. For `head_dim=512`, the atomic 16-row bucket
is limited to the measured `[4,096, 5,121)` safe interval; larger capacities
retain 8 rows because additional grid-wave occupancy steps showed regressions.
Deterministic plans switch at 4,096 and 2,048
rows for `head_dim=128` and `512`, respectively, and keep 16 rows thereafter.
Other `coff`/head dimensions and architecture families retain the 8-row schedule.
The deterministic 16-row bucket halves the partial workspace. The selected row
count is bound by the prepared plan, so `execute()` does not query device
capability or specialize on device-side sequence metadata.

#### Deterministic dAPE performance

CUDA-event medians over 100 iterations after 30 warmups on one GB200
(capability `(10, 0)`, CUDA 12.9, PyTorch 2.9.1, CUTLASS DSL 4.7) are below. Atomic and
deterministic variants share all operand addresses, measurement order alternates AB/BA,
and the whole run is repeated with plan construction order reversed. `operation`
includes the atomic path's required dAPE zero-fill; deterministic dAPE overwrites its
output and therefore needs no fill. JIT and allocations are outside the measurement.

| configuration | workspace | atomic kernel | deterministic kernel | operation ratio, deterministic / atomic (both construction orders) |
|---|---:|---:|---:|---:|
| r4, c2, d128, 1x8192 | 1.031 MiB | 32.5–34.0 us | 39.8–42.9 us | 0.96–0.98x |
| r4, c2, d128, 3x8192 | 3.094 MiB | 40.6–41.1 us | 45.7–47.1 us | 0.94–0.95x |
| r4, c2, d512, 1x8192 | 4.125 MiB | 42.5–42.8 us | 50.8–51.2 us | 0.97–0.99x |
| r4, c2, d512, 3x8192 | 12.375 MiB | 90.7–91.5 us | 89.6–91.0 us | 0.92–0.93x |
| r128, c1, d128, 1x8192 | 4 MiB | 31.1–32.6 us | 38.2–41.8 us | 0.91–0.93x |
| r128, c2, d128, 3x8192 | 8 MiB | 47.5–48.0 us | 53.1–54.5 us | 0.93–0.94x |
| r128, c1, d512, 1x8192 | 16 MiB | 40.8–41.0 us | 42.2–44.0 us | 0.84–0.86x |
| r128, c2, d512, 3x8192 | 16 MiB | 111.7–112.1 us | 115.9–116.9 us | 0.95–0.96x |

For the production `coff=1`, `head_dim=128`, ratio-4 long-context path with local
pooling disabled, the compressor runs after the raw CP shards are gathered and therefore
still sees the global packed token count. CP32 and CP64 therefore present the same 256K
compressor shape. On one L20D (capability `(10, 3)`, CUDA 13.0, PyTorch 2.12.1,
CUTLASS DSL 4.8.1), the current unequal-document shapes measured as follows; each range
covers both plan-construction orders:

| configuration | workspace | atomic operation | deterministic operation | deterministic / atomic |
|---|---:|---:|---:|---:|
| r4, c1, d128, packed 256K, CP32/64 | 5.502 MiB | 116.6–118.0 us | 100.6–101.1 us | 0.85–0.86x |
| r4, c1, d128, packed 1M, CP64 | 22.002 MiB | 372.0–373.9 us | 308.5–309.2 us | 0.83x |

Compared with the 8-row deterministic producer on the same source and host, the
24-row schedule reduces workspace from 16.5 to 5.502 MiB at 256K and from 66 to
22.002 MiB at 1M. Deterministic operation time also falls from 105.0 to about 100.9 us
and from 328.5 to about 309.0 us, respectively. dAPE remains within the benchmark's
fp64 error gate (maximum absolute errors `1.80e-5` and `4.52e-5`).

Reproduce the long-context rows with
`benchmark/csa/bench_csa_dape_deterministic.py --long-context`. For the compact
matrix above, omit that option; use
`--construct-order deterministic-first` for the reversed build and `--atomic-only`
against an unchanged-source checkout for the independent control.

### Support surface (`check_support`)

| architecture | ratio | coff | head dimension | forward | backward | deterministic backward |
|---|---:|---:|---:|---|---|---|
| SM90 | 2 or 4 | 1 or 2 | generic launch bound below | yes | yes | yes |
| SM100+ | 2 or 4 | 1 or 2 | generic launch bound below | yes | yes | yes |
| SM100+ | 128 | 1 or 2 | 128 or 512 | yes | yes | yes |
| SM90 | 128 | any | any | no | no | no |

The primary SM90 training configuration is packed THD, `ratio=4`, `coff=1`,
`head_dim=128`, BF16 KV/score/output, FP32 APE and int32 prefix metadata. Both dAPE
policies use the same forward/backward math; `deterministic=True` selects the
caller-workspace fixed-order reduction.

- SM90 and compute-capability major **>= 10** for the generic ratio `{2, 4}`
  kernels. Ratio 128 remains SM100-or-newer only; its schedule and tolerance
  contract have not yet been validated on SM90.
- `ratio in {2, 4}`, `coff in {1, 2}` (`coff == 2` is the production CSA/HCA configuration,
  `coff == 1` the own-block window form; `ratio == 2` is the configuration used in
  production training) — served by the generic kernels, which are
  generic over `(ratio, head_dim, coff in {1, 2})` but keep the whole pooling window in
  registers (register-bound beyond `ratio = 32`)
- `ratio == 128`, `coff in {1, 2}`, `head_dim in {128, 512}` — served by dedicated
  kernels (see [ratio=128](#ratio128) below); the wrappers and class APIs
  route by `ratio` transparently, and the gate can be widened per configuration once
  validated. **The ratio=128 numerics contract differs** (see [Numerics](#numerics))
- BF16 `kv` / `score` / `out`, FP32 `ape`, int32 `cu_seqlens` / `cu_seqlens_comp`
- int32 flat offsets: `total_tokens * coff * head_dim < 2**31` and
  `total_comp * head_dim < 2**31`
- Capacity-only rows do not require a token window. Device prefix metadata still owns
  the logical contract: every valid compressed row must describe a complete `ratio`
  token block.
- `head_dim <= 8388480` (forward launch `gridDim.y` bound)
- contiguous tensors on one CUDA device, with 16-byte-aligned base pointers (4-byte
  for the int32 cu_seqlens) — contiguity does not imply base alignment for
  storage-offset views, so this is checked per call

## Installation

```bash
pip install nvidia-cudnn-frontend
```

For a source checkout in a CUDA 13 environment, keep the CUDA extra when installing or
pinning CuTe DSL:

```bash
python -m pip install 'nvidia-cutlass-dsl[cu13]>=4.8.0'
python -m pip install -e .
```

## API Usage

### JAX ops (SM100 ratio-4 CSA / ratio-128 HCA)

```python
from cudnn.ops import dsa

out = dsa.compressor_forward_jax(
    kv, score, ape, cu_seqlens, cu_seqlens_comp, total_comp=static_output_rows
)
# DSv4 HCA: non-overlapping ratio-128 pooling, kv/score width == head_dim.
out = dsa.compressor_forward_jax(
    kv, score, ape, cu_seqlens, cu_seqlens_comp, total_comp=static_output_rows, ratio=128, coff=1
)
grad_kv, grad_score, grad_ape = dsa.compressor_backward_jax(
    kv, score, ape, cu_seqlens, cu_seqlens_comp, grad_out, ratio=128, coff=1
)
```

`cudnn.ops.dsa.compressor_forward_jax` and `cudnn.ops.dsa.compressor_backward_jax`
accept JAX arrays, eagerly or under `jax.jit`, and require no PyTorch. They support
`ratio=4, coff=2` (CSA) and `ratio=128, coff in {1, 2}` with `head_dim in {128, 512}`
(HCA), BF16 KV/scores, FP32 APE and int32 packed-sequence metadata, on SM100-family
GPUs. Forward and backward are separate calls; the caller owns the VJP. The
PyTorch counterparts are `cudnn.ops.dsa.compressor_forward` and
`cudnn.ops.dsa.compressor_backward`, the same functions as
`cudnn.csa.csa_compressor_forward_wrapper` and `csa_compressor_backward_wrapper`.

### High-level wrappers

```python
from cudnn import CSA

# forward: (total_tokens, coff*head_dim) BF16 kv/score, (ratio, coff*head_dim) FP32 ape,
# (B+1,) int32 cu_seqlens / cu_seqlens_comp
result = CSA.csa_compressor_forward_wrapper(
    kv, score, ape, cu_seqlens, cu_seqlens_comp,
    ratio=4, head_dim=128, coff=2,
    total_comp=None,  # defaults to cu_seqlens_comp[-1] (synchronizes); pass a static
                      # capacity explicitly to stay CUDA-graph capture-safe
    stream=None,
)
out = result["out"]  # (total_comp, head_dim) BF16

grads = CSA.csa_compressor_backward_wrapper(
    kv, score, ape, cu_seqlens, cu_seqlens_comp, grad_out,
    ratio=4, head_dim=128, coff=2, stream=None,
)
grad_kv, grad_score, grad_ape = grads  # BF16, BF16, FP32
```

Set `coff=1` for the `ratio`-token own-block window (and use `kv` / `score` tensors whose
packed width is `head_dim`); `coff=2` selects the `2 * ratio` overlapping window shown
above.

The wrappers keep a bounded LRU of prepared API instances. The generic forward
artifact is shared per `(ratio, head_dim, coff, device)`; generic backward adds
`deterministic` and the bounded `rows_per_cta` schedule bucket. In particular,
the generic row-count domain is the finite set `{8, 16, 24}`: SM90 ratio-2 uses
the documented capacity bucket, and ratio-4 deterministic backward has only its
documented short/long capacity buckets. The SM90 and SM100-family artifacts
remain device-scoped and are lowered for the active device; packed token totals
and per-sequence length distributions within a schedule bucket are runtime
data, not compile specializations.

### Class API

```python
from cudnn import CSACompressorForward, CSACompressorBackward

op = CSACompressorForward(
    sample_kv, sample_score, sample_ape, sample_cu_seqlens, sample_cu_seqlens_comp,
    sample_out, ratio=4, coff=2,
)
op.check_support()
op.compile()
op.execute(kv, score, ape, cu_seqlens, cu_seqlens_comp, out, current_stream=None)
```

`CSACompressorBackward.execute` additionally takes `grad_out` and the
`grad_kv` / `grad_score` / `grad_ape` buffers. `grad_kv` / `grad_score` may be
**uninitialized**: the kernel writes every position (disjoint, atomic-free stores;
never-consumed positions — segment tails, the last block's first-half columns
(`coff == 2` only; `coff == 1` has no first-half columns), segments shorter than `ratio`,
token-capacity padding beyond `cu_seqlens[-1]` — get exact zeros
from their unique owning CTA, matching autograd). When `total_comp == 0` the kernel is
not launched and the buffers are left untouched (zero them yourself if you need
autograd's exact zeros; the high-level wrapper does). With the default
`deterministic=False`, `grad_ape` must be **zero-initialized before every `execute`
call and before every CUDA-graph replay that reuses the buffer**: the kernel only
accumulates into it with fp32 atomics. The high-level wrapper supplies this zero-fill,
including during graph replay. With
`CSACompressorBackward(..., deterministic=True)`, call
`scratch_workspace_bytes()` after `check_support()`, then provide
`execute(..., workspace=...)` a same-device, contiguous CUDA uint8 buffer that is
16-byte aligned, does not overlap any input or gradient tensor, and has at least the
reported capacity. The deterministic path fully overwrites `grad_ape` without a
zero-fill.

### CUDA graphs

The launch path is capture-compatible once the kernels for a `(ratio, head_dim, coff)`
configuration are compiled: run one warmup call (or `compile()`) per configuration
before capturing, and pass `total_comp` explicitly. A call that would JIT under capture
raises a `RuntimeError` instead of corrupting the capture.

For deterministic backward capture, construct and compile
`CSACompressorBackward(..., deterministic=True)`, query
`scratch_workspace_bytes()`, and allocate the returned workspace before capture. The
captured `execute()` performs no JIT, allocation, host synchronization or device-prefix
readback. Prefix contents may change at the captured device addresses on replay;
tensor addresses, prepared capacities and launch topology remain fixed.

### Environment variables

- `CUDNNFE_CSA_COMPRESSOR_FAST_LAUNCH=0` — disable the cached-launch host optimization
  (a per-config snapshot of the CuTe-DSL launch state, replayed with in-place argument
  mutation; it removes tens of microseconds of per-call host overhead for these
  microsecond-scale kernels). The snapshot construction introspects
  private-but-stable DSL internals; on any structural mismatch (e.g. a future
  `nvidia-cutlass-dsl` upgrade) it falls back to the regular launch path automatically.

## Performance

Reproduce the primary SM90 configuration with:

```bash
python benchmark/csa/bench_csa_compressor.py --ratio 4 --coff 1 --head-dim 128
python benchmark/csa/bench_csa_dape_deterministic.py --sm90-primary
python benchmark/csa/bench_csa_dape_deterministic.py --sm90-primary --construct-order deterministic-first
```

The first command reports eager/fused forward, backward and CUDA Graph replay timing;
the latter two compare atomic and deterministic dAPE with JIT and allocation excluded.

Measured on one SM90 GPU (CC 9.0, driver 570.148.08, CUDA 13.0, cuDNN 9.20, PyTorch
2.12.1, CUTLASS DSL 4.8.1); BF16 `kv`/`score`, FP32 `ape`; `ratio = 4`,
`coff = 1`, `head_dim = 128`; median of 50 iterations after 20 warmup:

| THD pack | eager fwd | fused fwd | fwd | eager bwd | fused bwd | bwd | eager graph total | fused graph total | graph total |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 1 x 8192 | 205.7 us | 40.9 us | **5.0x** | 270.0 us | 54.4 us | **5.0x** | 202.2 us | 17.9 us | **11.3x** |
| 3 x 8192 | 222.9 us | 39.8 us | **5.6x** | 261.6 us | 57.2 us | **4.6x** | 306.4 us | 28.4 us | **10.8x** |

The opt-in deterministic backward used 0.52 MiB and 1.55 MiB of caller workspace for
the two packs. Its operation time was 35.87 us and 35.49 us, respectively, versus
29.84 us and 35.04 us for atomic dAPE (1.20x and 1.01x). The deterministic dAPE maximum
absolute error versus the fp64 oracle was `3.41e-6` and `5.46e-6`.

Measured on 1x B200 (CC 10.0, driver 590.48.01); BF16 `kv`/`score`, FP32 `ape`; `ratio = 4`, `coff = 2`;
THD packs of 8192-token sequences; eager baseline = the exact replaced region of
Megatron-LM `Compressor._forward_thd` on identical inputs.

*Isolated GPU kernel time* (nsys, sum of kernel durations per iteration, 50 iterations
after 20 warmup; no launch/host overhead; backward includes its `dAPE` zero-fill):

| THD pack | head_dim | eager fwd | fused fwd | fwd | eager bwd | fused bwd | bwd |
|---|---|---|---|---|---|---|---|
| 1 x 8192 | 128 | 117.8 us | 4.5 us | **26.5x** | 187.2 us | 12.8 us | **14.6x** |
| 3 x 8192 | 128 | 229.8 us | 10.0 us | **23.0x** | 352.7 us | 22.2 us | **15.9x** |
| 1 x 8192 | 512 | 263.3 us | 12.4 us | **21.2x** | 425.0 us | 22.8 us | **18.6x** |
| 3 x 8192 | 512 | 664.3 us | 35.0 us | **19.0x** | 1155.8 us | 66.0 us | **17.5x** |

*End-to-end wall clock of the same region* (CUDA events, median of 100 after 30 warmup;
includes launch overhead; eager backward goes through torch autograd with the forward
outside the timed region, fused backward is the explicit backward wrapper call — not
comparable to the kernel-time numbers above):

| THD pack | head_dim | eager fwd | fused fwd | fwd | eager bwd | fused bwd | bwd |
|---|---|---|---|---|---|---|---|
| 1 x 8192 | 128 | 343.7 us | 37.4 us | **9.2x** | 558.7 us | 51.9 us | **10.8x** |
| 3 x 8192 | 128 | 389.3 us | 38.2 us | **10.2x** | 585.8 us | 62.4 us | **9.4x** |
| 1 x 8192 | 512 | 423.3 us | 39.9 us | **10.6x** | 666.5 us | 62.8 us | **10.6x** |
| 3 x 8192 | 512 | 831.7 us | 62.7 us | **13.3x** | 1503.6 us | 108.3 us | **13.9x** |

The previously published per-call wall clock (commit `b3ceb7c`) was `333.7 / 37.8 / 506.7
/ 47.2 us` at `1 x 8192 / 128` (and analogously for the other packs). The re-run above
reproduces the forward columns within ~±4% and the fused/backward columns within +2-14%:
the fused per-call wall clock carries the cached-launcher host optimization, whose
snapshot replay jitters run-to-run (the published fused-forward is even non-monotonic in
pack size, `37.8 -> 35.1 us`), and eager backward drifts with GPU boost-clock state. The
CUDA-graph replay numbers below collapse that host jitter into a single replay.

*CUDA-graph replay of the same region, both implementations captured symmetrically as a
forward-only graph and a forward+backward graph* (median of 100 after 30 warmup, replay
timing via `benchmark/csa/bench_csa_compressor.py`; capturing each side into a graph
collapses its per-operation launches into a single replay, so this is the fairest
wall-clock basis for launch-bound shapes):

| THD pack | head_dim | eager fwd | fused fwd | fwd | eager total | fused total | total |
|---|---|---|---|---|---|---|---|
| 1 x 8192 | 128 | 119.2 us | 10.8 us | **11.0x** | 364.6 us | 22.8 us | **15.9x** |
| 3 x 8192 | 128 | 235.3 us | 14.8 us | **15.8x** | 668.4 us | 38.6 us | **17.3x** |
| 1 x 8192 | 512 | 269.8 us | 17.1 us | **15.7x** | 802.5 us | 40.0 us | **20.0x** |
| 3 x 8192 | 512 | 673.1 us | 39.2 us | **17.1x** | 2057.6 us | 107.8 us | **19.0x** |

`eager total` / `fused total` capture forward + backward together. The eager total graph
captures the autograd backward of the captured forward against stable, pre-allocated zero
`.grad` buffers (the captured region zeros them in place, then runs forward + backward, so
every replay accumulates into a zeroed buffer — numerically identical to a single fresh
backward, verified per shape in the harness); the fused total graph captures the forward
wrapper immediately followed by the backward wrapper. The backward replay alone is not a
separately captured quantity — it is approximately `total - fwd` and is reported only as
that reference, never as a measured column. Graph speedups are `eager / fused` of the
displayed µs, truncated to one decimal.

Capturing each side into a graph narrows the absolute eager-vs-fused forward gap most on
the smallest, launch-bound shape (about 2.8x at `1 x 8192 / 128` forward) and barely on
the largest, less launch-bound shape (1.2x at `3 x 8192 / 512`); the speedup ratio is
similar to or larger than per-call because a graph replay collapses each side's
per-operation launches into one.

Environment: driver 590.48.01, PyTorch 2.13.0 (CUDA 13.3), `nvidia-cutlass-dsl` 4.6.1.
Measurement basis: identical inputs over exactly the replaced region for both
implementations; per-call eager backward = the torch autograd backward with the forward
outside the timed region; graph eager backward = the autograd backward of the captured
forward, captured together with it against stable zero `.grad` buffers; fused backward =
the backward wrapper (kernel + the fp32 `dAPE` zero-fill + host validation, no autograd
engine). Both wall-clock tables above (per-call and graph) are from a single run of
`benchmark/csa/bench_csa_compressor.py`; the kernel-time table is from nsys.

An ncu hardware-ceiling audit of the **ported kernels (prior to the two optimization
commits)** — cache-flushed, `--set full`, all four benchmark shapes — showed measured
DRAM read volume matching the algorithmically necessary bytes within 1% (the THD gather
adds no over-read; stores fully coalesced at 32/32 bytes per sector, loads 29-30/32),
with neither L2 (&lt;27% of peak) nor DRAM (&lt;33%) close to saturation at these
microsecond-scale sizes: the gap to a pure DRAM-floor time was a mix of sub-wave grid
width / occupancy, memory latency, and (at the largest shape) issue pressure — not
wasted traffic. The two optimizations that audit identified are folded in here: 32-bit
vectorized forward accesses and backward kernel-side zero-writes replacing the two bf16
grad-buffer fills. They do not change the bytes the kernels must read, so the
traffic-optimality conclusion carries over; the utilization percentages above predate
them.

## ratio=128

`ratio = 128` is served by dedicated kernels (`compressor_sm100_r128.py`): the generic
kernels keep the whole `coff * ratio` window in per-thread registers, which spills
kilobytes of local memory per thread at `ratio = 128` (255-register cap). The dedicated
forward streams the window with a chunked softmax (one CTA per output row, the window
split over `threadIdx.y` chunk-rows, per-column `(max, denom, acc)` triples, one
fixed-order smem merge); its launch schedule is **bucketed by the output-row count**
(small / default / large, all precompiled for CUDA-graph safety) and per bucket selects
the online-rescale or two-phase accumulation form and, where it measured faster, an
`ex2.approx`-based fast exp. The dedicated backward stages each row's window into
shared memory chunk-parallel, computes `exp` and per-chunk partial `den`/`S` sums in
the same pass, merges the partials per column in a fixed chunk order, and stores the
gradients with a hoisted `1/den` multiply — plus kernel-side zero-writes to
never-consumed slots and, by default, fp32-atomic `dAPE`, exactly as at ratio=4. The
opt-in deterministic policy shares the generic family's CTA-partial plus fixed-order
reduction design. The backward defaults to the same fast exp outside the d=128
small-pack (vec=1) buckets.

**The ratio=128 numerics contract is the deterministic tolerance contract described in
[Numerics](#numerics), NOT the generic-family (ratio \{2, 4\}) bitwise-`dKV`/`dScore` contract.** The
reduction orders (forward chunk merge, backward `den`/`S` partial merge), the
backward's hoisted reciprocal, and the fast-exp buckets all differ from the eager op
order by design, each adopted on a measured same-GPU win and gated on tolerance +
fp64-oracle parity (worst observed deviations in [Numerics](#numerics)). Everything
stays bitwise run-to-run: fixed chunk boundaries and merge orders, no atomics in
forward/`dKV`/`dScore`.

Both kernels are register-flat in the window length (ptxas sm_100a: forward 32-51
registers, backward 64-128 registers, 0 spill / 0 stack across every shipped
(config, schedule) kernel — 16 kernels total; reproduce the per-kernel table with
`benchmark/csa/reg_probe_csa_compressor_r128.py`) and JIT in ~0.4-1.0 s per
configuration.
The backward picks its `rows_per_cta` at launch time (a runtime argument — no
recompile) so the grid fits one resident wave; the static row capacity fixes it under
CUDA-graph capture. Small packs (`nb_total <= 192`, d=128) switch the backward to a
vec=1 schedule bucket (also precompiled) for grid-fill. Every backward bucket except
the c1d128 default additionally loads `grad_out` once per row and keeps the register
vec live across the phase barriers (the `goreuse` schedule field): adopted per bucket
on measured pure-kernel wins — up to ~26% on the vec=1 small buckets, whose small
grids underfill the machine, so the +16/+1 register cost is not the binding
constraint — while the c1d128 default measured ~1.3% slower with it at 131k tokens
and keeps the phase-4 reload. `dKV`/`dScore` are bitwise-identical with and without
the field (the same read-only values move into registers).

Measured on 1x B200 (CC 10.0, driver 590.48.01), torch 2.13.0 / CUDA 13.3 /
`nvidia-cutlass-dsl` 4.6.1; eager baseline = the fp32-intermediate reference region of
the test suite on identical inputs (the numerics-contract reference; the upstream eager
region differs only by a bf16 weight-rounding cast at ~equal cost). Packs are
single-sequence THD unless noted.

*Isolated GPU kernel time* (nsys, sum of kernel durations inside a
`cudaProfilerApi`-gated 100-iteration unsynced loop / 100; forward and backward each
measured in their own loop; the fwd+bwd total is one combined loop per training step —
forward + backward + the ~1 us fp32 `dAPE` zero-fill — and is not the sum of the
standalone columns, because the interleaved backward changes L2 residency for the
forward at long context):

| config | tokens | eager fwd | fused fwd | fwd | eager bwd | fused bwd | bwd | fused fwd+bwd total |
|---|---|---|---|---|---|---|---|---|
| coff 1, d 128 | 8192 | 84.7 us | 5.7 us | **15.0x** | 127.3 us | 6.3 us | **20.2x** | 13.0 us |
| coff 1, d 128 | 131072 | 451.8 us | 14.9 us | **30.2x** | 471.9 us | 63.7 us | **7.4x** | 82.5 us |
| coff 2, d 128 | 8192 | 187.1 us | 6.4 us | **29.1x** | 246.2 us | 10.5 us | **23.4x** | 18.1 us |
| coff 2, d 128 | 65536 | 726.3 us | 16.4 us | **44.4x** | 960.0 us | 61.2 us | **15.7x** | 76.7 us |
| coff 1, d 512 | 65536 | 641.1 us | 48.4 us | **13.2x** | 795.0 us | 104.2 us | **7.6x** | 150.0 us |
| coff 2, d 512 | 65536 | 2315.2 us | 78.8 us | **29.4x** | 3261.9 us | 205.8 us | **15.9x** | 287.7 us |
| coff 2, d 512 | 131072 | 4526.5 us | 117.6 us | **38.5x** | 6304.3 us | 457.6 us | **13.8x** | 580.7 us |

*End-to-end wall clock per call* (CUDA events, median of 100 after 30 warmup; fused
backward includes a `grad_ape.zero_()` per call; not comparable to the kernel-time
numbers above — at small packs the ~9 us python launch path dominates the 5.7-11 us
kernels):

| config | tokens | fused fwd | fused bwd |
|---|---|---|---|
| coff 1, d 128 | 8192 | 14.1 us | 20.8 us |
| coff 1, d 128 | 131072 | 24.4 us | 82.1 us |
| coff 2, d 128 | 8192 | 15.2 us | 25.3 us |
| coff 2, d 128 | 65536 | 25.5 us | 75.6 us |
| coff 1, d 512 | 65536 | 56.3 us | 118.9 us |
| coff 2, d 512 | 65536 | 88.1 us | 220.8 us |

The ratio=128 numerics contract (21-case gate: 15 union + 1 scaled-input + 4 padding
cases, each running the determinism, tolerance and fp64-parity gates, plus 1
overflow-intermediate case that gates bit-stable NaN replay and NaN-pattern equality
vs the eager fp32 reference with the fp64 comparator explicitly skipped, with shipped
schedule coverage asserted) and the per-kernel ptxas table are reproducible from the
committed scripts `benchmark/csa/gate_csa_compressor_r128.py` and
`benchmark/csa/reg_probe_csa_compressor_r128.py`.

## Testing

```bash
(cd test/python && pytest compressed_sparse_attention/cutedsl/test_CSA_compressor.py)
```

The tests validate numerics against an fp32-intermediate eager reference (bitwise
`dKV`/`dScore` at ratio \{2, 4\}; the deterministic tolerance contract at ratio=128, with
fp64-oracle parity on finite-intermediate inputs), the upstream eager numerics, plus ragged packs, static-capacity
padding, kernel-side zero-writes into uninitialized gradient buffers (NaN-canary with
exact-zero assertions on every never-consumed slot class, and the `total_comp == 0`
zeros fallback), run-to-run determinism for both dAPE policies, deterministic
workspace validation and CUDA-graph overwrite semantics, atomic-path `grad_ape`
zeroing ownership, the ratio=128
dispatch envelope (schedule selection at every `nb_total` bucket boundary at L0, and
one execution of every shipped (config, schedule) kernel against the full contract —
tolerance, fp64-oracle parity, determinism — at L1; run
`pytest -m "L0 or L1"`), CUDA-graph capture/replay, and `check_support` boundaries.