CSA Fused Compressor
This is an experimental API and subject to change.
Overview
The CSA module hosts CuTe-DSL kernels for the CSA/HCA experimental attention variants
(the components that are not shared with the DSA module). Its first operation
is the fused Compressor: one forward and one backward kernel for the Compressor
gated-softmax pooling region (THD packed layout) used by CSA/HCA in Megatron-LM.
The kernels were ported from Megatron-LM at the maintainers’ request
(Megatron-LM PR #5984; measurements
and numerics in
Megatron-LM issue #5968). The eager
region they replace decomposes into ~39 forward and ~51 backward kernel launches per
call (at compress_ratio = 4, coff = 2) and materializes (total_comp, 2*ratio, 1, head_dim)
window intermediates. The default fused path is one forward and one backward kernel
plus one fp32 dAPE-buffer zero-fill. Deterministic dAPE replaces that fill with a
fixed-order reduction stage (two stages for ratio 4). Both backward modes write
dKV/dScore in full, including
exact zeros to never-consumed positions, so those buffers need no fills.
Semantics
For each THD segment s (cu_seqlens[s]..cu_seqlens[s+1]) and each output block b of
ratio tokens, with the overlapping window form (coff == 2, window size 2 * ratio):
k in [0, ratio): previous block’s token, first-half projection column, APE rowk— invalid for the segment’s first block (score-inf, kv0);k in [ratio, 2*ratio): own block’s token, second-half projection column, APE rowk - ratio.
The own-block window form (coff == 1, window size ratio) drops the overlap: every
k in [0, ratio) is the block’s own token on projection column j with APE row k, and
every window is fully valid (no first-block exception).
computed in fp32 with a single final bf16 rounding. Per-segment tail tokens
(seqlen % ratio) are dropped, as in the eager code. Output rows beyond
cu_seqlens_comp[-1] (a static CUDA-graph capacity) are exact zeros on SM90 and
SM100+; the forward skips their APE/KV/score loads and softmax, and the backward
ignores incoming gradients on those rows. Consequently a positive static capacity is
valid even when every segment is shorter than ratio.
Numerics
All arithmetic is fp32 with one final bf16 rounding; mul.rn.f32 / fma.rn.f32 are
pinned in PTX so results do not depend on compiler FMA contraction. The numerics
contract is per ratio family (the two families intentionally differ — do not
assume the ratio {2, 4} guarantees at ratio=128):
-
ratio in {2, 4}(generic kernels; the bitwise contract — ratio=4 as originally validated, ratio=2 added on the same terms): against an fp32-intermediate eager reference (same op order, fp32 throughout),dKV/dScoreare bit-identical and the forward matches within one bf16 rounding step on a tiny fraction of elements. -
ratio == 128(deterministic tolerance contract): the kernels are deterministic and faithful to the fp32-intermediate eager reference (the same eager region computed with fp32 intermediates and one final bf16 rounding — the comparator every number below is measured against):- Determinism — forward
outand backwarddKV/dScoreare bitwise run-to-run deterministic on all supported inputs (see below; NaN-prefill replay tested, including bit-stable replay of the NaN pattern on the gate’s overflow-intermediate case), - Same values within final-bf16 rounding on the gate tolerances —
out/dKV/dScorematch the eager reference within differing elements<= max(1, 0.1%)of the tensor andmax_abs <= 1.6e-2, thresholds calibrated on the gate’s documented input distribution (kv,grad_out~ N(0, 1) bf16,score~ N(0, 1.5²) bf16,ape~ N(0, 0.25²) fp32). Absolute bf16 deviations are NOT magnitude-free — bf16’s grid is relative, so scalingkv/grad_outby2^kscales every deviation exactly2^k, with differing-element counts and fp64 parity unchanged for as long as the scaled inputs keep every fp32 intermediate finite (see 3.). The gate’s ×64 scaled-input case commits that as evidence (deviations ×64 forward/dKV, ×4096dScore; counts and parity identical to its unit-scale twin), and - The eager reference’s non-finite propagation — both sides compute the
window math in fp32, so finite inputs that overflow the eager reference’s
fp32 intermediates (a
score + apesum or a backwardkv · grad_outproduct beyond fp32 range — elementwise ops both sides evaluate identically) poison the kernels’ outputs as they poison the reference’s, instead of the kernels returning clean-looking finite values. The gate’s overflow-intermediate case commits this:score= bf16 max withape= fp32 max drives fp32score + apeto +Inf, every fused output is NaN with the NaN mask equal to the eager reference’s on all four outputs, and the pattern replays bit-stable; the fp64 comparator is explicitly skipped there because the fp64 oracle stays finite (a NaN-vs-finite distance gates nothing). The mirror caveat: the fused evaluation order saturates EARLIER than the eager one near fp32 max — its forward chunk partials are un-normalized (up to2 · ratio ×the reference’s normalized weighted sums), so e.g. uniform-score inputs with|kv|within a factor~2 · ratioof fp32 max return ±Inf where the eager reference is still finite. Non-finite inputs follow the same rule wherever they fall in the forward’s window chunks: a NaN or +Infscore + apeat one window position makes that output NaN, and a -Inf position (a masked one) gets weight 0 (test_r128_forward_special_values_match_eager).
On inputs whose fp32 intermediates stay finite in BOTH evaluation orders — the eager reference’s and the fused kernels’ (which saturate earlier, see 3.) —
out/dKV/dScoreadditionally carry fp64-oracle parity: per case and per tensor they are at least as close to an fp64 oracle as the fp32-intermediate eager reference itself (err_fused <= err_eager * (1 + 1e-6) + 1e-4, asserted in the tests and on every finite-intermediate case of the committed gate script): the fused reorders and fast-exp buckets do not lose accuracy against the eager reference on the tested envelope.Worst observed on the gate distribution (19 unit-scale cases = 15 union — the 128k-token d=512 packs select the large-bucket schedules — + 4 padding, B200; reproduce with
benchmark/csa/gate_csa_compressor_r128.py, which also asserts the case list selects every shipped schedule): forward 0.0275% differing elements (9/32,768),dKV0.0041% (2,770/67,108,864),dScore0.0113% (104/921,856); worstmax_abs3.91e-3 for forward/dKV(the 131k-token d=512 cases) and 1.95e-3 fordScore(bf16-rounding magnitudes). The deviations come from measured-win reduction reorders (forward chunked-softmax merges; the backward’s fusedden/Spartial merge with a hoisted1/den) and anex2.approx-based fast exp enabled on specific schedule buckets in BOTH directions (forward: per-bucket table entries; backward: the default outside the d=128 vec=1 small-pack buckets).dAPE(fp32, atomic accumulation) is gated atmax_abs <= 1e-3vs the fp32 eager reference on the gate distribution — observed worst 2.0e-6, run-to-run replay deltas of the same magnitude — and scales the same way (observed 2.0e-3 under the ×64 case).dAPE’s distance from the fp64 oracle fluctuates run to run around the eager reference’s own (within ~1.5× either way — atomic reduction-order noise), so its parity check is asserted on the unit-scale cases, where the formula’s absolute term dominates, and recorded on scaled inputs. - Determinism — forward
In BOTH families the forward, dKV and dScore are bitwise run-to-run
deterministic (fixed chunk boundaries and merge orders, no atomics in those
outputs; NaN-prefill replay is part of the test suite). The backward has two dAPE
policies:
deterministic=False(default) keeps one fp32 atomic per(k, dim)per CTA. This fastest path is not bitwise deterministic and raises under stricttorch.use_deterministic_algorithms(True)(warn-only mode warns and runs). Its class plan requiresworkspace=None; workspace never switches a compiled atomic plan to deterministic execution.deterministic=Truewrites each CTA’s register-local dAPE sums to caller-owned fp32 partial planes and launches fixed-order reduction stages. Ratio 4 first reduces independent groups of 32 CTA planes: eight warps each sum four contiguous planes, then warp 0 merges those eight sums. A second kernel folds the resulting group sums with an eight-way contiguous tree. This hierarchical grid supplies thousands of CTAs on long packed sequences instead of assigning the whole partial axis to 16 CTAs. The ratio-4 producer accumulates 8 rows per CTA below 65,536 compressed rows and 24 rows per CTA at or above that boundary. The long-context schedule cuts producer partial storage by about two thirds while retaining enough CTA parallelism; the selected row count is part of the compiled/fast-launch cache key. Ratio 2 and ratio 128 use a serial ascending-CTA fold. Both forms contain no dAPE atomics, fully overwritegrad_ape, are bitwise replay-deterministic, and are accepted by strict torch deterministic mode. The class API exposesscratch_workspace_bytes()and requires a contiguous uint8workspaceof at least that size atexecute; the high-level wrapper allocates it. Both launches use the caller stream and are CUDA-graph capture/replay safe aftercompile().
On SM90, production coff=2, ratio-2 backward adds one measured 16-row producer
bucket while keeping 8 rows for smaller capacities. Atomic plans switch at 8,192
compressed rows for head_dim=128. For head_dim=512, the atomic 16-row bucket
is limited to the measured [4,096, 5,121) safe interval; larger capacities
retain 8 rows because additional grid-wave occupancy steps showed regressions.
Deterministic plans switch at 4,096 and 2,048
rows for head_dim=128 and 512, respectively, and keep 16 rows thereafter.
Other coff/head dimensions and architecture families retain the 8-row schedule.
The deterministic 16-row bucket halves the partial workspace. The selected row
count is bound by the prepared plan, so execute() does not query device
capability or specialize on device-side sequence metadata.
Deterministic dAPE performance
CUDA-event medians over 100 iterations after 30 warmups on one GB200
(capability (10, 0), CUDA 12.9, PyTorch 2.9.1, CUTLASS DSL 4.7) are below. Atomic and
deterministic variants share all operand addresses, measurement order alternates AB/BA,
and the whole run is repeated with plan construction order reversed. operation
includes the atomic path’s required dAPE zero-fill; deterministic dAPE overwrites its
output and therefore needs no fill. JIT and allocations are outside the measurement.
For the production coff=1, head_dim=128, ratio-4 long-context path with local
pooling disabled, the compressor runs after the raw CP shards are gathered and therefore
still sees the global packed token count. CP32 and CP64 therefore present the same 256K
compressor shape. On one L20D (capability (10, 3), CUDA 13.0, PyTorch 2.12.1,
CUTLASS DSL 4.8.1), the current unequal-document shapes measured as follows; each range
covers both plan-construction orders:
Compared with the 8-row deterministic producer on the same source and host, the
24-row schedule reduces workspace from 16.5 to 5.502 MiB at 256K and from 66 to
22.002 MiB at 1M. Deterministic operation time also falls from 105.0 to about 100.9 us
and from 328.5 to about 309.0 us, respectively. dAPE remains within the benchmark’s
fp64 error gate (maximum absolute errors 1.80e-5 and 4.52e-5).
Reproduce the long-context rows with
benchmark/csa/bench_csa_dape_deterministic.py --long-context. For the compact
matrix above, omit that option; use
--construct-order deterministic-first for the reversed build and --atomic-only
against an unchanged-source checkout for the independent control.
Support surface (check_support)
The primary SM90 training configuration is packed THD, ratio=4, coff=1,
head_dim=128, BF16 KV/score/output, FP32 APE and int32 prefix metadata. Both dAPE
policies use the same forward/backward math; deterministic=True selects the
caller-workspace fixed-order reduction.
- SM90 and compute-capability major >= 10 for the generic ratio
{2, 4}kernels. Ratio 128 remains SM100-or-newer only; its schedule and tolerance contract have not yet been validated on SM90. ratio in {2, 4},coff in {1, 2}(coff == 2is the production CSA/HCA configuration,coff == 1the own-block window form;ratio == 2is the configuration used in production training) — served by the generic kernels, which are generic over(ratio, head_dim, coff in {1, 2})but keep the whole pooling window in registers (register-bound beyondratio = 32)ratio == 128,coff in {1, 2},head_dim in {128, 512}— served by dedicated kernels (see ratio=128 below); the wrappers and class APIs route byratiotransparently, and the gate can be widened per configuration once validated. The ratio=128 numerics contract differs (see Numerics)- BF16
kv/score/out, FP32ape, int32cu_seqlens/cu_seqlens_comp - int32 flat offsets:
total_tokens * coff * head_dim < 2**31andtotal_comp * head_dim < 2**31 - Capacity-only rows do not require a token window. Device prefix metadata still owns
the logical contract: every valid compressed row must describe a complete
ratiotoken block. head_dim <= 8388480(forward launchgridDim.ybound)- contiguous tensors on one CUDA device, with 16-byte-aligned base pointers (4-byte for the int32 cu_seqlens) — contiguity does not imply base alignment for storage-offset views, so this is checked per call
Installation
For a source checkout in a CUDA 13 environment, keep the CUDA extra when installing or pinning CuTe DSL:
API Usage
JAX ops (SM100 ratio-4 CSA / ratio-128 HCA)
cudnn.ops.dsa.compressor_forward_jax and cudnn.ops.dsa.compressor_backward_jax
accept JAX arrays, eagerly or under jax.jit, and require no PyTorch. They support
ratio=4, coff=2 (CSA) and ratio=128, coff in {1, 2} with head_dim in {128, 512}
(HCA), BF16 KV/scores, FP32 APE and int32 packed-sequence metadata, on SM100-family
GPUs. Forward and backward are separate calls; the caller owns the VJP. The
PyTorch counterparts are cudnn.ops.dsa.compressor_forward and
cudnn.ops.dsa.compressor_backward, the same functions as
cudnn.csa.csa_compressor_forward_wrapper and csa_compressor_backward_wrapper.
High-level wrappers
Set coff=1 for the ratio-token own-block window (and use kv / score tensors whose
packed width is head_dim); coff=2 selects the 2 * ratio overlapping window shown
above.
The wrappers keep a bounded LRU of prepared API instances. The generic forward
artifact is shared per (ratio, head_dim, coff, device); generic backward adds
deterministic and the bounded rows_per_cta schedule bucket. In particular,
the generic row-count domain is the finite set {8, 16, 24}: SM90 ratio-2 uses
the documented capacity bucket, and ratio-4 deterministic backward has only its
documented short/long capacity buckets. The SM90 and SM100-family artifacts
remain device-scoped and are lowered for the active device; packed token totals
and per-sequence length distributions within a schedule bucket are runtime
data, not compile specializations.
Class API
CSACompressorBackward.execute additionally takes grad_out and the
grad_kv / grad_score / grad_ape buffers. grad_kv / grad_score may be
uninitialized: the kernel writes every position (disjoint, atomic-free stores;
never-consumed positions — segment tails, the last block’s first-half columns
(coff == 2 only; coff == 1 has no first-half columns), segments shorter than ratio,
token-capacity padding beyond cu_seqlens[-1] — get exact zeros
from their unique owning CTA, matching autograd). When total_comp == 0 the kernel is
not launched and the buffers are left untouched (zero them yourself if you need
autograd’s exact zeros; the high-level wrapper does). With the default
deterministic=False, grad_ape must be zero-initialized before every execute
call and before every CUDA-graph replay that reuses the buffer: the kernel only
accumulates into it with fp32 atomics. The high-level wrapper supplies this zero-fill,
including during graph replay. With
CSACompressorBackward(..., deterministic=True), call
scratch_workspace_bytes() after check_support(), then provide
execute(..., workspace=...) a same-device, contiguous CUDA uint8 buffer that is
16-byte aligned, does not overlap any input or gradient tensor, and has at least the
reported capacity. The deterministic path fully overwrites grad_ape without a
zero-fill.
CUDA graphs
The launch path is capture-compatible once the kernels for a (ratio, head_dim, coff)
configuration are compiled: run one warmup call (or compile()) per configuration
before capturing, and pass total_comp explicitly. A call that would JIT under capture
raises a RuntimeError instead of corrupting the capture.
For deterministic backward capture, construct and compile
CSACompressorBackward(..., deterministic=True), query
scratch_workspace_bytes(), and allocate the returned workspace before capture. The
captured execute() performs no JIT, allocation, host synchronization or device-prefix
readback. Prefix contents may change at the captured device addresses on replay;
tensor addresses, prepared capacities and launch topology remain fixed.
Environment variables
CUDNNFE_CSA_COMPRESSOR_FAST_LAUNCH=0— disable the cached-launch host optimization (a per-config snapshot of the CuTe-DSL launch state, replayed with in-place argument mutation; it removes tens of microseconds of per-call host overhead for these microsecond-scale kernels). The snapshot construction introspects private-but-stable DSL internals; on any structural mismatch (e.g. a futurenvidia-cutlass-dslupgrade) it falls back to the regular launch path automatically.
Performance
Reproduce the primary SM90 configuration with:
The first command reports eager/fused forward, backward and CUDA Graph replay timing; the latter two compare atomic and deterministic dAPE with JIT and allocation excluded.
Measured on one SM90 GPU (CC 9.0, driver 570.148.08, CUDA 13.0, cuDNN 9.20, PyTorch
2.12.1, CUTLASS DSL 4.8.1); BF16 kv/score, FP32 ape; ratio = 4,
coff = 1, head_dim = 128; median of 50 iterations after 20 warmup:
The opt-in deterministic backward used 0.52 MiB and 1.55 MiB of caller workspace for
the two packs. Its operation time was 35.87 us and 35.49 us, respectively, versus
29.84 us and 35.04 us for atomic dAPE (1.20x and 1.01x). The deterministic dAPE maximum
absolute error versus the fp64 oracle was 3.41e-6 and 5.46e-6.
Measured on 1x B200 (CC 10.0, driver 590.48.01); BF16 kv/score, FP32 ape; ratio = 4, coff = 2;
THD packs of 8192-token sequences; eager baseline = the exact replaced region of
Megatron-LM Compressor._forward_thd on identical inputs.
Isolated GPU kernel time (nsys, sum of kernel durations per iteration, 50 iterations
after 20 warmup; no launch/host overhead; backward includes its dAPE zero-fill):
End-to-end wall clock of the same region (CUDA events, median of 100 after 30 warmup; includes launch overhead; eager backward goes through torch autograd with the forward outside the timed region, fused backward is the explicit backward wrapper call — not comparable to the kernel-time numbers above):
The previously published per-call wall clock (commit b3ceb7c) was 333.7 / 37.8 / 506.7 / 47.2 us at 1 x 8192 / 128 (and analogously for the other packs). The re-run above
reproduces the forward columns within ~±4% and the fused/backward columns within +2-14%:
the fused per-call wall clock carries the cached-launcher host optimization, whose
snapshot replay jitters run-to-run (the published fused-forward is even non-monotonic in
pack size, 37.8 -> 35.1 us), and eager backward drifts with GPU boost-clock state. The
CUDA-graph replay numbers below collapse that host jitter into a single replay.
CUDA-graph replay of the same region, both implementations captured symmetrically as a
forward-only graph and a forward+backward graph (median of 100 after 30 warmup, replay
timing via benchmark/csa/bench_csa_compressor.py; capturing each side into a graph
collapses its per-operation launches into a single replay, so this is the fairest
wall-clock basis for launch-bound shapes):
eager total / fused total capture forward + backward together. The eager total graph
captures the autograd backward of the captured forward against stable, pre-allocated zero
.grad buffers (the captured region zeros them in place, then runs forward + backward, so
every replay accumulates into a zeroed buffer — numerically identical to a single fresh
backward, verified per shape in the harness); the fused total graph captures the forward
wrapper immediately followed by the backward wrapper. The backward replay alone is not a
separately captured quantity — it is approximately total - fwd and is reported only as
that reference, never as a measured column. Graph speedups are eager / fused of the
displayed µs, truncated to one decimal.
Capturing each side into a graph narrows the absolute eager-vs-fused forward gap most on
the smallest, launch-bound shape (about 2.8x at 1 x 8192 / 128 forward) and barely on
the largest, less launch-bound shape (1.2x at 3 x 8192 / 512); the speedup ratio is
similar to or larger than per-call because a graph replay collapses each side’s
per-operation launches into one.
Environment: driver 590.48.01, PyTorch 2.13.0 (CUDA 13.3), nvidia-cutlass-dsl 4.6.1.
Measurement basis: identical inputs over exactly the replaced region for both
implementations; per-call eager backward = the torch autograd backward with the forward
outside the timed region; graph eager backward = the autograd backward of the captured
forward, captured together with it against stable zero .grad buffers; fused backward =
the backward wrapper (kernel + the fp32 dAPE zero-fill + host validation, no autograd
engine). Both wall-clock tables above (per-call and graph) are from a single run of
benchmark/csa/bench_csa_compressor.py; the kernel-time table is from nsys.
An ncu hardware-ceiling audit of the ported kernels (prior to the two optimization
commits) — cache-flushed, --set full, all four benchmark shapes — showed measured
DRAM read volume matching the algorithmically necessary bytes within 1% (the THD gather
adds no over-read; stores fully coalesced at 32/32 bytes per sector, loads 29-30/32),
with neither L2 (<27% of peak) nor DRAM (<33%) close to saturation at these
microsecond-scale sizes: the gap to a pure DRAM-floor time was a mix of sub-wave grid
width / occupancy, memory latency, and (at the largest shape) issue pressure — not
wasted traffic. The two optimizations that audit identified are folded in here: 32-bit
vectorized forward accesses and backward kernel-side zero-writes replacing the two bf16
grad-buffer fills. They do not change the bytes the kernels must read, so the
traffic-optimality conclusion carries over; the utilization percentages above predate
them.
ratio=128
ratio = 128 is served by dedicated kernels (compressor_sm100_r128.py): the generic
kernels keep the whole coff * ratio window in per-thread registers, which spills
kilobytes of local memory per thread at ratio = 128 (255-register cap). The dedicated
forward streams the window with a chunked softmax (one CTA per output row, the window
split over threadIdx.y chunk-rows, per-column (max, denom, acc) triples, one
fixed-order smem merge); its launch schedule is bucketed by the output-row count
(small / default / large, all precompiled for CUDA-graph safety) and per bucket selects
the online-rescale or two-phase accumulation form and, where it measured faster, an
ex2.approx-based fast exp. The dedicated backward stages each row’s window into
shared memory chunk-parallel, computes exp and per-chunk partial den/S sums in
the same pass, merges the partials per column in a fixed chunk order, and stores the
gradients with a hoisted 1/den multiply — plus kernel-side zero-writes to
never-consumed slots and, by default, fp32-atomic dAPE, exactly as at ratio=4. The
opt-in deterministic policy shares the generic family’s CTA-partial plus fixed-order
reduction design. The backward defaults to the same fast exp outside the d=128
small-pack (vec=1) buckets.
The ratio=128 numerics contract is the deterministic tolerance contract described in
Numerics, NOT the generic-family (ratio {2, 4}) bitwise-dKV/dScore contract. The
reduction orders (forward chunk merge, backward den/S partial merge), the
backward’s hoisted reciprocal, and the fast-exp buckets all differ from the eager op
order by design, each adopted on a measured same-GPU win and gated on tolerance +
fp64-oracle parity (worst observed deviations in Numerics). Everything
stays bitwise run-to-run: fixed chunk boundaries and merge orders, no atomics in
forward/dKV/dScore.
Both kernels are register-flat in the window length (ptxas sm_100a: forward 32-51
registers, backward 64-128 registers, 0 spill / 0 stack across every shipped
(config, schedule) kernel — 16 kernels total; reproduce the per-kernel table with
benchmark/csa/reg_probe_csa_compressor_r128.py) and JIT in ~0.4-1.0 s per
configuration.
The backward picks its rows_per_cta at launch time (a runtime argument — no
recompile) so the grid fits one resident wave; the static row capacity fixes it under
CUDA-graph capture. Small packs (nb_total <= 192, d=128) switch the backward to a
vec=1 schedule bucket (also precompiled) for grid-fill. Every backward bucket except
the c1d128 default additionally loads grad_out once per row and keeps the register
vec live across the phase barriers (the goreuse schedule field): adopted per bucket
on measured pure-kernel wins — up to ~26% on the vec=1 small buckets, whose small
grids underfill the machine, so the +16/+1 register cost is not the binding
constraint — while the c1d128 default measured ~1.3% slower with it at 131k tokens
and keeps the phase-4 reload. dKV/dScore are bitwise-identical with and without
the field (the same read-only values move into registers).
Measured on 1x B200 (CC 10.0, driver 590.48.01), torch 2.13.0 / CUDA 13.3 /
nvidia-cutlass-dsl 4.6.1; eager baseline = the fp32-intermediate reference region of
the test suite on identical inputs (the numerics-contract reference; the upstream eager
region differs only by a bf16 weight-rounding cast at ~equal cost). Packs are
single-sequence THD unless noted.
Isolated GPU kernel time (nsys, sum of kernel durations inside a
cudaProfilerApi-gated 100-iteration unsynced loop / 100; forward and backward each
measured in their own loop; the fwd+bwd total is one combined loop per training step —
forward + backward + the ~1 us fp32 dAPE zero-fill — and is not the sum of the
standalone columns, because the interleaved backward changes L2 residency for the
forward at long context):
End-to-end wall clock per call (CUDA events, median of 100 after 30 warmup; fused
backward includes a grad_ape.zero_() per call; not comparable to the kernel-time
numbers above — at small packs the ~9 us python launch path dominates the 5.7-11 us
kernels):
The ratio=128 numerics contract (21-case gate: 15 union + 1 scaled-input + 4 padding
cases, each running the determinism, tolerance and fp64-parity gates, plus 1
overflow-intermediate case that gates bit-stable NaN replay and NaN-pattern equality
vs the eager fp32 reference with the fp64 comparator explicitly skipped, with shipped
schedule coverage asserted) and the per-kernel ptxas table are reproducible from the
committed scripts benchmark/csa/gate_csa_compressor_r128.py and
benchmark/csa/reg_probe_csa_compressor_r128.py.
Testing
The tests validate numerics against an fp32-intermediate eager reference (bitwise
dKV/dScore at ratio {2, 4}; the deterministic tolerance contract at ratio=128, with
fp64-oracle parity on finite-intermediate inputs), the upstream eager numerics, plus ragged packs, static-capacity
padding, kernel-side zero-writes into uninitialized gradient buffers (NaN-canary with
exact-zero assertions on every never-consumed slot class, and the total_comp == 0
zeros fallback), run-to-run determinism for both dAPE policies, deterministic
workspace validation and CUDA-graph overwrite semantics, atomic-path grad_ape
zeroing ownership, the ratio=128
dispatch envelope (schedule selection at every nb_total bucket boundary at L0, and
one execution of every shipped (config, schedule) kernel against the full contract —
tolerance, fp64-oracle parity, determinism — at L1; run
pytest -m "L0 or L1"), CUDA-graph capture/replay, and check_support boundaries.