Linear Attention
Chunked prefill and backward linear attention kernels for Mamba-2, GDN, GDP, KDA, and GDN-2 served by the open-source FROST kernels through the Python frontend. Every variant has a forward, a full backward, and a pair of summary operations for context parallelism.
The ops live in cudnn.linear_attention.ops and are PyTorch custom operators with autograd formulas; the summary ops are <op>_summary and <op>_summary_bwd. The same graph nodes are available on the Python graph API.
Support Matrix
Requires an SM100 / SM103 / SM107 GPU (Blackwell data center) and the CuTe DSL runtime the package depends on (nvidia-cutlass-dsl, a required dependency except on Windows ARM64; these kernels need 4.7.0 or newer and the engines decline with an error that names the installed version otherwise). On Windows ARM64, the CuTe DSL and TVM-FFI runtime dependencies are excluded from the base installation, so the base install does not enable these kernels there. One linear-attention node per graph.
T is the total token count of the packed batch, N the number of sequences.
Layout
All operands use the packed THD layout, where the tokens of every sequence are concatenated with cu_seqlens ([N + 1]) giving the sequence boundaries. For example, a dense batch of N equal-length sequences is cu_seqlens = [0, T, 2T, ...].
Operands may be strided views, as the engines build their TMA descriptors from the buffer strides, and the only requirement is a contiguous innermost (head-dim) axis. A column slice of a wider per-token buffer (for example the q, k, v pieces of one fused projection) is accepted directly, forward and backward, without .contiguous(). Tensors are rank 3 ([T, heads, dim]).
Additional Semantics
- States and checkpoints.
output_final_state=Truereturns the per-sequence state after the last token.checkpoint_every_n_tokens > 0also returns the state series, one entry everyntokens strictly before each sequence end, as a non-differentiable checkpoint that the backward can reuse instead of recomputing.state_indicesturnsinitial_stateinto a pool[N_pool, ...]addressed per sequence, written back in place. The rows are io dtype by default and fp32 withcheckpoint_dtype=torch.float32; in the forward the states and the rows may also be fp8 (e4m3fn, e5m2).state_indicestakes int32 or int64 indices and composes with checkpoints; the pool may be a strided view with a unit-stride innermost axis and 16-byte aligned, non-overlapping outer strides. - Forward context.
output_context=True(GDN, KDA, GDN-2, GDP) also returns the uint8 context the forward leaves for the backward of the same tokens: the piece transitions of a chained forward and, for GDN and GDP, its chunk factors and the normalizedq/krows underuse_qk_l2norm_in_kernel(GDP atV = 64). Autograd hands it to the backward, which reads them instead of recomputing; the gradients are bitwise the same with and without it. - Checkpoint conventions. By default row
jof a sequence ofLtokens is the state before blockj(afterj * ntokens): row 0 is the incoming state, the sequence end is excluded,ceil(L / n)rows per sequence andT // n + Nrows of capacity. Withcheckpoint_after_block=Truerowjis the state after blockj(after(j + 1) * ntokens): the incoming state is never stored, an aligned sequence end is stored and equalsfinal_state,floor(L / n)rows per sequence (none for a sequence shorter thann) andT // nrows of capacity. A backward node given after-block rows does not seed from them; it recomputes the series as it does without checkpoints. - batch_invariant. Each sequence’s results are bitwise independent of the batch composition (whole-sequence scheduling; split-K load balancing is disabled).
- Clamps and guards. Mamba-2’s
dtclamp zeroes the gate gradient where active. GDN-2’sbeta_guardis recomputed, not differentiated, in the backward. - Engines. The FROST engines are the only engines for these nodes and are pure pass-through:
check_supportverifies the kernel-native dtypes and head dims listed above and declines everything else. A plan can be pinned by name throughplan_name(for examplegdn_frost,mamba2_frost,mamba2_summary_frost). - Custom-op dispatch. In eager mode
kimi_delta_attentionandgated_delta_netskip thetorch.librarycustom-op dispatch, which costs more host time than the op’s own Python, and apply the same forward and autograd formula through a plaintorch.autograd.Function(the output’sgrad_fnisEagerKdaBackward/EagerGdnBackward). Undertorch.compile,torch.jit.trace, the autograd profiler (so traces showcudnn::<op>_fwd/_bwd), a torch function or dispatch mode, a functorch transform, or tensor subclasses they call the custom op, and acreate_graph=Truebackward runs the custom backward op, so differentiating it again raises as before. - C++ launcher. KDA’s and GDN’s forward and backward on the FROST engines run through the C++ launcher: once a call with the same tensor shapes, strides, dtypes and options has run 16 times, the kernels launch from C++ and the op’s Python (checks, graph and engine dispatch, workspace carving) is skipped. Recordings key on shapes only, so new
cu_seqlensvalues with the sameNreuse the recording. A backward whose incoming gradient is a broadcast view (out.sum().backward()) keeps running the op’s Python, which materializes it.CUDNN_FRONTEND_CPP_LAUNCHER=0turns the launcher off;=verifychecks every call against its recording.
Example
Benchmarks against the reference implementations are in benchmark/linear_attention/.
Context Parallelism
The five families support chunkwise context parallelism. We leave the all-gather of the summary to the framework-level, but we compute the summaries and combine them.
Each rank summarizes one boundary sequence. <op>_summary for the last local sequence from a zero state for the forward, and <op>_summary_bwd on the first local sequence from a zero outgoing-state gradient for the backward. Each returns a state row and the span’s transition operator. After the all-gather, the forward on rank rank takes gathered[preceding_rank_start:rank] and the backward gathered[rank + 1:following_rank_stop], in ascending rank order. The fold is inside the ops. The forward takes the preceding ranks’ slices as preceding_state_update and preceding_transition. The raw backward op torch.ops.cudnn.<op>_bwd takes the following ranks’ slices as following_state_grad and following_transition and folds them into the gradient state the last local sequence starts from.
Context Parallelism Example
Below, we show an example of using context parallelism in practice.