Fused GDN Decode / Verify Steps as a FlashInfer Backend
Experimental. cudnn.linear_attention.frost.gdn_decode_fused and
cudnn.linear_attention.frost.gdn_verify_fused are the fused Gated DeltaNet (GDN) decode step
and MTP verify step of a Qwen 3.5-architecture dense model’s GDN sub-layer, one CuTe DSL launch
each, for SM100-class (compute capability 10.x) and SM120-class (12.x) parts — validated on
cc 10.0 (GB200) and cc 12.0 (RTX 5090). cudnn.linear_attention.frost.flashinfer_gdn_backend registers
them as the backend "frost_fe" of FlashInfer’s experimental Qwen 3.5 GDN block
(flashinfer.experimental.qwen_3_5_gdn_block), so the block’s plans, plan cache, CUDA-graph
rules, reference tests and benchmark run the cuDNN Frontend kernels unchanged.
Each step fuses, after the FP8 QKVZ projection of a serving step: the BA projection, the causal
conv + SiLU over the slot’s history, the Q / K RMS scales, the gated delta-rule state update
(one token per request for decode, the request’s T draft tokens for verify, every token’s
state written to its own slot), the gated RMSNorm, the static per-tensor FP8 output quantization
and the conv-history shift. One CTA per (request, V head) with 256 threads; the 128 x 128 fp32
state rows are register-resident and travel through padded shared-memory boxes by
cp.async.bulk. The kernels serve the GDN sub-layer of one tensor-parallel rank as a geometry
Geometry(K_HEADS, V_HEADS, HIDDEN) (cudnn.linear_attention.frost.kernel.gdn_geometry): the
head counts and the hidden width move, head dim 128 and conv width 4 are pinned, and every
geometry is its own compiled form of the same kernel body (the default geometry — 16 K heads,
48 V heads, hidden 5120 — is the one every number on this page is written for; an entry takes
another with geometry=, a FlashInfer plan through its Qwen35GdnGeometry). The default form
follows three plan-time facts, the request count, the device’s SM count and the device’s
shared-memory readings (per SM and the per-CTA opt-in): a cluster of two CTAs per head (half the
state rows each) while the requests x 2 V cluster grid has an SM per CTA (requests x 96 at 48
V heads: one request on 148- / 152- / 170-SM parts), one CTA per SM while the requests x V grid
has one (requests x 48), and past that a register-capped two-CTAs-per-SM form so the grid
runs in one wave — where two of its CTAs can share an SM, i.e. 2 x (footprint + 1 KB) fits
the SM’s shared memory (228 KB on cc 10.0 parts). On a 100 KB-per-SM part (cc 12.0: 99 KB
per-CTA opt-in) every whole-slab form is one CTA per SM by shared memory alone, so the cap is
never picked there (it would only spill); the decode step runs the one-CTA form in two waves
past the one-CTA grid, and the verify step — whose cc 10.0 forms carry a 3-deep ring of 64 KB
state slabs that a 99 KB opt-in cannot hold — serves the same kernel body with shallower rings:
the two-CTAs-per-head pair with a 2-deep ring at one request and again past the one-CTA grid
(four and more requests on 170- / 188-SM parts: its one-per-SM cluster CTAs fill the waves the
one-CTA form’s nearly empty second wave would leave — measured at four requests on one 170-SM
RTX 5090 at boost clocks (the board allows no clock lock; three interleaved A/B/A rounds of 100
event-timed iterations, the L2 swept before every launch so code and data are cold, every arm
bitwise with the pick): 65.54 vs 71.68 us at T 6 and 50.90 vs 53.25 us at T 4, +4 to +9 % faster
than the one-CTA form across the cold, warm and in-situ readings; the other beyond-the-grid cells
take the order by the budget rule, unmeasured — see Performance), the one-CTA landing-slab
(NBUF 1) form at two and three requests, each up to 8 tokens per request, and the landing-slab
pair for 9 to 17 tokens per request at every requests count. The verify step traces its token
loop once (ROLLED_TOKENS) in the forms that keep the state in registers. Every knob is a compile-time
axis of one kernel body and bitwise with the default pick (no knob moves an fp32 addition), and
the outputs reproduce FlashInfer’s built-in frost backend byte for byte.
Register
Importing the module registers the backend; FlashInfer’s benchmark loads a backend
module with --register:
register_frost_fe(replace=False) is the explicit entry: idempotent for this
module’s own registration, replace=True re-registers (and drops the backend’s
cached plans). Nothing here is imported at import cudnn time; the two entries are also
reachable lazily as cudnn.GdnDecodeFused / cudnn.GdnVerifyFused (and under
cudnn.linear_attention.frost): direct-call kernels with the block’s argument contract, not
graph engines. Each entry resolves its device to a concrete CUDA ordinal at construction and
validates the binding (every operand on exactly that device, dtype, shape, stride, 16-byte
alignment, the step shape) before every compile and every launch, compile-cache hits included
— the compile key pins the plan-time facts only, and a compiled CuTe DSL entry accepts a tensor
of another dtype, footprint or device silently — and declines a step of more than 65535
requests (one grid row each; the decode kernel also forms its row addresses in 32-bit
arithmetic, so the row count times the widest of the qkv / z row stride, d_model and
the output row must stay below 2^31 elements — gdn_decode_fused.max_rows_for, a bound a
row-strided qkv / z view can lower below 65535, declined by type at prepare). CPU
operands or operands on another CUDA device are a ValueError naming the operand and both
devices.
Served and declined
Plans and CUDA graphs
A plan’s prepare compiles the kernel for the binding’s strides once (plan-time
facts only; T is a compile-time constant of the verify kernel) and is cheap
afterwards; run launches on the current stream, writes the caller’s FP8 out (and
bf16 out_bf16 when given) and advances conv_state / ssm_state in place. A
binding that was never run eagerly is refused under capture with a RuntimeError,
per the block’s contract. Compiled kernels are shared by every plan of one form,
device and norm epsilon.
Both entries take the launch knobs pdl / pdl_release (programmatic dependent launch:
the kernel waits for its predecessor only right before it reads the QKVZ projection’s rows and
the normed hidden, and releases its successor at entry). The caller then owes the obligation in
the config docstrings: the launch’s immediate predecessor must not write this layer’s state,
conv state or slot indices. The default forms keep PDL off.
Launch facts
Both entries state what a step launches without launching it: three read-only answers per served row count — (rows)
on GdnDecodeFused, the step shape (requests, tokens) on GdnVerifyFused. Both verify arguments are required:
cfg_for reads a missing tokens as the minimum (2) because one config can stand behind every tokens count, but the
launch differs per tokens count (shared memory, compiled kernel), so a fact never answers for an assumed step.
The symbol is the kernel’s name followed by the head of its config’s text, cut at a fixed length. What sits behind the
cut is not in the name — with CuTe DSL 4.8.0 the two-CTAs-per-head field, the geometry and the operand layout of both
kernels, and the launch knobs and the tokens per request of the verify kernel — so two forms can share a symbol.
Compare the form, not the name. On a cc 10.0 part at the default geometry, one request: decode
((96, 1, 1), (256, 1, 1), 42112), verify at six tokens ((96, 1, 1), (256, 1, 1), 126512), with or without pdl. A
step no launch of the entry serves is a typed decline: ValueError for a count that is not a positive int (or a
tokens that is not an int), NotImplementedError past the grid’s limit, past the 32-bit row-address bound of the
decode entry’s geometry, or for a tokens count outside the verify form’s bounds on the part. A call’s operand strides
can bound the decode rows lower still; that decline is the call’s. None of the three compiles, launches or touches
device memory.
Validation
test/python/linear_attention/frost/test_gdn_decode_fused.py and test_gdn_verify_fused.py
cover the host entries (typed declines, the plan-time compile key, the knob validation, the
shared-memory budget per tokens count on the cc 10.0 and cc 12.0 readings, the budget-derived
pick tables of both parts, the geometry envelope and the per-rank geometries’ derived constants,
operand shapes and BA-partition coverage, the DSL and architecture gates, the launch facts against the configuration
arithmetic) and, on an SM100-class or SM120-class device, the launch facts against the profiler’s record of a launch,
the null-slot and padding contracts, determinism, CUDA-graph replay over distinct
layers, a torch reference of the fused chain and the bitwise agreement of every knob arm the part’s
shared memory holds with the default pick (an arm the part does not hold is its typed decline,
reported as a skip). When CUDNN_FROST_GDN_VERIFY_REFERENCE_DIR names a directory of saved verify
operand sets with the reference kernel’s outputs (set_<tag>.pt / ref_<tag>.pt), every set is
replayed and compared byte for byte.
test/python/linear_attention/frost/test_flashinfer_gdn_backend.py runs
FlashInfer’s own decode and verify tests (tests/experimental/qwen_3_5_gdn_block/)
against frost_fe when a FlashInfer source tree is next to the installed package —
the verify reference also under the 148-SM launch policy and with the SM count held
below one request’s grid (the beyond-the-grid regime’s budget-fitting form at every shape: the
two-CTAs-per-SM form on a cc 10.0 part, the 2-deep-ring pair on a cc 12.0 part) — and checks the backend bitwise
against the built-in frost backend where that one is served: decode on its
reference cases, verify on the benchmark’s own input sets at requests 1 / 2 / 4 x 6
tokens (every token accepted, and random acceptance with a padding request).
Performance
One GPU class so far, GB200 (1200 W, 152 SMs, SM clock locked at 1830 MHz, one exclusive GPU) — the cc 12.0
admission ships with its functional validation only; a GeForce board allows no clock lock, so an RTX 5090 table
will be a separate table with its own (boost-clock, interleaved) protocol stated beside it (the one cc 12.0 figure on
this page, the verify pick order past the one-CTA grid, is such a boost-clock A/B/A reading with its protocol stated
inline, not a table entry). CUPTI
kernel-duration medians inside a serving step at batch 1 (cold code and data, as every layer’s
launch finds them), speed-ups positive = faster (baseline / new - 1):
Outputs were bitwise identical to the frost backend on every saved reference set and in
serving (identical tokens and acceptance counters). End to end on the same path (three
interleaved rounds, acceptance-length gate), the time per output token improved by +6.4 % with
MTP (five drafts, batch 1) and +6.9 % without; those figures include the PDL launch mode of the
fused steps and the GEMM tactic changes made in the same path, which are not part of this page’s
kernels (the kernels alone account for +2.0 % and +4.6 %, read against a same-snapshot control).