SwiGLU FP4 Gate / Up Product for Decode Rows (SM100-class)
Experimental. cudnn.SwigluFp4Decode (also cudnn.gemm.SwigluFp4Decode) is the gate / up product of a block-scaled FP4
(NVFP4) MLP at decode-time row counts, with SiLU(gate) * up and the FP4 re-quantization of the result fused into the
epilogue, as ONE CuTe DSL launch:
The product is accumulated in fp32 on the tensor core (tcgen05 block-scaled MMA) with the token rows on the narrow tile
of the instruction — 8, 16, 32 or 64 tokens — so 1 .. 64 rows do not pay a 128- or 256-row tile. The product itself is
never stored. Under dependent launch the first weight tiles are issued BEFORE the kernel waits for its predecessor, so the
weight bytes are already moving while the kernel that produces activation still runs.
It is a direct-call step entry, the same kind as the fused GDN decode step, the shared-expert decode pair and the FP8 projection for decode rows: an inference step with no backward, called by an engine that owns the tensors. It is not a graph node and it does not touch the GEMM engine.
Usage
Nothing is captured at construction: the eight tensors are launch arguments, so one entry serves every layer of its
geometry, and the entry owns no device memory. Every launch re-runs the binding checks (type, dtype, device, shape, byte
count, contiguity, alignment) before the compiled kernel is looked up; the host path makes views only — no conversion, no
copy, no device allocation, no device read. alpha and out_global_scale are read on the device. The launch goes to
torch’s current stream of the entry’s device. out or out_scale overlapping an input is undefined (not checked).
The row count is a runtime value. The kernel is compiled once per token tile, not per row count: compile() builds the
four forms, after which a launch at any served row count is a cache hit that compiles nothing; compile(rows) builds the
form of one row count. A launch of a form that was never compiled compiles it on first use; a serving path should not rely
on that. entry.cfg_for(rows) returns the form of a row count and entry.key(rows) its compile key (the form; the row
count is not part of it).
The weight layout
The kernel reads the weight in tiles of 128 rows: rows 128 j .. 128 j + 63 are gate channels 64 j .. 64 j + 63, rows
128 j + 64 .. 128 j + 127 the up channels of the same 64. interleave_gate_up(weight, weight_scale) builds that layout
from a weight whose gate rows 0 .. INTERMEDIATE - 1 are followed by its up rows, and permutes the scale bytes the same
way. It is a byte permutation — nothing is dequantized or requantized — that returns NEW tensors: one interleaved copy
per weight is the caller’s memory (100.3 MB at SwigluGeometry(5120, 17408)). It runs at load time, never on the launch
path, and its own preconditions are typed (TypeError for a non-tensor or a dtype; ValueError for a row count that is
not a multiple of 128, a column count that is not a multiple of 64, another scale byte count, two devices).
Numerics contract
There is no numerics knob: no split, no reduction across rows, no association choice. The output bytes are a function of the operand values on one runtime, and of nothing else:
- one fp32 accumulation chain per element in ascending K;
alpha *, then round-to-nearest-even to bf16 (gate and up);- SiLU as
x / (1 + 2**(x * fp32(-log2 e)))with an approximate exponential and an approximate division; * up, the second rounding to bf16;- the block maximum of
|v|over 16 channels; - the block scale
E4M3(gs * (max|v| * fp32(1/6))), round-to-nearest-even, saturating at 448; a block whose scale byte is zero writes zero codes; otherwise the codes are scaled byrcp(value(byte) * rcp(gs)), two approximate reciprocals; - E2M1 round-to-nearest-even, saturating at 6.
- A row’s bytes — its codes and each of its block scales — depend on that row’s operands only: they are the same alone and among up to 64 rows, at any position, with any neighbours (batch invariance), and identical across repeated launches. See determinism.
- Every field of the config is a performance form (below): the token tile, the weight stages issued ahead of the wait, the L2 policy of the weight stream, the cap of the persistent grid and the launch attribute. None of them changes a byte.
- Against an fp64 evaluation of the same chain on the exactly dequantized operands, every block scale lies in the interval of bytes its propagated error admits, and every code is within half a code step of its own block scale plus that error (the test utilities derive both bounds). Operands whose every intermediate is exact — integer sums, bf16 ties of both roundings in both directions, a gate of zero, power-of-two scales — give a host-built expectation byte for byte.
- This rounding chain is the entry’s own.
cudnn.ops.nvfp4_block_scale_quantizeapplies its encode scale to the elements before the block maximum, and it and the grouped GLU + Hadamard + quant fusion clamp the reciprocal of a zero block scale to the largest fp32 where this entry writes zero codes: the bytes of those APIs are not this entry’s. - Not a contract: equal bytes across GPU architectures or CuTe DSL compilers (the exponential, the division and the
reciprocals are approximate instructions); the codes of a row whose own activation scales hold a NaN code (such a row
leaves every other row’s bytes unchanged); and the codes under an
out_global_scaleoutside2**-100 .. 2**100, where a reciprocal can leave the normal range.
Forms and knobs
cudnn.gemm.frost.swiglu_decode.Cfg holds the performance fields; cfg=Cfg(...) passes a whole config.
The stage counts come from one constant capacity on every part, so a token tile is the same kernel everywhere; the part’s own readings (shared memory per SM, per-block maximum, SM count) only decide whether the part can hold a form, at construction. A thread block is 192 threads — four epilogue warps, one MMA warp, one loader warp — and allocates the whole 512 tensor-memory columns of its SM; shared memory places one thread block per SM.
Dependent launch
pdl=True (default off) puts the programmatic-stream-serialization attribute on every launch. Five of the six warps then
wait for the predecessor at once; the loader warp waits after it has issued its pre-wait weight stages.
As a dependent. The kernel reads activation, activation_scale, alpha and out_global_scale below its wait, so
they may be the predecessor’s outputs. Caller obligation: no kernel that can still be running when the launch starts
— the immediate stream predecessor and every earlier launch of an uninterrupted chain of early releases — writes weight or
weight_scale (static weights satisfy it). The tensor-memory allocation is below the wait: a waiting thread block holds
shared memory and no tensor memory.
As a predecessor. A thread block signals its dependents when its loader warp has finished, BEFORE the block’s last
output tile is written. A kernel launched behind this entry WITH the attribute may therefore start while out and
out_scale are still being written, and must issue its own wait before it reads them. A kernel launched without the
attribute starts after this one has completed. There is no release knob: the release point is the kernel’s own.
Launch facts
The entry states what a row count launches without launching it: three read-only answers, with cfg_for’s argument.
One call is one kernel launch. The symbol carries the form right behind the kernel’s name —
cudnn_kernel_frost_swiglu_fp4_decode_h5120_i17408_t8_pw2_evf_c148_nopdl_... — so two forms of one entry never share a
symbol. A row count outside 1 .. 64 is a typed decline (ValueError for one that is not a positive int,
NotImplementedError past 64), and none of the three compiles, launches or touches device memory.
Support matrix
Typed declines, all before any DSL import: TypeError for a non-tensor, a dtype, or a geometry or config of another
type; ValueError for a shape, a byte count, an alignment, a device mismatch, a knob outside its domain, two sources for
the geometry, or a row count that is not a positive int; NotImplementedError for a row count past 64, a geometry, a
memory layout, a part or a DSL the kernel does not serve. No copy is ever made to serve a layout.
The kernel is derived from FlashInfer’s persistent block-scaled dense GEMM (BSD-3-Clause notice retained, in an
Apache-2.0 project); the kernel file’s header and THIRD_PARTY_LICENSES.txt carry the notice and the list of changes.
Tests: test/python/gemm/frost/swiglu_decode/test_swiglu_fp4_decode.py (host declines, config and ledger pins, the
rendering pin per target, the launch facts against the configuration arithmetic and against the profiler’s record of a
launch, the device cells: the fp64 reference, exact cells, edge rows, every knob arm bitwise with the default, a waiting
dependent behind the entry, CUDA-graph replay, a warm stress with byte references) and
test_swiglu_fp4_decode_determinism.py (replay, batch invariance at every row count, the token-tile pick’s byte
neutrality, goldens). Source: python/cudnn/gemm/frost/swiglu_decode/.