Attention Gate + Quantize Decode Step (SM100-class)
Experimental. cudnn.AttentionGateQuantDecode (also cudnn.attention_gate.frost.AttentionGateQuantDecode) is the
output gate of an attention block fused with the static FP8 quantization of the following projection’s input, at
decode-time row counts, as ONE CuTe DSL launch:
attn (the attention output) and gate (the PRE-sigmoid gate) are [rows, width] bf16 with rows in 1 .. 160 (a
runtime argument) and width a positive multiple of 8 up to 32768 (heads x head_dim of the attention output, fixed at
construction). scale is one fp32 element (0-d or [1]): the static per-tensor dequantization scale of the consumer, a
projection with an E4M3 input. out is [rows, width] E4M3 (torch.float8_e4m3fn), written into a caller-owned
[>= rows, width] buffer or the entry’s own. Compute capability 10.x; a CuTe DSL >= 4.8.0.
The kernel is an element-wise map: thread i of a flat grid owns eight consecutive elements of the row-major slab (one
128-bit load of each operand, one convert and one byte store per element). It has no reduction, no shared memory and
no barrier, so every output byte is a function of its own (attn, gate) pair and of scale.
Numerics contract
The rounding chain is the contract, not a tolerance. Per element, in fp32 on the bf16 operands:
1 + exp(-gate)with the CuTe DSL’s precise exponential, then the IEEE divisionattn / (1 + exp(-gate))(div.rn.f32).- The quotient is rounded ONCE to bf16 (round-to-nearest-even).
- That bf16 value is multiplied by the correctly rounded fp32 reciprocal of
scale(div.rn.f32(1, scale)) and rounded ONCE to E4M3 (round-to-nearest-even, saturating at +-448: codes0x7E/0xFE).
This is the convention of a chain that stores the gated attention output as bf16 and quantizes it for a projection whose
static input scale is scale. The gated value is the DIVISION attn / (1 + exp(-gate)): a chain that rounds the
product attn * sigmoid(gate) instead writes a code one apart on rare elements, so a reference must spell the
division. The entry has no numerics argument: no knob, pick or row count changes a byte.
Against an fp64 evaluation of the bf16 operands, every byte is the E4M3 code of one of the two bf16 values that bracket
the exact gated value, and the code of the nearest one wherever the exact value is more than 2.5 fp32 units away from
the midpoint of that bracket. The test suite asserts both statements on drawn and adversarial operands and on every
finite gate value; its census module measures them over every (attn, gate) bf16 pair at five scales.
scale is a positive normal fp32 value by contract. Its value is never read on the host: a zero, subnormal, negative
or non-finite scale yields saturated, sign-flipped or NaN bytes. A NaN or infinite operand element yields the E4M3 NaN
byte or a saturated code for that element only.
The bytes follow the installed CuTe DSL’s lowering of the exponential and the part’s ex2.approx unit. Equal bytes
across DSL versions or across GPU architectures are not part of the contract (see
determinism).
Usage
One compiled kernel serves every row count and every layer of a model: all four tensors are launch arguments, the
slabs are bound with a runtime extent and the element count rows x width is a runtime argument, so no shape enters
the compiled kernel, the compile key is the config alone, and a captured CUDA graph holds pointers, never values. The
launch goes to torch’s current stream of the entry’s device. That device need not be the caller’s current one: the
compile and the launch run under the entry’s device, and the caller’s current device is restored afterwards. There is
no workspace; a warm call allocates nothing and reads no device memory to the host.
Everything the entry does not serve is declined by name: TypeError for a non-tensor, a dtype, a width that is not
an int, or a cfg of another type; ValueError for a shape, a non-contiguous operand, an alignment, a device
mismatch, an out that overlaps an input, an empty call, or a knob outside its vocabulary; NotImplementedError for a
row count above 160, an unserved width, a compute capability outside 10.x, and the CuTe DSL gates (not installed, below
4.8.0, without the part’s target, or with a compiler that cannot assemble it). The operand checks import no DSL module
and run before every compile and every launch; the architecture gate and the DSL gates run in compile(), before the
kernel module is imported.
Output ownership. With out the caller owns the storage and gets the same tensor back; rows past the call’s are
never written. Without it the entry returns a view of ONE entry-owned [160, width] buffer whose bytes are valid until
the entry’s NEXT call from any host thread. Stream order does not serialize host calls, so two host threads that share
one entry, on one stream or two, pass their own out (or build one entry each). The entry-owned buffer is allocated
at the first compile() or call that passes no out; compile without out before a CUDA-graph capture if a captured
call will use it.
Layout. attn, gate and out are contiguous and start at a 16-byte aligned address (scale: 4-byte); every
operand lives on the entry’s device.
Knobs and the caller’s obligation
cudnn.attention_gate.frost.Cfg(THREADS=256, PDL_RELEASE=False), passed as cfg=. Both fields are performance-only
and bit-neutral (the suite compares their bytes):
THREADS(128 or 256): threads per CTA, hence the grid. Withoutcfgthe entry takes a pick from the element count and the part (default_cfg); the pick table holds one row today, 256 threads everywhere.PDL_RELEASE, also the keyword-only launch knobpdl_release=of the entry (Nonekeeps the config’s field):griddepcontrol.launch_dependentsas the kernel’s first statement. Default off. The launch itself is ordinary: it never carries the programmatic-stream-serialization attribute and the kernel holds nogriddepcontrol.wait.
With the release on, every grid that can start before this kernel completes — a grid launched with the
programmatic-stream-serialization attribute, and transitively its own dependents — must issue its own
griddepcontrol.wait before it reads out and before it writes attn, gate or scale. A consumer launched without
the attribute is unaffected. The release is verified standalone only: its bytes equal the default’s with no dependent
grid on the stream; a chain with a dependent launched under the attribute is not covered by this suite.
Launch facts
The entry states what a row count launches without launching it: three read-only answers, with cfg_for’s argument.
A row count outside 1 .. 160 is a typed decline (ValueError for one that is not a positive int,
NotImplementedError past 160), and none of the three compiles, launches or touches device memory.
Support matrix
Tests: test/python/attention_gate/frost/test_gate_quant_decode.py (host declines, config and pick pins, the rendering
pinned per target and tied to the entry’s own launch arguments, the launch facts against the configuration arithmetic
and against the profiler’s record of a launch, the device cells against the fp64 reference, an entry
on a device that is not the caller’s current one), test_gate_quant_decode_determinism.py (replay, batch
invariance, goldens) and test_gate_quant_decode_census.py (pytest -m L2: every bf16 pair). Source:
python/cudnn/attention_gate/frost/.