Add + RMSNorm + Quantize Decode Step (SM100-class)
Experimental. cudnn.AddNormQuantDecode (also cudnn.norm.frost.AddNormQuantDecode) is the residual add, the
RMSNorm and the quantization of the next projection’s input, at decode-time row counts, as ONE CuTe DSL launch:
x and residual are [rows, hidden] bf16 with rows in 1 .. 1024 (a runtime argument) and hidden 5120 (fixed at
construction). weight is the zero-centred norm weight, [hidden] or [1, hidden] bf16: the kernel applies
1 + weight. scale is one fp32 element (0-d or [1]). Compute capability 10.x; a CuTe DSL >= 4.8.0.
It is a decode-step entry: the residual add, the normalization and the quantization the next projection consumes, in one launch. For a standalone normalization use the Normalizations graph API: its RMSNorm kernel sums a row in another order, so its bytes are not this entry’s, and it has no residual add and no quantized output.
Forms
Three residual modes times five output sets: fifteen forms, each one compiled kernel that serves every row count and
every layer. add, bf16 and quant are required keywords; the forms are peers and none is a default.
The returned tuple has the same three slots for every form, with None in a slot the form does not produce. quant is
one vocabulary (None and cudnn.norm.frost.kernel.add_norm_quant_decode_config.QUANT_FORMS). A form that writes no
output (bf16=False, quant=None) and add together with store_residual are declined at construction.
NVFP4 codes are two per byte with the even element in the low nibble. NVFP4 block scales are stored in the 128 x 4
scale layout over (rows, blocks): tiles of 128 rows, one after the other; the scale buffer therefore has
ceil(rows / 128) * 128 rows.
Numerics contract
The sum tree and the rounding chain are the contract, not a tolerance. Per row, in fp32 on the bf16 operands:
v = x + residual: one fp32 addition per element (underadd). The updated residual isbf16(v), nearest-even.- The sum of squares is ONE fixed tree whose shape depends on
hiddenalone: a thread owns eight contiguous elements and folds them with one fused multiply-add each, in element order, from+0; the 32 lanes of a warp are summed by a butterfly with offsets 1, 2, 4, 8, 16; the warps (20 athidden5120) by the same butterfly. inv = rsqrt( total / hidden + eps ): an IEEE division by the width, one addition, the part’s approximate reciprocal square root (rsqrt.approx.f32).y = bf16( (v * inv) * (1 + weight) ):1 + weightis one fp32 addition, then two multiplications and ONE rounding to bf16 (nearest-even). This bf16 value is the normalized value of record:outstores it and the codes are formed from it, so the codes are those of quantizingoutitself.quant="e4m3_static":code = e4m3( y * (1 / scale) )with the correctly rounded fp32 reciprocal (div.rn.f32(1, scale)), nearest-even, saturating at +-448 (codes0x7E/0xFE). A consumer readscode * scale.quant="nvfp4": per block of 16 elements, the block scale ise4m3( scale * (max|y| * fp32(1/6)) )(the stored byte; two fp32 multiplications and one nearest-even rounding). The codes aree2m1( y * q )(nearest-even, saturating at +-6) withq = rcp( block_scale * rcp(scale) ), wherercpis the part’s approximate reciprocal (rcp.approx.ftz.f32); a block whose scale byte is zero writes the codes of the signed zero. A consumer readscode * block_scale / scale.
What changes bytes: hidden (the tree and the divisor), eps, add (the function that is normalized), quant (which
codes exist) and the operand values; scale moves codes and block scales only. What never changes a byte: the row
count, a row’s position, the other rows, bf16, store_residual, pdl, pdl_release, and who owns an output.
A row alone writes the bytes it writes inside a call of up to 1024 rows. A block scale’s POSITION in the scale buffer
follows the row index (the layout); its value does not.
Against an fp64 evaluation of the bf16 operands, every out element is one of the two bf16 values that bracket the
exact normalized value (the kernel’s fp32 value is within 20 x 2^-24 relative of it), and on random operands the
count of elements that are not the nearest of the two is bounded from the same arithmetic (a share of at most
2 x 20 x 2^-24 x 2^8, asserted with a binomial slack). The updated residual is
bitwise bf16(fp32(x) + fp32(residual)). The E4M3 codes and the NVFP4 block scales are bitwise the host models above
applied to the kernel’s own out. Every NVFP4 code is one of at most two adjacent codes around the exact quotient
y * scale / block_scale (the bound of the two approximate reciprocals), and one code wherever the quotient is not
within that bound of an E2M1 rounding midpoint.
scale is by contract a finite positive fp32 in 2**-100 .. 2**100. Its value is never read on the host: outside that
range the reciprocal chains overflow or flush. A NaN or infinite element makes its own row’s out non-finite and
leaves that row’s codes unspecified; every other row is unaffected. A row whose squares underflow normalizes against
eps; a row whose sum of squares overflows fp32 normalizes to zeros.
The bytes follow the part’s rsqrt.approx and rcp.approx.ftz units. Equal bytes across GPU architectures are not
part of the contract; in particular an NVFP4 code at an exact rounding tie under a block scale that is not a power of
two follows the approximate reciprocal (see determinism).
Usage
Everything behind eps is keyword-only. One compiled kernel serves every row count and every layer of a model: all
tensors are launch arguments, the slabs are bound with a runtime extent and rows is a runtime argument, so no shape
enters the compiled kernel, the compile key is the config alone, and a captured CUDA graph holds pointers, never
values. compile() needs no operand. The launch goes to torch’s current stream of the entry’s device. That device need
not be the caller’s current one: the compile and the launch run under the entry’s device, and the caller’s current
device is restored afterwards. There is no workspace; a warm call allocates nothing and reads no device memory to the
host.
Every operand is contiguous, starts at a 16-byte aligned address (scale: 4-byte) and lives on the entry’s device; no
written operand may overlap another operand.
Output ownership is per output. An output the caller passes is the caller’s and is returned as given; rows past the
call’s are never written. An output the caller does not pass is the entry’s: one set per row count holding only the
form’s outputs, scales zero-filled, allocated by compile(rows) / outputs(rows) or at the first call of that row
count. Call compile(rows) for every row count before a CUDA-graph capture. Entry-owned outputs are valid until the
entry’s next call at that row count from ANY host thread: stream order does not serialize host calls, so two host
threads that share one entry pass their own outputs or build one entry each. The per-row-count sets are never freed, so
a caller that sweeps many row counts passes its own outputs. An entry-owned output is an operand of the launch like a
passed one, so the overlap rule covers it: a call whose x, residual, weight, scale or passed output overlaps an
entry-owned output that the call does not pass (a tensor an earlier call of that row count returned, fed back in) is
declined with ValueError before the launch. To feed a returned output back in, pass a caller-owned output.
The kernel writes the live rows’ scale bytes only. A caller-owned scales buffer serves one row count, or its
owner clears it when the row count shrinks: after a call with more rows, a call with fewer leaves the earlier call’s
bytes in the rows between the new rows and ceil(rows / 128) * 128.
Everything the entry does not serve is declined by name: TypeError for a non-tensor, a dtype, a hidden that is not
an int, a flag that is not a bool, or a rows of compile / outputs that is not an int; ValueError for a
width outside the kernel’s structure (a multiple of 256 in 256 .. 8192), an eps that is not a finite positive Python
float or int (the message names the type it got), add with store_residual, a quant outside the vocabulary, a
form without an output, pdl_release=True without pdl=True, an operand the form requires and did not get or got and
does not use, a shape, an alignment, a device mismatch, an overlap, or an empty call; NotImplementedError for a
structurally valid width other than 5120, a row count above 1024, a non-contiguous operand, a compute capability
outside 10.x, and the CuTe DSL gates (not installed, below 4.8.0, without the part’s target, or with a compiler that
cannot assemble it). The operand checks import no DSL module and run before every compile and every launch; the
architecture gate and the DSL gates run in compile(), before the kernel module is imported.
Launch knobs and the caller’s obligations
Both knobs default to off and neither changes a byte (the suite compares their bytes).
pdl=True: the launch carries the programmatic-stream-serialization attribute and the kernel issuesgriddepcontrol.waitAFTER it has loadedweightandscaleand BEFORE it readsx/residualor writes anything. Obligation:weightandscaleare not written by any launch that may still be running when this one starts (they are constants of the layer).pdl_release(Nonefollowspdl;Trueneedspdl=True):griddepcontrol.launch_dependentsdirectly behind this kernel’s OWN wait; on this entry the release never precedes the wait. Obligation: every launch that can start before this one completes issues its own wait before it reads this launch’s outputs and before it writes any operand of this call. Whether the release helps depends on the row count (a released consumer occupies SMs); a caller that wants it for some row counts only builds two entries.
pdl=True alone is wait + release; pdl=True, pdl_release=False is wait only. The suite pins the wait’s position in
the kernel’s text for every form and target: the loads of weight and scale are the only global accesses in front
of it, the loads of x and residual and every store follow it, and the release is the next instruction. On a
device it runs the step behind a predecessor that releases its dependents before it writes x and residual, the
release with a dependent launch on the stream that waits and copies the outputs, and two steps chained through one
buffer; each eager and captured.
Launch facts
The entry states what a row count launches without launching it: three read-only answers, with cfg_for’s argument.
A row count outside 1 .. 1024 is a typed decline (ValueError for one that is not a positive int,
NotImplementedError past 1024), and none of the three compiles, launches or touches device memory.
Support matrix
Tests: test/python/norm/frost/test_add_norm_quant_decode.py (host declines, the config’s limits and tables, the
rendering pinned per target and tied to the entry’s own launch arguments, the launch facts against the configuration
arithmetic and against the profiler’s record of a launch, the device cells against the references above, exact and
edge rows, rows whose sum of squares overflows, output ownership, the bit-neutral arms, the wait behind a predecessor
that released early, the release beside a dependent launch, a chain of two steps, CUDA-graph replay, an entry on a
device that is not the caller’s current one) and test_add_norm_quant_decode_determinism.py (replay, batch invariance,
goldens). Source: python/cudnn/norm/frost/.