Skip to navigation

Add + RMSNorm + Quantize Decode Step (SM100-class)

View as MarkdownOpen in Claude

Experimental. cudnn.AddNormQuantDecode (also cudnn.norm.frost.AddNormQuantDecode) is the residual add, the RMSNorm and the quantization of the next projection’s input, at decode-time row counts, as ONE CuTe DSL launch:

v = x + residual (add=True; fp32) residual <- bf16(v), in place
v = x (add=False) residual <- x when store_residual=True
y = bf16( v * rsqrt(mean(v^2) + eps) * (1 + weight) ) fp32 arithmetic, one rounding to bf16 (nearest-even)
out = y (bf16=True)
codes = e4m3( y * (1 / scale) ) (quant="e4m3_static": the consumer's static per-tensor scale)
codes, scales = nvfp4( y ; global scale ) (quant="nvfp4": 16-element blocks, E4M3 block scales)

x and residual are [rows, hidden] bf16 with rows in 1 .. 1024 (a runtime argument) and hidden 5120 (fixed at construction). weight is the zero-centred norm weight, [hidden] or [1, hidden] bf16: the kernel applies 1 + weight. scale is one fp32 element (0-d or [1]). Compute capability 10.x; a CuTe DSL >= 4.8.0.

It is a decode-step entry: the residual add, the normalization and the quantization the next projection consumes, in one launch. For a standalone normalization use the Normalizations graph API: its RMSNorm kernel sums a row in another order, so its bytes are not this entry’s, and it has no residual add and no quantized output.

Forms

Three residual modes times five output sets: fifteen forms, each one compiled kernel that serves every row count and every layer. add, bf16 and quant are required keywords; the forms are peers and none is a default.

residual modeargumentseffect
addadd=Truev = x + residual; residual is updated in place with bf16(v)
storeadd=False, store_residual=Truev = x; residual is written with x (the first layer’s residual)
noneadd=Falsev = x; the call takes no residual (None)
output setargumentsthe call returns (out, codes, scales)
bf16bf16=True, quant=Noneout bf16 [rows, hidden]; None; None
bf16 + E4M3bf16=True, quant="e4m3_static"out; codes float8_e4m3fn [rows, hidden]; None
E4M3bf16=False, quant="e4m3_static"None; codes; None
bf16 + NVFP4bf16=True, quant="nvfp4"out; codes uint8 [rows, hidden / 2]; scales float8_e4m3fn [ceil(rows / 128) * 128, hidden / 16]
NVFP4bf16=False, quant="nvfp4"None; codes; scales

The returned tuple has the same three slots for every form, with None in a slot the form does not produce. quant is one vocabulary (None and cudnn.norm.frost.kernel.add_norm_quant_decode_config.QUANT_FORMS). A form that writes no output (bf16=False, quant=None) and add together with store_residual are declined at construction.

NVFP4 codes are two per byte with the even element in the low nibble. NVFP4 block scales are stored in the 128 x 4 scale layout over (rows, blocks): tiles of 128 rows, one after the other; the scale buffer therefore has ceil(rows / 128) * 128 rows.

Numerics contract

The sum tree and the rounding chain are the contract, not a tolerance. Per row, in fp32 on the bf16 operands:

  1. v = x + residual: one fp32 addition per element (under add). The updated residual is bf16(v), nearest-even.
  2. The sum of squares is ONE fixed tree whose shape depends on hidden alone: a thread owns eight contiguous elements and folds them with one fused multiply-add each, in element order, from +0; the 32 lanes of a warp are summed by a butterfly with offsets 1, 2, 4, 8, 16; the warps (20 at hidden 5120) by the same butterfly.
  3. inv = rsqrt( total / hidden + eps ): an IEEE division by the width, one addition, the part’s approximate reciprocal square root (rsqrt.approx.f32).
  4. y = bf16( (v * inv) * (1 + weight) ): 1 + weight is one fp32 addition, then two multiplications and ONE rounding to bf16 (nearest-even). This bf16 value is the normalized value of record: out stores it and the codes are formed from it, so the codes are those of quantizing out itself.
  5. quant="e4m3_static": code = e4m3( y * (1 / scale) ) with the correctly rounded fp32 reciprocal (div.rn.f32(1, scale)), nearest-even, saturating at +-448 (codes 0x7E / 0xFE). A consumer reads code * scale.
  6. quant="nvfp4": per block of 16 elements, the block scale is e4m3( scale * (max|y| * fp32(1/6)) ) (the stored byte; two fp32 multiplications and one nearest-even rounding). The codes are e2m1( y * q ) (nearest-even, saturating at +-6) with q = rcp( block_scale * rcp(scale) ), where rcp is the part’s approximate reciprocal (rcp.approx.ftz.f32); a block whose scale byte is zero writes the codes of the signed zero. A consumer reads code * block_scale / scale.

What changes bytes: hidden (the tree and the divisor), eps, add (the function that is normalized), quant (which codes exist) and the operand values; scale moves codes and block scales only. What never changes a byte: the row count, a row’s position, the other rows, bf16, store_residual, pdl, pdl_release, and who owns an output. A row alone writes the bytes it writes inside a call of up to 1024 rows. A block scale’s POSITION in the scale buffer follows the row index (the layout); its value does not.

Against an fp64 evaluation of the bf16 operands, every out element is one of the two bf16 values that bracket the exact normalized value (the kernel’s fp32 value is within 20 x 2^-24 relative of it), and on random operands the count of elements that are not the nearest of the two is bounded from the same arithmetic (a share of at most 2 x 20 x 2^-24 x 2^8, asserted with a binomial slack). The updated residual is bitwise bf16(fp32(x) + fp32(residual)). The E4M3 codes and the NVFP4 block scales are bitwise the host models above applied to the kernel’s own out. Every NVFP4 code is one of at most two adjacent codes around the exact quotient y * scale / block_scale (the bound of the two approximate reciprocals), and one code wherever the quotient is not within that bound of an E2M1 rounding midpoint.

scale is by contract a finite positive fp32 in 2**-100 .. 2**100. Its value is never read on the host: outside that range the reciprocal chains overflow or flush. A NaN or infinite element makes its own row’s out non-finite and leaves that row’s codes unspecified; every other row is unaffected. A row whose squares underflow normalizes against eps; a row whose sum of squares overflows fp32 normalizes to zeros.

The bytes follow the part’s rsqrt.approx and rcp.approx.ftz units. Equal bytes across GPU architectures are not part of the contract; in particular an NVFP4 code at an exact rounding tie under a block scale that is not a power of two follows the approximate reciprocal (see determinism).

Usage

import torch
import cudnn
dev = torch.device("cuda", 0)
norm = cudnn.AddNormQuantDecode(dev, 5120, 1e-6, add=True, bf16=True, quant="e4m3_static") # one entry per form
norm.compile(rows) # plan time: typed declines, DSL gates, the compile, outputs(rows)
out, codes, _ = norm(x, residual, weight, scale) # residual <- bf16(x + residual); out bf16; codes E4M3
out, codes, _ = norm(x, residual, weight, scale, out=my_out, codes=my_codes) # or into caller-owned buffers
first = cudnn.AddNormQuantDecode(dev, 5120, 1e-6, add=False, store_residual=True, bf16=False, quant="e4m3_static")
_, codes, _ = first(x, residual, weight, scale) # residual <- x
mlp = cudnn.AddNormQuantDecode(dev, 5120, 1e-6, add=True, bf16=False, quant="nvfp4")
_, codes4, scales = mlp(x, residual, weight, global_scale) # codes uint8 [rows, 2560]; scales E4M3, 128 x 4 layout

Everything behind eps is keyword-only. One compiled kernel serves every row count and every layer of a model: all tensors are launch arguments, the slabs are bound with a runtime extent and rows is a runtime argument, so no shape enters the compiled kernel, the compile key is the config alone, and a captured CUDA graph holds pointers, never values. compile() needs no operand. The launch goes to torch’s current stream of the entry’s device. That device need not be the caller’s current one: the compile and the launch run under the entry’s device, and the caller’s current device is restored afterwards. There is no workspace; a warm call allocates nothing and reads no device memory to the host.

operanddtypeshapenotes
xbf16[rows, hidden], rows 1 .. 1024read only
residualbf16[>= rows, hidden]required in the add and store modes, None otherwise; in / out (add), out (store)
weightbf16[hidden] or [1, hidden]the zero-centred norm weight; read before the wait under pdl
scalefp32one element (0-d or [1])required with a quant, None otherwise; read before the wait under pdl
out (optional)bf16[>= rows, hidden]only with bf16=True
codes (optional)float8_e4m3fn / uint8[>= rows, hidden] / [>= rows, hidden / 2]only with a quant
scales (optional)float8_e4m3fn[>= ceil(rows / 128) * 128, hidden / 16]only with quant="nvfp4"

Every operand is contiguous, starts at a 16-byte aligned address (scale: 4-byte) and lives on the entry’s device; no written operand may overlap another operand.

Output ownership is per output. An output the caller passes is the caller’s and is returned as given; rows past the call’s are never written. An output the caller does not pass is the entry’s: one set per row count holding only the form’s outputs, scales zero-filled, allocated by compile(rows) / outputs(rows) or at the first call of that row count. Call compile(rows) for every row count before a CUDA-graph capture. Entry-owned outputs are valid until the entry’s next call at that row count from ANY host thread: stream order does not serialize host calls, so two host threads that share one entry pass their own outputs or build one entry each. The per-row-count sets are never freed, so a caller that sweeps many row counts passes its own outputs. An entry-owned output is an operand of the launch like a passed one, so the overlap rule covers it: a call whose x, residual, weight, scale or passed output overlaps an entry-owned output that the call does not pass (a tensor an earlier call of that row count returned, fed back in) is declined with ValueError before the launch. To feed a returned output back in, pass a caller-owned output.

The kernel writes the live rows’ scale bytes only. A caller-owned scales buffer serves one row count, or its owner clears it when the row count shrinks: after a call with more rows, a call with fewer leaves the earlier call’s bytes in the rows between the new rows and ceil(rows / 128) * 128.

Everything the entry does not serve is declined by name: TypeError for a non-tensor, a dtype, a hidden that is not an int, a flag that is not a bool, or a rows of compile / outputs that is not an int; ValueError for a width outside the kernel’s structure (a multiple of 256 in 256 .. 8192), an eps that is not a finite positive Python float or int (the message names the type it got), add with store_residual, a quant outside the vocabulary, a form without an output, pdl_release=True without pdl=True, an operand the form requires and did not get or got and does not use, a shape, an alignment, a device mismatch, an overlap, or an empty call; NotImplementedError for a structurally valid width other than 5120, a row count above 1024, a non-contiguous operand, a compute capability outside 10.x, and the CuTe DSL gates (not installed, below 4.8.0, without the part’s target, or with a compiler that cannot assemble it). The operand checks import no DSL module and run before every compile and every launch; the architecture gate and the DSL gates run in compile(), before the kernel module is imported.

Launch knobs and the caller’s obligations

Both knobs default to off and neither changes a byte (the suite compares their bytes).

  • pdl=True: the launch carries the programmatic-stream-serialization attribute and the kernel issues griddepcontrol.wait AFTER it has loaded weight and scale and BEFORE it reads x / residual or writes anything. Obligation: weight and scale are not written by any launch that may still be running when this one starts (they are constants of the layer).
  • pdl_release (None follows pdl; True needs pdl=True): griddepcontrol.launch_dependents directly behind this kernel’s OWN wait; on this entry the release never precedes the wait. Obligation: every launch that can start before this one completes issues its own wait before it reads this launch’s outputs and before it writes any operand of this call. Whether the release helps depends on the row count (a released consumer occupies SMs); a caller that wants it for some row counts only builds two entries.

pdl=True alone is wait + release; pdl=True, pdl_release=False is wait only. The suite pins the wait’s position in the kernel’s text for every form and target: the loads of weight and scale are the only global accesses in front of it, the loads of x and residual and every store follow it, and the release is the next instruction. On a device it runs the step behind a predecessor that releases its dependents before it writes x and residual, the release with a dependent launch on the stream that waits and copies the outputs, and two steps chained through one buffer; each eager and captured.

Launch facts

The entry states what a row count launches without launching it: three read-only answers, with cfg_for’s argument.

factanswer
kernel_symbols_for(rows)the symbol of the entry’s compiled kernel as a one-element tuple; empty until the entry is compiled. One compiled kernel serves every row count. The symbol carries the form directly behind the kernel’s name — cudnn_kernel_frost_add_rms_norm_quant_h5120_add_bf16_e4m3_static_wait_release_eps358637bd_...: the width, the residual mode, the outputs, the launch arm and the fp32 bits of eps — so two forms whose kernels differ never share a symbol. The eps text is the fp32 value the kernel adds: two eps values that round to one fp32 value give two entries that compile the same kernel under one symbol
form_for(rows)the form as plain data: the eight fields of the entry’s config (HIDDEN, EPS, ADD, STORE_RESIDUAL, BF16_OUT, QUANT, PDL, PDL_RELEASE). The compile key is the config alone, so nothing else is part of it. A fresh dict on every call
launch_shape_for(rows)(grid, block, shared_memory): one thread block per row, hidden / 8 threads, one fp32 slot of shared memory per warp — ((3, 1, 1), (640, 1, 1), 80) for three rows. A profiler’s kernel record of the launch reads the same three values

A row count outside 1 .. 1024 is a typed decline (ValueError for one that is not a positive int, NotImplementedError past 1024), and none of the three compiles, launches or touches device memory.

Support matrix

questionanswer
devicecompute capability 10.x; validated on cc 10.0. The kernel’s rendering is pinned for the sm_100a, sm_103a and sm_107a targets (one PTX body for the three): cc 10.3 and cc 10.7 are admitted on that basis and have not run on a device, and any other 10.x minor is admitted by the range alone. 9.0 and 12.x are declined by type
rows1 .. 1024 per call (runtime argument); 1025 and up declined by name. Three thread blocks of 640 threads share an SM, so one scheduling wave holds 444 rows on a 148-SM part and 636 on a 212-SM part: 1024 rows cross it
hidden5120, fixed at construction; another multiple of 256 up to 8192 is declined by name (each width is its own sum tree)
formsthe fifteen forms above; three launch arms (plain, wait, wait + release)
dtypesbf16 x / residual / weight / out; one fp32 scale; E4M3 or packed E2M1 codes; E4M3 scales; fp32 arithmetic, rounded once to bf16
layoutscontiguous slabs; 16-byte aligned operands (scale: 4-byte); no written operand overlapping another
determinismbitwise replay and batch invariance (the status page)
DSLnvidia-cutlass-dsl >= 4.8.0. On cc 10.0 / 10.3 the CUDA 12.9 and CUDA 13.4 compiler flavours of 4.8.0 render the same PTX and are both served (on cc 10.0 both ran and wrote the same bytes); on cc 10.7 the compiler must be CUDA 13.4 or later (the CUDA 12.9 flavour rejects the sm_107a target and is declined by name before JIT)

Tests: test/python/norm/frost/test_add_norm_quant_decode.py (host declines, the config’s limits and tables, the rendering pinned per target and tied to the entry’s own launch arguments, the launch facts against the configuration arithmetic and against the profiler’s record of a launch, the device cells against the references above, exact and edge rows, rows whose sum of squares overflows, output ownership, the bit-neutral arms, the wait behind a predecessor that released early, the release beside a dependent launch, a chain of two steps, CUDA-graph replay, an entry on a device that is not the caller’s current one) and test_add_norm_quant_decode_determinism.py (replay, batch invariance, goldens). Source: python/cudnn/norm/frost/.