> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/cudnn/latest/operations/Normalizations/llms.txt. # Normalizations ## Batchnorm Forward The batchnorm operation computes: $$ output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias $$ Optionally the operation also computes: $$next\_running\_mean = (1 - momentum)*previous\_running\_mean + momentum*current\_running\_mean$$ $$next\_running\_variance = (1 - momentum)*previous\_running\_variance + momentum*current\_running\_variance$$ ### C++ API ``` std::array, 5> batchnorm(std::shared_ptr& input, std::shared_ptr& scale, std::shared_ptr& bias, Batchnorm_attributes attributes); ``` Where the output array has tensors in order of: `[output, saved_mean, saved_invariance, next_running_mean, next_running_variance]` Batchnorm attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes: ``` Batchnorm_attributes& set_previous_running_stats(std::shared_ptr& previous_running_mean, std::shared_ptr& previous_running_variance, std::shared_ptr& momentum) Batchnorm_attributes& set_name(std::string const&) Batchnorm_attributes& set_compute_data_type(DataType_t value) ``` ### Python API - batchnorm - input - scale - bias - in_running_mean - in_running_var - epsilon - momentum - peer_stats (optional, list) - compute_data_type - name `peer_stats` is an optional list of per-peer statistics tensors for synchronized batch normalization; the frontend forwards each tensor to cuDNN's peer-stat descriptor list (`CUDNN_ATTR_OPERATION_NORM_FWD_PEER_STAT_DESCS` for the forward, `CUDNN_ATTR_OPERATION_NORM_BWD_PEER_STAT_DESCS` for the backward). See the cuDNN API documentation for the expected statistics layout. ## Batchnorm Finalize The `bn_finalize` operation calculates the statistics required for the next iteration from the statistics generated by the `genstats` operation. ### C++ API ``` std::array, 6> bn_finalize(std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr, BN_finalize_attributes); ``` The outputs are `[EQ_SCALE, EQ_BIAS, MEAN, INV_VARIANCE, NEXT_RUNNING_MEAN, NEXT_RUNNING_VAR]`. ## Batchnorm Backward (DBN) The DBN operation computes data gradient, scale gradient, bias gradient during backpropagation of batchnorm forward operation. ### C++ API ``` std::array, 3> batchnorm_backward(std::shared_ptr loss, std::shared_ptr input, std::shared_ptr scale, Batchnorm_backward_attributes); ``` The output array has tensors in order of: `[input gradient, scale gradient, bias gradient]`. DBN attributes is a lightweight structure with setters: ``` Batchnorm_backward_attributes& set_saved_mean_and_inv_variance(std::shared_ptr saved_mean, std::shared_ptr saved_inverse_variance) Batchnorm_backward_attributes& set_epsilon(std::shared_ptr epsilon) Batchnorm_backward_attributes& set_name(std::string const&) Batchnorm_backward_attributes& set_compute_data_type(DataType_t value) ``` Only setting either (saved mean and inverse_variance) or (epsilon) is necessary. ### Python API - batchnorm_backward - grad - input - scale - mean - inv_variance - peer_stats (optional, list) - compute_data_type - name `peer_stats` behaves as in [Batchnorm Forward](#batchnorm-forward). ## Batchnorm Inference The batchnorm inference operation applies precomputed statistics (the forward phase is `NormFwdPhase_t::INFERENCE`): $$ Y = scale*(X - mean)*inv\_variance + bias $$ It computes no statistics and updates no running statistics. ### C++ API ``` std::shared_ptr batchnorm_inference(std::shared_ptr x, std::shared_ptr mean, std::shared_ptr inv_variance, std::shared_ptr scale, std::shared_ptr bias, Batchnorm_inference_attributes); ``` The return value is `Y`. When they are not set, Y's dims and strides are inferred from X. `mean`, `inv_variance`, `scale` and `bias` are per-channel tensors of shape `[1, C, 1, 1]`. Batchnorm inference attributes is a lightweight structure with setters: ``` Batchnorm_inference_attributes& set_name(std::string const&) Batchnorm_inference_attributes& set_compute_data_type(DataType_t value) ``` ### Python API - batchnorm_inference - input - mean - inv_variance - scale - bias - compute_data_type - name ### Supported graphs The backend engines serve this node inside the fused training pattern batchnorm inference -> `relu_backward` -> `batchnorm_backward`, which needs cuDNN 8.9.4 or newer. The examples are `samples/cpp/norm/batchnorm.cpp` ("BN_inference DRelu DBN Graph") and `test/python/norm/graph/test_batchnorm.py::test_bn_infer_drelu_dbn`. A graph that holds only batchnorm inference, or batchnorm inference followed by a forward `relu`, is declined with `cudnnGraphNotSupportedError`. This was measured with cuDNN 9.27 on SM90 and SM103, using fp16 channels-last X and fp32 per-channel tensors. ## Generate Stats The Genstats operation computes the sum and sum of squares per-channel dimension. ### C++ API ``` std::array, 2> cudnn_frontend::graph::genstats(std::shared_ptr, Genstats_attributes); ``` The output array has tensors in order of: `[sum, square_sum]` Genstats attributes is a lightweight structure with setters: ``` Genstats_attributes& set_name(std::string const&) Genstats_attributes& set_compute_data_type(DataType_t value) ``` ### Python API - genstats - input - compute_data_type - name ## Layernorm Forward The layernorm operation computes $$ output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias $$ Normalization happens across features, independently for each sample. ### C++ API ``` std::array, 3> layernorm(std::shared_ptr& input, std::shared_ptr& scale, std::shared_ptr& bias, Layernorm_attributes attributes); ``` The output array has tensors in order of: `[output, mean, variance]` Layernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes: ``` Layernorm_attributes& set_name(std::string const&) Layernorm_attributes& set_compute_data_type(DataType_t value) ``` `scale` and `bias` are optional; pass `nullptr` (Python: `None`). Verified on Hopper with cuDNN 9.12, 9.14, 9.18, 9.20, 9.24 and 9.27, and on Ampere and Blackwell with 9.27; older backends were not qualified. - **Scale without bias** (`nn.LayerNorm(bias=False)`) works for any shape. As with a bias, the training `mean` and `variance` dims are `input`'s dims with 1 wherever `scale` is not 1. - **Bias without scale** is refused with `INVALID_VALUE`, because the backend rejects it. Pass a scale of ones. - **Neither:** nothing states the normalization axes, so the same rule as [RMSNorm without a scale](#rmsnorm-graph-api) applies. `input` may have at most one non-unit dimension after the first, or, in training, the caller sets the `mean` or `variance` dims. In training, a `mean` or `variance` whose dims are not set takes the dims set on the other one; `validate()` refuses the two with different dims. ### Python API - layernorm - norm_forward_phase - input - scale (optional) - bias (optional) - epsilon - compute_data_type - name ### Experimental PyTorch API `cudnn.experimental.ops.layer_norm(input, normalized_shape, weight=None, bias=None, eps=1e-5)` provides a graph-backed PyTorch custom op with autograd and `torch.compile` support. It currently supports normalization over one final dimension on nonempty CUDA tensors with FP16, BF16, or FP32 activations. ## Layernorm Backward (DLN) The DLN operation computes data gradient, scale gradient, and bias gradient during backpropagation of a layernorm forward operation. ### C++ API ``` std::array, 3> layernorm_backward(std::shared_ptr dy, std::shared_ptr x, std::shared_ptr scale, Layernorm_backward_attributes options); ``` The output array has tensors in order of: `[input gradient, scale gradient, bias gradient]`. `scale` is optional (Python: `scale=None`). Without it there is no scale or bias gradient (both returned as `nullptr` / `None`). The saved `mean` and `inv_variance` dims state the normalization axes, as in the forward. Verified on Hopper with cuDNN 9.12, 9.14, 9.18, 9.20, 9.24 and 9.27, and on Ampere and Blackwell with 9.27; older backends were not qualified. Layernorm_backward_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes: ``` Layernorm_backward_attributes& set_name(std::string const&) Layernorm_backward_attributes& set_compute_data_type(DataType_t value) ``` ### Python API - layernorm - input - scale - loss - compute_data_type - name ## RMSNorm Graph API `Graph::rmsnorm(x, scale, attributes)` (Python: `graph.rmsnorm(norm_forward_phase, input, scale=None, bias=None, epsilon, adaptive=False)`) normalizes `x` by its root mean square and multiplies by `scale`. Training also returns `INV_VARIANCE`. Its dims are `x`'s dims, with 1 on every axis where `scale` is not 1. Those are the normalization axes. `scale` is optional. This was verified on Hopper with cuDNN 9.20, 9.24 and 9.27, and on Ampere and Blackwell with 9.27; older backends were not qualified. Without a scale, nothing states the normalization axes, so the node accepts only these cases: - `x` has at most one non-unit dimension after the first, e.g. `{rows, hidden, 1, 1}`. That dimension is normalized. - In training, the caller sets the `INV_VARIANCE` dims explicitly, e.g. `{B, S, 1}` for `x = {B, S, H}`. Any other shape is refused with `INVALID_VALUE`. Pass a scale of ones, or reshape, to get the axes you want. `Graph::rmsnorm_backward(dy, x, scale, inv_variance, attributes)` also takes an optional `scale` (Python: `scale=None`). Without it there is no `DSCALE` or `DBIAS`, so `has_dbias(true)` is refused. The `INV_VARIANCE` dims state the normalization axes. Verified on Hopper with cuDNN 9.12, 9.14, 9.18, 9.20, 9.24 and 9.27, and on Ampere and Blackwell with 9.27; older backends were not qualified. ## RMSNorm PyTorch API `cudnn.experimental.ops.rms_norm(input, weight, bias=None, eps=1e-5)` applies RMS normalization over the final input dimension using a cuDNN graph and supports autograd and `torch.compile`. This experimental signature intentionally includes an optional additive bias and requires a weight tensor. It is therefore not a drop-in replacement for `torch.nn.functional.rms_norm`, whose signature includes `normalized_shape` and no bias argument. Inputs, weights, and optional bias must be CUDA tensors with matching FP16, BF16, or FP32 dtypes, and the input must contain at least one normalization row. ## RMSNorm + SiLU + Padding The experimental direct Python API `RmsNormSiluPadSm100` combines optional bias/residual addition, normalization, SiLU, padding, and history/cache copies for C160/C320/C640 BF16 tensors. It does not perform convolution. See [fused Conv3D + post-operations](/cudnn/operations/Convolutions#cute-dsl-fused-conv3d--post-operations) for requirements and the [API reference](/operations/Convolutions/conv3d_postops) for the exact normalization and tensor-layout contracts. ## Adaptive Layernorm Forward The adaptive layernorm operation computes $$ output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias $$ where the scale and bias vary across samples in a batch. Normalization happens across features, independently for each sample. ### C++ API ``` std::array, 3> adalayernorm(std::shared_ptr& input, std::shared_ptr& scale, std::shared_ptr& bias, AdaLayernorm_attributes attributes); ``` The output array has tensors in order of: `[output, mean, variance]` AdaLayernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes: ``` AdaLayernorm_attributes& set_name(std::string const&) AdaLayernorm_attributes& set_compute_data_type(DataType_t value) ``` `bias` is optional; pass `nullptr` (Python: `None`, the default). `scale` is required. ### Python API - adalayernorm - norm_forward_phase - input - scale - bias (optional) - epsilon - compute_data_type - name ## Adaptive Layernorm Backward (DADALN) The DADALN operation computes data gradient, scale gradient, and bias gradient during backpropagation of an adaptive layernorm forward operation. ### C++ API ``` std::array, 3> adalayernorm_backward(std::shared_ptr dy, std::shared_ptr x, std::shared_ptr scale, AdaLayernorm_backward_attributes options); ``` The output array has tensors in order of: `[input gradient, scale gradient, bias gradient]`. AdaLayernorm_backward_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes: ``` AdaLayernorm_backward_attributes& set_name(std::string const&) AdaLayernorm_backward_attributes& set_compute_data_type(DataType_t value) ``` ### Python API - adalayernorm_backward - grad - input - scale - mean - inv_variance - compute_data_type - name ## Instancenorm Forward The instancenorm operation computes $$ output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias $$ Normalization happens across each sample. ### C++ API ``` std::array, 3> instancenorm(std::shared_ptr& input, std::shared_ptr& scale, std::shared_ptr& bias, Instancenorm_attributes attributes); ``` The output array has tensors in order of: `[output, mean, variance]`. Instancenorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes: ``` Instancenorm_attributes& set_name(std::string const&) Instancenorm_attributes& set_compute_data_type(DataType_t value) ``` ### Python API - instancenorm - norm_forward_phase - adaptive (optional, bool) - input - scale - bias - epsilon - compute_data_type - name `adaptive` is a Python-only flag for adaptive InstanceNorm. The input must be channels-first contiguous (`[N, C, L]`, `[N, C, H, W]` or `[N, C, D, H, W]`), and `scale`/optional `bias` must be per sample and channel with singleton spatial dimensions -- `[N, C, 1]`, `[N, C, 1, 1]`, `[N, C, 1, 1, 1]`; the statistics have that same shape and the reduction runs over the spatial axes. Adaptive InstanceNorm is served by Python engines only (`adaptive=False` keeps backend eligibility). For the adaptive LayerNorm/RMSNorm layouts (`[B, S, D]` / `[S, B, D]` with `[B, 1, D]` / `[1, B, D]` scale) see the norm guide (`python/cudnn/norm/README.md`). ## Instancenorm Backward (DIN) The DIN operation computes data gradient, scale gradient, and bias gradient during backpropagation of an instancenorm forward operation. ### C++ API ``` std::array, 3> instancenorm_backward(std::shared_ptr dy, std::shared_ptr x, std::shared_ptr scale, Instancenorm_backward_attributes options); ``` The output array has tensors in order of: `[input gradient, scale gradient, bias gradient]`. Instancenorm_backward_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes: ``` Instancenorm_backward_attributes& set_saved_mean_and_inv_variance(std::shared_ptr mean, std::shared_ptr inv_variance) Instancenorm_backward_attributes& set_name(std::string const&) Instancenorm_backward_attributes& set_compute_data_type(DataType_t value) ``` ### Python API - instancenorm_backward - grad - input - scale - mean - inv_variance - adaptive (optional, bool) - compute_data_type - name ## Docs - [LayerNorm and RMSNorm in JAX](https://docs.nvidia.com/cudnn/latest/operations/Normalizations/norm_jax.md) - [RMSNorm + RHT + Amax (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Normalizations/rmsnorm_rht_amax.md) - [Fused RMSNorm + SiLU](https://docs.nvidia.com/cudnn/latest/operations/Normalizations/rmsnorm_silu.md) - [Add + RMSNorm + Quantize Decode Step (SM100-class)](https://docs.nvidia.com/cudnn/latest/operations/Normalizations/add_norm_quant_decode.md)