Skip to navigation

Normalizations

View as MarkdownOpen in Claude

Batchnorm Forward

The batchnorm operation computes:

output=scale∗input−meanvariance+epsilon+biasoutput = scale*{input - mean \over \sqrt{variance + epsilon}} + bias

Optionally the operation also computes:

next_running_mean=(1−momentum)∗previous_running_mean+momentum∗current_running_meannext\_running\_mean = (1 - momentum)*previous\_running\_mean + momentum*current\_running\_mean

next_running_variance=(1−momentum)∗previous_running_variance+momentum∗current_running_variancenext\_running\_variance = (1 - momentum)*previous\_running\_variance + momentum*current\_running\_variance

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 5> batchnorm(std::shared_ptr<Tensor_attributes>& input,
std::shared_ptr<Tensor_attributes>& scale,
std::shared_ptr<Tensor_attributes>& bias,
Batchnorm_attributes attributes);

Where the output array has tensors in order of: [output, saved_mean, saved_invariance, next_running_mean, next_running_variance]

Batchnorm attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Batchnorm_attributes&
set_previous_running_stats(std::shared_ptr<Tensor_attributes>& previous_running_mean,
std::shared_ptr<Tensor_attributes>& previous_running_variance,
std::shared_ptr<Tensor_attributes>& momentum)
Batchnorm_attributes&
set_name(std::string const&)
Batchnorm_attributes&
set_compute_data_type(DataType_t value)

Python API

  • batchnorm
    • input
    • scale
    • bias
    • in_running_mean
    • in_running_var
    • epsilon
    • momentum
    • peer_stats (optional, list)
    • compute_data_type
    • name

peer_stats is an optional list of per-peer statistics tensors for synchronized batch normalization; the frontend forwards each tensor to cuDNN’s peer-stat descriptor list (CUDNN_ATTR_OPERATION_NORM_FWD_PEER_STAT_DESCS for the forward, CUDNN_ATTR_OPERATION_NORM_BWD_PEER_STAT_DESCS for the backward). See the cuDNN API documentation for the expected statistics layout.

Batchnorm Finalize

The bn_finalize operation calculates the statistics required for the next iteration from the statistics generated by the genstats operation.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 6> bn_finalize(std::shared_ptr<Tensor_attributes>,
std::shared_ptr<Tensor_attributes>,
std::shared_ptr<Tensor_attributes>,
std::shared_ptr<Tensor_attributes>,
std::shared_ptr<Tensor_attributes>,
std::shared_ptr<Tensor_attributes>,
BN_finalize_attributes);

The outputs are [EQ_SCALE, EQ_BIAS, MEAN, INV_VARIANCE, NEXT_RUNNING_MEAN, NEXT_RUNNING_VAR].

Batchnorm Backward (DBN)

The DBN operation computes data gradient, scale gradient, bias gradient during backpropagation of batchnorm forward operation.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 3> batchnorm_backward(std::shared_ptr<Tensor_attributes> loss,
std::shared_ptr<Tensor_attributes> input,
std::shared_ptr<Tensor_attributes> scale,
Batchnorm_backward_attributes);

The output array has tensors in order of: [input gradient, scale gradient, bias gradient].

DBN attributes is a lightweight structure with setters:

Batchnorm_backward_attributes&
set_saved_mean_and_inv_variance(std::shared_ptr<Tensor_attributes> saved_mean,
std::shared_ptr<Tensor_attributes> saved_inverse_variance)
Batchnorm_backward_attributes&
set_epsilon(std::shared_ptr<Tensor_attributes> epsilon)
Batchnorm_backward_attributes&
set_name(std::string const&)
Batchnorm_backward_attributes&
set_compute_data_type(DataType_t value)

Only setting either (saved mean and inverse_variance) or (epsilon) is necessary.

Python API

  • batchnorm_backward
    • grad
    • input
    • scale
    • mean
    • inv_variance
    • peer_stats (optional, list)
    • compute_data_type
    • name

peer_stats behaves as in Batchnorm Forward.

Batchnorm Inference

The batchnorm inference operation applies precomputed statistics (the forward phase is NormFwdPhase_t::INFERENCE):

Y=scale∗(X−mean)∗inv_variance+biasY = scale*(X - mean)*inv\_variance + bias

It computes no statistics and updates no running statistics.

C++ API

std::shared_ptr<Tensor_attributes> batchnorm_inference(std::shared_ptr<Tensor_attributes> x,
std::shared_ptr<Tensor_attributes> mean,
std::shared_ptr<Tensor_attributes> inv_variance,
std::shared_ptr<Tensor_attributes> scale,
std::shared_ptr<Tensor_attributes> bias,
Batchnorm_inference_attributes);

The return value is Y. When they are not set, Y’s dims and strides are inferred from X. mean, inv_variance, scale and bias are per-channel tensors of shape [1, C, 1, 1].

Batchnorm inference attributes is a lightweight structure with setters:

Batchnorm_inference_attributes&
set_name(std::string const&)
Batchnorm_inference_attributes&
set_compute_data_type(DataType_t value)

Python API

  • batchnorm_inference
    • input
    • mean
    • inv_variance
    • scale
    • bias
    • compute_data_type
    • name

Supported graphs

The backend engines serve this node inside the fused training pattern batchnorm inference -> relu_backward -> batchnorm_backward, which needs cuDNN 8.9.4 or newer. The examples are samples/cpp/norm/batchnorm.cpp (“BN_inference DRelu DBN Graph”) and test/python/norm/graph/test_batchnorm.py::test_bn_infer_drelu_dbn.

A graph that holds only batchnorm inference, or batchnorm inference followed by a forward relu, is declined with cudnnGraphNotSupportedError. This was measured with cuDNN 9.27 on SM90 and SM103, using fp16 channels-last X and fp32 per-channel tensors.

Generate Stats

The Genstats operation computes the sum and sum of squares per-channel dimension.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 2>
cudnn_frontend::graph::genstats(std::shared_ptr<Tensor_attributes>, Genstats_attributes);

The output array has tensors in order of: [sum, square_sum]

Genstats attributes is a lightweight structure with setters:

Genstats_attributes&
set_name(std::string const&)
Genstats_attributes&
set_compute_data_type(DataType_t value)

Python API

  • genstats
    • input
    • compute_data_type
    • name

Layernorm Forward

The layernorm operation computes

output=scale∗input−meanvariance+epsilon+biasoutput = scale*{input - mean \over \sqrt{variance + epsilon}} + bias

Normalization happens across features, independently for each sample.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 3> layernorm(std::shared_ptr<Tensor_attributes>& input,
std::shared_ptr<Tensor_attributes>& scale,
std::shared_ptr<Tensor_attributes>& bias,
Layernorm_attributes attributes);

The output array has tensors in order of: [output, mean, variance]

Layernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Layernorm_attributes&
set_name(std::string const&)
Layernorm_attributes&
set_compute_data_type(DataType_t value)

scale and bias are optional; pass nullptr (Python: None). Verified on Hopper with cuDNN 9.12, 9.14, 9.18, 9.20, 9.24 and 9.27, and on Ampere and Blackwell with 9.27; older backends were not qualified.

  • Scale without bias (nn.LayerNorm(bias=False)) works for any shape. As with a bias, the training mean and variance dims are input’s dims with 1 wherever scale is not 1.
  • Bias without scale is refused with INVALID_VALUE, because the backend rejects it. Pass a scale of ones.
  • Neither: nothing states the normalization axes, so the same rule as RMSNorm without a scale applies. input may have at most one non-unit dimension after the first, or, in training, the caller sets the mean or variance dims.

In training, a mean or variance whose dims are not set takes the dims set on the other one; validate() refuses the two with different dims.

Python API

  • layernorm
    • norm_forward_phase
    • input
    • scale (optional)
    • bias (optional)
    • epsilon
    • compute_data_type
    • name

Experimental PyTorch API

cudnn.experimental.ops.layer_norm(input, normalized_shape, weight=None, bias=None, eps=1e-5) provides a graph-backed PyTorch custom op with autograd and torch.compile support. It currently supports normalization over one final dimension on nonempty CUDA tensors with FP16, BF16, or FP32 activations.

Layernorm Backward (DLN)

The DLN operation computes data gradient, scale gradient, and bias gradient during backpropagation of a layernorm forward operation.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 3>
layernorm_backward(std::shared_ptr<Tensor_attributes> dy,
std::shared_ptr<Tensor_attributes> x,
std::shared_ptr<Tensor_attributes> scale,
Layernorm_backward_attributes options);

The output array has tensors in order of: [input gradient, scale gradient, bias gradient].

scale is optional (Python: scale=None). Without it there is no scale or bias gradient (both returned as nullptr / None). The saved mean and inv_variance dims state the normalization axes, as in the forward. Verified on Hopper with cuDNN 9.12, 9.14, 9.18, 9.20, 9.24 and 9.27, and on Ampere and Blackwell with 9.27; older backends were not qualified.

Layernorm_backward_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Layernorm_backward_attributes&
set_name(std::string const&)
Layernorm_backward_attributes&
set_compute_data_type(DataType_t value)

Python API

  • layernorm
    • input
    • scale
    • loss
    • compute_data_type
    • name

RMSNorm Graph API

Graph::rmsnorm(x, scale, attributes) (Python: graph.rmsnorm(norm_forward_phase, input, scale=None, bias=None, epsilon, adaptive=False)) normalizes x by its root mean square and multiplies by scale. Training also returns INV_VARIANCE. Its dims are x’s dims, with 1 on every axis where scale is not 1. Those are the normalization axes.

scale is optional. This was verified on Hopper with cuDNN 9.20, 9.24 and 9.27, and on Ampere and Blackwell with 9.27; older backends were not qualified. Without a scale, nothing states the normalization axes, so the node accepts only these cases:

  • x has at most one non-unit dimension after the first, e.g. {rows, hidden, 1, 1}. That dimension is normalized.
  • In training, the caller sets the INV_VARIANCE dims explicitly, e.g. {B, S, 1} for x = {B, S, H}.

Any other shape is refused with INVALID_VALUE. Pass a scale of ones, or reshape, to get the axes you want.

Graph::rmsnorm_backward(dy, x, scale, inv_variance, attributes) also takes an optional scale (Python: scale=None). Without it there is no DSCALE or DBIAS, so has_dbias(true) is refused. The INV_VARIANCE dims state the normalization axes. Verified on Hopper with cuDNN 9.12, 9.14, 9.18, 9.20, 9.24 and 9.27, and on Ampere and Blackwell with 9.27; older backends were not qualified.

RMSNorm PyTorch API

cudnn.experimental.ops.rms_norm(input, weight, bias=None, eps=1e-5) applies RMS normalization over the final input dimension using a cuDNN graph and supports autograd and torch.compile.

This experimental signature intentionally includes an optional additive bias and requires a weight tensor. It is therefore not a drop-in replacement for torch.nn.functional.rms_norm, whose signature includes normalized_shape and no bias argument. Inputs, weights, and optional bias must be CUDA tensors with matching FP16, BF16, or FP32 dtypes, and the input must contain at least one normalization row.

RMSNorm + SiLU + Padding

The experimental direct Python API RmsNormSiluPadSm100 combines optional bias/residual addition, normalization, SiLU, padding, and history/cache copies for C160/C320/C640 BF16 tensors. It does not perform convolution. See fused Conv3D + post-operations for requirements and the API reference for the exact normalization and tensor-layout contracts.

Adaptive Layernorm Forward

The adaptive layernorm operation computes

output=scale∗input−meanvariance+epsilon+biasoutput = scale*{input - mean \over \sqrt{variance + epsilon}} + bias

where the scale and bias vary across samples in a batch.

Normalization happens across features, independently for each sample.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 3> adalayernorm(std::shared_ptr<Tensor_attributes>& input,
std::shared_ptr<Tensor_attributes>& scale,
std::shared_ptr<Tensor_attributes>& bias,
AdaLayernorm_attributes attributes);

The output array has tensors in order of: [output, mean, variance]

AdaLayernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

AdaLayernorm_attributes&
set_name(std::string const&)
AdaLayernorm_attributes&
set_compute_data_type(DataType_t value)

bias is optional; pass nullptr (Python: None, the default). scale is required.

Python API

  • adalayernorm
    • norm_forward_phase
    • input
    • scale
    • bias (optional)
    • epsilon
    • compute_data_type
    • name

Adaptive Layernorm Backward (DADALN)

The DADALN operation computes data gradient, scale gradient, and bias gradient during backpropagation of an adaptive layernorm forward operation.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 3>
adalayernorm_backward(std::shared_ptr<Tensor_attributes> dy,
std::shared_ptr<Tensor_attributes> x,
std::shared_ptr<Tensor_attributes> scale,
AdaLayernorm_backward_attributes options);

The output array has tensors in order of: [input gradient, scale gradient, bias gradient].

AdaLayernorm_backward_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

AdaLayernorm_backward_attributes&
set_name(std::string const&)
AdaLayernorm_backward_attributes&
set_compute_data_type(DataType_t value)

Python API

  • adalayernorm_backward
    • grad
    • input
    • scale
    • mean
    • inv_variance
    • compute_data_type
    • name

Instancenorm Forward

The instancenorm operation computes

output=scale∗input−meanvariance+epsilon+biasoutput = scale*{input - mean \over \sqrt{variance + epsilon}} + bias

Normalization happens across each sample.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 3> instancenorm(std::shared_ptr<Tensor_attributes>& input,
std::shared_ptr<Tensor_attributes>& scale,
std::shared_ptr<Tensor_attributes>& bias,
Instancenorm_attributes attributes);

The output array has tensors in order of: [output, mean, variance].

Instancenorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Instancenorm_attributes&
set_name(std::string const&)
Instancenorm_attributes&
set_compute_data_type(DataType_t value)

Python API

  • instancenorm
    • norm_forward_phase
    • adaptive (optional, bool)
    • input
    • scale
    • bias
    • epsilon
    • compute_data_type
    • name

adaptive is a Python-only flag for adaptive InstanceNorm. The input must be channels-first contiguous ([N, C, L], [N, C, H, W] or [N, C, D, H, W]), and scale/optional bias must be per sample and channel with singleton spatial dimensions — [N, C, 1], [N, C, 1, 1], [N, C, 1, 1, 1]; the statistics have that same shape and the reduction runs over the spatial axes. Adaptive InstanceNorm is served by Python engines only (adaptive=False keeps backend eligibility). For the adaptive LayerNorm/RMSNorm layouts ([B, S, D] / [S, B, D] with [B, 1, D] / [1, B, D] scale) see the norm guide (python/cudnn/norm/README.md).

Instancenorm Backward (DIN)

The DIN operation computes data gradient, scale gradient, and bias gradient during backpropagation of an instancenorm forward operation.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 3>
instancenorm_backward(std::shared_ptr<Tensor_attributes> dy,
std::shared_ptr<Tensor_attributes> x,
std::shared_ptr<Tensor_attributes> scale,
Instancenorm_backward_attributes options);

The output array has tensors in order of: [input gradient, scale gradient, bias gradient].

Instancenorm_backward_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Instancenorm_backward_attributes&
set_saved_mean_and_inv_variance(std::shared_ptr<Tensor_attributes> mean,
std::shared_ptr<Tensor_attributes> inv_variance)
Instancenorm_backward_attributes&
set_name(std::string const&)
Instancenorm_backward_attributes&
set_compute_data_type(DataType_t value)

Python API

  • instancenorm_backward
    • grad
    • input
    • scale
    • mean
    • inv_variance
    • adaptive (optional, bool)
    • compute_data_type
    • name