Normalizations
Batchnorm Forward
The batchnorm operation computes:
Optionally the operation also computes:
C++ API
Where the output array has tensors in order of: [output, saved_mean, saved_invariance, next_running_mean, next_running_variance]
Batchnorm attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:
Python API
- batchnorm
- input
- scale
- bias
- in_running_mean
- in_running_var
- epsilon
- momentum
- peer_stats (optional, list)
- compute_data_type
- name
peer_stats is an optional list of per-peer statistics tensors for synchronized batch normalization; the frontend
forwards each tensor to cuDNN’s peer-stat descriptor list (CUDNN_ATTR_OPERATION_NORM_FWD_PEER_STAT_DESCS for the
forward, CUDNN_ATTR_OPERATION_NORM_BWD_PEER_STAT_DESCS for the backward). See the cuDNN API documentation for the
expected statistics layout.
Batchnorm Finalize
The bn_finalize operation calculates the statistics required for the next iteration from the statistics generated by the genstats operation.
C++ API
The outputs are [EQ_SCALE, EQ_BIAS, MEAN, INV_VARIANCE, NEXT_RUNNING_MEAN, NEXT_RUNNING_VAR].
Batchnorm Backward (DBN)
The DBN operation computes data gradient, scale gradient, bias gradient during backpropagation of batchnorm forward operation.
C++ API
The output array has tensors in order of: [input gradient, scale gradient, bias gradient].
DBN attributes is a lightweight structure with setters:
Only setting either (saved mean and inverse_variance) or (epsilon) is necessary.
Python API
- batchnorm_backward
- grad
- input
- scale
- mean
- inv_variance
- peer_stats (optional, list)
- compute_data_type
- name
peer_stats behaves as in Batchnorm Forward.
Batchnorm Inference
The batchnorm inference operation applies precomputed statistics (the forward phase is NormFwdPhase_t::INFERENCE):
It computes no statistics and updates no running statistics.
C++ API
The return value is Y. When they are not set, Y’s dims and strides are inferred from X. mean, inv_variance, scale and bias are per-channel tensors of shape [1, C, 1, 1].
Batchnorm inference attributes is a lightweight structure with setters:
Python API
- batchnorm_inference
- input
- mean
- inv_variance
- scale
- bias
- compute_data_type
- name
Supported graphs
The backend engines serve this node inside the fused training pattern batchnorm inference -> relu_backward -> batchnorm_backward, which needs cuDNN 8.9.4 or newer. The examples are samples/cpp/norm/batchnorm.cpp (“BN_inference DRelu DBN Graph”) and test/python/norm/graph/test_batchnorm.py::test_bn_infer_drelu_dbn.
A graph that holds only batchnorm inference, or batchnorm inference followed by a forward relu, is declined with cudnnGraphNotSupportedError. This was measured with cuDNN 9.27 on SM90 and SM103, using fp16 channels-last X and fp32 per-channel tensors.
Generate Stats
The Genstats operation computes the sum and sum of squares per-channel dimension.
C++ API
The output array has tensors in order of: [sum, square_sum]
Genstats attributes is a lightweight structure with setters:
Python API
- genstats
- input
- compute_data_type
- name
Layernorm Forward
The layernorm operation computes
Normalization happens across features, independently for each sample.
C++ API
The output array has tensors in order of: [output, mean, variance]
Layernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:
scale and bias are optional; pass nullptr (Python: None). Verified on Hopper with cuDNN 9.12, 9.14, 9.18, 9.20, 9.24 and
9.27, and on Ampere and Blackwell with 9.27; older backends were not qualified.
- Scale without bias (
nn.LayerNorm(bias=False)) works for any shape. As with a bias, the trainingmeanandvariancedims areinput’s dims with 1 whereverscaleis not 1. - Bias without scale is refused with
INVALID_VALUE, because the backend rejects it. Pass a scale of ones. - Neither: nothing states the normalization axes, so the same rule as RMSNorm without a scale
applies.
inputmay have at most one non-unit dimension after the first, or, in training, the caller sets themeanorvariancedims.
In training, a mean or variance whose dims are not set takes the dims set on the other one; validate() refuses
the two with different dims.
Python API
- layernorm
- norm_forward_phase
- input
- scale (optional)
- bias (optional)
- epsilon
- compute_data_type
- name
Experimental PyTorch API
cudnn.experimental.ops.layer_norm(input, normalized_shape, weight=None, bias=None, eps=1e-5) provides a graph-backed PyTorch custom op with autograd
and torch.compile support. It currently supports normalization over one final
dimension on nonempty CUDA tensors with FP16, BF16, or FP32 activations.
Layernorm Backward (DLN)
The DLN operation computes data gradient, scale gradient, and bias gradient during backpropagation of a layernorm forward operation.
C++ API
The output array has tensors in order of: [input gradient, scale gradient, bias gradient].
scale is optional (Python: scale=None). Without it there is no scale or bias gradient (both returned as nullptr /
None). The saved mean and inv_variance dims state the normalization axes, as in the forward. Verified on Hopper with cuDNN 9.12, 9.14, 9.18, 9.20, 9.24 and 9.27, and on
Ampere and Blackwell with 9.27; older backends were not qualified.
Layernorm_backward_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:
Python API
- layernorm
- input
- scale
- loss
- compute_data_type
- name
RMSNorm Graph API
Graph::rmsnorm(x, scale, attributes) (Python: graph.rmsnorm(norm_forward_phase, input, scale=None, bias=None, epsilon, adaptive=False))
normalizes x by its root mean square and multiplies by scale. Training also returns INV_VARIANCE. Its dims are x’s
dims, with 1 on every axis where scale is not 1. Those are the normalization axes.
scale is optional. This was verified on Hopper with cuDNN 9.20, 9.24 and 9.27, and on Ampere and Blackwell with 9.27;
older backends were not qualified. Without a scale, nothing states the normalization axes, so the node accepts only these cases:
xhas at most one non-unit dimension after the first, e.g.{rows, hidden, 1, 1}. That dimension is normalized.- In training, the caller sets the
INV_VARIANCEdims explicitly, e.g.{B, S, 1}forx = {B, S, H}.
Any other shape is refused with INVALID_VALUE. Pass a scale of ones, or reshape, to get the axes you want.
Graph::rmsnorm_backward(dy, x, scale, inv_variance, attributes) also takes an optional scale (Python: scale=None).
Without it there is no DSCALE or DBIAS, so has_dbias(true) is refused. The INV_VARIANCE dims state the
normalization axes. Verified on Hopper with cuDNN 9.12, 9.14, 9.18, 9.20, 9.24 and 9.27, and on
Ampere and Blackwell with 9.27; older backends were not qualified.
RMSNorm PyTorch API
cudnn.experimental.ops.rms_norm(input, weight, bias=None, eps=1e-5) applies
RMS normalization over the final input dimension using a cuDNN graph and
supports autograd and torch.compile.
This experimental signature intentionally includes an optional additive bias
and requires a weight tensor. It is therefore not a drop-in replacement for
torch.nn.functional.rms_norm, whose signature includes normalized_shape
and no bias argument. Inputs, weights, and optional bias must be CUDA tensors
with matching FP16, BF16, or FP32 dtypes, and the input must contain at least
one normalization row.
RMSNorm + SiLU + Padding
The experimental direct Python API RmsNormSiluPadSm100 combines optional
bias/residual addition, normalization, SiLU, padding, and history/cache copies
for C160/C320/C640 BF16 tensors. It does not perform convolution. See
fused Conv3D + post-operations for
requirements and the API reference for the
exact normalization and tensor-layout contracts.
Adaptive Layernorm Forward
The adaptive layernorm operation computes
where the scale and bias vary across samples in a batch.
Normalization happens across features, independently for each sample.
C++ API
The output array has tensors in order of: [output, mean, variance]
AdaLayernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:
bias is optional; pass nullptr (Python: None, the default). scale is required.
Python API
- adalayernorm
- norm_forward_phase
- input
- scale
- bias (optional)
- epsilon
- compute_data_type
- name
Adaptive Layernorm Backward (DADALN)
The DADALN operation computes data gradient, scale gradient, and bias gradient during backpropagation of an adaptive layernorm forward operation.
C++ API
The output array has tensors in order of: [input gradient, scale gradient, bias gradient].
AdaLayernorm_backward_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:
Python API
- adalayernorm_backward
- grad
- input
- scale
- mean
- inv_variance
- compute_data_type
- name
Instancenorm Forward
The instancenorm operation computes
Normalization happens across each sample.
C++ API
The output array has tensors in order of: [output, mean, variance].
Instancenorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:
Python API
- instancenorm
- norm_forward_phase
- adaptive (optional, bool)
- input
- scale
- bias
- epsilon
- compute_data_type
- name
adaptive is a Python-only flag for adaptive InstanceNorm. The input must be channels-first contiguous
([N, C, L], [N, C, H, W] or [N, C, D, H, W]), and scale/optional bias must be per sample and channel
with singleton spatial dimensions — [N, C, 1], [N, C, 1, 1], [N, C, 1, 1, 1]; the statistics have that
same shape and the reduction runs over the spatial axes. Adaptive InstanceNorm is served by Python engines only
(adaptive=False keeps backend eligibility). For the adaptive LayerNorm/RMSNorm layouts ([B, S, D] / [S, B, D]
with [B, 1, D] / [1, B, D] scale) see the norm guide (python/cudnn/norm/README.md).
Instancenorm Backward (DIN)
The DIN operation computes data gradient, scale gradient, and bias gradient during backpropagation of an instancenorm forward operation.
C++ API
The output array has tensors in order of: [input gradient, scale gradient, bias gradient].
Instancenorm_backward_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:
Python API
- instancenorm_backward
- grad
- input
- scale
- mean
- inv_variance
- adaptive (optional, bool)
- compute_data_type
- name