Normalizations

Batchnorm Forward

The batchnorm operation computes:

$output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias$

Optionally the operation also computes:

$next\_running\_mean = (1 - momentum)*previous\_running\_mean + momentum*current\_running\_mean$

$next\_running\_variance = (1 - momentum)*previous\_running\_variance + momentum*current\_running\_variance$

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 5> batchnorm(std::shared_ptr<Tensor_attributes>& input,
                                                            std::shared_ptr<Tensor_attributes>& scale,
                                                            std::shared_ptr<Tensor_attributes>& bias,
                                                            Batchnorm_attributes attributes); 

Where the output array has tensors in order of: [output, saved_mean, saved_invariance, next_running_mean, next_running_variance]

Batchnorm attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Batchnorm_attributes&
set_previous_running_stats(std::shared_ptr<Tensor_attributes>& previous_running_mean,
                            std::shared_ptr<Tensor_attributes>& previous_running_variance,
                            std::shared_ptr<Tensor_attributes>& momentum)
Batchnorm_attributes&
set_name(std::string const&)
Batchnorm_attributes&
set_compute_data_type(DataType_t value)

Python API

• batchnorm
  • input
  • scale
  • bias
  • in_running_mean
  • in_running_var
  • epsilon
  • momentum
  • compute_data_type
  • name

Batchnorm Finalize

The bn_finalize operation calculates the statistics required for the next iteration from the statistics generated by the genstat operation.

C++ API

    std::array<std::shared_ptr<Tensor_attributes>, 6> bn_finalize(std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  BN_finalize_attributes);

The outputs are [EQ_SCALE, EQ_BIAS, MEAN, INV_VARIANCE, NEXT_RUNNING_MEAN, NEXT_RUNNING_VAR].

Batchnorm Backward (DBN)

The DBN operation computes data gradient, scale gradient, bias gradient during backpropagation of batchnorm forward operation.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 3> batchnorm_backward(std::shared_ptr<Tensor_attributes> loss,
                                                                         std::shared_ptr<Tensor_attributes> input,
                                                                         std::shared_ptr<Tensor_attributes> scale,
                                                                         Batchnorm_backward_attributes);

The output array has tensors in order of: [input gradient, scale gradient, bias gradient].

DBN attributes is a lightweight structure with setters:

Batchnorm_backward_attributes&
set_saved_mean_and_inv_variance(std::shared_ptr<Tensor_attributes> saved_mean,
                                std::shared_ptr<Tensor_attributes> saved_inverse_variance)
                                
Batchnorm_backward_attributes&
set_epsilon(std::shared_ptr<Tensor_attributes> epsilon)
Batchnorm_backward_attributes&
set_name(std::string const&)
Batchnorm_backward_attributes&
set_compute_data_type(DataType_t value)

Only setting either (saved mean and inverse_variance) or (epsilon) is necessary.

Generate Stats

The Genstats operation computes the sum and sum of squares per-channel dimension.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 2>
cudnn_frontend::graph::genstats(std::shared_ptr<Tensor_attributes>, Genstats_attributes);

The output array has tensors in order of: [sum, square_sum]

Genstats attributes is a lightweight structure with setters:

Genstats_attributes&
set_name(std::string const&)
Genstats_attributes&
set_compute_data_type(DataType_t value)

Python API

• genstats
  • input
  • compute_data_type
  • name

Layernorm Forward

The layernorm operation computes

$output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias$

Normalization happens across features, independently for each sample.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 3> layernorm(std::shared_ptr<Tensor_attributes>& input,
                                                            std::shared_ptr<Tensor_attributes>& scale,
                                                            std::shared_ptr<Tensor_attributes>& bias,
                                                            Layernorm_attributes attributes); 

The output array has tensors in order of: [output, mean, variance]

Layernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Layernorm_attributes&
set_name(std::string const&)
Layernorm_attributes&
set_compute_data_type(DataType_t value)

Python API

• layernorm
  • norm_forward_phase
  • input
  • scale
  • bias
  • epsilon
  • compute_data_type
  • name

Experimental PyTorch API

cudnn.experimental.ops.layer_norm(input, normalized_shape, weight=None, bias=None, eps=1e-5) provides a graph-backed PyTorch custom op with autograd and torch.compile support. It currently supports normalization over one final dimension on nonempty CUDA tensors with FP16, BF16, or FP32 activations.

Layernorm Backward (DLN)

The DLN operation computes data gradient, scale gradient, and bias gradient during backpropagation of a layernorm forward operation.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 3>
            layernorm_backward(std::shared_ptr<Tensor_attributes> dy,
                          std::shared_ptr<Tensor_attributes> x,
                          std::shared_ptr<Tensor_attributes> scale,
                          Layernorm_backward_attributes options);

The output array has tensors in order of: [input gradient, scale gradient, bias gradient].

Layernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Layernorm_attributes&
set_name(std::string const&)
Layernorm_attributes&
set_compute_data_type(DataType_t value)

Python API

• layernorm
  • input
  • scale
  • loss
  • compute_data_type
  • name

RMSNorm PyTorch API

cudnn.experimental.ops.rms_norm(input, weight, bias=None, eps=1e-5) applies RMS normalization over the final input dimension using a cuDNN graph and supports autograd and torch.compile.

This experimental signature intentionally includes an optional additive bias and requires a weight tensor. It is therefore not a drop-in replacement for torch.nn.functional.rms_norm, whose signature includes normalized_shape and no bias argument. Inputs, weights, and optional bias must be CUDA tensors with matching FP16, BF16, or FP32 dtypes, and the input must contain at least one normalization row.

Adaptive Layernorm Forward

The adaptive layernorm operation computes

$output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias$

where the scale and bias vary across samples in a batch.

Normalization happens across features, independently for each sample.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 3> adalayernorm(std::shared_ptr<Tensor_attributes>& input,
                                                            std::shared_ptr<Tensor_attributes>& scale,
                                                            std::shared_ptr<Tensor_attributes>& bias,
                                                            Layernorm_attributes attributes); 

The output array has tensors in order of: [output, mean, variance]

AdaLayernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

AdaLayernorm_attributes&
set_name(std::string const&)
AdaLayernorm_attributes&
set_compute_data_type(DataType_t value)

Python API

• adalayernorm
  • norm_forward_phase
  • input
  • scale
  • bias
  • epsilon
  • compute_data_type
  • name

Adaptive Layernorm Backward (DADALN)

The DADALN operation computes data gradient, scale gradient, and bias gradient during backpropagation of an adaptive layernorm forward operation.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 3>
            adalayernorm_backward(std::shared_ptr<Tensor_attributes> dy,
                          std::shared_ptr<Tensor_attributes> x,
                          std::shared_ptr<Tensor_attributes> scale,
                          AdaLayernorm_backward_attributes options);

The output array has tensors in order of: [input gradient, scale gradient, bias gradient].

AdaLayernorm_backward_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

AdaLayernorm_backward_attributes&
set_name(std::string const&)
AdaLayernorm_backward_attributes&
set_compute_data_type(DataType_t value)

Python API

• adalayernorm_backward
  • dy
  • x
  • scale
  • mean
  • inv_variance
  • compute_data_type
  • name

Instancenorm Forward

The instancenorm operation computes

$output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias$

Normalization happens across each sample.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 3> instancenorm(std::shared_ptr<Tensor_attributes>& input,
                                                            std::shared_ptr<Tensor_attributes>& scale,
                                                            std::shared_ptr<Tensor_attributes>& bias,
                                                            Instancenorm_attributes attributes); 

The output array has tensors in order of: [output, mean, variance].

Instancenorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Instancenorm_attributes&
set_name(std::string const&)
Instancenorm_attributes&
set_compute_data_type(DataType_t value)

Python API

• instancenorm
  • norm_forward_phase
  • input
  • scale
  • bias
  • epsilon
  • compute_data_type
  • name

Instancenorm Backward (DIN)

The DIN operation computes data gradient, scale gradient, and bias gradient during backpropagation of an instancenorm forward operation.

C++ API

std::array<std::shared_ptr<Tensor_attributes>, 3>
            instancenorm_backward(std::shared_ptr<Tensor_attributes> dy,
                          std::shared_ptr<Tensor_attributes> x,
                          std::shared_ptr<Tensor_attributes> scale,
                          Instancenorm_backward_attributes options);

The output array has tensors in order of: [input gradient, scale gradient, bias gradient].

Instancenorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Instancenorm_attributes&
set_name(std::string const&)
Instancenorm_attributes&
set_compute_data_type(DataType_t value)

Python API

• layernorm
  • input
  • scale
  • loss
  • compute_data_type
  • name