Normalizations#

Batchnorm Forward#

The batchnorm operation computes:

\( output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias \)

Optionally the operation also computes:

\(next\_running\_mean = (1 - momentum)*previous\_running\_mean + momentum*current\_running\_mean\)

\(next\_running\_variance = (1 - momentum)*previous\_running\_variance + momentum*current\_running\_variance\)

C++ API#

std::array<std::shared_ptr<Tensor_attributes>, 5> batchnorm(std::shared_ptr<Tensor_attributes>& input,
                                                            std::shared_ptr<Tensor_attributes>& scale,
                                                            std::shared_ptr<Tensor_attributes>& bias,
                                                            Batchnorm_attributes attributes);

Where the output array has tensors in order of: [output, saved_mean, saved_invariance, next_running_mean, next_running_variance]

Batchnorm attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Batchnorm_attributes&
set_previous_running_stats(std::shared_ptr<Tensor_attributes>& previous_running_mean,
                            std::shared_ptr<Tensor_attributes>& previous_running_variance,
                            std::shared_ptr<Tensor_attributes>& momentum)

Batchnorm_attributes&
set_name(std::string const&)

Batchnorm_attributes&
set_compute_data_type(DataType_t value)

Python API#

  • batchnorm

    • input

    • scale

    • bias

    • in_running_mean

    • in_running_var

    • epsilon

    • momentum

    • compute_data_type

    • name

Batchnorm Finalize#

The bn_finalize operation calculates the statistics required for the next iteration from the statistics generated by the genstat operation.

C++ API#

    std::array<std::shared_ptr<Tensor_attributes>, 6> bn_finalize(std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  std::shared_ptr<Tensor_attributes>,
                                                                  BN_finalize_attributes);

The outputs are [EQ_SCALE, EQ_BIAS, MEAN, INV_VARIANCE, NEXT_RUNNING_MEAN, NEXT_RUNNING_VAR].

Batchnorm Backward (DBN)#

The DBN operation computes data gradient, scale gradient, bias gradient during backpropagation of batchnorm forward operation.

C++ API#

std::array<std::shared_ptr<Tensor_attributes>, 3> batchnorm_backward(std::shared_ptr<Tensor_attributes> loss,
                                                                         std::shared_ptr<Tensor_attributes> input,
                                                                         std::shared_ptr<Tensor_attributes> scale,
                                                                         Batchnorm_backward_attributes);

The output array has tensors in order of: [input gradient, scale gradient, bias gradient].

DBN attributes is a lightweight structure with setters:

Batchnorm_backward_attributes&
set_saved_mean_and_inv_variance(std::shared_ptr<Tensor_attributes> saved_mean,
                                std::shared_ptr<Tensor_attributes> saved_inverse_variance)
                                
Batchnorm_backward_attributes&
set_epsilon(std::shared_ptr<Tensor_attributes> epsilon)

Batchnorm_backward_attributes&
set_name(std::string const&)

Batchnorm_backward_attributes&
set_compute_data_type(DataType_t value)

Only setting either (saved mean and inverse_variance) or (epsilon) is necessary.

Generate Stats#

The Genstats operation computes the sum and sum of squares per-channel dimension.

C++ API#

std::array<std::shared_ptr<Tensor_attributes>, 2>
cudnn_frontend::graph::genstats(std::shared_ptr<Tensor_attributes>, Genstats_attributes);

The output array has tensors in order of: [sum, square_sum]

Genstats attributes is a lightweight structure with setters:

Genstats_attributes&
set_name(std::string const&)

Genstats_attributes&
set_compute_data_type(DataType_t value)

Python API#

  • genstats

    • input

    • compute_data_type

    • name

Layernorm Forward#

The layernorm operation computes

\( output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias \)

Normalization happens across features, independently for each sample.

C++ API#

std::array<std::shared_ptr<Tensor_attributes>, 3> layernorm(std::shared_ptr<Tensor_attributes>& input,
                                                            std::shared_ptr<Tensor_attributes>& scale,
                                                            std::shared_ptr<Tensor_attributes>& bias,
                                                            Layernorm_attributes attributes);

The output array has tensors in order of: [output, mean, variance]

Layernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Layernorm_attributes&
set_name(std::string const&)

Layernorm_attributes&
set_compute_data_type(DataType_t value)

Python API#

  • layernorm

    • norm_forward_phase

    • input

    • scale

    • bias

    • epsilon

    • compute_data_type

    • name

Layernorm Backward (DLN)#

The DLN operation computes data gradient, scale gradient, and bias gradient during backpropagation of a layernorm forward operation.

C++ API#

std::array<std::shared_ptr<Tensor_attributes>, 3>
            layernorm_backward(std::shared_ptr<Tensor_attributes> dy,
                          std::shared_ptr<Tensor_attributes> x,
                          std::shared_ptr<Tensor_attributes> scale,
                          Layernorm_backward_attributes options);

The output array has tensors in order of: [input gradient, scale gradient, bias gradient].

Layernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Layernorm_attributes&
set_name(std::string const&)

Layernorm_attributes&
set_compute_data_type(DataType_t value)

Python API#

  • layernorm

    • input

    • scale

    • loss

    • compute_data_type

    • name

Adaptive Layernorm Forward#

The adaptive layernorm operation computes

\( output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias \)

where the scale and bias vary across samples in a batch.

Normalization happens across features, independently for each sample.

C++ API#

std::array<std::shared_ptr<Tensor_attributes>, 3> adalayernorm(std::shared_ptr<Tensor_attributes>& input,
                                                            std::shared_ptr<Tensor_attributes>& scale,
                                                            std::shared_ptr<Tensor_attributes>& bias,
                                                            Layernorm_attributes attributes);

The output array has tensors in order of: [output, mean, variance]

AdaLayernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

AdaLayernorm_attributes&
set_name(std::string const&)

AdaLayernorm_attributes&
set_compute_data_type(DataType_t value)

Python API#

  • adalayernorm

    • norm_forward_phase

    • input

    • scale

    • bias

    • epsilon

    • compute_data_type

    • name

Adaptive Layernorm Backward (DADALN)#

The DADALN operation computes data gradient, scale gradient, and bias gradient during backpropagation of an adaptive layernorm forward operation.

C++ API#

std::array<std::shared_ptr<Tensor_attributes>, 3>
            adalayernorm_backward(std::shared_ptr<Tensor_attributes> dy,
                          std::shared_ptr<Tensor_attributes> x,
                          std::shared_ptr<Tensor_attributes> scale,
                          AdaLayernorm_backward_attributes options);

The output array has tensors in order of: [input gradient, scale gradient, bias gradient].

AdaLayernorm_backward_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

AdaLayernorm_backward_attributes&
set_name(std::string const&)

AdaLayernorm_backward_attributes&
set_compute_data_type(DataType_t value)

Python API#

  • adalayernorm_backward

    • dy

    • x

    • scale

    • mean

    • inv_variance

    • compute_data_type

    • name

Instancenorm Forward#

The instancenorm operation computes

\( output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias \)

Normalization happens across each sample.

C++ API#

std::array<std::shared_ptr<Tensor_attributes>, 3> instancenorm(std::shared_ptr<Tensor_attributes>& input,
                                                            std::shared_ptr<Tensor_attributes>& scale,
                                                            std::shared_ptr<Tensor_attributes>& bias,
                                                            Instancenorm_attributes attributes);

The output array has tensors in order of: [output, mean, variance].

Instancenorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Instancenorm_attributes&
set_name(std::string const&)

Instancenorm_attributes&
set_compute_data_type(DataType_t value)

Python API#

  • instancenorm

    • norm_forward_phase

    • input

    • scale

    • bias

    • epsilon

    • compute_data_type

    • name

Instancenorm Backward (DIN)#

The DIN operation computes data gradient, scale gradient, and bias gradient during backpropagation of an instancenorm forward operation.

C++ API#

std::array<std::shared_ptr<Tensor_attributes>, 3>
            instancenorm_backward(std::shared_ptr<Tensor_attributes> dy,
                          std::shared_ptr<Tensor_attributes> x,
                          std::shared_ptr<Tensor_attributes> scale,
                          Instancenorm_backward_attributes options);

The output array has tensors in order of: [input gradient, scale gradient, bias gradient].

Instancenorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes:

Instancenorm_attributes&
set_name(std::string const&)

Instancenorm_attributes&
set_compute_data_type(DataType_t value)

Python API#

  • layernorm

    • input

    • scale

    • loss

    • compute_data_type

    • name

On this page