RNN and multihead attention API calls may exhibit nondeterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This happens when two buffer sizes (16 KB and 4 MB) are used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the nondeterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

• cudnnBatchNormalizationBackward()
• cudnnBatchNormalizationBackwardEx()
• cudnnBatchNormalizationForwardTraining()
• cudnnBatchNormalizationForwardTrainingEx()
• cudnnGetBatchNormalizationBackwardExWorkspaceSize()
• cudnnGetBatchNormalizationForwardTrainingExWorkspaceSize()
• cudnnGetBatchNormalizationTrainingExReserveSpaceSize()
• cudnnGetNormalizationBackwardWorkspaceSize()
• cudnnGetNormalizationForwardTrainingWorkspaceSize()
• cudnnGetNormalizationTrainingReserveSpaceSize()
• cudnnNormalizationBackward()
• cudnnNormalizationForwardTraining()

• Resampling forward: CUDNN_RESAMPLE_AVGPOOL_EXCLUDE_PADDING
• Resampling backward: CUDNN_RESAMPLE_AVGPOOL_EXCLUDE_PADDING, CUDNN_RESAMPLE_AVGPOOL_INCLUDE_PADDING, and CUDNN_RESAMPLE_MAXPOOL

RNN and multihead attention API calls may exhibit nondeterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This happens when two buffer sizes (16 KB and 4 MB) are used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the nondeterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

• cudnnBatchNormalizationBackward()
• cudnnBatchNormalizationBackwardEx()
• cudnnBatchNormalizationForwardTraining()
• cudnnBatchNormalizationForwardTrainingEx()
• cudnnGetBatchNormalizationBackwardExWorkspaceSize()
• cudnnGetBatchNormalizationForwardTrainingExWorkspaceSize()
• cudnnGetBatchNormalizationTrainingExReserveSpaceSize()
• cudnnGetNormalizationBackwardWorkspaceSize()
• cudnnGetNormalizationForwardTrainingWorkspaceSize()
• cudnnGetNormalizationTrainingReserveSpaceSize()
• cudnnNormalizationBackward()
• cudnnNormalizationForwardTraining()

• CUDNN_POINTWISE_GELU_APPROX_TANH_FWD and CUDNN_POINTWISE_GELU_APPROX_TANH_BWD, which are used for approximating GELU in the forward and backward pass, respectively.
• CUDNN_POINTWISE_ERF, which can be used to piecewise create the GELU operator.
• CUDNN_POINTWISE_IDENTITY, which can be used for explicitly converting between formats.

• Added support for performant adaptive pooling for NHWC layout supporting flexible I/O datatypes. It also supports large tensors with more than 4 trillion elements.
• Added support for passing host scalars by value as B tensor in the pointwise operations.
• Added support for generating code for more broadcasting patterns in the pointwise operations.
• The CPU overhead associated with the subgraph execution has been reduced by 30 - 40%.
• NVIDIA Ampere Architecture INT8 conv fusion heuristics have been updated to recommend more performant kernel configs for smaller problem sizes.

RNN and multihead attention API calls may exhibit nondeterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This happens when two buffer sizes (16 KB and 4 MB) are used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the nondeterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

• cudnnBatchNormalizationBackward()
• cudnnBatchNormalizationBackwardEx()
• cudnnBatchNormalizationForwardTraining()
• cudnnBatchNormalizationForwardTrainingEx()
• cudnnGetBatchNormalizationBackwardExWorkspaceSize()
• cudnnGetBatchNormalizationForwardTrainingExWorkspaceSize()
• cudnnGetBatchNormalizationTrainingExReserveSpaceSize()
• cudnnGetNormalizationBackwardWorkspaceSize()
• cudnnGetNormalizationForwardTrainingWorkspaceSize()
• cudnnGetNormalizationTrainingReserveSpaceSize()
• cudnnNormalizationBackward()
• cudnnNormalizationForwardTraining()

• Added support for CUDNN_BACKEND_OPERATION_RESAMPLE_FWD for the CUDNN_ATTR_RESAMPLE_MODE set to CUDNN_RESAMPLE_AVGPOOL and CUDNN_RESAMPLE_MAXPOOL through the runtime fusion engine. It can achieve up to 3x speed up compared to the legacy cudnnPoolingForward() API. Pointwise fusions to the output of this operation are also supported. Documentation about the patterns supported can be found in the Supported Graph Patterns section in the cuDNN Developer Guide.
• Newly added micro tile sizes for pointwise fusions that provide significantly improved performance on smaller problem sizes.

RNN and multihead attention API calls may exhibit nondeterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This happens when two buffer sizes (16 KB and 4 MB) are used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the nondeterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

• cudnnBatchNormalizationBackward()
• cudnnBatchNormalizationBackwardEx()
• cudnnBatchNormalizationForwardTraining()
• cudnnBatchNormalizationForwardTrainingEx()
• cudnnGetBatchNormalizationBackwardExWorkspaceSize()
• cudnnGetBatchNormalizationForwardTrainingExWorkspaceSize()
• cudnnGetBatchNormalizationTrainingExReserveSpaceSize()
• cudnnGetNormalizationBackwardWorkspaceSize()
• cudnnGetNormalizationForwardTrainingWorkspaceSize()
• cudnnGetNormalizationTrainingReserveSpaceSize()
• cudnnNormalizationBackward()
• cudnnNormalizationForwardTraining()

• Added API and support for the GEN_INDEX capability. CUDNN_POINTWISE_GEN_INDEX returns the position of an element in an input tensor along a given axis. This operation is similar to NumPy’s mesh grid operation as it returns a tensor with the index of all elements calculated according to the specified axis in the original tensor dimensions.
• Added API and support for the BINARY_SELECT capability. CUDNN_POINTWISE_BINARY_SELECT is similar to the ternary operation and selects between two input elements based on a predicate element.
• Experimentally supports serialization of execution plans to or from a string representation to enable the user to avoid recompilation of the fusion kernels. This feature only supports the runtime fusion engine currently. Generalized support for additional engines is planned for future releases.

• Previous versions of the runtime fusion engine only supported a minimum 128-bit alignment for tensors in all the operations. From this release onwards, the minimum alignment requirement has been relaxed down to 32 bit for input tensors in matrix multiplication and convolution for NVIDIA Ampere Architecture GPUs. For output tensors in any operation and input tensors for pointwise operations, the minimum alignment requirement has been relaxed down to 8 bit.
• Added support for ARM servers.

• We added documentation for the following data types and API.
  • CUDNN_BACKEND_OPERATION_REDUCTION_DESCRIPTOR
  • CUDNN_BACKEND_POINTWISE_DESCRIPTOR
  • CUDNN_BACKEND_REDUCTION_DESCRIPTOR
  • cudnnBackendBehaviorNote_t
  • cudnnBackendTensorReordering_t
  • cudnnBnFinalizeStatsMode_t
  • cudnnPaddingMode_t
  • cudnnResampleMode_t

RNN and multihead attention API calls may exhibit nondeterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This happens when two buffer sizes (16 KB and 4 MB) are used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

• cudnnBatchNormalizationBackward()
• cudnnBatchNormalizationBackwardEx()
• cudnnBatchNormalizationForwardTraining()
• cudnnBatchNormalizationForwardTrainingEx()
• cudnnGetBatchNormalizationBackwardExWorkspaceSize()
• cudnnGetBatchNormalizationForwardTrainingExWorkspaceSize()
• cudnnGetBatchNormalizationTrainingExReserveSpaceSize()
• cudnnGetNormalizationBackwardWorkspaceSize()
• cudnnGetNormalizationForwardTrainingWorkspaceSize()
• cudnnGetNormalizationTrainingReserveSpaceSize()
• cudnnNormalizationBackward()
• cudnnNormalizationForwardTraining()

• Added heuristics for convolution + x fusion and matmul + x fusion for NVIDIA Volta and NVIDIA Turing architectures.
• Updated the heuristics for matmul + x fusion for NVIDIA Ampere Architecture.
• Small performance improvement for the matmul + x fusion.
• Compilation time reduction.

• cudnnBackendAttributeName_t
• cudnnBackendAttributeType_t
• cudnnBackendDescriptorType_t
• cudnnBackendNumericalNote_t

RNN and multihead attention API calls may exhibit nondeterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This happens when two buffer sizes (16 KB and 4 MB) are used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

• cudnnBatchNormalizationBackward()
• cudnnBatchNormalizationBackwardEx()
• cudnnBatchNormalizationForwardTraining()
• cudnnBatchNormalizationForwardTrainingEx()
• cudnnGetBatchNormalizationBackwardExWorkspaceSize()
• cudnnGetBatchNormalizationForwardTrainingExWorkspaceSize()
• cudnnGetBatchNormalizationTrainingExReserveSpaceSize()
• cudnnGetNormalizationBackwardWorkspaceSize()
• cudnnGetNormalizationForwardTrainingWorkspaceSize()
• cudnnGetNormalizationTrainingReserveSpaceSize()
• cudnnNormalizationBackward()
• cudnnNormalizationForwardTraining()

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

• cudnnBatchNormalizationBackward()
• cudnnBatchNormalizationBackwardEx()
• cudnnBatchNormalizationForwardTraining()
• cudnnBatchNormalizationForwardTrainingEx()
• cudnnGetBatchNormalizationBackwardExWorkspaceSize()
• cudnnGetBatchNormalizationForwardTrainingExWorkspaceSize()
• cudnnGetBatchNormalizationTrainingExReserveSpaceSize()
• cudnnGetNormalizationBackwardWorkspaceSize()
• cudnnGetNormalizationForwardTrainingWorkspaceSize()
• cudnnGetNormalizationTrainingReserveSpaceSize()
• cudnnNormalizationBackward()
• cudnnNormalizationForwardTraining()

1. When the user stream is in capture mode (that is, cudaStreamCaptureStatusActive==1), the cuDNN-owned streams will still have default priority, and
2. RNN functions cudnnRNNForward(), cudnnRNNBackwardData_v8(), cudnnRNNBackwardWeights_v8(), and their legacy counterparts still use default priority CUDA streams or higher priority streams to launch concurrent and cooperative grids.

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

• cudnnBatchNormalizationBackward()
• cudnnBatchNormalizationBackwardEx()
• cudnnBatchNormalizationForwardTraining()
• cudnnBatchNormalizationForwardTrainingEx()
• cudnnGetBatchNormalizationBackwardExWorkspaceSize()
• cudnnGetBatchNormalizationForwardTrainingExWorkspaceSize()
• cudnnGetBatchNormalizationTrainingExReserveSpaceSize()
• cudnnGetNormalizationBackwardWorkspaceSize()
• cudnnGetNormalizationForwardTrainingWorkspaceSize()
• cudnnGetNormalizationTrainingReserveSpaceSize()
• cudnnNormalizationBackward()
• cudnnNormalizationForwardTraining()

• libcudnn_ops_infer_static.a
• libcudnn_ops_train_static.a
• libcudnn_cnn_infer_static.a
• libcudnn_cnn_train_static.a
• libcudnn_adv_infer_static.a
• libcudnn_adv_train_static.a

• Added HSH support - FP16 data type with FP32 math precision. Allow to achieve FP16 mixed-precision Tensor Core performance without sacrificing accuracy.
• Added support to bias gradient computation. Before cuDNN 8.3.0, bias was supported only for inference.
• Multihead attention has two dropout layers active in training mode. The first dropout operation is applied directly to the softmax output. The second dropout operation alters the multihead attention output, just before the point where residual connections are added. Before cuDNN 8.3.0, only the first dropout layer was supported.
• Significant performance improvement out of the box (no changes are required from users) for both multihead attention forward and backward paths.

• The cuDNN runtime fusion engine now supports resample operations of upsample and downsample. Support has been added for 2*2 average pooling with stride 2 and upsample by a factor of 2 using bilinear interpolation. The datatype supported is FP32 and the compute datatype is FP32. The resample operations can also be fused with other operations provided the input to the resample operation is located in global memory.
• The cuDNN runtime fusion engine is now generalized to accurately obey the intermediate storage datatype users specified in the operation graph. The support datatypes include INT8, BF16, FP16, INT32, FP32. As a general rule, we recommend users to use FP32 as the intermediate storage type that provides balanced numerical precision and performance.
• The cuDNN runtime fusion engine now supports batched matmul operation with row/column broadcast or row/column reduction operations in the epilog.
• The cuDNN runtime fusion engine now does numeric clamping while converting from a data type with a larger dynamic range to one with a relatively smaller dynamic range to avoid numeric overflows at all times.
• The cuDNN runtime fusion engine extends the support for broadcast pointwise operations in the epilogue to now include those between a tensor and a scalar value as well.
• Extended general fusion heuristic support to convolution forward and backward data and weight gradient operation patterns, with FP16, TF32, INT8 I/O data types, to ensure a good heuristic selection to improve out-of-the-box performance.

cuDNN 8.3 compiled with CUDA 10.2 must still rely on a regular method of launching kernels. Deadlocks are mitigated by employing higher priority CUDA streams. Currently, cuDNN RNN APIs still use higher priority streams, however, in future cuDNN versions, priorities of auxiliary streams will match the priority of the user stream defined by the cudnnSetStream() call. Future cuDNN versions will also use the cudaLaunchCooperativeKernel() API to launch cooperative grids in forward RNN functions such as cudnnRNNForward().

1. When the user stream is in capture mode (that is, cudaStreamCaptureStatusActive==1), the cuDNN-owned streams will still have default priority, and
2. RNN functions cudnnRNNForward(), cudnnRNNBackwardData_v8(), cudnnRNNBackwardWeights_v8(), and their legacy counterparts still use default priority CUDA streams or higher priority streams to launch concurrent and cooperative grids.

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

• cudnnBatchNormalizationBackward()
• cudnnBatchNormalizationBackwardEx()
• cudnnBatchNormalizationForwardTraining()
• cudnnBatchNormalizationForwardTrainingEx()
• cudnnGetBatchNormalizationBackwardExWorkspaceSize()
• cudnnGetBatchNormalizationForwardTrainingExWorkspaceSize()
• cudnnGetBatchNormalizationTrainingExReserveSpaceSize()
• cudnnGetNormalizationBackwardWorkspaceSize()
• cudnnGetNormalizationForwardTrainingWorkspaceSize()
• cudnnGetNormalizationTrainingReserveSpaceSize()
• cudnnNormalizationBackward()
• cudnnNormalizationForwardTraining()

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

• Bfloat16 type and compute precision of FP32 (requires compute capability 8.0 or later). For Bfloat16 support, convolution I/O channels are required to be a multiple of 8.
• INT8 and compute precision of INT32 (requires compute capability 7.5 or later) datatype and in NHWC layout. For INT8 support, the convolution I/O channels are required to be a multiple of 16, and unlike the NCHW_VECT_C kernels, filter and bias reordering is not required.

In the fused pointwise/reduction operations, FP32 is the compute precision supported.

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

• When there is a scale+bias+ReLU pattern in the graph fused to the x input of a convolution forward operation
• When the graph contains 3D convolution forward, backward data, or backward filter operation
• When the graph contains a convolution backward data operation with non-unit convolution strides

We are working on further generalization of this support.

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

• an operation graph that contains more than one convolution of matrix multiplication operations
• convolutions with compute type that is not FP32
• grouped convolutions
• when the convolution mode is not CUDNN_CROSS_CORRELATION

• cudnnRNNForwardInference()
• cudnnRNNForwardInferenceEx()
• cudnnRNNForwardTraining()
• cudnnRNNForwardTrainingEx()
• cudnnRNNBackwardData()
• cudnnRNNBackwardDataEx()

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multistream setups.

• For EfficientNet, when run using NHWC FP16 Tenor Core configurations on V100 and A100 GPU architectures.
• For PilotNet, AH-Net, MobileNet V3 on V100 and A100 GPU architectures.
• For various 3D convolution cases on NVIDIA RTX 8000.

When calling cudnnConvolutionBiasActivationForward() with INT8x4 or INT8x32 I/O tensors, it could result in CUDNN_STATUS_BAD_PARAM in 8.0.4. This issue has been fixed in this release.

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN 8.0.4 library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN 8.0.3 library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

Occasionally, inaccurate results were observed in outputs of the cudnnRNNBackwardWeights() and cudnnRNNBackwardWeightsEx() functions when the RNN cell type was GRU and the NVIDIA Ampere Architecture GPU was used with FP32 I/O and mathType of CUDNN_DEFAULT_MATH or CUDNN_TENSOR_OP_MATH. Users may switch to CUDNN_FMA_MATH as a temporary workaround. This issue is being investigated.

cudnnRNN*() with LSTM mode may produce inaccurate results on the cy outputs when clipping is enabled on all GPUs. This issue exists in previous cuDNN releases as well.

On NVIDIA Volta and NVIDIA Pascal architectures, performance regressions may be present for various TRUE_HALF convolutions.

Some ResNet-50 and SSD mixed precision inference use-cases may have performance regressions compared to cuDNN 7.6 on V100. V-Net 3D models might have performance regressions on NVIDIA Turing based architectures.

When using cudnnRNN*Ex() APIs, if the user used CUDNN_RNN_DATA_LAYOUT_SEQ_MAJOR_UNPACKED or CUDNN_RNN_DATA_LAYOUT_BATCH_MAJOR_UNPACKED as the layout of the RNN data descriptors, and if the batch size is larger than 6144 on NVIDIA Volta or NVIDIA Ampere Architecture A100 GPUs, or larger than 4096 on NVIDIA Turing GPUs, CUDNN_STATUS_EXECUTION_FAILED would be returned.

Documentation of the backend API is not complete. The CUDNN_BACKEND_OPERATION_GEN_STATS_DESCRIPTOR and CUDNN_BACKEND_OPERATION_POINTWISE_DESCRIPTOR descriptor types will be documented in a future release.

The conv_sample_v8.0 sample is not included in the Debian and RPM packages. This will be fixed in a future release.

The libfreeimage.a library in the RHEL 8 ppc64le RPM is for the wrong architecture. This will be fixed in a future release.

When the user is upgrading from cuDNN 8.0.2 to 8.0.3 through the Debian or RPM package, before installing libcudnn8-samples_*.deb/rpm, users should manually uninstall the old libcudnn8-doc package, otherwise a file conflict may happen.

Samples can crash unless they are installed in a writable location.

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN 8.0.2 library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

The tensor pointers and the filter pointers require at a minimum 4-byte alignment, including INT8 data in the cuDNN library.

Some computational options in cuDNN 8.0.2 now require increased alignment on tensors in order to run efficiently. As always, cuDNN recommends users to align tensors to 128-bit boundaries that will be sufficiently aligned for any computational option in cuDNN. Doing otherwise may cause performance regressions in cuDNN 8.0.2 compared to cuDNN v7.6.

For certain algorithms, when the computation is in float (32-bit float) and the output is in FP16 (half float), there are cases where the numerical accuracy between the different algorithms might differ. cuDNN 8.0.2 users can target the backend API to query the numerical notes of the algorithms to get the information programmatically. There are cases where algo0 and algo1 will have a reduced precision accumulation when users target the legacy API. In all cases, these numerical differences are not known to affect training accuracy even though they might show up in unit tests.

For the _ALGO_0 algorithm of convolution backward data and backward filter, grouped convolution with groups larger than 1 and with odd product of dimensions C, D (if 3D convolution), H, and W is not supported on devices older than NVIDIA Volta. To prevent a potential illegal memory access by an instruction that only has a 16-bit version in NVIDIA Volta and above, pad at least one of the dimensions to an even value.

On K80 GPUs, when cudnnConvolutionForward() is used with CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_PRECOMP_GEMM algorithm and half I/O data types a silent error might occur when the output width Q is 1 and both height and width padding are zero.

In INT8x32 Tensor Core cases, the parameters supported by cuDNN v7.6 are limited to W >= (R-1) * dilationW && H >= (S-1) * dilationH, whereas, in cuDNN v8.0.x, W == (R-1) * dilationW || H == (S-1) * dilationH cases are no longer supported.

In prior versions of cuDNN, some convolution algorithms can use texture-based load structure for performance improvements particularly in older hardware architectures. Users can opt out of using texture using the environmental variable CUDNN_TEXOFF_DBG. In cuDNN 8.x, this variable is removed. Texture loading is turned off by default. Users who want to continue to use texture-based load, can adapt the new backend API, and toggle the engine knob CUDNN_KNOB_TYPE_USE_TEX to 1 for engines that support texture-based load instructions.

The implementation of cuDNNLRNCrossChannelBackward() for even-sized normalization windows was incorrect in all previous releases. This issue has been fixed in this release.

There is not a dedicated API to query the supported or the most performant algo for cudnnConvolutionBiasActivationForward() in cuDNN. It is not recommended to query w using cudnnGetConvolutionForwardAlgorithm_v7. Instead, we recommend using the cuDNN version 8 backend API. The number of supported engines can be queried using enum CUDNN_ATTR_OPERATIONGRAPH_ENGINE_GLOBAL_COUNT from an operation graph descriptor using cudnnBackendGetAttribute().

A memcheck error may have occurred on cuDNN version 7.x builds when calling cudnnConvolutionBackwardFilter() on NVIDIA Volta or NVIDIA Turing GPUs. This issue has been fixed in this release.

Various convolutions that exhibited sub-optimal performance on GA100 GPUs are now achieving ideal performance. (not applicable for Jetson platforms)

cudnnCnnTrainVersionCheck() and cudnnCnnInferVersionCheck() were missing in past releases. This issue has been fixed in this release.

Documentation of RNN new APIs and deprecations is not complete. The cudnnRNNBackwardData_v8() and cudnnRNNBackwardWeights_v8() have been added to this release.

cuDNN 8.0.1 built with Windows and CUDA 11.0 RC had reduced performance on 2D, 3D, and grouped convolutions compared to Linux. This issue has been fixed in this release. (not applicable for Jetson platforms)

There was a known issue in cuDNN 8.0.1 when linking statically to cuDNN and using the library's 3D algo1 backward filter convolutions. Users would see that the library emits an internal error or incorrectly state that a shared library was missing. This issue has been fixed in this release.

When using an RPM file on RedHat for installation, upgrading from cuDNN v7 to cuDNN v8 directly or indirectly using TensorRT 7.1.3 would cause installation errors. This issue has been fixed in this release.

The implementation of cuDNNLRNCrossChannelBackward was inconsistent with the implementation of cuDNNLRNCrossChannelForward and returned incorrect results when the normalization window was even. This issue has been fixed in this release.

RNN APIs in cuDNN v8.0.1, compiled with CUDA 11.0, used an incorrect default down-conversion on GPUs with CUDA SM version SM80 (NVIDIA Ampere Architecture GPU family) when supplied input data and weights have the CUDNN_DATA_FLOAT type and cudnnMathType_t set using cudnnSetRNNMatrixMathType() is CUDNN_DEFAULT_MATH or CUDNN_TENSOR_OP_MATH. Instead of using the default TF32 computation when Tensor Cores are used, a down conversion to FP16 (half-precision) was performed; same as in the CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION mode. This introduced a lower dynamic range of intermediate data but possibly faster execution. To disable the automatic down conversion of CUDNN_DATA_FLOAT weights and data in RNN APIs, the user needed to set the environmental variable NVIDIA_TF32_OVERRIDE to 0 (notice this would have disabled the use of TF32 in the entire library, which might have a performance impact on CNNs that are not affected by this issue). Another workaround was to assign the CUDNN_FMA_MATH mode to the cudnnMathType_t argument in cudnnSetRNNMatrixMathType(). Due to this, the A100 GPU TF32 feature was not accessible for RNNs in cuDNN v8.0.1. This issue has been fixed in this release. (not applicable for Jetson platforms)

cuDNN convolution APIs may return CUDNN_STATUS_EXECUTION_FAILED when the number of input or output channels equals to or exceeds 2097152. This issue exists for all cuDNN 8.0.x releases. This issue has been fixed in this release.

Since version 8.0.0 Preview, cudnnConvolutionForward(), cudnnConvolutionBackwardData(), and cudnnConvolutionBackwardFilter() erroneously returned CUDNN_STATUS_INTERNAL_ERROR when the workspace size argument value was less than the required workspace size as returned by their respective cudnnGetWorkspace() API. This issue has been fixed and CUDNN_STATUS_BAD_PARAMS is returned as documented.

In this release, the performance of cudnnConvolutionBiasActivationForward() for true-half use cases on NVIDIA Pascal, INT8x4 use cases on NVIDIA Volta, and NVIDIA Turing, compared to version 7.6 is still lower. In addition, FP32 and pseudo-FP16 performance on NVIDIA Volta, NVIDIA Turing, and the NVIDIA Ampere Architecture GPU is still not fully optimized.

The new RNN APIs: cudnnRNNForward(), cudnnRNNBackwardData_v8(), and cudnnRNNBackwardWeights_v8() are available as a preview in the cuDNN 8.0.2 release.

Occasionally, inaccurate results were observed in outputs of the cudnnRNNBackwardWeights() and cudnnRNNBackwardWeightsEx() functions when the RNN cell type was GRU and the NVIDIA Ampere Architecture GPU was used with FP32 I/O and mathType of CUDNN_DEFAULT_MATH or CUDNN_TENSOR_OP_MATH. Users may switch to CUDNN_FMA_MATH as a temporary workaround. This issue is being investigated.

cudnnRNN*() with LSTM mode may produce inaccurate results on the cy outputs when clipping is enabled on all GPUs. This issue exists in previous cuDNN releases as well.

On NVIDIA Volta and NVIDIA Pascal architectures, performance regressions may be present for TRUE_HALF convolution backward filter.

When using cudnnRNN*Ex() APIs, if the user uses CUDNN_RNN_DATA_LAYOUT_SEQ_MAJOR_UNPACKED or CUDNN_RNN_DATA_LAYOUT_BATCH_MAJOR_UNPACKED as the layout of the RNN data descriptors, and if the batch size is larger than 6144 on NVIDIA Volta or NVIDIA Ampere Architecture A100 GPUs, or larger than 4096 on NVIDIA Turing GPUs, CUDNN_STATUS_EXECUTION_FAILED may be returned.

Currently, there are libcudnn_ops/cnn/adv_infer/train_static.a binaries in the cuDNN Debian and tgz packages. Users are advised not to link against those and link against libcudnn_static.a instead. Those binaries will be removed from the release packages in the next release.

When using cudnnRNN*Ex() APIs, if the user plans to use CUDNN_RNN_DATA_LAYOUT_SEQ_MAJOR_UNPACKED or CUDNN_RNN_DATA_LAYOUT_BATCH_MAJOR_UNPACKED as the layout of the RNN data descriptors, the user should call cudnnSetRNNPaddingMode() to set the mode to CUDNN_RNN_PADDED_IO_ENABLED after initializing an RNNDescriptor but before calling cudnnGetRNNWorkspaceSize(). Not doing this may result in CUDNN_STATUS_EXECUTION_FAILED.

Fused convolution-scale-bias-activation with per-channel α1 and α2 scaling gives incorrect results when the reorder type in the convolution descriptor is set to CUDNN_NO_REORDER.

When the user is using cudnnRNN* APIs with the problem sizes (input size, hidden size) being not multiples of 16 for FP16 tensors or multiples of 8 for FP32 tensors, users may encounter a return status of CUDNN_STATUS_EXECUTION_FAILED.

For some 3D spatial non-Tensor-Core convolutions on Maxwell, NVIDIA Pascal, NVIDIA Volta, and NVIDIA Turing architectures, cudnnBackwardFilter() can return incorrect results when the convolution width padding exceeds the value (filterWidth - 1)/2. Likewise, users of cudnnBackendExecute() can experience the same issue when using the engine with CUDNN_ATTR_ENGINE_GLOBAL_INDEX 32 for backward filter. The issue affecting cudnnBackwardFilter() has been fixed in this release. With cudnnBackendFinalize(), an engine descriptor with CUDNN_ATTR_ENGINE_GLOBAL_INDEX 32 and a backward filter operation that satisfies the above condition will return CUDNN_STATUS_NOT_SUPPORTED.

Samples can crash unless they are installed in a writable location.

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN 8.0.1 library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

Some data types are not widely supported by all cuDNN API. For example, CUDNN_DATA_INT8x4 is not supported by many functions. In such cases, support is available by using cudnnTransformTensor() to transform the tensors from the desired type to a type supported by the API. For example, a user is able to transform input tensors from CUDNN_DATA_INT8x4 to CUDNN_DATA_INT8, run the desired API and then transform output tensors from CUDNN_DATA_INT8 to CUDNN_DATA_INT8x4. Note that this transformation will incur an extra round trip to memory.

The tensor pointers and the filter pointers require at a minimum 4-byte alignment, including INT8 data in the cuDNN library.

Some computational options in cuDNN 8.0.1 now require increased alignment on tensors in order to run efficiently. As always, cuDNN recommends users to align tensors to 128-bit boundaries that will be sufficiently aligned for any computational option in cuDNN. Doing otherwise may cause performance regressions in cuDNN 8.0.1 compared to cuDNN v7.6.

For certain algorithms, when the computation is in float (32-bit float) and the output is in FP16 (half float), there are cases where the numerical accuracy between the different algorithms might differ. cuDNN 8.0.1 users can target the backend API to query the numerical notes of the algorithms to get the information programmatically. There are cases where algo0 and algo1 will have a reduced precision accumulation when users target the legacy API. In all cases, these numerical differences are not known to affect training accuracy even though they might show up in unit tests.

For the _ALGO_0 algorithm of convolution backward data and backward filter, grouped convolution with groups larger than 1 and with odd product of dimensions C, D (if 3D convolution), H, and W is not supported on devices older than NVIDIA Volta. To prevent a potential illegal memory access by an instruction that only has a 16-bit version in Volta and later, pad at least one of the dimensions to an even value.

On K80 GPUs, when cudnnConvolutionForward() is used with CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_PRECOMP_GEMM algorithm and half I/O data types a silent error might occur when the output width Q is 1 and both height and width padding are zero.

On pre-Volta, there are significant performance issues on convolution layers when the compute type is half.

Sub-optimal performance is present in this release for all INT8 convolutions for all GPUs.

The performance of cudnnConvolutionBiasActivationForward() is slower than v7.6 in most cases. This is being actively worked on and performance optimizations will be available in the upcoming releases.

There are some peer-to-peer documentation links that are broken within the cuDNN API Reference. These links will be fixed in the next release.

cudnnCnnTrainVersionCheck() and cudnnCnnInferVersionCheck() are missing in this release and will be added in the GA release.

Documentation of RNN new APIs and deprecations is not complete. The cudnnRNNBackwardData_v8() and cudnnRNNBackwardWeights_v8() functions will be implemented in the next release.

cuDNN 8.0.1 Preview build with Windows and CUDA 11.0 RC has reduced performance on 2D, 3D, and grouped convolutions compared to Linux.

There is a known issue in cuDNN 8.0.1 when linking statically to cuDNN and using the library's 3D algo1 backward filter convolutions. Users will see that the libraries emit an internal error or incorrectly state that a shared library is missing. This is a bug that will be fixed in a future release.

Steps 1 and 2 are required for the user to be able to switch between v7 and v8 installations. After steps 1 and 2 are performed once, step 3 can be used repeatedly and the user can choose the appropriate cuDNN version to work with. For more information, refer to the Installing From An RPM File and Upgrading From v7 To v8 sections in the cuDNN Installation Guide.

When FFT Tiled aglo (that is, CUDNN_CONVOLUTION_FWD_ALGO_FFT_TILING in forward convolution or CUDNN_CONVOLUTION_BWD_DATA_ALGO_FFT_TILING for backward data) is used for 3D convolution, an intermittent silent failure might happen due to an incorrect stream used for kernel execution. In some cases, this might be manifested as undefined values seen in the output.

The implementation of cuDNNLRNCrossChannelBackward is inconsistent with the implementation of cuDNNLRNCrossChannelForward and returns incorrect results when the normalization window is even. This will be fixed in a future release.

RNN APIs in cuDNN v8.0.1, compiled with CUDA 11.0, use an incorrect default down-conversion on GPUs with CUDA SM version SM80 (NVIDIA Ampere Architecture GPU family) when supplied input data and weights have the CUDNN_DATA_FLOAT type and cudnnMathType_t set using cudnnSetRNNMatrixMathType() is CUDNN_DEFAULT_MATH or CUDNN_TENSOR_OP_MATH. Instead of using the default TF32 computation when Tensor Cores are used, a down conversion to FP16 (half-precision) is performed; same as in the CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION mode. This introduces a lower dynamic range of intermediate data but possibly faster execution. To disable the automatic down conversion of CUDNN_DATA_FLOAT weights and data in RNN APIs, set the environmental variable NVIDIA_TF32_OVERRIDE to 0 (notice this will disable the use of TF32 in the entire library, which might have a performance impact on CNNs that are not affected by this issue). Another workaround is to assign the CUDNN_FMA_MATH mode to the cudnnMathType_t argument in cudnnSetRNNMatrixMathType(). Due to this, the A100 TF32 feature is not accessible for RNNs in cuDNN v8.0.1.

cuDNN convolution APIs may return CUDNN_STATUS_EXECUTION_FAILED when the number of input or output channels equals to or exceeds 2097152.

When the user is using cudnnRNN* APIs with the problem sizes (input size, hidden size) being not multiples of 16 for FP16 tensors or multiples of 8 for FP32 tensors, users may encounter a return status of CUDNN_STATUS_EXECUTION_FAILED.

Samples must be installed in a writable location, otherwise the samples can crash.

RNN and multihead attention API calls may exhibit non-deterministic behavior when the cuDNN 8.0.0 library is built with CUDA Toolkit 10.2 or higher. This is the result of a new buffer management and heuristics in the cuBLAS library. As described in the Results Reproducibility section in the cuBLAS Library User's Guide, numerical results may not be deterministic when cuBLAS APIs are launched in more than one CUDA stream using the same cuBLAS handle. This is caused by two buffer sizes (16 KB and 4 MB) used in the default configuration.

When a larger buffer size is not available at runtime, instead of waiting for a buffer of that size to be released, a smaller buffer may be used with a different GPU kernel. The kernel selection may affect numerical results. The user can eliminate the non-deterministic behavior of cuDNN RNN and multihead attention APIs, by setting a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, for example, :16:8 or :4096:2.

The first configuration instructs cuBLAS to allocate eight buffers of 16 KB each in GPU memory while the second setting creates two buffers of 4 MB each. The default buffer configuration in cuBLAS 10.2 and 11.0 is :16:8:4096:2, that is, we have two buffer sizes. In earlier cuBLAS libraries, such as cuBLAS 10.0, it used the :16:8 non-adjustable configuration. When buffers of only one size are available, the behavior of cuBLAS calls is deterministic in multi-stream setups.

Some data types are not widely supported by all cuDNN API. For example, CUDNN_DATA_INT8x4 is not supported by many functions. In such cases, support is available by using cudnnTransformTensor() to transform the tensors from the desired type to a type supported by the API. For example, a user is able to transform input tensors from CUDNN_DATA_INT8x4 to CUDNN_DATA_INT8, run the desired API and then transform output tensors from CUDNN_DATA_INT8 to CUDNN_DATA_INT8x4. Note that this transformation will incur an extra round trip to memory.

The tensor pointers and the filter pointers require at a minimum 4-byte alignment, including INT8 data in the cuDNN library.

Some computational options in cuDNN 8.0.0 now require increased alignment on tensors in order to run efficiently. As always, cuDNN recommends users to align tensors to 128-bit boundaries that will be sufficiently aligned for any computational option in cuDNN. Doing otherwise may cause performance regressions in cuDNN 8.0.0 compared to cuDNN v7.6.

For certain algorithms, when the computation is in float (32-bit float) and the output is in FP16 (half float), there are cases where the numerical accuracy between the different algorithms might differ. cuDNN 8.0.0 users can target the backend API to query the numerical notes of the algorithms to get the information programmatically. There are cases where algo0 and algo1 will have a reduced precision accumulation when users target the legacy API. In all cases, these numerical differences are not known to affect training accuracy even though they might show up in unit tests.

Support for Ubuntu 14.04 has been deprecated in this release. Upgrade to 16.04 or 18.04 for continued support.

Support for Mac OS X has been deprecated in this release. Linux and Windows OS are currently supported.

cuDNN version 8 introduces a new API deprecation policy to enable a faster pace of innovation. A streamlined, two-step, deprecation policy will be used for all API changes starting with cuDNN version 8. For details about this new deprecation policy, see Backward Compatibility And Deprecation Policy in the cuDNN Developer Guide.

Removed and deprecated API changes. For a list of removed and deprecated APIs, see API Changes For cuDNN 8.0.0.

Performance regressions on V100 are observed in this release on SSD inference use cases if not using TensorRT.

There are significant performance regressions on pre-Volta GPUs and some NVIDIA Turing GPUs based on the TU102 architecture. This performance regression is not applicable to T4, NVIDIA JetPack, and NVIDIA Tegra.

Sub-optimal performance is present in this release for all INT8 convolutions for all GPUs.

The performance of cudnnConvolutionBiasActivationForward() is slower than v7.6 in most cases. This is being actively worked on and performance optimizations will be available in the upcoming releases.

On K80 GPUs, when cudnnConvolutionForward() is used with CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_PRECOMP_GEMM algorithm and half I/O data types a silent error might occur.

There are some peer-to-peer documentation links that are broken within the cuDNN API Reference. These links will be fixed in the next release.

cudnnCnnTrainVersionCheck() and cudnnCnnInferVersionCheck() are missing in this release and will be added in the GA release.

Documentation of RNN new APIs and deprecations is not complete. The cudnnRNNBackwardData_v8() and cudnnRNNBackwardWeights_v8() functions will be implemented in the next release.

cuDNN 8.0.0 Preview will not work with GA10x NVIDIA Ampere Architecture GPUs. This will be fixed in the next release.

cuDNN 8.0.0 Preview build with Windows and CUDA 11.0 RC has reduced performance on 2D, 3D, and grouped convolutions compared to Linux.

There is a known issue in cuDNN 8.0.0 when linking statically to cuDNN and using the library's 3D algo1 backward filter convolutions. Users will see that the libraries emit an internal error or incorrectly state that a shared library is missing. This is a bug that will be fixed in a future release.

There is a known issue in cuDNN 8.0.0 when linking statically to cuDNN and using the library's 3D algo1 backward filter convolutions. Users will see that the library emit an internal error or incorrectly state that a shared library is missing. This is a bug that will be fixed in a future release.

Steps 1 and 2 are required for the user to be able to switch between v7 and v8 installations. After steps 1 and 2 are performed one time, step 3 can be used repeatedly and the user can choose the appropriate cuDNN version to work with. For more information, refer to the Installing From An RPM File and Upgrading From v7 To v8 sections in the cuDNN Installation Guide.

cudnnConvolutionForward(), cudnnConvolutionBackwardData(), and cudnnConvolutionBackwardFilter() erroneously returns CUDNN_STATUS_INTERNAL_ERROR when the workspace size argument value is less than the required workspace size as returned by their respective cudnnGetWorkspace() API.

cuDNN convolution APIs may return CUDNN_STATUS_EXECUTION_FAILED when the number of input or output channels equals to or exceeds 2097152.