Release Notes – Release 0.6.0 (BETA)¶
Key Features and Enhancements¶
Added ONNX export support for Transformer Engine modules.
Added the ability to calibrate the Linear layer for FP8 from a higher-precision checkpoint.
Added a zero_centered_gamma
option to LayerNorm()
(and other methods using LayerNorm()
) for increased precision when the parameters of LayerNorm()
are stored in FP16
or BFloat16
.
Added DotProductAttention()
, with optimized implementations of attention, including integration with FlashAttention()
.
Optimized handling of separate QKV parameters in TransformerLayer()
.
Fixed Issues in This Release¶
Fixed an issue with gradients not being propagated properly when training under AMP.
Fixed an issue occurring for very large inputs in the custom Softmax kernel.
Known Issues in This Release¶
There are no known issues in this release.
Breaking Changes in This Release¶
There are no breaking changes in this release.
Deprecated Features¶
There are no deprecated features in this release.