Transformer Engine
0.1.0
Home
Getting Started
Installation
Prerequisites
Transformer Engine in NGC Containers
pip - from GitHub
Additional Prerequisites
Installation (stable release)
Installation (development build)
Getting Started
Overview
Let’s build a Transformer layer!
Meet Transformer Engine
Fused TE Modules
Enabling FP8
Python API documentation
Common API
Classes
Framework-specific API
pyTorch
Modules
Functions
Examples and Tutorials
Using FP8 with Transformer Engine
Introduction to FP8
Structure
Mixed precision training - a quick introduction
Mixed precision training with FP8
Using FP8 with Transformer Engine
FP8 recipe
FP8 autocasting
Handling backward pass
Precision
Advanced
C/C++ API
activation.h
cast.h
gemm.h
layer_norm.h
transformer_engine.h
transpose.h
Transformer Engine
»
Index
Index
D
|
F
|
L
|
N
|
T
D
DelayedScaling (class in transformer_engine.common.recipe)
F
Format (class in transformer_engine.common.recipe)
forward() (transformer_engine.pytorch.LayerNormLinear method)
(transformer_engine.pytorch.LayerNormMLP method)
(transformer_engine.pytorch.Linear method)
(transformer_engine.pytorch.TransformerLayer method)
fp8_autocast() (in module transformer_engine.pytorch)
L
LayerNorm (class in transformer_engine.pytorch)
LayerNormLinear (class in transformer_engine.pytorch)
LayerNormMLP (class in transformer_engine.pytorch)
Linear (class in transformer_engine.pytorch)
N
nvte_cast_transpose (C++ function)
nvte_cast_transpose_dbias (C++ function)
nvte_cast_transpose_dbias_dgelu (C++ function)
nvte_create_tensor (C++ function)
nvte_cublas_gemm (C++ function)
nvte_destroy_tensor (C++ function)
nvte_fp8_dequantize (C++ function)
nvte_fp8_quantize (C++ function)
nvte_gelu (C++ function)
nvte_layernorm_bwd (C++ function)
nvte_layernorm_fwd (C++ function)
nvte_tensor_data (C++ function)
nvte_tensor_shape (C++ function)
nvte_tensor_type (C++ function)
nvte_transpose (C++ function)
NVTEDType (C++ enum)
NVTEDType::kNVTEBFloat16 (C++ enumerator)
NVTEDType::kNVTEByte (C++ enumerator)
NVTEDType::kNVTEFloat16 (C++ enumerator)
NVTEDType::kNVTEFloat32 (C++ enumerator)
NVTEDType::kNVTEFloat8E4M3 (C++ enumerator)
NVTEDType::kNVTEFloat8E5M2 (C++ enumerator)
NVTEDType::kNVTEInt32 (C++ enumerator)
NVTEDType::kNVTENumTypes (C++ enumerator)
NVTEShape (C++ struct)
NVTEShape::data (C++ member)
NVTEShape::ndim (C++ member)
NVTETensor (C++ type)
T
transformer_engine (C++ type)
transformer_engine::DType (C++ enum)
transformer_engine::DType::kBFloat16 (C++ enumerator)
transformer_engine::DType::kByte (C++ enumerator)
transformer_engine::DType::kFloat16 (C++ enumerator)
transformer_engine::DType::kFloat32 (C++ enumerator)
transformer_engine::DType::kFloat8E4M3 (C++ enumerator)
transformer_engine::DType::kFloat8E5M2 (C++ enumerator)
transformer_engine::DType::kInt32 (C++ enumerator)
transformer_engine::DType::kNumTypes (C++ enumerator)
transformer_engine::TensorWrapper (C++ struct)
transformer_engine::TensorWrapper::data (C++ function)
transformer_engine::TensorWrapper::dptr (C++ function)
transformer_engine::TensorWrapper::dtype (C++ function)
transformer_engine::TensorWrapper::operator= (C++ function)
,
[1]
transformer_engine::TensorWrapper::shape (C++ function)
transformer_engine::TensorWrapper::tensor_ (C++ member)
transformer_engine::TensorWrapper::TensorWrapper (C++ function)
,
[1]
,
[2]
,
[3]
,
[4]
transformer_engine::TensorWrapper::~TensorWrapper (C++ function)
TransformerLayer (class in transformer_engine.pytorch)