Transformer models are the backbone of language models from BERT to GPT-3 and require enormous computing resources. Transformer Engine is a library for accelerating Transformer models on NVIDIA’s newest generation of GPUs. Within this lab, we will show a variety of examples that show how TE has been integrated with PyTorch. TE will be integrated with more deep learning libraries in the future.
The TE APIs allow for existing training pipelines to benefit from the significant speedups with minimal changes to the current code.
To get started with Building and Extending Transformer API support for PyTorch, follow the steps below.
Using the Jupyter Notebook link on the left-hand navigation pane, open the Jupyter Notebook.
Run through the notebook
fp8_primer.ipynbto learn how to use the new datatype, FP8, with TE.Run through the notebook
Transformer-engine.ipynbto see the TE in action with a variety of examples.
To run a cell on the Jupyter Notebook, click on the cell you want to run and press enter Shift + Enter. Linux bash commands can be run inside the Jupyter Notebook by adding a bang symbol (!) before the command inside the Jupyter Notebook cell.