Important

NeMo 2.0 is an experimental feature and currently released in the dev container only: nvcr.io/nvidia/nemo:dev. Please refer to the Migration Guide for information on getting started.

Optimizations

• Attention Optimizations
• Sequence Packing
• Activation Recomputation
• Communication Overlap
• CPU Offloading