Large Language Models

To learn more about using NeMo to train Large Language Models at scale, please refer to the NeMo Framework User Guide.

• GPT-style models (decoder only)

• T5/BART/UL2-style models (encoder-decoder)

• BERT-style models (encoder only)

• RETRO model (decoder only)

• GPT Model Training
• Batching
• Positional embeddings
• Positional interpolation
• References
• Megatron Core Customization
• Reset Learning Rate
• Parameters
• Use Cases
• Ramp Up Batch Size
• Usage
• Ramp Up Stages and Training Interruption
• Automatic Node Scheduling
• Example

References