Core APIs

Low-level API reference for core Megatron components.

• transformer package
• tensor_parallel package
• Generalized Tensor Parallelism (GTP)
  • 1. Features
  • 2. Usage
  • 3. Implementation details
  • 4. Testing
• pipeline_parallel package
• fusions package
• distributed package
• datasets package
• Data Pipeline
  • Data pre-processing
  • Data loading: construction
  • Data loading: implementation
  • Offline cache preparation
  • Fast DataLoader initialization
• dist_checkpointing package
  • Safe Checkpoint Loading
  • Checkpointing Distributed Optimizer
  • Distributed Optimizer Checkpoint Formats
  • Async Checkpoint Saving Strategy
  • Subpackages
• dist_checkpointing.strategies package