Is this page helpful?

Megatron Discussions#

This directory contains in-depth guides, tutorials, and discussions about optimizing and using Megatron for various use cases.

Available Guides#

Training Guides#

Previous News#

  • [2025/05] Megatron Core v0.11.0 brings new capabilities for multi-data center LLM training (blog).

  • [2024/07] Megatron Core v0.7 improves scalability and training resiliency and adds support for multimodal training (blog).

  • [2024/06] Megatron Core added support for Mamba-based models. Review the paper An Empirical Study of Mamba-based Language Models and code example.

  • [2024/01 Announcement] NVIDIA has released the core capabilities in Megatron-LM into Megatron Core in this repository. Megatron Core expands upon Megatron-LM’s GPU-optimized techniques with more cutting-edge innovations on system-level optimizations, featuring composable and modular APIs.

Contributing#

To contribute a guide or tutorial, follow this structure:

  1. Create a new directory: docs/discussions/your-guide-name/

  2. Add your main guide: docs/discussions/your-guide-name/your-guide-name.md

  3. Create an images directory: docs/discussions/your-guide-name/images/

  4. Update this README.md with a link to your guide

Each guide should be self-contained with its own images and supporting files.