Unified Megatron-LM#
Overview#
We present our motivation, proposal, and timeline to unify Megatron-LM and Megatron-Bridge into a single repository.
Background#
We have two repositories under the āMegatronā name: Megatron-LM and Megatron-Bridge.
Megatron-LM has two primary components:
training: training loops, utils, and configs for launching distributed training at scale
core: composable library with GPU-optimized building blocks for custom training frameworks
Nemotron Pretraining Example#
Megatron-Bridge also has two primary components:
training: training loops, utils, and configs for launching distributed training at scale
Distinct from Megatron-LMās training
bridge: bidirectional HuggingFace/Megatron conversion, recipes for all leading open-source models
Megatron-Bridge leverages its own components as well as Megatron-LMās training and core components to enable pretraining, SFT, and PEFT of open-source models.
Motivation#
Duplication#
Even though both megatron.bridge.training (~28,000 lines of code) and megatron.training (~24,000 lines of code) leverage megatron.core, they each have their own training loops, dataloaders, and checkpoints. This has led to API and feature drift, redundant efforts, and inconsistent behaviors.
Both training loops continue to drift apart#
CI Testing#
Multiple entry points have created unnecessary strain on CI. We have many similar tests and a similar release process wasting valuable compute and engineering hours.
User Challenges#
Challenging to debug across two repositories.
Confusing to contribute to two different repositories when wanting to add an optimization (optimization goes into Megatron-LM, but the recipe needs to be updated in Megatron-Bridge).
Struggling to update to Megatron-Core top-of-tree at SOL because of added Megatron-Bridge dependency.
Difficult to manage configuration and FSDP fixes because of inconsistencies between Megatron-LM and Megatron Bridge.
Frustrating that MCore and Megatron Bridge use different configuration formats
Difficult to maintain and debug training across Megatron Bridge and Megatron-LM: Bridge recipe arguments stopped propagating after NeMo container updates, and diagnosis crossed into Megatron-LM.
Difficult to debug checkpointing across Megatron Bridge and its embedded Megatron-LM dependency, including patching Megatron-LMās distributed optimizer inside the Bridge container and encountering different configuration defaults between environments.
RL libraries take components from both Megatron-Bridge and Megatron-LM#
Proposal#
There are two components to our proposal:
Consolidate Megatron-Bridgeās
traininginto Megatron-LMāstrainingproviding a single training loop, configs, dataloader, and checkpoints for all Megatron users.Copy the rest of Megatron-Bridge into Megatron-LM.
Final design of a unified Megatron-LM#
Existing users, such as verl, already use the megatron namespace when using Megatron-Bridge (i.e. they use import megatron.bridge). This usage pattern will not change post-migration. Users will also continue to use the existing megatron-bridge pip package.
Timeline#
December 2026 (Megatron-Core 0.21.0 release)#
Training loop will be fully unified in Megatron-LMās
megatron/training
February 2027 (Megatron-Core 0.22.0 release)#
Megatron-Bridge will be fully migrated to Megatron-LM, most of the files will be found under
megatron/bridge
PR Tracker: NVIDIA/Megatron-LM#9363
User Impact#
megatron.bridge.training#
We are fully removing the Bridge training loop and consolidating it into megatron/training. Users who currently use megatron/bridge/training will need to migrate.
Checkpoints#
We are unifying the two existing checkpoint formats into a Bridge-like checkpoint format. The āoldā checkpoints will be backwards-compatible and will continue to work. Checkpoints generated in the new format will not be supported by āoldā code.
Bridge Containers#
The Bridge container will continue to be published and used in the same manner.
PyPi#
The Bridge pip package will be combined with Megatron LM. Users will be able to run pip install megatron-lm to install Megatron Core, Bridge and Training packages.
Megatron-Bridge Issues#
Existing issues will remain open until resolved. Starting in Q1 ā27, all new issues will need to be created in Megatron-LM.
Megatron-Bridge PRs#
We will be freezing the Megatron-Bridge repository on Nov 30, 2026. All PRs not merged in by then will need to be migrated to Megatron-LM by the user.