Important
NeMo 2.0 is an experimental feature and currently released in the dev container only: nvcr.io/nvidia/nemo:dev. Please refer to the Migration Guide for information on getting started.
The Bridge Between Lightning and Megatron Core
In PyTorch Lightning, a Strategy is responsible for managing the distributed execution of a model during training, validation, and testing. Strategies typically wrap the user-defined model with a class that can handle distributed execution. For instance, the standard DDPStrategy (Distributed Data Parallel Strategy) wraps the model with PyTorch’s DistributedDataParallel class. This wrapper handles the distribution of data across multiple GPUs or nodes, synchronizes gradients during the backward pass, and ensures that model parameters remain consistent across all processes. Strategies in Lightning abstract away much of the complexity of distributed training, allowing users to focus on their model architecture and training logic while the framework handles the intricacies of distributed execution.
MegatronStrategy
The MegatronStrategy is a PyTorch Lightning strategy that enables distributed training of large language models using NVIDIA’s Megatron-LM framework. It’s designed to handle models that exceed the memory capacity of a single GPU by implementing various forms of model parallelism.
To use the MegatronStrategy, you initialize it with parameters that define the parallelism setup:
from nemo import lightning as nl
strategy = nl.MegatronStrategy(
tensor_model_parallel_size=2,
pipeline_model_parallel_size=2,
virtual_pipeline_model_parallel_size=None,
context_parallel_size=1,
sequence_parallel=False,
expert_model_parallel_size=1,
)
These parameters determine how the model will be split across available GPUs. The strategy then sets up the necessary distributed environment, initializing process groups for each type of parallelism.
The strategy is also responsible for configuring the checkpoint IO interface that handles saving and loading checkpoints. For a full list of options that can be configured via MegatronStrategy, refer to the documentation.
When you create your PyTorch Lightning Trainer, you pass this strategy:
trainer = nl.Trainer(strategy=strategy, devices=8, accelerator="gpu")
The MegatronStrategy utilizes Megatron’s distributed checkpointing system for model I/O. This system efficiently manages checkpoints for models partitioned across multiple GPUs, maintaining consistency across various parallelism configurations. It enables correct model reconstruction even when GPU setups differ between saving and loading.
The MegatronStrategy wraps the user-defined training_step, validation_step, and test_step methods to make them compatible with Megatron’s forward-backward pass implementation. This wrapping process allows these steps to be executed within the context of Megatron’s distributed execution framework, ensuring that all forms of parallelism are properly handled during each phase of the training loop. By doing this, the strategy maintains the familiar PyTorch Lightning interface for users while seamlessly integrating the complex distributed operations required for large-scale model training.
The MegatronStrategy
employs the MegatronParallel
class to manage the distributed execution of the user-defined model. This class breaks down the execution process into three key steps:
Data Step: Prepares and distributes the input data across the model parallel groups.
Forward Step: Executes the forward pass across the partitioned model.
Loss Reduction: Computes and reduces the loss across the distributed setup.
MegatronParallel utilizes these steps to perform the forward-backward pass, which is derived from the user-defined training_step
, validation_step
, and test_step
methods. It orchestrates the flow of data and gradients through the partitioned model, manages inter-GPU communication, and ensures proper gradient synchronization. This approach enables efficient execution across multiple GPUs while preserving the logical structure of the user’s Lightning module.
MegatronParallel
The MegatronParallel
class is the core component that implements distributed model parallelism in the Megatron-LM framework. It manages the execution of the model across multiple GPUs, breaking down the process into three key steps:
Data Step: This step prepares and distributes the input data across the model parallel groups. For the GPT model, it uses the
gpt_data_step
function:def data_step(self, dataloader_iter): return gpt_data_step(dataloader_iter)
This function handles:
Fetching a batch from the dataloader
Moving required tensors to CUDA
Slicing the batch for context parallelism using
get_batch_on_this_context_parallel_rank
Preparing packed sequence parameters if necessary
Forward Step: This step executes the forward pass across the partitioned model. For the GPT model, it uses the
gpt_forward_step
function:def forward_step(self, model, batch): return gpt_forward_step(model, batch)
This function:
Prepares the forward arguments from the batch
Calls the model’s forward method with these arguments
Handles both standard and packed sequence inputs
Loss Reduction: After the forward pass, this step computes and reduces the loss across the distributed setup. The GPT model uses
MaskedTokenLossReduction
:def loss_reduction(self, model): return model.training_loss_reduction()
For validation:
def validation_loss_reduction(self, model): return model.validation_loss_reduction()
These methods handle:
Calculating the loss using masked token loss
Reducing the loss across data parallel groups
Handling special cases for validation (e.g., not dropping the last batch)
The MegatronParallel
class orchestrates these steps to perform the complete forward-backward pass.
By using these model-specific functions, MegatronParallel
allows the GPT model to define its own data processing, forward pass, and loss calculation logic while still benefiting from the distributed execution framework. This approach enables researchers and engineers to work with large language models using familiar PyTorch Lightning interfaces, while the underlying distributed execution is handled transparently.
MegatronMixedPrecision
The MegatronMixedPrecision class is a specialized precision plugin for Megatron-LM models in PyTorch Lightning. It extends the standard MixedPrecision plugin to handle the specific requirements of large language models trained with Megatron-LM.
from nemo import lightning as nl
precision = nl.MegatronMixedPrecision(precision="bf16-mixed")
trainer = nl.Trainer(strategy=strategy, plugins=precision)