Fine-Tune Large MoE LLMs

Introduction

Mixture-of-Experts (MoE) architectures have become the dominant design for frontier language models, activating only a fraction of their total parameters per token to deliver strong performance at reduced compute cost. This guide walks through fine-tuning four example MoE LLMs with NVIDIA NeMo Automodel. For a full list of supported architectures, see the LLM model coverage page.

To set up your environment to run NeMo Automodel, follow the installation guide.

Data

HellaSwag Dataset

All four recipes use the HellaSwag dataset, a commonsense natural language inference benchmark where the model must predict the most plausible continuation of a given scenario.

• Source: rowan/hellaswag
• Split: train (used for both training and validation in these recipes)
• Task: Next-token prediction on commonsense sentence completions

For details on how to swap in your own dataset, see the LLM Dataset Guide and the Dataset Overview.

Recipes

MiniMax-M2.5

examples/llm_finetune/minimax_m2/minimax_m2.5_hellaswag_pp.yaml — validated using 64 H100 GPUs (8 nodes x 8).

Key distributed settings:

1
distributed:
2
  strategy: fsdp2
3
  pp_size: 2
4
  ep_size: 32
5
  pipeline:
6
    pp_schedule: interleaved1f1b
7
    layers_per_stage: 2

GLM-5

examples/llm_finetune/glm/glm_5_hellaswag_pp.yaml — validated using 256 H100 GPUs (32 nodes x 8).

Key distributed settings:

1
distributed:
2
  strategy: fsdp2
3
  pp_size: 4
4
  ep_size: 64
5
  activation_checkpointing: true
6
  pipeline:
7
    pp_schedule: interleaved1f1b
8
    layers_per_stage: 2

Step-3.5 Flash (StepFun)

examples/llm_finetune/stepfun/step_3.5_flash_hellaswag_pp.yaml — validated using 64 H100 GPUs (8 nodes x 8).

Key distributed settings:

1
distributed:
2
  strategy: fsdp2
3
  pp_size: 2
4
  ep_size: 32
5
  pipeline:
6
    pp_schedule: interleaved1f1b
7
    layers_per_stage: 2

DeepSeek-V3.2

examples/llm_finetune/deepseek_v32/deepseek_v32_hellaswag_pp.yaml — validated using 256 H100 GPUs (32 nodes x 8).

Key distributed settings:

1
distributed:
2
  strategy: fsdp2
3
  pp_size: 4
4
  ep_size: 64
5
  activation_checkpointing: true
6
  pipeline:
7
    pp_schedule: interleaved1f1b
8
    layers_per_stage: 2

Launch Training

NeMo Automodel supports direct Slurm submission, local interactive runs through the Automodel CLI, and external torchrun. For full details on Slurm batch jobs, multi-node configuration, environment variables, and other launch options, see the Run on a Cluster guide.

Automodel CLI

$
uv run automodel your_single_node_config.yaml --nproc-per-node 8

The CLI takes the recipe config as its positional argument. Use this form only with a config whose distributed topology fits one node; the four configs above use multi-node PP/EP sizes and should be launched through Slurm or external torchrun as shown below.

torchrun

$
export TRANSFORMERS_OFFLINE=1
$
export HF_HOME=your/path/to/hf_cache
$
export HF_DATASETS_OFFLINE=1
$
export WANDB_API_KEY=your_wandb_key
$
export MASTER_ADDR=${MASTER_ADDR:?Set MASTER_ADDR to the rank-0 hostname}
$
export MASTER_PORT=${MASTER_PORT:-29500}
$
$
torchrun --nproc-per-node=8 \
>
         --nnodes=32 \
>
         --rdzv_backend=c10d \
>
         --rdzv_endpoint="${MASTER_ADDR}:${MASTER_PORT}" \
>
  -m nemo_automodel.recipes.llm.train_ft \
>
    -c examples/llm_finetune/glm/glm_5_hellaswag_pp.yaml \
>
    --model.pretrained_model_name_or_path=/your/local/model_weights

This command uses the GLM-5 config and its model.pretrained_model_name_or_path field. For another recipe, replace the -c path and --nnodes, then update the model and tokenizer path fields defined by that YAML.

Before you start:

• Hugging Face applies rate limits on downloads. We recommend cloning the model repository to your local filesystem beforehand.
• Ensure your Hugging Face cache (HF_HOME) is configured and that the dataset is already cached locally.
• To enable Weights & Biases logging, set your WANDB_API_KEY and configure the wandb section in the YAML file.