bridge.recipes.ernie.h100.ernie55#
ERNIE 5.5 recipe on H100: supervised fine-tuning of a Hugging Face checkpoint.
- func:
ernie55_sft_128gpu_h100_bf16_configtrains with the model, loss, optimizer and precision settings of- mod:
megatron.bridge.recipes.ernie.ernie55and changes what an 80 GB GPU needs: eight pipeline stages (exchanging fp32) and EP16 on 16 nodes of 8 GPUs, packed rows of at most 4096 tokens, a global batch of 32 rows, and four times the steps. Like the GB300 recipe it runs with the ERNIE 5.5 step (scripts/training/run_recipe.py --recipe <name> --step_func ernie55_step) and requires Megatron-Core’sdevbranch at commitf78809f6dor later.
The recipe keeps the cuDNN DSA backend, whose sparse forward is FlashMLA’s. The environment therefore needs a
FlashMLA build with its SM90 kernels; docker/Dockerfile.ci builds FlashMLA with FLASH_MLA_DISABLE_SM90=1,
so an image for H100 must build it without that variable (see examples/models/ernie55/README.md).
Module Contents#
Functions#
Fine-tune a Hugging Face ERNIE 5.5 checkpoint on 16 nodes of 8 H100 GPUs. |
Data#
API#
- bridge.recipes.ernie.h100.ernie55.__all__#
[‘ernie55_sft_128gpu_h100_bf16_config’]
- bridge.recipes.ernie.h100.ernie55._H100_SEQ_LENGTH#
4096
- bridge.recipes.ernie.h100.ernie55._H100_GLOBAL_BATCH_SIZE#
32
- bridge.recipes.ernie.h100.ernie55._H100_TRAIN_ITERS#
8260
- bridge.recipes.ernie.h100.ernie55._H100_LR_WARMUP_ITERS#
412
- bridge.recipes.ernie.h100.ernie55.ernie55_sft_128gpu_h100_bf16_config(
- *,
- hf_path: str | None = None,
Fine-tune a Hugging Face ERNIE 5.5 checkpoint on 16 nodes of 8 H100 GPUs.
TP1 PP8 CP1 EP16 (DP16), micro-batch 1, global batch 32, rows of at most 4096 tokens, 8260 steps with 412 warmup steps (the GB300 recipe’s schedule in tokens of full rows), the reference’s model, loss and optimizer settings (see :mod:
megatron.bridge.recipes.ernie.ernie55), with the model built from the checkpoint’sconfig.jsonand starting from its weights (checkpoint.pretrained_checkpoint). The data are SFT JSONL (dataset.path; text rows needdataset.hf_tokenizer), several samples packed into a row, samples longer than 4096 tokens split into rows; without a path, every row is one synthetic 4096-token document. The final checkpoint holds the weights only, as the GB300 recipe’s does.- Parameters:
hf_path – Directory of the Hugging Face ERNIE 5.5 checkpoint (
config.jsonand safetensors); default$ERNIE55_HF_PATH. Required to train.- Returns:
The training config; run it with
--step_func ernie55_step(run_recipe.pyinfersfinetune). Withouthf_pathand$ERNIE55_HF_PATHit builds the placeholder- data:
~megatron.bridge.recipes.ernie.ernie55.ERNIE55_FULL_SIZE_HF_CONFIGwith no checkpoint, as Bridge’s recipe matrix requires; finetuning then stops in setup.