bridge.recipes.ernie.gb200.ernie55#
ERNIE 5.5 recipe on GB200: supervised fine-tuning of a Hugging Face checkpoint.
- func:
ernie55_sft_64gpu_gb200_bf16_configfine-tunes a Hugging Face ERNIE 5.5 checkpoint on 16 nodes of 4 GB200 GPUs. Its model, loss, optimizer, schedule, precision and tokenizer settings are those of the GB300 recipe (:mod:megatron.bridge.recipes.ernie.ernie55, whose module docstring describes them); only the shape differs, because the GB300 recipe’s shape needs more memory per GPU than a GB200 has: CP8 instead of CP64, 16384-token packed rows instead of one 262144-token sample per row, global batch 32 instead of 2, and a process group whose object collectives run on CPU. The function’s docstring gives the reason for each.
Like the GB300 recipe it requires Megatron-Core’s dev branch at commit f78809f6d or later and trains with
the ERNIE 5.5 step: scripts/training/run_recipe.py --recipe <name> --step_func ernie55_step.
Module Contents#
Functions#
Fine-tune a Hugging Face ERNIE 5.5 checkpoint on 16 nodes of 4 GB200 GPUs. |
Data#
API#
- bridge.recipes.ernie.gb200.ernie55.__all__#
[‘ernie55_sft_64gpu_gb200_bf16_config’]
- bridge.recipes.ernie.gb200.ernie55._GB200_CONTEXT_PARALLEL_SIZE#
8
- bridge.recipes.ernie.gb200.ernie55._GB200_SEQ_LENGTH#
16384
- bridge.recipes.ernie.gb200.ernie55.ernie55_sft_64gpu_gb200_bf16_config(
- *,
- hf_path: str | None = None,
Fine-tune a Hugging Face ERNIE 5.5 checkpoint on 16 nodes of 4 GB200 GPUs.
TP1 PP1 CP8 EP64 (DP8), micro-batch 1, global batch 32, 16384-token rows split contiguously over the 8 context-parallel ranks (
cp_partition_mode="contiguous"), 2065 steps with 103 warmup steps, with the model, loss and optimizer settings of- Func:
~megatron.bridge.recipes.ernie.ernie55.ernie55_sft_64gpu_gb300_bf16_config, the model built from the checkpoint’sconfig.jsonand starting from its weights (checkpoint.pretrained_checkpoint). What differs from that recipe, and why:
CP8, so DP8. With CP64 on 64 GPUs the data-parallel size is 1 and every GPU holds the whole optimizer state of the parameters that expert parallelism does not split. With CP8 the layer-wise distributed optimizer spreads that state over 8 data-parallel ranks.
16384-token rows. Every context-parallel rank holds 2048 of a row’s tokens. Longer rows have not been validated on GB200.
Packed rows. Samples shorter than a row share it (
dataset.pack, the reference’s--packing); samples longer than 16384 tokens are split into rows, as in the GB300 recipe. Each document is padded to a multiple of 16 tokens (2 x CP). Withoutdataset.pathevery row is one synthetic 16384-token document.Global batch 32. Four micro-batches on each of the 8 data-parallel ranks: 524288 tokens per step, as many as the GB300 recipe’s two 262144-token rows, so its schedule carries over unchanged.
Process group
cpu:gloo,cuda:nccl. CUDA tensors still go through NCCL; the object collectives of a distributed checkpoint save go through gloo. On an NCCL-only group the save leaves about 24 GiB of point-to-point buffers on rank 0 at 64 ranks, and the next training step runs out of memory there.
The final checkpoint holds the weights only, as the GB300 recipe’s does.
Needs Megatron-Core dev with the
_post_o_group_projhook: Megatron-Core main has no CP partition modes.- Parameters:
hf_path – Directory of the Hugging Face ERNIE 5.5 checkpoint (
config.jsonand safetensors); default$ERNIE55_HF_PATH. Required to train.- Returns:
The training config; run it with
--step_func ernie55_step(run_recipe.pyinfersfinetune). Withouthf_pathand$ERNIE55_HF_PATHit builds the placeholder- data:
~megatron.bridge.recipes.ernie.ernie55.ERNIE55_FULL_SIZE_HF_CONFIGwith no checkpoint, as Bridge’s recipe matrix requires; finetuning then stops in setup.