bridge.recipes.ernie.ernie55#
ERNIE 5.5 recipe on GB300: supervised fine-tuning of a Hugging Face checkpoint.
This is an early SFT recipe and has not yet been optimized for training throughput or memory efficiency.
It requires Megatron-Core’s dev branch at commit f78809f6d or later. Kernel fusion, expert
communication, and activation recomputation settings may change as performance tuning progresses.
The recipe trains with the reference implementation’s settings: the per-row SFT loss
(ernie_per_row_loss), MTP loss scale 0.1, the sparse DSA indexer loss at 0.01, no auxiliary or z loss
(the z loss is still tracked), a router expert-bias update rate of 1e-4, full uniform recompute of every
layer, no fused kernels besides the grouped expert GEMM and the reference’s own Triton kernels for SiTU-GLU,
multimax and the context-parallel RoPE (ernie_fused_ops, which model.ernie_fused_ops=false turns off),
the cuDNN DSA backend, and dist_muon (Muon with
the layer-wise distributed optimizer; AdamW for the embedding, the output layer, the router matrices, the
compressors’ APE and every 1-D parameter; the reference’s per-head and gate/up Muon row blocks are per-parameter
qkv_layout blocks that Muon orthogonalizes one by one under muon_split_qkv,
see :mod:~megatron.bridge.models.ernie.modeling_ernie55.muon) with
weight decay 0.1 on every matrix, a 2e-5 to 2e-6 cosine schedule and gradient clipping at 1.0 (the multimax
head is clipped by its own norm). Train it with the ERNIE 5.5 step, which builds the packed THD rows and
selects the per-row loss: scripts/training/run_recipe.py --recipe <name> --step_func ernie55_step.
- func:
ernie55_sft_64gpu_gb300_bf16_configfine-tunes a Hugging Face ERNIE 5.5 checkpoint (config.json, safetensors and tokenizer files) at the reference’s production shape, CP64 EP64 at 256K tokens per row on 16 nodes of 4 GPUs. It runs Bridge’sfinetuneloop, whichrun_recipe.pyinfers from its name; it loads the checkpoint ofhf_path(or$ERNIE55_HF_PATH) ascheckpoint.pretrained_checkpointand reads its architecture. Like every Bridge recipe it builds without arguments; withouthf_pathand$ERNIE55_HF_PATHit uses the placeholder- data:
ERNIE55_FULL_SIZE_HF_CONFIG, which is not ERNIE 5.5’s layout, and finetuning then stops in setup.
The tokenizer is NullTokenizer with the model’s unpadded vocabulary; the data is SFT JSONL (dataset.path,
text rows with dataset.hf_tokenizer or pre-tokenized rows) or synthetic rows
(:class:~megatron.bridge.models.ernie.ernie55_data.Ernie55SFTDatasetConfig).
Module Contents#
Functions#
The ERNIE 5.5 |
|
Model, loss, optimizer, schedule, DDP, precision and tokenizer settings of the ERNIE 5.5 recipe. |
|
The reference’s production shape on 16 nodes: CP64 EP64, 256K-token rows, the reference’s schedule. |
|
Fine-tune a Hugging Face ERNIE 5.5 checkpoint at the reference’s production shape on 16 nodes of 4 GB300 GPUs. |
Data#
Environment variable naming the |
|
A placeholder full-size |
|
API#
- bridge.recipes.ernie.ernie55.__all__#
[‘ERNIE55_FULL_SIZE_HF_CONFIG’, ‘ERNIE55_HF_PATH_ENV’, ‘ernie55_sft_64gpu_gb300_bf16_config’]
- bridge.recipes.ernie.ernie55.ERNIE55_HF_PATH_ENV#
‘ERNIE55_HF_PATH’
Environment variable naming the
config.jsonsource when a recipe gets nohf_path.
- bridge.recipes.ernie.ernie55.ERNIE55_FULL_SIZE_HF_CONFIG: dict[str, Any]#
None
A placeholder full-size
config.json, not taken from a released checkpoint, that the recipe builds when it gets nohf_path: 21 alternating H and D layers, one more D layer and one shared-body D-layer MTP depth, all MoE with 256 routed experts (DeepSeek-V4-Flash backbone sizes), with the reference’s vocabulary and attention geometry. Passhf_pathto build the model from a real checkpoint’sconfig.json.
- bridge.recipes.ernie.ernie55._PRODUCTION_SEQ_LENGTH#
262144
- bridge.recipes.ernie.ernie55._SYNTHETIC_TOKEN_RANGE#
(1, 151643)
- bridge.recipes.ernie.ernie55._ernie55_hf_config(
- hf_path: str | None,
The ERNIE 5.5
config.jsonofhf_path, of$ERNIE55_HF_PATH, or :data:ERNIE55_FULL_SIZE_HF_CONFIG.- Returns:
The Hugging Face config and the path it was read from (
Nonefor the built-in config).
- bridge.recipes.ernie.ernie55._ernie55_common_config(
- hf_config: transformers.configuration_utils.PretrainedConfig,
- *,
- seq_length: int,
- context_parallel_size: int,
- expert_model_parallel_size: int,
- train_iters: int,
- lr_warmup_iters: int,
- global_batch_size: int,
Model, loss, optimizer, schedule, DDP, precision and tokenizer settings of the ERNIE 5.5 recipe.
Data, checkpointing and the process environment are left to the recipe.
- bridge.recipes.ernie.ernie55._ernie55_production_config(
- hf_config: transformers.configuration_utils.PretrainedConfig,
The reference’s production shape on 16 nodes: CP64 EP64, 256K-token rows, the reference’s schedule.
- bridge.recipes.ernie.ernie55.ernie55_sft_64gpu_gb300_bf16_config(
- *,
- hf_path: str | None = None,
Fine-tune a Hugging Face ERNIE 5.5 checkpoint at the reference’s production shape on 16 nodes of 4 GB300 GPUs.
TP1 PP1 CP64 EP64 (DP1), micro-batch 1, global batch 2, 262144-token rows split contiguously over the 64 context-parallel ranks (
cp_partition_mode="contiguous", the DSv4 layout), 2065 steps with 103 warmup steps, the reference’s model, loss and optimizer settings (see the module docstring), with the model built from the checkpoint’sconfig.jsonand starting from its weights (checkpoint.pretrained_checkpoint). The data are SFT JSONL (dataset.path; text rows needdataset.hf_tokenizer), one sample per row, samples longer than 262144 tokens split into rows (the reference’s--packing false --truncation_strategy split); without a path, every row is one synthetic 262144-token document. Each document is padded to a multiple of 128 tokens (2 x CP). The final checkpoint holds the weights only, as the reference’s does.Needs Megatron-Core dev with the
_post_o_group_projhook: Megatron-Core main has no CP partition modes.- Parameters:
hf_path – Directory of the Hugging Face ERNIE 5.5 checkpoint (
config.jsonand safetensors); default$ERNIE55_HF_PATH. Required to train.- Returns:
The training config; run it with
--step_func ernie55_step(run_recipe.pyinfersfinetune). Withouthf_pathand$ERNIE55_HF_PATHit builds the placeholder :data:ERNIE55_FULL_SIZE_HF_CONFIGwith no checkpoint, as Bridge’s recipe matrix requires; finetuning then stops in setup. Passhf_pathor set$ERNIE55_HF_PATHso the model is built from the checkpoint’s ownconfig.json.