bridge.recipes.ernie.ernie55#

ERNIE 5.5 recipe on GB300: supervised fine-tuning of a Hugging Face checkpoint.

This is an early SFT recipe and has not yet been optimized for training throughput or memory efficiency. It requires Megatron-Core’s dev branch at commit f78809f6d or later. Kernel fusion, expert communication, and activation recomputation settings may change as performance tuning progresses.

The recipe trains with the reference implementation’s settings: the per-row SFT loss (ernie_per_row_loss), MTP loss scale 0.1, the sparse DSA indexer loss at 0.01, no auxiliary or z loss (the z loss is still tracked), a router expert-bias update rate of 1e-4, full uniform recompute of every layer, no fused kernels besides the grouped expert GEMM and the reference’s own Triton kernels for SiTU-GLU, multimax and the context-parallel RoPE (ernie_fused_ops, which model.ernie_fused_ops=false turns off), the cuDNN DSA backend, and dist_muon (Muon with the layer-wise distributed optimizer; AdamW for the embedding, the output layer, the router matrices, the compressors’ APE and every 1-D parameter; the reference’s per-head and gate/up Muon row blocks are per-parameter qkv_layout blocks that Muon orthogonalizes one by one under muon_split_qkv, see :mod:~megatron.bridge.models.ernie.modeling_ernie55.muon) with weight decay 0.1 on every matrix, a 2e-5 to 2e-6 cosine schedule and gradient clipping at 1.0 (the multimax head is clipped by its own norm). Train it with the ERNIE 5.5 step, which builds the packed THD rows and selects the per-row loss: scripts/training/run_recipe.py --recipe <name> --step_func ernie55_step.

func:

ernie55_sft_64gpu_gb300_bf16_config fine-tunes a Hugging Face ERNIE 5.5 checkpoint (config.json, safetensors and tokenizer files) at the reference’s production shape, CP64 EP64 at 256K tokens per row on 16 nodes of 4 GPUs. It runs Bridge’s finetune loop, which run_recipe.py infers from its name; it loads the checkpoint of hf_path (or $ERNIE55_HF_PATH) as checkpoint.pretrained_checkpoint and reads its architecture. Like every Bridge recipe it builds without arguments; without hf_path and $ERNIE55_HF_PATH it uses the placeholder

data:

ERNIE55_FULL_SIZE_HF_CONFIG, which is not ERNIE 5.5’s layout, and finetuning then stops in setup.

The tokenizer is NullTokenizer with the model’s unpadded vocabulary; the data is SFT JSONL (dataset.path, text rows with dataset.hf_tokenizer or pre-tokenized rows) or synthetic rows (:class:~megatron.bridge.models.ernie.ernie55_data.Ernie55SFTDatasetConfig).

Module Contents#

Functions#

_ernie55_hf_config

The ERNIE 5.5 config.json of hf_path, of $ERNIE55_HF_PATH, or :data:ERNIE55_FULL_SIZE_HF_CONFIG.

_ernie55_common_config

Model, loss, optimizer, schedule, DDP, precision and tokenizer settings of the ERNIE 5.5 recipe.

_ernie55_production_config

The reference’s production shape on 16 nodes: CP64 EP64, 256K-token rows, the reference’s schedule.

ernie55_sft_64gpu_gb300_bf16_config

Fine-tune a Hugging Face ERNIE 5.5 checkpoint at the reference’s production shape on 16 nodes of 4 GB300 GPUs.

Data#

__all__

ERNIE55_HF_PATH_ENV

Environment variable naming the config.json source when a recipe gets no hf_path.

ERNIE55_FULL_SIZE_HF_CONFIG

A placeholder full-size config.json, not taken from a released checkpoint, that the recipe builds when it gets no hf_path: 21 alternating H and D layers, one more D layer and one shared-body D-layer MTP depth, all MoE with 256 routed experts (DeepSeek-V4-Flash backbone sizes), with the reference’s vocabulary and attention geometry. Pass hf_path to build the model from a real checkpoint’s config.json.

_PRODUCTION_SEQ_LENGTH

_SYNTHETIC_TOKEN_RANGE

API#

bridge.recipes.ernie.ernie55.__all__#

[‘ERNIE55_FULL_SIZE_HF_CONFIG’, ‘ERNIE55_HF_PATH_ENV’, ‘ernie55_sft_64gpu_gb300_bf16_config’]

bridge.recipes.ernie.ernie55.ERNIE55_HF_PATH_ENV#

‘ERNIE55_HF_PATH’

Environment variable naming the config.json source when a recipe gets no hf_path.

bridge.recipes.ernie.ernie55.ERNIE55_FULL_SIZE_HF_CONFIG: dict[str, Any]#

None

A placeholder full-size config.json, not taken from a released checkpoint, that the recipe builds when it gets no hf_path: 21 alternating H and D layers, one more D layer and one shared-body D-layer MTP depth, all MoE with 256 routed experts (DeepSeek-V4-Flash backbone sizes), with the reference’s vocabulary and attention geometry. Pass hf_path to build the model from a real checkpoint’s config.json.

bridge.recipes.ernie.ernie55._PRODUCTION_SEQ_LENGTH#

262144

bridge.recipes.ernie.ernie55._SYNTHETIC_TOKEN_RANGE#

(1, 151643)

bridge.recipes.ernie.ernie55._ernie55_hf_config(
hf_path: str | None,
) → tuple[transformers.configuration_utils.PretrainedConfig, str | None]#

The ERNIE 5.5 config.json of hf_path, of $ERNIE55_HF_PATH, or :data:ERNIE55_FULL_SIZE_HF_CONFIG.

Returns:

The Hugging Face config and the path it was read from (None for the built-in config).

bridge.recipes.ernie.ernie55._ernie55_common_config(
hf_config: transformers.configuration_utils.PretrainedConfig,
*,
seq_length: int,
context_parallel_size: int,
expert_model_parallel_size: int,
train_iters: int,
lr_warmup_iters: int,
global_batch_size: int,
) → megatron.bridge.training.config.ConfigContainer#

Model, loss, optimizer, schedule, DDP, precision and tokenizer settings of the ERNIE 5.5 recipe.

Data, checkpointing and the process environment are left to the recipe.

bridge.recipes.ernie.ernie55._ernie55_production_config(
hf_config: transformers.configuration_utils.PretrainedConfig,
) → megatron.bridge.training.config.ConfigContainer#

The reference’s production shape on 16 nodes: CP64 EP64, 256K-token rows, the reference’s schedule.

bridge.recipes.ernie.ernie55.ernie55_sft_64gpu_gb300_bf16_config(
*,
hf_path: str | None = None,
) → megatron.bridge.training.config.ConfigContainer#

Fine-tune a Hugging Face ERNIE 5.5 checkpoint at the reference’s production shape on 16 nodes of 4 GB300 GPUs.

TP1 PP1 CP64 EP64 (DP1), micro-batch 1, global batch 2, 262144-token rows split contiguously over the 64 context-parallel ranks (cp_partition_mode="contiguous", the DSv4 layout), 2065 steps with 103 warmup steps, the reference’s model, loss and optimizer settings (see the module docstring), with the model built from the checkpoint’s config.json and starting from its weights (checkpoint.pretrained_checkpoint). The data are SFT JSONL (dataset.path; text rows need dataset.hf_tokenizer), one sample per row, samples longer than 262144 tokens split into rows (the reference’s --packing false --truncation_strategy split); without a path, every row is one synthetic 262144-token document. Each document is padded to a multiple of 128 tokens (2 x CP). The final checkpoint holds the weights only, as the reference’s does.

Needs Megatron-Core dev with the _post_o_group_proj hook: Megatron-Core main has no CP partition modes.

Parameters:

hf_path – Directory of the Hugging Face ERNIE 5.5 checkpoint (config.json and safetensors); default $ERNIE55_HF_PATH. Required to train.

Returns:

The training config; run it with --step_func ernie55_step (run_recipe.py infers finetune). Without hf_path and $ERNIE55_HF_PATH it builds the placeholder :data:ERNIE55_FULL_SIZE_HF_CONFIG with no checkpoint, as Bridge’s recipe matrix requires; finetuning then stops in setup. Pass hf_path or set $ERNIE55_HF_PATH so the model is built from the checkpoint’s own config.json.