bridge.recipes.moonlight.h100.moonlight_16b#
Module Contents#
Functions#
Build the Moonlight architecture from its Hugging Face configuration. |
|
Build a checkpoint-compatible Moonlight provider for SFT or PEFT. |
|
Apply the shared bounded SFT/PEFT convergence contract. |
|
Get pipeline layout for Moonlight-16B based on PP and VP size. |
|
Return a pre-training config for Moonlight-16B. |
|
Return the bounded-convergence pre-training config for Moonlight-16B. |
|
Return the legacy TP4/PP2 full SFT config for Moonlight-16B. |
|
Return the bounded-convergence full SFT config for Moonlight-16B. |
|
Return the separate 8K-context SFT config for Moonlight-16B. |
|
Return a PEFT config for Moonlight-16B. |
|
Return the bounded-convergence PEFT config for Moonlight-16B. |
Data#
API#
- bridge.recipes.moonlight.h100.moonlight_16b._MOONLIGHT_16B_MODEL_ID#
âmoonshotai/Moonlight-16B-A3Bâ
- bridge.recipes.moonlight.h100.moonlight_16b._MOONLIGHT_16B_MODEL_REVISION#
â476b36a473d4467f94469414bef6cee75c9c8172â
- bridge.recipes.moonlight.h100.moonlight_16b._MOONLIGHT_16B_FINETUNING_UNPADDED_VOCAB_SIZE#
163842
- bridge.recipes.moonlight.h100.moonlight_16b._moonlight_16b_model_provider() megatron.bridge.models.mla_provider.MLAModelProvider#
Build the Moonlight architecture from its Hugging Face configuration.
- bridge.recipes.moonlight.h100.moonlight_16b._moonlight_16b_finetuning_model_provider(
- *,
- tensor_parallel_size: int,
- pipeline_parallel_size: int,
- context_parallel_size: int,
- expert_parallel_size: int,
- sequence_parallel: bool,
Build a checkpoint-compatible Moonlight provider for SFT or PEFT.
- bridge.recipes.moonlight.h100.moonlight_16b._apply_moonlight_16b_finetuning_convergence_contract(
- cfg: megatron.bridge.training.config.ConfigContainer,
Apply the shared bounded SFT/PEFT convergence contract.
- bridge.recipes.moonlight.h100.moonlight_16b._get_moonlight_pipeline_layout(pp_size: int, vp_size: int)#
Get pipeline layout for Moonlight-16B based on PP and VP size.
- bridge.recipes.moonlight.h100.moonlight_16b.moonlight_16b_pretrain_8gpu_h100_bf16_config() megatron.bridge.training.config.ConfigContainer#
Return a pre-training config for Moonlight-16B.
Recommended parallelism: TP=2, PP=1, EP=8 Uses precision-aware optimizer with bf16 gradients/moments.
- bridge.recipes.moonlight.h100.moonlight_16b.moonlight_16b_pretrain_16gpu_h100_bf16_config() megatron.bridge.training.config.ConfigContainer#
Return the bounded-convergence pre-training config for Moonlight-16B.
Recommended parallelism: TP=1, PP=1, CP=1, EP=8 on 16 H100 GPUs. The 100-step schedule uses GBS/MBS 1024/2 with 32-way gradient accumulation, two expert-data replicas, precision-aware bf16 optimizer state, and Moonlightâs model-native routing configuration.
- Returns:
ConfigContainer with the Moonlight-16B pre-training contract.
- bridge.recipes.moonlight.h100.moonlight_16b.moonlight_16b_sft_8gpu_h100_bf16_config() megatron.bridge.training.config.ConfigContainer#
Return the legacy TP4/PP2 full SFT config for Moonlight-16B.
Recommended parallelism: TP=4, PP=2, CP=1, EP=4 on 8 H100 GPUs. Sequence parallelism resolves offline packing to
pad_seq_to_mult=4, so this remains a support recipe rather than the shared pad-1 convergence cohort.- Returns:
ConfigContainer with all settings pre-configured for Moonlight-16B SFT.
- bridge.recipes.moonlight.h100.moonlight_16b.moonlight_16b_sft_8gpu_h100_bf16_tp1_config() megatron.bridge.training.config.ConfigContainer#
Return the bounded-convergence full SFT config for Moonlight-16B.
Recommended parallelism is TP=1, PP=1, CP=1, EP=8 on 8 H100 GPUs. This topology keeps sequence parallelism disabled so the shared SFT packing contract remains
pad_seq_to_mult=1. An 8K offline pack with GBS=8 and MBS=1 keeps the per-update token budget at 65,536 while dense DP=8 removes gradient accumulation.- Returns:
ConfigContainer with the Moonlight-16B full-SFT convergence contract.
- bridge.recipes.moonlight.h100.moonlight_16b.moonlight_16b_sft_8gpu_h100_bf16_8k_config() megatron.bridge.training.config.ConfigContainer#
Return the separate 8K-context SFT config for Moonlight-16B.
This recipe preserves the previously verified long-context execution and precision cohort instead of inheriting the 2K bounded-convergence SFT contract. Recommended parallelism is TP=2, PP=1, CP=2, EP=8 on 8 H100 GPUs.
- Returns:
ConfigContainer with the Moonlight-16B 8K-context SFT contract.
- bridge.recipes.moonlight.h100.moonlight_16b.moonlight_16b_peft_2gpu_h100_bf16_config(
- peft_scheme: str | megatron.bridge.peft.base.PEFT = 'lora',
Return a PEFT config for Moonlight-16B.
Default parallelism: TP=1, PP=1, EP=2, SP=False
- Parameters:
peft_scheme â PEFT scheme - âloraâ, âdoraâ, or a custom PEFT instance.
- Returns:
ConfigContainer with all settings pre-configured for Moonlight-16B PEFT.
- bridge.recipes.moonlight.h100.moonlight_16b.moonlight_16b_peft_4gpu_h100_bf16_config(
- peft_scheme: str | megatron.bridge.peft.base.PEFT = 'lora',
Return the bounded-convergence PEFT config for Moonlight-16B.
Recommended parallelism is TP=1, PP=1, CP=1, EP=4 on 4 H100 GPUs. This keeps sequence parallelism disabled and uses dense DP=4 with eight gradient accumulation steps. The default LoRA keeps the base model frozen and targets only the attention QKV and output projections with rank 8, alpha 16, and zero dropout.
- Parameters:
peft_scheme â PEFT scheme - âloraâ, âdoraâ, or a custom PEFT instance.
- Returns:
ConfigContainer with the Moonlight-16B PEFT convergence contract.
- bridge.recipes.moonlight.h100.moonlight_16b.__all__#
[âmoonlight_16b_peft_2gpu_h100_bf16_configâ, âmoonlight_16b_peft_4gpu_h100_bf16_configâ, âmoonlight_âŠ