bridge.recipes.muse_spark_open.gb200.muse_spark_open#
Bounded GB200 training recipes for Muse Spark Open.
These are early recipes: they are functional but not yet performance-optimized. The pretraining and full-SFT recipes fit the model with full activation recompute.
Each recipe is self-contained: it sets every model, data, and training option it relies on, even where that repeats another recipe.
Module Contents#
Functions#
Return a 100-step bounded text pretraining config on 192 GB200 GPUs. |
|
Return a 100-step bounded text pretraining config on 256 GB200 GPUs with eight pipeline stages. |
|
Return a one-epoch full SFT config on 100K packed Tulu 3 conversations with 192 GB200 GPUs. |
|
Return a 100-step full SFT config on packed Tulu 3 data with 256 GB200 GPUs and eight pipeline stages. |
|
Return a 20-step packed 32K Tulu 3 full SFT config with context parallelism on 192 GB200 GPUs. |
|
Return a 20-step packed 64K CoderForge agent-trajectory full SFT config with CP4 on 192 GB200 GPUs. |
|
Return a one-epoch LoRA config on 100K packed Tulu 3 conversations with 64 GB200 GPUs. |
|
Return a 100-step attention-projection LoRA config on image-text CORD-v2 data with 16 GB200 GPUs. |
|
Return a 100-step full SFT config on image-text CORD-v2 data with 192 GB200 GPUs. |
Data#
API#
- bridge.recipes.muse_spark_open.gb200.muse_spark_open._MODEL_ID#
‘meta-models/Muse-Spark-Open-816B-A42B’
- bridge.recipes.muse_spark_open.gb200.muse_spark_open._CORD_V2_REVISION#
‘7f0115a4b758a71d6473b8d085751692da2fef98’
- bridge.recipes.muse_spark_open.gb200.muse_spark_open._TULU3_REVISION#
‘b14afda60f1bbebe55d5d2fa1e4df5042f97f8be’
- bridge.recipes.muse_spark_open.gb200.muse_spark_open._CODERFORGE_REVISION#
‘060fca96cf723b2ebab3181e9e59fafd273df3cb’
- bridge.recipes.muse_spark_open.gb200.muse_spark_open.muse_spark_open_pretrain_192gpu_gb200_bf16_config() megatron.bridge.training.config.ConfigContainer#
Return a 100-step bounded text pretraining config on 192 GB200 GPUs.
TP1/PP6/EP32 gives every GPU a unique slice of the experts. The data is text-only, so the perception towers stay frozen rather than receiving only weight decay. Pass the imported checkpoint with
--pretrained_checkpointto start from natural expert routing; the recipe leaves it unset.The peak learning rate is 3e-5, decaying to 3e-6, instead of the bounded pretraining contract’s from-scratch 3e-4: at 3e-4 the warm-started model’s loss rises until the learning rate decays, while at 3e-5 it falls.
- bridge.recipes.muse_spark_open.gb200.muse_spark_open.muse_spark_open_pretrain_256gpu_gb200_bf16_config() megatron.bridge.training.config.ConfigContainer#
Return a 100-step bounded text pretraining config on 256 GB200 GPUs with eight pipeline stages.
TP1/PP8/EP32 keeps each stage’s expert all-to-all within 32 GPUs and gives every GPU a unique slice of the experts. The data-parallel size is 32, as in the 192-GPU recipe, so both see the same batches. The data is text-only, so the perception towers stay frozen. Pass the imported checkpoint with
--pretrained_checkpoint.The peak learning rate is 3e-5, decaying to 3e-6, instead of the bounded pretraining contract’s from-scratch 3e-4: at 3e-4 the warm-started model’s loss rises until the learning rate decays, while at 3e-5 it falls.
- bridge.recipes.muse_spark_open.gb200.muse_spark_open.muse_spark_open_sft_192gpu_gb200_bf16_config() megatron.bridge.training.config.ConfigContainer#
Return a one-epoch full SFT config on 100K packed Tulu 3 conversations with 192 GB200 GPUs.
The first 100,000 Tulu 3 conversations pack into 10,951 4K sequences; 85 steps at GBS 128 make one epoch. TP4/PP6/EP32 leaves eight data-parallel replicas, each running 16 microbatches per step. Pipeline stages hold 9, 11, 11, 11, 11, and 9 decoder blocks; the first and last also hold the embedding with the perception towers and the output layer. Full activation recompute keeps every stage in memory.
- bridge.recipes.muse_spark_open.gb200.muse_spark_open.muse_spark_open_sft_256gpu_gb200_bf16_config() megatron.bridge.training.config.ConfigContainer#
Return a 100-step full SFT config on packed Tulu 3 data with 256 GB200 GPUs and eight pipeline stages.
Data is Tulu 3 train[:10000] packed to 8K at GBS 8, the cross-model bounded SFT contract; the 192-GPU recipe instead trains one epoch of 100K conversations at 4K and GBS 128. TP4/PP8/EP32 leaves eight data-parallel replicas, one packed sequence each. The first and last stages hold seven decoder blocks because they also own the embedding with the perception towers and the output layer; the middle stages hold eight. With one microbatch per replica the extra stages lower memory but do not shorten the step.
- bridge.recipes.muse_spark_open.gb200.muse_spark_open.muse_spark_open_sft_192gpu_gb200_bf16_long_context_config() megatron.bridge.training.config.ConfigContainer#
Return a 20-step packed 32K Tulu 3 full SFT config with context parallelism on 192 GB200 GPUs.
TP4/PP6/CP2/EP32 with sequence parallelism leaves four data-parallel replicas. Context parallelism uses A2A because Transformer Engine rejects P2P for the sliding-window layers. Packing aligns each sequence to lcm(2CP, TPCP) = 8.
- bridge.recipes.muse_spark_open.gb200.muse_spark_open.muse_spark_open_sft_192gpu_gb200_bf16_coderforge_64k_config() megatron.bridge.training.config.ConfigContainer#
Return a 20-step packed 64K CoderForge agent-trajectory full SFT config with CP4 on 192 GB200 GPUs.
SWE-Rebench trajectories are long single sequences (median about 46K tokens), so they exercise the 2,048-token sliding window and long-range full attention under context parallelism, unlike packed short chat rows. TP4/PP6/CP4/EP32 with sequence parallelism keeps 4K tokens per GPU and leaves two data-parallel replicas. Packing aligns each sequence to lcm(2CP, TPCP) = 16. Only the SWE-Rebench files are downloaded; the dataset metadata lists every split, so split completeness is not verified.
- bridge.recipes.muse_spark_open.gb200.muse_spark_open.muse_spark_open_peft_64gpu_gb200_bf16_config(
- peft_scheme: str | megatron.bridge.peft.base.PEFT = 'lora',
Return a one-epoch LoRA config on 100K packed Tulu 3 conversations with 64 GB200 GPUs.
LoRA rank 32 / alpha 32 adapts the attention projections, every routed expert (one adapter per expert), and the latent MoE projections. The frozen base runs at TP4/PP4/EP16 with stages of 15, 16, 16, and 15 decoder blocks, leaving four data-parallel replicas that each run 32 microbatches of GBS 128 packed 4K sequences; 85 steps make one epoch of the 10,951 packs from the first 100,000 Tulu 3 conversations. The expert adapters keep the routed expert inputs for their weight gradients, which needs the four pipeline stages. DoRA has no per-expert adapter controls and keeps attention-only adapters.
- bridge.recipes.muse_spark_open.gb200.muse_spark_open.muse_spark_open_peft_16gpu_gb200_bf16_multimodal_config(
- peft_scheme: str | megatron.bridge.peft.base.PEFT = 'lora',
Return a 100-step attention-projection LoRA config on image-text CORD-v2 data with 16 GB200 GPUs.
The frozen base runs at TP4/EP16 with sequence parallelism. Context parallelism uses A2A because Transformer Engine rejects P2P for sliding-window layers.
- bridge.recipes.muse_spark_open.gb200.muse_spark_open.muse_spark_open_sft_192gpu_gb200_bf16_multimodal_config() megatron.bridge.training.config.ConfigContainer#
Return a 100-step full SFT config on image-text CORD-v2 data with 192 GB200 GPUs.
TP1/PP6/EP32 gives every GPU a unique slice of the experts. The first and last stages hold nine decoder blocks each because they also own the embedding with the perception towers and the output layer; the middle stages hold eleven. Full activation recompute and the distributed optimizer hold the complete parameter, gradient, and optimizer state. The audio tower stays frozen because the image-text dataset never exercises it.
- bridge.recipes.muse_spark_open.gb200.muse_spark_open.__all__#
[‘muse_spark_open_peft_16gpu_gb200_bf16_multimodal_config’, ‘muse_spark_open_peft_64gpu_gb200_bf16_c…