bridge.recipes.qwen.gb200.qwen35

GB200 text-only pretraining and SFT recipes for Qwen3.5 models.

Module Contents

Functions

_qwen35_text_35b_a3b_long_context_common	Build the shared 16-GB200 text-only long-context SFT configuration.
qwen35_text_35b_a3b_sft_long_context_16gpu_gb200_bf16_config	BF16 text SFT with packed chat data, TP1/CP8/EP16 and one MTP layer.
qwen35_text_35b_a3b_sft_long_context_16gpu_gb200_fp8mx_config	MXFP8 text SFT with packed chat data, TP1/CP8/EP16 and one MTP layer.
qwen35_text_9b_pretrain_8gpu_gb200_bf16_config	Return a text-only Qwen3.5-9B pretraining config for eight GB200 GPUs.
qwen35_text_9b_pretrain_8gpu_gb200_bf16_deterministic_config	Return a deterministic text-only Qwen3.5-9B pretraining config for eight GB200 GPUs.
qwen35_text_35b_a3b_pretrain_8gpu_gb200_bf16_config	Return a text-only Qwen3.5-35B-A3B pretraining config for eight GB200 GPUs.
qwen35_text_35b_a3b_sft_8gpu_gb200_bf16_dynamic_cp_config	Return a Qwen3.5-35B-A3B long-context SFT config with global-batch packing and dynamic CP.

Data

_QWEN35_9B_BASE
_QWEN35_35B_A3B_BASE
_QWEN35_35B_A3B_INSTRUCT_REVISION
_CODERFORGE_REVISION

API

bridge.recipes.qwen.gb200.qwen35._QWEN35_9B_BASE

‘Qwen/Qwen3.5-9B-Base’

bridge.recipes.qwen.gb200.qwen35._QWEN35_35B_A3B_BASE

‘Qwen/Qwen3.5-35B-A3B-Base’

bridge.recipes.qwen.gb200.qwen35._QWEN35_35B_A3B_INSTRUCT_REVISION

‘59d61f3ce65a6d9863b86d2e96597125219dc754’

bridge.recipes.qwen.gb200.qwen35._CODERFORGE_REVISION

‘060fca96cf723b2ebab3181e9e59fafd273df3cb’

bridge.recipes.qwen.gb200.qwen35._qwen35_text_35b_a3b_long_context_common(

seq_length: int,

*,

hf_path: str,

hf_revision: str | None,

) → megatron.bridge.training.config.ConfigContainer

Build the shared 16-GB200 text-only long-context SFT configuration.

bridge.recipes.qwen.gb200.qwen35.qwen35_text_35b_a3b_sft_long_context_16gpu_gb200_bf16_config(

seq_length: int = 131072,

*,

hf_path: str = 'Qwen/Qwen3.5-35B-A3B',

hf_revision: str | None = _QWEN35_35B_A3B_INSTRUCT_REVISION,

) → megatron.bridge.training.config.ConfigContainer

BF16 text SFT with packed chat data, TP1/CP8/EP16 and one MTP layer.

Set a pretrained text checkpoint before training. The default CoderForge source is normalized and packed by the dataset builder. Model config and tokenizer share a pinned revision; use hf_revision=None for an extracted local causal-LM checkpoint. No vision model is constructed.

Parameters:

• seq_length – Packed token budget, a positive multiple of 16 for CP8.

• hf_path – HF model ID or extracted local text checkpoint.

• hf_revision – Matching model/tokenizer revision, or None for local inputs.

Returns:

Training configuration with BF16 compute and parameter communication.

bridge.recipes.qwen.gb200.qwen35.qwen35_text_35b_a3b_sft_long_context_16gpu_gb200_fp8mx_config(

seq_length: int = 131072,

*,

hf_path: str = 'Qwen/Qwen3.5-35B-A3B',

hf_revision: str | None = _QWEN35_35B_A3B_INSTRUCT_REVISION,

) → megatron.bridge.training.config.ConfigContainer

MXFP8 text SFT with packed chat data, TP1/CP8/EP16 and one MTP layer.

Set a pretrained text checkpoint before training. The default CoderForge source is normalized and packed by the dataset builder. Model config and tokenizer share a pinned revision; use hf_revision=None for an extracted local causal-LM checkpoint. No vision model is constructed.

Parameters:

• seq_length – Packed token budget, a positive multiple of 16 for CP8.

• hf_path – HF model ID or extracted local text checkpoint.

• hf_revision – Matching model/tokenizer revision, or None for local inputs.

Returns:

Training configuration with MXFP8 parameter gather and CuTeDSL grouped MLP.

bridge.recipes.qwen.gb200.qwen35.qwen35_text_9b_pretrain_8gpu_gb200_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a text-only Qwen3.5-9B pretraining config for eight GB200 GPUs.

bridge.recipes.qwen.gb200.qwen35.qwen35_text_9b_pretrain_8gpu_gb200_bf16_deterministic_config() → megatron.bridge.training.config.ConfigContainer

Return a deterministic text-only Qwen3.5-9B pretraining config for eight GB200 GPUs.

Wraps :func:qwen35_text_9b_pretrain_8gpu_gb200_bf16_config and applies

Func:

~megatron.bridge.recipes.utils.determinism_utils.apply_determinism_overrides.

bridge.recipes.qwen.gb200.qwen35.qwen35_text_35b_a3b_pretrain_8gpu_gb200_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a text-only Qwen3.5-35B-A3B pretraining config for eight GB200 GPUs.

bridge.recipes.qwen.gb200.qwen35.qwen35_text_35b_a3b_sft_8gpu_gb200_bf16_dynamic_cp_config() → megatron.bridge.training.config.ConfigContainer

Return a Qwen3.5-35B-A3B long-context SFT config with global-batch packing and dynamic CP.

Fine-tunes on the CoderForge Preview SWE-Rebench trajectories (long, heavy-tailed chat samples) with Megatron-Core’s online sequence-packing scheduler: unpacked samples are packed into THD bins once per step across the DP x CP pool, and every bin runs on a context-parallel group sized for its longest sequence (default_dynamic_cp) instead of the static CP4 a 32k cap forces on every sample.

Topology: TP1 PP1 CP4 EP8 on eight GB200 GPUs. A sample of at most max_seqlen_per_dp_cp_rank tokens runs on one GPU, up to twice that on CP2, and so on up to the full pool. Set checkpoint.pretrained_checkpoint to a converted Megatron checkpoint before training.

Requires the Megatron-Core dev submodule pin (./scripts/switch_mcore.sh dev); ConfigContainer.validate reports what the main pin is missing. Qwen3.5 attention uses 256-wide heads, which Transformer Engine only trains with FlashAttention; until Megatron-Core derives pad_between_seqs from the actual cu_seqlens (https://github.com/NVIDIA/Megatron-LM/pull/7417), FlashAttention also needs bins without padding between sequences, which fold_alignment_padding provides. See docs/training/dynamic-context-parallel.md.