`bridge.recipes.qwen.qwen3_next`#

Module Contents#

Classes#

`Qwen3NextCommonKwargs`	Typed options accepted by Qwen3-Next recipe helpers.
`Qwen3NextFinetuneKwargs`	Typed options accepted by Qwen3-Next finetuning recipe helper functions.

Functions#

`qwen3_next_80b_a3b_pretrain_config`	Return a pre-training config for Qwen3-Next 80B-A3B.
`_qwen3_next_common`	Create a pre-training configuration for Qwen3-Next models using a given HuggingFace path.
`qwen3_next_80b_a3b_finetune_config`	Return a finetuning config for Qwen3-Next 80B-A3B.
`_qwen3_next_finetune_common`	Common finetuning configuration for Qwen3-Next model.

API#

class bridge.recipes.qwen.qwen3_next.Qwen3NextCommonKwargs#

Bases: typing_extensions.TypedDict

Typed options accepted by Qwen3-Next recipe helpers.

Initialization

Initialize self. See help(type(self)) for accurate signature.

hf_path: str#: None

dir: str | None#: None

name: str#: None

data_paths: list[str] | None#: None

data_args_path: str | None#: None

train_data_path: list[str] | None#: None

valid_data_path: list[str] | None#: None

test_data_path: list[str] | None#: None

per_split_data_args_path: str | None#: None

mock: bool#: None

tensor_model_parallel_size: int#: None

pipeline_model_parallel_size: int#: None

pipeline_dtype: torch.dtype | None#: None

virtual_pipeline_model_parallel_size: int | None#: None

context_parallel_size: int#: None

expert_model_parallel_size: int | None#: None

expert_tensor_parallel_size: int#: None

sequence_parallel: bool#: None

use_megatron_fsdp: bool#: None

enable_recompute: bool#: None

account_for_embedding_in_pipeline_split: bool#: None

account_for_loss_in_pipeline_split: bool#: None

mtp_num_layers: int | None#: None

mtp_loss_scaling_factor: float | None#: None

train_iters: int#: None

global_batch_size: int#: None

micro_batch_size: int#: None

seq_length: int#: None

lr: float#: None

min_lr: float#: None

lr_warmup_iters: int#: None

lr_decay_iters: int | None#: None

eval_interval: int#: None

save_interval: int#: None

use_null_tokenizer: bool#: None

precision_config: megatron.bridge.training.mixed_precision.MixedPrecisionConfig | str | None#: None

comm_overlap_config: megatron.bridge.training.comm_overlap.CommOverlapConfig | None#: None

moe_flex_dispatcher_backend: str | None#: None

disable_jit_fuser: bool#: None

class bridge.recipes.qwen.qwen3_next.Qwen3NextFinetuneKwargs#

Bases: bridge.recipes.qwen.qwen3_next.Qwen3NextCommonKwargs

Typed options accepted by Qwen3-Next finetuning recipe helper functions.

Initialization

Initialize self. See help(type(self)) for accurate signature.

pretrained_checkpoint: str | None#: None

peft: str | megatron.bridge.peft.base.PEFT | None#: None

packed_sequence: bool#: None

dataset_path: str | None#: None

finetune_lr: float#: None

wandb_project: str | None#: None

wandb_entity: str | None#: None

wandb_exp_name: str | None#: None

bridge.recipes.qwen.qwen3_next.qwen3_next_80b_a3b_pretrain_config(

**user_kwargs: typing_extensions.Unpack[bridge.recipes.qwen.qwen3_next.Qwen3NextCommonKwargs],

) → megatron.bridge.training.config.ConfigContainer#

Return a pre-training config for Qwen3-Next 80B-A3B.

See _qwen3_next_common for the full list of parameters.

bridge.recipes.qwen.qwen3_next._qwen3_next_common( hf_path: str, dir: str | None = None, name: str = 'default', data_paths: list[str] | None = None, data_args_path: str | None = None, train_data_path: list[str] | None = None, valid_data_path: list[str] | None = None, test_data_path: list[str] | None = None, per_split_data_args_path: str | None = None, mock: bool = False, path_to_cache: str | None = None, tensor_model_parallel_size: int = 4, pipeline_model_parallel_size: int = 2, pipeline_dtype: torch.dtype | None = torch.bfloat16, virtual_pipeline_model_parallel_size: int | None = None, context_parallel_size: int = 1, expert_model_parallel_size: int | None = 4, expert_tensor_parallel_size: int = 1, sequence_parallel: bool = True, use_megatron_fsdp: bool = False, enable_recompute: bool = False, account_for_embedding_in_pipeline_split: bool = False, account_for_loss_in_pipeline_split: bool = False, mtp_num_layers: int | None = 1, mtp_loss_scaling_factor: float | None = 0.1, train_iters: int = 300000, global_batch_size: int = 32, micro_batch_size: int = 2, seq_length: int = 4096, lr: float = 0.0003, min_lr: float = 3e-05, lr_warmup_iters: int = 500, lr_decay_iters: int | None = None, eval_interval: int = 500, save_interval: int = 500, use_null_tokenizer: bool = False, precision_config: megatron.bridge.training.mixed_precision.MixedPrecisionConfig | str | None = None, comm_overlap_config: megatron.bridge.training.comm_overlap.CommOverlapConfig | None = None, moe_flex_dispatcher_backend: str | None = None, disable_jit_fuser: bool | None = None, ) → megatron.bridge.training.config.ConfigContainer#

Create a pre-training configuration for Qwen3-Next models using a given HuggingFace path.

Parameters:

hf_path (str) – HuggingFace model path (e.g., “Qwen/Qwen3-Next-80B-A3B-Instruct”).
dir (str | None) – Base directory for saving logs and checkpoints.
name (str) – Name of the pre-training run.
data_paths (list[str] | None) – List of paths to dataset files. If None, mock data will be used.
data_args_path (str | None) – Path to file containing data arguments.
train_data_path (list[str] | None) – List of training data paths.
valid_data_path (list[str] | None) – List of validation data paths.
test_data_path (list[str] | None) – List of test data paths.
per_split_data_args_path (str | None) – Path to JSON file with per-split data configuration.
mock (bool) – Whether to use mock data. If True, ignores data_paths.
tensor_model_parallel_size (int) – Degree of tensor model parallelism.
pipeline_model_parallel_size (int) – Degree of pipeline model parallelism.
pipeline_dtype (torch.dtype | None) – Data type for pipeline parallelism.
virtual_pipeline_model_parallel_size (int | None) – Size of virtual pipeline parallelism.
context_parallel_size (int) – Degree of context parallelism to be passed to model_config.
expert_model_parallel_size (int | None) – Degree of expert parallelism for MoE.
expert_tensor_parallel_size (int) – Expert tensor parallelism for MoE.
sequence_parallel (bool) – Whether to use sequence parallelism.
use_megatron_fsdp (bool) – Whether to use Megatron FSDP.
enable_recompute (bool) – Whether to enable recompute for memory optimization.
account_for_embedding_in_pipeline_split (bool) – Whether to account for embedding in pipeline split.
account_for_loss_in_pipeline_split (bool) – Whether to account for loss in pipeline split.
mtp_num_layers (int | None) – Number of layers for MTP.
mtp_loss_scaling_factor (float | None) – Loss scaling factor for MTP.
train_iters (int) – Total number of training iterations.
global_batch_size (int) – Global batch size for training.
micro_batch_size (int) – Micro batch size for training.
seq_length (int) – Sequence length for training data.
lr (float) – Learning rate.
min_lr (float) – Minimum learning rate for cosine decay.
lr_warmup_iters (int) – Number of warmup iterations for the learning rate.
lr_decay_iters (int | None) – Number of iterations over which to decay the LR.
precision_config (MixedPrecisionConfig | str | None) – Precision configuration for the model.
comm_overlap_config (CommOverlapConfig | None) – Communication overlap configuration.
moe_flex_dispatcher_backend (str | None) – Token dispatcher type [deepep, hybridep].
disable_jit_fuser (bool) – Whether to disable the JIT fuser. Necessary for Qwen3-Next to work on Blackwell.

Returns:

Configuration for pre-training.

Return type:

ConfigContainer

bridge.recipes.qwen.qwen3_next.qwen3_next_80b_a3b_finetune_config(

**user_kwargs: typing_extensions.Unpack[bridge.recipes.qwen.qwen3_next.Qwen3NextFinetuneKwargs],

) → megatron.bridge.training.config.ConfigContainer#

Return a finetuning config for Qwen3-Next 80B-A3B.

Default configuration: 8 nodes, 64 GPUs total

Full SFT: TP=1, PP=1, EP=8, LR=5e-6 (with recompute)

bridge.recipes.qwen.qwen3_next._qwen3_next_finetune_common( hf_path: str, dir: str | None = None, name: str = 'default', dataset_path: str | None = None, tensor_model_parallel_size: int = 1, pipeline_model_parallel_size: int = 1, pipeline_dtype: torch.dtype | None = torch.bfloat16, virtual_pipeline_model_parallel_size: int | None = None, context_parallel_size: int = 1, expert_model_parallel_size: int | None = 8, expert_tensor_parallel_size: int = 1, sequence_parallel: bool = False, use_megatron_fsdp: bool = False, enable_recompute: bool = False, pretrained_checkpoint: str | None = None, peft: str | megatron.bridge.peft.base.PEFT | None = None, packed_sequence: bool = False, train_iters: int = 1000, global_batch_size: int | None = None, micro_batch_size: int = 1, seq_length: int = 2048, eval_interval: int = 30, save_interval: int = 50, finetune_lr: float = 5e-06, min_lr: float = 0.0, lr_warmup_iters: int = 50, lr_decay_iters: int | None = None, wandb_project: str | None = None, wandb_entity: str | None = None, wandb_exp_name: str | None = None, precision_config: megatron.bridge.training.mixed_precision.MixedPrecisionConfig | str | None = 'bf16_mixed', comm_overlap_config: megatron.bridge.training.comm_overlap.CommOverlapConfig | None = None, moe_flex_dispatcher_backend: str | None = None, disable_jit_fuser: bool | None = None, ) → megatron.bridge.training.config.ConfigContainer#: Common finetuning configuration for Qwen3-Next model.

bridge.recipes.qwen.qwen3_next#

Module Contents#

Classes#

Functions#

API#

`bridge.recipes.qwen.qwen3_next`#