bridge.models.minimax_m3.minimax_m3_bridge

Module Contents

Classes

MiniMaxM3ModelProvider	GPT provider that preserves MiniMax-M3’s FP32 router parameters.
MiniMaxM3Bridge	Megatron Bridge for the MiniMax-M3 language model.

Functions

_promote_router_weights_to_float32

Keep MiniMax-M3 router parameters in FP32 for every load path.

API

bridge.models.minimax_m3.minimax_m3_bridge._promote_router_weights_to_float32(

model: list[torch.nn.Module],

) → list[torch.nn.Module]

Keep MiniMax-M3 router parameters in FP32 for every load path.

Megatron initializes router parameters in params_dtype even when moe_router_dtype="fp32". Promoting them immediately after construction prevents truncation when loading either HF weights or a native Megatron checkpoint.

class bridge.models.minimax_m3.minimax_m3_bridge.MiniMaxM3ModelProvider

Bases: megatron.bridge.models.gpt_provider.GPTModelProvider

GPT provider that preserves MiniMax-M3’s FP32 router parameters.

__post_init__() → None

Install the router hook on fresh and deserialized providers.

class bridge.models.minimax_m3.minimax_m3_bridge.MiniMaxM3Bridge

Bases: megatron.bridge.models.conversion.model_bridge.MegatronModelBridge

Megatron Bridge for the MiniMax-M3 language model.

MiniMax-M3 ships as a natively multimodal checkpoint (MiniMaxM3SparseForConditionalGeneration): a CLIP-style vision tower plus a sparse-MoE text backbone. This bridge converts the language model (language_model.* weights) to a Megatron-Core GPTModel; the vision tower and multimodal projector are not yet bridged.

Text backbone architecture: - Mixed dense/MoE decoder: the first layers are dense (dense_intermediate_size MLP), the rest use 128 routed experts (top-4) plus one shared expert. - Sigmoid router scoring with expert-bias correction and routed_scaling_factor applied to the normalized top-k weights (same routing math as DeepSeek-V3). - SwiGLU-OAI expert/MLP activation: clamped gate/up projections with a +1 linear offset (same as GPT-OSS), expressed via activation_func_clamp_value and glu_linear_offset. - Gemma-style RMSNorm (x * (1 + w)) on every norm, expressed via layernorm_zero_centered_gamma. - GQA attention with per-head QK RMSNorm and partial RoPE (rotary_dim of head_dim channels rotated).

Known limitations: - The lightning-indexer block-sparse attention branch (self_attn.index_{q,k}_{proj,norm}) is not mapped; the Megatron model runs full causal attention on every layer. Selection happens at index_block_size granularity with index_topk_blocks kept per query, so full attention is mathematically identical for sequences up to index_topk_blocks * index_block_size tokens (2048 for the released checkpoint) and an approximation beyond that. - The vision tower, multimodal projector, and patch-merge MLP are not mapped (language model only). - MTP (Multi-Token Prediction) modules are not mapped. The released checkpoint advertises num_nextn_predict_layers in its config but ships no mtp.* weights, so mtp_num_layers is forced to None. - Standalone Hugging Face checkpoint export is not supported. The HF checkpoint is multimodal, while this bridge maps only its language_model.* tensors. In-memory HF-to-Megatron-to-HF weight verification remains supported.

.. rubric:: Example

from megatron.bridge import AutoBridge bridge = AutoBridge.from_hf_pretrained(“MiniMaxAI/MiniMax-M3”, trust_remote_code=True) provider = bridge.to_megatron_provider()

SUPPORTS_HF_PRETRAINED_EXPORT

False

classmethod hf_to_megatron_activation(hidden_act: str)

Convert HF activation name to Megatron activation function.

The released MiniMax-M3 checkpoint declares hidden_act="swigluoai", which is not a standard ACT2FN key (transformers normalizes it to silu and computes the gate inline from swiglu_alpha / swiglu_limit). Map it to quick_gelu — the SwiGLU-OAI gate is gate * sigmoid(1.702 * gate), i.e. exactly quick-GELU; the clamp and +1 offset are carried by separate provider fields.

provider_bridge(

hf_pretrained: megatron.bridge.models.hf_pretrained.causal_lm.PreTrainedCausalLM,

) → bridge.models.minimax_m3.minimax_m3_bridge.MiniMaxM3ModelProvider

Convert the HuggingFace MiniMax-M3 config to a GPTModelProvider.

abstractmethod classmethod megatron_to_hf_config(

provider: megatron.bridge.models.gpt_provider.GPTModelProvider,

) → dict

Reject standalone HF export until the full multimodal contract is mapped.

mapping_registry() → megatron.bridge.models.conversion.mapping_registry.MegatronMappingRegistry

Return the parameter mappings for the MiniMax-M3 language model.

All HF weights live under the language_model. prefix of the multimodal checkpoint. MoE weights use the legacy block_sparse_moe layout with per-expert w1 (gate), w3 (up), and w2 (down) tensors, the same on-disk format as MiniMax-M2.