bridge.models.glm5next.glm53flash_bridge#
Unified bridge for GLM-5.3-Flash (model_type glm5_next).
GLM-5.3-Flash is a VLM: a hybrid KDA/DSA + MoE + mHC language model paired with a
replicated HF vision tower. This bridge handles both the text and vision paths in a
single registered bridge and model — there is no separate “VL” split.
AutoBridge.get_model_config() returns GLM53FlashModelConfig, which
GLM53FlashModelBuilder builds; GLM53FlashModelProvider is the deprecated
provider counterpart built from the same kwargs.
Each HF decoder layer maps to two physical layers: attention at 2N and FFN at 2N+1.
Checkpoint keys use model.language_model; mixed FP8 weights are dequantized on
import. Vision weights (model.visual.**) are loaded replicated and stay BF16 even
under FP8. The VLM wrapper degrades to text-only when pixel_values is None.
Module Contents#
Classes#
Map separate HF Q/K/V tensors to rank-local [q_shard | k_shard | v_shard]. |
|
Split HF eh_proj [hidden, 2*hidden] into column-sharded Megatron e_proj/h_proj. |
|
Provider for the unified GLM-5.3-Flash VLM: hybrid language stack + HF vision. |
|
Inject HF image/video features into the hybrid language model’s embeddings. |
|
Unified bridge for GLM-5.3-Flash: hybrid language model + replicated vision tower. |
Functions#
Return the nested text_config for the GLM-5.3-Flash VLM. |
|
Return the normalized per-layer attention types used by the bridge. |
|
Per-HF-layer MLP type: ‘dense’ or ‘sparse’. |
|
Return the Hybrid layer pattern and MoE schedule of the main decoder. |
|
Whether |
|
Return per-sequence positions |
|
Select tokens along dim 1 of a |
|
Return the Hybrid stack with MLA q/kv norms wired into the DSA spec. |
API#
- bridge.models.glm5next.glm53flash_bridge._glm5next_text_config(hf_config)#
Return the nested text_config for the GLM-5.3-Flash VLM.
- bridge.models.glm5next.glm53flash_bridge._glm5next_layer_types(hf_config) list[str]#
Return the normalized per-layer attention types used by the bridge.
Transformers 5.19 normalizes the DSA layer name from
deepseek_sparse_attentiontoindexed_attention. Keep both spellings at the config boundary; the provider mapping below uses the canonical HF spelling.
- bridge.models.glm5next.glm53flash_bridge._glm5next_mlp_layer_types(hf_config) list[str]#
Per-HF-layer MLP type: ‘dense’ or ‘sparse’.
- bridge.models.glm5next.glm53flash_bridge._glm5next_hybrid_layout(hf_config) tuple[str, list[int]]#
Return the Hybrid layer pattern and MoE schedule of the main decoder.
Each HF block becomes an attention layer (KDA or DSA) followed by an FFN layer (dense MLP or MoE).
moe_layer_freqindexes those physical layers, as router replay does.
- bridge.models.glm5next.glm53flash_bridge._is_keep_mask(
- attention_mask: Optional[torch.Tensor],
- input_ids: torch.Tensor,
Whether
attention_maskis a[batch, seq]keep mask for denseinput_idsrows.
- bridge.models.glm5next.glm53flash_bridge._packed_position_ids(
- packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams,
- total_tokens: int,
- device: torch.device,
Return per-sequence positions
[1, total_tokens]for a packed THD stream.Positions restart at each physical sequence start; tokens past the last boundary get 0.
- bridge.models.glm5next.glm53flash_bridge._select_tokens(
- tensor: Optional[torch.Tensor],
- index: Optional[torch.Tensor],
Select tokens along dim 1 of a
[1, total_tokens, ...]stream; no-op without an index.
- class bridge.models.glm5next.glm53flash_bridge._ColumnParallelConcatMapping(
- megatron_param: str,
- hf_params: list[str],
Bases:
megatron.bridge.models.conversion.param_mapping.MegatronParamMapping[dict]Map separate HF Q/K/V tensors to rank-local [q_shard | k_shard | v_shard].
Initialization
- resolve(captures)#
- _shard_per_rank(
- merged: torch.Tensor,
- section_sizes: list[int],
- hf_to_megatron(
- hf_weights: dict,
- megatron_module: torch.nn.Module,
- megatron_to_hf(
- megatron_weights: Optional[torch.Tensor],
- megatron_module,
- _equal_sections(rows: int) list[int]#
- _section_sizes(config)#
- class bridge.models.glm5next.glm53flash_bridge._EhProjSplitMapping(megatron_param: str, hf_param: str, index: int)#
Bases:
megatron.bridge.models.conversion.param_mapping.MegatronParamMapping[torch.Tensor]Split HF eh_proj [hidden, 2*hidden] into column-sharded Megatron e_proj/h_proj.
Import: split along input dim (dim 1), shard column-parallel (dim 0). Export: gather both halves, concat along dim 1.
Initialization
- _export_cache: dict#
None
- resolve(captures)#
- hf_to_megatron(hf_weights, megatron_module)#
- _shard_column(weight, megatron_module)#
- megatron_to_hf(megatron_weights, megatron_module)#
- bridge.models.glm5next.glm53flash_bridge.glm5next_hybrid_stack_spec(
- config: megatron.bridge.models.hybrid_mla_provider.HybridMLAModelProvider,
Return the Hybrid stack with MLA q/kv norms wired into the DSA spec.
- class bridge.models.glm5next.glm53flash_bridge.GLM53FlashModelProvider#
Bases:
megatron.bridge.models.hybrid_mla_provider.HybridMLAModelProvider,megatron.core.ssm.kda_layer_config.KDALayerConfigProvider for the unified GLM-5.3-Flash VLM: hybrid language stack + HF vision.
- vision_config: Any#
None
- spatial_merge_size: int#
2
- scatter_embedding_sequence_parallel: bool#
False
- return_dict: bool#
True
- image_token_id: int#
154854
- video_token_id: int#
154855
- image_start_token_id: int#
154830
- image_end_token_id: int#
154831
- video_start_token_id: int#
154832
- video_end_token_id: int#
154833
- freeze_language_model: bool#
False
- freeze_vision_model: bool#
False
- freeze_vision_projection: bool#
False
- provide(
- pre_process: Optional[bool] = None,
- post_process: Optional[bool] = None,
- vp_stage: Optional[int] = None,
Build the VLM wrapper and apply explicitly requested freeze options.
Omitted
pre_process/post_processflags resolve to this rank’s first/last pipeline stage, as inHybridModelProvider.provide, so the wrapper and its language model agree on which stage owns the vision tower and embeddings.
- provide_language_model(
- pre_process: Optional[bool] = None,
- post_process: Optional[bool] = None,
- vp_stage: Optional[int] = None,
Build the Megatron HybridModel language model (delegates to the parent).
- class bridge.models.glm5next.glm53flash_bridge.GLM53FlashModel(
- config,
- pre_process: bool = True,
- post_process: bool = True,
- vp_stage: Optional[int] = None,
- *,
- build_language_model: Optional[Callable[..., megatron.core.transformer.module.MegatronModule]] = None,
- pg_collection: Optional[megatron.core.process_groups_config.ProcessGroupCollection] = None,
Bases:
megatron.core.transformer.module.MegatronModuleInject HF image/video features into the hybrid language model’s embeddings.
- Parameters:
config – The provider, or the transformer config of a builder-backed model config. Either one carries the Hugging Face vision config and the multimodal token IDs.
pre_process – Whether to build the vision tower and the embeddings.
post_process – Whether to build the language-model output layer.
vp_stage – Virtual pipeline stage.
build_language_model – Builds the language model from
pre_process,post_processandvp_stage. Defaults to the provider’sprovide_language_model().pg_collection – Process groups a builder passes. Defaults to the provider’s.
Initialization
- _process_groups() Optional[megatron.core.process_groups_config.ProcessGroupCollection]#
Return the process groups a builder passed, or the ones the provider carries.
- set_input_tensor(input_tensor) None#
Set model chunk input tensor.
- forward(
- input_ids: torch.LongTensor = None,
- attention_mask: Optional[torch.Tensor] = None,
- position_ids: Optional[torch.LongTensor] = None,
- inputs_embeds: Optional[torch.FloatTensor] = None,
- pixel_values: Optional[torch.Tensor] = None,
- pixel_values_videos: Optional[torch.FloatTensor] = None,
- image_grid_thw: Optional[torch.LongTensor] = None,
- video_grid_thw: Optional[torch.LongTensor] = None,
- mm_token_type_ids: Optional[torch.Tensor] = None,
- labels: Optional[torch.Tensor] = None,
- runtime_gather_output: Optional[bool] = None,
- packed_seq_params: Optional[megatron.core.packed_seq_params.PackedSeqParams] = None,
- *,
- loss_mask: Optional[torch.Tensor] = None,
Insert vision features, then run the language model on a THD token stream.
Two input layouts are accepted:
Dense rows without
packed_seq_params(Bridge BSHD training, generation).attention_maskis an optional[batch, seq]keep mask; every token is kept when it is omitted. Token IDs, embeddings, labels and the loss mask are packed with the same layout, so padding has zero loss, and the packed loss mask is returned with the output.A
[1, total_tokens]stream that is already packed, withpacked_seq_paramsandattention_mask=None(Bridge collate-time packing). Its boundaries are kept and only this context-parallel rank’s tokens are selected.
The language model applies no positional embedding, so
position_idsis not read; per-sequence positions in the packed layout are derived for MTP instead. Placeholder masks need whole sequences, so vision features are inserted before packing or context-parallel selection; sequence-parallel scattering comes last. Only the pre-process stage embeds tokens and runs the vision tower. Later stages read activations fromset_input_tensorand use token IDs only for layout.
- _insert_vision_features(
- input_ids: torch.Tensor,
- inputs_embeds: torch.Tensor,
- *,
- pixel_values: Optional[torch.Tensor],
- image_grid_thw: Optional[torch.Tensor],
- pixel_values_videos: Optional[torch.Tensor],
- video_grid_thw: Optional[torch.Tensor],
Replace image and video placeholder embeddings with HF vision features.
- _pack_rows(
- tensor: torch.Tensor,
- keep_mask: torch.Tensor,
Pack the kept tokens of dense rows into this rank’s TP/CP/FP8-aligned THD stream.
- _packed_cp_index(
- packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams,
- input_ids: torch.Tensor,
Return this CP rank’s token indices into a packed stream, or None without CP.
- freeze(
- freeze_language_model: bool,
- freeze_vision_model: bool,
- freeze_vision_projection: bool,
Freeze model modules (set requires_grad=False).
- class bridge.models.glm5next.glm53flash_bridge.GLM53FlashBridge#
Bases:
megatron.bridge.models.conversion.model_bridge.MegatronModelBridgeUnified bridge for GLM-5.3-Flash: hybrid language model + replicated vision tower.
- MODEL_CONFIG_CLASS#
None
- USE_MODEL_CONFIG_FOR_CONVERSION#
True
- DEQUANTIZED_SOURCE_KEY_SUFFIXES#
(‘_scale_inv’,)
- hf_config_to_model_config_kwargs(hf_config) dict[str, Any]#
Convert the Hugging Face GLM-5.3-Flash config to builder config kwargs.
The language-model fields come from the nested
text_config; the vision config, the multimodal token IDs andtie_word_embeddingscome from the top-level config.
- hf_config_to_provider_kwargs(hf_config) dict[str, Any]#
Adapt the builder kwargs to the deprecated provider path.
- classmethod megatron_to_hf_config(provider) dict#
- maybe_modify_loaded_hf_weight(
- hf_param,
- hf_state_dict: collections.abc.Mapping[str, torch.Tensor],
- static _maybe_dequantize_fp8(
- weight,
- param_name: str,
- hf_state_dict: collections.abc.Mapping[str, torch.Tensor],
- mapping_registry() megatron.bridge.models.conversion.mapping_registry.MegatronMappingRegistry#
- static _mhc_init_only_mappings(mg_layer: str) list#
Init-only mappings for an mHC hyper_connection with no HF source.
Used for the MTP layer’s inner physical layers, whose
hyper_connectionparams (mapping_proj, bias, alpha_pre/post/res) are created by MCore but have no counterpart in GLM-5.3-Flash’s HF checkpoint.
- static _mhc_mappings(mg_layer: str, hf_layer: str, kind: str) list#
mHC hyper-connection mappings for one physical layer.
kindis ‘attn’ or ‘ffn’. Withenable_mhc_connectionsMCore wraps each physical layer in aHyperConnectionHybridLayerexposing a singlehyper_connectionsubmodule (NOT separateself_attention_hyper_connection/mlp_hyper_connection). The checkpoint storeshc_{kind}_{fn,base,scale}as bare parameters (no.weightsuffix) on the HF layer.
- static _kda_attn_mappings(mg_layer: str, hf_layer: str) list#
KDA (linear attention, two-stage gates) weight mappings.
MCore KDA fuses q|k|v into
in_proj(order q|k|v along dim 0) and q|k|v conv into a singleconv1d; HF stores them separately. f_a/g_a are replicated; f_b/g_b, in_proj, beta_proj, conv1d, A_log, dt_bias are TP column-sharded.
- static _dsa_attn_mappings(mg_layer: str, hf_layer: str) list#
DSA (MLA + kpool indexer) weight mappings.
- static _dense_ffn_mappings(mg_layer: str, hf_layer: str) list#
Dense MLP FFN mappings.
- static _moe_ffn_mappings(mg_layer: str, hf_layer: str) list#
MoE FFN (router + shared expert + routed experts) mappings.