bridge.models.glm5next.glm53flash_bridge#

Unified bridge for GLM-5.3-Flash (model_type glm5_next).

GLM-5.3-Flash is a VLM: a hybrid KDA/DSA + MoE + mHC language model paired with a replicated HF vision tower. This bridge handles both the text and vision paths in a single registered bridge and model — there is no separate “VL” split. AutoBridge.get_model_config() returns GLM53FlashModelConfig, which GLM53FlashModelBuilder builds; GLM53FlashModelProvider is the deprecated provider counterpart built from the same kwargs.

Each HF decoder layer maps to two physical layers: attention at 2N and FFN at 2N+1. Checkpoint keys use model.language_model; mixed FP8 weights are dequantized on import. Vision weights (model.visual.**) are loaded replicated and stay BF16 even under FP8. The VLM wrapper degrades to text-only when pixel_values is None.

Module Contents#

Classes#

_ColumnParallelConcatMapping

Map separate HF Q/K/V tensors to rank-local [q_shard | k_shard | v_shard].

_EhProjSplitMapping

Split HF eh_proj [hidden, 2*hidden] into column-sharded Megatron e_proj/h_proj.

GLM53FlashModelProvider

Provider for the unified GLM-5.3-Flash VLM: hybrid language stack + HF vision.

GLM53FlashModel

Inject HF image/video features into the hybrid language model’s embeddings.

GLM53FlashBridge

Unified bridge for GLM-5.3-Flash: hybrid language model + replicated vision tower.

Functions#

_glm5next_text_config

Return the nested text_config for the GLM-5.3-Flash VLM.

_glm5next_layer_types

Return the normalized per-layer attention types used by the bridge.

_glm5next_mlp_layer_types

Per-HF-layer MLP type: ‘dense’ or ‘sparse’.

_glm5next_hybrid_layout

Return the Hybrid layer pattern and MoE schedule of the main decoder.

_is_keep_mask

Whether attention_mask is a [batch, seq] keep mask for dense input_ids rows.

_packed_position_ids

Return per-sequence positions [1, total_tokens] for a packed THD stream.

_select_tokens

Select tokens along dim 1 of a [1, total_tokens, ...] stream; no-op without an index.

glm5next_hybrid_stack_spec

Return the Hybrid stack with MLA q/kv norms wired into the DSA spec.

API#

bridge.models.glm5next.glm53flash_bridge._glm5next_text_config(hf_config)#

Return the nested text_config for the GLM-5.3-Flash VLM.

bridge.models.glm5next.glm53flash_bridge._glm5next_layer_types(hf_config) → list[str]#

Return the normalized per-layer attention types used by the bridge.

Transformers 5.19 normalizes the DSA layer name from deepseek_sparse_attention to indexed_attention. Keep both spellings at the config boundary; the provider mapping below uses the canonical HF spelling.

bridge.models.glm5next.glm53flash_bridge._glm5next_mlp_layer_types(hf_config) → list[str]#

Per-HF-layer MLP type: ‘dense’ or ‘sparse’.

bridge.models.glm5next.glm53flash_bridge._glm5next_hybrid_layout(hf_config) → tuple[str, list[int]]#

Return the Hybrid layer pattern and MoE schedule of the main decoder.

Each HF block becomes an attention layer (KDA or DSA) followed by an FFN layer (dense MLP or MoE). moe_layer_freq indexes those physical layers, as router replay does.

bridge.models.glm5next.glm53flash_bridge._is_keep_mask(
attention_mask: Optional[torch.Tensor],
input_ids: torch.Tensor,
) → bool#

Whether attention_mask is a [batch, seq] keep mask for dense input_ids rows.

bridge.models.glm5next.glm53flash_bridge._packed_position_ids(
packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams,
total_tokens: int,
device: torch.device,
) → torch.Tensor#

Return per-sequence positions [1, total_tokens] for a packed THD stream.

Positions restart at each physical sequence start; tokens past the last boundary get 0.

bridge.models.glm5next.glm53flash_bridge._select_tokens(
tensor: Optional[torch.Tensor],
index: Optional[torch.Tensor],
) → Optional[torch.Tensor]#

Select tokens along dim 1 of a [1, total_tokens, ...] stream; no-op without an index.

class bridge.models.glm5next.glm53flash_bridge._ColumnParallelConcatMapping(
megatron_param: str,
hf_params: list[str],
)#

Bases: megatron.bridge.models.conversion.param_mapping.MegatronParamMapping[dict]

Map separate HF Q/K/V tensors to rank-local [q_shard | k_shard | v_shard].

Initialization

resolve(captures)#
_shard_per_rank(
merged: torch.Tensor,
section_sizes: list[int],
) → list[torch.Tensor]#
hf_to_megatron(
hf_weights: dict,
megatron_module: torch.nn.Module,
) → torch.Tensor#
megatron_to_hf(
megatron_weights: Optional[torch.Tensor],
megatron_module,
) → dict#
_equal_sections(rows: int) → list[int]#
_section_sizes(config)#
class bridge.models.glm5next.glm53flash_bridge._EhProjSplitMapping(megatron_param: str, hf_param: str, index: int)#

Bases: megatron.bridge.models.conversion.param_mapping.MegatronParamMapping[torch.Tensor]

Split HF eh_proj [hidden, 2*hidden] into column-sharded Megatron e_proj/h_proj.

Import: split along input dim (dim 1), shard column-parallel (dim 0). Export: gather both halves, concat along dim 1.

Initialization

_export_cache: dict#

None

resolve(captures)#
hf_to_megatron(hf_weights, megatron_module)#
_shard_column(weight, megatron_module)#
megatron_to_hf(megatron_weights, megatron_module)#
bridge.models.glm5next.glm53flash_bridge.glm5next_hybrid_stack_spec(
config: megatron.bridge.models.hybrid_mla_provider.HybridMLAModelProvider,
) → megatron.core.transformer.ModuleSpec#

Return the Hybrid stack with MLA q/kv norms wired into the DSA spec.

class bridge.models.glm5next.glm53flash_bridge.GLM53FlashModelProvider#

Bases: megatron.bridge.models.hybrid_mla_provider.HybridMLAModelProvider, megatron.core.ssm.kda_layer_config.KDALayerConfig

Provider for the unified GLM-5.3-Flash VLM: hybrid language stack + HF vision.

vision_config: Any#

None

spatial_merge_size: int#

2

scatter_embedding_sequence_parallel: bool#

False

return_dict: bool#

True

image_token_id: int#

154854

video_token_id: int#

154855

image_start_token_id: int#

154830

image_end_token_id: int#

154831

video_start_token_id: int#

154832

video_end_token_id: int#

154833

freeze_language_model: bool#

False

freeze_vision_model: bool#

False

freeze_vision_projection: bool#

False

provide(
pre_process: Optional[bool] = None,
post_process: Optional[bool] = None,
vp_stage: Optional[int] = None,
) → bridge.models.glm5next.glm53flash_bridge.GLM53FlashModel#

Build the VLM wrapper and apply explicitly requested freeze options.

Omitted pre_process/post_process flags resolve to this rank’s first/last pipeline stage, as in HybridModelProvider.provide, so the wrapper and its language model agree on which stage owns the vision tower and embeddings.

provide_language_model(
pre_process: Optional[bool] = None,
post_process: Optional[bool] = None,
vp_stage: Optional[int] = None,
)#

Build the Megatron HybridModel language model (delegates to the parent).

class bridge.models.glm5next.glm53flash_bridge.GLM53FlashModel(
config,
pre_process: bool = True,
post_process: bool = True,
vp_stage: Optional[int] = None,
*,
build_language_model: Optional[Callable[..., megatron.core.transformer.module.MegatronModule]] = None,
pg_collection: Optional[megatron.core.process_groups_config.ProcessGroupCollection] = None,
)#

Bases: megatron.core.transformer.module.MegatronModule

Inject HF image/video features into the hybrid language model’s embeddings.

Parameters:
  • config – The provider, or the transformer config of a builder-backed model config. Either one carries the Hugging Face vision config and the multimodal token IDs.

  • pre_process – Whether to build the vision tower and the embeddings.

  • post_process – Whether to build the language-model output layer.

  • vp_stage – Virtual pipeline stage.

  • build_language_model – Builds the language model from pre_process, post_process and vp_stage. Defaults to the provider’s provide_language_model().

  • pg_collection – Process groups a builder passes. Defaults to the provider’s.

Initialization

_process_groups() → Optional[megatron.core.process_groups_config.ProcessGroupCollection]#

Return the process groups a builder passed, or the ones the provider carries.

set_input_tensor(input_tensor) → None#

Set model chunk input tensor.

forward(
input_ids: torch.LongTensor = None,
attention_mask: Optional[torch.Tensor] = None,
position_ids: Optional[torch.LongTensor] = None,
inputs_embeds: Optional[torch.FloatTensor] = None,
pixel_values: Optional[torch.Tensor] = None,
pixel_values_videos: Optional[torch.FloatTensor] = None,
image_grid_thw: Optional[torch.LongTensor] = None,
video_grid_thw: Optional[torch.LongTensor] = None,
mm_token_type_ids: Optional[torch.Tensor] = None,
labels: Optional[torch.Tensor] = None,
runtime_gather_output: Optional[bool] = None,
packed_seq_params: Optional[megatron.core.packed_seq_params.PackedSeqParams] = None,
*,
loss_mask: Optional[torch.Tensor] = None,
) → torch.Tensor | tuple[torch.Tensor, torch.Tensor]#

Insert vision features, then run the language model on a THD token stream.

Two input layouts are accepted:

  • Dense rows without packed_seq_params (Bridge BSHD training, generation). attention_mask is an optional [batch, seq] keep mask; every token is kept when it is omitted. Token IDs, embeddings, labels and the loss mask are packed with the same layout, so padding has zero loss, and the packed loss mask is returned with the output.

  • A [1, total_tokens] stream that is already packed, with packed_seq_params and attention_mask=None (Bridge collate-time packing). Its boundaries are kept and only this context-parallel rank’s tokens are selected.

The language model applies no positional embedding, so position_ids is not read; per-sequence positions in the packed layout are derived for MTP instead. Placeholder masks need whole sequences, so vision features are inserted before packing or context-parallel selection; sequence-parallel scattering comes last. Only the pre-process stage embeds tokens and runs the vision tower. Later stages read activations from set_input_tensor and use token IDs only for layout.

_insert_vision_features(
input_ids: torch.Tensor,
inputs_embeds: torch.Tensor,
*,
pixel_values: Optional[torch.Tensor],
image_grid_thw: Optional[torch.Tensor],
pixel_values_videos: Optional[torch.Tensor],
video_grid_thw: Optional[torch.Tensor],
) → torch.Tensor#

Replace image and video placeholder embeddings with HF vision features.

_pack_rows(
tensor: torch.Tensor,
keep_mask: torch.Tensor,
) → tuple[torch.Tensor, megatron.core.packed_seq_params.PackedSeqParams]#

Pack the kept tokens of dense rows into this rank’s TP/CP/FP8-aligned THD stream.

_packed_cp_index(
packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams,
input_ids: torch.Tensor,
) → Optional[torch.Tensor]#

Return this CP rank’s token indices into a packed stream, or None without CP.

freeze(
freeze_language_model: bool,
freeze_vision_model: bool,
freeze_vision_projection: bool,
)#

Freeze model modules (set requires_grad=False).

class bridge.models.glm5next.glm53flash_bridge.GLM53FlashBridge#

Bases: megatron.bridge.models.conversion.model_bridge.MegatronModelBridge

Unified bridge for GLM-5.3-Flash: hybrid language model + replicated vision tower.

MODEL_CONFIG_CLASS#

None

USE_MODEL_CONFIG_FOR_CONVERSION#

True

DEQUANTIZED_SOURCE_KEY_SUFFIXES#

(‘_scale_inv’,)

hf_config_to_model_config_kwargs(hf_config) → dict[str, Any]#

Convert the Hugging Face GLM-5.3-Flash config to builder config kwargs.

The language-model fields come from the nested text_config; the vision config, the multimodal token IDs and tie_word_embeddings come from the top-level config.

hf_config_to_provider_kwargs(hf_config) → dict[str, Any]#

Adapt the builder kwargs to the deprecated provider path.

classmethod megatron_to_hf_config(provider) → dict#
maybe_modify_loaded_hf_weight(
hf_param,
hf_state_dict: collections.abc.Mapping[str, torch.Tensor],
)#
static _maybe_dequantize_fp8(
weight,
param_name: str,
hf_state_dict: collections.abc.Mapping[str, torch.Tensor],
)#
mapping_registry() → megatron.bridge.models.conversion.mapping_registry.MegatronMappingRegistry#
static _mhc_init_only_mappings(mg_layer: str) → list#

Init-only mappings for an mHC hyper_connection with no HF source.

Used for the MTP layer’s inner physical layers, whose hyper_connection params (mapping_proj, bias, alpha_pre/post/res) are created by MCore but have no counterpart in GLM-5.3-Flash’s HF checkpoint.

static _mhc_mappings(mg_layer: str, hf_layer: str, kind: str) → list#

mHC hyper-connection mappings for one physical layer.

kind is ‘attn’ or ‘ffn’. With enable_mhc_connections MCore wraps each physical layer in a HyperConnectionHybridLayer exposing a single hyper_connection submodule (NOT separate self_attention_hyper_connection / mlp_hyper_connection). The checkpoint stores hc_{kind}_{fn,base,scale} as bare parameters (no .weight suffix) on the HF layer.

static _kda_attn_mappings(mg_layer: str, hf_layer: str) → list#

KDA (linear attention, two-stage gates) weight mappings.

MCore KDA fuses q|k|v into in_proj (order q|k|v along dim 0) and q|k|v conv into a single conv1d; HF stores them separately. f_a/g_a are replicated; f_b/g_b, in_proj, beta_proj, conv1d, A_log, dt_bias are TP column-sharded.

static _dsa_attn_mappings(mg_layer: str, hf_layer: str) → list#

DSA (MLA + kpool indexer) weight mappings.

static _dense_ffn_mappings(mg_layer: str, hf_layer: str) → list#

Dense MLP FFN mappings.

static _moe_ffn_mappings(mg_layer: str, hf_layer: str) → list#

MoE FFN (router + shared expert + routed experts) mappings.