`bridge.models.exaone.exaone45.exaone45_provider`#

EXAONE 4.5 VL model provider configuration for Megatron-Core.

Module Contents#

Classes#

Exaone45ModelProvider

Model provider for EXAONE 4.5 VL models.

Functions#

`exaone_45_transformer_layer_spec`	Create an EXAONE 4.5 layer spec backed by the EXAONE Post-LN layer pattern.
`exaone_45_mtp_block_spec`	Create an MTP block spec that preserves the EXAONE transformer layer.

API#

bridge.models.exaone.exaone45.exaone45_provider.exaone_45_transformer_layer_spec( config: Exaone45ModelProvider, ) → megatron.core.transformer.spec_utils.ModuleSpec#: Create an EXAONE 4.5 layer spec backed by the EXAONE Post-LN layer pattern.

bridge.models.exaone.exaone45.exaone45_provider.exaone_45_mtp_block_spec( config: Exaone45ModelProvider, vp_stage: int | None = None, ) → megatron.core.transformer.multi_token_prediction.MultiTokenPredictionBlockSubmodules | None#: Create an MTP block spec that preserves the EXAONE transformer layer.

class bridge.models.exaone.exaone45.exaone45_provider.Exaone45ModelProvider#

Bases: megatron.bridge.models.gpt_provider.GPTModelProvider

Model provider for EXAONE 4.5 VL models.

Note: num_query_groups in parent class corresponds to num_key_value_heads in HF config.

transformer_layer_spec: megatron.core.transformer.spec_utils.ModuleSpec | collections.abc.Callable[[bridge.models.exaone.exaone45.exaone45_provider.Exaone45ModelProvider], megatron.core.transformer.spec_utils.ModuleSpec]#: None

vision_config: transformers.models.exaone4_5.configuration_exaone4_5.Exaone4_5_VisionConfig#: ‘field(…)’

hf_text_config: transformers.models.exaone4_5.configuration_exaone4_5.Exaone4_5_Config | None#: None

image_token_id: int#: 67

video_token_id: int#: 68

vision_token_id: int#: 67

vision_start_token_id: int#: 73

vision_end_token_id: int#: 74

bos_token_id: int#: 1

eos_token_id: int#: 53

spatial_merge_size: int#: 2

rotary_base: float#: 1000000.0

rope_scaling: bool#: False

rope_scaling_factor: int#: 1

position_embedding_type: str#: ‘rope’

scatter_embedding_sequence_parallel: bool#: False

freeze_language_model: bool#: False

freeze_vision_model: bool#: False

freeze_vision_projection: bool#: False

freeze_mtp_model: bool#: False

qk_layernorm: bool#: True

activation_func: collections.abc.Callable#: None

normalization: str#: ‘RMSNorm’

add_bias_linear: bool#: False

add_qkv_bias: bool#: False

attention_softmax_in_fp32: bool#: True

hidden_dropout: float#: 0.0

attention_dropout: float#: 0.0

apply_rope_fusion: bool#: False

bias_activation_fusion: bool#: False

bias_dropout_fusion: bool#: False

masked_softmax_fusion: bool#: True

gradient_accumulation_fusion: bool#: False

provide( pre_process: bool | None = None, post_process: bool | None = None, vp_stage: int | None = None, ) → megatron.bridge.models.exaone.exaone45.modelling_exaone45.model.Exaone45Model#: Provide an EXAONE 4.5 VL model instance with vision and language components.

provide_language_model( pre_process: bool | None = None, post_process: bool | None = None, vp_stage: int | None = None, ) → megatron.core.models.gpt.GPTModel#: Provide just the language model component without vision.