bridge.models.exaone.exaone45.modelling_exaone45.text_model

EXAONE 4.5 GPT model implementation.

Module Contents

Classes

Exaone45GPTModel

EXAONE 4.5 GPT model with vision-language capabilities.

API

class bridge.models.exaone.exaone45.modelling_exaone45.text_model.Exaone45GPTModel(

config: megatron.bridge.models.transformer_config.TransformerConfig,

transformer_layer_spec: megatron.core.transformer.spec_utils.ModuleSpec,

vocab_size: int,

max_sequence_length: int,

pre_process: bool = True,

post_process: bool = True,

fp16_lm_cross_entropy: bool = False,

parallel_output: bool = True,

share_embeddings_and_output_weights: bool = False,

position_embedding_type: Literal[learned_absolute, rope, mrope, none] = 'learned_absolute',

rotary_percent: float = 1.0,

rotary_base: int = 10000,

rope_scaling: bool = False,

rope_scaling_factor: float = 8.0,

scatter_embedding_sequence_parallel: bool = True,

seq_len_interpolation_factor: Optional[float] = None,

mtp_block_spec: Optional[megatron.core.transformer.spec_utils.ModuleSpec] = None,

vp_stage: Optional[int] = None,

pg_collection: megatron.core.process_groups_config.ProcessGroupCollection = None,

)

Bases: megatron.core.models.gpt.gpt_model.GPTModel

EXAONE 4.5 GPT model with vision-language capabilities.

Initialization

forward(

input_ids: torch.Tensor,

position_ids: torch.Tensor,

attention_mask: torch.Tensor,

decoder_input: torch.Tensor = None,

labels: torch.Tensor = None,

inference_context: megatron.core.inference.contexts.BaseInferenceContext = None,

packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams = None,

extra_block_kwargs: dict = None,

runtime_gather_output: Optional[bool] = None,

*,

inference_params: Optional[megatron.core.inference.contexts.BaseInferenceContext] = None,

loss_mask: Optional[torch.Tensor] = None,

) → torch.Tensor

Forward function of the GPT Model This function passes the input tensors through the embedding layer, and then the decoeder and finally into the post processing layer (optional).

It either returns the Loss values if labels are given or the final hidden units

Parameters:

runtime_gather_output (bool) – Gather output at runtime. Default None means parallel_output arg in the constructor will be used.