`bridge.models.mistral.mistral_provider`#

Module Contents#

Classes#

MistralModelProvider

Shared Mistral settings, also inherited by the Ministral 3 provider.

Data#

API#

bridge.models.mistral.mistral_provider.logger#: ‘getLogger(…)’

class bridge.models.mistral.mistral_provider.MistralModelProvider#

Bases: megatron.bridge.models.gpt_provider.GPTModelProvider

Shared Mistral settings, also inherited by the Ministral 3 provider.

Architecture dimensions must be supplied from the Hugging Face config.

normalization: str#: ‘RMSNorm’

activation_func: Callable#: None

position_embedding_type: str#: ‘rope’

add_bias_linear: bool#: False

gated_linear_unit: bool#: True

seq_length: int#: 32768

attention_dropout: float#: 0.0

hidden_dropout: float#: 0.0

share_embeddings_and_output_weights: bool#: False

init_method_std: float#: 0.02

layernorm_epsilon: float#: 1e-05

window_size: List[int]#: None

rotary_base: float#: 1000000.0

params_dtype: torch.dtype#: None

bf16: bool#: True