nemo_automodel.components.models.mistral_large4.config

Configuration for Mistral Large 4.

Module Contents

Classes

Name	Description
`MistralLarge4Config`	Native Large 4 contract read from the released Transformers config.

Functions

Name	Description
`_config_from_hf`	Map the released Transformers composite config onto the native fields.
`_section`	Resolve a nested JSON object at the checkpoint-format boundary.

Data

ConfigValue

_TEXT_FIELDS

_VISION_FIELDS

API

class nemo_automodel.components.models.mistral_large4.config.MistralLarge4Config(
    hidden_size: int = 6144,
    intermediate_size: int = 24576,
    num_hidden_layers: int = 72,
    num_attention_heads: int = 64,
    vocab_size: int = 204800,
    kv_lora_rank: int = 512,
    qk_nope_head_dim: int = 64,
    qk_rope_head_dim: int = 64,
    v_head_dim: int = 128,
    rms_norm_eps: float = 1e-05,
    rope_theta: float = 10000.0,
    max_position_embeddings: int = 1048576,
    yarn_factor: float = 2048.0,
    yarn_original_max_position_embeddings: int = 512,
    yarn_beta_fast: float = 32.0,
    yarn_beta_slow: float = 1.0,
    final_logit_softcapping: float = 30.0,
    nope_layer_flags: list[bool] | None = None,
    kv_source_layers: list[int] | None = None,
    index_sharing: list[str] | None = None,
    indexer_nope_layer_flags: list[bool] | None = None,
    index_topk: int = 2048,
    index_n_heads: int = 32,
    index_head_dim: int = 128,
    first_k_dense_replace: int = 1,
    n_routed_experts: int = 128,
    n_shared_experts: int = 1,
    num_experts_per_tok: int = 4,
    moe_intermediate_size: int = 6144,
    routed_scaling_factor: float = 1.0,
    norm_topk_prob: bool = True,
    tie_word_embeddings: bool = False,
    vision_config: dict[str, nemo_automodel.components.models.mistral_large4.config.ConfigValue] | None = None,
    kwargs: nemo_automodel.components.models.mistral_large4.config.ConfigValue = {}
)

Bases: PretrainedConfig

Native Large 4 contract read from the released Transformers config.

Vision settings are consumed by the conditional-generation class. The causal class remains text-only; draft execution is not enabled.

flops_formula

= staticmethod(model_flops)

head_dim

= qk_nope_head_dim + qk_rope_head_dim

hidden_act

= 'silu'

index_sharing

indexer_nope_layer_flags

kv_source_layers

model_type

= 'mistral_large4'

nope_layer_flags

scoring_func

= 'softmax'

vision_config

= copy.deepcopy(vision_config)

nemo_automodel.components.models.mistral_large4.config.MistralLarge4Config.from_dict(
    config_dict: dict[str, nemo_automodel.components.models.mistral_large4.config.ConfigValue],
    kwargs: nemo_automodel.components.models.mistral_large4.config.ConfigValue = {}
) -> nemo_automodel.components.models.mistral_large4.config.MistralLarge4Config | tuple[nemo_automodel.components.models.mistral_large4.config.MistralLarge4Config, dict[str, nemo_automodel.components.models.mistral_large4.config.ConfigValue]]

classmethod

Read either the released Transformers config or a saved native config.

Parameters:

config_dict

dict[str, ConfigValue]

config.json contents. The released checkpoint nests text and vision settings under text_config/vision_config; configs saved by this class use the flat native fields.

**kwargs

ConfigValueDefaults to {}

Existing PretrainedConfig.from_dict overrides.

Returns: MistralLarge4Config | tuple[MistralLarge4Config, dict[str, ConfigValue]]

The configuration, plus unused kwargs when requested by the caller.

nemo_automodel.components.models.mistral_large4.config.MistralLarge4Config.validate_architecture() -> None

Reject settings that would silently change the checkpoint’s semantics.

nemo_automodel.components.models.mistral_large4.config._config_from_hf(
    config: dict[str, nemo_automodel.components.models.mistral_large4.config.ConfigValue]
) -> dict[str, nemo_automodel.components.models.mistral_large4.config.ConfigValue]

Map the released Transformers composite config onto the native fields.

nemo_automodel.components.models.mistral_large4.config._section(
    config: dict[str, nemo_automodel.components.models.mistral_large4.config.ConfigValue],
    name: str
) -> dict[str, nemo_automodel.components.models.mistral_large4.config.ConfigValue]

Resolve a nested JSON object at the checkpoint-format boundary.

nemo_automodel.components.models.mistral_large4.config.ConfigValue: TypeAlias = bool | int | float | str | list['ConfigValue'] | dict[str, 'ConfigValue'] | None

nemo_automodel.components.models.mistral_large4.config._TEXT_FIELDS = ('hidden_size', 'intermediate_size', 'num_hidden_layers', 'num_attention_heads',...

nemo_automodel.components.models.mistral_large4.config._VISION_FIELDS = ('hidden_size', 'intermediate_size', 'num_hidden_layers', 'num_attention_heads',...