nemo_automodel.components.models.glm_moe_dsa.config

Automodel’s own GLM MoE DSA config.

Module Contents

Classes

Name	Description
`GlmMoeDsaConfig`	Configuration for GLM MoE DSA (DeepSeek-style sparse attention).

Data

__all__

API

class nemo_automodel.components.models.glm_moe_dsa.config.GlmMoeDsaConfig(
    vocab_size: int = 154880,
    hidden_size: int = 6144,
    intermediate_size: int = 12288,
    moe_intermediate_size: int = 2048,
    num_hidden_layers: int = 78,
    num_attention_heads: int = 64,
    num_key_value_heads: int = 64,
    n_shared_experts: int = 1,
    n_routed_experts: int = 256,
    routed_scaling_factor: float = 2.5,
    n_group: int = 1,
    topk_group: int = 1,
    num_experts_per_tok: int = 8,
    norm_topk_prob: bool = True,
    mlp_layer_types: list[str] | None = None,
    kv_lora_rank: int = 512,
    q_lora_rank: int = 2048,
    qk_rope_head_dim: int = 64,
    qk_nope_head_dim: int = 192,
    v_head_dim: int = 256,
    index_topk: int = 2048,
    index_head_dim: int = 128,
    index_n_heads: int = 32,
    indexer_types: list[str] | None = None,
    index_topk_pattern: str | list[str] | None = None,
    index_topk_freq: int = 1,
    hidden_act: str = 'silu',
    max_position_embeddings: int = 202752,
    initializer_range: float = 0.02,
    rms_norm_eps: float = 1e-05,
    use_cache: bool = True,
    attention_bias: bool = False,
    attention_dropout: float = 0.0,
    rope_parameters: dict | None = None,
    pad_token_id: int | None = None,
    bos_token_id: int | None = 0,
    eos_token_id: int | list[int] | None = 1,
    tie_word_embeddings: bool = False,
    kwargs = {}
)

Bases: PretrainedConfig

Configuration for GLM MoE DSA (DeepSeek-style sparse attention).

Standalone rather than a subclass of the transformers config, matching the other Automodel-owned configs: Automodel ships its own GLM DSA modeling code, so its field protocol must not move when transformers releases do. Subclassing also inherited attribute_map entries, one of which is actively harmful — transformers maps head_dim onto qk_rope_head_dim while the released GLM-5.2 config.json sets both (head_dim: 192, the full QK head dim, and qk_rope_head_dim: 64), so the alias overwrote the rope dim with 192 and the DSA indexer’s nope split went negative (index_head_dim - qk_rope_head_dim = 128 - 192). Only the expert-count alias is kept here.

attribute_map

= {'num_local_experts': 'n_routed_experts'}

keys_to_ignore_at_inference

= ['past_key_values']

model_type

= 'glm_moe_dsa'

qk_head_dim

= qk_nope_head_dim + qk_rope_head_dim

nemo_automodel.components.models.glm_moe_dsa.config.__all__ = ['GlmMoeDsaConfig']