nemo_automodel.components.models.kimi_linear.config

Configuration for Moonshot Kimi Linear checkpoints.

Module Contents

Classes

Name	Description
`KimiLinear48BConfig`	HF-compatible configuration for Kimi Linear 48B A3B causal LM checkpoints.

API

class nemo_automodel.components.models.kimi_linear.config.KimiLinear48BConfig(
    vocab_size: int = 163840,
    hidden_size: int = 4096,
    head_dim: int | None = None,
    intermediate_size: int = 11008,
    num_hidden_layers: int = 32,
    num_attention_heads: int = 32,
    num_key_value_heads: int | None = None,
    hidden_act: str = 'silu',
    initializer_range: float = 0.02,
    rms_norm_eps: float = 1e-06,
    use_cache: bool = True,
    pad_token_id: int = 0,
    bos_token_id: int = 1,
    eos_token_id: int = 2,
    rope_theta: float = 10000.0,
    rope_scaling: dict[str, typing.Any] | None = None,
    tie_word_embeddings: bool = False,
    moe_intermediate_size: int | None = None,
    moe_renormalize: bool = True,
    moe_router_activation_func: str = 'sigmoid',
    num_experts: int | None = None,
    num_experts_per_token: int | None = None,
    num_shared_experts: int = 0,
    routed_scaling_factor: float = 1.0,
    first_k_dense_replace: int = 0,
    moe_layer_freq: int = 1,
    use_grouped_topk: bool = True,
    num_expert_group: int = 1,
    topk_group: int = 1,
    q_lora_rank: int | None = None,
    kv_lora_rank: int | None = None,
    qk_nope_head_dim: int | None = None,
    qk_rope_head_dim: int | None = None,
    v_head_dim: int | None = None,
    mla_use_nope: bool | None = False,
    num_nextn_predict_layers: int = 0,
    linear_attn_config: dict[str, typing.Any] | None = None,
    kda_mode: str = 'chunk',
    kda_unpad_inputs: bool = True,
    kda_use_fused_gate: bool = True,
    kda_use_qk_l2norm_in_kernel: bool = True,
    kwargs: typing.Any = {}
)

Bases: PretrainedConfig

HF-compatible configuration for Kimi Linear 48B A3B causal LM checkpoints.

Moonshot publishes both this model and the Kimi K3 text backbone under model_type: "kimi_linear" with architectures: ["KimiLinearForCausalLM"], so neither field tells the two families apart. Automodel gives this one a distinct identity (kimi_linear_48b_a3b / KimiLinear48BForCausalLM) and leaves kimi_linear to the K3 text config. A published Moonshot checkpoint therefore has to name this class explicitly, as the example recipes do; a checkpoint saved by Automodel already carries the distinct identity.

architectures

= list(type(self).architectures)

head_dim

is_linear_attn

bool

Return whether any decoder layer uses Kimi Delta Attention.

is_mla

bool

Return whether full-attention layers use Kimi MLA projection fields.

is_moe

bool

Return whether the checkpoint config declares routed experts.

keys_to_ignore_at_inference

= ['past_key_values']

model_type

= type(self).model_type

num_key_value_heads

nemo_automodel.components.models.kimi_linear.config.KimiLinear48BConfig.is_kda_layer(
    layer_idx: int
) -> bool

Return whether a zero-based layer index is configured as KDA.