bridge.models.bailing.configuration_bailing_moe_v2#
Bailing MoE V2 model configuration.
Local copy of the configuration from inclusionAI/Ling-mini-2.0 on HuggingFace hub, bundled here so that tests and conversion scripts do not require network access.
Module Contents#
Classes#
Configuration for the Bailing MoE V2 model (Ling series). |
API#
- class bridge.models.bailing.configuration_bailing_moe_v2.BailingMoeV2Config(
- vocab_size=157184,
- hidden_size=2048,
- intermediate_size=5120,
- num_hidden_layers=20,
- num_attention_heads=16,
- num_key_value_heads=4,
- hidden_act='silu',
- use_qkv_bias=False,
- use_bias=False,
- rms_norm_eps=1e-06,
- tie_word_embeddings=False,
- embedding_dropout=0.0,
- attention_dropout=0.0,
- output_dropout=0.0,
- initializer_range=0.02,
- max_position_embeddings=32768,
- rope_theta=600000.0,
- use_cache=True,
- max_window_layers=20,
- rope_scaling=None,
- pad_token_id=156892,
- eos_token_id=156892,
- num_experts=256,
- num_shared_experts=1,
- num_experts_per_tok=8,
- n_group=8,
- topk_group=4,
- moe_intermediate_size=512,
- first_k_dense_replace=1,
- head_dim=128,
- output_router_logits=False,
- use_qk_norm=True,
- num_nextn_predict_layers=0,
- mtp_loss_scaling_factor=0,
- moe_router_enable_expert_bias=True,
- routed_scaling_factor=1.0,
- **kwargs,
Bases:
transformers.configuration_utils.PretrainedConfigConfiguration for the Bailing MoE V2 model (Ling series).
Supported models:
inclusionAI/Ling-mini-2.0
inclusionAI/Ling-flash-2.0
inclusionAI/Ling-1T
Initialization
- model_type#
‘bailing_moe_v2’