bridge.models.stepfun.configuration_step35#

Module Contents#

Classes#

Step35Config

Configuration for the Step-3.5-Flash (Step35) Mixture-of-Experts model.

API#

class bridge.models.stepfun.configuration_step35.Step35Config(
hidden_size: int = 4096,
intermediate_size: int = 11264,
num_attention_heads: int = 64,
num_attention_groups: int = 8,
num_hidden_layers: int = 45,
max_seq_len: int = 128000,
vocab_size: int = 128815,
rms_norm_eps: float = 1e-05,
moe_intermediate_size: int = 1280,
moe_num_experts: int = 288,
moe_top_k: int = 8,
rope_theta: float = 10000,
rope_scaling: Optional[dict[str, Any]] = None,
max_position_embeddings: int = 128000,
share_expert_dims: int = 1280,
share_expert_dim: int | None = None,
head_dim: int = 128,
norm_expert_weight: bool = True,
layer_types: list[str] | None = None,
attention_other_setting: dict[str, Any] | None = None,
use_head_wise_attn_gate: bool = True,
sliding_window: Optional[int] = None,
num_nextn_predict_layers: int = 3,
moe_layers_enum: tuple[int] = (3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44),
**kwargs,
)#

Bases: transformers.configuration_utils.PretrainedConfig

Configuration for the Step-3.5-Flash (Step35) Mixture-of-Experts model.

Step35 is a decoder-only causal language model with grouped-query attention, rotary positional embeddings, and a configurable subset of transformer layers replaced by MoE FFN blocks (the remaining layers stay dense). It mirrors the architecture published under stepfun-ai/Step-3.5-Flash on the Hugging Face Hub.

Parameters:

  • hidden_size – Dimensionality of the hidden states.

  • intermediate_size – Dimensionality of the dense FFN intermediate states (used by non-MoE layers).

  • num_attention_heads – Number of query heads.

  • num_attention_groups – Number of key/value head groups for GQA.

  • num_hidden_layers – Total number of transformer layers.

  • max_seq_len – Maximum sequence length supported by the model.

  • vocab_size – Size of the tokenizer vocabulary.

  • rms_norm_eps – Epsilon used by RMSNorm.

  • moe_intermediate_size – Per-expert FFN intermediate size inside MoE layers.

  • moe_num_experts – Number of routed experts per MoE layer.

  • moe_top_k – Number of experts each token is routed to.

  • rope_theta – Base period of the rotary embeddings.

  • rope_scaling – Optional RoPE scaling configuration dict.

  • max_position_embeddings – Maximum positions supported by RoPE.

  • share_expert_dims – Hidden size of the shared-expert branch that runs alongside the routed experts.

  • share_expert_dim – Singular alias used by the published Step-3.5-Flash config.

  • head_dim – Per-head attention dimension.

  • norm_expert_weight – Whether to normalize the top-k expert routing weights so they sum to 1.

  • layer_types – Optional per-layer type override (e.g. attention variant); None uses the default for every layer.

  • attention_other_setting – Sliding-attention shape override from the HF config.

  • use_head_wise_attn_gate – Whether Step-3.5 uses per-head g_proj gates fused into QKV.

  • sliding_window – Sliding-window size for windowed attention; None disables windowing.

  • num_nextn_predict_layers – Number of MTP layers appended after the main decoder.

  • moe_layers_enum – Indices of layers that use the MoE FFN. Layers not listed here use the dense FFN of intermediate_size.

  • **kwargs – Forwarded to :class:~transformers.PretrainedConfig.

.. note::

model_type and architectures deliberately keep the step3p5 / Step3p5ForCausalLM spelling so this config stays compatible with the config.json shipped on stepfun-ai/Step-3.5-Flash. Only the Python class name uses the Step35 spelling internally.

Initialization

model_type#

‘step3p5’

architectures#

[‘Step3p5ForCausalLM’]