Skip to navigation

nemo_automodel.components.models.ernie5_5.config

View as MarkdownOpen in Claude

Configuration and layer schedule of ERNIE 5.5 causal language models.

Ernie5_5Config reads a released config.json without remote code. The functions of this module derive the layer schedule from it and hold the checks that the model applies before it allocates parameters:

  • Ernie5_5ModelPlan.from_config: the attention and feed-forward kind of every decoder layer, and the attention schedule of the MTP block that the released checkpoint carries.
  • decoder_only_config: a copy of a configuration without the MTP block. The model is built from this form only.
  • validate_architecture: rejects the settings that select an architecture the model does not implement.
  • main_rope_theta and hca_rope_scaling: the two rotary settings in the form the attention layers take.

Module Contents

Classes

NameDescription
Ernie5_5ConfigERNIE 5.5 configuration with the values of the released ERNIE-5.5-Flash checkpoint as defaults.
Ernie5_5LayerPlanAttention and feed-forward kind of one decoder layer.
Ernie5_5ModelPlanDecoder layer schedule, and the attention schedule of the MTP block.

Functions

NameDescription
_is_integerReturn whether value is an integer and not a boolean.
decoder_only_configReturn a copy of config that describes the decoder without the MTP block.
hca_rope_scalingReturn the YaRN settings of the HCA compress rotary embedding.
main_rope_thetaReturn the rotary base of the DSA layers.
validate_architectureReject the settings that select an architecture the ERNIE 5.5 model does not implement.
validate_dsa_rope_layoutReject DSA checkpoints whose rotary channels are not stored as adjacent pairs.
validate_hca_optionsReject the HCA configuration branches that are not implemented.

Data

HFConfigValue

_DSA_COMPRESS_RATIO

_HCA_COMPRESS_RATIO

_HCA_YARN_KEYS

API

class nemo_automodel.components.models.ernie5_5.config.Ernie5_5Config(
vocab_size: int = 201216,
hidden_size: int = 4096,
intermediate_size: int = 14336,
num_hidden_layers: int = 43,
max_position_embeddings: int = 1048576,
compress_ratios: list[int] | None = None,
first_k_dense_replace: int = 1,
moe_layer_freq: int = 1,
num_nextn_predict_layers: int = 0,
mtp_shared_last_layer: bool = True,
num_attention_heads: int = 64,
num_key_value_heads: int = 1,
head_dim: int = 512,
qk_rope_head_dim: int = 64,
q_lora_rank: int = 1024,
o_lora_rank: int = 1024,
o_groups: int = 8,
sliding_window: int = 128,
compress_rope_theta: float = 160000.0,
hca_rope_type: str = 'yarn',
use_qk_norm: bool = True,
sigmoid_gate_fusion: bool = False,
qk_norm_type: str = 'per_head',
use_gated_attn: bool = True,
gated_attn_act: str = 'sigmoid',
gated_attn_use_q_lora: bool = True,
use_vha_attention: bool = True,
add_full_attention_sink_bias: bool = True,
attention_bias: bool = False,
hybrid_mla_attention: str = 'mqa_dsa',
mqa_split_kv_b_proj: bool = True,
use_fast_hadamard: bool = True,
hybrid_mla_num_attention_heads: int = 64,
hybrid_mla_q_lora_rank: int = 1024,
hybrid_mla_kv_lora_rank: int = 512,
hybrid_mla_qk_nope_head_dim: int = 192,
hybrid_mla_qk_rope_head_dim: int = 64,
hybrid_mla_v_head_dim: int = 256,
index_n_heads: int = 64,
index_head_dim: int = 128,
index_topk: int = 2048,
rope_interleave: bool | None = None,
mqa_latent_rope_adjacent_pairing: bool | None = None,
rope_theta: float | None = None,
rope_parameters: dict[str, nemo_automodel.components.models.ernie5_5.config.HFConfigValue] | None = None,
rotary_base: float | None = None,
enable_hyper_connections: bool = True,
hc_mult: int = 4,
hc_sinkhorn_iters: int = 20,
hc_eps: float = 1e-06,
rms_norm_eps: float = 1e-06,
hidden_act: str = 'situ',
activation_situ_beta: float = 4.0,
activation_situ_linear_beta: float = 25.0,
n_routed_experts: int = 512,
n_shared_experts: int = 1,
num_experts_per_tok: int = 10,
moe_intermediate_size: int = 2048,
moe_latent_size: int = 2048,
routed_scaling_factor: float = 3.0,
routed_scaling_factor_learnable: bool = True,
scoring_func: str = 'sigmoid',
norm_topk_prob: bool = True,
n_group: int = 1,
topk_group: int = 1,
moe_split_feature_routing: bool = True,
latent_moe_use_norm: bool = True,
multimax: str = 'lm_head',
initializer_range: float = 0.02,
pad_token_id: int | None = None,
tie_word_embeddings: bool = False,
use_cache: bool = True,
)

Bases: PretrainedConfig

ERNIE 5.5 configuration with the values of the released ERNIE-5.5-Flash checkpoint as defaults.

Five fields do not default to the released value:

  • compress_ratios has no default. It lists one entry per decoder layer, 128 for an HCA layer and -2 for a DSA layer; the released file adds one entry for its MTP layer.
  • num_nextn_predict_layers defaults to 0 (released: 1). The model does not build the MTP block, so a configuration written by hand needs no override. The released config.json carries the explicit value 1 and is rejected until the recipe overrides it with 0: see decoder_only_config.
  • rope_interleave defaults to None (released: true). A configuration with a DSA layer must declare the rotary pairing of the checkpoint itself, here or in mqa_latent_rope_adjacent_pairing.
  • rotary_base defaults to None (released: 8000000). The released file repeats the main rotary base in this key. When it is set it must equal rope_theta; with a default, overriding rope_theta alone would be a conflict for every configuration.
  • pad_token_id defaults to None (released: 0). The model rejects input that contains the padding token, so a configuration must name it explicitly.

Two fields default to None, which selects the released value: rope_theta (8000000, unless rope_parameters carries a base) and rope_scaling (the YaRN settings of the HCA layers). The attribute rope_theta reads and writes rope_parameters["rope_theta"].

A configuration may carry the rotary settings twice, in rope_parameters and in the released keys rope_theta and rope_scaling; the two forms must agree. Fields of the released config.json that are not listed here are kept by PretrainedConfig, so loading and saving a checkpoint does not drop them.

model_type
= 'ernie5_5_moe'
rope_theta
float | None

Main rotary base, read from rope_parameters["rope_theta"].

Assigning the attribute writes the same entry, so a configuration override of rope_theta reaches the value that main_rope_theta returns.

nemo_automodel.components.models.ernie5_5.config.Ernie5_5Config.to_dict() -> dict[str, nemo_automodel.components.models.ernie5_5.config.HFConfigValue]

Serialize the configuration with the rotary and dtype keys of the released config.json.

Transformers keeps the rotary settings in rope_parameters and the dtype in dtype. Readers of the released format look up rope_theta, rope_scaling and torch_dtype, so these keys are written next to the standardized ones, with the same values.

Returns: dict[str, HFConfigValue]

The serialized configuration.

class nemo_automodel.components.models.ernie5_5.config.Ernie5_5LayerPlan(
layer_index: int,
attention: typing.Literal['hca', 'dsa'],
feed_forward: typing.Literal['dense', 'moe']
)
Dataclass

Attention and feed-forward kind of one decoder layer.

attention
Literal['hca', 'dsa']
feed_forward
Literal['dense', 'moe']
layer_index
int
class nemo_automodel.components.models.ernie5_5.config.Ernie5_5ModelPlan(
mtp_attention: tuple[typing.Literal['hca', 'dsa'], ...],
mtp_shared_last_layer: bool
)
Dataclass

Decoder layer schedule, and the attention schedule of the MTP block.

decoder_layers
tuple[Ernie5_5LayerPlan, ...]
mtp_attention
tuple[Literal['hca', 'dsa'], ...]
mtp_shared_last_layer
bool
classmethod

Derive the layer schedule, keeping the MTP entries of compress_ratios apart from the decoder layers.

Parameters:

config
Ernie5_5Config

Model configuration. compress_ratios holds one entry per decoder layer followed by one entry per MTP layer.

Returns: Ernie5_5ModelPlan

The attention and feed-forward kind of every decoder layer, the attention kind of every MTP layer and

Raises:

  • ValueError: If a layer count is not an integer in its valid range, moe_layer_freq is not 1, compress_ratios does not have one entry per decoder and MTP layer, an entry is neither 128 (HCA) nor -2 (DSA), or mtp_shared_last_layer is not a boolean.
nemo_automodel.components.models.ernie5_5.config._is_integer(
) -> bool

Return whether value is an integer and not a boolean.

Return a copy of config that describes the decoder without the MTP block.

The model does not build or train the MTP block of the released checkpoint, so it accepts a configuration only when num_nextn_predict_layers is 0. The released compress_ratios then still ends with the entry of the MTP layer, which shares the attention kind of the last decoder layer; the copy drops that entry and clears mtp_shared_last_layer, so that a configuration saved from the model matches the weights saved with it.

Parameters:

config
Ernie5_5Config

Model configuration. It is not modified.

Returns: Ernie5_5Config

A deep copy with num_nextn_predict_layers == 0, one compress_ratios entry per decoder layer and

Raises:

  • ValueError: If num_nextn_predict_layers is not 0, compress_ratios has neither one entry per decoder layer nor one additional entry of a shared MTP layer, or Ernie5_5ModelPlan.from_config rejects the result.
nemo_automodel.components.models.ernie5_5.config.hca_rope_scaling(

Return the YaRN settings of the HCA compress rotary embedding.

Parameters:

config
Ernie5_5Config

Model configuration.

Returns: dict[str, HFConfigValue]

Mapping with type set to "yarn" and the keys factor, original_max_position_embeddings,

Raises:

  • ValueError: If config.rope_parameters lacks one of the four keys.
nemo_automodel.components.models.ernie5_5.config.main_rope_theta(
) -> float

Return the rotary base of the DSA layers.

Parameters:

config
Ernie5_5Config

Model configuration. rope_parameters["rope_theta"] holds the base; rotary_base, when set, must repeat it.

Returns: float

The positive rotary base.

Raises:

  • ValueError: If the base is missing or not positive, or rotary_base differs from it.
nemo_automodel.components.models.ernie5_5.config.validate_architecture(
) -> None

Reject the settings that select an architecture the ERNIE 5.5 model does not implement.

The model implements one variant of the attention, feed-forward and output options of the released checkpoint. The options it reads are checked individually; the released flags it hard-codes are compared with the implemented value.

Parameters:

config
Ernie5_5Config

Model configuration.

Raises:

  • ValueError: If the layer schedule is invalid, the activation is not SiTU, the MoE does not have exactly one shared expert and one router group with split-feature routing and a latent norm, the mHC and RMSNorm epsilons differ, or a rotary setting is missing or inconsistent.
  • NotImplementedError: If a DSA layer uses the non-adjacent rotary layout, an HCA layer uses an option that is not implemented, or a released flag differs from the implemented variant.
nemo_automodel.components.models.ernie5_5.config.validate_dsa_rope_layout(
) -> None

Reject DSA checkpoints whose rotary channels are not stored as adjacent pairs.

mqa_latent_rope_adjacent_pairing takes precedence over rope_interleave when it is set.

Parameters:

config
Ernie5_5Config

Model configuration.

Raises:

  • ValueError: If neither field holds a boolean.
  • NotImplementedError: If the checkpoint declares the non-adjacent layout, which would need a permutation of the rotary rows that is not implemented.
nemo_automodel.components.models.ernie5_5.config.validate_hca_options(
) -> None

Reject the HCA configuration branches that are not implemented.

Parameters:

config
Ernie5_5Config

Model configuration.

Raises:

  • NotImplementedError: If Q/K normalization is disabled, the HCA rotary type is not YaRN, or the fused sigmoid gate is enabled.
  • ValueError: If the YaRN settings are incomplete.
nemo_automodel.components.models.ernie5_5.config.HFConfigValue: TypeAlias = str | int | float | bool | torch.dtype | list['HFConfigValue'] | dict[str, 'HFCo...
nemo_automodel.components.models.ernie5_5.config._DSA_COMPRESS_RATIO = -2
nemo_automodel.components.models.ernie5_5.config._HCA_COMPRESS_RATIO = 128
nemo_automodel.components.models.ernie5_5.config._HCA_YARN_KEYS = ('factor', 'original_max_position_embeddings', 'beta_fast', 'beta_slow')