nemo_automodel.components.models.ernie5_5.config
nemo_automodel.components.models.ernie5_5.config
Configuration and layer schedule of ERNIE 5.5 causal language models.
Ernie5_5Config reads a released config.json without remote code. The functions of this module derive the
layer schedule from it and hold the checks that the model applies before it allocates parameters:
Ernie5_5ModelPlan.from_config: the attention and feed-forward kind of every decoder layer, and the attention schedule of the MTP block that the released checkpoint carries.decoder_only_config: a copy of a configuration without the MTP block. The model is built from this form only.validate_architecture: rejects the settings that select an architecture the model does not implement.main_rope_thetaandhca_rope_scaling: the two rotary settings in the form the attention layers take.
Module Contents
Classes
Functions
Data
API
Bases: PretrainedConfig
ERNIE 5.5 configuration with the values of the released ERNIE-5.5-Flash checkpoint as defaults.
Five fields do not default to the released value:
compress_ratioshas no default. It lists one entry per decoder layer, 128 for an HCA layer and -2 for a DSA layer; the released file adds one entry for its MTP layer.num_nextn_predict_layersdefaults to 0 (released: 1). The model does not build the MTP block, so a configuration written by hand needs no override. The releasedconfig.jsoncarries the explicit value 1 and is rejected until the recipe overrides it with 0: seedecoder_only_config.rope_interleavedefaults to None (released: true). A configuration with a DSA layer must declare the rotary pairing of the checkpoint itself, here or inmqa_latent_rope_adjacent_pairing.rotary_basedefaults to None (released: 8000000). The released file repeats the main rotary base in this key. When it is set it must equalrope_theta; with a default, overridingrope_thetaalone would be a conflict for every configuration.pad_token_iddefaults to None (released: 0). The model rejects input that contains the padding token, so a configuration must name it explicitly.
Two fields default to None, which selects the released value: rope_theta (8000000, unless
rope_parameters carries a base) and rope_scaling (the YaRN settings of the HCA layers). The attribute
rope_theta reads and writes rope_parameters["rope_theta"].
A configuration may carry the rotary settings twice, in rope_parameters and in the released keys
rope_theta and rope_scaling; the two forms must agree. Fields of the released config.json that are
not listed here are kept by PretrainedConfig, so loading and saving a checkpoint does not drop them.
Main rotary base, read from rope_parameters["rope_theta"].
Assigning the attribute writes the same entry, so a configuration override of rope_theta reaches the
value that main_rope_theta returns.
Serialize the configuration with the rotary and dtype keys of the released config.json.
Transformers keeps the rotary settings in rope_parameters and the dtype in dtype. Readers of the
released format look up rope_theta, rope_scaling and torch_dtype, so these keys are written
next to the standardized ones, with the same values.
Returns: dict[str, HFConfigValue]
The serialized configuration.
Attention and feed-forward kind of one decoder layer.
Decoder layer schedule, and the attention schedule of the MTP block.
Derive the layer schedule, keeping the MTP entries of compress_ratios apart from the decoder layers.
Parameters:
Model configuration. compress_ratios holds one entry per decoder layer followed by one
entry per MTP layer.
Returns: Ernie5_5ModelPlan
The attention and feed-forward kind of every decoder layer, the attention kind of every MTP layer and
Raises:
ValueError: If a layer count is not an integer in its valid range,moe_layer_freqis not 1,compress_ratiosdoes not have one entry per decoder and MTP layer, an entry is neither 128 (HCA) nor -2 (DSA), ormtp_shared_last_layeris not a boolean.
Return whether value is an integer and not a boolean.
Return a copy of config that describes the decoder without the MTP block.
The model does not build or train the MTP block of the released checkpoint, so it accepts a configuration only
when num_nextn_predict_layers is 0. The released compress_ratios then still ends with the entry of the
MTP layer, which shares the attention kind of the last decoder layer; the copy drops that entry and clears
mtp_shared_last_layer, so that a configuration saved from the model matches the weights saved with it.
Parameters:
Model configuration. It is not modified.
Returns: Ernie5_5Config
A deep copy with num_nextn_predict_layers == 0, one compress_ratios entry per decoder layer and
Raises:
ValueError: Ifnum_nextn_predict_layersis not 0,compress_ratioshas neither one entry per decoder layer nor one additional entry of a shared MTP layer, orErnie5_5ModelPlan.from_configrejects the result.
Return the YaRN settings of the HCA compress rotary embedding.
Parameters:
Model configuration.
Returns: dict[str, HFConfigValue]
Mapping with type set to "yarn" and the keys factor, original_max_position_embeddings,
Raises:
ValueError: Ifconfig.rope_parameterslacks one of the four keys.
Return the rotary base of the DSA layers.
Parameters:
Model configuration. rope_parameters["rope_theta"] holds the base; rotary_base, when set,
must repeat it.
Returns: float
The positive rotary base.
Raises:
ValueError: If the base is missing or not positive, orrotary_basediffers from it.
Reject the settings that select an architecture the ERNIE 5.5 model does not implement.
The model implements one variant of the attention, feed-forward and output options of the released checkpoint. The options it reads are checked individually; the released flags it hard-codes are compared with the implemented value.
Parameters:
Model configuration.
Raises:
ValueError: If the layer schedule is invalid, the activation is not SiTU, the MoE does not have exactly one shared expert and one router group with split-feature routing and a latent norm, the mHC and RMSNorm epsilons differ, or a rotary setting is missing or inconsistent.NotImplementedError: If a DSA layer uses the non-adjacent rotary layout, an HCA layer uses an option that is not implemented, or a released flag differs from the implemented variant.
Reject DSA checkpoints whose rotary channels are not stored as adjacent pairs.
mqa_latent_rope_adjacent_pairing takes precedence over rope_interleave when it is set.
Parameters:
Model configuration.
Raises:
ValueError: If neither field holds a boolean.NotImplementedError: If the checkpoint declares the non-adjacent layout, which would need a permutation of the rotary rows that is not implemented.
Reject the HCA configuration branches that are not implemented.
Parameters:
Model configuration.
Raises:
NotImplementedError: If Q/K normalization is disabled, the HCA rotary type is not YaRN, or the fused sigmoid gate is enabled.ValueError: If the YaRN settings are incomplete.