nemo_automodel.components.models.ernie5_5.state_dict_adapter
nemo_automodel.components.models.ernie5_5.state_dict_adapter
State-dict adapter between the native ERNIE 5.5 model and the released Hugging Face checkpoint layout.
Ernie5_5StateDictAdapter is the object the checkpointer talks to. It serves an unsharded model with the
conversions of checkpoint_layout, whose released tensors are views of the native ones, and a sharded model
with the destinations and export views of checkpoint_io.
A checkpoint load is a session of two calls: to_hf(..., for_checkpoint_load=True) returns the destinations
that the reader fills, and from_hf receives the same mapping back, installs what could not be read in place
and returns the native state. The adapter keeps one session at a time and fails closed: after a session that was
abandoned, replaced or failed, every later load or save raises, because the model may be partially overwritten.
The tensors of a PEFT export keep their names: the LoRA adapter tensors, and the modules that the export stores
whole under the base_model.model. prefix (modules_to_save, such as an unfrozen output head). to_hf
and from_hf return a state of such tensors alone as it is, without a session, and to_hf appends them to
the converted base tensors of a state that also holds base tensors. In a load, from_hf raises ValueError
unless the tensors it receives have the keys, shapes and dtypes of the PEFT tensors that
to_hf(..., for_checkpoint_load=True) was given.
Limitations:
- Decoder-only: the configuration must have
num_nextn_predict_layers == 0and untied embedding and output-head weights. MTP weights are neither loaded nor exported. - No quantized checkpoints, no
v4_compatiblemetadata export and no streaming export of single tensors. - A sharded load or save needs the layouts listed in
checkpoint_io. All ranks must make the same calls in the same order. - A load is not transactional: after a failure the model and the adapter must be discarded. Concurrent sessions and use from several threads are not supported.
- An unsharded load through the checkpointer is supported only without a device mesh. Like a sharded load it reads into model storage; the fused K/V projection of each DSA layer is its only temporary.
- A conversion from the released layout without a load session (
from_hfon plain tensors) packs every routed expert into new tensors and so holds a second copy of the experts; it suits small checkpoints. - Supported stored dtypes of a checkpoint are BF16, FP16 and FP32. The adapter never sees the stored dtype, so it cannot reject another one: the reader would convert it silently (FP64 would be rounded).
Module Contents
Classes
Functions
Data
API
Bases: StateDictAdapter
Convert between the native state of Ernie5_5_MoeForCausalLM and the released checkpoint layout.
Native names carry the model. prefix of the registered class. For a sharded and for an unsharded state,
to_hf returns destinations or export views that alias model storage, so that a checkpoint read writes
the model in place and a distributed read or save moves each rank’s shard only.
Key planning (get_hf_state_dict_keys) is supported for the unsharded state only, which is how the
framework uses it: once, before the model is sharded. It converts a shape-only copy of the state with
to_hf. A sharded state is rejected, and like a save it raises while a load session is open and after a
session was abandoned, replaced or failed.
Loading into an unsharded model is supported without a device mesh, which is what a run in one process
passes. An unsharded state together with a device mesh is not supported: from_hf raises, and inside a
load session the adapter then refuses all further work.
_supports_low_memory_dcp_load is set because a load allocates only small temporaries, also in one
process: the fused K/V projection of each DSA layer and, for a sharded model, the three alphas of each mHC
unit. Every other destination, the routed experts included, is model storage.
Return an adapter for a deep copy of the configuration that shares no load session with this one.
A model is copied together with its adapter. A copy made while a load session is open, or after one was abandoned, replaced or failed, refuses all further work: the copied model may be partially overwritten, and the open session belongs to the original.
Parameters:
The memo dictionary of copy.deepcopy.
Returns: Self
A new adapter, usable when this adapter is idle. It refuses the destinations of this adapter’s
Return the state under the names of the registered class.
Parameters:
Names relative to the inner model mapped to tensors of any shape.
Returns: dict[str, torch.Tensor]
The same tensor objects, in the same order, with the model. prefix.
Return whether a mapping holds a tensor that was created as a destination of an ended session.
Such a mapping is a copy of finished destinations. Its tensors hold what an earlier load left in them, and the entries that alias model storage hold the current model values.
Parameters:
Hugging Face names mapped to plain tensors of any shape.
Returns: bool
True when at least one tensor object of the mapping is a view or a temporary of an ended session.
Make destinations the one active load session.
The adapter keeps only a weak link to the mapping. If the mapping is dropped before from_hf
receives it, the session counts as abandoned and the adapter refuses all further work.
Parameters:
Hugging Face names mapped to the destination tensors of the load, of any shape.
Rename the feed-forward tensors of every decoder layer between the two native spellings.
The eager conversions name the feed-forward feed_forward; the model registers it as mlp (dense)
or moe (MoE).
Parameters:
Names relative to the inner model mapped to tensors of any shape.
Rename layers.{index}.feed_forward.* to the module names when true, the
module names to feed_forward when false.
Returns: dict[str, torch.Tensor]
The same tensor objects. The renamed tensors of each layer follow all other entries, layer by
Raises:
ValueError: If a layer has no feed-forward tensor under the source name or already has one under the target name.
Return the state named relative to the inner model.
Parameters:
Native names with the model. prefix mapped to tensors of any shape.
Returns: dict[str, torch.Tensor]
The same tensor objects, in the same order, without the prefix.
Raises:
ValueError: If a name does not start withmodel..
Return the callbacks that complete a checkpoint read into an unsharded model.
A read fills the destinations in place, and all of them alias model storage except the fused K/V projection of each DSA layer. Each callback splits one filled fused projection into the native key and value projections.
Parameters:
Native tensors named relative to the inner model, in the shapes of the model’s parameters
and buffers. The callbacks overwrite layers.{index}.attention.k_b_proj
[heads * latent, key_channels] and layers.{index}.attention.v_b_proj
[heads * value_channels, latent] of every DSA layer in place.
Released Hugging Face names mapped to the destinations of the load. The callbacks read
model.layers.{index}.self_attn.kv_b_proj.weight of shape
[heads * (key_channels + value_channels), latent] of every DSA layer.
Returns: list[Callable[[], None]]
One callback per DSA layer, in layer order.
Convert a complete unsharded state in the released layout to the native state.
Parameters:
Exactly the released tensors of the decoder-only model, in the released layouts; plain tensors.
Mesh of a distributed load; must be None. Loading into an unsharded model with a
device mesh is not supported, because this conversion does not shard.
Returns: dict[str, torch.Tensor]
Native model. names mapped to tensors in the shapes of the model’s parameters and buffers.
Raises:
ValueError: If a device mesh is given, a tensor is missing or unexpected, or a component conversion rejects its tensors.
Convert a complete unsharded native state to the released layout.
Parameters:
Exactly the native tensors of the model under their model. names, in the shapes of
the model’s parameters and buffers; plain tensors on any device, the meta device included.
Returns: dict[str, torch.Tensor]
Released Hugging Face names mapped to tensors in the released layouts, in the order of
Raises:
ValueError: If a name lacks themodel.prefix, a tensor is missing or unexpected, or a component conversion rejects its tensors.
Reject the conversion of a single tensor.
The released layout fuses the key and value projections of a DSA layer and splits the grouped expert tensors, so one native tensor cannot be converted without the others.
Parameters:
Native name of the tensor.
Tensor of any shape.
Options of the caller; not read.
Raises:
NotImplementedError: Always.
Return the native state for tensors in the released layout, closing a load session if one is open.
Without a session, a complete mapping of plain tensors is converted on its own. Otherwise the mapping
must be the object that to_hf(..., for_checkpoint_load=True) returned for the open session, with the
same keys and the same tensor objects. Anything else makes the adapter refuse all further work: a copy
of the destinations, made during the session or after it, the destinations of another adapter or of a
session that has already ended, a mapping of DTensors or an empty mapping without a session.
Parameters:
Released Hugging Face names mapped to tensors in the released layouts: either the destinations of the active load session after the read, or, when no session is open, exactly the released tensors of the decoder-only model as plain tensors, none of which was a destination of an earlier session.
Mesh of a distributed load. A sharded session does not need it. An unsharded session
and a conversion without a session require None: loading into an unsharded model with a
device mesh is not supported.
Other options of the caller. None applies to this model; they are not read.
Returns: dict[str, torch.Tensor]
Native model. names mapped to tensors in the shapes of the model’s parameters and buffers. For
Raises:
RuntimeError: If the mapping is not the destinations of the active session, or an earlier session was abandoned, replaced or failed.ValueError: If the keys or tensors of the destinations were exchanged, a device mesh is given for an unsharded conversion, or a conversion rejects its tensors.
Return the Hugging Face keys of the unsharded model, in the order of an unsharded save.
The framework plans the keys once, before the model is sharded. A sharded state is rejected instead of planned: its export names only the experts that one rank holds and orders the keys differently, so it cannot stand in for the plan of the whole model.
Parameters:
Exactly the native tensors of the unsharded model under their model. names, in the
shapes of the model’s parameters and buffers; plain tensors on any device. Only shapes and
dtypes are read.
Returns: list[str]
The released Hugging Face names, in the order in which to_hf returns them for an unsharded
Raises:
ValueError: If a tensor of the state is a DTensor, or the unsharded conversion rejects the state.RuntimeError: If a load session is open, or an earlier session was abandoned, replaced or failed.NotImplementedError: If the configuration has been changed to MTP layers or tied weights.
Return the released layout of the native state, as load destinations or as tensors to save.
The state is treated as sharded when its tensors are DTensors. With for_checkpoint_load the result
opens a load session that from_hf must close; without it the result is what a save writes.
Parameters:
Exactly the native tensors of the model under their model. names, in the shapes of
the model’s parameters and buffers. Either all plain tensors, or all DTensors in the layouts
required by model_hf_dcp_load_destinations (load) and model_hf_distributed_export
(save): for a DTensor the shapes are global and each rank holds its shard.
Must be false; quantized checkpoints are not supported.
Return destinations for a checkpoint read instead of tensors to save.
Must be false; the legacy metadata export is not supported.
Other options of the checkpointer (exclude_key_regex, device_mesh,
legacy_paramwrapper_layout). They do not apply to this model and are not read.
Returns: dict[str, torch.Tensor]
Released Hugging Face names mapped to tensors in the released layouts.
Raises:
NotImplementedError: Ifquantizationorv4_compatibleis set, or the configuration has been changed to MTP layers or tied weights.RuntimeError: If a load session is active, or an earlier session was abandoned, replaced or failed.ValueError: If the state is empty (save), mixes plain tensors and DTensors (save), has a name without themodel.prefix, or a conversion rejects its tensors.TypeError: If a sharded state holds a plain tensor where a DTensor is required.
Bases: dict[str, Tensor]
Destinations of one checkpoint load, keyed by Hugging Face name.
The mapping owns the temporaries of the load and the callbacks that install them. The adapter holds it weakly, so dropping the mapping releases the temporaries.
Weak references to the destination tensors that are not tensors of the given state: the views and temporaries that the adapter created for this load.
Callbacks that install the temporaries into the native tensors after the read, in order.
The native tensors that the load fills, named relative to the inner model: the tensors of the
state that to_hf was given.
The destinations as created, to detect a mapping whose keys or tensors were exchanged.
Whether the state is unsharded (plain tensors) rather than sharded (DTensors).
Identity of one load session; two tokens are never the same object.
Return whether a state key names a tensor of a PEFT export: a LoRA adapter tensor, or a tensor of a module
that the export stores whole under the base_model.model. prefix.