Skip to navigation
ReferenceFull Library ReferenceNemo AutomodelNemo AutomodelComponentsModelsErnie5 5nemo_automodel.components.models.ernie5_5.parallelization

nemo_automodel.components.models.ernie5_5.parallelization

View as MarkdownOpen in Claude

Model-owned distributed parallelization for ERNIE 5.5.

ERNIE 5.5 is distributed with FSDP2 and expert parallelism, with or without context parallelism and activation checkpointing. The sidecar rejects every other topology before the shared dispatcher runs, and with it a model whose routed experts exchange tokens through HybridEP unless the model and its FSDP policies compute in BF16. It keeps the shared expert-parallel flow, and adds two things while that flow builds the FSDP units:

  • Every HCA compressor receives the process group of the mesh over which its parameters are sharded. A row shorter than one compression window does not use the compressor. With the group, the ranks that share the compressor parameters agree, for every microbatch, on whether the compressor takes part in backward, so they issue the same FSDP collectives in the same order.
  • The hyper-head becomes an FP32 unit of its own before the root unit, which computes in the bulk dtype, claims its parameters.

No other trainable parameter of the model is used by some ranks and not by others, so the sidecar does not enable FSDP’s reduction of unused parameters.

After the shared flow has sharded the routed experts, the sidecar records the expert-parallel size on them: a PEFT checkpoint save reads that attribute to decide whether every rank collects the adapter tensors itself.

Module Contents

Classes

NameDescription
Ernie5_5ModelParallelizerDistribute ERNIE 5.5 with FSDP2 and expert parallelism, and reject every other topology.
_FullyShardKwargsKeyword arguments with which the shared flows call _fully_shard_module.

Functions

NameDescription
_axis_namesReturn the axis names of a mesh as text, for example dp_replicate, dp_shard.
_is_pipeline_stageReturn whether pipeline-stage surgery removed modules or decoder layers from an ERNIE 5.5 model.

Data

PARALLELIZER

_COMPUTE_DTYPES

_HYPER_HEAD_FP32_NAMES

logger

API

class nemo_automodel.components.models.ernie5_5.parallelization.Ernie5_5ModelParallelizer()

Bases: ModelParallelizer

Distribute ERNIE 5.5 with FSDP2 and expert parallelism, and reject every other topology.

The supported topology is distributed.strategy: fsdp2 with ep_size greater than one, FSDP sharding over all ranks of the job (tp_size, pp_size and dp_replicate_size equal to one) and FSDP compute in FP32 or BF16. cp_size splits every row over that many ranks, and activation_checkpointing: true recomputes whole decoder layers in backward; the selective and MoE-only modes are not supported. A model built with the HybridEP dispatcher (model.backend.dispatcher: hybridep) is admitted for BF16 compute only: a BF16 model whose FSDP policies have no compute dtype or BF16. The model was validated with HybridEP in BF16 only, and the HybridEP combine fails a C++ assertion, which aborts the process, for activations that are not 16 bits wide.

A model with a DSA layer also restricts the number of ranks: it must divide config.hybrid_mla_num_attention_heads. A checkpoint stores the key and value projections of a DSA layer as one matrix with the rows of each attention head together, and the checkpoint adapter loads and saves that matrix only when every rank holds the rows of whole heads. With 64 heads the supported jobs have 2, 4, 8, 16, 32 or 64 ranks.

nemo_automodel.components.models.ernie5_5.parallelization.Ernie5_5ModelParallelizer._apply_moe(
model: torch.nn.Module,
) -> torch.nn.Module

Run the shared expert-parallel flow, verify it, and record the expert-parallel size on the experts.

For this model, which has neither pipeline stages nor quantized modules, a PEFT checkpoint save lets every rank collect the adapter tensors itself only when a module carries an ep_size attribute greater than one; otherwise only rank 0 receives them and the other ranks hand an empty state to the checkpoint adapter. Expert modules with a token dispatcher set the attribute when the shared flow initializes the dispatcher. The torch experts of this model have no dispatcher, so the sidecar sets it once the shared flow has sharded them and its post-condition holds. A save without PEFT does not read the attribute.

Parameters:

model
nn.Module

The Ernie5_5_MoeForCausalLM to distribute, admitted by _prepare_model.

mesh_context
MeshContext

Meshes, parallelism sizes and strategy of the run.

Returns: nn.Module

The model, with its parameters sharded by the shared expert-parallel flow and ep_size set on the

Raises:

  • RuntimeError: If the shared flow returned without making the hyper-head an FSDP unit.
nemo_automodel.components.models.ernie5_5.parallelization.Ernie5_5ModelParallelizer._fully_shard_module(
module: torch.nn.Module,
) -> torch.nn.Module

Shard module after giving ERNIE-specific units inside it their boundary and group.

The shared flow calls this hook for each decoder block, for the modules inside a block that compute in FP32, for the embedding, the output head and the roots. Whichever call contains an HCA compressor sets the compressor’s synchronization group from the mesh of that call; setting the same group again is harmless. The call that contains the hyper-head shards it first as an FP32 unit.

The check of the mesh is a post-condition of the shared flow, not an admission check: when it fails, the flow has already distributed the expert weights. A mesh context built by MeshContext.build that _prepare_model admitted yields a one-dimensional mesh.

Parameters:

module
nn.Module

Module that becomes an FSDP unit, with parameters of any shape.

**kwargs
Unpack[_FullyShardKwargs]Defaults to {}

Arguments of fully_shard. mesh is the one-dimensional data-parallel mesh over which the parameters of module are sharded on their leading axis. ignored_params holds parameters of any shape that another unit already owns; it is passed to fully_shard unchanged, as are mp_policy, offload_policy and reshard_after_forward.

Returns: nn.Module

module as an FSDP unit.

Raises:

  • ValueError: If mesh has more than one dimension.
nemo_automodel.components.models.ernie5_5.parallelization.Ernie5_5ModelParallelizer._prepare_model(
model: torch.nn.Module,
) -> None

Reject every unsupported topology before an executor changes the model.

The shared dispatcher has not run when this hook is called, so a rejected model keeps its parameters, its modules and its synchronization groups as they were handed to the hook. The hook runs for every strategy, including DDP and Megatron-FSDP. Under pipeline parallelism the framework has already copied the model and cut the copies into stages when it calls the hook, once per stage; the hook rejects the stage.

One condition is not decided here. The mesh over which the FSDP units are sharded is derived inside the shared expert-parallel flow, so _fully_shard_module checks its number of dimensions as a post-condition.

Parameters:

model
nn.Module

The Ernie5_5_MoeForCausalLM to distribute.

mesh_context
MeshContext

Meshes, parallelism sizes, strategy, precision policies and activation-checkpointing mode of the run.

Raises:

  • ValueError: If the strategy is not FSDP2; if the tensor- or pipeline-parallel size or the replicate size is not one; if model is a pipeline stage; if expert parallelism is off; if activation checkpointing is neither off nor true; if an FSDP compute dtype is neither FP32 nor BF16; if the model was built with the HybridEP dispatcher and its dtype or an FSDP compute dtype is FP32; if the data-parallel mesh does not cover all ranks of the job; or if the model has a DSA layer and the number of ranks does not divide its number of DSA attention heads.
class nemo_automodel.components.models.ernie5_5.parallelization._FullyShardKwargs

Bases: typing.TypedDict

Keyword arguments with which the shared flows call _fully_shard_module.

ignored_params
set[Parameter] | None
mesh
Required[DeviceMesh]
mp_policy
MixedPrecisionPolicy | None
offload_policy
OffloadPolicy | None
reshard_after_forward
bool | int | None
nemo_automodel.components.models.ernie5_5.parallelization._axis_names(
mesh: torch.distributed.device_mesh.DeviceMesh
) -> str

Return the axis names of a mesh as text, for example dp_replicate, dp_shard.

nemo_automodel.components.models.ernie5_5.parallelization._is_pipeline_stage(
model: torch.nn.Module
) -> bool

Return whether pipeline-stage surgery removed modules or decoder layers from an ERNIE 5.5 model.

Parameters:

model
nn.Module

The Ernie5_5_MoeForCausalLM handed to the sidecar.

Returns: bool

True when the embedding, the hyper-head, the final norm, the output head or MultiMax is missing, or the

nemo_automodel.components.models.ernie5_5.parallelization.PARALLELIZER = Ernie5_5ModelParallelizer()
nemo_automodel.components.models.ernie5_5.parallelization._COMPUTE_DTYPES = (None, torch.float32, torch.bfloat16)
nemo_automodel.components.models.ernie5_5.parallelization._HYPER_HEAD_FP32_NAMES = ('hc_fn', 'hc_base', 'hc_scale')
nemo_automodel.components.models.ernie5_5.parallelization.logger = logging.getLogger(__name__)