> This page is for version 0.4.0 · 26.04.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.models.ernie5_5.parallelization

Model-owned distributed parallelization for ERNIE 5.5.

ERNIE 5.5 is distributed with FSDP2 and expert parallelism, with or without context parallelism and activation
checkpointing. The sidecar rejects every other topology before the shared dispatcher runs, and with it a model
whose routed experts exchange tokens through HybridEP unless the model and its FSDP policies compute in BF16. It
keeps the shared expert-parallel flow, and adds two things while that flow builds the FSDP units:

* Every HCA compressor receives the process group of the mesh over which its parameters are sharded. A row shorter
  than one compression window does not use the compressor. With the group, the ranks that share the compressor
  parameters agree, for every microbatch, on whether the compressor takes part in backward, so they issue the same
  FSDP collectives in the same order.
* The hyper-head becomes an FP32 unit of its own before the root unit, which computes in the bulk dtype, claims its
  parameters.

No other trainable parameter of the model is used by some ranks and not by others, so the sidecar does not enable
FSDP's reduction of unused parameters.

After the shared flow has sharded the routed experts, the sidecar records the expert-parallel size on them: a PEFT
checkpoint save reads that attribute to decide whether every rank collects the adapter tensors itself.

## Module Contents

### Classes

| Name                                                                                                                | Description                                                                              |
| ------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| [`Ernie5_5ModelParallelizer`](#nemo_automodel-components-models-ernie5_5-parallelization-Ernie5_5ModelParallelizer) | Distribute ERNIE 5.5 with FSDP2 and expert parallelism, and reject every other topology. |
| [`_FullyShardKwargs`](#nemo_automodel-components-models-ernie5_5-parallelization-_FullyShardKwargs)                 | Keyword arguments with which the shared flows call `_fully_shard_module`.                |

### Functions

| Name                                                                                                  | Description                                                                                      |
| ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| [`_axis_names`](#nemo_automodel-components-models-ernie5_5-parallelization-_axis_names)               | Return the axis names of a mesh as text, for example `dp_replicate, dp_shard`.                   |
| [`_is_pipeline_stage`](#nemo_automodel-components-models-ernie5_5-parallelization-_is_pipeline_stage) | Return whether pipeline-stage surgery removed modules or decoder layers from an ERNIE 5.5 model. |

### Data

[`PARALLELIZER`](#nemo_automodel-components-models-ernie5_5-parallelization-PARALLELIZER)

[`_COMPUTE_DTYPES`](#nemo_automodel-components-models-ernie5_5-parallelization-_COMPUTE_DTYPES)

[`_HYPER_HEAD_FP32_NAMES`](#nemo_automodel-components-models-ernie5_5-parallelization-_HYPER_HEAD_FP32_NAMES)

[`logger`](#nemo_automodel-components-models-ernie5_5-parallelization-logger)

### API

```python
class nemo_automodel.components.models.ernie5_5.parallelization.Ernie5_5ModelParallelizer()
```

**Bases:** `ModelParallelizer`

Distribute ERNIE 5.5 with FSDP2 and expert parallelism, and reject every other topology.

The supported topology is `distributed.strategy: fsdp2` with `ep_size` greater than one, FSDP sharding
over all ranks of the job (`tp_size`, `pp_size` and `dp_replicate_size` equal to one) and FSDP compute in
FP32 or BF16. `cp_size` splits every row over that many ranks, and `activation_checkpointing: true`
recomputes whole decoder layers in backward; the selective and MoE-only modes are not supported.
A model built with the HybridEP dispatcher (`model.backend.dispatcher: hybridep`) is admitted for BF16
compute only: a BF16 model whose FSDP policies have no compute dtype or BF16. The model was validated with
HybridEP in BF16 only, and the HybridEP combine fails a C++ assertion, which aborts the process, for
activations that are not 16 bits wide.

A model with a DSA layer also restricts the number of ranks: it must divide
`config.hybrid_mla_num_attention_heads`. A checkpoint stores the key and value projections of a DSA layer as
one matrix with the rows of each attention head together, and the checkpoint adapter loads and saves that
matrix only when every rank holds the rows of whole heads. With 64 heads the supported jobs have 2, 4, 8, 16,
32 or 64 ranks.

```python
nemo_automodel.components.models.ernie5_5.parallelization.Ernie5_5ModelParallelizer._apply_moe(
    model: torch.nn.Module,
    mesh_context: nemo_automodel.components.distributed.MeshContext
) -> torch.nn.Module
```

Run the shared expert-parallel flow, verify it, and record the expert-parallel size on the experts.

For this model, which has neither pipeline stages nor quantized modules, a PEFT checkpoint save lets every
rank collect the adapter tensors itself only when a module carries an `ep_size` attribute greater than
one; otherwise only rank 0 receives them and the other ranks hand an empty state to the checkpoint
adapter. Expert modules with a token dispatcher set the attribute when the shared flow initializes the
dispatcher. The torch experts of this model have no dispatcher, so the sidecar sets it once the shared
flow has sharded them and its post-condition holds. A save without PEFT does not read the attribute.

**Parameters:**

**`model`** `nn.Module`

The `Ernie5_5_MoeForCausalLM` to distribute, admitted by `_prepare_model`.

---

**`mesh_context`** `MeshContext`

Meshes, parallelism sizes and strategy of the run.

---

**Returns:** `nn.Module`

The model, with its parameters sharded by the shared expert-parallel flow and `ep_size` set on the

**Raises:**

* `RuntimeError`: If the shared flow returned without making the hyper-head an FSDP unit.

```python
nemo_automodel.components.models.ernie5_5.parallelization.Ernie5_5ModelParallelizer._fully_shard_module(
    module: torch.nn.Module,
    kwargs: typing.Unpack[nemo_automodel.components.models.ernie5_5.parallelization._FullyShardKwargs] = {}
) -> torch.nn.Module
```

Shard `module` after giving ERNIE-specific units inside it their boundary and group.

The shared flow calls this hook for each decoder block, for the modules inside a block that compute in
FP32, for the embedding, the output head and the roots. Whichever call contains an HCA compressor sets the
compressor's synchronization group from the mesh of that call; setting the same group again is harmless.
The call that contains the hyper-head shards it first as an FP32 unit.

The check of the mesh is a post-condition of the shared flow, not an admission check: when it fails, the
flow has already distributed the expert weights. A mesh context built by `MeshContext.build` that
`_prepare_model` admitted yields a one-dimensional mesh.

**Parameters:**

**`module`** `nn.Module`

Module that becomes an FSDP unit, with parameters of any shape.

---

**`**kwargs`** `Unpack[_FullyShardKwargs]` — default: \{}

Arguments of `fully_shard`. `mesh` is the one-dimensional data-parallel mesh over which
the parameters of `module` are sharded on their leading axis. `ignored_params` holds
parameters of any shape that another unit already owns; it is passed to `fully_shard`
unchanged, as are `mp_policy`, `offload_policy` and `reshard_after_forward`.

---

**Returns:** `nn.Module`

`module` as an FSDP unit.

**Raises:**

* `ValueError`: If `mesh` has more than one dimension.

```python
nemo_automodel.components.models.ernie5_5.parallelization.Ernie5_5ModelParallelizer._prepare_model(
    model: torch.nn.Module,
    mesh_context: nemo_automodel.components.distributed.MeshContext
) -> None
```

Reject every unsupported topology before an executor changes the model.

The shared dispatcher has not run when this hook is called, so a rejected model keeps its parameters, its
modules and its synchronization groups as they were handed to the hook. The hook runs for every strategy,
including DDP and Megatron-FSDP. Under pipeline parallelism the framework has already copied the model
and cut the copies into stages when it calls the hook, once per stage; the hook rejects the stage.

One condition is not decided here. The mesh over which the FSDP units are sharded is derived inside the
shared expert-parallel flow, so `_fully_shard_module` checks its number of dimensions as a
post-condition.

**Parameters:**

**`model`** `nn.Module`

The `Ernie5_5_MoeForCausalLM` to distribute.

---

**`mesh_context`** `MeshContext`

Meshes, parallelism sizes, strategy, precision policies and activation-checkpointing
mode of the run.

---

**Raises:**

* `ValueError`: If the strategy is not FSDP2; if the tensor- or pipeline-parallel size or the
  replicate size is not one; if `model` is a pipeline stage; if expert parallelism is off; if
  activation checkpointing is neither off nor `true`; if an FSDP compute dtype is neither FP32 nor
  BF16; if the model was built with the HybridEP dispatcher and its dtype or an FSDP compute dtype
  is FP32; if the data-parallel mesh does not cover all ranks of the job; or if the model has a
  DSA layer and the number of ranks does not divide its number of DSA attention heads.

```python
class nemo_automodel.components.models.ernie5_5.parallelization._FullyShardKwargs
```

**Bases:** `typing.TypedDict`

Keyword arguments with which the shared flows call `_fully_shard_module`.

**`ignored_params`** `set[Parameter] | None`

---

**`mesh`** `Required[DeviceMesh]`

---

**`mp_policy`** `MixedPrecisionPolicy | None`

---

**`offload_policy`** `OffloadPolicy | None`

---

**`reshard_after_forward`** `bool | int | None`

---

```python
nemo_automodel.components.models.ernie5_5.parallelization._axis_names(
    mesh: torch.distributed.device_mesh.DeviceMesh
) -> str
```

Return the axis names of a mesh as text, for example `dp_replicate, dp_shard`.

```python
nemo_automodel.components.models.ernie5_5.parallelization._is_pipeline_stage(
    model: torch.nn.Module
) -> bool
```

Return whether pipeline-stage surgery removed modules or decoder layers from an ERNIE 5.5 model.

**Parameters:**

**`model`** `nn.Module`

The `Ernie5_5_MoeForCausalLM` handed to the sidecar.

---

**Returns:** `bool`

True when the embedding, the hyper-head, the final norm, the output head or MultiMax is missing, or the

```python
nemo_automodel.components.models.ernie5_5.parallelization.PARALLELIZER = Ernie5_5ModelParallelizer()
```

```python
nemo_automodel.components.models.ernie5_5.parallelization._COMPUTE_DTYPES = (None, torch.float32, torch.bfloat16)
```

```python
nemo_automodel.components.models.ernie5_5.parallelization._HYPER_HEAD_FP32_NAMES = ('hc_fn', 'hc_base', 'hc_scale')
```

```python
nemo_automodel.components.models.ernie5_5.parallelization.logger = logging.getLogger(__name__)
```