> This page is for version 0.4.0 · 26.04.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.moe.nccl_ep

Eager, dropless NCCL-EP token transport through Transformer Engine.

## Module Contents

### Classes

| Name                                                                    | Description                                                                               |
| ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| [`NcclEpManager`](#nemo_automodel-components-moe-nccl_ep-NcclEpManager) | Layer-local routing state for the flex dispatcher's eager NCCL-EP path.                   |
| [`_NcclTeam`](#nemo_automodel-components-moe-nccl_ep-_NcclTeam)         | NCCL's `ncclTeam_t`: the team's rank count, this rank's index in it, and its rank stride. |
| [`_Runtime`](#nemo_automodel-components-moe-nccl_ep-_Runtime)           | -                                                                                         |

### Functions

| Name                                                                                                | Description                                                                                 |
| --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| [`_join_transport`](#nemo_automodel-components-moe-nccl_ep-_join_transport)                         | Bootstrap the process-wide transport once, or check that it matches these settings.         |
| [`_legacy_group_for`](#nemo_automodel-components-moe-nccl_ep-_legacy_group_for)                     | Create a classic NCCL group over `group`'s ranks when `group` cannot lend its communicator. |
| [`_nvlink_domain_size`](#nemo_automodel-components-moe-nccl_ep-_nvlink_domain_size)                 | Return how many ranks of `group` share this rank's NVLink domain, as NCCL-EP sees it.       |
| [`_ordered_layout_environment`](#nemo_automodel-components-moe-nccl_ep-_ordered_layout_environment) | Select NCCL-EP's ordered expert layout for groups created inside the block.                 |
| [`finalize_nccl_ep`](#nemo_automodel-components-moe-nccl_ep-finalize_nccl_ep)                       | Release the owned NCCL-EP context before destroying its process group.                      |

### Data

[`_ORDERED_LAYOUT_ENV`](#nemo_automodel-components-moe-nccl_ep-_ORDERED_LAYOUT_ENV)

[`_runtime`](#nemo_automodel-components-moe-nccl_ep-_runtime)

### API

```python
class nemo_automodel.components.moe.nccl_ep.NcclEpManager(
    config: nemo_automodel.components.moe.config.NcclEpConfig,
    group: torch.distributed.ProcessGroup,
    hidden_dim: int,
    num_experts: int,
    top_k: int
)
```

Layer-local routing state for the flex dispatcher's eager NCCL-EP path.

**`_num_tokens`** `= 0`

---

**`api`**

---

**`capacity`** `= (config.max_tokens_per_rank + 63) // 64 * 64`

---

**`dispatched_probs`** `Tensor | None = None`

---

**`max_tokens`** `= config.max_tokens_per_rank`

---

**`num_local_experts`** `= num_experts // group.size()`

---

**`token_indices`** `Tensor | None = None`

---

**`token_probs`** `Tensor | None = None`

---

**`tokens_per_expert`** `Tensor | None = None`

---

```python
nemo_automodel.components.moe.nccl_ep.NcclEpManager._release_dispatch_state() -> None
```

Drop the handle buffer and routing tensors of this manager's latest dispatch.

```python
nemo_automodel.components.moe.nccl_ep.NcclEpManager.combine(
    hidden_states: torch.Tensor,
    async_finish: bool = False,
    allocate_on_comm_stream: bool = False
) -> torch.Tensor
```

Combine expert outputs back into the dispatched token order.

**Parameters:**

**`hidden_states`** `torch.Tensor`

BF16 tensor of shape \[received_tokens, hidden],
expert-packed and already multiplied by routing weights.

---

**`async_finish`** `bool` — default: False

Must be false; TE owns stream ordering.

---

**`allocate_on_comm_stream`** `bool` — default: False

Must be false; TE owns allocations.

---

**Returns:** `torch.Tensor`

BF16 tensor of shape \[local_tokens, hidden] in original token order.

```python
nemo_automodel.components.moe.nccl_ep.NcclEpManager.dispatch(
    hidden_states: torch.Tensor,
    async_finish: bool = False,
    allocate_on_comm_stream: bool = False
) -> torch.Tensor
```

Dispatch into a fresh expert-packed buffer with differentiable weights.

**Parameters:**

**`hidden_states`** `torch.Tensor`

BF16 CUDA tensor of shape \[local_tokens, hidden], with at
most `max_tokens_per_rank` rows.

---

**`async_finish`** `bool` — default: False

Must be false; TE owns stream ordering.

---

**`allocate_on_comm_stream`** `bool` — default: False

Must be false; TE owns allocations.

---

**Returns:** `torch.Tensor`

BF16 tensor of shape \[received_tokens, hidden], packed by local expert and,

```python
nemo_automodel.components.moe.nccl_ep.NcclEpManager.get_number_of_tokens_per_expert() -> torch.Tensor
```

Return device counts of shape \[local_experts] for the packed rows.

```python
nemo_automodel.components.moe.nccl_ep.NcclEpManager.get_permuted_hidden_states_by_experts(
    hidden_states: torch.Tensor
) -> tuple[torch.Tensor, torch.Tensor]
```

Return the already packed dispatch result without a second permutation.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape \[received_tokens, hidden], expert-packed.

---

**Returns:** `torch.Tensor`

The input activation tensor of shape \[received_tokens, hidden], returned by

```python
nemo_automodel.components.moe.nccl_ep.NcclEpManager.get_restored_hidden_states_by_experts(
    hidden_states: torch.Tensor
) -> torch.Tensor
```

Keep expert-packed outputs in the layout consumed by combine.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape \[received_tokens, hidden], expert-packed.

---

**Returns:** `torch.Tensor`

The input tensor of shape \[received_tokens, hidden], returned by reference

```python
nemo_automodel.components.moe.nccl_ep.NcclEpManager.setup_metadata(
    num_local_tokens: int,
    probs: torch.Tensor
) -> None
```

Select top-k from dense per-rank routing weights.

**Parameters:**

**`num_local_tokens`** `int`

Local input token count.

---

**`probs`** `torch.Tensor`

Tensor of shape \[tokens, ep_size, local_experts], with zero weights
for inactive routes. The expert axes enumerate contiguous global expert
IDs; ep_size is the number of ranks in the EP group.

---

```python
class nemo_automodel.components.moe.nccl_ep._NcclTeam()
```

**Bases:** `Structure`

NCCL's `ncclTeam_t`: the team's rank count, this rank's index in it, and its rank stride.

**`_fields_`**

---

```python
class nemo_automodel.components.moe.nccl_ep._Runtime(
    api: types.ModuleType | None = None,
    group: torch.distributed.ProcessGroup | None = None,
    settings: tuple[int, ...] | None = None,
    legacy_source: torch.distributed.ProcessGroup | None = None,
    legacy_group: torch.distributed.ProcessGroup | None = None,
    pending: nemo_automodel.components.moe.nccl_ep.NcclEpManager | None = None
)
```

Dataclass

**`api`** `ModuleType | None = None`

---

**`group`** `ProcessGroup | None = None`

---

**`legacy_group`** `ProcessGroup | None = None`

---

**`legacy_source`** `ProcessGroup | None = None`

---

**`pending`** `NcclEpManager | None = None`

---

**`settings`** `tuple[int, ...] | None = None`

---

```python
nemo_automodel.components.moe.nccl_ep._join_transport(
    group: torch.distributed.ProcessGroup,
    capacity: int,
    hidden_dim: int,
    num_experts: int,
    top_k: int
) -> types.ModuleType
```

Bootstrap the process-wide transport once, or check that it matches these settings.

**Parameters:**

**`group`** `dist.ProcessGroup`

NCCL expert-parallel group with contiguous expert ownership.

---

**`capacity`** `int`

Group-uniform input extent, a multiple of 64.

---

**`hidden_dim`** `int`

Width of each dispatched token.

---

**`num_experts`** `int`

Global expert count across the EP group.

---

**`top_k`** `int`

Distinct expert selections per token.

---

**Returns:** `ModuleType`

Transformer Engine's NCCL-EP API module.

```python
nemo_automodel.components.moe.nccl_ep._legacy_group_for(
    group: torch.distributed.ProcessGroup
) -> torch.distributed.ProcessGroup | None
```

Create a classic NCCL group over `group`'s ranks when `group` cannot lend its communicator.

Transformer Engine 2.20 borrows the raw NCCL communicator through the CUDA backend's
`_comm_ptr()`. PyTorch 2.15 builds `backend="nccl"` groups on `ProcessGroupNCCL2`,
which does not expose it, so NCCL-EP then runs on a `nccl-legacy` group of the same ranks.

**Parameters:**

**`group`** `dist.ProcessGroup`

NCCL expert-parallel group.

---

**Returns:** `dist.ProcessGroup | None`

An initialized `nccl-legacy` group, or None when `group` lends its communicator.

```python
nemo_automodel.components.moe.nccl_ep._nvlink_domain_size(
    group: torch.distributed.ProcessGroup
) -> int
```

Return how many ranks of `group` share this rank's NVLink domain, as NCCL-EP sees it.

NCCL-EP partitions its group into NCCL's load/store-accessible (LSA) teams, the ranks that
reach each other's memory over NVLink, and runs cross-domain kernels when there are several.
`ncclTeamLsa` reports this rank's team for the communicator that NCCL-EP borrows. The query
is local; NCCL-EP repeats it when it creates its group.

**Parameters:**

**`group`** `dist.ProcessGroup`

Classic NCCL group whose communicator NCCL-EP borrows.

---

**Returns:** `int`

The LSA team size, a divisor of `group.size()`.

```python
nemo_automodel.components.moe.nccl_ep._ordered_layout_environment() -> collections.abc.Iterator[None]
```

Select NCCL-EP's ordered expert layout for groups created inside the block.

```python
nemo_automodel.components.moe.nccl_ep.finalize_nccl_ep() -> None
```

Release the owned NCCL-EP context before destroying its process group.

```python
nemo_automodel.components.moe.nccl_ep._ORDERED_LAYOUT_ENV = 'NCCL_EP_HT_EM_AG_SCAN_MODE'
```

```python
nemo_automodel.components.moe.nccl_ep._runtime = _Runtime()
```