Skip to navigation

nemo_automodel.components.moe.nccl_ep

View as MarkdownOpen in Claude

Eager, dropless NCCL-EP token transport through Transformer Engine.

Module Contents

Classes

NameDescription
NcclEpManagerLayer-local routing state for the flex dispatcher’s eager NCCL-EP path.
_NcclTeamNCCL’s ncclTeam_t: the team’s rank count, this rank’s index in it, and its rank stride.
_Runtime-

Functions

NameDescription
_join_transportBootstrap the process-wide transport once, or check that it matches these settings.
_legacy_group_forCreate a classic NCCL group over group’s ranks when group cannot lend its communicator.
_nvlink_domain_sizeReturn how many ranks of group share this rank’s NVLink domain, as NCCL-EP sees it.
_ordered_layout_environmentSelect NCCL-EP’s ordered expert layout for groups created inside the block.
finalize_nccl_epRelease the owned NCCL-EP context before destroying its process group.

Data

_ORDERED_LAYOUT_ENV

_runtime

API

class nemo_automodel.components.moe.nccl_ep.NcclEpManager(
group: torch.distributed.ProcessGroup,
hidden_dim: int,
num_experts: int,
top_k: int
)

Layer-local routing state for the flex dispatcher’s eager NCCL-EP path.

_num_tokens
= 0
api
capacity
= (config.max_tokens_per_rank + 63) // 64 * 64
dispatched_probs
Tensor | None = None
max_tokens
= config.max_tokens_per_rank
num_local_experts
= num_experts // group.size()
token_indices
Tensor | None = None
token_probs
Tensor | None = None
tokens_per_expert
Tensor | None = None
nemo_automodel.components.moe.nccl_ep.NcclEpManager._release_dispatch_state() -> None

Drop the handle buffer and routing tensors of this manager’s latest dispatch.

nemo_automodel.components.moe.nccl_ep.NcclEpManager.combine(
hidden_states: torch.Tensor,
async_finish: bool = False,
allocate_on_comm_stream: bool = False
) -> torch.Tensor

Combine expert outputs back into the dispatched token order.

Parameters:

hidden_states
torch.Tensor

BF16 tensor of shape [received_tokens, hidden], expert-packed and already multiplied by routing weights.

async_finish
boolDefaults to False

Must be false; TE owns stream ordering.

allocate_on_comm_stream
boolDefaults to False

Must be false; TE owns allocations.

Returns: torch.Tensor

BF16 tensor of shape [local_tokens, hidden] in original token order.

nemo_automodel.components.moe.nccl_ep.NcclEpManager.dispatch(
hidden_states: torch.Tensor,
async_finish: bool = False,
allocate_on_comm_stream: bool = False
) -> torch.Tensor

Dispatch into a fresh expert-packed buffer with differentiable weights.

Parameters:

hidden_states
torch.Tensor

BF16 CUDA tensor of shape [local_tokens, hidden], with at most max_tokens_per_rank rows.

async_finish
boolDefaults to False

Must be false; TE owns stream ordering.

allocate_on_comm_stream
boolDefaults to False

Must be false; TE owns allocations.

Returns: torch.Tensor

BF16 tensor of shape [received_tokens, hidden], packed by local expert and,

nemo_automodel.components.moe.nccl_ep.NcclEpManager.get_number_of_tokens_per_expert() -> torch.Tensor

Return device counts of shape [local_experts] for the packed rows.

nemo_automodel.components.moe.nccl_ep.NcclEpManager.get_permuted_hidden_states_by_experts(
hidden_states: torch.Tensor
) -> tuple[torch.Tensor, torch.Tensor]

Return the already packed dispatch result without a second permutation.

Parameters:

hidden_states
torch.Tensor

Tensor of shape [received_tokens, hidden], expert-packed.

Returns: torch.Tensor

The input activation tensor of shape [received_tokens, hidden], returned by

nemo_automodel.components.moe.nccl_ep.NcclEpManager.get_restored_hidden_states_by_experts(
hidden_states: torch.Tensor
) -> torch.Tensor

Keep expert-packed outputs in the layout consumed by combine.

Parameters:

hidden_states
torch.Tensor

Tensor of shape [received_tokens, hidden], expert-packed.

Returns: torch.Tensor

The input tensor of shape [received_tokens, hidden], returned by reference

nemo_automodel.components.moe.nccl_ep.NcclEpManager.setup_metadata(
num_local_tokens: int,
probs: torch.Tensor
) -> None

Select top-k from dense per-rank routing weights.

Parameters:

num_local_tokens
int

Local input token count.

probs
torch.Tensor

Tensor of shape [tokens, ep_size, local_experts], with zero weights for inactive routes. The expert axes enumerate contiguous global expert IDs; ep_size is the number of ranks in the EP group.

class nemo_automodel.components.moe.nccl_ep._NcclTeam()

Bases: Structure

NCCL’s ncclTeam_t: the team’s rank count, this rank’s index in it, and its rank stride.

_fields_
class nemo_automodel.components.moe.nccl_ep._Runtime(
api: types.ModuleType | None = None,
group: torch.distributed.ProcessGroup | None = None,
settings: tuple[int, ...] | None = None,
legacy_source: torch.distributed.ProcessGroup | None = None,
legacy_group: torch.distributed.ProcessGroup | None = None,
)
Dataclass
api
ModuleType | None = None
group
ProcessGroup | None = None
legacy_group
ProcessGroup | None = None
legacy_source
ProcessGroup | None = None
pending
NcclEpManager | None = None
settings
tuple[int, ...] | None = None
nemo_automodel.components.moe.nccl_ep._join_transport(
group: torch.distributed.ProcessGroup,
capacity: int,
hidden_dim: int,
num_experts: int,
top_k: int
) -> types.ModuleType

Bootstrap the process-wide transport once, or check that it matches these settings.

Parameters:

group
dist.ProcessGroup

NCCL expert-parallel group with contiguous expert ownership.

capacity
int

Group-uniform input extent, a multiple of 64.

hidden_dim
int

Width of each dispatched token.

num_experts
int

Global expert count across the EP group.

top_k
int

Distinct expert selections per token.

Returns: ModuleType

Transformer Engine’s NCCL-EP API module.

nemo_automodel.components.moe.nccl_ep._legacy_group_for(
group: torch.distributed.ProcessGroup
) -> torch.distributed.ProcessGroup | None

Create a classic NCCL group over group’s ranks when group cannot lend its communicator.

Transformer Engine 2.20 borrows the raw NCCL communicator through the CUDA backend’s _comm_ptr(). PyTorch 2.15 builds backend="nccl" groups on ProcessGroupNCCL2, which does not expose it, so NCCL-EP then runs on a nccl-legacy group of the same ranks.

Parameters:

group
dist.ProcessGroup

NCCL expert-parallel group.

Returns: dist.ProcessGroup | None

An initialized nccl-legacy group, or None when group lends its communicator.

nemo_automodel.components.moe.nccl_ep._nvlink_domain_size(
group: torch.distributed.ProcessGroup
) -> int

Return how many ranks of group share this rank’s NVLink domain, as NCCL-EP sees it.

NCCL-EP partitions its group into NCCL’s load/store-accessible (LSA) teams, the ranks that reach each other’s memory over NVLink, and runs cross-domain kernels when there are several. ncclTeamLsa reports this rank’s team for the communicator that NCCL-EP borrows. The query is local; NCCL-EP repeats it when it creates its group.

Parameters:

group
dist.ProcessGroup

Classic NCCL group whose communicator NCCL-EP borrows.

Returns: int

The LSA team size, a divisor of group.size().

nemo_automodel.components.moe.nccl_ep._ordered_layout_environment() -> collections.abc.Iterator[None]

Select NCCL-EP’s ordered expert layout for groups created inside the block.

nemo_automodel.components.moe.nccl_ep.finalize_nccl_ep() -> None

Release the owned NCCL-EP context before destroying its process group.

nemo_automodel.components.moe.nccl_ep._ORDERED_LAYOUT_ENV = 'NCCL_EP_HT_EM_AG_SCAN_MODE'
nemo_automodel.components.moe.nccl_ep._runtime = _Runtime()