nemo_automodel.components.moe.nccl_ep
nemo_automodel.components.moe.nccl_ep
Eager, dropless NCCL-EP token transport through Transformer Engine.
Module Contents
Classes
Functions
Data
API
Layer-local routing state for the flex dispatcher’s eager NCCL-EP path.
Drop the handle buffer and routing tensors of this manager’s latest dispatch.
Combine expert outputs back into the dispatched token order.
Parameters:
BF16 tensor of shape [received_tokens, hidden], expert-packed and already multiplied by routing weights.
Must be false; TE owns stream ordering.
Must be false; TE owns allocations.
Returns: torch.Tensor
BF16 tensor of shape [local_tokens, hidden] in original token order.
Dispatch into a fresh expert-packed buffer with differentiable weights.
Parameters:
BF16 CUDA tensor of shape [local_tokens, hidden], with at
most max_tokens_per_rank rows.
Must be false; TE owns stream ordering.
Must be false; TE owns allocations.
Returns: torch.Tensor
BF16 tensor of shape [received_tokens, hidden], packed by local expert and,
Return device counts of shape [local_experts] for the packed rows.
Return the already packed dispatch result without a second permutation.
Parameters:
Tensor of shape [received_tokens, hidden], expert-packed.
Returns: torch.Tensor
The input activation tensor of shape [received_tokens, hidden], returned by
Keep expert-packed outputs in the layout consumed by combine.
Parameters:
Tensor of shape [received_tokens, hidden], expert-packed.
Returns: torch.Tensor
The input tensor of shape [received_tokens, hidden], returned by reference
Select top-k from dense per-rank routing weights.
Parameters:
Local input token count.
Tensor of shape [tokens, ep_size, local_experts], with zero weights for inactive routes. The expert axes enumerate contiguous global expert IDs; ep_size is the number of ranks in the EP group.
Bases: Structure
NCCL’s ncclTeam_t: the team’s rank count, this rank’s index in it, and its rank stride.
Bootstrap the process-wide transport once, or check that it matches these settings.
Parameters:
NCCL expert-parallel group with contiguous expert ownership.
Group-uniform input extent, a multiple of 64.
Width of each dispatched token.
Global expert count across the EP group.
Distinct expert selections per token.
Returns: ModuleType
Transformer Engine’s NCCL-EP API module.
Create a classic NCCL group over group’s ranks when group cannot lend its communicator.
Transformer Engine 2.20 borrows the raw NCCL communicator through the CUDA backend’s
_comm_ptr(). PyTorch 2.15 builds backend="nccl" groups on ProcessGroupNCCL2,
which does not expose it, so NCCL-EP then runs on a nccl-legacy group of the same ranks.
Parameters:
NCCL expert-parallel group.
Returns: dist.ProcessGroup | None
An initialized nccl-legacy group, or None when group lends its communicator.
Return how many ranks of group share this rank’s NVLink domain, as NCCL-EP sees it.
NCCL-EP partitions its group into NCCL’s load/store-accessible (LSA) teams, the ranks that
reach each other’s memory over NVLink, and runs cross-domain kernels when there are several.
ncclTeamLsa reports this rank’s team for the communicator that NCCL-EP borrows. The query
is local; NCCL-EP repeats it when it creates its group.
Parameters:
Classic NCCL group whose communicator NCCL-EP borrows.
Returns: int
The LSA team size, a divisor of group.size().
Select NCCL-EP’s ordered expert layout for groups created inside the block.
Release the owned NCCL-EP context before destroying its process group.