Skip to navigation

MoE Grouped Matmul

View as MarkdownOpen in Claude

Overview

The MoE Grouped Matmul operation computes a grouped matrix multiplication across experts, as used in Mixture-of-Experts (MoE) layers. Each expert has its own weight matrix, and tokens are routed to experts via first_token_offset.

Four routing modes are defined; COMBINE currently requires the FROST frost_gemm engine (see FROST COMBINE support below):

None mode (tokens already routed per expert):

Output[1, S×topK, N]=Token[1, S×topK, K] × Weight[E, K, N]\text{Output}[1,\ S \times \text{topK},\ N] = \text{Token}[1,\ S \times \text{topK},\ K]\ \times\ \text{Weight}[E,\ K,\ N]

Gather mode (gather tokens from unrouted layout before matmul):

Output[1, S×topK, N]=Token[1, S, K] × Weight[E, K, N]\text{Output}[1,\ S \times \text{topK},\ N] = \text{Token}[1,\ S,\ K]\ \times\ \text{Weight}[E,\ K,\ N]

Scatter mode (scatter output back to token order after matmul):

First compute the grouped matmul for each routed row, then scatter each result back to the expanded token/top-k output slot using TokenIndex and TokenKs:

GroupedResult[r, N]=Token[1, r, K] × Weight[expert(r), K, N]\text{GroupedResult}[r,\ N] = \text{Token}[1,\ r,\ K]\ \times\ \text{Weight}[\text{expert}(r),\ K,\ N]

Output[1, TokenIndex[r]×topK+TokenKs[r], N]=GroupedResult[r, N]\text{Output}[1,\ \text{TokenIndex}[r] \times \text{topK} + \text{TokenKs}[r],\ N] = \text{GroupedResult}[r,\ N]

Combine mode performs FC2, routing-score multiplication, and summation across selected experts, producing [1, S, N]. See the FROST COMBINE contract below.

where EE = number of experts, SS = number of tokens, KK = hidden size, NN = output (weight) size.

Tensor Roles by Mode

TensorShapeModes
Token[1, S*topK, K] (None/Scatter/Combine) or [1, S, K] (Gather)All
Weight[E, K, N]All
FirstTokenOffset[B*E+1, 1, 1] explicit boundaries; INT32/INT64All
TokenIndex[1, S*topK, 1], INT32Gather, Scatter, Combine
TokenKs[1, S*topK, 1], INT32Scatter, Combine
TopKscalar int32Scatter, Combine
TopKScores[1, S, topK], FP32Combine only

Support Matrix

The support matrix is based on the latest cuDNN backend.

OperationMinimum cuDNNMinimum cublasLtDatatype (I/O)Compute typeFusion pattern
MoE Grouped Matmul (forward)9.18.0—int8, fp8, fp16, bf16, fp32, nvfp4/mxfp8 (9.21.0)FLOAT9.21.0: SwiGLU, AMAX, etc.
MoE Grouped Matmul Bwd9.22.013.5fp16, bf16FLOAT-

Important Notes

  1. Let G=B*E. Use G+1 nondecreasing boundaries starting at zero. Group g spans [offset[g], offset[g+1]) and uses expert g % E. The final boundary may be smaller than the routed capacity (TokenIndex.shape[1] in Gather mode, otherwise Token.shape[1]); trailing routed rows are unused. In None/Gather mode their output rows remain untouched; in Scatter mode only destinations of used routed rows are written. E=1 requires at least two boundaries. FROST supports only explicit boundaries; native backend explicit-offset support requires cuDNN 9.28.0 or newer.
  2. In Scatter mode, both TokenIndex and TokenKs are required, and top_k must be explicitly provided.
  3. In Gather mode, TokenIndex is required.

Deprecated implicit offsets

The length-G format, which infers the final endpoint from routed capacity, is deprecated. It remains supported by native engines for E>1 for compatibility; FROST declines it. For E>1, length modulo E selects the format: 1 for explicit boundaries, 0 for deprecated implicit offsets. No mode parameter is needed.

To migrate, append the old routed-capacity endpoint to the G starts and declare FirstTokenOffset as [G+1,1,1]. This preserves the old group ranges. Use a smaller final endpoint when trailing capacity is intentionally unused. This applies to both forward and backward APIs.

FROST GATHER support

The opt-in frost_gemm engine supports ordinary and block-scaled GATHER on SM100 and SM120, in both orientations: token-by-weight and weight-by-token (SWAP_AB=1). Enable it with CUDNN_FRONTEND_ENABLE_FROST_ENGINES=1 before planning.

Let T be the source token count and R the routed row count. Supply token [1,T,K], weight [E,K,N], INT32 token_index [1,R,1] with contiguous rows, and INT32 or INT64 first_token_offset [G+1,1,1]. The result is [1,R,N]. Offsets are nondecreasing boundaries beginning at zero, with final endpoint <= R; group g uses expert g % E, and empty groups are supported. Each index is a source row in [0,T), so repeated source tokens are allowed.

For block-scaled GATHER, token scale factors (SFA) use reorder=NONE and logical shape [1,T,K/block], with contiguous K scales. Each scale occupies one byte; the row stride must be at least round_up(K/block,16) bytes and a multiple of 16 bytes, and the base pointer must be 16-byte aligned. SFA uses source-token order and is gathered with the same token_index as the token data. Token SFA in F8_128x4 is not supported for GATHER. Weight scale factors (SFB) retain the per-expert F8_128x4 layout. These public operand roles and layout requirements also apply with SWAP_AB=1.

For all block-scaled FROST MoE modes, bind weight SF as a rank-3 byte buffer with E as its outermost axis. Each expert plane must be a dense packed F8_128x4 blob containing at least 512 * ceil(N/128) * ceil((K/block)/4) bytes. Its two inner axes may be permuted. The expert stride must be a multiple of 16 bytes and at least the full inner-plane byte span; padding between experts is supported. The base pointer must also be 16-byte aligned. Views that interleave experts, overlap planes, or expose insufficient plane capacity raise ValueError before launch.

fc1 = graph.moe_grouped_matmul(
token, weight, first_token_offset,
token_index=token_index,
mode=cudnn.moe_grouped_matmul_mode.GATHER,
)
activated = graph.swish(input=fc1)
activated.set_output(True).set_data_type(cudnn.data_type.BFLOAT16)

Routing logits, top-k selection, offsets, and indices are computed by the caller. FROST uses TMA tile::gather4 to load four indexed source rows directly into shared memory, then runs grouped matmul and the supported epilogue in the same kernel. No globally materialized permutation of the token matrix is needed. With SWAP_AB=1, tokens are the internal B operand and GATHER4 loads the token columns of the transposed matmul; the public tensors and routed output order stay the same. SM120 uses the shared::cta GATHER4 instruction with its existing warp-MMA mainloop and STG epilogue; its workspace remains the 128-byte scheduler counter. Both SM100 and SM120 support parallel GEMMs, including the gate/up pair for SwiGLU, sharing the same offsets and indices, in both orientations and for ordinary and block-scaled inputs. Output rows and row-wise epilogue auxiliaries use routed order. The existing dtype, layout, alignment, and fusion support gates still apply.

FROST SCATTER support

FROST supports SCATTER for ordinary and block-scaled MoE on SM100 and SM120, in both orientations (SWAP_AB=0/1). The input and its block scales keep their expert-grouped layout. The epilogue writes each routed row r to the expanded token slot:

destination = token_index[r] * top_k + token_ks[r]
output[0, destination, :] = grouped_result[r, :]

For R = S * top_k, supply token [1,R,K], output [1,R,N], and contiguous INT32 token_index and token_ks, both [1,R,1]. The first index is in [0,S); token_ks is the position within the token’s top-k list, in [0,top_k). Destinations of active routed tokens must be unique; unused output slots are left untouched. 1 \<= top_k \<= E; offsets retain the GATHER/NONE group convention with G+1 explicit boundaries and support empty groups. Routing metadata is supplied by the caller and may change between executions, including graph replay.

fc2 = graph.moe_grouped_matmul(
token, weight, first_token_offset,
token_index=token_index, token_ks=token_ks, top_k=top_k,
mode=cudnn.moe_grouped_matmul_mode.SCATTER,
)
fc2.set_output(True).set_data_type(cudnn.data_type.BFLOAT16)

SM100 uses TMA tile::scatter4 for eligible N-major outputs, including swapAB, and indexed STG for other supported layouts. SM120 uses indexed STG. Pointwise epilogues and multiple materialized outputs share this store logic; token-indexed auxiliary tensors and gen_index use the scattered output coordinates. Input block scales retain their existing segmented SFA/per-expert SFB contract. Output block quantization, reductions, and packed sub-byte outputs are currently declined for SCATTER.

SCATTER preserves each top-k contribution separately. Multiplication by routing weights and reduction across top-k require a subsequent operation to produce the final [1,S,N] result.


FROST COMBINE support

cudnn.moe_grouped_matmul_mode.COMBINE fuses FC2 with weighted scatter-add. It is supported only by the opt-in frost_gemm engine, whose SM100 and SM120 source families both run it for ordinary and block-scaled inputs and both SWAP_AB=0/1 orientations. Native cuDNN engines decline this mode.

Let R be routed input capacity, T the original token count, and k=top_k. Supply token [1,R,K], weight [E,K,N], contiguous INT32 token_index and token_ks of shape [1,R,1], and compact FP32 top_k_scores [1,T,k]. The scores remain in original token/top-k-slot order. 1 \<= k \<= E. Offsets retain the explicit [G+1,1,1] contract; R may include unused capacity and need not equal T*k.

For each active routed row r in group g, the operation computes:

t = token_index[0, r, 0]
j = token_ks[0, r, 0]
output[0, t, :] += (token[0, r, :] @ weight[g % E, :, :]) * top_k_scores[0, t, j]

The output is [1,T,N]; every execution starts from zero, including empty groups and repeated CUDA Graph replays. Out-of-range token/slot indices are ignored. Repeated destinations accumulate; destinations without contributions are zero. N-major and M-major output layouts, including padding, are supported.

For NVFP4 on SM107, the non-swap, single-CTA/singleton-cluster FROST path can read compact token rows through immutable tensor maps. Expert token counts do not need additional padding: COMBINE masks group-tail rows before reading routing metadata or accumulating outputs. Token scales retain their existing per-group 128-row blocked layout and use independent scale-block offsets. The scale tensor map is bounded by the declared segmented-capacity contract, including when the last TMA box extends beyond that capacity. Other layouts and configurations retain their existing descriptor path.

The SM107 NVFP4 swapped 256x256x128, 128x256x64 MMA, cluster 2x1, 2-CTA tile also reuses immutable B/SFB maps with BF16 feature-contiguous output. It transposes weighted fragments into a 128-token by 256-feature shared tile, then bulk-reduces each token’s contiguous feature segment. Warp-shared routing loads and 64-column TMEM loads limit metadata traffic and live registers. The compiler reserves this staging space before sizing the AB pipeline. Partial feature segments or unaligned destinations use scalar reductions. This path does not require expert token counts or output widths to match the tile.

output = graph.moe_grouped_matmul(
fc2_input, fc2_weight, first_token_offset,
token_index=token_index, token_ks=token_ks,
top_k=top_k, top_k_scores=top_k_scores,
mode=cudnn.moe_grouped_matmul_mode.COMBINE,
)
output.set_output(True).set_data_type(cudnn.data_type.BFLOAT16)

GEMM accumulation and score multiplication use FP32. Each weighted contribution is converted to the declared output dtype (BF16, FP16, or FP32), and atomic addition accumulates in that dtype. Atomic order is unspecified, so results are not bitwise deterministic and can differ from an FP32 top-k reduction followed by a single output cast. Choose FP32 output when FP32 combining is required. Scores are neither normalized nor modified by COMBINE.

The first implementation accepts one grouped GEMM, with optional input block-scale dequantization, and one final output. Post-COMBINE pointwise, reduction, quantization, and parallel-GEMM fusion are declined: applying those operations before summing experts would change the graph’s meaning.

The FC2 epilogue issues weighted atomic additions directly to the final output; it does not materialize [T*k,N] results or launch a separate finalizer. For token-by-weight GEMM with contiguous output features, each lane packs up to eight BF16/FP16 or four FP32 contributions into a 16-byte vector atomic add. The destination address selects narrower accesses when the output pointer or row stride is unaligned. Single-CTA SM100-family block-scaled token-by-weight kernels with 128-row MMA slices instead stage weighted feature tiles in shared memory and issue cp.reduce.async.bulk once per row. Unaligned destinations and partial feature tiles retain vector/scalar atomics. Shared-memory staging is included in the GEMM pipeline budget. Normal 2-CTA configurations retain the existing atomic path pending memory-safety qualification. Outside the SM107 swapped bulk path described above, weight-by-token (SWAP_AB) and M-major outputs retain scalar atomics because adjacent registers do not address adjacent features of the same token. All paths preserve per-element atomicity and output-dtype accumulation.

Bulk reduction trades shared-memory staging and synchronization for fewer global atomics; it does not guarantee a speedup for every workload. A review spot check on SM100 with NVFP4, two experts, top-k 2 and a single-CTA 128x256 tile measured BF16 COMBINE at 18.96 us versus 18.33 us for forced atomics at S/N/K = 258/256/256, and 59.57 us versus 66.37 us at 8192/2048/2048. These warmed CUDA Graph measurements include output zeroing and compare fixed configurations, rather than end-to-end serving performance. Output zeroing uses cudaMemsetAsync for contiguous storage and cudaMemset2DAsync for padded row/column-major storage, preserving padding. The execute-time stream orders output memset, the existing scheduler-counter initialization kernel, and grouped GEMM. Routing metadata and scores may change on graph replay, and runtime token/output extents remain independent of routed capacity.

MoE Grouped Matmul Forward

C++ API

std::shared_ptr<Tensor_attributes>
moe_grouped_matmul(std::shared_ptr<Tensor_attributes> token,
std::shared_ptr<Tensor_attributes> weight,
std::shared_ptr<Tensor_attributes> first_token_offset,
std::shared_ptr<Tensor_attributes> token_index, // optional, pass nullptr for None mode
std::shared_ptr<Tensor_attributes> token_ks, // optional, pass nullptr unless Scatter mode
Moe_grouped_matmul_attributes options);

Moe_grouped_matmul_attributes is a lightweight structure with setters:

Moe_grouped_matmul& set_name(std::string const&);
// Required: selects the routing mode
Moe_grouped_matmul& set_mode(MoeGroupedMatmulMode_t mode);
// MoeGroupedMatmulMode_t::NONE — tokens already routed
// MoeGroupedMatmulMode_t::GATHER — gather before matmul
// MoeGroupedMatmulMode_t::SCATTER — scatter after matmul
// Required for SCATTER mode
Moe_grouped_matmul& set_top_k(int32_t top_k_value);
Moe_grouped_matmul& set_compute_data_type(DataType_t value);

Python API

Low-level graph API (cudnn.pygraph)

output = graph.moe_grouped_matmul(
token, # Token tensor
weight, # Weight tensor
first_token_offset, # Expert routing offsets
token_index=None, # Required for Gather/Scatter modes
token_ks=None, # Required for Scatter mode
mode=cudnn.moe_grouped_matmul_mode.NONE, # NONE, GATHER, or SCATTER
top_k=1, # Top-k value; required for Scatter mode
compute_data_type=cudnn.data_type.NOT_SET,
name=None,
)

Args:

  • token (cudnn_tensor): Token data.
    • None/Scatter modes: shape (1, S*topK, K)
    • Gather mode: shape (1, S, K)
  • weight (cudnn_tensor): Expert weight data with shape (E, K, N).
  • first_token_offset (cudnn_tensor): INT32/INT64 tensor of shape (B*E+1, 1, 1) including the final endpoint. See deprecated implicit offsets for compatibility.
  • token_index (Optional[cudnn_tensor]): INT32 tensor of shape (1, S*topK, 1). Maps each routed slot to a source token index. Required for Gather and Scatter modes.
  • token_ks (Optional[cudnn_tensor]): INT32 tensor of shape (1, S*topK, 1). The top-k slot index in [0, top_k) for each routed token. Required for Scatter mode.
  • mode (cudnn.moe_grouped_matmul_mode): Routing mode — NONE, GATHER, or SCATTER.
  • top_k (int): Top-k routing value. Must be provided for Scatter mode.
  • compute_data_type (Optional[cudnn.data_type]): Data type for internal computation. Defaults to FLOAT.
  • name (Optional[str]): Name for the operation.

Returns:

  • output (cudnn_tensor): Output tensor of shape (1, M_out, N), where M_out = token_index.shape[1] for Gather mode, otherwise token.shape[1].

High-level experimental API

from cudnn.experimental.ops import moe_grouped_matmul
output = moe_grouped_matmul(
token, # (1, M, K) torch.Tensor, fp16 or bf16
weight, # (E, K, N) torch.Tensor, column-major inner dims
first_token_offset, # (B*E+1, 1, 1) torch.Tensor, INT32
token_index=None, # (1, S*topK, 1) INT32; required for gather/scatter
token_ks=None, # (1, S*topK, 1) INT32; required for scatter
mode="none", # "none", "gather", or "scatter"
top_k=1, # required for scatter mode
)

The high-level API handles cuDNN handle management and graph caching automatically. cuDNN graphs are built once per unique (shape, dtype, mode, top_k) configuration and reused across subsequent calls.

Configurable Options

  • Mode (mode): Controls how tokens are routed to and from expert weight matrices.

    • NONE: Tokens are already ordered by expert (pre-routed). Direct grouped matmul with no reordering.
    • GATHER: Tokens are in original (un-routed) order. TokenIndex specifies which source token each expert slot reads from.
    • SCATTER: Tokens are pre-routed, but the output is scattered back to the original token order. Requires both TokenIndex and TokenKs.
  • TopK (top_k): The number of experts each token is routed to. Required in Scatter mode for the scatter-back computation.

  • Compute data type (compute_data_type): Sets the precision for internal accumulation. Defaults to FLOAT for fp16/bf16 I/O.

Example (Python)

import torch
import cudnn
num_experts = 8
token_num = 1024 # S
hidden_size = 512 # K
weight_size = 256 # N
graph = cudnn.pygraph(
intermediate_data_type=cudnn.data_type.FLOAT,
compute_data_type=cudnn.data_type.FLOAT,
)
# Token: [1, S, K], bf16, row-major
token_t = graph.tensor(
name="token",
dim=[1, token_num, hidden_size],
stride=[token_num * hidden_size, hidden_size, 1],
data_type=cudnn.data_type.BFLOAT16,
)
# Weight: [E, K, N], bf16, column-major inner dims (stride[1] == 1)
weight_t = graph.tensor(
name="weight",
dim=[num_experts, hidden_size, weight_size],
stride=[hidden_size * weight_size, 1, hidden_size],
data_type=cudnn.data_type.BFLOAT16,
)
# FirstTokenOffset: [E+1, 1, 1], INT32, including the final endpoint
fto_t = graph.tensor(
name="first_token_offset",
dim=[num_experts + 1, 1, 1],
stride=[1, 1, 1],
data_type=cudnn.data_type.INT32,
)
output_t = graph.moe_grouped_matmul(
token_t, weight_t, fto_t,
mode=cudnn.moe_grouped_matmul_mode.NONE,
compute_data_type=cudnn.data_type.FLOAT,
name="moe_fwd",
)
output_t.set_output(True).set_data_type(cudnn.data_type.BFLOAT16)
graph.validate()
graph.build_operation_graph()
graph.create_execution_plans([cudnn.heur_mode.A])
graph.check_support()
graph.build_plans()

MoE Grouped Matmul Backward

The backward operation computes the weight gradient dWeightd\text{Weight} given the upstream gradient dOutputd\text{Output} and the forward token activations:

dWeight[E, K, N]=TokenT[1, S, K] × dOutput[1, S, N]d\text{Weight}[E,\ K,\ N] = \text{Token}^T[1,\ S,\ K]\ \times\ d\text{Output}[1,\ S,\ N]

per expert, where the per-expert token slices are determined by FirstTokenOffset. Declare the dweight dimensions explicitly when using G+1 boundaries. Backward has no weight input from which to recover E; omitted output dimensions retain the deprecated implicit format’s inference from the offset length.

C++ API

std::shared_ptr<Tensor_attributes>
moe_grouped_matmul_bwd(std::shared_ptr<Tensor_attributes> doutput,
std::shared_ptr<Tensor_attributes> token,
std::shared_ptr<Tensor_attributes> first_token_offset,
Moe_grouped_matmul_bwd_attributes options);

Moe_grouped_matmul_bwd_attributes is a lightweight structure with setters:

Moe_grouped_matmul_bwd& set_name(std::string const&);
Moe_grouped_matmul_bwd& set_compute_data_type(DataType_t value);

Python API

Low-level graph API (cudnn.pygraph)

dweight = graph.moe_grouped_matmul_bwd(
doutput, # Upstream gradient tensor
token, # Forward token activations
first_token_offset, # Expert routing offsets
compute_data_type=cudnn.data_type.NOT_SET,
name=None,
)

Args:

  • doutput (cudnn_tensor): Upstream gradient with shape (1, S, N), same layout as the forward output.
  • token (cudnn_tensor): Forward token activations with shape (1, S, K).
  • first_token_offset (cudnn_tensor): INT32/INT64 tensor of shape (B*E+1, 1, 1), same as used in the forward pass. See deprecated implicit offsets for compatibility.
  • compute_data_type (Optional[cudnn.data_type]): Data type for internal accumulation.
  • name (Optional[str]): Name for the operation.

Returns:

  • dweight (cudnn_tensor): Weight gradient with shape (E, K, N).

Example (Python)

import torch
import cudnn
num_experts = 8
token_num = 1024
hidden_size = 512
weight_size = 256
graph = cudnn.pygraph(
intermediate_data_type=cudnn.data_type.FLOAT,
compute_data_type=cudnn.data_type.FLOAT,
)
# dOutput: [1, S, N], bf16, row-major
doutput_t = graph.tensor(
name="doutput",
dim=[1, token_num, weight_size],
stride=[token_num * weight_size, weight_size, 1],
data_type=cudnn.data_type.BFLOAT16,
)
# Token: [1, S, K], bf16, row-major
token_t = graph.tensor(
name="token",
dim=[1, token_num, hidden_size],
stride=[token_num * hidden_size, hidden_size, 1],
data_type=cudnn.data_type.BFLOAT16,
)
# FirstTokenOffset: [E+1, 1, 1], INT32, including the final endpoint
fto_t = graph.tensor(
name="first_token_offset",
dim=[num_experts + 1, 1, 1],
stride=[1, 1, 1],
data_type=cudnn.data_type.INT32,
)
dweight_t = graph.moe_grouped_matmul_bwd(
doutput_t, token_t, fto_t,
compute_data_type=cudnn.data_type.FLOAT,
name="moe_bwd",
)
# dweight: [E, K, N]
dweight_t.set_dim([num_experts, hidden_size, weight_size]).set_output(True).set_data_type(cudnn.data_type.BFLOAT16)
graph.validate()
graph.build_operation_graph()
graph.create_execution_plans([cudnn.heur_mode.A])
graph.check_support()
graph.build_plans()