Grouped GEMM + Quant — Unified (SM100)
Grouped GEMM + Quant — Unified (SM100)
This is an experimental API and subject to change.
JAX support
Canonical block-scaled (dense weights): the jax.jit-compatible entry point cudnn.grouped_gemm_quant_jax_forward takes A (m, k), B (experts, n, k) or column-wise (experts, k, n) with b_major="n" (the dFC1 layout, MXFP8 only), already MMA-packed sfa_tensor/sfb_tensor bytes (physical or flat), int32 padded_offsets, fp32 alpha, and an optional prob (m,) fp32/bf16. A/B are MXFP8 with E8M0 scales (sf_vec_size=32; uint8 accepted), or packed FP4. Packed FP4 operands travel as uint8 with two values per byte (JAX has no packed FP4 dtype), so K is twice the array extent; their scale buffers are E4M3 with sf_vec_size=16 for NVFP4, or E8M0. With a BF16/FP16 d_dtype (default BF16) it produces D (m, n), plus the per-expert amax (experts, 1) with generate_amax. With an FP8 d_dtype, which needs MXFP8 inputs, it requires norm_const_tensor (1,) and also produces D_col and the physical 6-D SFD buffers; discrete_col_sfd=True packs column scales by expert. It returns a TupleDict with the eager wrapper’s keys. Outputs are not initialized, so rows at or past padded_offsets[-1] are unspecified, as in the Torch path. It runs the Rubin quant kernel on SM107 and the SM100 kernel otherwise. grouped_gemm_quant_wrapper_sm100 dispatches JAX inputs with a dense b_tensor to it; pass sf_vec_size. Bias, row scale, a caller-provided d_tensor, dynamic scheduling, and sf_fp8_dtype_override raise ValueError. Discrete (b_ptrs) weights and the GroupedGemmQuantSm100 class stay torch-only.
Overview
Unified Grouped GEMM + Quant fusion: A block-scaled grouped GEMM with output quantization and per-row gating on NVIDIA Blackwell GPUs (SM100+), designed for MoE (Mixture of Experts) workloads. Implemented with CUTLASS/CUTE. Used for FC2 (forward down-projection) and dFC1 (backward FC1 GEMMs).
This kernel uses the unified BlockScaledMoEGroupedGemmQuantKernel which supports the MoEWeightMode abstraction:
- Dense mode (
MoEWeightMode.DENSE): all expert weights packed into a single contiguous(N, K, L)tensor - Discrete mode (
MoEWeightMode.DISCRETE): each expert weight and scale-factor tensor provided through per-expert device pointer arrays
Groups are contiguous in the M dimension and described by padded_offsets (cumulative aligned end offsets).
This kernel performs:
- Block-scaled grouped GEMM: Low-precision GEMM (FP4, FP8) with per-block scale factors across multiple expert groups
- Per-row gating: Multiplies output by per-row gating probability
- Optional quantized output: Produces row and column scale factors for downstream quantization
Shapes
- Inputs
A: contiguous activation tensor across all groups, shape(valid_m, K, 1)B(dense): weight tensor across all groups, shape(N, K, L)B(discrete): per-expert weight tensors, each with shape(N, K), passed viab_ptrsSFA: scale factor tensor for A, shape(32, 4, ceil(valid_m/128), 4, ceil(ceil(K/sf_vec_size)/4), 1)SFB(dense): scale factor tensor for B, shape(32, 4, ceil(N/128), 4, ceil(ceil(K/sf_vec_size)/4), L)SFB(discrete): per-expert scale factor tensors, passed viasfb_ptrspadded_offsets: cumulative sum of aligned group M sizes, shape(L,).valid_m = padded_offsets[-1]alpha: per-group scaling factors, shape(L,)prob: per-row gating probabilities, shape(valid_m, 1, 1). Optional; omission disables gating.norm_const: normalization constant for FP8 quantization, shape(1,)
- Outputs
D: row-quantized output, shape(valid_m, N, 1)D_col: column-quantized output, shape(valid_m, N, 1)SFD_row: row scale factors (when SFD outputs are enabled), shape(32, 4, ceil(valid_m/128), 4, ceil(ceil(N/sf_vec_size)/4), 1)SFD_col: column scale factors (when SFD outputs are enabled), shape(32, 4, ceil(N/128), 4, ceil(ceil(valid_m/sf_vec_size)/4), 1)amax: per-group amax (whend_dtypeis bf16/float16), shape(L, 1)
Canonical layouts and optional amax
The unified wrapper and API class also accept contiguous A (M, K), dense B
(L, N, K), and probability (M,). Operands may independently use canonical or
legacy layouts. Scale buffers may be contiguous tensors of any rank, provided
their bytes are already packed in the required MMA-tiled physical order; reshaping
ordinary logical scales does not perform this packing. A dense canonical B may
instead be an N-major (L, K, N) contiguous buffer, such as a column-wise weight,
when b_major="n" (FP8 only).
With 2-D A, the wrapper returns D/D_col (M, N) and contiguous SFD buffers with
physical shape (1, ceil(rows/128), ceil(ceil(cols/sf_vec_size)/4), 32, 4, 4).
Preallocated D must follow the layout selected by A. Legacy 3-D A retains the
existing output layouts. Canonical normalization happens during compilation.
generate_amax=False disables the BF16/FP16 amax output, including its allocation,
initialization, reduction, and atomic updates. The default is True for compatibility;
when disabled, amax_tensor is None. D values are unchanged. Probability is
optional: prob_tensor=None removes the probability load and multiply.
The wrapper advertises supports_canonical_layouts and supports_optional_amax.
Its metadata memo retains compiled APIs and output allocation parameters, never
input tensors or their data pointers. Runtime data, routing, and streams remain
per-call arguments.
Equations
Step 1: Block-scaled grouped GEMM (per group g with rows m in [padded_offsets[g-1], padded_offsets[g])):
Step 2: Per-row gating:
Step 3: Optional output quantization (when SFD outputs are generated):
Diagram
API Usage
High-level Wrapper
Class API
Parameters
Input/Output Tensors
-
Input tensor A:
a_tensor/sample_a- Shape:
(valid_m, K, 1), Stride: K-major - Dtype:
{float4_e2m1fn_x2, uint8, float8_e4m3fn, float8_e5m2}
- Shape:
-
Input tensor B:
b_tensor/sample_b- Shape:
(N, K, L), Stride: K-major (FP8 also supports N-major) - Dtype: Must match A
- Shape:
-
Output tensor D:
d_tensor/sample_d- Shape:
(valid_m, N, 1), Stride: N-major - Dtype:
{float16, bfloat16, float32}for FP4;{float16, bfloat16, float8_e4m3fn, float8_e5m2, float4_e2m1fn_x2}for FP8
- Shape:
-
Output tensor D_col:
d_col_tensor/sample_d_col- Shape/Dtype: Must match D
- Wrapper behavior: returned only when
d_dtype ∈ {float8_e4m3fn, float8_e5m2, float4_e2m1fn_x2}; forbfloat16,float16, andfloat32,outputs["d_col_tensor"]isNone
-
Input tensor prob:
prob_tensor/sample_prob- Shape:
(valid_m, 1, 1), dtype:float32 - Optional: pass
Nonewhen no gating is needed
- Shape:
-
Scale factor tensors: SFA, SFB, SFD_row, SFD_col — block-scaled 6-D layout
-
Group offsets:
padded_offsetsshape(L,), dtypeint32 -
Scaling tensors:
alphashape(L,),amaxshape(L, 1),norm_constshape(1,)
Common Parameters
acc_dtype: Must betorch.float32mma_tiler_mn: Default(256, 256); supported tiles areTILE_M ∈ {128, 256}andTILE_N = 256cluster_shape_mn: Default(2, 1)whenTILE_M=256,(1, 1)otherwisesf_vec_size:{16, 32}. Default:16vector_f32: Default:Falsem_aligned: Must be256discrete_col_sfd: Default:False
Wrapper Return Values
Returns TupleDict: d_tensor, d_col_tensor (optional; None for bfloat16/float16/float32 outputs), amax_tensor, sfd_row_tensor, sfd_col_tensor
Support Surface and Constraints
Data Types
Key Constraints
AandBmust have same dtype;DandD_colmust have same dtype- All scale factor tensors must have same dtype
- Expert count
<= 1024; M aligned to 256 - SM100+ compute capability required
prob_tensor=Noneomits per-row gatinguse_single_group_runtime_offsets=Truerequires exactly one expert. The kernel derivespadded_offsets[0]from runtimeA.shape[0]and does not load its value from device memory; the argument must still be an int32 tensor with shape(1,).
Usage Examples
For usage examples, see test/python/gemm/cutedsl/test_grouped_gemm_quant.py + test/python/gemm/cutedsl/test_grouped_gemm_quant_utils.py (dense and discrete unified API coverage)
Prepared execution
Dense canonical MXFP8 E4M3 inputs with E8M0 packed scales support a prepared
BF16/FP16 output call with generate_amax=False:
Preparation does not execute the GEMM or retain sample inputs. Pass current
operands on each call. D is fresh unless supplied through d_tensor or
outputs={"d_tensor": output_buffer}; other quant outputs are None in this
prepared configuration. A D supplied during preparation must be passed again
at execution. reuse_row_outputs applies only to GLU.
The shared prepared execution contract
requires fixed metadata/configuration apart from M, a separate plan per stream, and the
preparation stream current during every call. check=False is only for callers
that enforce those invariants themselves.
Caller-owned scheduler counter
Dense block-scaled dynamic scheduling accepts scheduler_counter_tensor, a
nonempty contiguous one-dimensional CUDA int32 tensor on A’s device. Zero its
first element on the execution stream before every launch; keep it alive
until execution completes. Use distinct counters for overlapping invocations
and GEMMs. The class constructor uses sample_scheduler_counter and execution
must preserve its optional presence.
Omitting the counter retains internal initialization. Supplying it bypasses
the counter-initialization launch, so callers may combine the reset with
existing preparation work. The same contract applies to prepared calls;
see counter ownership.
Detect support with
grouped_gemm_quant_wrapper_sm100.supports_external_scheduler_counter.