Skip to navigation

Grouped GEMM + Quant — Unified (SM100)

View as MarkdownOpen in Claude

This is an experimental API and subject to change.

JAX support

Canonical block-scaled (dense weights): the jax.jit-compatible entry point cudnn.grouped_gemm_quant_jax_forward takes A (m, k), B (experts, n, k) or column-wise (experts, k, n) with b_major="n" (the dFC1 layout, MXFP8 only), already MMA-packed sfa_tensor/sfb_tensor bytes (physical or flat), int32 padded_offsets, fp32 alpha, and an optional prob (m,) fp32/bf16. A/B are MXFP8 with E8M0 scales (sf_vec_size=32; uint8 accepted), or packed FP4. Packed FP4 operands travel as uint8 with two values per byte (JAX has no packed FP4 dtype), so K is twice the array extent; their scale buffers are E4M3 with sf_vec_size=16 for NVFP4, or E8M0. With a BF16/FP16 d_dtype (default BF16) it produces D (m, n), plus the per-expert amax (experts, 1) with generate_amax. With an FP8 d_dtype, which needs MXFP8 inputs, it requires norm_const_tensor (1,) and also produces D_col and the physical 6-D SFD buffers; discrete_col_sfd=True packs column scales by expert. It returns a TupleDict with the eager wrapper’s keys. Outputs are not initialized, so rows at or past padded_offsets[-1] are unspecified, as in the Torch path. It runs the Rubin quant kernel on SM107 and the SM100 kernel otherwise. grouped_gemm_quant_wrapper_sm100 dispatches JAX inputs with a dense b_tensor to it; pass sf_vec_size. Bias, row scale, a caller-provided d_tensor, dynamic scheduling, and sf_fp8_dtype_override raise ValueError. Discrete (b_ptrs) weights and the GroupedGemmQuantSm100 class stay torch-only.

Overview

Unified Grouped GEMM + Quant fusion: A block-scaled grouped GEMM with output quantization and per-row gating on NVIDIA Blackwell GPUs (SM100+), designed for MoE (Mixture of Experts) workloads. Implemented with CUTLASS/CUTE. Used for FC2 (forward down-projection) and dFC1 (backward FC1 GEMMs).

This kernel uses the unified BlockScaledMoEGroupedGemmQuantKernel which supports the MoEWeightMode abstraction:

  • Dense mode (MoEWeightMode.DENSE): all expert weights packed into a single contiguous (N, K, L) tensor
  • Discrete mode (MoEWeightMode.DISCRETE): each expert weight and scale-factor tensor provided through per-expert device pointer arrays

Groups are contiguous in the M dimension and described by padded_offsets (cumulative aligned end offsets).

This kernel performs:

  1. Block-scaled grouped GEMM: Low-precision GEMM (FP4, FP8) with per-block scale factors across multiple expert groups
  2. Per-row gating: Multiplies output by per-row gating probability
  3. Optional quantized output: Produces row and column scale factors for downstream quantization

Shapes

  • Inputs
    • A: contiguous activation tensor across all groups, shape (valid_m, K, 1)
    • B (dense): weight tensor across all groups, shape (N, K, L)
    • B (discrete): per-expert weight tensors, each with shape (N, K), passed via b_ptrs
    • SFA: scale factor tensor for A, shape (32, 4, ceil(valid_m/128), 4, ceil(ceil(K/sf_vec_size)/4), 1)
    • SFB (dense): scale factor tensor for B, shape (32, 4, ceil(N/128), 4, ceil(ceil(K/sf_vec_size)/4), L)
    • SFB (discrete): per-expert scale factor tensors, passed via sfb_ptrs
    • padded_offsets: cumulative sum of aligned group M sizes, shape (L,). valid_m = padded_offsets[-1]
    • alpha: per-group scaling factors, shape (L,)
    • prob: per-row gating probabilities, shape (valid_m, 1, 1). Optional; omission disables gating.
    • norm_const: normalization constant for FP8 quantization, shape (1,)
  • Outputs
    • D: row-quantized output, shape (valid_m, N, 1)
    • D_col: column-quantized output, shape (valid_m, N, 1)
    • SFD_row: row scale factors (when SFD outputs are enabled), shape (32, 4, ceil(valid_m/128), 4, ceil(ceil(N/sf_vec_size)/4), 1)
    • SFD_col: column scale factors (when SFD outputs are enabled), shape (32, 4, ceil(N/128), 4, ceil(ceil(valid_m/sf_vec_size)/4), 1)
    • amax: per-group amax (when d_dtype is bf16/float16), shape (L, 1)

Canonical layouts and optional amax

The unified wrapper and API class also accept contiguous A (M, K), dense B (L, N, K), and probability (M,). Operands may independently use canonical or legacy layouts. Scale buffers may be contiguous tensors of any rank, provided their bytes are already packed in the required MMA-tiled physical order; reshaping ordinary logical scales does not perform this packing. A dense canonical B may instead be an N-major (L, K, N) contiguous buffer, such as a column-wise weight, when b_major="n" (FP8 only).

With 2-D A, the wrapper returns D/D_col (M, N) and contiguous SFD buffers with physical shape (1, ceil(rows/128), ceil(ceil(cols/sf_vec_size)/4), 32, 4, 4). Preallocated D must follow the layout selected by A. Legacy 3-D A retains the existing output layouts. Canonical normalization happens during compilation.

generate_amax=False disables the BF16/FP16 amax output, including its allocation, initialization, reduction, and atomic updates. The default is True for compatibility; when disabled, amax_tensor is None. D values are unchanged. Probability is optional: prob_tensor=None removes the probability load and multiply.

The wrapper advertises supports_canonical_layouts and supports_optional_amax. Its metadata memo retains compiled APIs and output allocation parameters, never input tensors or their data pointers. Runtime data, routing, and streams remain per-call arguments.

Equations

Step 1: Block-scaled grouped GEMM (per group g with rows m in [padded_offsets[g-1], padded_offsets[g])):

ref[m,n]=αg∑kdequantize(A[m,k],SFA)⋅dequantize(B[n,k,g],SFB)\text{ref}[m, n] = \alpha_g \sum_{k} \text{dequantize}(A[m, k], \text{SFA}) \cdot \text{dequantize}(B[n, k, g], \text{SFB})

Step 2: Per-row gating:

D[m,n]=prob[m]⋅ref[m,n]D[m, n] = \text{prob}[m] \cdot \text{ref}[m, n]

Step 3: Optional output quantization (when SFD outputs are generated):

SFD_row[m,n]=norm_const⋅max⁡k∈block∣D[m,k]∣⋅rcp_max\text{SFD\_row}[m, n] = \text{norm\_const} \cdot \max_{k \in \text{block}} |D[m, k]| \cdot \text{rcp\_max} Dquantized[m,n]=D[m,n]⋅norm_constSFD_row[m,n]D_{\text{quantized}}[m, n] = D[m, n] \cdot \frac{\text{norm\_const}}{\text{SFD\_row}[m, n]}

Diagram

A (valid_m×K×1) B (N×K×L) padded_offsets
SFA SFB |
| | |
| +------------+ |
| | |
v v v
Dequantize → Grouped GEMM (per group ranges) → Select B[:,:,group_idx]
|
| × alpha[group_idx]
v
ref (valid_m×N×1)
|
| × prob
v
D (valid_m×N×1)
|
+----------+-----------+
| |
v v
Row Quantize Col Quantize
| |
v v
D_row, SFD_row D_col, SFD_col

API Usage

High-level Wrapper

from cudnn import grouped_gemm_quant_wrapper_sm100
from cuda.bindings import driver as cuda
stream = cuda.CUstream(torch.cuda.current_stream().cuda_stream)
outputs = grouped_gemm_quant_wrapper_sm100(
a_tensor=a,
b_tensor=b,
sfa_tensor=sfa,
sfb_tensor=sfb,
padded_offsets=padded_offsets,
alpha_tensor=alpha,
norm_const_tensor=norm_const,
prob_tensor=prob,
acc_dtype=torch.float32,
d_dtype=torch.bfloat16,
cd_major="n",
mma_tiler_mn=(256, 256),
cluster_shape_mn=(2, 1),
sf_vec_size=16,
vector_f32=False,
m_aligned=256,
discrete_col_sfd=False,
current_stream=stream,
)
# dictionary access:
d = outputs["d_tensor"]
d_col = outputs["d_col_tensor"] # None for bf16/fp16/fp32 outputs; tensor only for low-precision outputs
amax = outputs["amax_tensor"]
sfd_row = outputs["sfd_row_tensor"]
sfd_col = outputs["sfd_col_tensor"]
# or tuple unpacking:
d, d_col, amax, sfd_row, sfd_col = outputs # d_col is None for bf16/fp16/fp32 outputs

Class API

from cudnn import GroupedGemmQuantSm100
from cuda.bindings import driver as cuda
stream = cuda.CUstream(torch.cuda.current_stream().cuda_stream)
api = GroupedGemmQuantSm100(
sample_a=a,
sample_b=b,
sample_d=d,
sample_sfa=sfa,
sample_sfb=sfb,
sample_padded_offsets=padded_offsets,
sample_alpha=alpha,
sample_d_col=d_col,
sample_sfd_row=sfd_row,
sample_sfd_col=sfd_col,
sample_amax=amax,
sample_norm_const=norm_const,
sample_prob=prob,
acc_dtype=torch.float32,
mma_tiler_mn=(256, 256),
cluster_shape_mn=(2, 1),
sf_vec_size=16,
vector_f32=False,
m_aligned=256,
discrete_col_sfd=False,
)
assert api.check_support()
api.compile()
api.execute(
a_tensor=a, b_tensor=b, d_tensor=d,
sfa_tensor=sfa, sfb_tensor=sfb,
padded_offsets=padded_offsets, alpha_tensor=alpha,
d_col_tensor=d_col, sfd_row_tensor=sfd_row, sfd_col_tensor=sfd_col,
amax_tensor=amax, norm_const_tensor=norm_const, prob_tensor=prob,
current_stream=stream,
)

Parameters

Input/Output Tensors

  • Input tensor A: a_tensor / sample_a

    • Shape: (valid_m, K, 1), Stride: K-major
    • Dtype: {float4_e2m1fn_x2, uint8, float8_e4m3fn, float8_e5m2}
  • Input tensor B: b_tensor / sample_b

    • Shape: (N, K, L), Stride: K-major (FP8 also supports N-major)
    • Dtype: Must match A
  • Output tensor D: d_tensor / sample_d

    • Shape: (valid_m, N, 1), Stride: N-major
    • Dtype: {float16, bfloat16, float32} for FP4; {float16, bfloat16, float8_e4m3fn, float8_e5m2, float4_e2m1fn_x2} for FP8
  • Output tensor D_col: d_col_tensor / sample_d_col

    • Shape/Dtype: Must match D
    • Wrapper behavior: returned only when d_dtype ∈ {float8_e4m3fn, float8_e5m2, float4_e2m1fn_x2}; for bfloat16, float16, and float32, outputs["d_col_tensor"] is None
  • Input tensor prob: prob_tensor / sample_prob

    • Shape: (valid_m, 1, 1), dtype: float32
    • Optional: pass None when no gating is needed
  • Scale factor tensors: SFA, SFB, SFD_row, SFD_col — block-scaled 6-D layout

  • Group offsets: padded_offsets shape (L,), dtype int32

  • Scaling tensors: alpha shape (L,), amax shape (L, 1), norm_const shape (1,)

Common Parameters

  • acc_dtype: Must be torch.float32
  • mma_tiler_mn: Default (256, 256); supported tiles are TILE_M ∈ {128, 256} and TILE_N = 256
  • cluster_shape_mn: Default (2, 1) when TILE_M=256, (1, 1) otherwise
  • sf_vec_size: {16, 32}. Default: 16
  • vector_f32: Default: False
  • m_aligned: Must be 256
  • discrete_col_sfd: Default: False

Wrapper Return Values

Returns TupleDict: d_tensor, d_col_tensor (optional; None for bfloat16/float16/float32 outputs), amax_tensor, sfd_row_tensor, sfd_col_tensor


Support Surface and Constraints

Data Types

Formatab_dtypesf_dtypesf_vec_sized_dtype
MXFP8float8_e4m3fn or float8_e5m2float8_e8m0fnu32{float16, bfloat16, float8_e4m3fn, float8_e5m2, float4_e2m1fn_x2}
NVF4float4_e2m1fn_x2 or uint8{float8_e4m3fn, float8_e8m0fnu}{16, 32}{float16, bfloat16, float32}

Key Constraints

  • A and B must have same dtype; D and D_col must have same dtype
  • All scale factor tensors must have same dtype
  • Expert count <= 1024; M aligned to 256
  • SM100+ compute capability required
  • prob_tensor=None omits per-row gating
  • use_single_group_runtime_offsets=True requires exactly one expert. The kernel derives padded_offsets[0] from runtime A.shape[0] and does not load its value from device memory; the argument must still be an int32 tensor with shape (1,).

Usage Examples

For usage examples, see test/python/gemm/cutedsl/test_grouped_gemm_quant.py + test/python/gemm/cutedsl/test_grouped_gemm_quant_utils.py (dense and discrete unified API coverage)

Prepared execution

Dense canonical MXFP8 E4M3 inputs with E8M0 packed scales support a prepared BF16/FP16 output call with generate_amax=False:

from cudnn.gemm.cutedsl.grouped.prepared import prepare_grouped_gemm
kwargs = dict(
a_tensor=a, b_tensor=b, sfa_tensor=sfa, sfb_tensor=sfb,
padded_offsets=padded_offsets, alpha_tensor=alpha,
prob_tensor=prob, d_dtype=torch.bfloat16, sf_vec_size=32,
generate_amax=False, use_dynamic_sched=True,
)
plan = prepare_grouped_gemm("quant", **kwargs)
outputs = plan.run(d_tensor=output_buffer, **kwargs)
assert outputs["d_tensor"] is output_buffer

Preparation does not execute the GEMM or retain sample inputs. Pass current operands on each call. D is fresh unless supplied through d_tensor or outputs={"d_tensor": output_buffer}; other quant outputs are None in this prepared configuration. A D supplied during preparation must be passed again at execution. reuse_row_outputs applies only to GLU.

The shared prepared execution contract requires fixed metadata/configuration apart from M, a separate plan per stream, and the preparation stream current during every call. check=False is only for callers that enforce those invariants themselves.

Caller-owned scheduler counter

Dense block-scaled dynamic scheduling accepts scheduler_counter_tensor, a nonempty contiguous one-dimensional CUDA int32 tensor on A’s device. Zero its first element on the execution stream before every launch; keep it alive until execution completes. Use distinct counters for overlapping invocations and GEMMs. The class constructor uses sample_scheduler_counter and execution must preserve its optional presence.

Omitting the counter retains internal initialization. Supplying it bypasses the counter-initialization launch, so callers may combine the reset with existing preparation work. The same contract applies to prepared calls; see counter ownership. Detect support with grouped_gemm_quant_wrapper_sm100.supports_external_scheduler_counter.