Note
This page is a rendered version of 010_moe_grouped_gemm.ipynb available on GitHub.
Block-scaled grouped GEMM using the CUTLASS Operator API#
This notebook covers the operation behind a quantized Mixture-of-Experts layer: a grouped GEMM whose operands are block-scaled.
It sits at the intersection of two earlier tutorials, and assumes both:
grouped GEMM with contiguous offsets, which introduced packing several problems into one tensor plus an
offsetsvector;block-scaled GEMM, which introduced
ScaledOperandand theSwizzle32x4x4scale-factor layout.
Key takeaway: these two flows compose for block-scaled grouped GEMM. Most of this notebook focuses on assembling one scale-factor tensor from per-expert scale factors. After that, the flow remains the same as for other Operator API operations: describe the operation and operands, discover an Operator that implements it, provide its workspace, and run it.
[1]:
import cutlass.operators as ops
if not (status := ops.utils.device.device_or_env_supports("100f")):
print(f"This notebook requires a Blackwell GPU (sm_100f family).\n{status.error}")
import sys
sys.exit(0)
The operation#
The earlier grouped GEMM tutorial introduced contiguous offsets for non-block-scaled inputs. Here we use the two variants supported by the SM100 block-scaled grouped GEMM Operators:
represent scale factors for one expert;
assemble per-expert scale factors into one tensor;
describe the operation, discover an Operator, and run it.
For an MoE forward pass, offsets_along="m" partitions rows. The token counts are decided at run time, the tokens are concatenated, and offsets records each cumulative endpoint:
A (total_m, K) activations, tokens of all experts concatenated
B (G, K, N) one weight matrix per expert
offsets (G,) cumulative end row of each expert's slice
out (total_m, N) out[start:end] = A[start:end] @ B[g]
For a weight-gradient pass, offsets_along="k" partitions the reduction dimension:
A (M, total_k) activations, transposed
B (total_k, N) incoming gradients
out (G, M, N) out[g] = A[:, start:end] @ B[start:end, :]
In both variants, the data tensors are tightly concatenated and offsets describes their group boundaries. Atom-sized padding appears only in the scale-factor tensors.
[2]:
import torch
def ceil_div(value, divisor):
return -(-value // divisor)
def round_up(value, multiple):
return ceil_div(value, multiple) * multiple
def cumulative_offsets(group_sizes):
running = 0
values = []
for size in group_sizes:
running += size
values.append(running)
return values
# MXFP8: E4M3FN data, one E8M0 scale factor per 32 elements along K, tcgen05 layout.
data_dtype = torch.float8_e4m3fn
scale_dtype = torch.float8_e8m0fnu
scale_mode = ops.ScaleMode.Blockwise1x32
scale_swizzle = ops.ScaleSwizzleMode.Swizzle32x4x4
scale_vector_size = ops.ScaleMode.numel(scale_mode) # V = 32 elements per scale factor
Assembling scale tensors for grouped GEMM#
Recap: scale tensors for one expert#
The block-scaled GEMM tutorial describes how Blackwell MMAs require scale factors in the layout represented by ScaleSwizzleMode.Swizzle32x4x4.
For one GEMM, scale factors are stored in atoms of 128 rows by 4 columns. A block with fewer rows or columns must be padded to whole atoms. Within one atom, the row index splits as (m1, m0) with m0 < 32 and m1 < 4, the column index contributes c < 4, and the storage order from slowest to fastest is:
atom_row, atom_col, m0, m1, c
The helpers below pad one expert’s plain scale-factor block and arrange it in this atom order.
Assembling per-expert scale tensors#
A grouped scale tensor is the concatenation of its per-expert scale tensors. The correct order is pad each expert, arrange that expert in atom order, then concatenate. Concatenating first and padding the total is incorrect because each expert must independently occupy whole atoms.
For offsets_along="m" and "k", rows and cols describe different logical dimensions:
|
Grouped axis |
|
|
|---|---|---|---|
|
rows |
|
|
|
columns |
|
|
Here V = ScaleMode.numel(mode) is the number of quantized values represented by one scale factor. The assemble helper applies the required pad-arrange-concatenate order.
[3]:
atom_rows, atom_cols = 128, 4 # one tcgen05 scale-factor atom
def pad_to_atoms(block):
# Grow one expert's block to whole atoms. Padding values are never read.
rows, cols = block.shape
padded = torch.zeros(
(round_up(rows, atom_rows), round_up(cols, atom_cols)),
dtype=torch.uint8,
device=block.device,
).view(block.dtype)
padded[:rows, :cols] = block
return padded
def atom_tile_block(block):
# (rows, cols) logical -> flat atom order: (atom_row, atom_col, m0, m1, c).
rows, cols = block.shape
assert rows % atom_rows == 0 and cols % atom_cols == 0
return (
block.reshape(rows // atom_rows, 4, 32, cols // atom_cols, atom_cols)
.permute(0, 3, 2, 1, 4)
.reshape(-1)
)
def assemble(per_expert_blocks):
return torch.cat([atom_tile_block(pad_to_atoms(block)) for block in per_expert_blocks])
def e8m0(exponents):
# E8M0 holds an exponent only, so a scale factor is exactly 2 ** exponent.
return (127 + exponents).to(torch.uint8).view(torch.float8_e8m0fnu)
def dequantize(quantized, plain_scale, scale_along_rows=True):
# Recover the logical operand from plain, unpadded scales for the reference.
expanded = plain_scale.float().repeat_interleave(scale_vector_size, dim=1)
if scale_along_rows:
return quantized.float() * expanded[:, : quantized.shape[1]]
return quantized.float() * expanded[:, : quantized.shape[0]].t()
2Dx3D: the grouped axis is rows#
Every expert shares the same K, so each scale-factor block has the same number of columns and a different number of rows. Assume the router produced unbalanced experts and the quantizer returned one plain (m_g, ceil_div(K, V)) scale tensor per expert. We pad and arrange each block before concatenating them.
[4]:
torch.manual_seed(7)
group_sizes = [96, 160, 32, 224] # unbalanced routing, the normal case
group_count = len(group_sizes)
total_m = sum(group_sizes)
hidden_k, intermediate_n = 128, 128
scale_columns = ceil_div(hidden_k, scale_vector_size)
offsets_list = cumulative_offsets(group_sizes)
offsets = torch.tensor(offsets_list, dtype=torch.int32, device="cuda")
quantized_a = torch.randint(-1, 2, (total_m, hidden_k), device="cuda").to(data_dtype)
quantized_b = torch.randint(
-1, 2, (group_count, hidden_k, intermediate_n), device="cuda"
).to(data_dtype)
# What a quantizer hands you: one plain, unpadded block per expert. Varied values, so that a
# wrong assembly cannot pass the check below -- with all-ones scales any permutation would.
plain_a = [
e8m0(torch.randint(-1, 2, (size, scale_columns), device="cuda")) for size in group_sizes
]
plain_b = [
e8m0(torch.randint(-1, 2, (intermediate_n, scale_columns), device="cuda"))
for _ in range(group_count)
]
scale_a = assemble(plain_a).view(
sum(round_up(size, atom_rows) for size in group_sizes),
round_up(scale_columns, atom_cols),
)
scale_b = assemble(plain_b).view(group_count, -1)
print(f"scale_a {tuple(scale_a.shape)} from blocks {[tuple(block.shape) for block in plain_a]}")
print(f"scale_b {tuple(scale_b.shape)} from blocks {[tuple(block.shape) for block in plain_b]}")
scale_a (768, 4) from blocks [(96, 4), (160, 4), (32, 4), (224, 4)]
scale_b (4, 512) from blocks [(128, 4), (128, 4), (128, 4), (128, 4)]
The two view calls do not rearrange or validate the scale factors; assemble has already put them in the required storage order. This Operator uses the compact rank-2 shapes as extent carriers: their dimensions communicate the padded rows and scale columns used to reconstruct the kernel’s Swizzle32x4x4 layout. They are not logical views of that swizzled layout. Callers remain responsible for arranging the values according to the declared swizzle.
Describe the operation#
Everything from here follows the ordinary Operator API flow. First, describe the grouped GEMM and its scaled operands.
[5]:
out = torch.empty((total_m, intermediate_n), dtype=torch.bfloat16, device="cuda")
args = ops.IndexPtrGroupedGemmArguments(
A=ops.ScaledOperand(quantized_a, scale_a, scale_mode, scale_swizzle),
B=ops.ScaledOperand(quantized_b, scale_b, scale_mode, scale_swizzle),
out=out,
offsets=offsets,
offsets_along="m",
accumulator_type=torch.float32,
)
Discover an Operator#
Ask the registered providers for compatible implementations and select one candidate. Discovery does not compile the Operator.
[6]:
operators = ops.get_operators(args)
assert operators, "No operator supports these arguments"
print(f"{len(operators)} candidates, first is\n {operators[0].metadata.operator_name}")
operator = operators[0]
90 candidates, first is
cutedsl.Sm100PersistentScaledGroupedGemmOperator_2Dx3D_mxfp8_ttt_outbf16_accf32_offsetdiv1_overwrite_output_1cta_cluster1x1x1_tile128x64x128
Allocate the workspace#
This Operator needs device workspace for per-expert TMA descriptors. The requirement depends on the number of groups and includes the required pointer alignment. The caller owns this allocation; the Operator does not allocate device memory implicitly.
[ ]:
requirement = operator.get_workspace_size(args)
print(f"workspace: {requirement.size_bytes} bytes, {requirement.ptr_alignment}B aligned")
if requirement.size_bytes:
workspace = ops.workspace.Workspace(
torch.empty(requirement.size_bytes, dtype=torch.uint8, device="cuda"),
alignment_bytes=requirement.ptr_alignment,
)
operator.initialize_workspace(args, workspace)
else:
workspace = None
workspace: 528 bytes, 128B aligned
Run the Operator#
Pass the same arguments and the caller-owned workspace to the selected Operator.
[8]:
operator.run(args, workspace=workspace)
Verify the result#
Build the reference directly from the plain per-expert scale factors, independently of the assembled scale storage.
[9]:
reference = torch.empty_like(out)
for index, (start, end) in enumerate(zip([0, *offsets_list[:-1]], offsets_list)):
a_real = dequantize(quantized_a[start:end], plain_a[index])
b_real = dequantize(quantized_b[index], plain_b[index], scale_along_rows=False)
reference[start:end] = (a_real @ b_real).to(torch.bfloat16)
torch.testing.assert_close(out, reference, rtol=0, atol=0)
print("2Dx3D matches the plain-scale-factor reference")
2Dx3D matches the plain-scale-factor reference
2Dx2D: the grouped axis is scale columns#
Here M and N are shared, while each expert owns a slice of the reduction dimension. The per-expert scale-factor blocks therefore differ in columns instead of rows. The assembly rule is unchanged: pad and arrange each expert independently, then concatenate the resulting blocks. Do not concatenate the plain blocks before padding.
[10]:
torch.manual_seed(11)
# Multiples of 16 but not of 128, so the per-expert column padding is visible.
wgrad_group_sizes = [64, 160, 96]
wgrad_group_count = len(wgrad_group_sizes)
total_k = sum(wgrad_group_sizes)
hidden_m, output_n = 128, 128
wgrad_offsets_list = cumulative_offsets(wgrad_group_sizes)
wgrad_a = torch.randint(-1, 2, (hidden_m, total_k), device="cuda").to(data_dtype)
wgrad_b = torch.randint(-1, 2, (total_k, output_n), device="cuda").to(data_dtype)
# Plain per-expert blocks: rows are the shared M (or N), columns are this expert's share of K.
wgrad_plain_a = [
e8m0(torch.randint(-1, 2, (hidden_m, ceil_div(size, scale_vector_size)), device="cuda"))
for size in wgrad_group_sizes
]
wgrad_plain_b = [
e8m0(torch.randint(-1, 2, (output_n, ceil_div(size, scale_vector_size)), device="cuda"))
for size in wgrad_group_sizes
]
grouped_columns = sum(
round_up(ceil_div(size, scale_vector_size), atom_cols) for size in wgrad_group_sizes
)
wgrad_scale_a = assemble(wgrad_plain_a).view(round_up(hidden_m, atom_rows), grouped_columns)
wgrad_scale_b = assemble(wgrad_plain_b).view(round_up(output_n, atom_rows), grouped_columns)
print(f"blocks {[tuple(b.shape) for b in wgrad_plain_a]} -> scale_a {tuple(wgrad_scale_a.shape)}")
wgrad_out = torch.empty(
(wgrad_group_count, hidden_m, output_n), dtype=torch.bfloat16, device="cuda"
)
wgrad_args = ops.IndexPtrGroupedGemmArguments(
A=ops.ScaledOperand(wgrad_a, wgrad_scale_a, scale_mode, scale_swizzle),
B=ops.ScaledOperand(wgrad_b, wgrad_scale_b, scale_mode, scale_swizzle),
out=wgrad_out,
offsets=torch.tensor(wgrad_offsets_list, dtype=torch.int32, device="cuda"),
offsets_along="k",
accumulator_type=torch.float32,
# Required here, unlike 2Dx3D -- see the guarantee section below.
offsets_value_divisibility=16,
)
wgrad_operators = ops.get_operators(wgrad_args)
assert wgrad_operators, "No operator supports these arguments"
wgrad_operator = wgrad_operators[0]
wgrad_requirement = wgrad_operator.get_workspace_size(wgrad_args)
wgrad_workspace = ops.workspace.Workspace(
torch.empty(wgrad_requirement.size_bytes, dtype=torch.uint8, device="cuda"),
alignment_bytes=wgrad_requirement.ptr_alignment,
)
wgrad_operator.initialize_workspace(wgrad_args, wgrad_workspace)
wgrad_operator.run(wgrad_args, workspace=wgrad_workspace)
wgrad_reference = torch.empty_like(wgrad_out)
for index, (start, end) in enumerate(
zip([0, *wgrad_offsets_list[:-1]], wgrad_offsets_list)
):
a_real = dequantize(wgrad_a[:, start:end], wgrad_plain_a[index])
b_real = dequantize(wgrad_b[start:end], wgrad_plain_b[index], scale_along_rows=False)
wgrad_reference[index] = (a_real @ b_real).to(torch.bfloat16)
torch.testing.assert_close(wgrad_out, wgrad_reference, rtol=0, atol=0)
print(f"2Dx2D matches the plain-scale-factor reference ({len(wgrad_operators)} candidates)")
blocks [(128, 2), (128, 5), (128, 3)] -> scale_a (128, 16)
2Dx2D matches the plain-scale-factor reference (90 candidates)
Sizing, as a consequence#
Once the assembly is clear the size follows from it and needs no separate rule. A block of (rows, cols) plain scale factors occupies round_up(rows, 128) * round_up(cols, 4) elements, and the buffer is the blocks back to back:
grouped extent = sum over experts of round_up(extent_g, P)
with P = 128 rows for a grouped m axis and 4 scale columns for a grouped k axis. Note that this is not round_up(sum(extent_g), P): the rounding happens per block and then accumulates. That is the whole content of the “padding” this notebook keeps mentioning – there is no padding between tokens, only whole-atom padding inside each block.
[11]:
def grouped_extent_exact(per_group_extents, padding):
return sum(round_up(extent, padding) for extent in per_group_extents)
def grouped_extent_upper_bound(group_count, total_extent, padding):
return round_up(total_extent, padding) + (group_count - 1) * padding
print("2Dx3D, grouped rows:")
print(f" assembled {scale_a.shape[0]}")
print(f" exact formula {grouped_extent_exact(group_sizes, atom_rows)}")
print(
f" upper bound {grouped_extent_upper_bound(group_count, total_m, atom_rows)}"
)
wgrad_columns = [ceil_div(size, scale_vector_size) for size in wgrad_group_sizes]
print("2Dx2D, grouped scale columns:")
print(f" assembled {wgrad_scale_a.shape[1]}")
print(f" exact formula {grouped_extent_exact(wgrad_columns, atom_cols)}")
print(
f" upper bound "
f"{grouped_extent_upper_bound(wgrad_group_count, sum(wgrad_columns), atom_cols)}"
)
2Dx3D, grouped rows:
assembled 768
exact formula 768
upper bound 896
2Dx2D, grouped scale columns:
assembled 16
exact formula 16
upper bound 20
When assembling on the host, use the exact sum because every per-group extent is available. When a device-side quantizer owns those extents, allocate the upper bound round_up(total, P) + (G - 1) * P from the known total and group count.
Over-allocation is accepted: the Operator checks the grouped extent’s granularity and lower bound, and any unused capacity remains at the end of the buffer.
Setting offsets_value_divisibility#
IndexPtrGroupedGemmArguments accepts the optional promise offsets_value_divisibility=D, meaning offsets[g] % D == 0 for every group on every invocation. Operator discovery uses this guarantee to match implementations whose addressing requires a particular divisibility.
Examples include:
TMA alignment. With
offsets_along="k", TMA-based Operators may require aligned group boundaries. A promise that is too weak can leave no matching Operators.Scale-factor blocks. When
Dcovers the per-group block paddingP, an Operator can reuse the data boundaries as scale-factor boundaries.
Always pass the strongest value guaranteed by the producer of offsets. For example, a router that pads every expert to a multiple of 128 guarantees offsets_value_divisibility=128. The default is 1, meaning no stronger invariant is known. This caller promise is not verified on the host; violating it can produce incorrect behavior.
The next cell shows how the promise selects different Operators for the same padded problem.
[12]:
def mxfp8_args(group_sizes, divisibility, scale_rows):
total = sum(group_sizes)
count = len(group_sizes)
padded_columns = round_up(scale_columns, atom_cols)
return ops.IndexPtrGroupedGemmArguments(
A=ops.ScaledOperand(
torch.randint(-1, 2, (total, hidden_k), device="cuda").to(data_dtype),
torch.ones((scale_rows, padded_columns), dtype=scale_dtype, device="cuda"),
scale_mode,
scale_swizzle,
),
B=ops.ScaledOperand(
torch.randint(-1, 2, (count, hidden_k, intermediate_n), device="cuda").to(
data_dtype
),
torch.ones(
(count, round_up(intermediate_n, atom_rows) * padded_columns),
dtype=scale_dtype,
device="cuda",
),
scale_mode,
scale_swizzle,
),
out=torch.empty((total, intermediate_n), dtype=torch.bfloat16, device="cuda"),
accumulator_type=torch.float32,
offsets=torch.tensor(
cumulative_offsets(group_sizes), dtype=torch.int32, device="cuda"
),
offsets_along="m",
offsets_value_divisibility=divisibility,
)
def survey(label, probe_args):
found = ops.get_operators(probe_args)
requirements = sorted(
{operator.metadata.operands.offsets_value_divisibility for operator in found}
)
print(f"{label:34s} {len(found):3d} candidates, requirements {requirements}")
padded_sizes = [atom_rows] * group_count
padded_rows = grouped_extent_exact(padded_sizes, atom_rows)
survey("128-padded offsets, promise D=1", mxfp8_args(padded_sizes, 1, padded_rows))
survey("128-padded offsets, promise D=128", mxfp8_args(padded_sizes, 128, padded_rows))
128-padded offsets, promise D=1 90 candidates, requirements [1]
128-padded offsets, promise D=128 90 candidates, requirements [128]
Both promises are truthful for these 128-padded offsets. With D=128, discovery can select the implementation that reuses the data offsets for scale-factor blocks. With D=1, it selects the general implementation instead.
The caller does not choose this implementation detail directly. Supplying the strongest truthful promise lets discovery consider every compatible Operator.
NVFP4: per-group global scales#
NVFP4 uses a second, coarser scale on top of the block scales: one FP32 value per operand per group, folded into the result. It is not part of ScaledOperand, which describes scale * quantized for one tensor. Instead, PerGroupScaleEpilogueArguments describes a fixed epilogue:
out[start:end] = (A[start:end] @ B[g]) * global_scale_a[g] * global_scale_b[g]
The fixed epilogue carries two (G,) FP32 tensors. It is required for NVFP4 and rejected for the MX formats, so discovery enforces the pairing.
NVFP4 also uses E4M3 block scales over 16 values, so V = 16. The assembly and sizing rules remain unchanged, and the example reuses assemble.
[13]:
nvfp4_scale_mode = ops.ScaleMode.Blockwise1x16
nvfp4_vector_size = ops.ScaleMode.numel(nvfp4_scale_mode) # 16
nvfp4_columns = ceil_div(hidden_k, nvfp4_vector_size)
def fp4_ones(logical_shape, packed_dim):
# FP4 stores two logical values per byte along the K-major packed dimension.
packed_dim %= len(logical_shape)
shape = list(logical_shape)
needs_permute = packed_dim != len(logical_shape) - 1
if needs_permute:
shape[packed_dim], shape[-1] = shape[-1], shape[packed_dim]
if shape[-1] % 2:
raise ValueError("The packed FP4 dimension must contain an even number of values")
shape[-1] //= 2
packed = torch.full(shape, 0x22, dtype=torch.uint8, device="cuda").view(
torch.float4_e2m1fn_x2
)
if not needs_permute:
return packed
permutation = list(range(len(logical_shape)))
permutation[packed_dim], permutation[-1] = permutation[-1], permutation[packed_dim]
return packed.permute(permutation)
def nvfp4_unit_scales(rows):
return torch.ones((rows, nvfp4_columns), dtype=torch.float8_e4m3fn, device="cuda")
nvfp4_scale_a = assemble([nvfp4_unit_scales(size) for size in group_sizes]).view(
grouped_extent_exact(group_sizes, atom_rows), round_up(nvfp4_columns, atom_cols)
)
nvfp4_scale_b = assemble(
[nvfp4_unit_scales(intermediate_n) for _ in range(group_count)]
).view(group_count, -1)
global_scale_a = torch.tensor([1.0, 2.0, 3.0, 4.0], device="cuda")
global_scale_b = torch.tensor([2.0, 3.0, 4.0, 5.0], device="cuda")
nvfp4_out = torch.empty((total_m, intermediate_n), dtype=torch.bfloat16, device="cuda")
nvfp4_args = ops.IndexPtrGroupedGemmArguments(
A=ops.ScaledOperand(
fp4_ones((total_m, hidden_k), packed_dim=-1),
nvfp4_scale_a,
nvfp4_scale_mode,
scale_swizzle,
),
B=ops.ScaledOperand(
fp4_ones((group_count, hidden_k, intermediate_n), packed_dim=1),
nvfp4_scale_b,
nvfp4_scale_mode,
scale_swizzle,
),
out=nvfp4_out,
accumulator_type=torch.float32,
offsets=offsets,
offsets_along="m",
epilogue=ops.PerGroupScaleEpilogueArguments(
global_scale_a=global_scale_a,
global_scale_b=global_scale_b,
),
)
nvfp4_operators = ops.get_operators(nvfp4_args)
assert nvfp4_operators, "No operator supports these NVFP4 arguments"
nvfp4_operator = nvfp4_operators[0]
nvfp4_requirement = nvfp4_operator.get_workspace_size(nvfp4_args)
nvfp4_workspace = ops.workspace.Workspace(
torch.empty(nvfp4_requirement.size_bytes, dtype=torch.uint8, device="cuda"),
alignment_bytes=nvfp4_requirement.ptr_alignment,
)
nvfp4_operator.initialize_workspace(nvfp4_args, nvfp4_workspace)
nvfp4_operator.run(nvfp4_args, workspace=nvfp4_workspace)
# Every data element and block scale is 1.0, so each output element is K before the global
# scales are applied.
nvfp4_reference = torch.empty_like(nvfp4_out)
for index, (start, end) in enumerate(zip([0, *offsets_list[:-1]], offsets_list)):
nvfp4_reference[start:end] = (
hidden_k * global_scale_a[index] * global_scale_b[index]
).to(torch.bfloat16)
torch.testing.assert_close(nvfp4_out, nvfp4_reference, rtol=0, atol=0)
print("NVFP4 with per-group global scales matches the reference")
NVFP4 with per-group global scales matches the reference
Dropping the epilogue makes the same arguments undiscoverable, which is how the requirement is enforced:
[14]:
nvfp4_without_epilogue = ops.IndexPtrGroupedGemmArguments(
A=nvfp4_args.A,
B=nvfp4_args.B,
out=nvfp4_args.out,
accumulator_type=torch.float32,
offsets=nvfp4_args.offsets,
offsets_along="m",
)
print(
f"NVFP4 without the fixed epilogue: "
f"{len(ops.get_operators(nvfp4_without_epilogue))} candidates"
)
NVFP4 without the fixed epilogue: 0 candidates
Summary#
The contract for a block-scaled grouped GEMM is:
Pad and arrange each expert’s scale-factor block into
128 x 4atoms.Build the grouped scale tensor by padding each expert first and then concatenating the arranged blocks. Concatenate-then-pad is not correct.
Size the grouped extent as
sum(round_up(extent_g, P)), whereP = 128rows for a groupedmaxis andP = 4scale columns for a groupedkaxis. If device-only extents prevent exact sizing, allocateround_up(total, P) + (G - 1) * Pand leave the tail unused.Pass the strongest truthful
offsets_value_divisibility. Some Operators require a minimum promise for correctness, while others use a stronger promise to avoid extra work. The promise is not verified on the host.For NVFP4, provide per-group global scales through
PerGroupScaleEpilogueArguments.Query
get_workspace_sizeand provide the requested workspace when running the Operator.
The examples validate assembled scale factors against references built directly from the plain per-expert factors.