Arguments and Operands#
An operation is expressed as a RuntimeArguments
object that describes both the operation kind (e.g. GEMM) and the operands it
runs on. Each operation has its own RuntimeArguments subclass. Operands are
described by Operand subclasses such as
DenseTensor and
ScaledOperand. Tensor- and numeric-typed fields
within Arguments and Operands accept any object matching the
TensorLike / NumericLike
protocols (e.g. torch.Tensor, cute.Tensor, cutlass.Numeric,
torch.dtype).
RuntimeArguments#
- class cutlass.operators.arguments.base.RuntimeArguments(
- *,
- performance: PerformanceControls | None | ellipsis = Ellipsis,
Bases:
SupportsConversionToInternalTypesDescribes the operands and all other arguments passed to an Operator at runtime.
It contains runtime operands (usually tensors) passed to the operation, as well as any custom epilogue fusions, and runtime performance controls.
This is an abstract base class, whose subclass describes the operation type itself (e.g. GemmArguments, GroupedGemmArguments, etc.). All operators that implement the same operation type accept the same RuntimeArguments subclass.
- performance: PerformanceControls | None | ellipsis = Ellipsis#
Optional runtime performance controls passed to the Operator.
...(the default) expresses no opinion, and every operator matches.Noneasks for an operator that exposes no performance controls at all.
- class cutlass.operators.arguments.gemm.GemmProblemSize(M: int, N: int, K: int, L: int)#
Bases:
NamedTupleProblem size for a GEMM operation.
A GEMM with problem size (M, N, K, L) has operands with the following shapes (batch, rows, columns):
A: (L, M, K)B: (L, K, N)out: (L, M, N)
- M: int#
Number of rows in A and out
- N: int#
Number of columns in B and out
- K: int#
Number of columns in A and rows in B
- L: int#
Number of batches of matrix multiplications
- class cutlass.operators.arguments.gemm.GemmArguments(
- A: TensorLike | Operand,
- B: TensorLike | Operand,
- out: TensorLike | Operand,
- accumulator_type: NumericLike,
- epilogue: EpilogueArguments | None = None,
- *,
- performance: PerformanceControls | None | EllipsisType = Ellipsis,
Bases:
RuntimeArgumentsArguments for a Generalized Matrix Multiplication (GEMM) operation:
out = A @ B.The tensors must be all rank-3 or all rank-2. *
L: Number of batches *M: Number of rows in A and out *K: Number of columns in A and rows in B *N: Number of columns in B and outFor convenience, construction for
A,B, andoutthat are operands to a dense GEMM can be passed in without wrapping them inDenseTensor.GemmArguments(A, B, out, accumulator_type) # is equivalent to: GemmArguments(DenseTensor(A), DenseTensor(B), DenseTensor(out), accumulator_type)
Other operand types must explicitly wrap tensors in a
Operandsubclass. For example, a scaled GEMM can be constructed as:GemmArguments( ScaledOperand(A, ScaleATensor, scale_mode, scale_swizzle), ScaledOperand(B, ScaleBTensor, scale_mode, scale_swizzle), out, # No need to wrap in a `DenseTensor` accumulator_type, )
- accumulator_type: NumericLike#
Data type of the accumulator
- epilogue: EpilogueArguments | None#
Optional custom epilogue fusion to be performed after the GEMM
- property problem_size: GemmProblemSize#
Problem size for a GEMM operation.
- class cutlass.operators.arguments.grouped_gemm.IndexPtrGroupedGemmProblemSize(
- M: int,
- N: int,
- K: int,
- G: int,
- offsets: tuple[int, ...],
- offsets_along: Literal['m', 'n', 'k'],
Bases:
NamedTupleProblem shape for contiguous-offset grouped GEMM.
Exactly one of M, N, or K is packed across groups, as selected by
offsets_along. Each value inoffsetsis the exclusive endpoint of one group along that dimension, matching the representation accepted byIndexPtrGroupedGemmArguments.- M: int#
M extent, summed across groups when
offsets_along == "m".
- N: int#
N extent, summed across groups when
offsets_along == "n".
- K: int#
K extent, summed across groups when
offsets_along == "k".
- G: int#
Number of independent GEMMs.
- offsets: tuple[int, ...]#
Exclusive endpoint for each group along
offsets_along.
- offsets_along: Literal['m', 'n', 'k']#
Logical GEMM dimension partitioned by
offsets.
- class cutlass.operators.arguments.grouped_gemm.IndexPtrGroupedGemmArguments(
- A: TensorLike | Operand,
- B: TensorLike | Operand,
- out: TensorLike | Operand,
- accumulator_type: NumericLike,
- offsets: TensorLike | Operand,
- offsets_along: Literal['m', 'n', 'k'],
- epilogue: EpilogueArgumentsBase | None = None,
- *,
- offsets_value_divisibility: int = 1,
- accumulate_on_output: bool = False,
Bases:
RuntimeArgumentsArguments for a Grouped GEMM with contiguous tensors concatenated along M, N, or K.
A grouped GEMM performs a series of independent GEMM operations, where each GEMM can have different matrix dimensions. In this particular class, we represent grouped GEMMs where exactly one dimension can vary across groups; the other two remain constant. All operands are dense tensors, packed and contiguous. For the operand with the varying dimension, its total size is
sum(changing_dim) * fixed_dims.offsetsdescribes the problem boundaries alongoffsets_along. It is a tensor delineating the ending positions of each problem in the group.When
offsets_along == "m"the operands and computation are:A : DenseTensor of shape (TotalM, K) # TotalM = sum(M) over groups B : DenseTensor of shape (group_count, K, N) out : DenseTensor of shape (TotalM, N) offsets : DenseTensor of shape (group_count,) = [M0, M0+M1, ..., TotalM] start = 0 for g in range(group_count): end = offsets[g] out[start:end, :] = A[start:end, :] @ B[g, :, :] start = end
This is equivalent to Pytorch’s 2Dx3D Grouped GEMM.
Rows of
Aandoutare sliced byoffsets;Bis per group:A (TotalM x K) B[g,:,:] (K x N, per group) out (TotalM x N) K N N <---------> <-------> <-------> 0 ┌───────────┐ ┐ ┌─────────┐ ┐ ┌─────────┐ ┐ │ A0 │ │ M0 ─ @ ─ │ B0 │ │ K ────────► │ out0 │ │ M0 M0 ├───────────┤ ┤ └─────────┘ ┘ ├─────────┤ ┤ │ A1 │ │ M1 ─ @ ─ ┌─────────┐ ┐ │ out1 │ │ M1 M0+M1 ├───────────┤ ┤ │ B1 │ │ K ────────► ├─────────┤ ┤ │ A2 │ │ M2 ─ @ ─ └─────────┘ ┘ │ out2 │ │ M2 TotalM└───────────┘ ┘ ┌─────────┐ ┐ └─────────┘ ┘ ▲ │ B2 │ │ K ────────► offsets slice rows └─────────┘ ┘ [0:M0], [M0:M0+M1], ... per group g: out[start:end, :] = A[start:end, :] @ B[g, :, :] (Mg x N) (Mg x K) (K x N)
When
offsets_along == "n"the operands and computation are:A : DenseTensor of shape (group_count, M, K) B : DenseTensor of shape (K, TotalN) # TotalN = sum(N) over groups out : DenseTensor of shape (M, TotalN) offsets : DenseTensor of shape (group_count,) = [N0, N0+N1, ..., TotalN] start = 0 for g in range(group_count): end = offsets[g] out[:, start:end] = A[g, :, :] @ B[:, start:end] start = end
This is equivalent to Pytorch’s 3Dx2D Grouped GEMM.
Columns of
Bandoutare sliced byoffsets;Ais per group:A[g,:,:] (M x K, per group) B (K x TotalN) out (M x TotalN) K N0 N1 N2 N0 N1 N2 <─────────> <────><────><────> <────><────><────> ┌─────────────┐ ┐ ┌─────┬─────┬─────┐ ┌─────┬─────┬─────┐ │ A0 │ │ M ─ @ ─► │ │ │ │ │ │ │ │ └─────────────┘ ┘ K │ B0 │ B1 │ B2 │ K M │out0 │out1 │out2 │ M ┌─────────────┐ ┐ │ │ │ │ │ │ │ │ │ A1 │ │ M ─ @ ─► └─────┴─────┴─────┘ └─────┴─────┴─────┘ └─────────────┘ ┘ ▲ ▲ ▲ ┌─────────────┐ ┐ offsets slice columns │ A2 │ │ M ─ @ ─► [0:N0], [N0:N0+N1], ... └─────────────┘ ┘ per group g: out[:, start:end] = A[g, :, :] @ B[:, start:end] (M x Ng) (M x K) (K x Ng)
When
offsets_along == "k"each K-slice is an independent GEMM whose M x N product is written to its own section in the output tensor:A : DenseTensor of shape (M, TotalK) # TotalK = sum(K) over groups B : DenseTensor of shape (TotalK, N) out : DenseTensor of shape (group_count, M, N) # one M x N result per group offsets : DenseTensor of shape (group_count,) = [K0, K0+K1, ..., TotalK] start = 0 for g in range(group_count): end = offsets[g] out[g, :, :] = A[:, start:end] @ B[start:end, :] start = end
This is equivalent to Pytorch’s 2Dx2D Grouped GEMM.
offsetsslice columns ofAand rows ofB; each slice is one group:A (M x TotalK) B (TotalK x N) out (group_count, M, N) K0 K1 K2 N N <────><─────><─────> <──────> <──────> ┌─────┬──────┬──────┐ ┐ 0 ┌─────────┐ ┐ g=0 ┌─────────┐ ┐ M │ A0 │ A1 │ A2 │ │ M │ B0 │ │ K0 │ A0@B0 │ │ M └─────┴──────┴──────┘ ┘ K0 ├─────────┤ ┤ └─────────┘ ┘ │ │ │ │ B1 │ │ K1 g=1 ┌─────────┐ ┐ │ │ └────@───► B2 K0+K1 ├─────────┤ ┤ │ A1@B1 │ │ M │ └───────────@───► B1 │ B2 │ │ K2 └─────────┘ ┘ └──────────────────@───► B0 TotalK └─────────┘ ┘ g=2 ┌─────────┐ ┐ │ A2@B2 │ │ M └─────────┘ ┘ per group g: out[g, :, :] = A[:, start:end] @ B[start:end, :] (M x N) (M x Kg) (Kg x N)
See also
- A: DenseTensor | ScaledOperand#
Input tensor A.
The shape of A is (sum(M across all groups or TotalM), K) if offsets_along is “m” or (M, sum(K across all groups or TotalK)) if offsets_along is “k” or (group_count, M, K) if offsets_along is “n” where M, N, K are the dimensions of the GEMM problem.
Pass a
ScaledOperandfor a block-scaled input: it adds one scale factor tensor, which the caller assembles as the concatenation of per-group blocks. Each block holds one group’s scale factors, grown to whole layout atoms and arranged in the order named byScaledOperand.swizzleon its own, so a block of(rows, cols)scale factors occupiesround_up(rows, 128) * round_up(cols, 4)elements. The grouped extent is thereforesum(round_up(extent_g, P))over groups – notround_up(sum(extent_g), P)– wherePis 128 rows for a groupedmaxis and 4 scale columns for a groupedkaxis.Such a tensor is not one large arranged array that the groups index into. For a grouped
maxis the two descriptions coincide, but for a groupedkaxis they do not, because the grouped axis is then interior to the atom tiling.A caller that assembles on the host knows the per-group extents and can size exactly. When a device-side quantizer fills the buffer instead they are not host-visible, so allocate the upper bound
round_up(total, P) + (group_count - 1) * P. Over-allocating is accepted: the Operator checksP-alignment and a lower bound rather than an exact size, and the kernel derives each block’s position fromoffsets, so the unused tail is ignored.
- B: DenseTensor | ScaledOperand#
Input tensor B.
The shape of B is (K, sum(N across all groups or TotalN)) if offsets_along is “n” or (sum(K across all groups or TotalK), N) if offsets_along is “k” or (group_count, K, N) if offsets_along is “m” where M, N, K are the dimensions of the GEMM problem.
Pass a
ScaledOperandfor a block-scaled input: it adds one scale factor tensor, assembled as described onA.
- out: DenseTensor | ScaledOperand#
Output tensor.
The shape of out is (sum(M across all groups or TotalM), N) if offsets_along is “m” or (M, sum(N across all groups or TotalN)) if offsets_along is “n” or (group_count, M, N) if offsets_along is “k” where M, N, K are the dimensions of the GEMM problem.
Pass a
ScaledOperandfor a quantized output: the operator writes one scale factor tensor, whichoffsetspartitions across groups just as it does the quantized tensor.
- accumulator_type: NumericLike#
Data type of the accumulator.
- offsets: DenseTensor#
Integer tensor describing grouped-problem boundaries.
- offsets_along: Literal['m', 'n', 'k']#
Logical GEMM dimension partitioned by
offsets.
- offsets_value_divisibility: int = 1#
A caller-provided divisor of every cumulative boundary in
offsets.A value
Dpromises thatoffsets[g] % D == 0for every group and every invocation using these arguments. A weaker truthful promise is safe, but it may exclude Operators that require a stronger alignment guarantee. Callers should normally provide the strongest guarantee implied by howoffsetsare produced, without considering any particular kernel’s requirement. Leave this as1when no stronger invariant is known. Device offsets are not copied to the host to verify this promise.For a block-scaled grouped GEMM the promise has a second consequence. When
Dis a multiple of the scale-factor paddingPdescribed onA, no group needs padding, so the scale-factor block boundaries coincide with the data boundaries; Operators that therefore skip deriving them on the device become available, and the exact grouped scale extent collapses toround_up(total, P). Such Operators compute the same result, so a stronger promise here can only widen the set of candidates.
- accumulate_on_output: bool = False#
Whether to add each group’s product to that group’s existing output slice.
This option is valid only for
offsets_along="k". For groupg,Falsecomputesout[g, :, :] = A_g @ B_gandTruecomputesout[g, :, :] += A_g @ B_g. Results from different groups are never accumulated into one another.
- epilogue: EpilogueArgumentsBase | None = None#
Optional custom or fixed epilogue to perform after each grouped GEMM.
- class cutlass.operators.arguments.grouped_gemm.GroupedGemmArguments(
- A: TensorLike | Operand,
- B: TensorLike | Operand,
- out: TensorLike | Operand,
- accumulator_type: NumericLike,
- offsets: TensorLike | Operand,
- epilogue: EpilogueArgumentsBase | None = None,
Bases:
IndexPtrGroupedGemmArgumentsDeprecated compatibility interface for M-offset grouped GEMM.
Use
IndexPtrGroupedGemmArgumentswithoffsets_along="m".
- class cutlass.operators.arguments.epilogue.EpilogueArgumentsBase(*args, **kwargs)#
Bases:
ABC,SupportsConversionToInternalTypesCommon interface for custom and fixed epilogue arguments.
- abstract copy() EpilogueArgumentsBase#
Return an independent epilogue-argument wrapper.
- class cutlass.operators.arguments.epilogue.EpilogueArguments(epilogue_fn: Callable | str, **kwargs)#
Bases:
EpilogueArgumentsBaseDescribes a user-defined epilogue that is fused on top of the operation described by the primary RuntimeArguments.
An epilogue fusion is a custom function that performs tensor-level transformations on the result of a matrix multiplication. This transformation is fused into the kernel’s epilogue, which stores the final output.
EpilogueArgumentsencapsulates the epilogue function describing the transformation along with its arguments.To support flexible definition of epilogues,
EpilogueArgumentsis defined generically as taking in anepilogue_fnand additionalkwargs.Under the hood, the AST for
epilogue_fnis parsed to determine the operands and outputs of the epilogue.kwargsmust contain Tensors or scalars for all operands and outputs in the provided epilogue.Structure of ``epilogue_fn``
The
epilogue_fnis a function describing the custom transformation on the accumulator tensor (intermediate result before the epilogue).The general structure of these functions is:
def custom_epi_name(accum, *args) -> TensorType | tuple[TensorType, ...]: '''Compute the epilogue. # Args: accum (TensorType): Result of the primary operation (e.g. ``A @ B`` for a GEMM) before the epilogue. *args: Additional tensors or scalars used in the epilogue (e.g. aux tensors). # Returns: At least one tensor resulting from the epilogue computation. ''' # Do some compute return D # and potentially other values
epilogue_fnmust be a Python callable (or its string representation) that must satisfy the following constraints:Takes a first positional argument named
accum– the result of the operation just before the epilogue. For a GEMM,accum = A @ B.Returns at least one tensor resulting from the epilogue. Currently the return list must contain at least one output named
D.Each argument following
accumis a tensor or scalar to be loaded.Each variable in the return statement is a tensor or scalar to be stored.
Operations are represented in static single assignment (SSA) form. This means that each variable can be assigned exactly once.
The epilogue is parsed, never executed
The function body is a small domain-specific language: its source is parsed (a callable’s source is read back with
inspect.getsource, so callables and source strings are equivalent) and names are matched against the fixed vocabulary below. They do not resolve to Python objects –maxhere is notbuiltins.max– and calling arbitrary Python functions is not supported.Arithmetic:
+,-,*,/– elementwise, with scalar and row/column-vector broadcasting of the non-accumoperand.Elementwise functions:
relu,tanh,sigmoid,silu,hardswish,gelu,exp,maximum,minimum,multiply_add,identity.Gated linear units:
swiglu,geglu,reglu(see below).Layout:
permute,reshape.Reductions:
sum,maxandmin(see below).
Gated linear units
swiglu,gegluandreglucomputeup * activation(gate)from a single matmul whoseBinterleaves the gate and up projections along N – gate at the even N indices, up at the odd ones:def swiglu_epi(accum): D = swiglu(accum) return D
They pair two accumulator elements into one result, so unlike every other epilogue the output is not the accumulator’s shape:
D(and therefore the operation’sout) has half of B’s N. Nothing else in the epilogue changes shape, so a gated result may be scaled or combined with a tensor of its own half-width N.Reductions
A returned value may reduce the accumulator instead of storing a tile; the reduction is fused into the same kernel, so the
M x Nresult is never re-read:def epi_with_reductions(accum): D = accum total = sum(accum) # scalar: all axes folded col_max = max(accum, dim=[0]) # per-column: destination (N,) row_max = max(accum, dim=[1]) # per-row: column-major (M, 1) return D, total, col_max, row_max
The folded-vs-kept axes are derived from the layout of the destination tensor passed in
kwargs(thedim=keyword documents intent): a single-element tensor selects the scalar reduction, a 1-D(N,)tensor keeps the columns, and a column-major(M, 1)tensor keeps the rows – a contiguous(M, 1)or a bare(M,)is ambiguous and rejected. Reductions combine onto the destination with atomics, so initialise it to the operation’s identity (0forsum,-infformax,+infformin), or to an existing partial result to accumulate into it. Reduction destinations must be 32-bit wide (e.g.torch.float32): the fold and the atomics run in the destination’s dtype, and discovery offers no operator for a narrower destination. Batch axes are folded into the same destination; per-batch reduction outputs are not supported yet.Choosing data transports
By default the kernel decides how each operand is moved between global memory and the compute; wrap a tensor kwarg in
LoadorStore(e.g.C=Load(C, via=Transport.ASYNC_GMEM_LOAD)) to select a specific transport. See those classes below.Structure of ``kwargs``
kwargsmust contain sample Tensors or scalars for all operands and outputs in the provided epilogue. For example, with an epilogue of:def my_epi(accum, alpha, C, beta): F = (accum * alpha) + (C * beta) D = relu(F) return D, F
A user would need to construct epilogue arguments as follows:
epi_args = EpilogueArguments( my_epi, alpha=..., C=..., beta=..., D=..., F=... )
- copy() EpilogueArguments#
Return a copy of these arguments. Does not copy the underlying tensors.
Constructing a
RuntimeArgumentstraces the epilogue and converts its tensors, both of which mutate the object. Operating on a copy keeps a caller-suppliedEpilogueArgumentsreusable across several operations, mirroring howGemmArgumentscopies itsA/B/outoperands.- Returns:
An independent, untraced copy.
- Return type:
- property parameters: list[cute.Tensor | Numeric]#
Returns the list of input and output parameters of the epilogue.
- property parameter_names: list[str]#
Returns the list of names of the input and output parameters of the epilogue.
- to_tensor_wrappers(permute: list[int] | None = None)#
Converts the input and output parameters of the epilogue to TensorWrappers.
- trace(
- accumulator_shape: tuple[int, ...],
- accumulator_type: Numeric,
Traces the epilogue function and generates an internal representation of the epilogue.
- Parameters:
accumulator_shape (tuple[int, ...]) – The shape of the accumulator tensor. For example, for a GEMM, this would be the shape of the output tensor.
accumulator_type (Numeric) – The datatype of the accumulator tensor.
- class cutlass.operators.arguments.epilogue.PerGroupScaleEpilogueArguments(
- global_scale_a: TensorLike | DenseTensor | None = None,
- global_scale_b: TensorLike | DenseTensor | None = None,
Bases:
EpilogueArgumentsBaseA fixed epilogue that applies scale factors to each grouped GEMM result.
It describes a fixed epilogue that, for each group
g, computesscaled_accum_g = accum_g * global_scale_a[g] * global_scale_b[g]before output conversion. Eitherglobal_scale_aorglobal_scale_bmay be omitted, but not both. The converted result overwrites that group’s output slice, or is added to the existing slice when the grouped operation requests output accumulation.This is a fixed epilogue with operator-specific implementation. Unlike
EpilogueArguments, it does not support arbitrary transformations.- global_scale_a: DenseTensor | None = None#
Optional contiguous rank-1 FP32 scale tensor for A; None skips this scaling.
- global_scale_b: DenseTensor | None = None#
Optional contiguous rank-1 FP32 scale tensor for B; None skips this scaling.
- copy() PerGroupScaleEpilogueArguments#
Return an independent wrapper around the same scale tensors.
- class cutlass.operators.arguments.epilogue.Transport(value)#
Bases:
EnumHow a supplemental epilogue tensor moves between GMEM and registers.
TMAstages through shared memory; the*_GMEM_*variants use direct GMEM addressing, either synchronous to the issuing thread (SYNC_GMEM_LOAD/SYNC_GMEM_STORE) or asynchronous through shared memory viacp.async(ASYNC_GMEM_LOAD).
- class cutlass.operators.arguments.epilogue.Load(
- tensor: Any,
- via: Transport | str = Transport.TMA,
- num_bits_per_copy: int | None = None,
Bases:
objectDescribe how an epilogue input tensor is read.
Wrap an
EpilogueArgumentstensor kwarg to override the default (TMA) read transport, e.g.C=ops.Load(C, via=ops.Transport.ASYNC_GMEM_LOAD). Passing a bare tensor is equivalent toLoad(tensor)(a TMA read).- Parameters:
tensor (TensorLike) – The tensor to read.
via (Transport | str) – Read transport; one of
TMA,SYNC_GMEM_LOADorASYNC_GMEM_LOAD. Strings are accepted.num_bits_per_copy (int | None) – Transaction width for non-TMA transports;
Noneauto-derives it.
- class cutlass.operators.arguments.epilogue.Store(
- tensor: Any,
- via: Transport | str = Transport.TMA,
- num_bits_per_copy: int | None = None,
Bases:
objectDescribe how an epilogue output tensor is written.
Wrap an
EpilogueArgumentstensor kwarg to override the default (TMA) write transport, e.g.D=ops.Store(D, via=ops.Transport.SYNC_GMEM_STORE). Passing a bare tensor is equivalent toStore(tensor)(a TMA write).- Parameters:
tensor (TensorLike) – The tensor to write.
via (Transport | str) – Write transport; one of
TMAorSYNC_GMEM_STORE. Strings are accepted.num_bits_per_copy (int | None) – Transaction width for the direct store;
Noneauto-derives it.
PerformanceControls#
- class cutlass.operators.arguments.base.PerformanceControls#
Bases:
objectOptional runtime performance controls passed to the Operator.
Some operators may support performance options that can be controlled at runtime. This class is the general container for all such controls.
- class cutlass.operators.arguments.pdl.ProgrammaticDependentLaunchControl(
- early_launch_dependent: EarlyLaunchDependent | None = None,
- early_start: EarlyStart | None = None,
Bases:
objectControls for Programmatic Dependent Launch (PDL) at runtime.
PDL allows a dependent kernel to begin executing before its producer has completed. This class exposes the runtime controls for the PDL launch. A launch can use same-stream serialization, a programmatic CUDA event, or both.
For same-stream serialization, the producer and consumer kernels must use the same stream. The consumer must enable
pdl_stream_serializationto receive the early-launch signal.For a programmatic CUDA event, the producer must specify
pdl_programmatic_eventto signal the consumer.early_launch_dependentcontrols when and how the producer signals consumer kernels. The signal may use a programmatic CUDA event.early_startcontrols how a consumer starts before producer completion. It selects same-stream serialization and declares independent operands.
- For more details about PDL launches, see:
Leave both controls as
Noneto disable PDL behavior.- early_launch_dependent: EarlyLaunchDependent | None#
Optional runtime controls for launching dependent work early.
- early_start: EarlyStart | None#
Optional runtime controls for starting before producer completion.
- is_enabled() bool#
Whether controls describe any PDL support.
- class cutlass.operators.arguments.pdl.EarlyLaunchDependent(
- pdl_programmatic_event: PDLProgrammaticEvent | None = None,
Bases:
objectRuntime controls for launching dependent work early.
- pdl_programmatic_event: PDLProgrammaticEvent | None#
Optional programmatic CUDA event to signal from this launch.
- class cutlass.operators.arguments.pdl.EarlyStart(
- pdl_stream_serialization: bool = False,
- independent_operands: Iterable[str] = (),
Bases:
objectRuntime controls for starting before producer completion.
- pdl_stream_serialization: bool#
Whether to use same-stream serialization for this PDL launch.
- independent_operands: Iterable[str]#
Operands that are independent of the producer for this launch.
All input operands are dependent by default.
Declaring independent operands allows the consumer kernel to load them without waiting for the producer to complete, which may improve performance.
griddepcontrol_waitguards dependent operands but not independent operands. Incorrectly declaring a dependent operand as independent may cause a race condition.The runtime verifies that the selected kernel’s compiled early-start behavior supports this declaration. An empty list means that all operands depend on the producer.
- class cutlass.operators.arguments.pdl.PDLProgrammaticEvent(
- programmatic_event: cuda_driver.CUevent | cuda_runtime.cudaEvent_t,
- programmatic_event_flags: int = 0,
- programmatic_event_trigger_at_block_start: bool = False,
Bases:
objectProgrammatic CUDA event launch attributes for PDL.
- programmatic_event: cuda_driver.CUevent | cuda_runtime.cudaEvent_t#
CUDA event to signal the dependent kernel.
- programmatic_event_flags: int#
Optional event-record flags.
See https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__EVENT.html.
- programmatic_event_trigger_at_block_start: bool#
Whether to trigger the programmatic event automatically at block start.
Truetriggers the event automatically at block start.Falserequires the kernel to trigger the event manually.This affects the constructed graph edge. See https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/programmatic-dependent-launch.html.
- class cutlass.operators.arguments.data_driven_sync.DataDrivenSyncControls(
- *,
- producer: DataDrivenSyncProducer | None = None,
- consumer: DataDrivenSyncConsumer | None = None,
Bases:
objectRuntime controls for data-driven producer-consumer synchronization.
Data-driven synchronization allows a consumer operator to read the producer’s outputs before the producer operator has completed. The controls may request a producer role, a consumer role, or both.
Using data-driven synchronization requires additional launch configuration by the user. See the data-driven synchronization example notebook for more details.
Typically paired with Programmatic Dependent Launch (PDL) to overlap the producer and consumer launches for concurrent reads/writes to the synchronized tensor.
Specifying
Nonemeans role is not implemented.- producer: DataDrivenSyncProducer | None#
Optional controls for a launch that produces data consumers load with
report_valid.
- consumer: DataDrivenSyncConsumer | None#
Optional controls for a launch that consumes producer output with
report_validloads.
- is_enabled() bool#
Return whether either data-driven synchronization role is requested.
- class cutlass.operators.arguments.data_driven_sync.DataDrivenSyncProducer(*, assert_user_will_prefill: bool = False)#
Bases:
objectRuntime controls for a data-driven sync producer.
A data-driven sync producer writes data that a consumer may read before the producer operator completes. The producer output must use the TMA
report_validsentinel pattern for unwritten data and must never produce a legitimate-0.0(NaNfor scale factors) in completed output.- assert_user_will_prefill: bool#
Whether the user assumes responsibility for prefilling the output with the sentinel pattern.
When
True, the operator’s caller must fill its output with the sentinel pattern before launching dependent consumers. The producer will not perform the fill.When
False, the producer operator will initialize the output with the sentinel pattern. The user does not need to prefill the output.
- class cutlass.operators.arguments.data_driven_sync.DataDrivenSyncConsumer(*, consumed_operands: tuple[str, ...] = ())#
Bases:
objectRuntime controls for a data-driven sync consumer.
- consumed_operands: tuple[str, ...]#
Operands that user wants consumer to load with
report_validenabled from a producer.Any operand in this list cannot contain the sentinel pattern as a legitimate computed value.
Example: GemmArguments(A, B, out) -> consumed_operands=(“B”,)
Operands#
- class cutlass.operators.arguments.base.Operand(*args, **kwargs)#
Bases:
ABC,SupportsConversionToInternalTypesBase class for all operands to Operators, which encapsulates one or more TensorLike objects.
In the most basic case, an Operand enacapsulates a single tensor.
In more complex cases, an Operand may encapsulate multiple tensors that encapsulate a single logical operand. For instance, a
ScaledOperandencapsulates a quantized and scale tensor, that together reconstruct the logical value of the operand.
- class cutlass.operators.arguments.operand.DenseTensor(tensor: cutlass.operators.typing.TensorLike)#
Bases:
OperandAn operand encapsulating a simple, single dense tensor.
- class cutlass.operators.arguments.operand.ScaledOperand(
- quantized: TensorLike | DenseTensor,
- scale: TensorLike | DenseTensor,
- mode: ScaleMode | tuple[int, ...],
- swizzle: ScaleSwizzleMode,
Bases:
OperandAn operand whose logical value is a quantized tensor multiplied by a tensor of scale factors.
A
ScaledOperandencapsulates physicalquantizedandscaletensors that together represent the logical valuescale * quantized: each entry ofscalebroadcasts and multiplies a contiguous block ofquantized, where the block shape is given bymode.This Operand is typically used to express narrow-precision formats (such as OCP MXFP8 / MXFP4 and NVIDIA NVFP4), which store the data as a quantized narrow-precision tensor multiplied by a separate tensor of scale factors that recover dynamic range.
The scale
modeandswizzledescribe the physical layout of the scale tensor.The
modedescribes how scale factors are broadcast over the quantized tensor, and therefore determines its size. It’s usually dictated by the data format you seek to represent – MXFP8 / MXFP4 / NVFP4 require specific scale modes. SeeScaleModefor more details.The
swizzledescribes the in-memory layout of the scale tensor. It’s usually dictated by the hardware architecture – for example, Blackwell block-scaled MMAs require a specific swizzle layout, and most kernels using them require the input to be pre-arranged in this layout. SeeScaleSwizzleModefor more details.Example
quantized_A = torch.randn(M, K, dtype=torch.float8_e4m3fn, device="cuda") scale_A = torch.randn(M, K // 32, dtype=torch.float8_e8m0fnu, device="cuda") A = ScaledOperand( quantized_A, scale_A, ScaleMode.Blockwise1x32, ScaleSwizzleMode.Swizzle32x4x4, )
See also
ScaleMode: granularity of the scaling, and the format-to-mode mapping.ScaleSwizzleMode: in-memory layout of the scale tensor.
References
- quantized: DenseTensor#
Narrow-precision tensor holding the quantized values.
The dequantized logical value is
scale * quantized: each scale factor broadcasts over a block of this tensor (block shape given bymode) and multiplies its values.
- scale: DenseTensor#
Tensor of scale factors.
Each entry of this tensor broadcasts over a block of
quantizedand scales it.modedefines that broadcast, whileswizzlenames the required physical element order. Together they determine the required element count, but they do not define one canonical host tensor shape.This tensor must be contiguous and already arranged in the layout named by
swizzle.ScaledOperand.numel_scale()returns the required element count for one logical quantized shape. A consuming Operator may validate capacity and may require a carrier shape that communicates padded extents. Neither check validates the physical element order; callers must ensure that the values satisfy the swizzle contract.
- mode: ScaleMode | tuple[int, ...]#
Block shape over which each scale factor is broadcast.
Typically, a block of quantized-tensor elements share the same scale factor value. The logical value of the scaled operand is the result of broadcasting the scale factor along the block shape described by
mode, and then multiplying it element-wise with the quantized tensor.This may be a
ScaleModeenum for commonly used block shapes, or a bare(L, M, K)tuple for custom block shapes. For instance,(1, 1, 32)means that each scale factor covers 32 elements along the K axis, and 1 element along the L and M axes.
- swizzle: ScaleSwizzleMode#
In-memory layout of the scale factor tensor.
The scale factor tensor is sometimes required to be stored in a specific swizzled layout, often dictated by the hardware MMA’s contract.
- static numel_scale(
- quantized_shape: tuple[int, ...],
- mode: ScaleMode | tuple[int, ...],
- swizzle: ScaleSwizzleMode,
Return the canonical scale element count for
quantized_shape.quantized_shapeis(L, outer, K)for a 3D operand, whereouterisMfor an A-side scale andNfor a B-side scale. One scale factor coversVelements alongK, so the scale tensor has logical shape(L, outer, ceil_div(K, V)).Returned value depends on
swizzle:ScaleSwizzleMode.SwizzleNone:L * outer * ceil_div(K, V)ScaleSwizzleMode.Swizzle32x4x4:L * round_up(outer, 128) * round_up(ceil_div(K, V), 4)
where
V = ScaleMode.numel(mode)is the number of quantized-tensor elements covered by one scale factor.This is the count corresponding to one logical
quantized_shape. When grouped scale storage is assembled by padding and concatenating per-group blocks, callnumel_scalefor each group’s logical shape and sum the results. This generally differs from callingnumel_scaleonce with the concatenated shape. SeeIndexPtrGroupedGemmArguments.Afor details.- Parameters:
quantized_shape (tuple[int, ...]) – Logical shape of the operand the scale will be paired with, as
(L, outer, K)or(outer, K).mode (ScaleMode | tuple[int, ...]) – Block shape of the scaling.
swizzle (ScaleSwizzleMode) – In-memory layout of the scale tensor.
- Returns:
Required number of scale-factor elements.
- Return type:
int
- Raises:
ValueError – If
quantized_shapeis not rank 2 or 3, orswizzleis not a recognisedScaleSwizzleMode.
- class cutlass.operators.arguments.operand.ScaleMode(value)#
Bases:
EnumAn enum over commonly used block scaling modes for a
ScaledOperand.Each member’s value is a
(batch, row, col)tuple giving the shape of the block of quantized-tensor elements that share a scale factor. Each scale factor is broadcast & multiplied over this block of quantized-tensor elements.For block shapes that are not enumerated here, bare
(batch, row, col)tuples are also accepted asScaledOperand.modeSee also
ScaledOperand: the operand that consumes this enum.ScaleSwizzleMode: in-memory layout of the scale tensor.
- Blockwise1x16 = (1, 1, 16)#
Scale over
(1, 1, 16)block. Used by NVIDIA NVFP4 (FP8 E4M3 scale dtype).
- Blockwise1x32 = (1, 1, 32)#
Scale over
(1, 1, 32)block. Used by OCP MXFP8 / MXFP4 (E8M0 scale dtype).
- classmethod from_string(
- value: str,
Return the scale mode matching a profiler CLI spelling.
- Accepted values (case-insensitive):
"1x16"maps toBlockwise1x16."1x32"maps toBlockwise1x32.
- Parameters:
value (str) – Scale-mode spelling to parse.
- Returns:
Matching scale mode.
- Return type:
- Raises:
ValueError – If
valuedoes not name a supported scale mode.
- static compare( ) bool#
Return whether two scale modes describe the same scaling.
Allows mixed comparisons between
ScaleModeenums and bare tuples, and tolerates differing tuple lengths as long as the longer tuple has leading1s for the extra positions (so that, e.g.,(1, 1, 16) == (1, 16)).
- static numel(
- scale: ScaleMode | tuple[int, ...],
Return the number of quantized-tensor elements scaled by one scale factor.
For a scale mode of block shape
(L, M, K), this is the block volumeL * M * K– the number of contiguous elements in a quantized tensor that share (and are multiplied by) the same scale factor.- Parameters:
scale (ScaleMode | tuple[int, ...]) – Scale mode whose tuple entries give the block shape.
- Returns:
Product of the entries of
scale(e.g.numel(Blockwise1x32) == 32).- Return type:
int
- class cutlass.operators.arguments.operand.ScaleSwizzleMode(value)#
Bases:
EnumIn-memory layout of the scale tensor in a
ScaledOperand.Hardware MMAs that consume block-scaled inputs typically require their scale tensors to be stored in a specific non-trivial layout.
ScaleSwizzleModeindicates that scale factors inside aScaledOperandare already stored in one of those specific layouts.Each Operator validates the contiguity and capacity required by its addressing scheme, but does not inspect or rearrange scale values. The producer of the scale tensor, typically a quantizer, is responsible for writing values in the named layout.
See also
ScaledOperand: the operand that consumes this enum.ScaleMode: granularity of the scaling.PyTorch
torch.nn.functional.SwizzleType, which this enum closely resembles.
- SwizzleNone = 1#
Scale tensor stored in the natural ordering implied by
ScaleMode.For an
(L, M, K)operand at mode(1, 1, V), this is a tensor ofL * M * (K // V)scale factors with one value permodeblock.
- Swizzle32x4x4 = 2#
1D block-scaling layout dictated by Blackwell
tcgen05.mmablock-scaled MMAs.This is used for MXFP8 / MXFP4 / NVFP4 GEMMs on Blackwell. Under this layout, each tile contains 128x4 scale factors, and the 128 rows are interleaved in groups of 32, matching the warp-group structure of the Blackwell tensor core.
For a quantized tensor of shape
(L, M, K)and a scale vector size of V, the scale tensor is required to haveL * round_up(M, 128) * round_up(ceil_div(K, V), 4)elements.References
- classmethod from_string(
- value: str,
Return the scale swizzle matching a profiler CLI spelling.
- Accepted values (case-insensitive):
"none"maps toSwizzleNone."32x4x4"maps toSwizzle32x4x4.
- Parameters:
value (str) – Scale-swizzle spelling to parse.
- Returns:
Matching scale swizzle.
- Return type:
- Raises:
ValueError – If
valuedoes not name a supported scale swizzle.
Layout markers#
Normalized major-mode markers.
- cutlass.operators.utils.layout.normalized_row_major() tuple[int, int, int]#
Returns the normalized row major strides for a rank 3 tensor.
Leading dimension has stride=1. All others are 0. Example: shape (L=batch, M=row, K=column) => stride (0, 0, 1)
Uses definition provided by normalized_major_stride.
- cutlass.operators.utils.layout.normalized_col_major() tuple[int, int, int]#
Returns the normalized column major strides for a rank 3 tensor.
Leading dimension has stride=1. All others are 0. Example: shape (L=batch, M=row, K=column) => stride (0, 1, 0)
Uses definition provided by normalized_major_stride.
Type markers#
Type markers for CUTLASS Operator API field annotations.
- cutlass.operators.typing.TensorLike#
Tensor-like inputs accepted by the Operator API.
- Includes:
Any DLPack-compatible tensor (
torch.Tensor,jax.Array,numpy.ndarray, …).cutlass.cute.Tensor(CuTe DSL host tensor; does not implement DLPack but is handled natively byTensorWrapper).cutlass.operators.utils.tensor.TensorWrapper(implements DLPack).
alias of
_SupportsDLPack|Tensor|TensorWrapper
- class cutlass.operators.typing.NumericLike(*args, **kwargs)#
Bases:
ProtocolType marker for fields that accept numeric-like types.
- Fields annotated with NumericLike accept the following:
cutlass.Numerictorch.dtype
- class cutlass.operators.typing.AnyValue(value_type: type[_ValueT])#
Bases:
Generic[_ValueT]A domain that contains every value of type
_ValueT.
- class cutlass.operators.typing.SupportsConversionToInternalTypes(*args, **kwargs)#
Bases:
ProtocolProtocol for objects that support conversion to internal Cutlass Operator types.
Used by RuntimeArguments and Operand classes.