Operators#

Operators are used to describe the properties and to configure the execution of the problem we want to solve. They are divided into Description Operators and Execution Operators.


Description Operators#

Operator

Default Value

Description

Size<M, N, K>

Not set

Defines the problem size.

Function<function>

Not set

BLAS function. Use function::MM for GEMM.

Arrangement<arrangement, arrangement, arrangement>

row_major, col_major, col_major

Arrangement of each matrix in global memory and default arrangement in shared memory.

TransposeMode<transpose_mode, transpose_mode>

transposed, non-transposed

Transpose mode of the each matrix (Deprecated since 0.2.0).

Precision<PA, PB, PC>

float, float, float

Computation Precision of GEMM A, B, and C - either all floating point or all integral.

RequiredMantissaBits<X>

Not set

Requests flexible-precision floating-point emulation with X mantissa bits, with native fallback based on architecture and precision.

Type<type>

type::real

Type of input and output data (type::real or type::complex).

LeadingDimension<LDA, LDB, LDC>

As defined by Size, and Arrangement or TransposeMode operators

Leading dimensions for matrices A, B, C.

Alignment<a_align, b_align, c_align>

alignof(BLAS::a_value_type), alignof(BLAS::b_value_type), alignof(BLAS::c_value_type)

Alignments (in bytes) of matrices A, B, C.

SM<CC>

Not set

Target CUDA architecture for which the BLAS function should be generated.

Side<side>

side::left

Side of the equation on which the triangular matrix A appears. See TRSM.

FillMode<fill_mode>

fill_mode::lower

Which triangle of matrix A contains the data. See TRSM.

Diag<diag>

diag::non_unit

Whether the diagonal of A is stored (non_unit) or implicitly ones (unit). See TRSM.

BatchesPerBlock<N>

1

Number of independent TRSM instances solved per CUDA block. See TRSM.

Description operators define the problem we want to solve. Combined with Execution Operators, they form a complete function descriptor that can be executed on a GPU.

Operators are added (in arbitrary order) to construct the operation descriptor type. For example, to describe a matrix multiplication for non-transposed matrices A (m x k), B (k x n), C (m x n) with complex double precision values where m = 8, n = 16, k = 32 for execution on Ada architecture, one would write:

#include <cublasdx.hpp>

using GEMM = decltype(cublasdx::Size<8, 16, 32>()
              + cublasdx::Precision<double>()
              + cublasdx::Type<cublasdx::type::complex>()
              + cublasdx::Arrangement<cublasdx::col_major, cublasdx::col_major>()
              + cublasdx::Function<cublasdx::function::MM>()
              + cublasdx::SM<890>());

For a function descriptor to be complete, the following is required:

If neither Arrangement Operator nor the deprecated TransposeMode Operator is specified, cuBLASDx uses the default global-memory arrangement Arrangement<row_major, col_major, col_major>.

Size Operator#

cublasdx::Size<unsigned int M, unsigned int N, unsigned int K>()

Sets the problem size of the function to be executed.

For GEMM:

  • M - logical number of rows in matrices op(A) and C.

  • N - logical number of columns in matrices op(B) and C.

  • K - logical number of columns in matrix op(A) and rows in op(B).

For example, for GEMM M, N, and K specify that the A (M x K) matrix is multiplied by B (K x N) matrix which results in C (M x N) matrix (assuming A and B are non-transposed). See Arrangement, TransposeMode and GEMM.

Type Operator#

cublasdx::Type<cublasdx::type T>;

namespace cublasdx {
  enum class type
  {
    real,
    complex
  };
}

Sets the type of input and output data used in computation. Use type::real for real data type, and type::complex for complex data type.

Precision Operator#

cublasdx::Precision<PA, PB=PA, PC=PA>;

Sets the precision of computation for A, B, and C.

The precision type can be either floating :

  • __half

  • float

  • double

  • __nv_fp8_e5m2

  • __nv_fp8_e4m3

  • __nv_bfloat16

  • cublasdx::tfloat32_t

or integral:

  • int8_t

  • uint8_t

  • int16_t

  • uint16_t

  • int32_t

  • uint32_t

  • int64_t

  • uint64_t

It describes the computation precision for the values used for input and output. This means that this is the type to which the input values will be converted just before executing the multiplication instruction.

Note

Starting from cuBLASDx 0.3.0, computational precision has been decoupled from data precision, i.e. the input / output data for each matrix can be of arbitrary type (even integral input for floating point GEMM) provided that Alignment Operator is set and at least one of those conditions is met:

  1. It’s implicitly convertible to the data type chosen with Precision Operator and Type Operator.

  2. For inputs: An appropriate converting loading operation is provided as one of the arguments. It takes the input type value. Its result must be at least implicitly convertible to the compute type.

  3. For output: An appropriate converting storing operation is provided as one of the arguments. It takes the result computational type (usually C type as defined by Precision Operator and Type Operator). Its result must be at least implicitly convertible to the output type.

Warning

If using computation precision decoupled from input types, the Alignment Operator must be explicitly set.

RequiredMantissaBits Operator#

cublasdx::RequiredMantissaBits<unsigned int X>()

Enables flexible-precision floating-point emulation for pipelined GEMM. The value X is the requested number of mantissa bits preserved by the emulation path.

X may exceed the type’s stored mantissa width: 52 bits for double or 23 bits for float. The number of eight-bit emulation slices is ceil((X + 1) / 8). For example, RequiredMantissaBits<75>() is valid for either type and selects 10 slices. Extra slices can preserve smaller input contributions under the shared exponent of a row of A or a column of B. The output still uses the descriptor’s floating-point type.

When this operator is present, SM89, SM100, SM110 (called SM101 before CUDA 13), SM120, and SM121 always use emulation. On other architectures, suggest_pipeline uses the regular FP32 pipeline for Precision<float>() when X <= 24, or the regular FP64 pipeline for Precision<double>() when X <= 53. These thresholds include the implicit leading significand bit. Requests above those thresholds use emulation on every architecture. Native fallback uses the descriptor’s full precision; it does not truncate the result to X bits.

Note

cublasdx::emulated_stage_size_ratio_v<BLAS> is the ratio of the descriptor’s input element size to the emulation slice element size. It is sizeof(double) / sizeof(slice_type) == 8 for FP64 emulation, 4 for FP32 emulation, and 1 when the regular precision pipeline is selected. An emulated A/B pipeline stage can therefore use less shared memory than a native FP64 stage, permitting larger tiles within the same shared-memory limit. The actual buffer size also depends on layout, pipeline depth, and other storage.

Warning

When emulation is selected, descriptors using RequiredMantissaBits<> perform device memory allocations and deallocations for temporary emulation storage. The temporary storage is on the order of the combined size of the A and B input matrices.

Requirements:

  • X > 0.

  • Use exactly one RequiredMantissaBits operator in a descriptor.

  • Use with Function<function::MM>, Block, and WithPipeline.

  • Use Type<type::real>().

  • Use either Precision<float, float, float>() or Precision<double, double, double>().

See Using Pipelined GEMM for the full pipeline creation requirements and error reporting behavior.

For example, the following descriptor requests an emulated double-precision GEMM pipeline:

using BLAS = decltype(cublasdx::Size<64, 64, 64>()
              + cublasdx::Precision<double, double, double>()
              + cublasdx::Type<cublasdx::type::real>()
              + cublasdx::Arrangement<cublasdx::row_major, cublasdx::col_major>()
              + cublasdx::Function<cublasdx::function::MM>()
              + cublasdx::SM<900>()
              + cublasdx::Block()
              + cublasdx::WithPipeline()
              + cublasdx::RequiredMantissaBits<52>());

Arrangement Operator#

cublasdx::Arrangement<cublasdx::arrangement A, cublasdx::arrangement B = cublasdx::col_major, cublasdx::arrangement C = cublasdx::col_major>;

namespace cublasdx {
  enum class arrangement
  {
    col_major,
    row_major
  };

  inline constexpr auto col_major   = arrangement::col_major;
  inline constexpr auto left_layout = arrangement::col_major;
  inline constexpr auto row_major   = arrangement::row_major;
  inline constexpr auto right_layout = arrangement::row_major;
}

Sets the order for global A, B, and C matrices used in the function. The order can be either column-major or row-major. Arrangement operator directly influences Get Memory Layout and Suggested Shared Memory Layout methods, by explicitly setting the default Get Memory Layout value and choosing Suggested Shared Memory Layout for optimized global-shared transfers.

Selecting a specific order for matrices doesn’t mean the function doesn’t accept matrices with different layouts, but it may influence the overall performance.

Warning

Arrangement and TransposeMode operators can’t be defined at the same time.

TransposeMode Operator#

Warning

TransposeMode operator is deprecated since 0.2.0 and may be removed in future versions.

cublasdx::TransposeMode<cublasdx::transpose_mode ATransposeMode, cublasdx::transpose_mode BTransposeMode>;

namespace cublasdx {
  enum class transpose_mode
  {
    non_transposed,
    transposed,
    conj_transposed,
  };

  inline constexpr auto N = transpose_mode::non_transposed;
  inline constexpr auto T = transpose_mode::transposed;
  inline constexpr auto C = transpose_mode::conj_transposed;
}

Sets the transpose mode for the A and B matrices used in the function. For example, TransposeMode<N, N>() sets transpose mode of A and B matrix as non-transposed for GEMM. Possible values for transpose mode are:

  • transpose_mode::non_transposed,

  • transpose_mode::transposed, and

  • transpose_mode::conj_transposed (conjugated transposed).

Warning

  • Arrangement and TransposeMode operators can’t be defined at the same time.

  • Using transpose_mode::non_transposed for a matrix in TransposeMode operator corresponds to arrangement::col_major in Arrangement operator.

  • Using transpose_mode::transposed for a matrix in TransposeMode operator corresponds to arrangement::row_major in Arrangement operator.

  • Using transpose_mode::conj_transposed for a matrix in TransposeMode operator corresponds to arrangement::row_major in Arrangement operator and cublasdx::conjugate passed in execute() method as transform operator for that matrix.

LeadingDimension Operator#

cublasdx::LeadingDimension<unsigned int LDA, unsigned int LDB, unsigned int LDC>()

Defines leading dimensions for matrices A, B, and C. The leading dimension of a matrix is a stride (in elements) to the beginning of the next column for a column-major matrix or the next row for a row-major matrix.

If Arrangement is used in the description of a general matrix multiplication operation, A, B, C matrices can be described in a following way:

Table 3 GEMM leading-dimension requirements with Arrangement#

Matrix

Column-major storage

Row-major storage

A (M x K)

Real dimensions are \(LDA\times K\); require LDA >= M.

Real dimensions are \(LDA\times M\); require LDA >= K.

B (K x N)

Real dimensions are \(LDB\times N\); require LDB >= K.

Real dimensions are \(LDB\times K\); require LDB >= N.

C (M x N)

Real dimensions are \(LDC\times N\); require LDC >= M.

Real dimensions are \(LDC\times M\); require LDC >= N.

A matrix can also be described using a layout (see CuTe: Layout), i.e. a pair of integer tuples: shape and stride (distance between elements). In simple terms, a shape represents the number of elements in each dimension, and stride represents distance between elements in each dimension.

  • A - matrix of \(M\times K\) shape, with 1 stride in 1st dimension and LDA in 2nd dimension if A is column-major, or with LDA in 1st and 1 in 2nd dimension otherwise.

  • B - matrix of \(K\times N\) shape, with 1 stride in 1st dimension and LDB in 2nd dimension if B is column-major, or with LDB in 1st and 1 in 2nd dimension otherwise.

  • C - matrix of \(M\times N\) shape, with 1 stride in 1st dimension and LDC in 2nd dimension if C is column-major, or with LDC in 1st and 1 in 2nd dimension otherwise.

Warning

TransposeMode operator is deprecated since 0.2.0 and may be removed in future versions.

If TransposeMode (deprecated) is used in the description of a general matrix multiplication operation, in BLAS nomenclature the dimensions of the A, B, C matrices can be described in a following way:

  • Real dimensions of matrix A are \(LDA\times K\) with LDA >= M if A is non-transposed, and \(LDA\times M\) with LDA >= K otherwise.

  • Real dimensions of matrix B are \(LDB\times N\) with LDB >= K if B is non-transposed, and \(LDB\times K\) with LDB >= N otherwise.

  • Real dimensions of matrix C are \(LDC\times N\) with LDC >= M.

See also, suggested_leading_dimension_of.

Alignment Operator#

cublasdx::Alignment<unsigned int AAlignment, unsigned int BAlignment, unsigned int CAlignment>()

cublasdx::MaxAlignment = Alignment<16, 16, 16> // alias of maximum supported alignments

Defines the alignments (in bytes) of the pointers to matrices A, B, and C (either raw pointers or wrapped in CuTe tensors) that are passed to the execute(...) method.

Alignment operator is “sticky”, which means that it will dictate alignment wherever it’s not specified explicitly, some examples are:

  • Alignment in bytes of dynamic leading dimensions (e.g. it’s incorrect to set Alignment to 16 and use leading dimension of 7 for fp16 datatype).

  • In Pipelining API it will describe:
    • byte alignment of global memory pointers,

    • byte alignment of dynamic global memory strides, and

    • byte alignment of shared memory pointers (may be bumped up in some cases, as pipeline owns the shared memory).

For the tile API copy paths (cublasdx::copy, execute), statically known strides constrain the effective copy alignment at compile time; they need not be multiples of the declared alignment. For example, a packed FP16 layout with a static leading dimension of 7 can use Alignment<16, 16, 16>: the base pointers remain 16-byte aligned, while the library derives a narrower copy alignment from the layout. Pipelines are stricter: pipeline creation rejects, at compile time, any static leading dimension or batch stride of a global A/B tensor whose size in bytes is not a multiple of the declared Alignment<>. Dynamic strides (and base pointers) are validated at pipeline creation against the effective copy alignment (the declared alignment as retained by the selected copies) and return pipeline_error_code::alignment_unfulfilled when the promise is not met.

Note that the alignments have direct implication for how much shared memory is required for given a description operator. Additionally, it may also impact the performance.

Requirements:

  • AAlignments, BAlignments, CAlignments should be powers-of-two and less than or equal to the maximum supported alignments, i.e., 16 bytes.

  • AAlignments, BAlignments, CAlignments are multiple of alignment of either chosen compute value types, or input value types if decoupled precision is used.

Warning

If using computation precision decoupled from input types, the Alignment Operator must be explicitly set.

See also, Suggested Alignment Trait.

Function Operator#

cublasdx::Function<cublasdx::function F>()

namespace cublasdx {
  enum class function
  {
    MM,    // General matrix multiply (GEMM)
    TRSM   // Triangular solve
  };
}

Sets the BLAS function to be executed.

General Matrix Multiply#

Function<function::MM> sets the operation to general matrix multiply, defined as one of:

  1. \(\mathbf{C}_{m\times n} = {\alpha} \times \mathbf{A}_{m\times k} \times \mathbf{B}_{k\times n} + {\beta} \times \mathbf{C}_{m\times n}\)

  2. \(\mathbf{C}_{m\times n} = \mathbf{A}_{m\times k} \times \mathbf{B}_{k\times n}\)

where \({\alpha}\) and \({\beta}\) are scalars (real or complex), and A, B, and C are matrices with dimensions \(A: m\times k\), \(B: k\times n\), and \(C: m\times n\), respectively.

The matrices can be column-major, row-major, have a custom or suggested layout. See GEMM Execution Methods, Get Memory Layout, Suggested Shared Memory Layout, and Arrangement.

Triangular Solve (TRSM)#

Function<function::TRSM> sets the operation to triangular solve. It solves one of:

  1. \(A \times X = B\) — left-side: Side<side::left>

  2. \(X \times A = B\) — right-side: Side<side::right>

where A is a triangular matrix and B is overwritten in-place with the solution X. There is no alpha or beta scalar — the solve is always \(B \leftarrow A^{-1} B\) (left) or \(B \leftarrow B A^{-1}\) (right).

A TRSM descriptor requires Size, SM, and an execution operator. The following operators are optional and default to the values shown:

  • Side — default: side::left

  • FillMode — default: fill_mode::lower

  • Diag — default: diag::non_unit

Size for TRSM takes only two dimensions Size<M, N>: M rows and N columns of B. The triangular matrix A is square with dimension M (left-side) or N (right-side).

Note

TRSM requires linking against the pre-built libcublasdx.fatbin device-code library. A compile-time static_assert is emitted if the CMake target is set up incorrectly.

See Quick Installation Guide for more information regarding linking setup in your CMake project or directly with NVCC.

Example descriptor for a left-side, lower-triangular, non-unit TRSM:

#include <cublasdx.hpp>

using TRSM = decltype(cublasdx::Size<32, 16>()
              + cublasdx::Precision<float>()
              + cublasdx::Type<cublasdx::type::real>()
              + cublasdx::Function<cublasdx::function::TRSM>()
              + cublasdx::Side<cublasdx::side::left>()
              + cublasdx::FillMode<cublasdx::fill_mode::lower>()
              + cublasdx::Diag<cublasdx::diag::non_unit>()
              + cublasdx::SM<900>()
              + cublasdx::Block());

SM Operator#

cublasdx::SM<unsigned int CC, sm_modifier Modifier = cublasdx::generic>()

Sets the target architecture CC for the underlying BLAS function to use. Supported architectures are:

  • Turing: 750 (sm_75).

  • Ampere: 800, 860 and 870 (sm_80, sm_86, sm_87).

  • Ada: 890 (sm_89).

  • Hopper: 900 (sm_90).

  • Blackwell: 1000, 1100, 1030, 1200, 1210 (sm_100, sm_103, sm_110, sm_120, sm_121).

  • Rubin: 1070 (sm_107).

Each architecture can have a modifier:

  • cublasdx::generic, portable PTX generation, default option.

  • cublasdx::arch_specific, potentially non-portable PTX generation, accelerated mode enabling WGMMA / 1SM UTCMMA.
    • This implies the use of accelerated target in compilation, e.g. sm_100a instead of sm_100, or sm_107a instead of sm_107.

    • BlocksPerCluster can additionally enable 2SM UTCMMA on supported architectures.

  • cublasdx::family_specific, potentially non-portable PTX generation, family.
    • This implies the use of family target in compilation, e.g. sm_100f instead of sm_100.

  • The vectorized fp32 FMA (fma.rn.f32x2) used on 1000, 1030, 1070 and 1100 is generic PTX and does not require a modifier. SM110 has no FFMA2 unit and executes it as two FFMA.

Warning

It is not guaranteed that executions of exactly the same BLAS function with exactly the same inputs on GPUs of different CUDA architectures will produce bit-identical results.

Note

If the created descriptor type (that includes the SM<CC> operator) is passed to a CUDA kernel using a template parameter, wrapping the body of the CUDA kernel in the compile-time macro CUBLASDX_SKIP_IF_NOT_APPLICABLE_SM_BLOCK(GEMM) is recommended. This practice avoids kernel compilation for unintended device architectures, thereby reducing compilation time and the final binary size. This is only necessary if the compilation unit is compiled for multiple CUDA architectures.

#include <cublasdx.hpp>

template<typename GEMM>
__global__ void user_kernel(...) {
  CUBLASDX_SKIP_IF_NOT_APPLICABLE_SM_BLOCK(GEMM) {
    // rest of the kernel code ...
  }
}

The macro expands to an if constexpr statement, so the entire kernel body must be placed inside its braces. Two limitations follow from that:

  • It only prevents instantiation inside a template. if constexpr discards a branch without instantiating it only when the condition depends on a template parameter, so in a non-template kernel that uses a concrete descriptor (e.g. using GEMM = decltype(... + SM<900>()) at namespace scope) the discarded body is still instantiated for every targeted architecture and can fail to compile there. Make such kernels templates on the descriptor.

  • An else branch after the block is not supported. In the host compilation pass the macro expands to nothing, leaving a plain compound statement, so a following else is a syntax error. Code placed after the block is not a fallback: it is compiled for, and runs on, every targeted architecture, including the matching one.

The older CUBLASDX_SKIP_IF_NOT_APPLICABLE_SM(GEMM) macro is still available and takes a trailing semicolon instead of a block:

template<typename GEMM>
__global__ void user_kernel(...) {
  CUBLASDX_SKIP_IF_NOT_APPLICABLE_SM(GEMM);

  // rest of the kernel code ...
}

It expands to an early return, so it skips code generation but not instantiation: the statements following it are still instantiated during the device pass of every targeted architecture. A translation unit that instantiates descriptors for several architectures at once - for example SM<900> and SM<1200> - can therefore fail to compile on the architecture checks of the other architecture’s internals. Prefer the _BLOCK form in that case.

EnableInputStreaming Operator#

cublasdx::EnableInputStreaming()

Informs the library that no load transforms will be applied to inputs, thus allowing streaming of input data from shared memory directly to MMA units without passing through registers.

cuBLASDx can apply non-destructive in-flight per-element transforms to each element of the A and B inputs, just before they are passed to actual multiplication. This requires all elements to pass through register memory, which prevents the use of WGMMA and UTCMMA instructions (both 1SM UTCMMA and cluster-enabled 2SM UTCMMA).

The use of this operator limits preprocessing capabilities but unlocks potentially greater performance of data movement by letting the library handle it fully.

When this operator is present, pipelines require the global A and B tensor value types to match the descriptor’s Precision<> and Type<> exactly (BLAS::a_value_type / BLAS::b_value_type); other types are rejected at compile time.


TRSM-Specific Operators#

The following operators are used exclusively with Function<function::TRSM>.

Side Operator#

cublasdx::Side<cublasdx::side S>()

namespace cublasdx {
  enum class side
  {
    left,   // Solve A * X = B
    right,  // Solve X * A = B
  };
}

Specifies on which side of the equation the triangular matrix A appears.

  • side::left — solves \(A \times X = B\).

  • side::right — solves \(X \times A = B\).

FillMode Operator#

cublasdx::FillMode<cublasdx::fill_mode FM>()

namespace cublasdx {
  enum class fill_mode
  {
    lower,  // Lower triangle of A is stored
    upper,  // Upper triangle of A is stored
  };
}

Specifies which triangular part of matrix A holds the meaningful data. The other triangle is not accessed.

Diag Operator#

cublasdx::Diag<cublasdx::diag D>()

namespace cublasdx {
  enum class diag
  {
    non_unit,  // Diagonal of A is read from the matrix
    unit,      // Diagonal of A is implicitly 1; diagonal elements are not accessed
  };
}

Specifies whether the diagonal elements of A are explicitly stored (non_unit) or treated as ones without reading them (unit).

BatchesPerBlock Operator#

cublasdx::BatchesPerBlock<unsigned int N>()

Controls how many independent TRSM problem instances (batches) a single CUDA block processes. When omitted, defaults to 1. Use BLAS::suggested_batches_per_block to query the library-recommended value for your descriptor and tune accordingly.

When N > 1, the shared memory tensors passed to execute() must carry a batch mode as their third index. Use BLAS::get_layout_smem_a() and BLAS::get_layout_smem_b() to obtain layouts that include the batch stride.

Traits

  • BLAS::batches_per_block — the selected (or default) value of N.

  • BLAS::suggested_batches_per_block — library-recommended value for this descriptor.


Execution Operators#

Execution operators configure how the function will run on the GPU. Combined with Description Operators, they form a complete function descriptor that can be executed on a GPU.

Operator

Description

Block

Creates block execution object. See Block Configuration Operators.

Thread

Creates thread execution object (TRSM only).

Block Operator#

cublasdx::Block()

Generates a collective operation to run in a single CUDA block. Threads will cooperate to compute the collective operation. The layout and the number of threads participating in the execution, can be configured using Block Configuration Operators.

For example, the following code example creates a function descriptor for GEMM function that will run in a single CUDA block:

#include <cublasdx.hpp>

using GEMM = decltype(cublasdx::Size<32, 32, 64>()
              + cublasdx::Precision<double, __half, double>()
              + cublasdx::Type<cublasdx::type::real>()
              + cublasdx::Arrangement<cublasdx::row_major, cublasdx::col_major>()
              + cublasdx::Function<cublasdx::function::MM>()
              + cublasdx::SM<890>()
              + cublasdx::Block());

Thread Operator#

cublasdx::Thread()

Generates an independent operation to run in a single CUDA thread. Each thread solves its own problem instance independently, without cooperation with other threads. Currently only supported for TRSM.

For example, the following code example creates a function descriptor for TRSM function that will run independently per thread:

#include <cublasdx.hpp>

using TRSM = decltype(cublasdx::Size<32, 16>()
              + cublasdx::Precision<float>()
              + cublasdx::Type<cublasdx::type::real>()
              + cublasdx::Function<cublasdx::function::TRSM>()
              + cublasdx::Side<cublasdx::side::left>()
              + cublasdx::FillMode<cublasdx::fill_mode::lower>()
              + cublasdx::Diag<cublasdx::diag::non_unit>()
              + cublasdx::SM<900>()
              + cublasdx::Thread());

Block Configuration Operators#

Block-configuration operators allow the user to configure block size of a single CUDA block.

Operators

Default value

Description

BlockDim<X, Y, Z>

Based on heuristics

Number of threads used to perform BLAS function.

Note

Block configuration operators can only be used with Block Operator.

Warning

It is not guaranteed that executions of exactly the same BLAS function with exactly the same inputs but with different

will produce bit-identical results.

BlockDim Operator#

struct cublasdx::BlockDim<unsigned int X, unsigned int Y, unsigned int Z>()

Sets the CUDA block size to (X, Y, Z) to configure the execution, meaning it sets number of threads participating in the execution and their layout. Using this operator, user can run the BLAS function in 1D, 2D or 3D block with different number of threads. Set block dimension can be accessed via BLAS::block_dim trait.

Adding BlockDim<X, Y, Z> to the description puts the following requirements on the execution of the BLAS function:

For GEMM:

  • Kernel must be launched with 3D block dimensions dim3(X1, Y1, Z1) where X1 >= X, Y1 >= Y, and Z1 >= Z, also:

    • For 1D BlockDim<X> kernel must be launched with dim3(X1, Y1, Z1) where X1 >= X.

    • For 2D BlockDim<X, Y> kernel must be launched with dim3(X, Y1, Z1) where Y1 >= Y.

    • For 3D BlockDim<X, Y, Z> kernel must be launched with dim3(X, Y, Z1) where Z1 >= Z.

  • X * Y * Z threads must be participating in the execution.

  • The participating threads must be consecutive (adjacent) threads.

The listed requirements may be lifted or loosened in the future releases of cuBLASDx.

For TRSM:

  • The kernel must be launched with exactly the block dimensions BLAS::block_dim, which equals BlockDim<X, Y, Z> if the BlockDim operator is specified, or BLAS::suggested_block_dim if not. Unlike GEMM, TRSM does not allow launching with block dimensions other than specified.

Note

cuBLASDx can’t validate all kernel launch configuration at runtime and check that all requirements are met, thus it is user responsibility to adhere to the rules listed above. Violating those rules is considered undefined behavior and can lead to incorrect results and/or failures.

Examples

BlockDim<64>, kernel launched with block dimensions dim3(128, 1, 1) - OK
BlockDim<64>, kernel launched with block dimensions dim3(64, 4, 1) - OK
BlockDim<64>, kernel launched with block dimensions dim3(64, 2, 2) - OK
BlockDim<16, 16>, kernel launched with block dimensions dim3(16, 32, 1) - OK
BlockDim<16, 16>, kernel launched with block dimensions dim3(16, 16, 2) - OK
BlockDim<8, 8, 8>, kernel launched with block dimensions dim3(8, 8, 16) - OK

BlockDim<64>, kernel launched with block dimensions dim3(32, 1, 1) - INCORRECT
BlockDim<64>, kernel launched with block dimensions dim3(32, 2, 1) - INCORRECT
BlockDim<16, 16>, kernel launched with block dimensions dim3(256, 1, 1) - INCORRECT
BlockDim<8, 8, 8>, kernel launched with block dimensions dim3(512, 2, 1) - INCORRECT

A GEMM may use fewer threads than BlockDim (for example a matrix-vector product, which never needs more threads than it has outputs; this does not apply to complex integer types, which keep all BlockDim threads in the GEMM). The remaining threads are inactive: they do not take part in the GEMM (see is_thread_active() of the register accumulator). The StaticBlockDim requirements still apply.

The value of BlockDim can be accessed from BLAS description via BLAS::block_dim trait. When BlockDim is not set, the default block dimensions are used (the default value is BLAS::suggested_block_dim).

If the default block dimensions provided by cuBLASDx is smaller than the ones optimal for a kernel, it may still be optimal to try the default before increasing the number of threads contributing to the calculations.

Restrictions

  • X * Y * Z must be greater than or equal to 32.

Note

  • It’s recommended that X * Y * Z is 32, 64, 128, 256, 512, or 1024.

  • It’s recommended that X * Y * Z is a multiple of 32.

StaticBlockDim Operator#

struct cublasdx::StaticBlockDim()

Informs the library that the runtime configuration of CUDA block will be exactly the same as BlockDim passed to the descriptor.

This may slightly improve performance, as no extra runtime checks are necessary.

WithPipeline Operator#

cublasdx::WithPipeline()

Informs the library that this tile matrix multiplication description will be used only with cublasdx Pipelining extension. A WithPipeline descriptor is executed through suggest_pipeline / make_pipeline; the kernel must provide a dynamic shared memory buffer of buffer_size() bytes aligned to buffer_alignment() and pass it to device_pipeline.get_tile(smem, ...), see Pipeline Creation Result.

Warning

It is incorrect to use WithPipeline operator and execute GEMM with regular tile interface.

For example, the following code example creates a function descriptor for GEMM function that will run in a pipeline:

#include <cublasdx.hpp>

using GEMM = decltype(cublasdx::Size<32, 32, 64>()
                      + cublasdx::Precision<double, __half, double>()
                      + cublasdx::Type<cublasdx::type::real>()
                      + cublasdx::Arrangement<cublasdx::row_major, cublasdx::col_major>()
                      + cublasdx::Function<cublasdx::function::MM>()
                      + cublasdx::SM<890>()
                      + cublasdx::Block()
                      + cublasdx::WithPipeline());

BlocksPerCluster Operator#

cublasdx::BlocksPerCluster<unsigned int M, unsigned int N = 1, unsigned int K = 1>()

Asks the library to run the tile in a CUDA thread-block cluster of M x N x K CTAs. N and K default to 1. K must be 1; any other value fails a compile-time assertion because clustering along K is not portable across architectures that support cluster execution. Thus, BlocksPerCluster<2>, BlocksPerCluster<2, 1>, and BlocksPerCluster<2, 1, 1> are equivalent.

The extents are in CTAs, and cluster mates are expected to be neighbouring output tiles: mates along M share the same n, and mates along N share the same m. The level of operation remains Block() and Size<> stays per-block. The cluster must not exceed 16 CTAs (M * N <= 16); larger extents fail to compile.

Unlike cuSolverDx’s Cluster() execution operator, this does not change the execution level or make a matrix operation collective across the whole cluster. Each cuBLASDx CTA still computes its own Size<> output tile. The cluster is an optional pipeline optimization for sharing operand loads and, where supported, for issuing a two-CTA MMA atom.

The cluster layout is part of the contract and is as simple as possible: CTAs are laid out column-major, with blockIdx.x running over the M extent (M-fastest) and blockIdx.y over N. A kernel can therefore decode its position inside the cluster as rank_m = blockIdx.x % M and rank_n = blockIdx.y % N, and a grid scheduler that hands every cluster a contiguous M x N block of output tiles satisfies all of the library’s alignment expectations.

Clusters can enable hardware optimizations such as TMA multicast and 2SM UTCMMA on supported architectures, alongside the existing 1SM UTCMMA support. The operator is a hint; inspect the resulting pipeline’s uses_cluster member to determine whether clustered execution is active.

Warning

BlocksPerCluster<> only has an effect on a Function<function::MM> descriptor that also contains WithPipeline. Without WithPipeline, or for a non-GEMM function, the operator remains available for introspection but is otherwise ignored. In those cases has_blocks_per_cluster_v<BLAS> is true and the blocks_per_cluster_of_v_* traits report the specified extents, while uses_blocks_per_cluster_v<BLAS> is false.

Explicitly specifying BlocksPerCluster<1> (equivalent to BlocksPerCluster<1, 1, 1>) also does not enable clustered execution because a single CTA is not a cluster. The operator is present, so has_blocks_per_cluster_v<BLAS> is nevertheless true, while uses_blocks_per_cluster_v<BLAS> is false. When BlocksPerCluster<> is absent, the cluster extents default to 1 x 1 x 1 and has_blocks_per_cluster_v<BLAS> is false.

When clustered execution is active, BlocksPerCluster<M, N> requires the global per-CTA tile grid to contain only complete clusters; that is, (global_m / SizeM) must be divisible by M and (global_n / SizeN) must be divisible by N. The K extent must remain 1: it is not a batch or split-K dimension.

A pipeline that reports clustered execution must be launched with its matching cluster dimension. Otherwise, launch it without a cluster dimension.

Clustered execution is driven through suggest_pipeline / make_pipeline. If WithPipeline is omitted, BlocksPerCluster<> has no effect and regular BLAS::execute(tensor_a, tensor_b, accumulator) uses the same backend it would use without BlocksPerCluster<>.

For example:

#include <cublasdx.hpp>

using GEMM = decltype(cublasdx::Size<128, 256, 64>()
                      + cublasdx::Precision<__half, __half, float>()
                      + cublasdx::Type<cublasdx::type::real>()
                      + cublasdx::Function<cublasdx::function::MM>()
                      + cublasdx::Arrangement<cublasdx::row_major, cublasdx::col_major>()
                      + cublasdx::SM<1000, cublasdx::arch_specific>()
                      + cublasdx::Block()
                      + cublasdx::BlockDim<128>()
                      + cublasdx::Alignment<16, 16, 16>()
                      + cublasdx::EnableInputStreaming()
                      + cublasdx::WithPipeline()
                      + cublasdx::BlocksPerCluster<2, 1>());