GEMM Execution Methods#

Execution methods are used to run the BLAS functions as defined by user with cuBLASDx operators.

Note

GEMM supports only execution on CUDA thread block level (block execution); thread-level execution is available for TRSM, see TRSM Thread Execute Methods.

Block Execute Method#

The block execution methods are available if the descriptor has been constructed using the Block Operator and is_complete_blas_execution Trait is true.

Shared Memory API#

Method execute(...) runs the calculations defined by the BLAS descriptor, accepting three types of arguments. It computes:

\(\mathbf{C}_{m\times n} = \alpha \cdot \mathbf{A}_{m\times k} \times \mathbf{B}_{k\times n} + \beta \cdot \mathbf{C}_{m\times n}\)

C is read only if beta is non-zero, and is overwritten in place; C must not alias A or B.

using BLAS = decltype(cublasdx::Size<M, N, K>() + ...);

// #1 - Tensor API

template<class Alpha,                         // Must be convertible to BLAS::c_value_type
         class AEngine, class ALayout,        // Types derived from pointer and layout used to create tensor_a
         class BEngine, class BLayout,        // Types derived from pointer and layout used to create tensor_b
         class Beta,                          // Must be convertible to BLAS::c_value_type
         class CEngine, class CLayout,        // Types derived from pointer and layout used to create tensor_c
         class ALoadOp = cublasdx::identity,  // Transform operation applied when data is loaded from matrix A
         class BLoadOp = cublasdx::identity,  // Transform operation applied when data is loaded from matrix B
         class CLoadOp = cublasdx::identity,  // Transform operation applied when data is loaded from matrix C
         class CStoreOp = cublasdx::identity> // Transform operation applied when data is stored to matrix C
inline __device__ void execute(const Alpha&                               alpha,
                               const cublasdx::tensor<AEngine, ALayout>&  tensor_a,
                               const cublasdx::tensor<BEngine, BLayout>&  tensor_b,
                               const Beta&                                beta,
                               cublasdx::tensor<CEngine, CLayout>&        tensor_c,
                               const ALoadOp&                             a_load_op  = {},
                               const BLoadOp&                             b_load_op  = {},
                               const CLoadOp&                             c_load_op  = {},
                               const CStoreOp&                            c_store_op = {})


// #2 - Pointer API
template<
  class Alpha, // Must be convertible to BLAS::c_value_type
  class TA,    // Value type of matrix A
  class TB,    // Value type of matrix B
  class Beta,  // Must be convertible to BLAS::c_value_type
  class TC,    // Value type of matrix C
  class ALoadTransformOp  = cublasdx::identity, // Transform operation applied when data is loaded from matrix A
  class BLoadTransformOp  = cublasdx::identity, // Transform operation applied when data is loaded from matrix B
  class CLoadTransformOp  = cublasdx::identity, // Transform operation applied when data is loaded from matrix C
  class CStoreTransformOp = cublasdx::identity> // Transform operation applied when data is stored to matrix C
 inline __device__ auto execute(const Alpha              alpha,
                                TA*                      matrix_a,
                                TB*                      matrix_b,
                                const Beta               beta,
                                TC*                      matrix_c,
                                const ALoadTransformOp&  a_load_op  = {},
                                const BLoadTransformOp&  b_load_op  = {},
                                const CLoadTransformOp&  c_load_op  = {},
                                const CStoreTransformOp& c_store_op = {})

// #3 - Pointer API, which allows providing runtime/dynamic leading dimensions for matrices A, B, and C
template<
  class Alpha,
  class TA,
  class LDA,
  class TB,
  class LDB,
  class Beta,
  class TC,
  class LDC,
  class ALoadTransformOp  = cublasdx::identity,
  class BLoadTransformOp  = cublasdx::identity,
  class CLoadTransformOp  = cublasdx::identity,
  class CStoreTransformOp = cublasdx::identity>
inline __device__ void execute(const Alpha&             alpha,
                               TA*                      matrix_a,
                               const LDA                lda,
                               TB*                      matrix_b,
                               const LDB                ldb,
                               const Beta&              beta,
                               TC*                      matrix_c,
                               const LDC                ldc,
                               const ALoadTransformOp&  a_load_op  = {},
                               const BLoadTransformOp&  b_load_op  = {},
                               const CLoadTransformOp&  c_load_op  = {},
                               const CStoreTransformOp& c_store_op = {})

Method #1 accepts cublasdx::tensor as representations of shared memory storage for matrices A, B, and C. cublasdx::tensor is essentially CuTe tensor (cute::Tensor), a representation of the multi-dimensional array, with rich functionality abstracting away the details of how the array’s elements are organized and stored in memory.

See cuBLASDx Tensor, Tensor Creation, and Get Memory Layout for how to create tensor using raw memory pointer and CuTe layout, and if needed, dynamically defined leading dimensions. If necessary, user can pass tensors with custom layouts.

In method #2 and #3 pointers matrix_a, matrix_b, matrix_c must point to shared memory regions aligned to BLAS::<a/b/c>_alignment. If Alignment operator was not used, BLAS::<a/b/c>_alignment is equal to alignof(BLAS::<a/b/c>_value_type).

Methods #2 and #3 assume the layout of each matrix corresponds to the arrangement set for that matrix, i.e. if Arrangement<col_major, row_major, col_major> was used in the BLAS description A matrix should be column-major, B matrix - row-major, and C matrix - column-major. The default arrangement corresponds to using Arrangement<row_major, col_major, col_major>.

Method #3 allows user to provide custom dynamic leading dimensions via lda, ldb, and ldc arguments. In this case, leading dimension values set via LeadingDimension operator are ignored. Values lda, ldb, and ldc have to follow the same rules as presented in LeadingDimension operator. Leading dimensions may be runtime integers or compile-time integral constants such as cute::Int<N>.

After the execution function user has to perform CUDA block synchronization before accessing A, B, or C.

Before calling execute(...), all threads must have finished writing A, B, and C to shared memory, i.e., user has to perform CUDA block synchronization (__syncthreads()) or call cublasdx::copy_wait() (which includes __syncthreads()) after the last write. All threads of the CUDA block (BlockDim) must call execute(...); calling it from only a subset of threads (e.g., in warp-specialized code) is not supported.

The code example below shows how the three execute(...) methods can be used.

#include <cublasdx.hpp>

using GEMM = decltype(cublasdx::Size<32, 32, 32>()
              + cublasdx::Precision<cublasdx::tfloat32_t, cublasdx::tfloat32_t, float>()
              + cublasdx::Type<cublasdx::type::real>()
              + cublasdx::Arrangement<cublasdx::row_major, cublasdx::col_major>()
              + cublasdx::Function<cublasdx::function::MM>()
              + cublasdx::MaxAlignment() // max alignment (16, 16, 16) is the default
              + cublasdx::SM<800>()
              + cublasdx::Block());

using a_data_type = typename GEMM::a_value_type;
using b_data_type = typename GEMM::b_value_type;
using c_data_type = typename GEMM::c_value_type;

extern __shared__ __align__(16) cublasdx::byte smem[];

// smem_<a/b/c> are aligned to cublasdx::alignment_of<GEMM>::<a/b/c>
auto [smem_a, smem_b, smem_c] = cublasdx::slice_shared_memory<GEMM>(smem);

//*********** Method #1, using cublasdx tensor APIs
{
    // Make global memory tensor
    auto a_global_tensor = cublasdx::make_tensor(a, GEMM::get_layout_gmem_a());
    auto b_global_tensor = cublasdx::make_tensor(b, GEMM::get_layout_gmem_b());
    auto c_global_tensor = cublasdx::make_tensor(c, GEMM::get_layout_gmem_c());

    // Make shared memory tensor
    auto a_shared_tensor = cublasdx::make_tensor(smem_a, GEMM::get_layout_smem_a());
    auto b_shared_tensor = cublasdx::make_tensor(smem_b, GEMM::get_layout_smem_b());
    auto c_shared_tensor = cublasdx::make_tensor(smem_c, GEMM::get_layout_smem_c());

    // Load data from global to shared memory using cublasdx::copy API
    using alignment = cublasdx::alignment_of<GEMM>;
    cublasdx::copy<GEMM, alignment::a>(a_global_tensor, a_shared_tensor);
    cublasdx::copy<GEMM, alignment::b>(b_global_tensor, b_shared_tensor);
    cublasdx::copy<GEMM, alignment::c>(c_global_tensor, c_shared_tensor);
    cublasdx::copy_wait();

    // Execute
    GEMM().execute(alpha, a_shared_tensor, b_shared_tensor, beta, c_shared_tensor);
    __syncthreads();

    // Store back to global memory using cublasdx::copy API
    cublasdx::copy<GEMM, alignment::c>(c_shared_tensor, c_global_tensor);
    cublasdx::copy_wait(); // Only needed if further operations in this kernel read c_global_tensor or reuse the shared memory of c_shared_tensor; otherwise the kernel's finalization is the final synchronization point.
}

//*********** Method #1, cublasdx tensor APIs, with dynamic leading dimensions
{
    // Make global memory tensor
    auto a_global_tensor = cublasdx::make_tensor(a, GEMM::get_layout_gmem_a(lda));
    auto b_global_tensor = cublasdx::make_tensor(b, GEMM::get_layout_gmem_b(ldb));
    auto c_global_tensor = cublasdx::make_tensor(c, GEMM::get_layout_gmem_c(ldc));

    // Make shared memory tensor
    auto a_shared_tensor = cublasdx::make_tensor(smem_a, GEMM::get_layout_smem_a(lda));
    auto b_shared_tensor = cublasdx::make_tensor(smem_b, GEMM::get_layout_smem_b(ldb));
    auto c_shared_tensor = cublasdx::make_tensor(smem_c, GEMM::get_layout_smem_c(ldc));

    // Load data from global to shared memory using cublasdx::copy API
    using alignment = cublasdx::alignment_of<GEMM>;
    cublasdx::copy<GEMM, alignment::a>(a_global_tensor, a_shared_tensor);
    cublasdx::copy<GEMM, alignment::b>(b_global_tensor, b_shared_tensor);
    cublasdx::copy<GEMM, alignment::c>(c_global_tensor, c_shared_tensor);
    cublasdx::copy_wait();

    // Execute
    GEMM().execute(alpha, a_shared_tensor, b_shared_tensor, beta, c_shared_tensor);
    __syncthreads();

    // Store back to global memory using cublasdx::copy API
    cublasdx::copy<GEMM, alignment::c>(c_shared_tensor, c_global_tensor);
    cublasdx::copy_wait(); // Only needed if further operations in this kernel read c_global_tensor or reuse the shared memory of c_shared_tensor; otherwise the kernel's finalization is the final synchronization point.
}

//*********** Method #2, using raw shared memory pointers
{
    // User code to load data from global to shared memory
    // smem_a <-- a, smem_b <-- b, smem_c <-- c
    __syncthreads(); // all writes to shared memory must complete before execute

    // Execute
    GEMM().execute(alpha, smem_a, smem_b, beta, smem_c);
    __syncthreads();

    // User code to store back to global memory
    // smem_c --> c
}

//*********** Method #3, with dynamic leading dimensions
{
    // User code to load data from global to shared memory
    // smem_a <-- a, smem_b <-- b, smem_c <-- c
    __syncthreads(); // all writes to shared memory must complete before execute

    // Execute
    GEMM().execute(alpha, smem_a, lda, smem_b, ldb, beta, smem_c, ldc);
    __syncthreads();

    // User code to store back to global memory
    // smem_c --> c
}

Register API#

Method execute(...) runs the calculations defined by the BLAS descriptor, accepting two types of arguments.

using BLAS = decltype(cublasdx::Size<M, N, K>() + ...);

// #1 - Registers with accumulator API

template<class AEngine, class ALayout,       // Types derived from pointer and layout used to create tensor_a
         class BEngine, class BLayout,       // Types derived from pointer and layout used to create tensor_b
         class BlasAccumulator,              // Object created from get_accumulator / suggest_accumulator
         class ALoadOp = cublasdx::identity, // Transform operation applied when data is loaded from matrix A
         class BLoadOp = cublasdx::identity> // Transform operation applied when data is loaded from matrix B
inline __device__ void execute(const cublasdx::tensor<AEngine, ALayout>&  tensor_a,
                               const cublasdx::tensor<BEngine, BLayout>&  tensor_b,
                               BlasAccumulator                         && blas_accumulator,
                               const ALoadOp&                             a_load_op  = {},
                               const BLoadOp&                             b_load_op  = {})

// #2 - Registers without accumulator API

template<class AEngine, class ALayout,
         class BEngine, class BLayout,
         class ALoadOp = cublasdx::identity,
         class BLoadOp = cublasdx::identity>
inline __device__ auto execute(const cublasdx::tensor<AEngine, ALayout>&  tensor_a,
                               const cublasdx::tensor<BEngine, BLayout>&  tensor_b,
                               const ALoadOp&                             a_load_op  = {},
                               const BLoadOp&                             b_load_op  = {})

Method #1 accepts shared memory tensors for A and B matrices, and an accumulator. It returns nothing, as it accumulates the result of multiplying A with B into the passed accumulator, resulting in:

\(\mathbf{C}_{m\times n} \mathrel{+}= \mathbf{A}_{m\times k} \times \mathbf{B}_{k\times n}\)

A freshly created accumulator is zero-initialised, so the first call yields \(\mathbf{C} = \mathbf{A} \times \mathbf{B}\); passing the same accumulator to further calls keeps accumulating. The accumulator must exist beforehand, either from a previous execution of method #2, or created from a BLAS object (see Data Accumulator). It must match exactly the precision and partitioning of GEMM which it is used for.

Method #2 accepts only shared memory tensors for A and B matrices. It returns an accumulator. The results correspond to the result of:

\(\mathbf{C}_{m\times n} = \mathbf{A}_{m\times k} \times \mathbf{B}_{k\times n}\)

execute(...) returns once the calling thread has finished its part; other threads may still be reading A and B. Call __syncthreads() before any thread overwrites the shared memory A or B tensors.

The code example below shows how the two execute(...) methods can be used.

#include <cublasdx.hpp>

using GEMM = decltype(cublasdx::Size<32, 32, 32>()
              + cublasdx::Precision<cublasdx::tfloat32_t, cublasdx::tfloat32_t, float>()
              + cublasdx::Type<cublasdx::type::real>()
              + cublasdx::Arrangement<cublasdx::row_major, cublasdx::col_major>()
              + cublasdx::Function<cublasdx::function::MM>()
              + cublasdx::MaxAlignment() // max alignment (16, 16, 16) is the default
              + cublasdx::SM<800>()
              + cublasdx::Block());

using a_data_type = typename GEMM::a_value_type;
using b_data_type = typename GEMM::b_value_type;
using c_data_type = typename GEMM::c_value_type;

extern __shared__ __align__(16) cublasdx::byte smem[];

// smem_<a/b> are aligned to cublasdx::alignment_of<GEMM>::<a/b>
auto [smem_a, smem_b] = cublasdx::slice_shared_memory_ab<GEMM>(smem);

//*********** Method #1, register API with accumulator
{
    // Make global memory tensor
    auto a_global_tensor = cublasdx::make_tensor(a, GEMM::get_layout_gmem_a());
    auto b_global_tensor = cublasdx::make_tensor(b, GEMM::get_layout_gmem_b());
    auto c_global_tensor = cublasdx::make_tensor(c, GEMM::get_layout_gmem_c());

    // Make shared memory tensor
    auto a_shared_tensor = cublasdx::make_tensor(smem_a, GEMM::get_layout_smem_a());
    auto b_shared_tensor = cublasdx::make_tensor(smem_b, GEMM::get_layout_smem_b());

    // Load data from global to shared memory using cublasdx::copy API
    using alignment = cublasdx::alignment_of<GEMM>;
    cublasdx::copy<GEMM, alignment::a>(a_global_tensor, a_shared_tensor);
    cublasdx::copy<GEMM, alignment::b>(b_global_tensor, b_shared_tensor);
    cublasdx::copy_wait();

    // Execute
    auto accumulator = GEMM::get_accumulator();

    GEMM().execute(a_shared_tensor, b_shared_tensor, accumulator);

    // Store back to global memory using accumulator.partition_and_store
    accumulator.partition_and_store(c_global_tensor);
}

//*********** Method #2, register API without accumulator
{
    // Make global memory tensor
    auto a_global_tensor = cublasdx::make_tensor(a, GEMM::get_layout_gmem_a());
    auto b_global_tensor = cublasdx::make_tensor(b, GEMM::get_layout_gmem_b());
    auto c_global_tensor = cublasdx::make_tensor(c, GEMM::get_layout_gmem_c());

    // Make shared memory tensor
    auto a_shared_tensor = cublasdx::make_tensor(smem_a, GEMM::get_layout_smem_a());
    auto b_shared_tensor = cublasdx::make_tensor(smem_b, GEMM::get_layout_smem_b());

    // Load data from global to shared memory using cublasdx::copy API
    using alignment = cublasdx::alignment_of<GEMM>;
    cublasdx::copy<GEMM, alignment::a>(a_global_tensor, a_shared_tensor);
    cublasdx::copy<GEMM, alignment::b>(b_global_tensor, b_shared_tensor);
    cublasdx::copy_wait();

    // Execute
    auto accumulator = GEMM().execute(a_shared_tensor, b_shared_tensor);
    // alternatively
    // auto accumulator = GEMM::get_accumulator();
    // GEMM().execute(a_shared_tensor, b_shared_tensor, accumulator);

    // Store back to global memory using accumulator.partition_and_store
    accumulator.partition_and_store(c_global_tensor);
}

Input Data Properties#

Note

Starting from cuBLASDx 0.3.0, computational precision has been decoupled from data precision, i.e. the input / output data for each matrix can be of arbitrary type (even integral input for floating point GEMM) provided that Alignment Operator is set and at least one of those conditions is met:

  1. It’s implicitly convertible to the data type chosen with Precision Operator and Type Operator.

  2. For inputs: An appropriate converting loading operation is provided as one of the arguments. It takes the input type value. Its result must be at least implicitly convertible to the compute type.

  3. For output: An appropriate converting storing operation is provided as one of the arguments. It takes the result computational type (usually C type as defined by Precision Operator and Type Operator). Its result must be at least implicitly convertible to the output type.

Warning

If using computation precision decoupled from input types, the Alignment Operator must be explicitly set.

The underlying element types of scalars (alpha and beta) by default are assumed to be BLAS::c_value_type, but they can be any types convertible to it. The scalars are converted by value to the computation type of C (for cublasdx::axpby(alpha, x, beta, y): alpha to the type of x, beta to the type of y), with the usual conversion rules:

  • a real scalar can be used with a complex C, it is taken as (scalar, 0),

  • a floating-point scalar for an integral C is truncated toward zero, for example alpha = 2.7f scales by 2,

  • a scalar wider than C is rounded to the precision of C; a double scalar for a half or bfloat16 C is converted through float, so it can be double-rounded: one ulp away from __double2half(alpha) or __double2bfloat16(alpha), and just below the largest finite value it can overflow (65520 - 2^-20 becomes inf in half, where __double2half gives 65504). Pass a BLAS::c_value_type scalar to control the rounding,

  • an integer scalar for a bfloat16 or tfloat32_t value (with cublasdx::axpby on such tensors) is not rounded to nearest: it is first rounded to nearest float and then truncated toward zero to the target precision. Integers with |x| <= 2^24 are exact in float, so they are simply truncated (259 becomes 258 in bfloat16, 4099 becomes 4096 in tfloat32_t); larger int and 64-bit integer values are double-rounded (2^24 + 2^16 + 1 becomes 2^24 in bfloat16, where __ll2bfloat16_rn gives 2^24 + 2^17). Convert such scalars to the target type yourself to control the rounding.

These rules are the same for the register, emulation and TMEM accumulators.

The same value type in another representation (__half for half) is reinterpreted bitwise. Every other scalar is converted by value as described above; a type whose constructor the computation type accepts also converts (float2 / cuFloatComplex become a complex<float> through its constructor, with the same components). A scalar with no value conversion to the computation type (for example __nv_bfloat16 for a half C, or __half2 for a complex<half> C) is rejected at compile time with a descriptive static_assert; convert such scalars explicitly. The same rule gates execute()’s alpha/beta overloads, so all entry points accept the same scalar types.

Transform Operation Inputs#

All GEMM methods accept transform functors. a_load_op, b_load_op, c_load_op are applied as elements are read from each matrix, and c_store_op is applied before the results of matrix multiplication are stored in the C matrix. Each functor has to represent an element-wise transform which:

  1. For load transformations: accepts respective input type to execute(...) method and returns value of type implicitly convertible to BLAS::<a/b/c>_value_type.

  2. For store transformations: accepts BLAS::c_value_type and returns respective input type to execute(...) method.

  3. Functors must be stateless (empty class types, e.g. capture-less lambdas or structs without data members); the library default-constructs the functor type and does not use the passed object, so function pointers and capturing lambdas are rejected.

When beta is zero, C is not read (it may be uninitialized; its contents, including NaNs, do not affect the result), so c_load_op has no effect; c_store_op is always applied.

cuBLASDx provides two built-in stateless functors; see cublasdx::identity:

  • cublasdx::identity — passes the value through unchanged (default for all positions).

  • cublasdx::conjugate — applies complex conjugation; is a no-op for real types.

Example

using GEMM = decltype(Size<128, 128, 128>() + Type<type::real>() + Precision<float, float, double>() + Block() + ...);

struct multiply_by_2 {
  template<class T>
  __device__ constexpr T operator()(const T arg) const {
    return arg * static_cast<T>(2.0f);
  }
};

struct negate {
  template <class T>
  __device__ constexpr T operator()(const T arg) const {
    return -arg;
  }
};

GEMM().execute(..., multiply_by_2{}, cublasdx::conjugate{}, cublasdx::identity{}, negate{});

Warning

It is not guaranteed that executions of exactly the same BLAS function with exactly the same inputs but with different

will produce bit-identical results.

Warning

It is not guaranteed that executions of exactly the same BLAS function with exactly the same inputs on GPUs of different CUDA architectures will produce bit-identical results.