GEMM Execution Methods#
Execution methods are used to run the BLAS functions as defined by user with cuBLASDx operators.
Note
GEMM supports only execution on CUDA thread block level (block execution); thread-level execution is available for TRSM, see TRSM Thread Execute Methods.
Block Execute Method#
The block execution methods are available if the descriptor has been constructed using the Block Operator
and is_complete_blas_execution Trait is true.
Register API#
Method execute(...) runs the calculations defined by the BLAS descriptor, accepting two types of arguments.
using BLAS = decltype(cublasdx::Size<M, N, K>() + ...);
// #1 - Registers with accumulator API
template<class AEngine, class ALayout, // Types derived from pointer and layout used to create tensor_a
class BEngine, class BLayout, // Types derived from pointer and layout used to create tensor_b
class BlasAccumulator, // Object created from get_accumulator / suggest_accumulator
class ALoadOp = cublasdx::identity, // Transform operation applied when data is loaded from matrix A
class BLoadOp = cublasdx::identity> // Transform operation applied when data is loaded from matrix B
inline __device__ void execute(const cublasdx::tensor<AEngine, ALayout>& tensor_a,
const cublasdx::tensor<BEngine, BLayout>& tensor_b,
BlasAccumulator && blas_accumulator,
const ALoadOp& a_load_op = {},
const BLoadOp& b_load_op = {})
// #2 - Registers without accumulator API
template<class AEngine, class ALayout,
class BEngine, class BLayout,
class ALoadOp = cublasdx::identity,
class BLoadOp = cublasdx::identity>
inline __device__ auto execute(const cublasdx::tensor<AEngine, ALayout>& tensor_a,
const cublasdx::tensor<BEngine, BLayout>& tensor_b,
const ALoadOp& a_load_op = {},
const BLoadOp& b_load_op = {})
Method #1 accepts shared memory tensors for A and B matrices, and an accumulator. It returns nothing, as it accumulates the result
of multiplying A with B into the passed accumulator, resulting in:
\(\mathbf{C}_{m\times n} \mathrel{+}= \mathbf{A}_{m\times k} \times \mathbf{B}_{k\times n}\)
A freshly created accumulator is zero-initialised, so the first call yields \(\mathbf{C} = \mathbf{A} \times \mathbf{B}\); passing the same accumulator to further calls keeps accumulating. The accumulator must exist beforehand, either from a previous execution of method #2, or created from a BLAS object (see Data Accumulator). It must match exactly the precision and partitioning of GEMM which it is used for.
Method #2 accepts only shared memory tensors for A and B matrices. It returns an accumulator. The results correspond to the result of:
\(\mathbf{C}_{m\times n} = \mathbf{A}_{m\times k} \times \mathbf{B}_{k\times n}\)
execute(...) returns once the calling thread has finished its part; other threads may still be reading A and B.
Call __syncthreads() before any thread overwrites the shared memory A or B tensors.
The code example below shows how the two execute(...) methods can be used.
#include <cublasdx.hpp>
using GEMM = decltype(cublasdx::Size<32, 32, 32>()
+ cublasdx::Precision<cublasdx::tfloat32_t, cublasdx::tfloat32_t, float>()
+ cublasdx::Type<cublasdx::type::real>()
+ cublasdx::Arrangement<cublasdx::row_major, cublasdx::col_major>()
+ cublasdx::Function<cublasdx::function::MM>()
+ cublasdx::MaxAlignment() // max alignment (16, 16, 16) is the default
+ cublasdx::SM<800>()
+ cublasdx::Block());
using a_data_type = typename GEMM::a_value_type;
using b_data_type = typename GEMM::b_value_type;
using c_data_type = typename GEMM::c_value_type;
extern __shared__ __align__(16) cublasdx::byte smem[];
// smem_<a/b> are aligned to cublasdx::alignment_of<GEMM>::<a/b>
auto [smem_a, smem_b] = cublasdx::slice_shared_memory_ab<GEMM>(smem);
//*********** Method #1, register API with accumulator
{
// Make global memory tensor
auto a_global_tensor = cublasdx::make_tensor(a, GEMM::get_layout_gmem_a());
auto b_global_tensor = cublasdx::make_tensor(b, GEMM::get_layout_gmem_b());
auto c_global_tensor = cublasdx::make_tensor(c, GEMM::get_layout_gmem_c());
// Make shared memory tensor
auto a_shared_tensor = cublasdx::make_tensor(smem_a, GEMM::get_layout_smem_a());
auto b_shared_tensor = cublasdx::make_tensor(smem_b, GEMM::get_layout_smem_b());
// Load data from global to shared memory using cublasdx::copy API
using alignment = cublasdx::alignment_of<GEMM>;
cublasdx::copy<GEMM, alignment::a>(a_global_tensor, a_shared_tensor);
cublasdx::copy<GEMM, alignment::b>(b_global_tensor, b_shared_tensor);
cublasdx::copy_wait();
// Execute
auto accumulator = GEMM::get_accumulator();
GEMM().execute(a_shared_tensor, b_shared_tensor, accumulator);
// Store back to global memory using accumulator.partition_and_store
accumulator.partition_and_store(c_global_tensor);
}
//*********** Method #2, register API without accumulator
{
// Make global memory tensor
auto a_global_tensor = cublasdx::make_tensor(a, GEMM::get_layout_gmem_a());
auto b_global_tensor = cublasdx::make_tensor(b, GEMM::get_layout_gmem_b());
auto c_global_tensor = cublasdx::make_tensor(c, GEMM::get_layout_gmem_c());
// Make shared memory tensor
auto a_shared_tensor = cublasdx::make_tensor(smem_a, GEMM::get_layout_smem_a());
auto b_shared_tensor = cublasdx::make_tensor(smem_b, GEMM::get_layout_smem_b());
// Load data from global to shared memory using cublasdx::copy API
using alignment = cublasdx::alignment_of<GEMM>;
cublasdx::copy<GEMM, alignment::a>(a_global_tensor, a_shared_tensor);
cublasdx::copy<GEMM, alignment::b>(b_global_tensor, b_shared_tensor);
cublasdx::copy_wait();
// Execute
auto accumulator = GEMM().execute(a_shared_tensor, b_shared_tensor);
// alternatively
// auto accumulator = GEMM::get_accumulator();
// GEMM().execute(a_shared_tensor, b_shared_tensor, accumulator);
// Store back to global memory using accumulator.partition_and_store
accumulator.partition_and_store(c_global_tensor);
}
Input Data Properties#
Note
Starting from cuBLASDx 0.3.0, computational precision has been decoupled from data precision, i.e. the input / output data for each matrix can be of arbitrary type (even integral input for floating point GEMM) provided that Alignment Operator is set and at least one of those conditions is met:
It’s implicitly convertible to the data type chosen with Precision Operator and Type Operator.
For inputs: An appropriate converting loading operation is provided as one of the arguments. It takes the input type value. Its result must be at least implicitly convertible to the compute type.
For output: An appropriate converting storing operation is provided as one of the arguments. It takes the result computational type (usually
Ctype as defined by Precision Operator and Type Operator). Its result must be at least implicitly convertible to the output type.
Warning
If using computation precision decoupled from input types, the Alignment Operator must be explicitly set.
The underlying element types of scalars (alpha and beta) by default are assumed to be BLAS::c_value_type, but
they can be any types convertible to it. The scalars are converted by value to the computation type of C (for
cublasdx::axpby(alpha, x, beta, y): alpha to the type of x, beta to the type of y), with the usual
conversion rules:
a real scalar can be used with a complex
C, it is taken as(scalar, 0),a floating-point scalar for an integral
Cis truncated toward zero, for examplealpha = 2.7fscales by2,a scalar wider than
Cis rounded to the precision ofC; adoublescalar for ahalforbfloat16Cis converted throughfloat, so it can be double-rounded: one ulp away from__double2half(alpha)or__double2bfloat16(alpha), and just below the largest finite value it can overflow (65520 - 2^-20becomesinfinhalf, where__double2halfgives65504). Pass aBLAS::c_value_typescalar to control the rounding,an integer scalar for a
bfloat16ortfloat32_tvalue (withcublasdx::axpbyon such tensors) is not rounded to nearest: it is first rounded to nearestfloatand then truncated toward zero to the target precision. Integers with|x| <= 2^24are exact infloat, so they are simply truncated (259becomes258inbfloat16,4099becomes4096intfloat32_t); largerintand 64-bit integer values are double-rounded (2^24 + 2^16 + 1becomes2^24inbfloat16, where__ll2bfloat16_rngives2^24 + 2^17). Convert such scalars to the target type yourself to control the rounding.
These rules are the same for the register, emulation and TMEM accumulators.
The same value type in another representation (__half for half) is reinterpreted bitwise. Every other scalar
is converted by value as described above; a type whose constructor the computation type accepts also converts
(float2 / cuFloatComplex become a complex<float> through its constructor, with the same components). A
scalar with no value conversion to the computation type (for example __nv_bfloat16 for a half C, or
__half2 for a complex<half> C) is rejected at compile time with a descriptive static_assert; convert
such scalars explicitly. The same rule gates execute()’s alpha/beta overloads, so all entry points accept the
same scalar types.
Transform Operation Inputs#
All GEMM methods accept transform functors. a_load_op, b_load_op, c_load_op are applied as elements are read from each matrix,
and c_store_op is applied before the results of matrix multiplication are stored in the C matrix. Each functor has to represent an element-wise
transform which:
For load transformations: accepts respective input type to
execute(...)method and returns value of type implicitly convertible toBLAS::<a/b/c>_value_type.For store transformations: accepts
BLAS::c_value_typeand returns respective input type toexecute(...)method.Functors must be stateless (empty class types, e.g. capture-less lambdas or structs without data members); the library default-constructs the functor type and does not use the passed object, so function pointers and capturing lambdas are rejected.
When beta is zero, C is not read (it may be uninitialized; its contents, including NaNs, do not affect the result),
so c_load_op has no effect; c_store_op is always applied.
cuBLASDx provides two built-in stateless functors; see cublasdx::identity:
cublasdx::identity— passes the value through unchanged (default for all positions).cublasdx::conjugate— applies complex conjugation; is a no-op for real types.
Example
using GEMM = decltype(Size<128, 128, 128>() + Type<type::real>() + Precision<float, float, double>() + Block() + ...);
struct multiply_by_2 {
template<class T>
__device__ constexpr T operator()(const T arg) const {
return arg * static_cast<T>(2.0f);
}
};
struct negate {
template <class T>
__device__ constexpr T operator()(const T arg) const {
return -arg;
}
};
GEMM().execute(..., multiply_by_2{}, cublasdx::conjugate{}, cublasdx::identity{}, negate{});
Warning
It is not guaranteed that executions of exactly the same BLAS function with exactly the same inputs but with different
leading dimensions (LeadingDimension),
CUDA architecture (SM), or
number of threads (BlockDim)
will produce bit-identical results.
Warning
It is not guaranteed that executions of exactly the same BLAS function with exactly the same inputs on GPUs of different CUDA architectures will produce bit-identical results.