cuSPARSELt Functions¶
Library Management Functions¶
cusparseLtInit¶
cusparseStatus_t
cusparseLtInit(cusparseLtHandle_t* handle)
cusparseLtHandle_t) which holds the cuSPARSELt library context. It allocates light hardware resources on the host, and must be called prior to making any other cuSPARSELt library calls. Calling any cusparseLt function which uses cusparseLtHandle_t without a previous call of cusparseLtInit() will return an error.Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
OUT |
cuSPARSELt library handle |
See cusparseStatus_t for the description of the return status.
cusparseLtDestroy¶
cusparseStatus_t
cusparseLtDestroy(const cusparseLtHandle_t* handle)
cusparseLtHandle_t after cusparseLtDestroy() will return an error.Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
See cusparseStatus_t for the description of the return status.
Matrix Descriptor Functions¶
cusparseLtDenseDescriptorInit¶
cusparseStatus_t
cusparseLtDenseDescriptorInit(const cusparseLtHandle_t* handle,
cusparseLtMatDescriptor_t* matDescr,
int64_t rows,
int64_t cols,
int64_t ld,
uint32_t alignment,
cudaDataType valueType,
cusparseOrder_t order)
The function initializes the descriptor of a dense matrix.
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
Host |
OUT |
Dense matrix description |
|
|
Host |
IN |
Number of rows |
|
Host |
IN |
Number of columns |
|
Host |
IN |
Leading dimension |
|
Host |
IN |
Memory alignment in bytes |
Host |
IN |
Data type of the matrix |
|
Host |
IN |
Memory layout |
Constrains:
where
valueTypecan beCUDA_R_16F,CUDA_R_16BF,CUDA_R_I8,CUDA_R_32F.
rows,cols, andldmust be a multiple of
16 if
valueTypeisCUDA_R_I88 if
valueTypeisCUDA_R_16ForCUDA_R_16BF4 if
valueTypeisCUDA_R_32FThe total size of the matrix cannot exceed:
elements for
CUDA_R_8I
elements for
CUDA_R_16ForCUDA_R_16BF
elements for
CUDA_R_32F
See cusparseStatus_t for the description of the return status.
cusparseLtStructuredDescriptorInit¶
cusparseStatus_t
cusparseLtStructuredDescriptorInit(const cusparseLtHandle_t* handle,
cusparseLtMatDescriptor_t* matDescr,
int64_t rows,
int64_t cols,
int64_t ld,
uint32_t alignment,
cudaDataType valueType,
cusparseOrder_t order,
cusparseLtSparsity_t sparsity)
The function initializes the descriptor of a structured matrix.
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
Host |
OUT |
Dense matrix description |
|
|
Host |
IN |
Number of rows |
|
Host |
IN |
Number of columns |
|
Host |
IN |
Leading dimension |
|
Host |
IN |
Memory alignment in bytes |
Host |
IN |
Data type of the matrix |
|
Host |
IN |
Memory layout |
|
Host |
IN |
Matrix sparsity ratio |
Constrains:
where
valueTypecan beCUDA_R_16F,CUDA_R_16BF,CUDA_R_I8,CUDA_R_32F.
rows,cols, andldmust be a multiple of
16 if
valueTypeisCUDA_R_I88 if
valueTypeisCUDA_R_16ForCUDA_R_16BF4 if
valueTypeisCUDA_R_32FThe total size of the matrix cannot exceed:
elements for
CUDA_R_8I
elements for
CUDA_R_16ForCUDA_R_16BF
elements for
CUDA_R_32F
Sparsity ratio
Value |
Description |
|---|---|
|
50% Sparsity Ratio |
See cusparseStatus_t for the description of the return status.
cusparseLtMatDescriptorDestroy¶
cusparseStatus_t
cusparseLtMatDescriptorDestroy(const cusparseLtMatDescriptor_t* matDescr)
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
Matrix descriptor |
See cusparseStatus_t for the description of the return status.
Matmul Functions¶
cusparseLtMatmulDescriptorInit¶
cusparseStatus_t
cusparseLtMatmulDescriptorInit(const cusparseLtHandle_t* handle,
cusparseLtMatmulDescriptor_t* matMulDescr,
cusparseOperation_t opA,
cusparseOperation_t opB,
const cusparseLtMatDescriptor_t* matA,
const cusparseLtMatDescriptor_t* matB,
const cusparseLtMatDescriptor_t* matC,
const cusparseLtMatDescriptor_t* matD,
cusparseComputeType computeType)
The function initializes the matrix multiplication descriptor.
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
Host |
OUT |
Matrix multiplication descriptor |
|
Host |
IN |
Operation applied to the matrix |
|
Host |
IN |
Operation applied to the matrix |
|
Host |
IN |
Structured or dense matrix descriptor |
|
Host |
IN |
Structured or dense matrix descriptor |
|
Host |
IN |
Dense matrix descriptor |
|
Host |
IN |
Dense matrix descriptor |
|
Host |
IN |
Compute precision |
The structured matrix descriptor can used for matA or matB but not both.
Data types Supported:
Input |
Output |
Compute |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Constrains:
Given
Aof size,
Bof size, and
Cof size(regardless
opA,opB),must be a multiple of 32
CUDA_R_8Idata type only supports (the opposite ifBis structured):
opA/opB = TNif the matrix orders areorderA/orderB = Col/Col
opA/opB = NTif the matrix orders areorderA/orderB = Row/Row
opA/opB = NNif the matrix orders areorderA/orderB = Row/Col
opA/opB = TTif the matrix orders areorderA/orderB = Col/Row
See cusparseStatus_t for the description of the return status.
cusparseLtMatmulAlgSelectionInit¶
cusparseStatus_t
cusparseLtMatmulAlgSelectionInit(const cusparseLtHandle_t* handle,
cusparseLtMatmulAlgSelection_t* algSelection,
const cusparseLtMatmulDescriptor_t* matmulDescr,
cusparseLtMatmulAlg_t alg)
The function initializes the algorithm selection descriptor.
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
Host |
OUT |
Algorithm selection descriptor |
|
Host |
IN |
Matrix multiplication descriptor |
|
Host |
IN |
Algorithm mode |
See cusparseStatus_t for the description of the return status.
cusparseLtMatmulAlgSetAttribute¶
cusparseStatus_t
cusparseLtMatmulAlgSetAttribute(const cusparseLtHandle_t* handle,
cusparseLtMatmulAlgSelection_t* algSelection,
cusparseLtMatmulAlgAttribute_t attribute,
const void* data,
size_t dataSize)
The function sets the value of the specified attribute belonging to algorithm selection descriptor.
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
Host |
OUT |
Algorithm selection descriptor |
|
Host |
IN |
The attribute that will be set by this function |
|
|
Host |
IN |
Pointer to the value to which the specified attribute will be set |
|
Host |
IN |
Size in bytes of the attribute value used for verification |
See cusparseStatus_t for the description of the return status.
cusparseLtMatmulAlgGetAttribute¶
cusparseStatus_t
cusparseLtMatmulAlgGetAttribute(const cusparseLtHandle_t* handle,
const cusparseLtMatmulAlgSelection_t* algSelection,
cusparseLtMatmulAlgAttribute_t attribute,
void* data,
size_t dataSize)
The function returns the value of the queried attribute belonging to algorithm selection descriptor.
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
Host |
IN |
Algorithm selection descriptor |
|
Host |
IN |
The attribute that will be retrieved by this function |
|
|
Host |
OUT |
Memory address containing the attribute value retrieved by this function |
|
Host |
IN |
Size in bytes of the attribute value used for verification |
See cusparseStatus_t for the description of the return status.
cusparseLtMatmulGetWorkspace¶
cusparseStatus_t
cusparseLtMatmulGetWorkspace(const cusparseLtHandle_t* handle,
const cusparseLtMatmulAlgSelection_t* algSelection,
size_t* workspaceSize)
The function determines the required workspace size associated to the selected algorithm.
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
Host |
IN |
Algorithm selection descriptor |
|
|
Host |
OUT |
Workspace size in bytes |
See cusparseStatus_t for the description of the return status.
cusparseLtMatmulPlanInit¶
cusparseStatus_t
cusparseLtMatmulPlanInit(const cusparseLtHandle_t* handle,
cusparseLtMatmulPlan_t* plan,
const cusparseLtMatmulDescriptor_t* matmulDescr,
const cusparseLtMatmulAlgSelection_t* algSelection,
size_t workspaceSize)
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
Host |
OUT |
Matrix multiplication plan |
|
Host |
IN |
Matrix multiplication descriptor |
|
Host |
IN |
Algorithm selection descriptor |
|
|
Host |
IN |
Workspace size in bytes |
See cusparseStatus_t for the description of the return status.
cusparseLtMatmulPlanDestroy¶
cusparseStatus_t
cusparseLtMatmulPlanDestroy(const cusparseLtMatmulPlan_t* plan)
cusparseLtMatmulPlan_t after cusparseLtMatmulPlanDestroy() will return an error.Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
Matrix multiplication plan |
See cusparseStatus_t for the description of the return status.
cusparseLtMatmul¶
cusparseStatus_t
cusparseLtMatmul(const cusparseLtHandle_t* handle,
const cusparseLtMatmulPlan_t* plan,
const void* alpha,
const void* d_A,
const void* d_B,
const void* beta,
const void* d_C,
void* d_D,
void* workspace,
cudaStream_t* streams,
int32_t numStreams)
The function computes the matrix multiplication of matrices A and B to produce the the output matrix D, according to the following operation:
A, B, and C are input matrices, and D has the same shape of C
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
Host |
IN |
Matrix multiplication plan |
|
|
Host |
IN |
|
|
Device |
IN |
Pointer to the structured or dense matrix |
|
Device |
IN |
Pointer to the structured or dense matrix |
|
Host |
IN |
|
|
Device |
OUT |
Pointer to the dense matrix |
|
Device |
OUT |
Pointer to the dense matrix |
|
Device |
IN |
Pointer to workspace |
|
Host |
IN |
Pointer to CUDA stream array for the computation |
|
Host |
IN |
Number of CUDA streams in |
Data types Supported:
Input |
Output |
Compute |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
CUSPARSE_COMPUTE_TF32kernels perform the conversion from 32-bit IEEE754 floating-point to TensorFloat-32 by applying round toward plus infinity rounding mode before the computation.CUSPARSE_COMPUTE_TF32_FASTkernels suppose that the data are already represented in TensorFloat-32 (32-bit per value). If 32-bit IEEE754 floating-point are used as input, the values are truncated to TensorFloat-32 before the computation.CUSPARSE_COMPUTE_TF32_FASTkernels provide better performance thanCUSPARSE_COMPUTE_TF32but could produce less accurate results.
The structured matrix A or B (compressed) must respect the following constrains depending on the operation applied on it:
-
For
op = CUSPARSE_NON_TRANSPOSECUDA_R_16F,CUDA_R_16BF,CUDA_R_8Ieach row must have at least two non-zero values every four elementsCUDA_R_32Feach row must have at least one non-zero value every two elements
-
For
op = CUSPARSE_TRANSPOSECUDA_R_16F,CUDA_R_16BF,CUDA_R_8Ieach column must have at least two non-zero values every four elementsCUDA_R_32Feach column must have at least one non-zero value every two elements
The correctness of the pruning result (matrix A/B) can be check with the function cusparseLtSpMMAPruneCheck().
Constrains:
All pointers must be aligned to 16 bytes
Properties
The routine requires no extra storage
The routine supports asynchronous execution with respect to
streams[0]Provides deterministic (bit-wise) results for each run
cusparseLtMatmul supports the following optimizations:
CUDA graph capture
Hardware Memory Compression
See cusparseStatus_t for the description of the return status.
cusparseLtMatmulSearch¶
cusparseStatus_t
cusparseLtMatmulSearch(const cusparseLtHandle_t* handle,
cusparseLtMatmulPlan_t* plan,
const void* alpha,
const void* d_A,
const void* d_B,
const void* beta,
const void* d_C,
void* d_D,
void* workspace,
cudaStream_t* streams,
int32_t numStreams)
plan by selecting the fastest one. The functionality is intended to be used for auto-tuning purposes when the same operation is repeated multiple times over different inputs.
The function is NOT asynchronous with respect to
streams[0](blocking call)The number of iterations for the evaluation can be set by using cusparseLtMatmulAlgSetAttribute() with
CUSPARSELT_MATMUL_SEARCH_ITERATIONS.The selected algorithm id can be retrieved by using cusparseLtMatmulAlgGetAttribute() with
CUSPARSELT_MATMUL_ALG_CONFIG_ID.
Helper Functions¶
cusparseLtSpMMAPrune¶
cusparseStatus_t
cusparseLtSpMMAPrune(const cusparseLtHandle_t* handle,
const cusparseLtMatmulDescriptor_t* matmulDescr,
const void* d_in,
void* d_out,
cusparseLtPruneAlg_t pruneAlg,
cudaStream_t stream)
The function prunes a dense matrix d_in according to the specified algorithm pruneAlg.
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
Host |
IN |
Matrix multiplication descriptor |
|
|
Device |
IN |
Pointer to the dense matrix |
|
Device |
OUT |
Pointer to the pruned matrix |
Device |
IN |
Pruning algorithm |
|
|
Host |
IN |
CUDA stream for the computation |
Properties
The routine requires no extra storage
The routine supports asynchronous execution with respect to
streamProvides deterministic (bit-wise) results for each run
cusparseLtSpMMAPrune supports the following optimizations:
CUDA graph capture
Hardware Memory Compression
See cusparseStatus_t for the description of the return status.
cusparseLtSpMMAPrune2¶
cusparseStatus_t
cusparseLtSpMMAPrune2(const cusparseLtHandle_t* handle,
const cusparseLtMatDescriptor_t* sparseMatDescr,
int isSparseA,
cusparseOperation_t op,
const void* d_in,
void* d_out,
cusparseLtPruneAlg_t pruneAlg,
cudaStream_t stream);
The function prunes a dense matrix d_in according to the specified algorithm pruneAlg.
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
|
Host |
IN |
Specify if the structured (sparse) matrix is in the first position ( |
|
Host |
IN |
Operation that will be applied to the structured (sparse) matrix in the multiplication |
|
Device |
IN |
Pointer to the dense matrix |
|
Device |
OUT |
Pointer to the pruned matrix |
Device |
IN |
Pruning algorithm |
|
|
Host |
IN |
CUDA stream for the computation |
The function has the same properties of cusparseLtSpMMAPrune()
cusparseLtSpMMAPruneCheck¶
cusparseStatus_t
cusparseLtSpMMAPruneCheck(const cusparseLtHandle_t* handle,
const cusparseLtMatmulDescriptor_t* matmulDescr,
const void* d_in,
int* d_valid,
cudaStream_t stream)
The function checks the correctness of the pruning structure for a given matrix.
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
Host |
IN |
Matrix multiplication descriptor |
|
|
Device |
IN |
Pointer to the matrix to check |
|
Device |
OUT |
Validation results ( |
|
Host |
IN |
CUDA stream for the computation |
See cusparseStatus_t for the description of the return status.
cusparseLtSpMMAPruneCheck2¶
cusparseStatus_t
cusparseLtSpMMAPruneCheck2(const cusparseLtHandle_t* handle,
const cusparseLtMatDescriptor_t* sparseMatDescr,
int isSparseA,
cusparseOperation_t op,
const void* d_in,
int* d_valid,
cudaStream_t stream)
The function checks the correctness of the pruning structure for a given matrix.
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
|
Host |
IN |
Specify if the structured (sparse) matrix is in the first position ( |
|
Host |
IN |
Operation that will be applied to the structured (sparse) matrix in the multiplication |
|
Device |
IN |
Pointer to the matrix to check |
|
Device |
OUT |
Validation results ( |
|
Host |
IN |
CUDA stream for the computation |
The function has the same properties of cusparseLtSpMMAPruneCheck()
cusparseLtSpMMACompressedSize¶
cusparseStatus_t
cusparseLtSpMMACompressedSize(const cusparseLtHandle_t* handle,
const cusparseLtMatmulPlan_t* plan,
size_t* compressedSize)
The function provides the size of the compressed matrix to be allocated before calling cusparseLtSpMMACompress().
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
Host |
IN |
Matrix plan descriptor |
|
|
Host |
OUT |
Size in bytes of the compressed matrix |
See cusparseStatus_t for the description of the return status.
cusparseLtSpMMACompressedSize2¶
cusparseStatus_t
cusparseLtSpMMACompressedSize2(const cusparseLtHandle_t* handle,
const cusparseLtMatDescriptor_t* sparseMatDescr,
size_t* compressedSize)
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
|
Host |
IN |
Structured (sparse) matrix descriptor |
|
Host |
OUT |
Size in bytes of the compressed matrix |
The function has the same properties of cusparseLtSpMMACompressedSize()
cusparseLtSpMMACompress¶
cusparseStatus_t
cusparseLtSpMMACompress(const cusparseLtHandle_t* handle,
const cusparseLtMatmulPlan_t* plan,
const void* d_dense,
void* d_compressed,
cudaStream_t stream)
The function compresses a dense matrix d_dense. The compressed matrix is intended to be used as the first/second operand A/B in the cusparseLtMatmul() function.
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
Host |
IN |
Matrix multiplication plan |
|
|
Device |
IN |
Pointer to the dense matrix |
|
Device |
OUT |
Pointer to the compressed matrix |
|
Host |
IN |
CUDA stream for the computation |
Properties
The routine requires no extra storage
The routine supports asynchronous execution with respect to
streamProvides deterministic (bit-wise) results for each run
cusparseLtSpMMACompress supports the following optimizations:
CUDA graph capture
Hardware Memory Compression
See cusparseStatus_t for the description of the return status.
cusparseLtSpMMACompress2¶
cusparseStatus_t
cusparseLtSpMMACompress2(const cusparseLtHandle_t* handle,
const cusparseLtMatDescriptor_t* sparseMatDescr,
int isSparseA,
cusparseOperation_t op,
const void* d_dense,
void* d_compressed,
cudaStream_t stream)
Parameter |
Memory |
In/Out |
Description |
|---|---|---|---|
Host |
IN |
cuSPARSELt library handle |
|
|
Host |
IN |
Specify if the structured (sparse) matrix is in the first position ( |
|
Host |
IN |
Operation that will be applied to the structured (sparse) matrix in the multiplication |
|
Device |
IN |
Pointer to the dense matrix |
|
Device |
OUT |
Pointer to the compressed matrix |
|
Host |
IN |
CUDA stream for the computation |
The function has the same properties of cusparseLtSpMMACompress()