cuSPARSELt Data Types

Opaque Data Structures

cusparseLtHandle_t

The structure holds the cuSPARSELt library context (device properties, system information, etc.).

The handle must be initialized and destroyed with cusparseLtInit() and cusparseLtDestroy() functions respectively.

cusparseLtMatDescriptor_t

The structure captures the shape and characteristics of a matrix.

It is initialized with cusparseLtDenseDescriptorInit() or cusparseLtStructuredDescriptorInit() functions and destroyed with cusparseLtMatDescriptorDestroy().

cusparseLtMatmulDescriptor_t

The structure holds the description of the matrix multiplication operation.

It is initialized with cusparseLtMatmulDescriptorInit() function.

cusparseLtMatmulAlgSelection_t

The structure holds the description of the matrix multiplication algorithm.

It is initialized with cusparseLtMatmulAlgSelectionInit() function.

cusparseLtMatmulPlan_t

The structure holds the matrix multiplication execution plan, namely all the information necessary to execute the cusparseLtMatmul() operation.

It is initialized and destroyed with cusparseLtMatmulPlanInit() and cusparseLtMatmulPlanDestroy() functions respectively.

---

Enumerators

cusparseLtSparsity_t

The enumerator specifies the sparsity ratio of the structured matrix as

Value	Description
CUSPARSELT_SPARSITY_50_PERCENT	50% Sparsity Ratio: - 2:4 for half, bfloat16, int, int8, e4m3, e5m2 - 1:2 for float

The sparsity property is used in the cusparseLtStructuredDescriptorInit() function.

---

cusparseComputeType

The enumerator specifies the compute precision modes of the matrix

Value	Description
CUSPARSE_COMPUTE_32I	- Element-wise multiplication of matrix A and B, and accumulation of the intermediate values are performed with 32-bit integer precision. - Alpha and beta coefficients, and epilogue are performed with single precision floating-point. - Tensor Cores will be used whenever possible.
CUSPARSE_COMPUTE_32F	- Element-wise multiplication of matrix A and B, and accumulation of the intermediate values are performed with single precision floating-point. - Alpha and beta coefficients, and epilogue are performed with single precision floating-point. - Tensor Cores will be used whenever possible.
CUSPARSE_COMPUTE_16F	- Element-wise multiplication of matrix A and B, and accumulation of the intermediate values are performed with half precision floating-point. - Alpha and beta coefficients, and epilogue are performed with single precision floating-point. - Tensor Cores will be used whenever possible.

The compute precision is used in the cusparseLtMatmulDescriptorInit() function.

---

cusparseLtMatDescAttribute_t

The enumerator specifies the additional attributes of a matrix descriptor

Value	Description
CUSPARSELT_MAT_NUM_BATCHES	Number of matrices in a batch
CUSPARSELT_MAT_BATCH_STRIDE	Stride between consecutive matrices in a batch expressed in terms of matrix elements

The algorithm enumerator is used in the cusparseLtMatDescSetAttribute() and cusparseLtMatDescGetAttribute() functions.

---

cusparseLtMatmulDescAttribute_t

The enumerator specifies the additional attributes of a matrix multiplication descriptor

Value	Type	Default Value	Description
CUSPARSELT_MATMUL_ACTIVATION_RELU	int 0: false, true otherwise	false	ReLU activation function
CUSPARSELT_MATMUL_ACTIVATION_RELU_UPPERBOUND	float	inf	Upper bound of the ReLU activation function
CUSPARSELT_MATMUL_ACTIVATION_RELU_THRESHOLD	float	0.0f	Lower threshold of the ReLU activation function
CUSPARSELT_MATMUL_ACTIVATION_GELU	int 0: false, true otherwise	false	Enable/Disable GeLU activation function
CUSPARSELT_MATMUL_ACTIVATION_GELU_SCALING	float	1.0f	Scaling coefficient for the GeLU activation function. It implies CUSPARSELT_MATMUL_ACTIVATION_GELU
CUSPARSELT_MATMUL_ALPHA_VECTOR_SCALING	int 0: false, true otherwise	false	Enable/Disable alpha vector (per-channel) scaling
CUSPARSELT_MATMUL_BETA_VECTOR_SCALING	int 0: false, true otherwise	false	Enable/Disable beta vector (per-channel) scaling
CUSPARSELT_MATMUL_BIAS_POINTER	void*	NULL (disabled)	Bias pointer. The bias vector size must equal to the number of rows of the output matrix (D)
CUSPARSELT_MATMUL_BIAS_STRIDE	int64_t	0 (disabled)	Bias stride between consecutive bias vectors. 0 means broadcast the first bias vector
CUSPARSELT_MATMUL_SPARSE_MAT_POINTER	void*	NULL (disabled)	Pointer to the prunned sparse matrix.

where the ReLU activation function is defined as:

• The data type of the bias vector is the same as the matric C except the following case:

  • INT8 input/output, INT32 Tensor Core compute kernels

  • INT8 input, INT32 output, INT32 Tensor Core compute kernels

  • INT8 input, FP16 output, INT32 Tensor Core compute kernels on pre-SM 9.0

  • INT8 input, BF16 output, INT32 Tensor Core compute kernels on pre-SM 9.0

in which the data type of the bias is FP32.

• The GeLU activation function is available only with

  • INT8 input/output, INT32 Tensor Core compute kernels

  • E4M3 input, E4M3 output, FP32 Tensor Core compute kernels

  • E4M3 input, BF16 output, FP32 Tensor Core compute kernels

  • E5M2 input, E5M2 output, FP32 Tensor Core compute kernels

  • E5M2 input, BF16 output, FP32 Tensor Core compute kernels

• CUSPARSELT_MATMUL_BETA_VECTOR_SCALING implies CUSPARSELT_MATMUL_ALPHA_VECTOR_SCALING

• CUSPARSELT_MATMUL_SPARSE_MAT_POINTER provides more flexibility for cusparseLtMatmulSearch() to select the best algorithm. The referenced memory cannot be modified until cusparseLtMatmulSearch() is called.

The algorithm enumerator is used in the cusparseLtMatmulDescSetAttribute() and cusparseLtMatmulDescGetAttribute() functions.

---

cusparseLtMatmulAlg_t

The enumerator specifies the algorithm for matrix-matrix multiplication

Value	Description
CUSPARSELT_MATMUL_ALG_DEFAULT	Default algorithm

The algorithm enumerator is used in the cusparseLtMatmulAlgSelectionInit() function.

---

cusparseLtMatmulAlgAttribute_t

The enumerator specifies the matrix multiplication algorithm attributes

Value	Description	Possible Values
CUSPARSELT_MATMUL_ALG_CONFIG_ID	Algorithm ID	[0, MAX) (see CUSPARSELT_MATMUL_ALG_CONFIG_MAX_ID)
CUSPARSELT_MATMUL_ALG_CONFIG_MAX_ID	Algorithm ID limit (query only)
CUSPARSELT_MATMUL_SEARCH_ITERATIONS	Number of iterations (kernel launches per algorithm) for cusparseLtMatmulSearch()	> 0 (default=5)
CUSPARSELT_MATMUL_SPLIT_K	Split-K factor (number of slices)	On pre-SM 9.0, [1, K], 1: Split-K disabled (default=not set); on SM 9.0, -1 (segment-K enabled) or 1 (segment-K disabled)
CUSPARSELT_MATMUL_SPLIT_K_MODE	Number of kernels for the Split-K algorithm	CUSPARSELT_SPLIT_K_MODE_ONE_KERNEL, CUSPARSELT_SPLIT_K_MODE_TWO_KERNELS
CUSPARSELT_MATMUL_SPLIT_K_BUFFERS	Device memory buffers to store partial results for the reduction	On pre-SM 9.0, [0, SplitK - 1]; on SM 9.0, 0

The algorithm attribute enumerator is used in the cusparseLtMatmulAlgGetAttribute() and cusparseLtMatmulAlgSetAttribute() functions.

Split-K parameters allow users to split the GEMM computation along the K dimension so that more CTAs will be created with a better SM utilization when N or M dimensions are small. However, this comes with the cost of reducing the operation of K slides to the final results. The cusparseLtMatmulSearch() function can be used to find the optimal combination of Split-K parameters.

Segment-K is a split-K method on SM 9.0 that utilizes warp-specialized persistent CTAs for enhanced efficiency and replaces the tranditional split-K method.

---

cusparseLtSplitKMode_t

The enumerator specifies the Split-K mode values corresponding to CUSPARSELT_MATMUL_SPLIT_K_MODE attribute in cusparseLtMatmulAlgAttribute_t

Value	Description	Details
CUSPARSELT_SPLIT_K_MODE_ONE_KERNEL	Use a single kernel for Split-K	Use the same GEMM kernel to do the final reduction
CUSPARSELT_SPLIT_K_MODE_TWO_KERNELS	Use two kernels for Split-K	Launch another GPU kernel to do the final reduction

---

cusparseLtPruneAlg_t

The enumerator specifies the pruning algorithm to apply to the structured matrix before the compression

Value	Description
CUSPARSELT_PRUNE_SPMMA_TILE	- half, bfloat16, int8, e4m3, e5m2: Zero-out eight values in a 4x4 tile to maximize the L1-norm of the resulting tile, under the constraint of selecting exactly two elements for each row and column  - float: Zero-out two values in a 2x2 tile to maximize the L1-norm of the resulting tile, under the constraint of selecting exactly one element for each row and column
CUSPARSELT_PRUNE_SPMMA_STRIP	- half, bfloat16, int8, e4m3, e5m2: Zero-out two values in a 1x4 strip to maximize the L1-norm of the resulting strip  - float: Zero-out one value in a 1x2 strip to maximize the L1-norm of the resulting strip  The strip direction is chosen according to the operation op and matrix layout applied to the structured (sparse) matrix

The pruning algorithm is used in the cusparseLtSpMMAPrune() function.