cuSPARSELt Data Types

Opaque Data Structures

cusparseLtHandle_t

The structure holds the cuSPARSELt library context (device properties, system information, etc.).
The handle must be initialized and destroyed with cusparseLtInit() and cusparseLtDestroy() functions respectively.

cusparseLtMatDescriptor_t

The structure captures the shape and characteristics of a matrix.
It is initialized with cusparseLtDenseDescriptorInit() or cusparseLtStructuredDescriptorInit() functions and destroyed with cusparseLtMatDescriptorDestroy().

cusparseLtMatmulDescriptor_t

The structure holds the description of the matrix multiplication operation.
It is initialized with cusparseLtMatmulDescriptorInit() function.

cusparseLtMatmulAlgSelection_t

The structure holds the description of the matrix multiplication algorithm.
It is initialized with cusparseLtMatmulAlgSelectionInit() function.

cusparseLtMatmulPlan_t

The structure holds the matrix multiplication execution plan, namely all the information necessary to execute the cusparseLtMatmul() operation.
It is initialized and destroyed with cusparseLtMatmulPlanInit() and cusparseLtMatmulPlanDestroy() functions respectively.

Enumerators

cusparseLtSparsity_t

The enumerator specifies the sparsity ratio of the structured matrix as

sparsity\ ratio = \frac{nnz}{num\_rows * num\_cols}

Value

Description

CUSPARSELT_SPARSITY_50_PERCENT

50% Sparsity Ratio:

- 2:4 for half, bfloat16, int

- 1:2 for tf32 and float
The sparsity property is used in the cusparseLtStructuredDescriptorInit() function.

cusparseComputeType

The enumerator specifies the compute precision modes of the matrix

Value

Description

CUSPARSE_COMPUTE_16F

- Default mode for 16-bit floating-point precision

- All computations and intermediate storage ensure at least 16-bit precision

- Tensor Cores will be used whenever possible

CUSPARSE_COMPUTE_32I

- Default mode for 32-bit integer precision

- All computations and intermediate storage ensure at least 32-bit integer precision

- Tensor Cores will be used whenever possible

CUSPARSE_COMPUTE_TF32_FAST

- Default mode for 32-bit floating-point precision

- The inputs are supposed to be directly represented in TensorFloat-32 precision. The 32-bit floating-point values are truncated to TensorFloat-32 before the computation

- All computations and intermediate storage ensure at least TensorFloat-32 precision

- Tensor Cores will be used whenever possible

CUSPARSE_COMPUTE_TF32

- All computations and intermediate storage ensure at least TensorFloat-32 precision

- The inputs are rounded to TensorFloat-32 precision. This mode is slower than CUSPARSE_COMPUTE_TF32_FAST, but could provide more accurate results

- Tensor Cores will be used whenever possible
The compute precision is used in the cusparseLtMatmulDescriptorInit() function.

cusparseLtMatDescAttribute_t

The enumerator specifies the additional attributes of a matrix descriptor

Value

Description

CUSPARSELT_MAT_NUM_BATCHES

Number of matrices in a batch

CUSPARSELT_MAT_BATCH_STRIDE

Stride between consecutive matrices in a batch expressed in terms of matrix elements
The algorithm enumerator is used in the cusparseLtMatDescSetAttribute() and cusparseLtMatDescGetAttribute() functions.

cusparseLtMatmulDescAttribute_t

The enumerator specifies the additional attributes of a matrix multiplication descriptor

Value

Type

Default Value

Description

CUSPARSELT_MATMUL_ACTIVATION_RELU

int 0: false, true otherwise

false

ReLU activation function

CUSPARSELT_MATMUL_ACTIVATION_RELU_UPPERBOUND

float

inf

Upper bound of the ReLU activation function

CUSPARSELT_MATMUL_ACTIVATION_RELU_THRESHOLD

float

0.0f

Lower threshold of the ReLU activation function

CUSPARSELT_MATMUL_ACTIVATION_GELU

int 0: false, true otherwise

false

Enable/Disable GeLU activation function

CUSPARSELT_MATMUL_ACTIVATION_GELU_SCALING

float

1.0f

Scaling coefficient for the GeLU activation function. It implies CUSPARSELT_MATMUL_ACTIVATION_GELU

CUSPARSELT_MATMUL_ALPHA_VECTOR_SCALING

int 0: false, true otherwise

false

Enable/Disable alpha vector (per-channel) scaling

CUSPARSELT_MATMUL_BETA_VECTOR_SCALING

int 0: false, true otherwise

false

Enable/Disable beta vector (per-channel) scaling

CUSPARSELT_MATMUL_BIAS_POINTER

void*

NULL (disabled)

Bias pointer. The bias vector size must equal to the number of rows of the output matrix (D)

CUSPARSELT_MATMUL_BIAS_STRIDE

int64_t

0 (disabled)

Bias stride between consecutive bias vectors. 0 means broadcast the first bias vector

where the ReLU activation function is defined as:

workflow

  • The GeLU activation function is available only with INT8 input/output, INT32 Tensor Core compute kernels

  • CUSPARSELT_MATMUL_BETA_VECTOR_SCALING implies CUSPARSELT_MATMUL_ALPHA_VECTOR_SCALING

The algorithm enumerator is used in the cusparseLtMatmulDescSetAttribute() and cusparseLtMatmulDescGetAttribute() functions.

cusparseLtMatmulAlg_t

The enumerator specifies the algorithm for matrix-matrix multiplication

Value

Description

CUSPARSELT_MATMUL_ALG_DEFAULT

Default algorithm
The algorithm enumerator is used in the cusparseLtMatmulAlgSelectionInit() function.

cusparseLtMatmulAlgAttribute_t

The enumerator specifies the matrix multiplication algorithm attributes

Value

Description

CUSPARSELT_MATMUL_ALG_CONFIG_ID

Algorithm ID (set and query)

CUSPARSELT_MATMUL_ALG_CONFIG_MAX_ID

Algorithm ID limit (query only)

CUSPARSELT_MATMUL_SEARCH_ITERATIONS

Number of iterations (kernel launches per algorithm) for cusparseLtMatmulSearch(), default=10

CUSPARSELT_MATMUL_SPLIT_K

Split-K factor, default=not set. Valid range: [1, K]. Value 1 is equivalent to the Split-K feature is disabled

CUSPARSELT_MATMUL_SPLIT_K_MODE

Number of kernels to call for Split-K. Values are specified in cusparseLtSplitKMode_t.

CUSPARSELT_MATMUL_SPLIT_BUFFERS

Device memory buffers to store partial results for the reduction. The valid range is [1, SplitK - 1]
The algorithm attribute enumerator is used in the cusparseLtMatmulAlgGetAttribute() and cusparseLtMatmulAlgSetAttribute() functions.
Split-K parameters allow users to split the GEMM computation along the K dimension so that more CTAs will be created with a better SM utilization when N or M dimensions are small. However, this comes with the cost of reducing the operation of K slides to the final results. The cusparseLtMatmulSearch() function can be used to find the optimal combination of Split-K parameters when CUSPARSELT_MATMUL_SPLIT_K=1.

cusparseLtSplitKMode_t

The enumerator specifies the Split-K mode values corresponding to CUSPARSELT_MATMUL_SPLIT_K_MODE attribute in cusparseLtMatmulAlgAttribute_t

Value

Description

Details

CUSPARSELT_SPLIT_K_MODE_ONE_KERNEL

Use a single kernel for Split-K

Use the same GEMM kernel to do the final reduction

CUSPARSELT_SPLIT_K_MODE_TWO_KERNELS

Use two kernels for Split-K

Launch another GPU kernel to do the final reduction

cusparseLtPruneAlg_t

The enumerator specifies the pruning algorithm to apply to the structured matrix before the compression

Value

Description

CUSPARSELT_PRUNE_SPMMA_TILE

- half, bfloat16, int8: Zero-out eight values in a 4x4 tile to maximize the L1-norm of the resulting tile, under the constraint of selecting exactly two elements for each row and column

 

- float, tf32: Zero-out two values in a 2x2 tile to maximize the L1-norm of the resulting tile, under the constraint of selecting exactly one element for each row and column

CUSPARSELT_PRUNE_SPMMA_STRIP

- half, bfloat16, int8: Zero-out two values in a 1x4 strip to maximize the L1-norm of the resulting strip

 

- float, tf32: Zero-out one value in a 1x2 strip to maximize the L1-norm of the resulting strip

 

The strip direction is chosen according to the operation op and matrix layout applied to the structured (sparse) matrix
The pruning algorithm is used in the cusparseLtSpMMAPrune() function.