Release Notes#

cuBLASMp v0.4.0#

Added support for NVIDIA Blackwell GPU architecture.
Added support for GEMM + AllReduce using the cublasMpMatmul API.
Added support for NN cases in AllGather + GEMM and GEMM + ReduceScatter variants of the cublasMpMatmul API.
Bug fixes.
Deprecated functionality: atomic Matmul with multicast communication (i.e., cublasMpMatmul with cublasMpMatmulAlgoType_t = CUBLASMP_MATMUL_ALGO_TYPE_ATOMIC_MULTICAST). The functional implementation remains available but it is not performant and will be removed in a future release.

Add option to set the amount of SMs to be used for communication (currently relevant only for Atomic GEMM + ReduceScatter).
Decrease workspace size requirement in TP overlap GEMMs.
Remove extra synchronization in TP overlap GEMMs.
Allow C matrix to be null when beta is 0.
Fix GEMM implementation for complex types with transA / transB being CUBLAS_OP_T.

Added new cublasMpMatmul API.
Added GEMM/Matmul fast paths required for tensor parallelism (communication-computation-overlapped AllGather+GEMM and GEMM+ReduceScatter).
Added FP8 support.
Added cublasMpStatus_t.
Added cublasMpStreamSet and cublasMpStreamGet.
Added cublasMpMatrixDescriptorInit API to allow reusing matrix descriptors.
Added NVSHMEM dependency.
Added Matmul helper APIs: cublasMpMatmulDescriptorCreate, cublasMpMatmulDescriptorDestroy, cublasMpMatmulDescriptorAttributeSet, cublasMpMatmulDescriptorAttributeGet.
Dropped support for CUDA 11.x.
Bug fixes.

Removed the cublasMpHandle_t parameter from cublasMpGridCreate, cublasMpGridDestroy, cublasMpMatrixDescriptorCreate, cublasMpMatrixDescriptorDestroy, cublasMpGetVersion APIs.
Changed the return status of all functions to cublasMpStatus_t.
Removed cublasMpSetMathMode and cublasMpGetMathMode APIs.