Getting Started¶
Introduction¶
cudaMalloc()
, cudaFree()
, cudaMemcpy()
, and cudaMemcpyAsync()
.Hardware and Software requirements¶
Hardware requirements¶
x86_64 CPU architecture
NVIDIA data center GPUs: with Volta (
SM 7.0
), Ampere (SM 8.0
), or Hopper (SM 9.0
) architecturesRecommended NVIDIA Infiniband solutions for accelerated inter-node communication
Software requirements¶
Supported OS: Linux x86_64
Supported CUDA: 11.8, 12.1.1
Required packages
CUDA 11.8.0
CUDA Toolkit 11.8.0 (https://developer.nvidia.com/cuda-11-8-0-download-archive)
HPC-X v2.14 (https://developer.nvidia.com/networking/hpc-x) - contains OpenUCC and OpenUCX that satisfy cuSOLVERMp requirements.
NCCL v2.16.x (https://developer.nvidia.com/nccl) - required to achieve good performance
CUDA 12.1.1
CUDA Toolkit 12.1.1 (https://developer.nvidia.com/cuda-12-1-1-download-archive)
HPC-X v2.16 (https://developer.nvidia.com/networking/hpc-x) - contains OpenUCC and OpenUCX that satisfy cuSOLVERMp requirements.
NCCL v2.16.x (https://developer.nvidia.com/nccl) - required to achieve good performance
Recommended packages
OpenUCX v1.10+ (https://github.com/openucx/ucx/releases) and OpenUCC v1.1+ (https://github.com/openucx/ucc/releases) - alternative to HPC-X you can install OpenUCX and OpenUCC manually. Both needs to be configured with CUDA support
GDRCopy v2.0+ (https://github.com/NVIDIA/gdrcopy) and nv_peer_mem (https://github.com/Mellanox/nv_peer_memory) - Allows underlying communication packages use GPUDirect RDMA. If you install OpenUCX yourself - it should be configured with GDRCopy support.
Mellanox OFED (https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed) - drivers for NVIDIA Infiniband Adapters (https://www.nvidia.com/en-us/networking/products/infiniband). If you install OpenUCX yourself - it should be configured with IB communications support.
Synchronous Execution¶
Data Layout of Local Matrices¶
Workflow¶
1. Bootstrap CAL communicator: cal_comm_create().2. Initialize the library handle: cusolverMpCreate().3. Initialize grid descriptors: cusolverMpCreateDeviceGrid().4. Initialize matrix descriptors: cusolverMpCreateMatrixDesc().5. Query the host and device buffer sizes for a given routine.6. Allocate host and device workspace buffers for a given routine.6. Execute the routine to perform the desired computation.7. Synchronize local stream to make sure the result is available, if required: cal_stream_sync().8. Deallocate host and device workspace.9. Destroy matrix descriptors: cusolverMpDestroyMatrixDesc().10. Destroy grid descriptors: cusolverMpDestroyGrid().11. Destroy cuSOLVERMp library handle: cusolverMpDestroy().12. Destroy CAL library handle: cal_comm_destroy().