cuBLASMp: A High-Performance CUDA Library for Distributed Dense Linear Algebra

NVIDIA cuBLASMp is a high-performance, multi-process, GPU-accelerated library for distributed basic dense linear algebra.

cuBLASMp is compatible with 2D block-cyclic data layout and provides PBLAS-like C APIs.

Download: cuBLASMp library is available through NVIDIA Developer Zone , NVIDIA HPC SDK, PyPI and conda-forge.

Key Features

• Multi-process, multi-GPU.

• One process per GPU.

• PBLAS-like C functionalities and interfaces to facilitate porting.

• NCCL and NVSHMEM communication backends.

• Logging and tracing.

• Tensor-core accelerated.

Index

• Getting Started
  • Introduction
  • Hardware and Software Requirements
  • Data Layout of Local Matrices
  • Workflow
  • Code Samples
• How to Use cuBLASMp
  • cuBLASMp Initialization
  • Using cuBLASMp for Tensor Parallelism in Distributed Machine Learning
  • cuBLASMp Logging
  • cuBLASMp Data Types
  • cuBLASMp C API
• Release Notes
  • cuBLASMp v0.5.0
  • cuBLASMp v0.4.0
  • cuBLASMp v0.3.1
  • cuBLASMp v0.3.0
  • cuBLASMp v0.2.1
  • cuBLASMp v0.2.0
  • cuBLASMp v0.1.2
  • cuBLASMp v0.1.1
  • cuBLASMp v0.1.0
• Troubleshooting
  • Hints
  • Known Issues
• Software License Agreement
  • Third Party License Agreements