Troubleshooting

Hints

• If any issues need to be debugged, the following environment variables are very useful:

  • CUSOLVERMP_LOG_LEVEL environment variable to enable detailed logging of cuSOLVERMp operations

  • NCCL_DEBUG. NCCL provides extensive debugging capabilities through environment variables such as NCCL_DEBUG. For a comprehensive list of NCCL debugging options, refer to the NCCL Environment Variables documentation.

Known Issues

• Some users on multi-node NVLink+RDMA platforms may experience hangs with versions of NCCL older than 2.31. To work around this issue, update NCCL to 2.31 or newer, or set the NCCL_PXN_DISABLE=1 environment variable.

• cuSOLVERMp versions older than 0.9.0: Workspace queries may under-report the required device workspace with cuSOLVER from CUDA Toolkit 13.2 Update 1 and newer, which can cause failures, illegal memory accesses, or hangs in cusolverMpGeqrf(), cusolverMpGels(), cusolverMpOrgqr(), cusolverMpOrmqr(), cusolverMpGetrf(), cusolverMpPotrf(), and cusolverMpSygvd(). To avoid this issue, update to cuSOLVERMp 0.9.0 or newer.

• cuSOLVERMp versions older than 0.7.0: Some users may face hangs due to lazy initialization of NCCL in UCC. To disable the lazy NCCL initialization, please set UCC_TL_NCCL_LAZY_INIT environment variable to no.

• cuSOLVERMp versions older than 0.7.0: Some users may see errors with HPC-X v2.18 caused by a clash of UCC being initialized in OMPI and cuSOLVERMp. To disable UCC initialization in OMPI, please set OMPI_MCA_coll_ucc_enable environment variable to 0.