Troubleshooting

Known Issues

  • Some users may face hangs due to lazy initialization of NCCL in UCC. To disable the lazy NCCL initialization, please set UCC_TL_NCCL_LAZY_INIT environment variable to no

  • Some users may see errors with HPC-X v2.18 caused by a clash of UCC being initialized in OMPI and cuBLASMp. To disable UCC initialization in OMPI, please set OMPI_MCA_coll_ucc_enable environment variable to 0