Mellanox SHARP binary distribution is available as part of HPC-X, MLNX_OFED or UFM packages. UFM package includes only the Aggregation Manager.
- In case of HPC-X package, please refer to HPC-X User Manual for installation and configuration procedures.
This Deployment Guide and examples refer to the following environment variables HPCX_SHARP_DIR, OMPI_HOME and assumes that HPC-X installation is in a shared folder accessible from all compute nodes.
- In case of MLNX_OFED distribution or custom installation, you have to set the HPCX_SHARP_DIR environment variable to point to the directory in which it was installed (/opt/mellanox/sharp is a default directory for Mellanox SHARP software in MLNX_OFED package). OMPI_HOME should point to the MPI installation folder.
- In case of using the Aggregation Manager from the UFM distribution, you have to enable Mellanox SHARP support in UFM. For further information, refer to the UFM User Manual.
The rest of the Mellanox SHARP components should be installed from either the HPC-X or MLNX_OFED packages.
Make sure the following are set prior to configuring the setup:
- Mellanox SHARP configuration files must be created in the same location (please refer to serctions 5 TBD and 6 TBD). Make sure that you have write permission to HPCX_SHARP_DIR/.
- Aggregation Manager and Mellanox SHARP daemons are trusted entities and should run by a privilege user (root user).
Aggregation Manager is a central entity running on a dedicated server with SM. This dedicated server cannot serve as a compute node and host Mellanox SHARP daemon. Therefore, make sure onboard Subnet Manager is disabled in managed switches.
- Mellanox SHARP daemons and Aggregation Manager communicate over TCP/IP which should be configured before running Mellanox SHARP. Please check firewall settings.
- Run ibdiagnet and check the Switch-IB firmware installed. See section Prerequisites TBD for the supported switch firmware version.
- Using UD Mcast feature requires IPoIB interface enabled in compute servers.
