Installing and Loading HPC-X
Extract hpcx.tbz into your current working directory.
tar -xvf hpcx.tbz
Update shell variable of the location of HPC-X installation.
$ cd hpcx $ export HPCX_HOME=$PWD
HPC-X includes Open MPI v4.1.x. Each Open MPI version has its own module file which can be used to load the desired version.
The symbolic links hpcx-init.sh and modulefiles/hpcx point to the default version (Open MPI v4.1.x).
% source $HPCX_HOME/hpcx-init.sh
% hpcx_load
% env | grep HPCX
% mpicc $HPCX_MPI_TESTS_DIR/examples/hello_c.c -o $HPCX_MPI_TESTS_DIR/examples/hello_c
% mpirun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_c
% oshcc $HPCX_MPI_TESTS_DIR/examples/hello_oshmem_c.c -o $HPCX_MPI_TESTS_DIR/examples/hello_oshmem_c
% oshrun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_oshmem_c
% hpcx_unload
As of version 1.7, HPC-X builds are no longer distributed based on the Intel compiler suite. However, after following the HPC-X deployment example below, HPC-X can subsequently be rebuilt from source with your Intel compiler suite as follows:
$ tar xfp ${HPCX_HOME}/sources/openmpi-gitclone.tar.gz
$ cd ${HPCX_HOME}/sources/openmpi-gitclone
$ ./configure CC=icc CXX=icpc F77=ifort FC=ifort --prefix=${HPCX_HOME}/ompi-icc \
--with-hcoll=${HPCX_HOME}/hcoll \
--with-ucx=${HPCX_HOME}/ucx \
--with-platform=contrib/platform/mellanox/optimized \
2>&1 | tee config-icc-output.log
$ make -j32 all 2>&1 | tee build_icc.log && make -j24 install 2>&1 | tee install_icc.log
In the above example, 4 switches are used to specify the compiler suite:
|
CC: |
Specifies the C compiler |
|
CXX: |
Specifies the C++ compiler |
|
F77: |
Specifies the Fortran 77 compiler |
|
FC: |
Specifies the Fortran 90 compiler |
We strongly recommend using a single compiler suite whenever possible. Unexpected or undefined behavior can occur when you mix compiler suites in unsupported ways (e.g., mixing Fortran 77 and Fortran 90 compilers between different compiler suite is almost guaranteed not to work.)
In all cases, the Intel compiler suite must be found in your PATH and be able to successfully compile and link non-MPI applications before Open MPI will be able to be built properly.
For rebuilding HPC-X open-source components, please use the helper script as described in "Rebuilding Open MPI Using a Helper Script" section.
To load Open MPI/OpenSHMEM v4.1.x based package:
% module use $HPCX_HOME/modulefiles
% module load hpcx
% mpicc $HPCX_MPI_TESTS_DIR/examples/hello_c.c -o $HPCX_MPI_TESTS_DIR/examples/hello_c
% mpirun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_c
% oshcc $HPCX_MPI_TESTS_DIR/examples/hello_oshmem_c.c -o $HPCX_MPI_TESTS_DIR/examples/hello_oshmem_c
% oshrun -np 2 $HPCX_MPI_TESTS_DIR/examples/hello_oshmem_c
% module unload hpcx
Starting from version 2.1, HPC-X toolkit is provided with a set of environments. You are to select the environment that meets your needs best.
HPC-X with CUDA® support - hpcx
WarningCuda support in SLES 11, RHEL 6 and RHEL OSs lower than 7.4 with PPC arch is no longer available.
This is the default option which is optimized for best performance for the single-thread mode. This option supports both GPU and non-GPU setups.
WarningStarting with CUDA 11.0, the minimum recommended GCC compiler is at least GCC 5 due to C++11 requirements in CUDA libraries e.g. cuFFT and CUB.
HPC-X with multi-threading support - hpcx-mt
This option enables multi-threading support in all of the HPC-X components. Please use this module in order to run multi-threaded applications.HPC-X for profiling - hpcx-prof
This option enables UCX compiled with profiling information.HPC-X for debug - hpcx-debug
This option enables UCX/HCOLL/SHARP compiled in debug mode.HPC-X stack - hpcx-stack
This environment contains all the libraries that 'Vanilla HPCX' has, except for OMPI.
When HPC-X is launched with Open MPI without a resource manager job environment (slurm,pbs, etc.), or when it is launched from a compute node, the default rsh/ssh-based launcher will be used. This launcher does not propagate environment variables to the compute nodes. Thus, it is important to ensure the propagation of LD_LIBRARY_PATH variable from HPC-x is done as follows.
%mpirun -x LD_LIBRARY_PATH -np 2 -H host1,host2 $HPCX_MPI_TESTS_DIR/examples/hello_c
Note that only one of the environments can be loaded to be run.
For information on how to load and use the additional environments, please refer to the HPC-X README file (embedded in the HPC-X package).
HPC-X supports Singularity containerization technology, which help deploying and running distributed applications without launching an entire virtual machine (VM) for each application.
For instructions on the technology and how to create a standalone Singularity container with MLNX_OFED and HPC-X inside, please visit:
$HPCX_HOME/utils/singularity/hpcx-singularity.md file inside HPC-X package