Using MPI

3.1 Introduction

The Message Passing Interface (MPI) is a standardized and portable message passing system designed by a group of researchers from academia and industry to function on a wide variety of parallel computers. The standard defines the syntax and semantics of a core of library routines useful to a wide range of users writing portable message-passing programs in Fortran or the C programming language. MPI libraries allow the compilation of code so that it can be used over a variety of multi-processor systems from SMP nodes to NUMA (non-Uniform Memory Access) systems and interconnected cluster nodes .

Depending on the cluster hardware, the interconnect available may be Ethernet, InfiniBand/OmniPath.

Typically, the cluster administrator has already custom-configured the cluster in a way to suit the workflow of users. However, users that are interested in understanding and using the MPI options available, should find this chapter helpful.

3.2 MPI Libraries

MPI libraries that are commonly integrated by the cluster administrator with NVIDIA Base Command Manager are

• MPICH (https://www.mpich.org/)

• MVAPICH (http://mvapich.cse.ohio-state.edu/)

• OpenMPI (https://www.open-mpi.org/)

The preceding MPI libaries can be used with compilers from GCC, or Intel.

The following MPI library implementations may also be integrated with BCM:

• Intel MPI Library (https://software.intel.com/en-us/mpi-library), which works with the Intel compiler suite.

• Open MPI version 4 library with CUDA-awareness. NVIDIA GPUs using CUDA can make use of this library.

3.3 MPI Packages And Module Paths

By default, the only MPI implementations installed are the ones that work using GCC.

The cluster administrator can make available for the user a variety of different MPI implementations for different compilers, by installing the appropriate packages. The actual combination of compiler and MPI implementation that a user needs for job runs can then be loaded by the user with the help of modules (section 2.3.2).

3.3.1 MPI Packages That Can Be Installed, And Their Corresponding Module Paths

The following packages are made available from the BCM repositories at the time of writing (October 2022):

3.3.2 Finding The Installed MPI Packages And Their Available Module Paths

3.3.2.1 Finding The Installed MPI Packages

The packages installed on the cluster can be found with the rpm -qa query on a RHEL system.

Example

[fred@basecm11 ~]$ # search for packages starting with (open)mpi, mvapich, intel-mpi, cm-openmpi)
[fred@basecm11 ~]$ rpm -qa | egrep '(^mpi|^openmpi|^mvapich|^intel-mpi|cm-openmpi)'
openmpi-geib-gcc-64-1.10.7-656_cm9.2.x86_64
mpich-ge-gcc-64-3.4.2-197_cm9.2.x86_64
mvapich2-gcc-64-2.3.7-213_cm9.2.x86_64
openmpi4-gcc-4.1.2-100042_cm9.2_d17293b429.x86_64
intel-mpi-2020-2019.9-100008_cm9.2_aca4120975.x86_64
cm-openmpi4-cuda11.7-ofed47-gcc9-4.1.4-100012_cm9.2_a9f8de3740.x86_64
openmpi4-intel-4.1.2-100042_cm9.2_d17293b429.x86_64

The interconnect and compiler implementation of a package can be worked out from looking at the name of the package.

Here, for example,

openmpi-geib-gcc-64-1.10.7-656_cm9.2.x86_64

implies: Open MPI version 1.10.7, compiled for both Gigabit Ethernet (ge) and InfiniBand (ib), with GCC (gcc, the GNU project cross-compiler) for a 64-bit architecture, packaged as a (BCM) cluster manager (cm) package, for version 11 for the x86_64 architecture.

3.3.2.2 Finding The Available MPI Modules

The corresponding module paths can be found by a search through the available modules:

Example

[fred@basecm11 ~]$ # search for modules starting with (open)mpi or mvapich
[fred@basecm11 ~]$ module -l avail | egrep '(^openmpi|^mpi|^mvapich|^intel/mpi)'
intel/mpi/64/                           default
intel/mpi/64/2020/4.304                                     2021/10/23 22:18:12
mpich/ge/gcc/64/3.4.2                                       2021/11/26 22:33:47
mvapich2/gcc/64/2.3.7                                       2022/07/14 13:02:23
openmpi/gcc/64/1.10.7                                       2022/10/13 22:45:54
openmpi/gcc/64/4.1.2                                        2022/09/09 22:52:03
openmpi/intel/64/4.1.2                                      2022/09/09 23:35:29
openmpi4-cuda11.7-ofed47-gcc9/4.1.4                         2022/10/10 22:37:29
openmpi4/gcc/4.1.2                                          2022/09/09 22:52:03
openmpi4/intel/4.1.2                                        2022/09/09 23:35:29

Tab-completion when searching for modules is another approach.

As in the case for the MPI library, for a module the interconnect and compiler implementation can likewise be worked out from looking at the name of the module. So, if a user would like to use an Open MPI implementation that works with an AMD64 node, using GCC, then loading the module openmpi/gcc/64 should be suitable.

3.4 The Appropriate Interconnect, Compiler, And MPI Implementation For A Module

3.4.1 Interconnects

Jobs can use particular networks for inter-node communication. The hardware for these networks can be Ethernet or InfiniBand, while the software can be a particular MPI implementation.

3.4.1.1 Gigabit Ethernet

Gigabit Ethernet is an interconnect that is commonly available in consumer computing. For HPC the next generation 10 Gigabit Ethernet and beyond have also been in use for some time. For Ethernet, no additional modules or libraries are needed. The Open MPI, MPICH and MVAPICH implementations all work with any Ethernet technology.

3.4.1.2 InfiniBand

InfiniBand is a high-performance switched fabric that has come to dominate the connectivity in HPC applications. An InfiniBand interconnect has a lower latency and somewhat higher throughput than a comparably-priced Ethernet interconnect. This is because of special hardware, as well as special software that shortcuts the networking stack in the operating system layer. This typically provides significantly greater performance for most HPC applications. Open MPI and MVAPICH are suitable MPI implementations for InfiniBand.

3.4.2 Selecting A Compiler And MPI implementation

Once the appropriate compiler module has been loaded, the associated MPI implementation is selected by loading the corresponding library module. In the following simplified list, <compiler> indicates a choice of gcc, or intel:

• mpich/ge/ <compiler>

• mvapich2/ <compiler>

• openmpi/ <compiler>

• openmpi3/ <compiler>

• openmpi4/ <compiler>

Section 3.3.1 should be refered to for a more complete list.

3.5 Compiling And Carrying Out An MPI Run

After the appropriate MPI module has been added to the user environment, the user can start compiling applications.

For a cluster using Ethernet interconnectivity, the mpich and openmpi implementations may be used. For a cluster using InfiniBand, the mvapich, mvapich2 and openmpi implementations may be used. Open MPI’s openmpi implementations first attempt to use InfiniBand, but revert to Ethernet if InfiniBand is not available.

In this section the loading of modules, compilation, and runs are illustrated.

3.5.1 Example MPI Run

This example covers an MPI run, which can be run inside and outside of a queuing system.

Depending on the libraries and compilers installed on the system by the cluster administrator, the availability of the packages providing these modules may differ. To see a full list of modules on the system the command module avail can be typed.

3.5.1.1 Examples Of Loading MPI Modules

To use mpirun, the relevant environment modules must be loaded. For example, to use the mpich over Gigabit Ethernet (ge) GCC implementation:

$ module add mpich/ge/gcc

or to use the openmpi4 Open MPI version 4 MPI-3 Intel implementation:

$ module add openmpi4/intel

Similarly, to use the mvapich2 InfiniBand GCC implementation:

$ module add mvapich2/gcc

3.5.1.2 Keeping Loading Of Modules Clean

The preceding modules can actually be loaded concurrently, and it works as expected. Paths supplied by the most recently-loaded module override the paths of any previous modules.

For example, if mpich/ge/gcc is loaded first, then openmpi4/intel, and then mvapich2/gcc, as suggested in the preceding excerpts, then the modules might be listed as:

$ module list
Currently Loaded Modulefiles:
Currently Loaded Modulefiles:
 1) shared   3) cmd                 5) cm-setup/9.2            7) openmpi4/intel/4.1.2
 2) cmsh     4) cluster-tools/9.2   6) mpich/ge/gcc/64/3.4.2   8) mvapich2/gcc/64/2.3.7

The path of the MPI compiler mpicc is defined by the last module in the modulefiles stack, which is the MVAPICH GCC implementation, as can be seen with:

$ which mpicc
/cm/shared/apps/mvapich2/gcc/64/2.3.7/bin/mpicc

After removing it, then the path changes to the path supplied by the previous module in the stack, the Open MPI Intel implementation:

$ module remove mvapich2/gcc/64/2.3.7
$ which mpicc
/cm/shared/apps/openmpi4/intel/4.1.2/bin/mpicc

After removing that module too, the path again changes to the path supplied by the previous module in the stack, the MPICH GCC implementation:

$ module remove openmpi4/intel/4.1.2
$ which mpicc
/cm/shared/apps/mpich/ge/gcc/64/3.4.2/bin/mpicc

However, because loading modules on top of each other can cause confusion, a user should generally try adding modules in a simple, clean manner.

3.5.1.3 Examples Compiling And Preparing The MPI Application

The code must be compiled with MPI libraries and an underlying compiler. The correct library command can be found in the following table:

An MPI application myapp.c, built in C, could then be compiled as:

$ mpicc myapp.c

The mpicc compilation requires the underlying compiler (GCC, Intel) already be available. By default, Linux systems have a version of the GCC compiler already in their paths, even if no modules have been loaded. So if the module for an MPI implementation based on GCC is loaded without explicitly loading the GCC compiler, then the mpicc compilation still works in this case.

The a.out binary that is created can then be executed using the mpirun command (section 3.5.1.5).

3.5.1.4 Creating A Machine File

A machine file contains a list of nodes which can be used by MPI programs.

The workload management system creates a machine file based on the nodes allocated for a job when the job is submitted with the workload manager job submission tool. So if the user chooses to have the workload management system allocate nodes for the job, then creating a machine file is not needed.

However, if an MPI application is being run “by hand” outside the workload manager, then the user is responsible for creating a machine file manually. Depending on the MPI implementation, the layout of this file may differ.

Machine files can generally be created in two ways:

• Listing the same node several times to indicate that more than one process should be started on each node:

  node001
  node001
  node002
  node002

• Listing nodes once, but with a suffix for the number of CPU cores to use on each node:

  node001:2
  node002:2

3.5.1.5 Running The Application

3.5.1.6 Creating A Simple Parallel Processing Executable

A simple “hello world” program designed for parallel processing can be built with MPI. After compiling it, it can be used to send a message about how and where it is running:

[fred@basecm11 ~]$ cat hello.c
#include <stdio.h>
#include <mpi.h>
int main (int argc, char *argv[])
{
    int id, np, i;
    char processor_name[MPI_MAX_PROCESSOR_NAME];
    int processor_name_len;
    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &np);
    MPI_Comm_rank(MPI_COMM_WORLD, &id);
    MPI_Get_processor_name(processor_name, &processor_name_len);
    for(i=1;i<2;i++)
    {printf(
    "Hello world from process %03d out of %03d, processor name %s\n",
    id, np, processor_name
    );}
    MPI_Finalize();
    return 0;
}
[fred@basecm11 ~]$ module add openmpi/gcc   #or as appropriate depending on installed package
[fred@basecm11 ~]$ mpicc hello.c -o hello

The preceding compilation works for MPI based on gcc because a version of gcc is already available by default with the distribution. Its version can be found with, for example:

fred@basecm11:~$ which gcc; gcc --version | head -1
/usr/bin/gcc
gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0

If the user has a need for another version of the gcc compiler, then that can be loaded before the compilation.

fred@basecm11:~$ module load gcc/14.2.0
fred@basecm11:~$ which gcc; gcc --version | head -1
/cm/local/apps/gcc/14.2.0/bin/gcc
gcc (GCC) 14.2.0
fred@basecm11:~$ mpicc hello.c -o hello

If the user would like to use the Intel compiler with MPI instead, then the steps would instead be something like:

[fred@basecm11 ~]$ module load intel/compiler/64/2020/19.1.3 intel/mpi/64/2020/4.304
[fred@basecm11 ~]$ mpicc hello.c -o helloforintel

If in doubt about how things are compiled, then the libraries that the binary was compiled with can be checked using ldd. For example, the GCC-related libraries used could be checked with:

[fred@basecm11 ~]$ ldd hello |grep gcc |awk '{print $1,$2,$3}'
libmpi.so.12 => /cm/shared/apps/openmpi/gcc/64/1.10.7/lib64/libmpi.so.12
libopen-rte.so.12 => /cm/shared/apps/openmpi/gcc/64/1.10.7/lib64/libopen-rte.so.12
libopen-pal.so.13 => /cm/shared/apps/openmpi/gcc/64/1.10.7/lib64/libopen-pal.so.13

The compiled binary can be run:

[fred@basecm11 ~]$ ./hello
Hello world from process 000 out of 001, processor name basecm11

However, it still runs on a single processor, and on the node it is started on, unless it is submitted to the system in a special way.

3.5.1.7 Running An MPI Executable In Parallel Without A Workload Manager

Compute node environment provided by user’s .bashrc: After the relevant module files are chosen (section 3.5.1) for MPI, an executable compiled with MPI libraries runs on nodes in parallel when submitted with mpirun. The executable running on other nodes loads environment modules on those other nodes by sourcing the .bashrc file of the user (section 2.3.3) and flags passed by the mpirun command. It is a good idea to ensure that the environment module stack used on the compute node is not confusing.

The environment of the user from the interactive shell prompt is not normally carried over automatically to the compute nodes during an mpirun submission. That is, compiling and running the executable normally only works on the local node without a special treatment. To have the executable run on the compute nodes, the right environment modules for the job must be made available on the compute nodes too, as part of the user login process to the compute nodes for that job. Usually the system administrator takes care of such matters in the default user configuration by setting up a default user environment (section 2.3.3), with reasonable initrm and initadd options. Users are then typically allowed to set up their personal default overrides to the default administrator settings, by placing their own initrm and initadd options to the module command according to their needs, or by specifying an overriding environment in the job submissions.

Running mpirun outside a workload manager: When using mpirun manually, outside a workload manager environment, the number of processes (-np) as well as the number of hosts (-machinefile) should be specified. For example, on a cluster with 2 compute-nodes and a machine file as specified here:

Example

[fred@basecm11 ~]$ module add mvapich2/gcc gcc/11.2.0  #or as appropriate
[fred@basecm11 ~]$ mpicc hello.c -o hello
[fred@basecm11 ~]$ mpirun -np 4 -machinefile mpirun.hosts ~/hello
Hello world from process 0 of 4 on node001.cm.cluster
Hello world from process 1 of 4 on node002.cm.cluster
Hello world from process 2 of 4 on node001.cm.cluster
Hello world from process 3 of 4 on node002.cm.cluster

If the cluster has no InfiniBand connectors, then the preceding mpirun fails, because MVAPICH requires InfiniBand. That kind of failure displays an output such as:

rdma_open_hca(575)...............: No IB device found

Open MPI implementations are more forgiving. They check for InfiniBand and if that is unavailable they go ahead with Ethernet. However they have their own quirks. For example, an attempt to carry out an MPI run with Open MPI might be as follows:

Example

[fred@basecm11 ~]$ module initclear; module initadd openmpi/gcc
[fred@basecm11 ~]$ module add openmpi/gcc   #or as appropriate
[fred@basecm11 ~]$ mpicc hello.c -o hello
[fred@basecm11 ~]$ mpirun -np 4 -machinefile mpirun.hosts hello
bash: orted: command not found
--------------------------------------------------------------------------
ORTE was unable to reliably start one or more daemons.
This usually is caused by:
...
    output snipped

It is generally a good idea to read through the man page for mpirun for the MPI implementation that the user is using. In this case, the man page for mpirun for Open MPI reveals that here the problem is that the environment in the interactive shell is not carried over to the compute nodes during an mpirun submission. So the path for mpirun should be specified with with either the --prefix option, or as an absolute path:

Example

[fred@basecm11 ~]$ /cm/shared/apps/openmpi/gcc/64/1.10.7/bin/mpirun -np 4 -machinefile mpirun.hosts hello
Hello world from process 002 out of 004, processor name node002.cm.cluster
Hello world from process 003 out of 004, processor name node001.cm.cluster
Hello world from process 000 out of 004, processor name node002.cm.cluster
Hello world from process 001 out of 004, processor name node001.cm.cluster

The output of the preceding hello.c program is actually printed in random order. This can be modified as follows, so that only process 0 prints to the standard output, and other processes communicate their output to process 0:

#include "mpi.h"
#include "string.h"
#include <stdio.h>
int main( int argc, char *argv[] )
{
    int  numprocs, myrank, namelen, i;
    char processor_name[MPI_MAX_PROCESSOR_NAME];
    char greeting[MPI_MAX_PROCESSOR_NAME + 80];
    MPI_Status status;
    MPI_Init( &argc, &argv );
    MPI_Comm_size( MPI_COMM_WORLD, &numprocs );
    MPI_Comm_rank( MPI_COMM_WORLD, &myrank );
    MPI_Get_processor_name( processor_name, &namelen );
    sprintf( greeting, "Hello world, from process %d of %d on %s",
             myrank, numprocs, processor_name );
    if ( myrank == 0 ) {
        printf( "%s\n", greeting );
        for ( i = 1; i < numprocs; i++ ) {
            MPI_Recv( greeting, sizeof( greeting ), MPI_CHAR,
                      i, 1, MPI_COMM_WORLD, &status );
            printf( "%s\n", greeting );
        }
    }
    else {
        MPI_Send( greeting, strlen( greeting ) + 1, MPI_CHAR,
                  0, 1, MPI_COMM_WORLD );
    }
    MPI_Finalize( );
    return 0;
}

Example

[fred@basecm11 ~]$ module clear
[fred@basecm11 ~]$  module add intel/compiler/64 intel/mpi
[fred@basecm11 ~]$  module list
Currently Loaded Modulefiles:
 1) intel/compiler/64/2020/19.1.3   2) intel/mpi/64/2020/4.304
[fred@basecm11 ~]$ mpicc hello.c -o hello
[fred@basecm11 ~]$  mpirun -np 4 -machinefile mpirun.hosts ./hello
Hello world, from process 0 of 4 on node002
Hello world, from process 1 of 4 on node002
Hello world, from process 2 of 4 on node002
Hello world, from process 3 of 4 on node003

Running the executable with mpirun outside the workload manager as shown does not take the resources of the cluster into account. To handle running jobs with cluster resources is of course what workload managers such as Slurm are designed to do. Workload managers also typically take care of what environment modules should be loaded on the compute nodes for a job, via additions that the user makes to a job script.

Running an application through a workload manager via a job script is introduced in Chapter 7.

Appendix A contains a number of simple MPI programs.

3.5.2 Hybridization

OpenMP is an implementation of multi-threading. This is a method of parallelizing whereby a parent thread—a series of instructions executed consecutively—forks a specified number of child threads, and a task is divided among them. The threads then run concurrently, with the runtime environment allocating threads to different processors and accessing the shared memory of an SMP system.

MPI can be mixed with OpenMP to achieve high performance on a cluster/supercomputer of multi-core nodes or servers. MPI creates processes that reside on the level of node, while OpenMP forks threads on the level of a core within an SMP node. Each process executes a portion of the overall computation, while inside each process, a team of threads is created through OpenMP directives to further divide the problem. This kind of execution makes sense due to:

• the ease of programming that OpenMP provides

• OpenMP not necessarily requiring copies of data structure, which allows for designs that overlap computation and communication

• overcoming the limits of parallelism within the SMP node is of course still possible by using the power of other nodes via MPI.

Example

#include<mpi.h>
#include <omp.h>
#include <stdio.h>
#include<stdlib.h>
int main(int argc , char** argv) {
  int size, myrank,namelength;
  char processor_name[MPI_MAX_PROCESSOR_NAME];
  MPI_Init(&argc,&argv);
  MPI_Comm_rank(MPI_COMM_WORLD,&myrank);
  MPI_Comm_size(MPI_COMM_WORLD,&size);
  MPI_Get_processor_name(processor_name,&namelength);
  printf("Hello I am Processor %d on %s of %d\n",myrank,processor_name,\
size);
    int tid = 0;  int n_of_threads = 1;
    #pragma omp parallel default(shared) private(tid, n_of_threads)
    {
    #if defined (_OPENMP)
          n_of_threads= omp_get_num_threads();
          tid = omp_get_thread_num();
    #endif
    printf("Hybrid Hello World: I am thread # %d out of %d\n", tid, n_o\
f_threads);
    }
 MPI_Finalize();
 return 0;
}

The program can be compiled as follows:

fred@basecm11 ~]$ mpicc -o hybridhello omphello.c -fopenmp

To specify the number of OpenMP threads per MPI task the environment variable OMP_NUM_THREADS must be set.

Example

fred@basecm11 ~]$ export OMP_NUM_THREADS=3

The number of threads specified by the variable can then be run over the hosts specified by the mpirun.hosts file:

fred@basecm11 ~]$ mpirun -np 2 -hostfile mpirun.hosts ./hybridhello
Hello I am Processor 0 on node001 of 2
Hello I am Processor 1 on node002 of 2
Hybrid Hello World: I am thread # 0 out of 3
Hybrid Hello World: I am thread # 2 out of 3
Hybrid Hello World: I am thread # 1 out of 3
Hybrid Hello World: I am thread # 0 out of 3
Hybrid Hello World: I am thread # 2 out of 3
Hybrid Hello World: I am thread # 1 out of 3

3.5.2.1 Benefits And Drawbacks Of Using OpenMP

The main benefit to using OpenMP is that it can decrease memory requirements, with usually no reduction in performance. Other benefits include:

• Potential additional parallelization opportunities besides those exploited by MPI.

• Less domain decomposition, which can help with load balancing as well as allowing for larger messages and fewer tasks participating in MPI collective operations.

• OpenMP is a standard, so any modifications introduced into an application are portable and appear as comments on systems not using OpenMP.

• By adding annotations to existing code and using a compiler option, it is possible to add OpenMP to a code somewhat incrementally, almost on a loop-by-loop basis. The vector loops in a code that vectorize well are good candidates for OpenMP.

There are also some potential drawbacks:

• OpenMP can be hard to program and/or debug in some cases.

• Effective usage can be complicated on NUMA systems due to locality considerations

• If an application is network- or memory- bandwidth-bound, then threading it is not going to help. In this case it will be OK to leave some cores idle.

• In some cases a serial portion may be essential, which can inhibit performance.

• In most MPI codes, synchronization is implicit and happens when messages are sent and received. However, with OpenMP, much synchronization must be added to the code explicitly. The programmer must also explicitly determine which variables can be shared among threads and which ones cannot (parallel scoping). OpenMP codes that have errors introduced by incomplete or misplaced synchronization or improper scoping can be difficult to debug because the error can introduce race conditions which cause the error to happen only intermittently.

3.5.3 Support Thread Levels

MPI defines four “levels” of thread safety. The maximum thread support level is returned by the MPI_Init_thread call in the “provided” argument.

An environment variable MPICH_MAX_THREAD_SAFETY can be set to different values to increase the thread safety:

3.5.4 Further Recommendations

Users face various challenges with running and scaling large scale jobs on peta-scale production systems. For example: certain applications may not have enough memory per core, the default environment variables may need to be adjusted, or I/O may dominate run time.

Possible ways to deal with these are:

• Trying out various compilers and compiler flags, and finding out which options are best for particular applications.

• Changing the default MPI rank ordering. This is a simple, yet sometimes effective, runtime tuning option that requires no source code modification, recompilation or re-linking. The default MPI rank placement on the compute nodes is SMP style. However, other choices are round-robin, folded rank, and custom ranking.

• Using fewer cores per node is helpful when more memory per process than the default is needed. Having fewer processes to share the memory and interconnect bandwidth is also helpful in this case. For NUMA nodes, extra care must be taken.

• Hybrid MPI/OpenMP reduces the memory footprint. Overlapping communication with computation in hybrid MPI/OpenMP can be considered.

• Some applications may perform better when large memory pages are used.