> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Using MPI ## 3.1 Introduction The Message Passing Interface (MPI) is a standardized and portable message passing system designed by a group of researchers from academia and industry to function on a wide variety of parallel computers. The standard defines the syntax and semantics of a core of library routines useful to a wide range of users writing portable message-passing programs in Fortran or the C programming language. MPI libraries allow the compilation of code so that it can be used over a variety of multi-processor systems from SMP nodes to NUMA (non-Uniform Memory Access) systems and interconnected cluster nodes . Depending on the cluster hardware, the interconnect available may be Ethernet, InfiniBand/OmniPath. Typically, the cluster administrator has already custom-configured the cluster in a way to suit the workflow of users. However, users that are interested in understanding and using the MPI options available, should find this chapter helpful. ## 3.2 MPI Libraries MPI libraries that are commonly integrated by the cluster administrator with NVIDIA Base Command Manager are * MPICH ([https://www.mpich.org/](https://www.mpich.org/)) * MVAPICH ([http://mvapich.cse.ohio-state.edu/](http://mvapich.cse.ohio-state.edu/)) * OpenMPI ([https://www.open-mpi.org/](https://www.open-mpi.org/)) The preceding MPI libaries can be used with compilers from GCC, or Intel. The following MPI library implementations may also be integrated with BCM: * Intel MPI Library ([https://software.intel.com/en-us/mpi-library](https://software.intel.com/en-us/mpi-library)), which works with the Intel compiler suite. * Open MPI version 4 library with CUDA-awareness. NVIDIA GPUs using CUDA can make use of this library. ## 3.3 MPI Packages And Module Paths By default, the only MPI implementations installed are the ones that work using GCC. The cluster administrator can make available for the user a variety of different MPI implementations for different compilers, by installing the appropriate packages. The actual combination of compiler and MPI implementation that a user needs for job runs can then be loaded by the user with the help of modules (section [2.3.2](/user/clusterusage#232-managing-environment-modules-as-a-user)). ### 3.3.1 MPI Packages That Can Be Installed, And Their Corresponding Module Paths The following packages are made available from the BCM repositories at the time of writing (October 2022):
**Argonne National Laboratory Ethernet implementation for MPI-1, MPI-2, MPI-3**
**package name** **module path**
`mpich-ge-gcc-64` `mpich/ge/gcc`
`mpich-ge-intel-64` `mpich/ge/intel`
**MVAPICH2 InfiniBand implementation for MPI-3**
**package name** **module path**
`mvapich2-gcc-64` `mvapich2/gcc`
`mvapich2-intel-64` `mvapich2/intel`
**MVAPICH2 InfiniBand implementation with performance scaled messaging for MPI-3**
**package name** **module path**
`mvapich2-psmgcc-64` `mvapich2/psmgcc`
`mvapich2-psmintel-64` `mvapich2/psmintel`
**Open MPI version 1 implementation for Ethernet with MPI-3**
**package name** **module path**
`openmpi-ge-gcc-64` `openmpi/gcc`
`openmpi-ge-intel-64` `openmpi/intel`
**Open MPI version 1 implementation for Ethernet and InfiniBand with MPI-3**
**package name** **module path**
`openmpi-geib-gcc-64` `openmpi/gcc`
`openmpi-geib-intel-64` `openmpi/intel`
**Open MPI version 3 implementation for Ethernet and InfiniBand with MPI-3**
**package name** **module path**
`openmpi3-geib-gcc-64` `openmpi3/gcc`
`openmpi3-ge-gcc-64` (only Ethernet, only GCC) `openmpi3/gcc`
`openmpi3-ge-intel-64` (only Ethernet, only Intel compiler) `openmpi3/intel`
**Open MPI version 4 implementation for InfiniBand with MPI-3**
**package name** **module path**
`openmpi4-gcc` (GCC) `openmpi4/gcc`
`openmpi4-intel` (Intel compiler) `openmpi4/intel`
**Open MPI version 4 implementation for OFED with MPI-3**
**package name** **module path**
`cm-openmpi4-cuda11.2-ofed47-gcc9` `openmpi4-cuda11.2-ofed47-gcc9`
`cm-openmpi4-cuda11.2-ofed50-gcc9` `openmpi4-cuda11.2-ofed50-gcc9`
`cm-openmpi4-cuda11.2-ofed51-gcc9` `openmpi4-cuda11.2-ofed51-gcc9`
`cm-openmpi4-cuda11.7-ofed47-gcc9` `openmpi4-cuda11.7-ofed47-gcc9`
`cm-openmpi4-cuda11.7-ofed50-gcc9` `openmpi4-cuda11.7-ofed50-gcc9`
`cm-openmpi4-cuda11.7-ofed51-gcc9` `openmpi4-cuda11.7-ofed51-gcc9`
**Intel MPI library implementation for the Intel compiler**
**package name** **module path**
`intel-mpi-2019` `intel/mpi/64`
`intel-mpi-2020` `intel/mpi/64`
### 3.3.2 Finding The Installed MPI Packages And Their Available Module Paths #### 3.3.2.1 Finding The Installed MPI Packages The packages installed on the cluster can be found with the `rpm -qa` query on a RHEL system. **Example** ```bash [fred@basecm10 ~]$ # search for packages starting with (open)mpi, mvapich, intel-mpi, cm-openmpi) [fred@basecm10 ~]$ rpm -qa | egrep '(^mpi|^openmpi|^mvapich|^intel-mpi|cm-openmpi)' openmpi-geib-gcc-64-1.10.7-656_cm9.2.x86_64 mpich-ge-gcc-64-3.4.2-197_cm9.2.x86_64 mvapich2-gcc-64-2.3.7-213_cm9.2.x86_64 openmpi4-gcc-4.1.2-100042_cm9.2_d17293b429.x86_64 intel-mpi-2020-2019.9-100008_cm9.2_aca4120975.x86_64 cm-openmpi4-cuda11.7-ofed47-gcc9-4.1.4-100012_cm9.2_a9f8de3740.x86_64 openmpi4-intel-4.1.2-100042_cm9.2_d17293b429.x86_64 ``` The interconnect and compiler implementation of a package can be worked out from looking at the name of the package. Here, for example, ```bash openmpi-geib-gcc-64-1.10.7-656_cm9.2.x86_64 ``` implies: Open MPI version `1.10.7`, compiled for both Gigabit Ethernet (`ge`) and InfiniBand (`ib`), with GCC (`gcc`, the GNU project cross-compiler) for a 64-bit architecture, packaged as a (BCM) cluster manager (`cm`) package, for version 10.0 for the `x86_64` architecture. #### 3.3.2.2 Finding The Available MPI Modules The corresponding module paths can be found by a search through the available modules: **Example** ```bash [fred@basecm10 ~]$ # search for modules starting with (open)mpi or mvapich [fred@basecm10 ~]$ module -l avail | egrep '(^openmpi|^mpi|^mvapich|^intel/mpi)' intel/mpi/64/ default intel/mpi/64/2020/4.304 2021/10/23 22:18:12 mpich/ge/gcc/64/3.4.2 2021/11/26 22:33:47 mvapich2/gcc/64/2.3.7 2022/07/14 13:02:23 openmpi/gcc/64/1.10.7 2022/10/13 22:45:54 openmpi/gcc/64/4.1.2 2022/09/09 22:52:03 openmpi/intel/64/4.1.2 2022/09/09 23:35:29 openmpi4-cuda11.7-ofed47-gcc9/4.1.4 2022/10/10 22:37:29 openmpi4/gcc/4.1.2 2022/09/09 22:52:03 openmpi4/intel/4.1.2 2022/09/09 23:35:29 ``` Tab-completion when searching for modules is another approach. As in the case for the MPI library, for a module the interconnect and compiler implementation can likewise be worked out from looking at the name of the module. So, if a user would like to use an Open MPI implementation that works with an AMD64 node, using GCC, then loading the module `openmpi/gcc/64` should be suitable. ## 3.4 The Appropriate Interconnect, Compiler, And MPI Implementation For A Module ### 3.4.1 Interconnects Jobs can use particular networks for inter-node communication. The hardware for these networks can be Ethernet or InfiniBand, while the software can be a particular MPI implementation. #### 3.4.1.1 Gigabit Ethernet Gigabit Ethernet is an interconnect that is commonly available in consumer computing. For HPC the next generation 10 Gigabit Ethernet and beyond have also been in use for some time. For Ethernet, no additional modules or libraries are needed. The Open MPI, MPICH and MVAPICH implementations all work with any Ethernet technology. #### 3.4.1.2 InfiniBand InfiniBand is a high-performance switched fabric that has come to dominate the connectivity in HPC applications. An InfiniBand interconnect has a lower latency and somewhat higher throughput than a comparably-priced Ethernet interconnect. This is because of special hardware, as well as special software that shortcuts the networking stack in the operating system layer. This typically provides significantly greater performance for most HPC applications. Open MPI and MVAPICH are suitable MPI implementations for InfiniBand. ### 3.4.2 Selecting A Compiler And MPI implementation Once the appropriate compiler module has been loaded, the associated MPI implementation is selected by loading the corresponding library module. In the following simplified list, \<*`compiler`*> indicates a choice of `gcc`, or `intel`: * `mpich/ge/` \<*`compiler`*> * `mvapich2/` \<*`compiler`*> * `openmpi/` \<*`compiler`*> * `openmpi3/` \<*`compiler`*> * `openmpi4/` \<*`compiler`*> Section [3.3.1](#331-mpi-packages-that-can-be-installed-and-their-corresponding-module-paths) should be refered to for a more complete list. ## 3.5 Compiling And Carrying Out An MPI Run After the appropriate MPI module has been added to the user environment, the user can start compiling applications. For a cluster using Ethernet interconnectivity, the `mpich` and `openmpi` implementations may be used. For a cluster using InfiniBand, the `mvapich`, `mvapich2` and `openmpi` implementations may be used. Open MPI’s `openmpi` implementations first attempt to use InfiniBand, but revert to Ethernet if InfiniBand is not available. In this section the loading of modules, compilation, and runs are illustrated. ### 3.5.1 Example MPI Run This example covers an MPI run, which can be run inside and outside of a queuing system. Depending on the libraries and compilers installed on the system by the cluster administrator, the availability of the packages providing these modules may differ. To see a full list of modules on the system the command `module avail` can be typed. #### 3.5.1.1 Examples Of Loading MPI Modules To use `mpirun`, the relevant environment modules must be loaded. For example, to use the `mpich` over Gigabit Ethernet (`ge`) GCC implementation: ```bash $ module add mpich/ge/gcc ``` or to use the `openmpi4` Open MPI version 4 MPI-3 Intel implementation: ```bash $ module add openmpi4/intel ``` Similarly, to use the `mvapich2` InfiniBand GCC implementation: ```bash $ module add mvapich2/gcc ``` #### 3.5.1.2 Keeping Loading Of Modules Clean The preceding modules can actually be loaded concurrently, and it works as expected. Paths supplied by the most recently-loaded module override the paths of any previous modules. For example, if `mpich/ge/gcc` is loaded first, then `openmpi4/intel`, and then `mvapich2/gcc`, as suggested in the preceding excerpts, then the modules might be listed as: ```bash $ module list Currently Loaded Modulefiles: Currently Loaded Modulefiles: 1) shared 3) cmd 5) cm-setup/9.2 7) openmpi4/intel/4.1.2 2) cmsh 4) cluster-tools/9.2 6) mpich/ge/gcc/64/3.4.2 8) mvapich2/gcc/64/2.3.7 ``` The path of the MPI compiler `mpicc` is defined by the last module in the modulefiles stack, which is the MVAPICH GCC implementation, as can be seen with: ```bash $ which mpicc /cm/shared/apps/mvapich2/gcc/64/2.3.7/bin/mpicc ``` After removing it, then the path changes to the path supplied by the previous module in the stack, the Open MPI Intel implementation: ```bash $ module remove mvapich2/gcc/64/2.3.7 $ which mpicc /cm/shared/apps/openmpi4/intel/4.1.2/bin/mpicc ``` After removing that module too, the path again changes to the path supplied by the previous module in the stack, the MPICH GCC implementation: ```bash $ module remove openmpi4/intel/4.1.2 $ which mpicc /cm/shared/apps/mpich/ge/gcc/64/3.4.2/bin/mpicc ``` However, because loading modules on top of each other can cause confusion, a user should generally try adding modules in a simple, clean manner. #### 3.5.1.3 Examples Compiling And Preparing The MPI Application The code must be compiled with MPI libraries and an underlying compiler. The correct library command can be found in the following table:
**Language** **C** **C++** **Fortran77** **Fortran90**
**Command** `mpicc ` `mpixx` `mpif77` `mpif90`
An MPI application `myapp.c`, built in C, could then be compiled as: ```bash $ mpicc myapp.c ``` The `mpicc` compilation requires the underlying compiler (GCC, Intel) already be available. By default, Linux systems have a version of the GCC compiler already in their paths, even if no modules have been loaded. So if the module for an MPI implementation based on GCC is loaded without explicitly loading the GCC compiler, then the `mpicc` compilation still works in this case. The `a.out` binary that is created can then be executed using the `mpirun` command (section [3.5.1.5](#3515-running-the-application)). #### 3.5.1.4 Creating A Machine File A machine file contains a list of nodes which can be used by MPI programs. The workload management system creates a machine file based on the nodes allocated for a job when the job is submitted with the workload manager job submission tool. So if the user chooses to have the workload management system allocate nodes for the job, then creating a machine file is not needed. However, if an MPI application is being run “by hand” outside the workload manager, then the user is responsible for creating a machine file manually. Depending on the MPI implementation, the layout of this file may differ. Machine files can generally be created in two ways: * Listing the same node several times to indicate that more than one process should be started on each node: ```bash node001 node001 node002 node002 ``` * Listing nodes once, but with a suffix for the number of CPU cores to use on each node: ```bash node001:2 node002:2 ``` #### 3.5.1.5 Running The Application #### 3.5.1.6 Creating A Simple Parallel Processing Executable A simple “hello world” program designed for parallel processing can be built with MPI. After compiling it, it can be used to send a message about how and where it is running: ```bash [fred@basecm10 ~]$ cat hello.c #include #include int main (int argc, char *argv[]) { int id, np, i; char processor_name[MPI_MAX_PROCESSOR_NAME]; int processor_name_len; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &np); MPI_Comm_rank(MPI_COMM_WORLD, &id); MPI_Get_processor_name(processor_name, &processor_name_len); for(i=1;i<2;i++) {printf( "Hello world from process %03d out of %03d, processor name %s\n", id, np, processor_name );} MPI_Finalize(); return 0; } [fred@basecm10 ~]$ module add openmpi/gcc #or as appropriate depending on installed package [fred@basecm10 ~]$ mpicc hello.c -o hello ``` The preceding compilation works for MPI based on `gcc` because a version of `gcc` is already available by default with the distribution. Its version can be found with, for example: ```bash [fred@basecm10 ~]$ gcc --version | head -1 gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-10) ``` If the user has a need for another version of the `gcc` compiler, then that can be loaded before the compilation. ```bash [fred@basecm10 ~]$ module load gcc/11.2.0 [fred@basecm10 ~]$ mpicc hello.c -o hello ``` If the user would like to use the Intel compiler with MPI instead, then the steps would instead be something like: ```bash [fred@basecm10 ~]$ module load intel/compiler/64/2020/19.1.3 intel/mpi/64/2020/4.304 [fred@basecm10 ~]$ mpicc hello.c -o helloforintel ``` If in doubt about how things are compiled, then the libraries that the binary was compiled with can be checked using `ldd`. For example, the GCC-related libraries used could be checked with: ```bash [fred@basecm10 ~]$ ldd hello |grep gcc |awk '{print $1,$2,$3}' libmpi.so.12 => /cm/shared/apps/openmpi/gcc/64/1.10.7/lib64/libmpi.so.12 libopen-rte.so.12 => /cm/shared/apps/openmpi/gcc/64/1.10.7/lib64/libopen-rte.so.12 libopen-pal.so.13 => /cm/shared/apps/openmpi/gcc/64/1.10.7/lib64/libopen-pal.so.13 ``` The compiled binary can be run: ```bash [fred@basecm10 ~]$ ./hello Hello world from process 000 out of 001, processor name basecm10 ``` However, it still runs on a single processor, and on the node it is started on, unless it is submitted to the system in a special way. #### 3.5.1.7 Running An MPI Executable In Parallel Without A Workload Manager **Compute node environment provided by user’s `.bashrc`:** After the relevant module files are chosen (section [3.5.1](#351-example-mpi-run)) for MPI, an executable compiled with MPI libraries runs on nodes in parallel when submitted with `mpirun`. The executable running on other nodes loads environment modules on those other nodes by sourcing the `.bashrc` file of the user (section [2.3.3](/user/clusterusage#233-changing-the-default-environment-modules)) and flags passed by the `mpirun` command. It is a good idea to ensure that the environment module stack used on the compute node is not confusing. The environment of the user from the interactive shell prompt is not normally carried over automatically to the compute nodes during an `mpirun` submission. That is, compiling and running the executable normally only works on the local node without a special treatment. To have the executable run on the compute nodes, the right environment modules for the job must be made available on the compute nodes too, as part of the user login process to the compute nodes for that job. Usually the system administrator takes care of such matters in the default user configuration by setting up a default user environment (section [2.3.3](/user/clusterusage#233-changing-the-default-environment-modules)), with reasonable `initrm` and `initadd` options. Users are then typically allowed to set up their personal default overrides to the default administrator settings, by placing their own `initrm` and `initadd` options to the `module` command according to their needs, or by specifying an overriding environment in the job submissions. **Running `mpirun` outside a workload manager:** When using `mpirun` manually, outside a workload manager environment, the number of processes (`-np`) as well as the number of hosts (`-machinefile`) should be specified. For example, on a cluster with 2 compute-nodes and a machine file as specified [here](#3514-creating-a-machine-file): **Example** ```bash [fred@basecm10 ~]$ module add mvapich2/gcc gcc/11.2.0 #or as appropriate [fred@basecm10 ~]$ mpicc hello.c -o hello [fred@basecm10 ~]$ mpirun -np 4 -machinefile mpirun.hosts ~/hello Hello world from process 0 of 4 on node001.cm.cluster Hello world from process 1 of 4 on node002.cm.cluster Hello world from process 2 of 4 on node001.cm.cluster Hello world from process 3 of 4 on node002.cm.cluster ``` If the cluster has no InfiniBand connectors, then the preceding `mpirun` fails, because MVAPICH requires InfiniBand. That kind of failure displays an output such as: ```bash rdma_open_hca(575)...............: No IB device found ``` Open MPI implementations are more forgiving. They check for InfiniBand and if that is unavailable they go ahead with Ethernet. However they have their own quirks. For example, an attempt to carry out an MPI run with Open MPI might be as follows: **Example** ```bash [fred@basecm10 ~]$ module initclear; module initadd openmpi/gcc [fred@basecm10 ~]$ module add openmpi/gcc #or as appropriate [fred@basecm10 ~]$ mpicc hello.c -o hello [fred@basecm10 ~]$ mpirun -np 4 -machinefile mpirun.hosts hello bash: orted: command not found -------------------------------------------------------------------------- ORTE was unable to reliably start one or more daemons. This usually is caused by: ... output snipped ``` It is generally a good idea to read through the man page for `mpirun` for the MPI implementation that the user is using. In this case, the man page for `mpirun` for Open MPI reveals that here the problem is that the environment in the interactive shell is not carried over to the compute nodes during an `mpirun` submission. So the path for `mpirun` should be specified with with either the `--prefix` option, or as an absolute path: **Example** ```bash [fred@basecm10 ~]$ /cm/shared/apps/openmpi/gcc/64/1.10.7/bin/mpirun -np 4 -machinefile mpirun.hosts hello Hello world from process 002 out of 004, processor name node002.cm.cluster Hello world from process 003 out of 004, processor name node001.cm.cluster Hello world from process 000 out of 004, processor name node002.cm.cluster Hello world from process 001 out of 004, processor name node001.cm.cluster ``` The output of the preceding `hello.c` program is actually printed in random order. This can be modified as follows, so that only process 0 prints to the standard output, and other processes communicate their output to process 0: ```bash #include "mpi.h" #include "string.h" #include int main( int argc, char *argv[] ) { int numprocs, myrank, namelen, i; char processor_name[MPI_MAX_PROCESSOR_NAME]; char greeting[MPI_MAX_PROCESSOR_NAME + 80]; MPI_Status status; MPI_Init( &argc, &argv ); MPI_Comm_size( MPI_COMM_WORLD, &numprocs ); MPI_Comm_rank( MPI_COMM_WORLD, &myrank ); MPI_Get_processor_name( processor_name, &namelen ); sprintf( greeting, "Hello world, from process %d of %d on %s", myrank, numprocs, processor_name ); if ( myrank == 0 ) { printf( "%s\n", greeting ); for ( i = 1; i < numprocs; i++ ) { MPI_Recv( greeting, sizeof( greeting ), MPI_CHAR, i, 1, MPI_COMM_WORLD, &status ); printf( "%s\n", greeting ); } } else { MPI_Send( greeting, strlen( greeting ) + 1, MPI_CHAR, 0, 1, MPI_COMM_WORLD ); } MPI_Finalize( ); return 0; } ``` **Example** ```bash [fred@basecm10 ~]$ module clear [fred@basecm10 ~]$ module add intel/compiler/64 intel/mpi [fred@basecm10 ~]$ module list Currently Loaded Modulefiles: 1) intel/compiler/64/2020/19.1.3 2) intel/mpi/64/2020/4.304 [fred@basecm10 ~]$ mpicc hello.c -o hello [fred@basecm10 ~]$ mpirun -np 4 -machinefile mpirun.hosts ./hello Hello world, from process 0 of 4 on node002 Hello world, from process 1 of 4 on node002 Hello world, from process 2 of 4 on node002 Hello world, from process 3 of 4 on node003 ``` Running the executable with `mpirun` outside the workload manager as shown does not take the resources of the cluster into account. To handle running jobs with cluster resources is of course what workload managers such as Slurm are designed to do. Workload managers also typically take care of what environment modules should be loaded on the compute nodes for a job, via additions that the user makes to a job script. Running an application through a workload manager via a job script is introduced in Chapter [7](/admin/workloadmanagement). Appendix [A](/user/mpiexample) contains a number of simple MPI programs. ### 3.5.2 Hybridization OpenMP is an implementation of multi-threading. This is a method of parallelizing whereby a parent thread—a series of instructions executed consecutively—forks a specified number of child threads, and a task is divided among them. The threads then run concurrently, with the runtime environment allocating threads to different processors and accessing the shared memory of an SMP system. MPI can be mixed with OpenMP to achieve high performance on a cluster/supercomputer of multi-core nodes or servers. MPI creates processes that reside on the level of node, while OpenMP forks threads on the level of a core within an SMP node. Each process executes a portion of the overall computation, while inside each process, a team of threads is created through OpenMP directives to further divide the problem. This kind of execution makes sense due to: * the ease of programming that OpenMP provides * OpenMP not necessarily requiring copies of data structure, which allows for designs that overlap computation and communication * overcoming the limits of parallelism within the SMP node is of course still possible by using the power of other nodes via MPI. **Example** ```bash #include #include #include #include int main(int argc , char** argv) { int size, myrank,namelength; char processor_name[MPI_MAX_PROCESSOR_NAME]; MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_COMM_WORLD,&myrank); MPI_Comm_size(MPI_COMM_WORLD,&size); MPI_Get_processor_name(processor_name,&namelength); printf("Hello I am Processor %d on %s of %d\n",myrank,processor_name,\ size); int tid = 0; int n_of_threads = 1; #pragma omp parallel default(shared) private(tid, n_of_threads) { #if defined (_OPENMP) n_of_threads= omp_get_num_threads(); tid = omp_get_thread_num(); #endif printf("Hybrid Hello World: I am thread # %d out of %d\n", tid, n_o\ f_threads); } MPI_Finalize(); return 0; } ``` The program can be compiled as follows: ```bash fred@basecm10 ~]$ mpicc -o hybridhello omphello.c -fopenmp ``` To specify the number of OpenMP threads per MPI task the environment variable `OMP_NUM_THREADS` must be set. **Example** ```bash fred@basecm10 ~]$ export OMP_NUM_THREADS=3 ``` The number of threads specified by the variable can then be run over the hosts specified by the `mpirun.hosts` file: ```bash fred@basecm10 ~]$ mpirun -np 2 -hostfile mpirun.hosts ./hybridhello Hello I am Processor 0 on node001 of 2 Hello I am Processor 1 on node002 of 2 Hybrid Hello World: I am thread # 0 out of 3 Hybrid Hello World: I am thread # 2 out of 3 Hybrid Hello World: I am thread # 1 out of 3 Hybrid Hello World: I am thread # 0 out of 3 Hybrid Hello World: I am thread # 2 out of 3 Hybrid Hello World: I am thread # 1 out of 3 ``` #### 3.5.2.1 Benefits And Drawbacks Of Using OpenMP The main benefit to using OpenMP is that it can decrease memory requirements, with usually no reduction in performance. Other benefits include: * Potential additional parallelization opportunities besides those exploited by MPI. * Less domain decomposition, which can help with load balancing as well as allowing for larger messages and fewer tasks participating in MPI collective operations. * OpenMP is a standard, so any modifications introduced into an application are portable and appear as comments on systems not using OpenMP. * By adding annotations to existing code and using a compiler option, it is possible to add OpenMP to a code somewhat incrementally, almost on a loop-by-loop basis. The vector loops in a code that vectorize well are good candidates for OpenMP. There are also some potential drawbacks: * OpenMP can be hard to program and/or debug in some cases. * Effective usage can be complicated on NUMA systems due to locality considerations * If an application is network- or memory- bandwidth-bound, then threading it is not going to help. In this case it will be OK to leave some cores idle. * In some cases a serial portion may be essential, which can inhibit performance. * In most MPI codes, synchronization is implicit and happens when messages are sent and received. However, with OpenMP, much synchronization must be added to the code explicitly. The programmer must also explicitly determine which variables can be shared among threads and which ones cannot (parallel scoping). OpenMP codes that have errors introduced by incomplete or misplaced synchronization or improper scoping can be difficult to debug because the error can introduce race conditions which cause the error to happen only intermittently. ### 3.5.3 Support Thread Levels MPI defines four “levels” of thread safety. The maximum thread support level is returned by the `MPI_Init_thread` call in the “provided” argument. An environment variable `MPICH_MAX_THREAD_SAFETY` can be set to different values to increase the thread safety: | **MPICH_MAX_THREAD_SAFETY** | **Supported Thread Level** | | :-------------------------- | :------------------------- | | not set | MPI_THREAD_SINGLE | | single | MPI_THREAD_SINGLE | | funneled | MPI_THREAD_FUNNELED | | serialized | MPI_THREAD_SERIALIZED | | multiple | MPI_THREAD_MULTIPLE | ### 3.5.4 Further Recommendations Users face various challenges with running and scaling large scale jobs on peta-scale production systems. For example: certain applications may not have enough memory per core, the default environment variables may need to be adjusted, or I/O may dominate run time. Possible ways to deal with these are: * Trying out various compilers and compiler flags, and finding out which options are best for particular applications. * Changing the default MPI rank ordering. This is a simple, yet sometimes effective, runtime tuning option that requires no source code modification, recompilation or re-linking. The default MPI rank placement on the compute nodes is SMP style. However, other choices are round-robin, folded rank, and custom ranking. * Using fewer cores per node is helpful when more memory per process than the default is needed. Having fewer processes to share the memory and interconnect bandwidth is also helpful in this case. For NUMA nodes, extra care must be taken. * Hybrid MPI/OpenMP reduces the memory footprint. Overlapping communication with computation in hybrid MPI/OpenMP can be considered. * Some applications may perform better when large memory pages are used.