Cluster Usage

2.1 Login To The Cluster Environment

The login node is the node where the user logs in and works from. Simple clusters have a single login node, but large clusters sometimes have multiple login nodes to improve the availability of the cluster. In most clusters, the login node is also the head node from where the cluster is monitored and installed. On the login node:

• applications can be developed

• code can be compiled and debugged

• applications can be submitted to the cluster for execution

• running applications can be monitored

To carry out an ssh login to the cluster, a terminal session can be started from Unix-like operating systems:

Example

$ ssh myname@cluster.hostname

On a Windows operating system, an SSH client such as PuTTY (http://www.putty.org) can be downloaded. Another standard possibility is to run a Unix-like environment such as Cygwin (http://www.cygwin.com) within the Windows operating system, and then run the SSH client from within it.

A Mac OS X user can use the Terminal application from the Finder, or under Application/Utilities/Terminal.app. An X11 windowing environment must be installed for it to work. XQuartz is the recommended X11 windowing environment, and is indeed the only Apple-backed X11 version available from OS X 10.8 onward.

When using the SSH connection, the cluster’s address must be added. When the connection is made, a username and password must be entered at the prompt.

If the administrator has changed the default SSH port from 22 to something else, the port can be specified with the -p <port> option:

$ ssh -X -p <port> <user>@<cluster>

The -X option can be dropped if no X11-forwarding is required. X11-forwarding allows a GUI application from the cluster to be displayed locally.

Optionally, after logging in, the password used can be changed using the passwd command:

$ passwd

2.2 Setting Up The User Environment

By default, each user uses the bash shell interpreter. In that case, each time a user login takes place, a file named .bashrc is executed to set up the shell environment for the user. The shell and its environment can be customized to suit user preferences. For example,

• the prompt can be changed to indicate the current username, host, and directory, for example: by setting the prompt string variable:

  PS1='[\u@\h \W]\$ '

• the size of the command history file can be increased, for example: export HISTSIZE=100

• aliases can be added for frequently used command sequences, for example: alias lart='ls -alrt'

• environment variables can be created or modified, for example: export MYVAR=MYSTRING

• the location of software packages and versions that are to be used by a user (the path to a package) can be set.

Because there is a huge choice of software packages and versions, it can be hard to set up the right environment variables and paths for software that is to be used. Collisions between different versions of the same package and non-matching dependencies on other packages must also be avoided. To make setting up the environment easier, BCM provides preconfigured environment modules (section 2.3).

2.3 Environment Modules

For a user to compile and run computational jobs on a cluster, a special shell environment is typically set up for the software that is used.

However, setting up the right environment for a particular software package and version can be tricky, and it can be hard to keep track of how it was set up.

For example, users want to have a clean way to bring up the right environment for compiling code according to the various MPI implementations, but can easily get confused about which libraries have been used, and can end up with multiple libraries with similar names installed in a disorganized manner.

A user might also like to conveniently test new versions of a software package before permanently installing the package.

Within a Linux distribution running without special utilities, setting up environments can be complex. However, BCM makes use of the environment modules package, which provides the module command. The module command is a special utility to make taking care of the shell environment much easier.

2.3.1 Available Commands

Practical use of the modules commands is given in sections 2.3.2 and 2.3.3.

For reference, the help text for the module command can be viewed as follows:

Example

[me@cluster ~]$ module --help
Modules Release 4.5.3 (2020-08-31)
Usage: module [options] [command] [args ...]
Loading / Unloading commands:
  add | load      modulefile [...]  Load modulefile(s)
  rm | unload     modulefile [...]  Remove modulefile(s)
  purge                             Unload all loaded modulefiles
  reload | refresh                  Unload then load all loaded modulefiles
  switch | swap   [mod1] mod2       Unload mod1 and load mod2
Listing / Searching commands:
  list            [-t|-l|-j]        List loaded modules
  avail   [-d|-L] [-t|-l|-j] [-S|-C] [--indepth|--no-indepth] [mod ...]
                                    List all or matching available modules
  aliases                           List all module aliases
  whatis  [-j]    [modulefile ...]  Print whatis information of modulefile(s)
  apropos | keyword | search [-j] str
                                    Search all name and whatis containing str
  is-loaded       [modulefile ...]  Test if any of the modulefile(s) are loaded
  is-avail        modulefile [...]  Is any of the modulefile(s) available
  info-loaded     modulefile        Get full name of matching loaded module(s)
Collection of modules handling commands:
  save            [collection|file] Save current module list to collection
  restore         [collection|file] Restore module list from collection or file
  saverm          [collection]      Remove saved collection
  saveshow        [collection|file] Display information about collection
  savelist        [-t|-l|-j]        List all saved collections
  is-saved        [collection ...]  Test if any of the collection(s) exists
Shell's initialization files handling commands:
  initlist                          List all modules loaded from init file
  initadd         modulefile [...]  Add modulefile to shell init file
  initrm          modulefile [...]  Remove modulefile from shell init file
  initprepend     modulefile [...]  Add to beginning of list in init file
  initswitch      mod1 mod2         Switch mod1 with mod2 from init file
  initclear                         Clear all modulefiles from init file
Environment direct handling commands:
  prepend-path [-d c] var val [...] Prepend value to environment variable
  append-path [-d c] var val [...]  Append value to environment variable
  remove-path [-d c] var val [...]  Remove value from environment variable
Other commands:
  help            [modulefile ...]  Print this or modulefile(s) help info
  display | show  modulefile [...]  Display information about modulefile(s)
  test            [modulefile ...]  Test modulefile(s)
  use     [-a|-p] dir [...]         Add dir(s) to MODULEPATH variable
  unuse           dir [...]         Remove dir(s) from MODULEPATH variable
  is-used         [dir ...]         Is any of the dir(s) enabled in MODULEPATH
  path            modulefile        Print modulefile path
  paths           modulefile        Print path of matching available modules
  clear           [-f]              Reset Modules-specific runtime information
  source          scriptfile [...]  Execute scriptfile(s)
  config [--dump-state|name [val]]  Display or set Modules configuration
Switches:
  -t | --terse    Display output in terse format
  -l | --long     Display output in long format
  -j | --json     Display output in JSON format
  -d | --default  Only show default versions available
  -L | --latest   Only show latest versions available
  -S | --starts-with
                  Search modules whose name begins with query string
  -C | --contains Search modules whose name contains query string
  -i | --icase    Case insensitive match
  -a | --append   Append directory to MODULEPATH
  -p | --prepend  Prepend directory to MODULEPATH
  --auto          Enable automated module handling mode
  --no-auto       Disable automated module handling mode
  -f | --force    By-pass dependency consistency or confirmation dialog
Options:
  -h | --help     This usage info
  -V | --version  Module version
  -D | --debug    Enable debug messages
  -v | --verbose  Enable verbose messages
  -s | --silent   Turn off error, warning and informational messages
  --paginate      Pipe mesg output into a pager if stream attached to terminal
  --no-pager      Do not pipe message output into a pager
  --color[=WHEN]  Colorize the output; WHEN can be 'always' (default if
                  omitted), 'auto' or 'never'

2.3.2 Managing Environment Modules As A User

There is a good chance the cluster administrator has set up the user’s account, fred for example, so that some modules are loaded already by default. In that case, the modules loaded into the user’s environment can be seen with the module list command:

Example

[fred@basecm11 ~]$ module list
Currently Loaded Modulefiles:
 1) shared   2) cmsh   3) cmd   4) cluster-tools/11   5) cm-setup/11   6) gcc/14.2.0

If there are no modules loaded by default, then the module list command just returns nothing.

How does a user know what modules are available? The “module avail” command lists all modules that are available for loading (some output elided):

Example

fred@basecm11:~$ module avail
---------------------------- /cm/local/modulefiles -----------------------------
boost/1.81.0              dot              module-git   python312
cluster-tools/11.0        freeipmi/1.6.14  module-info  rocm-smi/4.3.0
cm-bios-tools             gcc/14.2.0       modules      sedutil/1.16.0
cm-image/11             gdb/16.2         mpc/1.3.1    shared
cm-nvfwupd/2.1.2          gmp/6.3.0        mpfr/4.2.1   slurm/slurm/25.11
cm-scale/cm-scale.module  ipmitool/1.8.19  null         use.own
cm-setup/11             lua/5.4.7        openldap
cmd                       luajit           prs
cmsh                      mariadb-libs     python3
---------------------------- /cm/shared/modulefiles ----------------------------
cm-pmix3/3.1.7       hpl/2.3        mvapich2/gcc/64/2.3.7    ucx/1.18.0
cm-pmix4/4.1.3       hwloc/1.11.13  openblas/dynamic/0.3.28
default-environment  hwloc2/2.8.0   openmpi/gcc/64/4.1.5
hdf5_18/1.8.21       iperf/3.17.1   openmpi4/gcc/4.1.5

In the list there are two kinds of modules:

• local modules, which are specific to the node, or head node only

• shared modules, which are made available from a shared storage, and which only become available for loading after the shared module is loaded.

Modules can be loaded using the add or load options. A list of modules can be added by spacing them:

Example

[fred@basecm11 ~]$ module add shared gcc openmpi/gcc

Tab completion works for suggesting modules for the add/load commands. If the tab completion suggestion is unique, even though it is not the full path, then it is still enough to specify the module. For example, looking at the possibile available modules listed by the avail command previously, it turns out that specifying gcc is enough to specify gcc/14.2.0 because there is no other directory path under gcc/ besides 14.2.0 anyway.

To remove one or more modules, the module unload or module rm command is used.

To remove all modules from the user’s environment, the module purge command is used.

The user should be aware that some loaded modules can conflict with others loaded at the same time. This can happen with MPI modules. For example, loading openmpi/gcc without removing an already loaded intel/mpi/64 can result in conflicts about which compiler should be used.

2.3.2.1 The shared Module

The shared module provides access to shared libraries. By default these are under /cm/shared.

The shared module is special because often other modules, as seen under /cm/shared/modulefiles, depend on it. So, if it is to be loaded, then it is usually loaded first, so that the dependent modules can use it.

The shared module is obviously a useful local module, and is therefore often configured to be loaded for the user by default. Setting the default environment modules is discussed in section 2.3.3.

2.3.3 Changing The Default Environment Modules

.

If a user has to manually load up the same modules every time upon login it would be inefficient. That is why an initial default state for modules can be set up by the user, by using the module init* subcommands:

The more useful ones of these are:

• module initadd: add a module to the initial state

• module initrm: remove a module from the initial state

• module initlist: list all modules loaded initially

• module initclear: clear all modules from the list of modules loaded initially

Example

[fred@basecm11 ~]$ module initclear
[fred@basecm11 ~]$ module initlist
bash initialization file $HOME/.bashrc loads modules:
[fred@basecm11 ~]$ module initadd shared gcc openmpi/gcc
[fred@basecm11 ~]$ module initlist
bash initialization file $HOME/.bashrc loads modules:
   shared gcc openmpi/gcc

In the preceding example, the modules defined for the new initial environment for the user are loaded from the next login onward.

Example

[fred@basecm11 ~]$ module list
No Modulefiles Currently Loaded.
[fred@basecm11 ~]$ exit
logout
Connection to basecm11 closed
[root@basejumper ~]# ssh fred@basecm11
fred@basecm11's password:
...
[fred@basecm11 ~]$ module list
Currently Loaded Modulefiles:
 1) shared   2) gcc/9.2.0   3) openmpi/gcc/64/1.10.7
[fred@basecm11 ~]$

If the user is unsure about what the module does, it can be checked using “module whatis”:

$ module whatis openmpi/gcc
----------------------------------- /cm/shared/modulefiles ------------------------------------
openmpi/gcc/64/1.10.7: adds OpenMPI to your environment variables

The man pages for module and modulefile give further details on usage.

2.4 Compiling Applications

Compiling an application is usually done on the head node or login node. Typically, there are several compilers available on the head node in BCM. These compilers provide different levels of optimization, standards conformance, and support for accelerators.

For example: The GNU compiler collection, Intel compilers, and the NVIDIA HPC SDK compilers. The following table summarizes the available compiler commands on a cluster with these compilers:

GNU compilers are the de facto standard on Linux and are installed by default. They are provided under the terms of the GNU General Public License. Compilers by NVIDIA HPC and Intel are available as packages via the BCM YUM repository, and require the purchase of a license to use them. To make a compiler available to be used in a user’s shell commands, the appropriate environment module (section 2.3) must be loaded first. On most clusters two versions of GCC are available:

1. The version of GCC that comes along with the Linux distribution. For example, for Ubuntu 24.04:

   Example

   fred@basecm11:~$ which gcc; gcc --version | head -1
   /usr/bin/gcc
   gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0

2. The latest version suitable for general use that is packaged as a module by BCM:

   Example

   fred@basecm11:~$ module load gcc
   fred@basecm11:~$ which gcc; gcc --version | head -1
   /cm/local/apps/gcc/14.2.0/bin/gcc
   gcc (GCC) 14.2.0

To use the latest version of GCC, the gcc module in a default BCM cluster must be loaded. To revert to the version of GCC that comes natively with the Linux distribution, the gcc module must be unloaded.

The compilers—GCC, Intel, NVIDIA HPC—in the preceding table are ordinarily used for applications that run on a single node. However, the applications used may fork, thread, and run across as many nodes and processors as they can access if the application is designed that way.

The standard, structured way of running applications in parallel is to use the MPI-based libraries (Chapter 3). These are the MPICH, MVAPIC, Open MPI, and Intel MPI libraries. The libraries link to the underlying compilers of the preceding table.

If the parallel library environment has been loaded, then the following MPI compiler commands automatically use the underlying compilers:

2.4.1 Open MPI And Mixing Compilers

BCM comes with multiple Open MPI packages corresponding to the different available compilers. However, sometimes mixing compilers is desirable. For example, C-compilation may be preferred using icc from Intel, while Fortran90-compilation may be preferred using gfortran from the GNU Project. In such cases it is possible to override the default compiler path environment variable, for example:

[fred@basecm11 ~]$ module list
Currently Loaded Modulefiles:
 1) shared   2) gcc/9.2.0   3) openmpi/gcc/64/1.10.7
[fred@basecm11 ~]$ mpicc --version --showme; mpif90 --version --showme
gcc --version
gfortran --version
[fred@basecm11 ~]$ export OMPI_CC=icc; export OMPI_FC=openf90
[fred@basecm11 ~]$ mpicc --version --showme; mpif90 --version --showme
icc --version
openf90 --version

Variables that may be set are OMPI_CC, OMPI_CXX, OMPI_FC, and OMPI_F77. More on overriding the Open MPI wrapper settings is documented in the man pages of mpicc in the environment section.