> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Cluster Usage ## 2.1 Login To The Cluster Environment The login node is the node where the user logs in and works from. Simple clusters have a single login node, but large clusters sometimes have multiple login nodes to improve the availability of the cluster. In most clusters, the login node is also the head node from where the cluster is monitored and installed. On the login node: * applications can be developed * code can be compiled and debugged * applications can be submitted to the cluster for execution * running applications can be monitored To carry out an ssh login to the cluster, a terminal session can be started from Unix-like operating systems: **Example** ```bash $ ssh myname@cluster.hostname ``` On a Windows operating system, an SSH client such as PuTTY ([http://www.putty.org](http://www.putty.org)) can be downloaded. Another standard possibility is to run a Unix-like environment such as Cygwin ([http://www.cygwin.com](http://www.cygwin.com)) within the Windows operating system, and then run the SSH client from within it. A Mac OS X user can use the Terminal application from the Finder, or under `Application/Utilities/Terminal.app`. An X11 windowing environment must be installed for it to work. XQuartz is the recommended X11 windowing environment, and is indeed the only Apple-backed X11 version available from OS X 10.8 onward. When using the SSH connection, the cluster’s address must be added. When the connection is made, a username and password must be entered at the prompt. If the administrator has changed the default SSH port from 22 to something else, the port can be specified with the `-p ` option: ```bash $ ssh -X -p @ ``` The -X option can be dropped if no X11-forwarding is required. X11-forwarding allows a GUI application from the cluster to be displayed locally. Optionally, after logging in, the password used can be changed using the `passwd` command: ```bash $ passwd ``` ## 2.2 Setting Up The User Environment By default, each user uses the bash shell interpreter. In that case, each time a user login takes place, a file named `.bashrc` is executed to set up the shell environment for the user. The shell and its environment can be customized to suit user preferences. For example, * the prompt can be changed to indicate the current username, host, and directory, for example: by setting the prompt string variable: ```bash PS1='[\u@\h \W]\$ ' ``` * the size of the command history file can be increased, for example: `export HISTSIZE=100` * aliases can be added for frequently used command sequences, for example: `alias lart='ls -alrt'` * environment variables can be created or modified, for example: `export MYVAR=MYSTRING` * the location of software packages and versions that are to be used by a user (the path to a package) can be set. Because there is a huge choice of software packages and versions, it can be hard to set up the right environment variables and paths for software that is to be used. Collisions between different versions of the same package and non-matching dependencies on other packages must also be avoided. To make setting up the environment easier, BCM provides preconfigured environment modules (section [2.3](#23-environment-modules)). ## 2.3 Environment Modules For a user to compile and run computational jobs on a cluster, a special shell environment is typically set up for the software that is used. However, setting up the right environment for a particular software package and version can be tricky, and it can be hard to keep track of how it was set up. For example, users want to have a clean way to bring up the right environment for compiling code according to the various MPI implementations, but can easily get confused about which libraries have been used, and can end up with multiple libraries with similar names installed in a disorganized manner. A user might also like to conveniently test new versions of a software package before permanently installing the package. Within a Linux distribution running without special utilities, setting up environments can be complex. However, BCM makes use of the environment modules package, which provides the `module` command. The `module` command is a special utility to make taking care of the shell environment much easier. ### 2.3.1 Available Commands Practical use of the `modules` commands is given in sections [2.3.2](#232-managing-environment-modules-as-a-user) and [2.3.3](#233-changing-the-default-environment-modules). For reference, the help text for the `module` command can be viewed as follows: **Example** ```bash [me@cluster ~]$ module --help Modules Release 4.5.3 (2020-08-31) Usage: module [options] [command] [args ...] Loading / Unloading commands: add | load modulefile [...] Load modulefile(s) rm | unload modulefile [...] Remove modulefile(s) purge Unload all loaded modulefiles reload | refresh Unload then load all loaded modulefiles switch | swap [mod1] mod2 Unload mod1 and load mod2 Listing / Searching commands: list [-t|-l|-j] List loaded modules avail [-d|-L] [-t|-l|-j] [-S|-C] [--indepth|--no-indepth] [mod ...] List all or matching available modules aliases List all module aliases whatis [-j] [modulefile ...] Print whatis information of modulefile(s) apropos | keyword | search [-j] str Search all name and whatis containing str is-loaded [modulefile ...] Test if any of the modulefile(s) are loaded is-avail modulefile [...] Is any of the modulefile(s) available info-loaded modulefile Get full name of matching loaded module(s) Collection of modules handling commands: save [collection|file] Save current module list to collection restore [collection|file] Restore module list from collection or file saverm [collection] Remove saved collection saveshow [collection|file] Display information about collection savelist [-t|-l|-j] List all saved collections is-saved [collection ...] Test if any of the collection(s) exists Shell's initialization files handling commands: initlist List all modules loaded from init file initadd modulefile [...] Add modulefile to shell init file initrm modulefile [...] Remove modulefile from shell init file initprepend modulefile [...] Add to beginning of list in init file initswitch mod1 mod2 Switch mod1 with mod2 from init file initclear Clear all modulefiles from init file Environment direct handling commands: prepend-path [-d c] var val [...] Prepend value to environment variable append-path [-d c] var val [...] Append value to environment variable remove-path [-d c] var val [...] Remove value from environment variable Other commands: help [modulefile ...] Print this or modulefile(s) help info display | show modulefile [...] Display information about modulefile(s) test [modulefile ...] Test modulefile(s) use [-a|-p] dir [...] Add dir(s) to MODULEPATH variable unuse dir [...] Remove dir(s) from MODULEPATH variable is-used [dir ...] Is any of the dir(s) enabled in MODULEPATH path modulefile Print modulefile path paths modulefile Print path of matching available modules clear [-f] Reset Modules-specific runtime information source scriptfile [...] Execute scriptfile(s) config [--dump-state|name [val]] Display or set Modules configuration Switches: -t | --terse Display output in terse format -l | --long Display output in long format -j | --json Display output in JSON format -d | --default Only show default versions available -L | --latest Only show latest versions available -S | --starts-with Search modules whose name begins with query string -C | --contains Search modules whose name contains query string -i | --icase Case insensitive match -a | --append Append directory to MODULEPATH -p | --prepend Prepend directory to MODULEPATH --auto Enable automated module handling mode --no-auto Disable automated module handling mode -f | --force By-pass dependency consistency or confirmation dialog Options: -h | --help This usage info -V | --version Module version -D | --debug Enable debug messages -v | --verbose Enable verbose messages -s | --silent Turn off error, warning and informational messages --paginate Pipe mesg output into a pager if stream attached to terminal --no-pager Do not pipe message output into a pager --color[=WHEN] Colorize the output; WHEN can be 'always' (default if omitted), 'auto' or 'never' ``` ### 2.3.2 Managing Environment Modules As A User There is a good chance the cluster administrator has set up the user’s account, `fred` for example, so that some modules are loaded already by default. In that case, the modules loaded into the user’s environment can be seen with the `module list` command: **Example** ```bash [fred@basecm11 ~]$ module list Currently Loaded Modulefiles: 1) shared 2) cmsh 3) cmd 4) cluster-tools/11 5) cm-setup/11 6) gcc/14.2.0 ``` If there are no modules loaded by default, then the `module list` command just returns nothing. How does a user know what modules are available? The “`module avail`” command lists all modules that are available for loading (some output elided): **Example** ```bash fred@basecm11:~$ module avail ---------------------------- /cm/local/modulefiles ----------------------------- boost/1.81.0 dot module-git python312 cluster-tools/11.0 freeipmi/1.6.14 module-info rocm-smi/4.3.0 cm-bios-tools gcc/14.2.0 modules sedutil/1.16.0 cm-image/11 gdb/16.2 mpc/1.3.1 shared cm-nvfwupd/2.1.2 gmp/6.3.0 mpfr/4.2.1 slurm/slurm/25.11 cm-scale/cm-scale.module ipmitool/1.8.19 null use.own cm-setup/11 lua/5.4.7 openldap cmd luajit prs cmsh mariadb-libs python3 ---------------------------- /cm/shared/modulefiles ---------------------------- cm-pmix3/3.1.7 hpl/2.3 mvapich2/gcc/64/2.3.7 ucx/1.18.0 cm-pmix4/4.1.3 hwloc/1.11.13 openblas/dynamic/0.3.28 default-environment hwloc2/2.8.0 openmpi/gcc/64/4.1.5 hdf5_18/1.8.21 iperf/3.17.1 openmpi4/gcc/4.1.5 ``` In the list there are two kinds of modules: * **local modules**, which are specific to the node, or head node only * **shared modules**, which are made available from a shared storage, and which only become available for loading after the `shared` module is loaded. Modules can be loaded using the `add` or `load` options. A list of modules can be added by spacing them: **Example** ```bash [fred@basecm11 ~]$ module add shared gcc openmpi/gcc ``` Tab completion works for suggesting modules for the `add`/`load` commands. If the tab completion suggestion is unique, even though it is not the full path, then it is still enough to specify the module. For example, looking at the possibile available modules listed by the `avail` command previously, it turns out that specifying `gcc` is enough to specify `gcc/14.2.0` because there is no other directory path under `gcc/` besides `14.2.0` anyway. To remove one or more modules, the `module unload` or `module rm` command is used. To remove all modules from the user’s environment, the `module purge` command is used. The user should be aware that some loaded modules can conflict with others loaded at the same time. This can happen with MPI modules. For example, loading `openmpi/gcc` without removing an already loaded `intel/mpi/64` can result in conflicts about which compiler should be used. #### 2.3.2.1 The `shared` Module The `shared` module provides access to shared libraries. By default these are under `/cm/shared`. The `shared` module is special because often other modules, as seen under `/cm/shared/modulefiles`, depend on it. So, if it is to be loaded, then it is usually loaded first, so that the dependent modules can use it. The `shared` module is obviously a useful local module, and is therefore often configured to be loaded for the user by default. Setting the default environment modules is discussed in section [2.3.3](#233-changing-the-default-environment-modules). ### 2.3.3 Changing The Default Environment Modules . If a user has to manually load up the same modules every time upon login it would be inefficient. That is why an initial default state for modules can be set up by the user, by using the `module init*` subcommands: The more useful ones of these are: * `module initadd`: add a module to the initial state * `module initrm`: remove a module from the initial state * `module initlist`: list all modules loaded initially * `module initclear`: clear all modules from the list of modules loaded initially **Example** ```bash [fred@basecm11 ~]$ module initclear [fred@basecm11 ~]$ module initlist bash initialization file $HOME/.bashrc loads modules: [fred@basecm11 ~]$ module initadd shared gcc openmpi/gcc [fred@basecm11 ~]$ module initlist bash initialization file $HOME/.bashrc loads modules: shared gcc openmpi/gcc ``` In the preceding example, the modules defined for the new initial environment for the user are loaded from the next login onward. **Example** ```bash [fred@basecm11 ~]$ module list No Modulefiles Currently Loaded. [fred@basecm11 ~]$ exit logout Connection to basecm11 closed [root@basejumper ~]# ssh fred@basecm11 fred@basecm11's password: ... [fred@basecm11 ~]$ module list Currently Loaded Modulefiles: 1) shared 2) gcc/9.2.0 3) openmpi/gcc/64/1.10.7 [fred@basecm11 ~]$ ``` If the user is unsure about what the module does, it can be checked using “`module whatis`”: ```bash $ module whatis openmpi/gcc ----------------------------------- /cm/shared/modulefiles ------------------------------------ openmpi/gcc/64/1.10.7: adds OpenMPI to your environment variables ``` The `man` pages for `module` and `modulefile` give further details on usage. ## 2.4 Compiling Applications Compiling an application is usually done on the head node or login node. Typically, there are several compilers available on the head node in BCM. These compilers provide different levels of optimization, standards conformance, and support for accelerators. For example: The GNU compiler collection, Intel compilers, and the NVIDIA HPC SDK compilers. The following table summarizes the available compiler commands on a cluster with these compilers:
**Language** **GNU** **NVIDIA HPC** **Intel**
C `gcc` nvc and nvcc `icc`
C++ `g++` nvc++ and nvcc `icc`
Fortran77 `gfortran` nvfortran `ifort`
Fortran90 `gfortran` nvfortran `ifort`
Fortran95 `gfortran` nvfortran `ifort`
GNU compilers are the de facto standard on Linux and are installed by default. They are provided under the terms of the GNU General Public License. Compilers by NVIDIA HPC and Intel are available as packages via the BCM YUM repository, and require the purchase of a license to use them. To make a compiler available to be used in a user’s shell commands, the appropriate environment module (section [2.3](#23-environment-modules)) must be loaded first. On most clusters two versions of GCC are available: 1. The version of GCC that comes along with the Linux distribution. For example, for Ubuntu 24.04: **Example** ```bash fred@basecm11:~$ which gcc; gcc --version | head -1 /usr/bin/gcc gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 ``` 2. The latest version suitable for general use that is packaged as a module by BCM: **Example** ```bash fred@basecm11:~$ module load gcc fred@basecm11:~$ which gcc; gcc --version | head -1 /cm/local/apps/gcc/14.2.0/bin/gcc gcc (GCC) 14.2.0 ``` To use the latest version of GCC, the `gcc` module in a default BCM cluster must be loaded. To revert to the version of GCC that comes natively with the Linux distribution, the `gcc` module must be unloaded. The compilers—GCC, Intel, NVIDIA HPC—in the preceding table are ordinarily used for applications that run on a single node. However, the applications used may fork, thread, and run across as many nodes and processors as they can access if the application is designed that way. The standard, structured way of running applications in parallel is to use the MPI-based libraries (Chapter [3](/user/mpi)). These are the MPICH, MVAPIC, Open MPI, and Intel MPI libraries. The libraries link to the underlying compilers of the preceding table. If the parallel library environment has been loaded, then the following MPI compiler commands automatically use the underlying compilers:
**Language** **C** **C++** **Fortran77** **Fortran90**
**Command** `mpicc ` `mpicxx` `mpif77` `mpif90`
### 2.4.1 Open MPI And Mixing Compilers BCM comes with multiple Open MPI packages corresponding to the different available compilers. However, sometimes mixing compilers is desirable. For example, C-compilation may be preferred using `icc` from Intel, while Fortran90-compilation may be preferred using `gfortran` from the GNU Project. In such cases it is possible to override the default compiler path environment variable, for example: ```bash [fred@basecm11 ~]$ module list Currently Loaded Modulefiles: 1) shared 2) gcc/9.2.0 3) openmpi/gcc/64/1.10.7 [fred@basecm11 ~]$ mpicc --version --showme; mpif90 --version --showme gcc --version gfortran --version [fred@basecm11 ~]$ export OMPI_CC=icc; export OMPI_FC=openf90 [fred@basecm11 ~]$ mpicc --version --showme; mpif90 --version --showme icc --version openf90 --version ``` Variables that may be set are OMPI_CC, OMPI_CXX, OMPI_FC, and OMPI_F77. More on overriding the Open MPI wrapper settings is documented in the man pages of `mpicc` in the environment section.