> This page is for version 10.
> For other versions, use one of these documentation indexes:
> - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt
> - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server.

# Using Apptainer

Apptainer is a tool that allows applications to be packaged and run in containers. Apptainer containers can include a simple binary and library stack, or a complicated work flow. The packaged applications are then called Apptainer images. These images are completely portable, with their only dependency requirement being that Apptainer must be running on the target system.

NVIDIA Base Command Manager provides the `cm-apptainer` package. This allows users to create container images, which are run in containers. The containers can also be used with MPI applications and with different workload managers. The containers run within user space and are always executed as the run-time user (non-root). The application within the container may have access to the filesystem inside or outside the container. The package also provides a local module file called `apptainer` that must be loaded, before using the `apptainer` command line utility or running a Apptainer container.

In this chapter several examples are given of how to create container images and start Apptainer containers.

## 11.1  How To Build A Simple Container Image

The Apptainer image is a single file which physically contains the container filesystem. Apptainer container images can be bootstrapped with a *definition file*. The definition file describes how the container is to be built. There are several Apptainer keywords (parameters) in the definition file, laid out in lines. If a line does not contain a Apptainer keyword, then the line is treated as a shell script line.

The main keywords in the definition file are:

* `BootStrap`: The name of the Linux distribution type module. This informs Apptainer which distribution module should be used to parse the commands in the definition file. At present the following 4 modules are supported:

  * `yum`: Bootstraps distributions such as Red Hat, Centos, and Scientific Linux.

  * `debootstrap`: Bootstraps Debian- and Ubuntu-based distributions.

  * `arch`. Bootstraps Arch Linux.

  * `docker`. Bootstraps Docker. It creates a core operating system image based on an image hosted on a particular Docker Registry server. For the `docker` module, several other keywords may also be defined:

    * `From`: this keyword defines the string of the registry name used for this image in the format `[`*`name`*`]:[`*`version`*`]`.

    * `IncludeCmd`: use the Docker-defined Cmd as the %runscript, if the Cmd is defined,

    * `From`: sets the docker registry name,

    * `Token`: sets the docker authorization token.

* `MirrorURL`: the URL to get packages from.

* `OSVersion`: this keyword must be defined as the alphabet-character string associated with the version of the distribution you wish to use. For example: `trusty` or `stable`.

* `Include`: install additional packages.

* `%setup`: this section blob is a bash scriptlet that is executed on the host outside the container, during bootstrapping.

* `%post`: this scriptlet section is executed once from inside the container, during bootstrapping.

* `%runscript`: the scriptlet that is executed inside the container when it is started.

* `%test`: this section is run at the very end of the boostrapping process and validates the container during the bootstrap process.

A very simple image definition file for Debian 7 (Wheezy) can look like this:

**Example**

```bash
[root@basecm10 ~]# cat debian7.def
Bootstrap: docker
From: debian:7
%post
  echo "ready"
%runscript
  echo "Hello from container!"
  cat /etc/os-release
```

Once the definition file is ready then a user can run the Apptainer command `build` to create and install the defined operating system into the container image.

**Example**

```bash
[root@basecm10 ~]# mkdir /cm/shared/appt-images
[root@basecm10 ~]# apptainer build /cm/shared/appt-images/debian7.img debian7.def
INFO:    Starting build...
Getting image source signatures
...
Writing manifest to image destination
Storing signatures
2024/12/20 16:14:19  info unpack layer: sha256:2b15b7abe8b39a409a8b29c5ce62b22ce091102ca25fbf49aa877cca40983717
INFO:    Running post scriptlet
+ echo ready
ready
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: /cm/shared/appt-images/debian7.img
[root@basecm10 ~]#
```

The image must be built by a privileged user, even if it is always supposed to be executed with regular user permissions. If, as is the usual case, users are not allowed to use root permissions on a cluster, then they can build the new image on their own computers. The image can then be transfered to the cluster.

The image can be placed in any directory, but it makes sense to share it among compute nodes. The directory `/cm/shared/appt-images` is therefore a sensible location for keeping the container images.

The image created can then be executed as a regular binary:

**Example**

```bash
[user@basecm10 ~]$ module load apptainer
[user@basecm10 ~]$ /cm/shared/appt-images/debian7.img
Hello from container!
PRETTY_NAME="Debian GNU/Linux 7 (wheezy)"
NAME="Debian GNU/Linux"
VERSION_ID="7"
VERSION="7 (wheezy)"
ID=debian
ANSI_COLOR="1;31"
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support/"
BUG_REPORT_URL="http://bugs.debian.org/"
[user@basecm10 ~]$
```

The following example shows an image definition file that adds the `/etc/services` file and `/bin/grep` binary from the filesystem of the host, to the container image. When a user runs the image that is created, it `grep`s the services file for arguments passed to the image:

**Example**

```bash
[root@basecm10 ~]# cat grep.def
BootStrap: yum
OSVersion: 7
MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
Include: grep
%setup
  cp /etc/services ${APPTAINER_ROOTFS}/etc/services
%runscript
  exec /bin/grep $@ /etc/services

[root@basecm10 ~]# apptainer build /cm/shared/appt-images/grep.img grep.def
INFO:    Starting build...
INFO:    Skipping GPG Key Import
INFO:    Running setup scriptlet
+ cp /etc/services /tmp/sbuild-102714887/fs/etc/services
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: /cm/shared/appt-images/grep.img
```

The container can now be run as if it were a simple script. If a string is passed as an argument, the `/etc/services` file that is packaged inside the container is searched for the string:

**Example**

```bash
[user@basecm10 ~]$ /cm/shared/appt-images/grep.img telnets
telnets         992/tcp
telnets         992/udp
[user@basecm10 ~]$
```

## 11.2  Using MPI

MPI applications can be run from a Apptainer container according to two models:

1. the Hybrid model: this relies on both the host and the container to provide MPI implementations for the application that is to run. (This is sometimes also called the Host MPI model).

2. the Bind model: this relies on only the host MPI implementation. The container itself has no MPI implementation, but bind mounts the host MPI implementation into the container, which tends to keep the container lighter.

This is discussed in the upstream Apptainer MPI documentation at [https://apptainer.org/docs/user/latest/mpi.html](https://apptainer.org/docs/user/latest/mpi.html).

BCM administrators typically keep the system software synchronized at the same version across all nodes, and use `/cm/shared/` and `/home` as a shared directory. Containers placed on the nodes therefore usually make use of the shared, synchronized software, which means that in BCM, bind mounting is mostly not needed.

A container can be defined and built on the head node for testing purposes. After adding an MPI implementation, the container can be placed onto the node image, and execution can be carried out via a workload manager such as Slurm.

### 11.2.1  Runnning An MPI Example

Apptainer at the time of writing (December 2024) supports the two main MPI implementations: OpenMPI ([https://www.open-mpi.org/](https://www.open-mpi.org/)), and MPICH ([https://www.mpich.org/](https://www.mpich.org/)).

This section shows a simple example of how an MPICH and OpenMPI run can be carried out.

#### 11.2.1.1  Container Definitions

**MPICH container definition:** An example of a container definition for MPICH is:

```bash
[root@basecm10 ~]# cat mpich.def
BootStrap: yum
OSVersion: 7
MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
Include: yum
%post
  echo "Installing extra packages..."
  yum install tcl -y
  yum install less -y
  yum install which -y
  yum install cm-config-yum -y
  yum install env-modules -y
  yum install gcc-recent -y
  yum install libgfortran -y
  yum install mpich-ge-gcc-64 -y
%files
#  <source> [<destination>]
   /etc/yum.repos.d/cm.repo
   /etc/pki/rpm-gpg/RPM-GPG-KEY-cm
%runscript
#run when container runs
# takes arguments after container name when container is run
  echo "Hello from container!"
  cat /etc/os-release
  echo "loaded modules are:"
  module list
  echo "available modules are:"
  module avail
  module load gcc
  module load mpich/ge/gcc/
%setup
#runs on host, as root
#  touch $APPTAINER_ROOTFS/var/log/yum.log
[root@basecm10 ~]#
```

The `%files` section takes care of some binding that is needed for the container to install the extra YUM packages in the `%post` section.

**OpenMPI container definition:** Similarly, an example of a container definition for OpenMPI could be:

```bash
BootStrap: yum
OSVersion: 7
MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
Include: yum
%post
  echo "Installing extra packages..."
  yum install tcl -y
  yum install less -y
  yum install cm-config-yum -y
  yum install env-modules -y
#  yum install openmpi -y
  yum install openmpi-geib-gcc-64 -y
  module load gcc
  module load openmpi
%files
#  <source> [<destination>]
   /etc/yum.repos.d/cm.repo
   /etc/pki/rpm-gpg/RPM-GPG-KEY-cm
%runscript
  echo "Hello from container!"
  cat /etc/os-release
  echo "loaded modules are:"
  module list
  echo "available modules are:"
  module avail
  echo "repos are:"
```

#### 11.2.1.2  Building The Container

If the directory `/cm/shared/appt-images` was created earler on, then:

* For MPICH, a Apptainer image can then be built by the `root` user with:

  ```bash
  apptainer build /cm/shared/appt-images/mpich.img mpich.def
  ```

* Similarly, for OpenMPI, a Apptainer image can be built by the `root` user with:

  ```bash
  apptainer build /cm/shared/appt-images/openmpi.img openmpi.def
  ```

#### 11.2.1.3  Execution With A Container On The Head Node For Testing

A hello world binary can be run.

The hello world example from Chapter [3](/user/mpi) was:

```bash
#include <stdio.h>
#include <mpi.h>
int main (int argc, char *argv[])
{
    int id, np, i;
    char processor_name[MPI_MAX_PROCESSOR_NAME];
    int processor_name_len;
    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &np);
    MPI_Comm_rank(MPI_COMM_WORLD, &id);
    MPI_Get_processor_name(processor_name, &processor_name_len);
    for(i=1;i<2;i++)
    {printf(
    "Hello world from process %03d out of %03d, processor name %s\n",
    id, np, processor_name
    );}
    MPI_Finalize();
    return 0;
}
```

This can be compiled by a user (`fred` in this case), after loading the appropriate MPICH or OpenMPI module:

**Example**

```bash
[fred@basecm10 ~]$ module load mpich/ge/gcc/64/     # or module load openmpi/gcc/64/ as appropriate
[fred@basecm10 ~]$ mpicc hello.c -o hello
```

For an MPICH implemented by the administrator on the regular nodes, the user can run the binary with a machine file (containing `node001` and `node002` here) with:

**Example**

```bash
[fred@basecm10 ~]$ module rm openmpi
[fred@basecm10 ~]$ module load mpich/ge/gcc/64/
[fred@basecm10 ~]$ mpirun --map-by slot -machinefile mpirun.hosts /home/fred/hello
Hello world from process 000 out of 002, processor name node001
Hello world from process 001 out of 002, processor name node002
```

For an OpenMPI implemented by the administrator, the corresponding run is:

**Example**

```bash
[fred@basecm10 ~]$ module rm mpich/ge/gcc/64/3.3.2
[fred@basecm10 ~]$ module load openmpi/gcc/64/1.10.7
[fred@basecm10 ~]$ /cm/shared/apps/openmpi/gcc/64/1.10.7/bin/mpirun --mca btl tcp,self --map-by slot\
 -machinefile mpirun.hosts /home/fred/hello
Hello world from process 001 out of 002, processor name node002
Hello world from process 000 out of 002, processor name node001
```

The `--mca btl tcp,self` option avoids warnings being displayed when Ethernet is used for MPI. The option can be dropped if InfiniBand has been configured by the cluster administrator.

#### 11.2.1.4  Running A Binary Via MPI From A Apptainer Image

The MPICH or OpenMPI binary can be run using Apptainer with the MPICH or OpenMPI container image on the head node: **Running a binary via MPI from an MPICH Apptainer image:**

**Example**

```bash
[fred@basecm10 ~]$ module rm openmpi/gcc/64/1.10.7
[fred@basecm10 ~]$ module load mpich/ge/gcc/64/
[fred@basecm10 ~]$ mpirun -np 4 -machinefile mpirun.hosts apptainer exec\
 /cm/shared/appt-images/mpich.img /home/fred/hello
Hello world from process 001 out of 002, processor name node002
Hello world from process 000 out of 002, processor name node001
```

**Running a binary via MPI from an OpenMPI Apptainer image:**

**Example**

```bash
[fred@basecm10 ~]$ module load openmpi/gcc/64/1.10.7
[fred@basecm10 ~]$ /cm/shared/apps/openmpi/gcc/64/1.10.7/bin/mpirun --mca btl tcp,self --map-by slot \
 -machinefile mpirun.hosts apptainer exec /cm/shared/appt-images/openmpi.img /home/fred/hello
Hello world from process 001 out of 002, processor name node002
Hello world from process 000 out of 002, processor name node001
[fred@basecm10 ~]$
```

#### 11.2.1.5  Execution Via Workload Manager On Regular Nodes With Slurm

If the user `fred` is meant to submit MPI jobs via a workload manager, such as Slurm, then a standard way to execute them is to create a job script (Chapter [7](/admin/workloadmanagement)). With Slurm, `srun` is generally recommended as an optimal way to run MPI tasks, in preference to `mpirun`.

A simple job script for the MPICH image is:

**Example**

```bash
[fred@basecm10 ~]$ cat slurm.job
#!/bin/bash
#SBATCH --ntasks=2
module load slurm
module load mpich/ge/gcc
srun apptainer exec /cm/shared/appt-images/mpich.img /home/fred/hello
[fred@basecm10 ~]$ module load slurm
[fred@basecm10 ~]$ sbatch slurm.job
Submitted batch job 2
[fred@basecm10 ~]$ cat slurm-2.out
Hello world from process 001 out of 002, processor name node002
Hello world from process 000 out of 002, processor name node001
```

A similar run with OpenMPI is:

```bash
[fred@basecm10 ~]$ cat slurm.job
#!/bin/bash
#SBATCH --ntasks=2
module load slurm
module load openmpi
module load apptainer
srun apptainer exec /cm/shared/appt-images/openmpi.img /home/fred/hello
[fred@basecm10 ~]$ module load slurm
[fred@basecm10 ~]$ sbatch slurm.job
Submitted batch job 8
[fred@basecm10 ~]$ cat slurm-8.out
Hello world from process 001 out of 002, processor name node002
Hello world from process 000 out of 002, processor name node001
```