> This page is for version 10.
> For other versions, use one of these documentation indexes:
> - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt
> - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server.

# Apptainer

BCM provides an application containerization tool called Apptainer. Apptainer is designed to execute containers as if they are just native applications on a host computer, and to work with HPC. Apptainer users can therefore run Apptainer containers just as they run any other program on an HPC cluster.

## 9.1  Use Cases

Adding Apptainer to BCM brings a stronger integration of containerization with HPC. While Docker and Kubernetes can work within HPC, some drawbacks still prevent the use of HPC resources in the way that HPC users and administrators are used to.

Besides the use of Apptainer containers in HPC jobs, Apptainer users can create portable images with their applications. Apptainer images are files that represent the container filesystem. These images can be copied from one environment (cluster) to another and executed without modification. Thus, when a user creates a container image file, it is up to the user what files, or which RPMs, to install in the image. For example, the user can create an image file that bundles Open MPI with the user’s application. This guarantees that the application will be able to run if it requires that MPI implementation, even if no MPI libraries are installed on the execution host or if there is some version incompatibility.

There is no need for a special configuration inside workload managers in order to use Apptainer. This is because the containers are designed to be run like any application on the system. Users need just to use the image file as the usual script or binary to be executed in their jobscripts or in a shell. The `apptainer` command can also be used to apply special options to the container, when executing the image file in the jobscript or shell.

## 9.2  Package `cm-apptainer`

Apptainer is packaged and distributed as a package called `cm-apptainer`. The package should be installed by the cluster administrator in the software image for each node. The user is able to run a Apptainer image only if the Apptainer package is installed on the node. In order to allow users to build an image, it makes sense to install the package on the head and login nodes as well. The tool does not provide services that run in the background, so a simple installation of the package is enough to start using it.

Apptainer contexts are always run as the user running them. This means that there is no risk in allowing the containers to have access to, and interact with, the file system of the host.

This means that, if an image is created by the root user on a machine, then the files that require root access inside the image, still need to be allowed root permissions on any other machine. Thus, if a user crates an image on a laptop, and adds a file that can be read only by the root user, then when the container is started on another machine by a regular user, that regular user has no access to the root-only readable file inside the container.

While there is no daemon running as root, nor any persistent processes that an attacker may use to escalate privileges, there is a need to run some system calls as root so that the container is encapsulated. For this part of the run flow, there is a single SUID binary called Sexec (Apptainer Exec). This is a simple binary that is as small as possible, and which the Apptainer developers claim has been audited by multiple security experts.

## 9.3  MPI Integration

Because of the nature of Apptainer, all MPI implementations should work fine inside a Apptainer container. The developers of the tool have spent a lot of effort in making Apptainer aware of Open MPI, as well as adding a Apptainer module into Open MPI so that running at extreme scale is as efficient as possible. However, in some cases, starting an MPI process may not be as optimal as execution outside the container. So, specifically for Open MPI, Apptainer provides a special mechanism to handle the execution of MPI processes. It adds all the MPI processes of the same MPI application to the same container on a host. This also reduces the application startup time. The Open MPI daemon `orted` in this case is not added to the running container, which means the overhead of starting up daemons is reduced.

When an Open MPI application that has been packaged to an image is started, the following steps take place:

1. `mpirun` is called;

2. `mpirun` forks and executes `orted`;

3. `orted` initializes the PMI (process management interface);

4. `orted` forks as many times as the number of processes per node requested;

5. the container image is started in each fork (because it is the original command specified in `mpirun` arguments);

6. each container process executes the command (that is, the MPI application) passed inside the given container;

7. each of the MPI process links to the dynamic Open MPI library, which loads shared libraries with `dlopen` system call;

8. Open MPI libraries connect back to the original `orted` process via PMI;

9. all non-shared memory communication then occurs through the PMI, and then passes on to local network interfaces.

Additional information about Apptainer usage can be found in [Chapter 11 of the *User Manual*](/user/apptainer). The official web site of the tool is [https://www.apptainer.org](https://www.apptainer.org).