Communicator#

Both the cuStateVec API and the cuStateVec Ex API use a communicator to abstract inter-process communication (IPC) for multi-process state vector operations. A communicator provides the common data-transfer functions used during distributed state vector simulation and implements them on top of an IPC library, typically MPI. This page describes the concepts shared by both APIs. For the API-specific functions and lifecycle, see Distributed index bit swap API for the cuStateVec API and cuStateVec Ex: Communicator for the cuStateVec Ex API.

Communicator interface#

In the cuStateVec API, a communicator is created with custatevecCommunicatorCreate() and referenced by a custatevecCommunicatorDescriptor_t handle. The communicator interface that a custom communicator implements is shown in the samples_mpi example. In the cuStateVec Ex API, the preliminary communicator interface is defined as custatevecExCommunicator_t in custatevecEx_ext.h. Both APIs use the communicator in the library implementation to execute inter-process data transfers.

The built-in communicators that implement these interfaces are provided for several MPI libraries: Open MPI, MPICH, and the MPI standard ABI. Because MPI libraries come in different application binary interfaces (ABIs), libcustatevec.so does not link against any specific MPI library. Instead, the communicator loads the chosen library at run time with dlopen() and resolves the required functions and predefined handles from it, so that ABI differences are handled inside cuStateVec.

Users can also develop custom communicators that implement the communicator interfaces for other IPC libraries; these are called external communicators.

Communicator types#

The communicator type is specified with custatevecCommunicatorType_t, which is shared by both APIs:

CUSTATEVEC_COMMUNICATOR_TYPE_OPENMPI – built-in communicator for Open MPI.
CUSTATEVEC_COMMUNICATOR_TYPE_MPICH – built-in communicator for MPICH.
CUSTATEVEC_COMMUNICATOR_TYPE_MPI_ABI – built-in communicator for any MPI library that provides the MPI standard ABI.
CUSTATEVEC_COMMUNICATOR_TYPE_EXTERNAL – a user-provided communicator plugin.

Only a single MPI library is used during the lifetime of an application.

Built-in communicators#

The built-in communicators require a CUDA-aware MPI library so that device-to-device data transfers are performed.

Open MPI and MPICH expose their own implementation-specific ABIs, selected with CUSTATEVEC_COMMUNICATOR_TYPE_OPENMPI and CUSTATEVEC_COMMUNICATOR_TYPE_MPICH. CUSTATEVEC_COMMUNICATOR_TYPE_MPI_ABI selects any MPI library that provides the MPI standard ABI, independent of the implementation.

The built-in communicators work with the following MPI libraries.

Open MPI version 4.x and 5.0. Validated by using 4.1.0 and 5.0.8.
MPICH version 4.x and 5.0. Validated by using 4.3.2 and 5.0.1.
Any MPI library providing version 1 of the MPI standard ABI, ratified in MPI 5.0. Validated by using MPICH 5.0.1.

Library resolution#

For the MPI communicator types, the library-name argument (soname in the cuStateVec API, libraryPath in the cuStateVec Ex API) names the shared library to load. An empty string ("") is equivalent to NULL.

If an MPI library is already available in the process, typically because the application links and uses MPI, that library is used and the library-name argument is ignored.
Otherwise, a non-NULL name is loaded as given, and NULL selects a default name according to the type: libmpi.so for CUSTATEVEC_COMMUNICATOR_TYPE_OPENMPI and CUSTATEVEC_COMMUNICATOR_TYPE_MPICH, and libmpi_abi.so for CUSTATEVEC_COMMUNICATOR_TYPE_MPI_ABI. If the library cannot be loaded with dlopen(), CUSTATEVEC_STATUS_LOADING_LIBRARY_FAILED is returned.

The loaded library is then checked against the requested type in two steps, which report different status codes:

The ABI is verified before the library is initialized. A library providing the MPI standard ABI is accepted only for CUSTATEVEC_COMMUNICATOR_TYPE_MPI_ABI, and a library using an implementation-specific ABI is accepted only for CUSTATEVEC_COMMUNICATOR_TYPE_OPENMPI and CUSTATEVEC_COMMUNICATOR_TYPE_MPICH. A mismatch returns CUSTATEVEC_STATUS_INVALID_VALUE.
The required function symbols and predefined MPI handles are then resolved following the convention of the type. If they cannot be resolved, CUSTATEVEC_STATUS_COMMUNICATOR_ERROR is returned.

Note

In this version, CUSTATEVEC_COMMUNICATOR_TYPE_MPI_ABI supports version 1 of the MPI standard ABI, which is ratified in MPI 5.0. A library reporting another version returns CUSTATEVEC_STATUS_NOT_SUPPORTED.

External communicator#

To support other MPI libraries or other IPC libraries, an application can provide a custom communicator plugin, selected with CUSTATEVEC_COMMUNICATOR_TYPE_EXTERNAL. The plugin is a shared library that wraps the IPC library of choice and exposes it through the communicator interface. The ABI and symbol checks used for the built-in MPI types do not apply to an external communicator.

The plugin interface and the example implementation differ between the two APIs; see the API-specific pages for details.