cuStateVec Ex: Communicator#

cuStateVec Ex API has communicator that abstracts inter-process communication (IPC) for multi-process state vector operations. Communicator has an interface that provides common data transfer functions utilized during state vector simulations and implemented by using IPC libraries.

Communicator workflow#

The following is the example workflow to use communicator. The following example uses the builtin communicator that wraps Open MPI.

// 1. Application-global initialization - initialize communicator library once
//    It uses Open MPI.  MPI_Init() is internally called.
custatevecExCommunicatorStatus_t commStatus;
custatevecExCommunicatorInitialize(CUSTATEVEC_COMMUNICATOR_TYPE_OPENMPI,
                                   nullptr,  // Use default library path
                                   &argc, &argv,  // argc, argv
                                   &commStatus);

// 2. Query global rank and size (before creating instance)
int32_t rank, size;
custatevecExCommunicatorGetSizeAndRank(&size, &rank, &commStatus);

// 3. Create communicator instance
custatevecExCommunicatorDescriptor_t communicator;
custatevecExCommunicatorCreate(&communicator);

// Use communicator for multi-process state vector operations
// ... state vector configuration and simulation logic ...

// 4. Clean up communicator instance
custatevecExCommunicatorDestroy(communicator);

// 5. On application shutdown - finalize communicator library once
// MPI_Finalize() is internally called.
custatevecExCommunicatorFinalize(&commStatus);

  1. Application-global initialization

    custatevecExCommunicatorInitialize() should be called to load external library or communicator implementation. The above example looks for Open MPI library named as “libmpi.so”. When CUSTATEVEC_COMMUNICATOR_TYPE_MPICH, that API will assume the library is MPICH. After the call to custatevecExCommunicatorInitialize(), the IPC library is initialized.

    The call to custatevecExCommunicatorInitialize() is an application global initialization. During the application lifetime, the application is able to utilize only a single library. The next call after the successful call to custatevecExCommunicatorInitialize() will return error.

  2. Getting rank and size.

    custatevecExCommunicatorGetSizeAndRank() returns the rank and size.

  3. Create communicator instance

    After the initialization, communicator instance is created by calling custatevecExCommunicatorCreate(). The instance is used as an input argument to create multi-process state vector.

  4. Destroy communicator instance

    After using the communicator instance, communicator is destroyed by custatevecExCommunicatorDestroy().

  5. Application-level finalization

    On application shutdown, custatevecExCommunicatorFinalize() is called to finalize the use of the IPC library.

Communicator implementations#

Built-in communicators#

The cuStateVec library provides built-in implementation of communicators by using Open MPI and MPICH. In order to use those communicators, MPI library should be CUDA-aware to receive device pointers.

The built-in communicators work with the following MPI libraries.

  • Open MPI version 5.0.8 and ABI compatible versions.

  • MPICH version 4.3.2 and ABI compatible versions

External communicator#

To support other MPI libraries and other IPC libraries, one can build custom communicators. An example implementation is provided in the mpiCommunicator.c sample.

Custom MPI initialization#

The builtin communicators with Open MPI and MPICH support the MPI_Init() and MPI_Finalize() called from applications.

// Call MPI_Init() here if an application needs the explicit call to MPI_Init(),
// or needing optional arguments to initialize the library.
MPI_Init(&argc, &argv);

// Application-global initialization
// It does not internally call MPI_Init().
custatevecExCommunicatorStatus_t commStatus;
custatevecExCommunicatorInitialize(CUSTATEVEC_COMMUNICATOR_TYPE_OPENMPI,
                                   nullptr,  // Use default library path
                                   nullptr, nullptr,  // argc, argv
                                   &commStatus);

// Application main...

// On application shutdown - It does not internally call MPI_Finalize()
custatevecExCommunicatorFinalize(&commStatus);

// Finalize MPI library
MPI_Finalize();

Communicator class and interfaces#

In custatevecEx_ext.h, one struct for class and two interfaces are defined.

  • custatevecExCommunicatorModule_t

    • Library-level initialization and finalization

    • Version management

  • custatevecExCommunicator_t

    • Communicator class

  • custatevecExCommunicatorInterface_t

    • Communicator interface

The following three functions in custatevecExCommunicatorInterface_t should accept device memory pointer to directly transfer data on devices.

  • sendAsync

  • recvAsync

  • sendRecvAsync

Please refer to the mpiCommunicator.c sample for the communicator implementation.

Note

The interface defined in custatevecEx_ext.h is a preliminary version and subject to change.

Use of communicator from application#

Communicator instance can be utilized from user applications. The following is an example to call communicator function.

if (isMultiProcess) {
    int rank;
    communicator->intf->getRank(communicator, &rank);
    if (rank != 0) {
        setOutputEnabled(false);  // From common.hpp
    }
}