Is this page helpful?

cuStateVec Ex: Communicator#

cuStateVec Ex API has a communicator that abstracts inter-process communication (IPC) for multi-process state vector operations. The communicator has an interface that provides the common data-transfer functions used during state vector simulations, implemented on top of an IPC library.

This page describes the cuStateVec Ex API communicator interface, its lifecycle, and how to use it from an application. See Communicator for the concepts shared with the cuStateVec API, including the communicator types, the built-in communicators and their supported MPI versions, library resolution, and external communicators.

Note

The cuStateVec Ex API communicator interface, defined in custatevecEx_ext.h, is a preliminary version and subject to change.

Communicator workflow#

The following is the example workflow to use communicator. The following example uses the builtin communicator that wraps Open MPI.

// 1. Application-global initialization - initialize communicator library once
//    It uses Open MPI.  MPI_Init() is internally called.
custatevecExCommunicatorStatus_t commStatus;
custatevecExCommunicatorInitialize(CUSTATEVEC_COMMUNICATOR_TYPE_OPENMPI,
                                   nullptr,  // Use default library path
                                   &argc, &argv,  // argc, argv
                                   &commStatus);

// 2. Query global rank and size (before creating instance)
int32_t rank, size;
custatevecExCommunicatorGetSizeAndRank(&size, &rank, &commStatus);

// 3. Create communicator instance
custatevecExCommunicatorDescriptor_t communicator;
custatevecExCommunicatorCreate(&communicator);

// Use communicator for multi-process state vector operations
// ... state vector configuration and simulation logic ...

// 4. Clean up communicator instance
custatevecExCommunicatorDestroy(communicator);

// 5. On application shutdown - finalize communicator library once
// MPI_Finalize() is internally called.
custatevecExCommunicatorFinalize(&commStatus);

  1. Application-global initialization

    custatevecExCommunicatorInitialize() selects and loads the communicator implementation and initializes the underlying IPC library. The communicatorType argument chooses the implementation: CUSTATEVEC_COMMUNICATOR_TYPE_OPENMPI, CUSTATEVEC_COMMUNICATOR_TYPE_MPICH, or CUSTATEVEC_COMMUNICATOR_TYPE_MPI_ABI for the built-in MPI communicators, or CUSTATEVEC_COMMUNICATOR_TYPE_EXTERNAL for a custom plugin.

    For the MPI types, the libraryPath argument names the shared library to load, and NULL or an empty string selects a default name. The rules for resolving the library, including the case where an MPI library is already available in the process, are described in Communicator.

    The call to custatevecExCommunicatorInitialize() is an application-global initialization. During the application lifetime, only a single library is used. The first successful call returns CUSTATEVEC_STATUS_SUCCESS, and subsequent calls return CUSTATEVEC_STATUS_ALREADY_INITIALIZED.

  2. Getting rank and size.

    custatevecExCommunicatorGetSizeAndRank() returns the rank and size.

  3. Create communicator instance

    After the initialization, communicator instance is created by calling custatevecExCommunicatorCreate(). The instance is used as an input argument to create multi-process state vector.

  4. Destroy communicator instance

    After using the communicator instance, communicator is destroyed by custatevecExCommunicatorDestroy().

  5. Application-level finalization

    On application shutdown, custatevecExCommunicatorFinalize() is called to finalize the use of the IPC library.

External communicator#

An external communicator plugin, selected with CUSTATEVEC_COMMUNICATOR_TYPE_EXTERNAL, exposes a factory function named custatevecExCommunicatorGetModuleEXT that returns its module function table. The libraryPath argument names the plugin to load, or may be NULL to resolve the factory function from the application binary itself, so that an application can provide its own communicator without a separate shared object. An example implementation is provided in the exMpiCommunicator.c sample.

Custom MPI initialization#

The builtin communicators with Open MPI and MPICH support the MPI_Init() and MPI_Finalize() called from applications.

// Call MPI_Init() here if an application needs the explicit call to MPI_Init(),
// or needing optional arguments to initialize the library.
MPI_Init(&argc, &argv);

// Application-global initialization
// It does not internally call MPI_Init().
custatevecExCommunicatorStatus_t commStatus;
custatevecExCommunicatorInitialize(CUSTATEVEC_COMMUNICATOR_TYPE_OPENMPI,
                                   nullptr,  // Use default library path
                                   nullptr, nullptr,  // argc, argv
                                   &commStatus);

// Application main...

// On application shutdown - It does not internally call MPI_Finalize()
custatevecExCommunicatorFinalize(&commStatus);

// Finalize MPI library
MPI_Finalize();

Communicator class and interfaces#

In custatevecEx_ext.h, one struct for class and two interfaces are defined.

  • custatevecExCommunicatorModule_t

    • Library-level initialization and finalization

    • Version management

  • custatevecExCommunicator_t

    • Communicator class

  • custatevecExCommunicatorInterface_t

    • Communicator interface

The following three functions in custatevecExCommunicatorInterface_t should accept device memory pointer to directly transfer data on devices.

  • sendAsync

  • recvAsync

  • sendRecvAsync

The setNativeCommunicator function replaces the native communicator that an instance uses for collectives. For the built-in MPI communicators this is an MPI_Comm, for example &MPI_COMM_WORLD. It takes a pointer to a native communicator handle; a NULL pointer resets the instance to the world communicator. It should be called before the state vector is instantiated, and the communicator must not be changed after a state vector instance is created.

Please refer to the exMpiCommunicator.c sample for the communicator implementation.

Use of communicator from application#

Communicator instance can be utilized from user applications. The following is an example to call communicator function.

if (isMultiProcess) {
    int rank;
    communicator->intf->getRank(communicator, &rank);
    if (rank != 0) {
        setOutputEnabled(false);  // From common.hpp
    }
}