Supported functionalities

The following limitations apply to cuFFTMp:

  • If defined, CUDA_VISIBLE_DEVICES should be identical on all processes within a node.

  • Callbacks are not supported.

  • Because NVSHMEM spawns a hidden thread to handle communications, each process should have exclusive access to at least 2 CPU cores.

  • Only 2D and 3D transforms are supported, with the following restrictions:

    • The first two dimensions have length greater than or equal to the number of GPUs

    • When using built-in data layouts (CUFFT_XT_FORMAT_INPLACE and CUFFT_XT_FORMAT_INPLACE_SHUFFLED):

      • in 2D, R2C only supports an CUFFT_XT_FORMAT_INPLACE input;

      • in 2D, C2R only supports an CUFFT_XT_FORMAT_INPLACE_SHUFFLED input;

      • no strides are allowed;

      • only in-place data layouts are allowed. In particular, for R2C, the real dimension has to be padded to accommodate the complex elements in the output.

  • Using different MPI communicators (for different processes in MPI_COMM_WORLD) is allowed, but those MPI communicators cannot overlap: for a given process, one cannot use cuFFTMp with two distinct MPI communicators.

  • Only NVSHMEM-allocated memory can be used for descriptors and workspace. In particular, cudaMalloc’ed memory cannot be used. Note that memory allocated using cufftXtMalloc is automatically NVSHMEM-allocated.