Other Methods

Runtime Database Query

cufftdx::experimental::utils::query_database(...) and cufftdx::experimental::utils::get_all_implementations(...) provide runtime access to the cuFFTDx internal database. Both methods are useful when FFT configuration is selected dynamically, for example based on user inputs, including checking whether a given configuration requires a workspace. To better understand the difference between non-workspace and workspace-required FFTs please check Supported Functionality. cufftdx::experimental::utils::query_database(...) returns the optimal implementation as std::optional<cufftdx::experimental::utils::frontend_impl_traits>. If no implementation is found for the provided configuration, it returns std::nullopt.

frontend_impl_traits describes the selected implementation:

Field	Description
elements_per_thread, ffts_per_block (ffts_per_block is 0 for thread execution)	Same as FFT::elements_per_thread and FFT::ffts_per_block.
suggested_elements_per_thread, suggested_ffts_per_block, suggested_block_dim_x/y/z (block execution only)	Same as FFT::suggested_elements_per_thread, FFT::suggested_ffts_per_block, and FFT::suggested_block_dim.
block_dim_x, block_dim_y, block_dim_z (block execution only)	Same as FFT::block_dim. Launch the kernel with these dimensions. They are 0 for thread execution.
shared_memory_size	Same as FFT::shared_memory_size.
storage_size, stride	Same as FFT::storage_size and FFT::stride.
input_elements_per_thread, output_elements_per_thread	Same as FFT::input_ept and FFT::output_ept.
workspace_size	Bytes of global memory the FFT reads; 0 if the implementation needs no workspace, same as FFT::workspace_size. A nonzero value means a workspace must be created on the host with cufftdx::make_workspace before calling execute(...). make_workspace is not available in NVRTC-compiled code, which cannot execute FFTs that require a workspace (such as the Bluestein fallback): in an NVRTC flow, treat a nonzero workspace_size as unsupported.
transposition_type	The Transposition layout used by the selected implementation, or transposition::any if it matches no known layout.
execution_size	The size the selected implementation actually runs at: halved under real_mode::folded, padded under a Bluestein fallback.
code	Which database answered: experimental::code_type::ptx or experimental::code_type::ltoir.

The code_type parameter is of type cufftdx::experimental::query_code_type and selects which database backend the call dispatches to:

• experimental::query_code_type::ptx – query the built-in offline PTX database that ships with cuFFTDx (default).

• experimental::query_code_type::ltoir_offline – query an offline LTO database that has been generated ahead of time with the LTO helper (see Custom LTO Helper) and included in the translation unit.

• experimental::query_code_type::ltoir_online – forward the request to the cuFFT Device API at runtime via query_all_cufft_implementations. This mode additionally requires CUFFTDX_ENABLE_CUFFT_DEPENDENCY to be defined and the application to be linked against cuFFT.

std::optional<cufftdx::experimental::utils::frontend_impl_traits>
cufftdx::experimental::utils::query_database(
    unsigned int                    fft_size,
    cufftdx::fft_direction          dir,
    cufftdx::fft_type               type,
    unsigned int                    sm,
    cufftdx::utils::execution_type  execution,
    cufftdx::precision              prec =
        cufftdx::precision::f32,
    unsigned int                    fft_ept = 0,
    unsigned int                    ffts_per_block = 0,
    std::tuple<unsigned int, unsigned int, unsigned int>
        block_dim = {0, 0, 0},
    cufftdx::complex_layout         layout =
        cufftdx::complex_layout::natural,
    cufftdx::real_mode              rmode =
        cufftdx::real_mode::normal,
    cufftdx::experimental::query_code_type code_type =
        cufftdx::experimental::query_code_type::ptx,
    bool                            has_thread_linearization = false,
    cufftdx::experimental::transposition trp =
        cufftdx::experimental::transposition::any,
    cufftdx::experimental::thread_layout thread_layout =
        cufftdx::experimental::thread_layout::regular);

cufftdx::experimental::utils::get_all_implementations(...) returns all matching implementations as std::vector<cufftdx::experimental::utils::frontend_impl_traits>. If no implementation is found, it returns an empty vector.

std::vector<cufftdx::experimental::utils::frontend_impl_traits>
cufftdx::experimental::utils::get_all_implementations(
    unsigned int                    fft_size,
    cufftdx::fft_direction          dir,
    cufftdx::fft_type               type,
    unsigned int                    sm,
    cufftdx::utils::execution_type  execution,
    cufftdx::precision              prec =
        cufftdx::precision::f32,
    unsigned int                    fft_ept = 0,
    unsigned int                    ffts_per_block = 0,
    std::tuple<unsigned int, unsigned int, unsigned int>
        block_dim = {0, 0, 0},
    cufftdx::complex_layout         layout =
        cufftdx::complex_layout::natural,
    cufftdx::real_mode              rmode =
        cufftdx::real_mode::normal,
    cufftdx::experimental::query_code_type code_type =
        cufftdx::experimental::query_code_type::ptx,
    bool                            has_thread_linearization = false,
    cufftdx::experimental::transposition trp =
        cufftdx::experimental::transposition::any,
    cufftdx::experimental::thread_layout thread_layout =
        cufftdx::experimental::thread_layout::regular);

has_thread_linearization and trp select, respectively, whether the returned implementation uses Thread Linearization and which Transposition shared-memory layout it uses. Both default to the values of a descriptor with neither operator applied.

thread_layout selects the thread-to-data mapping of the returned implementation and must match cufftdx::experimental::thread_layout_of<FFT>::value for the descriptor used to execute the FFT. It defaults to cufftdx::experimental::thread_layout::regular and determines the returned block_dim_x and block_dim_y. Both LTO query modes return no implementation when cufftdx::experimental::thread_layout::transposed is requested.

For an example on how to use these APIs, please check the nvrtc_query_database_fft_block example.

Note

These APIs are available only in host (non-NVRTC) code when CUFFTDX_ENABLE_RUNTIME_DATABASE is defined. The default experimental::query_code_type::ptx mode sees FFT sizes present in the PTX database that ships with cuFFTDx, including sizes with no direct (Cooley-Tukey) implementation: for block execution with real_mode::normal, if the size is covered by the built-in Bluestein padding table, the query transparently falls back to a Bluestein implementation instead of returning nothing. Thread execution and real_mode::folded never fall back to Bluestein. Check the returned implementation’s workspace_size field and create a workspace on the host with cufftdx::make_workspace before calling execute(...) when it is nonzero. In NVRTC-compiled code, which cannot use a workspace, treat a nonzero workspace_size as an unsupported configuration. The fallback applies to query_database and get_all_implementations in every query mode. get_database_and_ltoir and query_all_cufft_implementations resolve the implementation with the same logic (in ltoir_online mode) but only report what the cuFFT online LTOIR database provides. If the resolved implementation is a PTX record because cuFFT has no matching function, query_all_cufft_implementations returns an empty vector, and get_database_and_ltoir returns an empty database string and no LTOIR, with the traits (including workspace_size) of the PTX record.

To extend the queryable set of non-Bluestein sizes, use one of the LTO modes: experimental::query_code_type::ltoir_offline (after generating an offline LTO database with the LTO helper, see Custom LTO Helper) or experimental::query_code_type::ltoir_online (which delegates to the cuFFT Device API at runtime and additionally requires CUFFTDX_ENABLE_CUFFT_DEPENDENCY).

The LTO query modes are only supported for LTO databases generated with cuFFT 12.3 or later (from CUDA Toolkit 13.3). Requesting a trp other than transposition::any in ltoir_online mode additionally requires a cuFFT runtime 12.5 or later. On an older runtime the call does not fail: query_database and get_all_implementations return PTX implementations (check code), get_database_and_ltoir returns an empty database and no LTOIR, and query_all_cufft_implementations returns an empty vector.

Warning

Compilation time can significantly increase when using these APIs.

(online) LTO Database Creation

cufftdx::utils::get_database_and_ltoir() returns a tuple containing:

• Database string (std::string) to be inserted into the cuFFTDx headers.

• Vector of LTOIRs (std::vector<std::vector<char>>) for building device functions for the specified FFT operation.

• cufftdx::experimental::utils::frontend_impl_traits describing the selected implementation (block dimensions, shared memory size, whether a workspace is required, and so on) – see the field table in Runtime Database Query. The string and the LTOIR vector are empty, and code is not ltoir, when cuFFT has no LTO function for the configuration.

std::tuple<std::string, std::vector<std::vector<char>>,
           cufftdx::experimental::utils::frontend_impl_traits>
cufftdx::utils::get_database_and_ltoir(
    unsigned int                    fft_size,
    cufftdx::fft_direction          dir,
    cufftdx::fft_type               type,
    unsigned int                    sm,
    cufftdx::utils::execution_type  execution,
    cufftdx::precision              prec =
        cufftdx::precision::f32,
    cufftdx::complex_layout         layout =
        cufftdx::complex_layout::natural,
    cufftdx::real_mode              rmode =
        cufftdx::real_mode::normal,
    unsigned int                    fft_ept = 0
        /* use heuristic */,
    unsigned int                    ffts_per_block = 1
        /* 0: use suggested ffts_per_block */,
    dim3                            block_dim = dim3(0, 0, 0)
        /* 0: use suggested block_dim */,
    bool                            has_thread_linearization = false,
    cufftdx::experimental::transposition trp =
        cufftdx::experimental::transposition::any);

Example

// Assuming the following FFT operator is defined in the NVRTC-compiled code:
// using FFT = decltype(cufftdx::Block() +
//                      cufftdx::Size<128>() +
//                      cufftdx::Type<cufftdx::fft_type::c2c>() +
//                      cufftdx::Direction<cufftdx::fft_direction::forward>() +
//                      cufftdx::Precision<float>() +
//                      cufftdx::ElementsPerThread<8>() +
//                      cufftdx::FFTsPerBlock<2>() +
//                      cufftdx::SM<750>());

// You can get the database string and LTOIRs for the FFT operation by calling:
auto [lto_db, ltoirs, impl_traits] =
   cufftdx::utils::get_database_and_ltoir(128,
                                          cufftdx::fft_direction::forward,
                                          cufftdx::fft_type::c2c,
                                          750,
                                          cufftdx::utils::execution_type::block,
                                          cufftdx::precision::f32,
                                          cufftdx::complex_layout::natural,
                                          cufftdx::real_mode::normal,
                                          8,
                                          2);
// impl_traits.block_dim_x/y/z and impl_traits.shared_memory_size describe how to
// launch the kernel; impl_traits.workspace_size is nonzero if a workspace is required.

After obtaining the database string and LTOIRs, you can insert the database string into the cuFFTDx header file and link the LTOIRs to the user code as shown in Use Case II: Online Kernel Generation.

Note

The cufftdx::utils::get_database_and_ltoir() function is a wrapper around cuFFT Device APIs (see cuFFT Device API Reference). To use this function:

This function is available only in host (non-NVRTC) code.

1. Define CUFFTDX_ENABLE_CUFFT_DEPENDENCY before including the cuFFTDx headers.

2. Define CUFFTDX_ENABLE_RUNTIME_DATABASE before including the cuFFTDx headers.

3. Define CUFFT_ENABLE_EXPERIMENTAL_API (required by the cuFFT Device API headers).

4. Link against the cuFFT library.

Querying All cuFFT Online LTO Implementations

cufftdx::utils::query_all_cufft_implementations() is the “list all candidates” counterpart to get_database_and_ltoir. It returns all matching cuFFT LTOIR implementations using the same algorithm selection as query_database, or an empty vector if none are available.

std::vector<cufftdx::experimental::utils::frontend_impl_traits>
cufftdx::utils::query_all_cufft_implementations(
    unsigned int                    fft_size,
    cufftdx::fft_direction          dir,
    cufftdx::fft_type               type,
    unsigned int                    sm,
    cufftdx::utils::execution_type  execution,
    cufftdx::precision              prec =
        cufftdx::precision::f32,
    cufftdx::complex_layout         layout =
        cufftdx::complex_layout::natural,
    cufftdx::real_mode              rmode =
        cufftdx::real_mode::normal,
    unsigned int                    fft_ept = 0
        /* use heuristic */,
    unsigned int                    ffts_per_block = 1
        /* 0: use suggested ffts_per_block */,
    std::tuple<unsigned int, unsigned int, unsigned int>
        block_dim = {0, 0, 0},
    bool                            has_thread_linearization = false,
    cufftdx::experimental::transposition trp =
        cufftdx::experimental::transposition::any);

For a complete autotuning example that enumerates all implementations, benchmarks each one, and then calls get_database_and_ltoir() for the selected configuration, see example/cufftdx/04_nvrtc_fft/nvrtc_query_autotune_lto.cu.

has_thread_linearization and trp on both get_database_and_ltoir() and query_all_cufft_implementations() select, respectively, whether the returned implementation uses Thread Linearization and which Transposition shared-memory layout it uses.

Note

Like get_database_and_ltoir(), this function is available only in host (non-NVRTC) code. It is a wrapper around the cuFFT Device API and requires CUFFTDX_ENABLE_RUNTIME_DATABASE and CUFFTDX_ENABLE_CUFFT_DEPENDENCY to be defined before including cuFFTDx headers, CUFFT_ENABLE_EXPERIMENTAL_API to be defined, and the application to be linked against cuFFT.

Shared Memory Compute For Dynamic Batching

cufftdx::experimental::utils::get_shared_memory_size_for_dynamic_batching<FFT>(const unsigned int ffts_per_block) computes the total shared memory bytes required for computing multiple FFTs with the DynamicBatching operator.

template<class FFT>
constexpr unsigned int
get_shared_memory_size_for_dynamic_batching(
    const unsigned int ffts_per_block);

This function is useful when you need to determine shared memory requirements at runtime, particularly when the number of FFTs per block varies dynamically.

The returned value is the scratch space required by the register-input execute overloads (#1 and #2). For the shared-memory overloads (#3 and #4), the buffer must also hold all input and output values: use the maximum of the returned value, ffts_per_block * <FFT_input_size_in_bytes> and ffts_per_block * <FFT_output_size_in_bytes>.

Parameters:

• ffts_per_block - Number of FFTs to compute per block (user-defined, can be a runtime value)

cufftdx::experimental::utils::get_shared_memory_size_for_dynamic_batching(const unsigned int shared_memory_size_per_fft, const unsigned int ffts_per_block, const unsigned int implicit_type_batching) provides the same functionality at runtime, without creating a FFT Description.

constexpr unsigned int
get_shared_memory_size_for_dynamic_batching(
    const unsigned int shared_memory_size_per_fft,
    const unsigned int ffts_per_block,
    const unsigned int implicit_type_batching);

Parameters:

• shared_memory_size_per_fft - Shared memory size required for a single FFT (obtained from FFT::shared_memory_size)

• ffts_per_block - Number of FFTs to compute per block (user-defined, can be a runtime value)

• implicit_type_batching - Number of FFTs batched per value type (obtained from FFT::implicit_type_batching)

Example

using FFT = decltype(Size<128>() + Precision<float>() +
               Type<fft_type::c2c>() +
               Direction<fft_direction::forward>() +
               ElementsPerThread<8>() + DynamicBatching());

// Query compile-time traits
constexpr auto shared_memory_size_per_fft =
    FFT::shared_memory_size;
constexpr auto implicit_type_batching =
    FFT::implicit_type_batching;

// Runtime value chosen by user
unsigned int ffts_per_block = 8;

// Compute total shared memory requirement
auto total_shared_memory_size_bytes =
    cufftdx::experimental::utils::
        get_shared_memory_size_for_dynamic_batching(
            shared_memory_size_per_fft,
            ffts_per_block,
            implicit_type_batching);

auto total_shared_memory_size_bytes_with_desc =
    cufftdx::experimental::utils::
        get_shared_memory_size_for_dynamic_batching<FFT>(
            ffts_per_block);

Note

These functions only return a valid result when using traits from a description with the DynamicBatching operator. For obtaining the shared memory bytes required for other descriptions, see Shared Memory Size Trait.

Frontend to Backend Traits Conversion

cufftdx::experimental::utils::frontend_to_backend(...) converts a frontend FFT description to a effective backend traits that is used to query the cuFFTDx databases.

In combination with cuFFT Device API, this function can be used to generate the LTO database containing both device function code and metadata (a C++ header file) for a specified FFT operation. See Custom LTO Helper for an example.

#include "cufftdx/utils.hpp"

namespace cufftdx {
   namespace utils {
      enum class algorithm {
         ct,
         bluestein,
      };

      enum class execution_type {
         thread,
         block,
      };
   } // namespace utils

   namespace experimental {
      namespace utils {
         struct backend_impl_traits {
            unsigned int  size;
            fft_type      type;
            fft_direction direction;
            unsigned int  sm;
            unsigned int  elements_per_thread;
            unsigned int  min_elements_per_thread;
         };

         backend_impl_traits
         frontend_to_backend(
               cufftdx::utils::algorithm algo,
               cufftdx::utils::execution_type exec_type,
               unsigned int fft_size
                   /* size_of<FFT>::value */,
               fft_type type
                   /* type_of<FFT>::value */,
               fft_direction direction
                   /* direction_of<FFT>::value */,
               cufftdx::precision precision
                   /* the cufftdx::precision value matching
                    * precision_of_t<FFT> */,
               unsigned int ffts_per_block
                   /* the value of the FFTsPerBlock operator,
                    * or 0 if not set */,
               unsigned int sm
                   /* sm_of<FFT>::value */,
               real_mode real_mode
                   /* real_fft_mode_of<FFT>::value */,
               unsigned int elements_per_thread
                   /* the value of the ElementsPerThread
                    * operator, or 0 if not set */,
               bool has_thread_linearization
                   /* experimental::is_thread_linearization_enabled_v<FFT> */,
               unsigned int block_dim_x
                   /* x of the BlockDim operator, or 0 if not set */,
               unsigned int block_dim_y
                   /* y of the BlockDim operator, or 0 if not set */,
               unsigned int block_dim_z
                   /* z of the BlockDim operator, or 0 if not set */,
               experimental::code_type code_type
                   /* experimental::code_type_of<FFT>::value */,
               experimental::thread_layout thread_layout =
                   experimental::thread_layout::regular
                   /* experimental::thread_layout_of<FFT>::value */
            );
      } // namespace utils
   } // namespace experimental
} // namespace cufftdx