Traits#
Traits provide the user with information about the FFT description constructed using Operators. They are divided into three categories:
Description Traits#
Trait |
Default value |
Description |
|---|---|---|
None. |
Size of the FFT to compute. |
|
|
Type of the FFT operation, either |
|
See Direction Trait. |
Direction of the FFT operation, either |
|
|
Type of the underlying floating-point values used to compute the FFT: |
|
None. |
|
|
None. |
|
|
None. |
|
|
None. |
|
|
|
FFT’s input and output data layout. See Complex Element Layouts. |
|
|
|
|
|
Code type requested for the FFT compute device functions with CodeType Operator, either |
|
|
Shared-memory layout requested via Transposition. Echoes the requested value, including
|
|
|
Thread-to-data mapping used during block execution, as set by Thread Layout. |
Description traits can be retrieved from an FFT descriptor using the helper functions provided. For example:
#include <iostream>
#include <cufftdx.hpp>
using FFT = decltype( cufftdx::Size<8>() + cufftdx::Type<fft_type::c2c>()
+ cufftdx::Direction<fft_direction::forward>()
+ cufftdx::Precision<double>() + cufftdx::Thread() );
if(cufftdx::is_complete_fft<FFT>::value)
std::cout << "Size of the FFT operation: "
<< cufftdx::size_of<FFT>::value << std::endl;
Size Trait#
cufftdx::size_of<FFT>::value
cufftdx::size_of_v<FFT>
Size of the FFT to compute, as set by Size Operator.
There is no default size. If the descriptor was not created using a Size Operator, compilation will fail with an error message.
Type Trait#
cufftdx::type_of<FFT>::value
cufftdx::type_of_v<FFT>
Type of the FFT operation, as set by Type Operator.
The default type is complex-to-complex, fft_type::c2c.
Direction Trait#
cufftdx::direction_of<FFT>::value
cufftdx::direction_of_v<FFT>
Direction of the FFT operation, as set by Direction Operator.
Default direction:
If the FFT type is
fft_type::r2c, the default direction isfft_direction::forward.If the FFT type is
fft_type::c2r, the default direction isfft_direction::inverse.For any other type, there is no default direction. If the descriptor was not created using a Direction Operator, compilation will fail with an error message.
Precision Trait#
cufftdx::precision_of<FFT>::type
cufftdx::precision_of_t<FFT>
Floating-point precision of the FFT operation, as set by Precision Operator.
The default precision is float.
Is FFT? Trait#
cufftdx::is_fft<FFT>::value
cufftdx::is_fft_v<FFT>
Trait is true if the descriptor is an FFT description, formed with Description Operators.
There is no default value. The descriptor either is or is not an FFT description.
Is FFT Execution? Trait#
cufftdx::is_fft_execution<FFT>::value
cufftdx::is_fft_execution_v<FFT>
Trait is true if the descriptor is an FFT description, formed with Description Operators and a Execution Operators.
There is no default value. The descriptor either is or is not an FFT description including an Execution Operators.
Is FFT-complete? Trait#
cufftdx::is_complete_fft<FFT>::value
cufftdx::is_complete_fft_v<FFT>
Trait is true if the descriptor is a complete FFT description, formed with Description Operators.
Note
Complete in this context means that the descriptor has been formed with all the necessary Description Operators and it is only missing an Execution Operators to be able to run.
For an FFT descriptor to be complete, the following is required:
One, and only one, Size Operator.
One, and only one, Direction Operator unless either
cufftdx::Type<fft_type::r2c>()orcufftdx::Type<fft_type::c2r>()are added.One, and only one, SM Operator unless a Thread Operator is added.
There is no default value. The descriptor either is or is not an FFT-complete description.
Is FFT-complete Execution? Trait#
cufftdx::is_complete_fft_execution<FFT>::value
cufftdx::is_complete_fft_execution_v<FFT>
Trait is true if both cufftdx::is_fft_execution and cufftdx::is_complete_fft are true.
Note
If cufftdx::is_complete_fft_execution trait is true for a descriptor FFT, then we can use the Execution Methods
to compute the FFT.
There is no default value.
Real FFT layout Trait#
cufftdx::real_fft_layout_of<FFT>::value
cufftdx::real_fft_layout_of_v<FFT>
The input and output data layout for real-to-complex (fft_type::r2c) and complex-to-real (fft_type::c2r) FFT types, set with RealFFTOptions Operator.
The default value is complex_layout::natural.
Real FFT mode Trait#
cufftdx::real_fft_mode_of<FFT>::value
cufftdx::real_fft_mode_of_v<FFT>
The value represents optimization mode in which the FFT kernel will be executed, set with RealFFTOptions Operator.
The default value is real_mode::normal.
Code Type Trait#
cufftdx::experimental::code_type_of<Description>::value
cufftdx::experimental::code_type_of_v<Description>
Code type requested for the FFT compute device functions, as set by CodeType Operator. See Selected Code Type Trait for the code type of the implementation actually selected.
The default value is experimental::code_type::ptx.
Transposition Trait#
cufftdx::experimental::transposition_of<Description>::value
cufftdx::experimental::transposition_of_v<Description>
Shared-memory layout requested via Transposition. Echoes the requested value verbatim,
including experimental::transposition::any; it does not reflect what the resolved implementation actually uses
(see FFT::transposition, which never resolves to any).
The default value is experimental::transposition::any.
Thread Layout Trait#
cufftdx::experimental::thread_layout_of<Description>::value
cufftdx::experimental::thread_layout_of_v<Description>
Thread-to-data mapping used during block execution, as set by Thread Layout.
The default value is experimental::thread_layout::regular.
Execution Traits#
Execution traits can be retrieved directly from an FFT descriptor that has been configured with Execution Operators. The available execution traits depend on the operator used to build the descriptor; either a Thread Operator or a Block Operator.
Common Thread and Block Traits#
The following traits are available, with the same meaning, for descriptors built with either the Thread Operator or the Block Operator.
Selected Code Type Trait#
FFT::code
Code type of the implementation actually selected for thread or block execution. Unlike
experimental::code_type_of, which reports the requested CodeType,
this reflects the implementation that will execute. It may be experimental::code_type::ptx even when
CodeType<experimental::code_type::ltoir> was requested, if no matching LTO database entry was found.
See Using cuFFT LTO Database with cuFFTDx for LTO configuration and fallback behavior.
The default value is experimental::code_type::ptx.
Value Type Trait#
FFT::value_type
Complex type of the underlying data used for FFT computation.
The default type is cufftdx::detail::complex<float>, as defined in the types.hpp header file.
Input Type Trait#
FFT::input_type
The type of elements to be provided for FFT execution. This is the same as Value Type Trait for C2C and C2R configurations, but in R2C it becomes directly dependent on the used RealFFTOptions Operator values.
The default type is the same as Value Type Trait.
Output Type Trait#
FFT::output_type
The type of elements that will be returned from FFT execution. This is the same as Value Type Trait for C2C and R2C configurations, but in C2R it becomes directly dependent on the used RealFFTOptions Operator values.
The default type is the same as Value Type Trait.
Input Length Trait#
FFT::input_length
Complete length of a single batch input for this FFT. This value is
equivalent to Size Trait for C2C, but dependent on RealFFTOptions Operator
real_mode value for R2C and RealFFTOptions Operator complex_layout value
for C2R.
The default value is the same as Size Trait.
Output Length Trait#
FFT::output_length
Complete length of a single batch output for this FFT. This value is
equivalent to Size Trait for C2C, but dependent on RealFFTOptions Operator
real_mode value for C2R and RealFFTOptions Operator complex_layout value
for R2C.
The default value is the same as Size Trait.
Implicit Type Batching Trait#
FFT::implicit_type_batching
Number of values from different FFTs batched into one element of type Description::value_type used in FFT computation. If
it’s higher than one it means that Thread or Block FFT object calculates multiple FFTs in one go.
The value is 2 if cufftdx::precision_of<FFT>::type is __half, and 1 otherwise.
Note
Please note that in future releases of cuFFTDx FFT::implicit_type_batching may be replaced, and/or extended.
Elements Per Thread Trait#
FFT::elements_per_thread
Logical number of FFT elements that each thread will compute. This may differ from number of physical elements since RealFFTOptions Operator may change both the number of input or output elements and their logical layout among threads. Please refer to Input EPT Trait and Output EPT Trait for input and output memory operations and to Storage Size Trait for allocating register space.
The default value is the same as Size Trait.
Effective Elements Per Thread Trait#
FFT::effective_elements_per_thread
Number of FFT::value_type elements each thread actually processes during pre/post-processing.
This equals Elements Per Thread Trait except when real_mode::folded
is used, in which case it is half that value.
Use Input EPT Trait and Output EPT Trait for input/output memory operations, and Storage Size Trait to allocate register space.
Storage Size Trait#
FFT::storage_size
Number of Description::value_type elements that each thread must allocate to compute the FFT.
The default value is the same as Input EPT Trait.
Thread Traits#
Trait |
Default value |
Description |
|---|---|---|
|
Complex type of the underlying data used to compute the FFT. |
|
Same as Elements Per Thread Trait. |
Count of elements of type Input Type Trait to be loaded for this FFT |
|
Same as Elements Per Thread Trait. |
Count of elements of type Output Type Trait to be stored after this FFT |
|
None. |
Type of elements to be loaded. Provides the best possible vectorization and correctness. |
|
None. |
Type of elements to be stored. Provides the best possible vectorization and correctness. |
|
None. |
Length of the input of type Input Type Trait to be loaded as one batch.` |
|
None. |
Length of the output of type Output Type Trait to be stored as one batch. |
|
|
Number of values from different FFTs batched into one element of type |
|
|
Number of FFT elements to be computed per thread. |
|
Same as |
Number of |
|
|
Number of |
|
Always |
Stride between elements of the thread FFT held by each thread in its |
|
Empty workspace handle. |
Type accepted by the workspace-taking |
|
Always |
||
Same as |
Whether Thread Linearization is enabled. Has no effect on thread execution. |
|
|
Always |
|
See Working Group Trait. |
Working group of the FFT operation. Fixed, trivial semantics for thread execution. |
|
|
Code type requested for the FFT compute device functions, as set by CodeType Operator. |
|
|
Code type of the implementation actually selected; may differ from the requested Code Type Trait. |
Thread traits can be retrieved from descriptors built with Thread Operator.
For example:
#include <cufftdx.hpp>
using FFT = decltype(cufftdx::Size<8>() + cufftdx::Type<fft_type::c2c>()
+ cufftdx::Direction<fft_direction::forward>()
+ cufftdx::Precision<double>() + Thread());
// Retrieve the FFT data type
using complex_type = typename FFT::value_type;
// Retrieve the number of elements per thread
auto elements_per_thread = FFT::elements_per_thread;
Input EPT Trait#
FFT::input_ept
Maximal count of elements of type Input Type Trait to be provided by a single thread for FFT execution.
The default value is the same as Elements Per Thread Trait.
Output EPT Trait#
FFT::output_ept
Maximal count of elements of type Output Type Trait that will be returned from a single thread after FFT execution.
The default value is the same as Elements Per Thread Trait.
Stride Size Trait#
FFT::stride
Stride between elements of the FFT held by each thread in input.
For thread FFT FFT::stride is always 1.
Workspace Type Trait#
FFT::workspace_type
Type accepted by the execute(input, workspace) overload of the thread execute method. Since thread FFTs never
require a workspace, this is always the empty-workspace handle, and the argument is ignored. Do not use this overload to call
execute uniformly on Thread() and Block() descriptors: see Thread Execute Method for details.
Thread Layout Trait#
FFT::thread_layout
Always experimental::thread_layout::regular for thread execution, where one thread computes the entire FFT.
The ThreadLayout operator cannot be combined with Thread().
Thread Linearization Trait#
FFT::has_thread_linearization
true if the descriptor includes ThreadLinearization; otherwise false.
This has no effect on thread execution, where a single thread computes the entire FFT.
The default value is false.
Transposition Trait#
FFT::transposition
Always experimental::transposition::none for thread execution, which does not use shared-memory scratch.
Unlike transposition_of, this reports the resolved implementation rather than the requested
Transposition value.
Working Group Trait#
FFT::working_group
Working group of the thread FFT operation. It exposes the same interface as FFT::working_group of block execution,
with fixed semantics: is_thread_active() is true, is_predicated() is false, block_dim() is dim3(1, 1, 1),
batchid() and fftid() are 0, and implicit_bpb() and batch_stride() are 1.
Block Traits#
Trait |
Default value |
Description |
|---|---|---|
|
Complex type of the underlying data used to compute the FFT. |
|
None. |
Count of elements of type Input Type Trait for each thread to load. |
|
None. |
Count of elements of type Output Type Trait for each thread to store. |
|
None. |
Type of elements to be loaded by each thread. Provides the best possible vectorization and correctness. |
|
None. |
Type of elements to be stored by each thread. Provides the best possible vectorization and correctness. |
|
None. |
Length of the input of type Input Type Trait to be loaded as one batch.` |
|
None. |
Length of the output of type Output Type Trait to be stored as one batch. |
|
|
Device-side type of workspace required for FFT computation. |
|
|
Number of values from different FFTs batched into one element of type |
|
Heuristic. |
Number of FFT elements to be computed per thread. |
|
Same as |
Number of |
|
Determined by |
Number of |
|
Determined by |
Stride between elements of the block FFT held by each thread in its |
|
|
Number of FFTs to compute by a CUDA block in this FFT operation. |
|
Heuristic. |
Suggested number of FFTs to compute by a CUDA block to target maximum performance. |
|
Determined from the description. |
First available number of FFT elements per thread that satisfies the conditions of the description. |
|
Determined from |
Smallest CUDA block dimensions that fit this description. |
|
Determined from |
Size of the shared memory in bytes. |
|
See Block Dim Trait. |
|
|
Determined from |
Total number of threads in the CUDA block. |
|
|
Determines if it’s required to allocate extra workspace in global memory using |
|
|
Size of global memory required for workspace (in bytes). |
|
Thread-to-data mapping configured via Thread Layout. |
||
Same as |
Whether Thread Linearization is enabled. |
|
Never |
Shared-memory layout of the resolved implementation selected by Transposition. Unlike
|
|
|
Working group of the FFT operation. |
|
|
Code type of the implementation actually selected; may differ from the requested Code Type Trait. |
Block traits can be retrieved from descriptors built with Block Operator.
For example:
#include <cufftdx.hpp>
using FFT = decltype( cufftdx::Size<128>() + cufftdx::Type<fft_type::c2c>()
+ cufftdx::Direction<fft_direction::forward>()
+ cufftdx::Precision<float>() + cufftdx::Block()
+ cufftdx::ElementsPerThread<8>() + cufftdx::FFTsPerBlock<2>() );
// Retrieve the FFT data type
using complex_type = typename FFT::value_type;
// Allocate managed memory for input/output
complex_type* data;
auto size = FFT::ffts_per_block * cufftdx::size_of<FFT>::value;
auto size_bytes = size * sizeof(complex_type);
cudaMallocManaged(&data, size_bytes);
Input EPT Trait#
FFT::input_ept
Maximal count of elements of type Input Type Trait to be provided by a single thread for register API FFT execution.
Output EPT Trait#
FFT::output_ept
Maximal count of elements of type Output Type Trait that will be returned by a single thread from register API FFT execution.
Workspace Type Trait#
FFT::workspace_type
Type of a workspace required by execute(...) function of FFT. User should check if FFT requires a workspace
using Description::requires_workspace trait, and create one with cufftdx::make_workspace<FFT>(cudaError_t&, cudaStream_t).
See Make Workspace Function for more details about workspace.
Warning
FFT::workspace_type object doesn’t track lifetime of underlying memory, and is only valid within a lifetime of
workspace object it was casted from.
Warning
Type returned by cufftdx::make_workspace<FFT>(cudaError_t&, cudaStream_t) can be different for different FFT descriptions,
and is not the same as FFT::workspace_type. User should use auto when creating a workspace object.
Stride Size Trait#
FFT::stride
Stride between elements of the block FFT held by each thread in its input.
See also expected input data format.
Example
0-th thread of 8-point FFT with FFT::stride equal to 2 should have values 0, 2, 4, and 6 in its input.
FFTs Per Block Trait#
FFT::ffts_per_block
Number of FFTs to compute in parallel within a CUDA block, as part of the collective FFT operation.
The default value is 1.
Suggested FFTs Per Block Trait#
FFT::suggested_ffts_per_block
Suggested number of FFTs to compute in parallel within a CUDA block, as part of the collective FFT operation, to maximize performance.
The default value is heuristic, and depends on the size of the FFT, the number of elements per thread, and other parameters.
Suggested Elements Per Thread Trait#
FFT::suggested_elements_per_thread
Suggested number of FFT elements to be computed by each thread: the first available value that satisfies the conditions of the description (its size, precision, architecture and type, the Transposition operator, and the minimum value implied by BlockDim). It does not depend on the ElementsPerThread value set in the description. Relates to Elements Per Thread Operator the same way Suggested FFTs Per Block Trait relates to FFTs Per Block Operator.
The default value depends on the size of the FFT, the precision, and other parameters.
Suggested Block Dim Trait#
FFT::suggested_block_dim
Suggested CUDA block dimensions to launch the FFT kernel with: the smallest block that fits this description, that is the block required by its ElementsPerThread and FFTsPerBlock values, ignoring any larger BlockDim Operator. It is derived from the description’s own values, not from FFT::suggested_ffts_per_block or FFT::suggested_elements_per_thread.
The default value depends on the size of the FFT, the precision, and other parameters.
Block Dim Trait#
FFT::block_dim
Required CUDA block dimensions: the block dimensions set by the BlockDim operator, or the default block dimensions otherwise.
With stride = size_of<FFT>::value / FFT::elements_per_thread and bpb = FFT::ffts_per_block / FFT::implicit_type_batching, the default is:
(stride, bpb, 1)for thread_layout::regular (default),(bpb, stride, 1)forthread_layout::transposed, and(stride * bpb, 1, 1)with ThreadLinearization.
Note
The kernel containing the FFT execution must be launched with the block dimensions returned by this trait.
Max Threads Per Block Trait#
FFT::max_threads_per_block
Maximum number of threads for the FFT in the CUDA block.
The default value is determined from FFTs Per Block Trait and Elements Per Thread Trait.
Requires Workspace Trait#
FFT::requires_workspace
Boolean value. If true, a workspace must be created and passed to the FFT::execute(...) method (see block execute methods).
Otherwise, it’s not necessary to create and pass a workspace. Workspace can be created using
cufftdx::make_workspace<FFT>(cudaError_t&, cudaStream_t) function. Workspace created for FFT which does not require one
will be empty and won’t allocate any global memory.
cufftdx::is_workspace_required<FFT>() is the free-function equivalent of this trait for block descriptors. It is a
host-side constexpr function, so it can be used in host code (for example in an if constexpr), and it is available under NVRTC.
It is not SFINAE-friendly: it is a hard error for descriptors without this trait, such as Thread(). In device code, use
FFT::requires_workspace directly:
template<class FFT>
constexpr bool cufftdx::is_workspace_required();
Workspace Size Trait#
FFT::workspace_size
Informs how much global memory will be allocated by required workspace. If Description::requires_workspace is false
it’s 0; otherwise it’s greater than zero.
Thread Layout Trait#
FFT::thread_layout
Thread-to-data mapping configured by ThreadLayout. This is the same value as thread_layout_of. Use Working Group Trait to obtain batch and local work indices for the selected layout.
The default value is experimental::thread_layout::regular.
Thread Linearization Trait#
FFT::has_thread_linearization
true if the descriptor includes ThreadLinearization; otherwise false.
When enabled, Working Group Trait maps the linearized CUDA thread index to the batch and local
work indices according to FFT::thread_layout.
The default value is false.
Transposition Trait#
FFT::transposition
Shared-memory layout of the resolved FFT implementation selected by Transposition.
Unlike transposition_of, this is never experimental::transposition::any. When any is requested
or no Transposition operator is supplied, this reports the concrete layout selected by cuFFTDx.
The default value depends on the selected implementation.
Working Group Trait#
FFT::working_group
Working group of the FFT operation. This trait informs which threads will participate in the FFT execution. It provides the following functions:
is_thread_active()- Returns true if the thread is active in the FFT execution.block_dim()- Returns the block dimensions of the participating threads in the FFT execution. Returned dimensions may not be the same as the block dimensions retuned by theblock_dimtrait which is the case if not all threads in the block are participating in FFT computations.is_predicated()- Returns true if the FFT execution is predicated, that is, if some of the threads are not active in the FFT execution.batchid()- Returns the index, among the batches a thread participates in, of the batch it is currently processing. A batch holds FFT::implicit_type_batching FFTs (2 for__half, 1 otherwise).fftid()- Returns the thread’s local work index within the FFT it is currently processing.implicit_bpb()- Returns the number of batches (not FFTs) processed per block, that isFFT::ffts_per_block / FFT::implicit_type_batching, accounting for implicit type batching.batch_stride()- Returns the distance, in linear thread indices, between two threads processing the same position of different batches. It is1forthread_layout::transposed; forthread_layout::regularit is FFT::stride with ThreadLinearization andblockDim.xwithout it.
The values returned by batchid() and fftid() depend on the ThreadLayout and
ThreadLinearization operators; see their descriptions for the exact
thread-to-data mapping. Prefer these functions over reading threadIdx directly, so that the code remains correct for
every combination of the operators.
Only threads participating in the FFT execution (that is, is_thread_active() returns true) should contain valid data when calling and returning from the FFT execution.
The block execute method may contain block-wide barriers (always for the shared-memory overloads, and for the register overloads when
FFT::stride > 1) and must be called by every thread of the CUDA block,
including threads for which is_thread_active() returns false; use is_thread_active() to guard the data
accesses around the call, never the call itself.
Thread execution also exposes FFT::working_group, with the same interface but fixed, trivial semantics: a single
thread always computes the whole FFT, so is_thread_active() is always true, is_predicated() is always false,
block_dim() is always dim3(1, 1, 1), and batchid()/fftid() are always 0.
Other Traits#
Trait |
Default value |
Description |
|---|---|---|
|
|
|
None. |
The target architecture used by underlying FFT function. See SM Operator. |
|
None. |
Retrieves the working group of the FFT operation from a descriptor. |
|
|
|
|
|
|
|
cufftdx::is_supported#
// FFT - FFT description without CUDA architecture defined using SM operator
// Architecture - unsigned integer representing CUDA architecture (SM)
template<class FFT, unsigned int Architecture>
struct is_supported : std::bool_constant<...> { };
// Helper variable template
template<class FFT, unsigned int Architecture>
inline constexpr bool is_supported_v<FFT, Architecture> =
is_supported<FFT, Architecture>::value;
// true if FFT is supported on the provided CUDA architecture
cufftdx::is_supported<FFT, Architecture>::value;
cufftdx::is_supported checks whether an FFT is supported on Architecture CUDA architecture.
// true if FFT is supported on the provided CUDA architecture
cufftdx::is_supported<FFT, Architecture>::value;
Requirements:
FFTmust have defined size, direction (if it can’t be deduced from the type). See Description Operators section.
FFTcan’t have defined target CUDA architecture via SM operator.If
FFTdescription includes ElementsPerThread operator,cufftdx::is_supportedtakes that into account when verifying support.
Example
using FFT = decltype(Size<32768>() + Type<fft_type::c2c>() +
Direction<fft_direction::inverse>() + Block() +
Precision<float>());
cufftdx::is_supported<FFT, 800>::value; // true
using FFT = decltype(Size<8192>() + Type<fft_type::c2c>() +
Direction<fft_direction::forward>() + Block() +
Precision<double>());
cufftdx::is_supported<FFT, 800>::value; // true
cufftdx::is_supported<FFT, 750>::value; // false
using FFT = decltype(Size<4095>() + Type<fft_type::c2c>() +
Direction<fft_direction::inverse>() + Block() +
Precision<float>());
cufftdx::is_supported_v<FFT, 800>; // true
cufftdx::is_supported_v<FFT, 750>; // false
cufftdx::sm_of Trait#
sm_of<FFT>::value
sm_of_v<FFT>
The architecture used by underyling FFT function defined with SM Operator
There is no default value.
cufftdx::working_group_of Trait#
working_group_of<Description>::value
working_group_of_v<Description>
Free-trait equivalent of FFT::working_group, usable on a dependent descriptor type.
cufftdx::experimental::is_dynamic_batching_enabled Trait#
experimental::is_dynamic_batching_enabled<Description>::value
experimental::is_dynamic_batching_enabled_v<Description>
true if Description includes the DynamicBatching operator.
cufftdx::experimental::is_thread_linearization_enabled Trait#
experimental::is_thread_linearization_enabled<Description>::value
experimental::is_thread_linearization_enabled_v<Description>
true if Description includes the ThreadLinearization operator.