Achieving High Performance#

The compiler controls the lowest level of parallelism for MathDx Tile APIs. These choices still affect kernel performance:

  • The CTA size.

  • The number of problems processed at once.

  • The amount of surrounding computation inside the kernel.

Performance may be better or worse than the classic device APIs, depending on these choices. Measure performance as code is ported. The advice below uses the Examples to illustrate each topic.

Suggested FFT Configuration#

For FFTs, the CTA size and the number of transforms per block interact with the transform size, precision, and target architecture. A tuned configuration can be substantially faster than the default. The C++ and Python FFT APIs provide measured configurations through suggested_tile_config, which should be used as the starting point:

import cuda.tile as ct
import nvmath.tile.fft as mfft

config = mfft.suggested_tile_config("fft", rank=2, axis=1, size=1024,
                                    precision=ct.float32, arch="sm_100")

def make_fft_kernel(config):
    batches_per_block = config.batches_per_block

    @ct.kernel(num_worker_warps=config.num_worker_warps)
    def fft_kernel(source: ct.Array, destination: ct.Array):
        # Rows are independent contiguous 1024-point transforms along axis=1,
        # the last axis before the trailing (real, imaginary) pair.
        data = ct.load(source, (ct.bid(0), 0, 0),
                       shape=(batches_per_block, 1024, 2))
        ct.store(destination, (ct.bid(0), 0, 0), mfft.fft(data, axis=1))

    return fft_kernel
#include "cufftdx_tile_config.hpp"

constexpr auto config = cufftdx::tiles::suggested_tile_config<float>(
    cufftdx::tiles::transform_t::fft,
    /* rank = */ 2, /* axis = */ 1, /* size = */ 1024, /* sm_arch = */ 1000);

[[cutile::hint(0, num_worker_warps_per_cta = config.num_worker_warps_per_cta)]]
__tile_global__ void fft_kernel(const float* input, float* output) { /* ... */ }

In Python the value must reach the decorator at definition time, so the kernel is built by a factory function. The C++ form is constexpr, which allows the suggestion to drive both the hint and the tile shape without a host round trip.

For argument definitions and supported configurations, see nvmath.tile.fft.suggested_tile_config() and cufftdx::tiles::suggested_tile_config() in the API references.

Both APIs return a configuration even when no tuned entry exists for the key. In that case, found is false and the configuration contains a one-batch, one-warp fallback. A missing entry does not indicate a compilation error or guarantee that the fallback supports the requested transform.

Architecture matching#

The match parameter controls which architecture supplies the suggested configuration. The tables contain measurements for sm_80, sm_90, sm_100, and sm_120. By default, an architecture without its own table uses the tuned architecture with the same compute-capability major version: for example, sm_86 and sm_89 use sm_80, and sm_121 uses sm_120. A major version without a table uses sm_100. This provides a starting point for tuning. It does not change the kernel’s target architecture.

Behavior

Python match

C++ match

Map to a tuned architecture (default)

"mapped_to_major_arch"

cufftdx::tiles::arch_match_t::mapped_to_major_arch

Require a table for the requested architecture

"exact_match"

cufftdx::tiles::arch_match_t::exact_match

Python accepts match as a keyword-only argument. C++ accepts it as the optional final argument. To select mapping explicitly:

config = mfft.suggested_tile_config(
    "fft", rank=1, axis=0, size=1024,
    precision=ct.float32, arch="sm_89",
    match="mapped_to_major_arch",
)
constexpr auto config = cufftdx::tiles::suggested_tile_config<float>(
    cufftdx::tiles::transform_t::fft, 1, 0, 1024, 890,
    cufftdx::tiles::arch_match_t::mapped_to_major_arch);

These calls use the sm_80 table. Selecting exact_match for sm_89 returns found == false, because there is no table for that architecture. With either mode, found reports whether an entry exists in the selected table. A mapped suggestion can have found == true even if its measurements come from a different architecture.

Autotuning#

The shipped FFT table covers a set of architectures and shapes, so an FFT kernel whose performance matters is worth sweeping directly. cuTile Python provides autotuning [1] for this, and cuda.tile.tune.exhaustive_search() is the entry point: it compiles and times every configuration in a search space, reports the ones that fail to resolve instead of aborting, and returns the best.

Three parameters are worth sweeping:

  1. batches_per_block, the tile’s leading extent.

  2. num_worker_warps, the number of worker warps that determines the CTA size.

  3. occupancy, the number of CTAs the compiler should aim to keep resident on an SM, as an integer between 1 and 32, and left to the compiler when unset.

Only batches_per_block changes the kernel, because it is part of the tile shape, so the search takes a function that maps a configuration to a kernel. num_worker_warps and occupancy are compiler hints, so they go through hints_fn and cost no rebuild:

from cuda.tile import tune

kernels = {b: make_fft_kernel(b) for b in (1, 4)}
search_space = [(b, w, o) for b in kernels for w in (1, 4) for o in (None, 2)]

result = tune.exhaustive_search(
    search_space,
    stream,
    grid_fn=lambda config: (TOTAL_TRANSFORMS // config[0], 1, 1),
    kernel=lambda config: kernels[config[0]],
    args_fn=lambda config: (source, destination),
    hints_fn=lambda config: {"num_worker_warps": config[1]}
    | ({} if config[2] is None else {"occupancy": config[2]}),
)
print(result.summary())

The three interact, so the fastest combination is not generally the one that is fastest for any of them in isolation.

A tuner ranks configurations by time. It does not check correctness. Before adopting the winning configuration, apply its hints again and check its result. The kernel returned through kernel carries only the parameters fixed at definition time. The fft_autotuning example applies the hints and checks the result.

Array Shapes and Strides#

Tile extents are always compile-time constants. Global-memory arrays have separate shapes and strides, which can be runtime values. Specializing stable array dimensions and strides lets the compiler simplify addressing and shape calculations.

In Python, use typing.Annotated with cuda.tile.ArrayAnnotation. static_shape_dims and static_stride_dims select axes whose values are specialized at launch. For an interleaved complex array of shape (B, N, 2), the following keeps the batch count dynamic while specializing the transform length, complex-pair extent, and all strides:

from typing import Annotated

FFTArray = Annotated[
    ct.Array,
    ct.ArrayAnnotation(static_shape_dims=(1, 2),
                       static_stride_dims=(0, 1, 2)),
]

@ct.kernel(num_worker_warps=4)
def fft_kernel(source: FFTArray, destination: FFTArray):
    data = ct.load(source, (ct.bid(0), 0, 0),
                   shape=(1, source.shape[1], 2))
    ct.store(destination, (ct.bid(0), 0, 0), mfft.fft(data, axis=1))
// The tensor extent is a runtime int; the tile extent is not.
const auto tile_shape = ct::shape { 128_ic };
auto view = ct::partition_view { ct::tensor_span { a, ct::extents { n } }, tile_shape };

Include the contiguous axis explicitly when selecting static stride axes. Changing a specialized shape or stride produces another kernel specialization, so select values that are reused across launches. See cuTile array annotations [2]. ct.Constant[int] instead specializes an integer kernel parameter, for example when a tile extent is supplied as a separate argument:

@ct.kernel(num_worker_warps=4)
def fft_kernel(source: ct.Array, destination: ct.Array, n: ct.Constant[int]):
    data = ct.load(source, (0, 0), shape=(n, 2))
    ct.store(destination, (0, 0), mfft.fft(data, axis=0))

Here n is fixed for each kernel specialization, so it can be used in shape=(n, 2). An ordinary runtime integer parameter cannot be used as a tile extent.

The C++ side carries the same distinction in the types. A tensor extent may be a runtime int, as ct::extents { n } is above, while a tile extent is compile-time: either a template parameter, as in ct::shape<N, 2>, or an integral-constant value such as the 128_ic literal, which needs using namespace cuda::tiles::literals. ct::integral_constant<N> spells the same thing where a value is wanted, as the axis argument to ct::sum does.

Batching and Occupancy#

  • Batch small problems whenever possible. A leading batch extent solves that many independent problems within a single CTA, which is often the difference between a solver kernel that fully utilizes the GPU and one that does not. batches_per_block serves the same purpose for FFTs.

  • Ensure that enough blocks are launched to fill the GPU. One block per problem is the usual arrangement for batched solves. reg_least_squares solves 10000 independent 100x8 problems in this way.

  • Pad the tile rather than reducing the problem. Only the tile extent must be a compile-time power of two, not the tensor from which it is loaded, so a problem of an arbitrary size can retain its natural dimensions and pad the tile with zero rows. reg_least_squares applies this technique to preserve the 100x8 problem size it mirrors.

  • The best parameters for compute-bound and memory-bound kernels may differ.

Kernel Fusion#

  • Perform the pointwise work between the transforms in tiles. A convolution consists of fft, a complex multiplication, and ifft within a single kernel, without returning to global memory in between. fft_convolution_1d presents the minimal case.

  • Merge adjacent memory-bound kernels, such as pre- and post-processing, into the MathDx kernel to reduce global memory traffic. filtered_backprojection_3d fuses filtering and backprojection in this way, and retains the unfused two-kernel version for comparison.

  • Divide the computation into stages only when the working set requires it. A multi-dimensional transform whose full volume exceeds the capacity of a single CTA must stage through global memory between axes. fft_convolution_3d_split and fft_convolution_5d demonstrate this decomposition.

  • Reuse factorizations. The *_using_factor variants allow one kernel to factor and subsequent kernels to solve, rather than repeating the factorization for each right-hand side. Refer to factor_reuse.

  • Block problems that exceed a single call. blocked_potrf factors 512x512 matrices by traversing 32x32 tiles in left-looking order, combining cholesky_factor with triangular_solve.

Memory Management#

  • For tiles of a known shape, prefer ct::partition_view over the gather and scatter forms, so that the compiler can employ the Tensor Memory Accelerator.

  • Use masked gather and scatter when the amount of data is known only at runtime. They move the valid elements between global memory and a compile-time-sized tile, which is useful for compressed chunks with a runtime byte count. See Runtime-sized buffers for an example.

  • For the direct-memory variant, give each payload slot max_comp_chunk_size bytes, the worst case a call may write, and round the slot stride up to a multiple of buffer_alignment(). Give each concurrent chunk its own disjoint scratch slice, and stride by compress_scratch_size or decompress_scratch_size for the direction you run. The lz4_gmem and ans_gmem examples show this layout.

  • Use 16-byte aligned pointers and declare the alignment to the compiler with ct::assume_aligned<16> in C++, as fft_convolution_1d.cu does.

Important

Mark non-aliasing C++ pointers with __restrict__. This lets the compiler optimize memory accesses without accounting for overlapping buffers. Only use it when the pointers satisfy the non-aliasing requirement.

Runtime-sized buffers#

A tile extent cannot be a runtime length. Use masked gather to load the data into a fixed-size tile:

  1. Choose a compile-time capacity.

  2. Form a tile of indices.

  3. Mask indices beyond the runtime count.

Masked elements receive a padding value. Masked scatter writes only valid elements to global memory.

For example, this Python kernel copies a variable-length byte region in 256-element tiles. Launch ct.cdiv(count, 256) blocks for a positive count that fits both one-dimensional arrays:

@ct.kernel
def copy_bytes(source: ct.Array, destination: ct.Array, count: int):
    indices = ct.bid(0) * 256 + ct.arange(256, dtype=ct.int32)
    valid = indices < count
    data = ct.gather(source, indices, mask=valid, padding_value=0)
    ct.scatter(destination, indices, data, mask=valid)

Every data tile has shape (256,), including the final partial tile. The C++ gather/scatter forms use the same approach: fixed-size index and predicate tiles select which global memory elements are transferred. Refer to cuTile gather [3] and cuTile scatter [4] for the Python operations.

This is useful when transferring only the meaningful bytes of a compressed tile: scatter the returned byte count, then gather with the same count before decompression. Compression APIs also accept global memory inputs and outputs directly, so neither the uncompressed chunk nor its payload needs to pass through a tile. Use that path when no surrounding tile computation needs the data; see nvCOMPDx in Tile and Compression cutile-python API.

Precision#

  • Compute in the precision required by the target accuracy rather than the precision of the input data. mixed_precision_fft_1d performs the transform in one precision and stores the result in another.

  • Higher-precision tiles (for example, double instead of single) consume more registers and shared memory, which limits the number of batches that fit within a CTA. Repeat the procedure in Suggested FFT Configuration after changing precision, as the tuned entries differ.

Measuring and Diagnosing#

  • Set LIBMATHDX_LP_LOG_LEVEL=6 during compilation to report which SIMT implementation was selected for each call. This is the most direct way to determine that a kernel resolved to a different CTA size or specialization than intended.

  • Sweep the CTA size and the batch extent, at the sizes and precisions used by the application, before committing to a configuration.

  • Run the kernel before timing starts. In Python, the first call compiles the kernel. Measure with CUDA events.

  • Compare against the classic device API for the same problem. Relative performance varies with the shape, so porting an already-tuned SIMT kernel does not necessarily improve it.

Further Reading#

References#