12. Optimization Guide#

Tile IR programs run as portable bytecode that the compiler maps onto a specific GPU target. Most performance decisions — instruction selection, pipelining, register/shared-memory budget, warp specialization, and so on — are made by the compiler. This section describes the user-controlled knobs that influence those decisions, the remarks that explain selected compiler decisions, and the tools that optimize Tile IR before lowering.

Program-level tuning knobs are exposed as optimization hints attached to ops. Hints are advisory: they tell the compiler what to prefer when it has a choice, but the compiler may override a hint if it would produce an invalid or unprofitable program.

12.1. Optimization Hints#

Optimization hints are carried on the optimization_hints attribute (see the OptimizationHints attribute reference). The attribute is a dictionary keyed by architecture (e.g. sm_100, sm_120), and each value is a dictionary of hints for the operation carrying the attribute. The default dictionary may provide a fallback for a hint that is absent from the target-specific dictionary, provided the default value is valid for the current compilation target. A target-specific value takes precedence over the default value for the same hint.

For example, an cuda_tile.entry op might carry:

optimization_hints=<
  sm_100 = {num_cta_in_cga = 8, num_worker_warps_per_cta = 8},
  sm_120 = {num_cta_in_cga = 16, num_worker_warps_per_cta = 4},
  default = {occupancy = 1}
>

A memory operation might instead carry:

optimization_hints=<
  sm_100 = {latency = 4},
  default = {allow_tma = true}
>

The available hints are enumerated below. Each entry lists the hint name, where it applies, accepted values, and how it is used.

12.1.1. `num_cta_in_cga`#

Applies to: cuda_tile.entry
Type: integer
Accepted values: powers of 2 in the closed range [1, 16].
Default: 1 (single-CTA grouping).

Suggests the number of CTAs that the compiler should assign to each cooperative group array (CGA). Values greater than 1 permit inter-CTA cooperation, including dual-CTA MMA instructions, multicast TMA, distributed shared memory, and cluster-wide barriers, but place additional constraints on resource usage and the launch grid. On Ampere and Ada targets (sm_80, sm_86, sm_87, sm_88, and sm_89), the only accepted value is 1.

12.1.2. `occupancy`#

Applies to: cuda_tile.entry
Type: integer
Accepted values: integers in the closed range [1, 32].
Default: 1.

Suggests the target number of active CTAs per streaming multiprocessor (SM). The compiler uses the value when budgeting resources for each CTA. It may reduce the requested value to satisfy resident-thread, register, shared-memory, or tensor-memory constraints. The hint therefore does not guarantee that the requested occupancy will be achieved at run time.

12.1.3. `num_worker_warps_per_cta`#

Applies to: cuda_tile.entry
Type: integer
Accepted values: powers of 2 in the closed range [1, 32].
Default: 4.

Suggests the number of worker (non-specialized) warps in each CTA. The compiler uses this value for the baseline CUDA-core worker group and when partitioning work among those workers. A warp-specialized schedule may add separate warps for MMA, TMA, or control agents; those additional warps are not included in this value. Consequently, this hint does not specify the total number of warps in the final CTA. Increasing the worker-warp count can also reduce the register budget available to each thread.

12.1.4. `allow_tma`#

Applies to: cuda_tile.load_view_tko, cuda_tile.store_view_tko
Type: boolean
Accepted values: true or false.
Default: true.

Controls whether the Tensor Memory Accelerator (TMA) is permitted for the view-based load or store. A value of true makes the operation eligible for TMA but does not require the compiler to use it. A value of false requires the compiler to select a non-TMA implementation.

12.1.5. `latency`#

Applies to: cuda_tile.load_view_tko, cuda_tile.store_view_tko, cuda_tile.load_ptr_tko, cuda_tile.store_ptr_tko
Type: integer
Accepted values: integers in the closed range [1, 10].
Default: if omitted, the compiler infers a value from the memory request size.

Specifies a unitless scheduling-latency tier for the memory operation. The compiler translates the tier into a static latency estimate used when scheduling the operation. A larger value tells the scheduler to assume a greater latency and may result in more overlap with other work or a deeper prefetch pipeline; it does not require a particular pipeline depth. If the hint is omitted, the compiler derives the tier from the size of the memory request.

Note

Each hint above is advisory. If a hint conflicts with hardware constraints, with other hints, or with what the compiler determines to be a valid and efficient schedule, it may be ignored. Optimization remarks can report hints that were used, invalid, or outside the supported range, but do not necessarily report every case in which a valid hint is not applied.

12.2. Compiler Optimization Remarks#

tileiras provides opt-in optimization remarks that explain selected compiler decisions. Remarks are disabled by default. Each public remark has a kind and a category:

Passed remarks report an optimization that was applied or a hint that was used.
Missed remarks report a requested optimization or hint that was not applied exactly as specified; the compiler may use an adjusted value or alternative implementation.
Failed remarks report a specific optimization attempt or hint validation that could not be applied. Compilation may still continue using a fallback.

Neither Missed nor Failed indicates that compilation itself failed.

The available category selectors are memory and tensorcore. The all selector also enables optimization-hint remarks.

12.2.1. Enabling remarks#

Use --remarks=<category> to enable every public kind of remark in one category. The category may be memory, tensorcore, all, or none. For example, the following command enables all memory remarks:

$ tileiras --gpu-name sm_100 input.tilebc -o output.cubin --remarks=memory

Use a kind-specific option to enable only one kind of remark:

--remarks-passed=<category>
--remarks-missed=<category>
--remarks-failed=<category>

The kind-specific options accept the same category values. If both the general option and a kind-specific option are present, the kind-specific option overrides --remarks for that kind.

The following public remarks are available:

Remark	Kind	Category selector	Enabling option
A load instruction was selected	Passed	`memory`	`--remarks-passed=memory` or `--remarks=memory`
A store instruction was selected	Passed	`memory`	`--remarks-passed=memory` or `--remarks=memory`
TMA load instruction selection failed	Failed	`memory`	`--remarks-failed=memory` or `--remarks=memory`
TMA store instruction selection failed	Failed	`memory`	`--remarks-failed=memory` or `--remarks=memory`
An MMA operation was optimized to use Tensor Cores	Passed	`tensorcore`	`--remarks-passed=tensorcore` or `--remarks=tensorcore`
An MMA operation used FMA instructions instead of Tensor Cores	Failed	`tensorcore`	`--remarks-failed=tensorcore` or `--remarks=tensorcore`
An optimization hint, including a default hint, was used	Passed	None	`--remarks-passed=all` or `--remarks=all`
An optimization hint was invalid	Failed	None	`--remarks-failed=all` or `--remarks=all`
An optimization-hint value was outside the supported range	Missed	None	`--remarks-missed=all` or `--remarks=all`

By default, enabled remarks are displayed on the command line. Use --remark-format=yaml to write them as YAML. The default YAML output file is tileir_remarks.yaml; use --remarks-output-file=<file> to select a different path:

$ tileiras --gpu-name sm_100 input.tilebc -o output.cubin \
    --remarks=all --remark-format=yaml \
    --remarks-output-file=remarks.yaml

12.3. Cuda Tile Optimizations#

The cuda-tile-optimize tool runs optimization passes directly on Tile IR. It accepts either Tile IR bytecode or textual MLIR. Without an output option, it writes textual MLIR to standard output. Use -o to write textual MLIR to a file, or combine -o with --emit-bytecode to write Tile IR bytecode:

$ cuda-tile-optimize input.tilebc --opt-level=2 -o optimized.mlir
$ cuda-tile-optimize input.tilebc --opt-level=3 --emit-bytecode \
    -o optimized.tilebc

--opt-level=N selects optimization level 0 through 3 and defaults to 3. -O=N is an alias. The tool also accepts textual MLIR pass pipelines with --run-before-default-pipeline=<pipeline> and --run-after-default-pipeline=<pipeline> (abbreviated --before and --after).

These passes are part of the open-source NVIDIA CUDA Tile repository; their implementations and tests can be freely inspected there.

12.3.1. Available passes#

The default pipeline contains the following passes:

Canonicalization applies standard Tile IR simplification patterns.
Common-subexpression elimination (CSE) removes redundant, side-effect-free expressions.
Loop-invariant code motion (LICM) hoists safe, side-effect-free operations from cuda_tile.for and cuda_tile.loop. It also examines nested cuda_tile.if, cuda_tile.scan, and cuda_tile.reduce regions.
Loop splitting splits an eligible cuda_tile.for at a comparison of its induction variable with a loop-invariant value, allowing the condition to be removed from the resulting loop bodies.

The tool also provides an optional FMA fusion pass. Pass --fuse-fma to fuse eligible multiply-add and multiply-subtract sequences. This pass is not enabled by any optimization level because it changes floating-point rounding from two operations to a single-rounding FMA operation. This change can affect the bit-exact results of floating-point operations and violates Tile IR’s numerical stability guarantees; therefore, FMA fusion is strictly opt-in.

12.3.2. Optimization levels#

Both tileiras and the CUDA driver JIT compiler apply this Tile IR optimization pipeline before lowering Tile IR bytecode. cuda-tile-optimize, tileiras, and driver JIT compilation all default to optimization level 3. The levels include the following passes:

Level	`cuda-tile-optimize`	`tileiras` and driver JIT compilation
`-O0`	No level-controlled Tile IR passes	No Tile IR optimization passes
`-O1`	Canonicalization and CSE	Canonicalization and CSE
`-O2`	`-O1` plus Tile IR LICM	`-O1` plus Tile IR LICM
`-O3`	`-O2` plus loop splitting and a second canonicalization pass	`-O2` plus a second canonicalization pass; loop splitting runs only when requested by the input

cuda-tile-optimize --opt-level=3 uses a default loop-split threshold of 1, which enables splitting for structurally eligible loops. At level 3, the compiler pipeline used by tileiras and driver JIT compilation includes the pass with a default threshold of 0, so it does not split loops unless the input provides an explicit loop-split threshold. FMA fusion is not enabled automatically by either tool or by driver JIT compilation.

This subsection covers only Tile IR-level optimizations. In addition to these transformations, the selected optimization level can affect compiler heuristics during lowering. No optimization level affects the semantic stability of an input program.

12. Optimization Guide#

12.1. Optimization Hints#

12.1.1. num_cta_in_cga#

12.1.2. occupancy#

12.1.3. num_worker_warps_per_cta#

12.1.4. allow_tma#

12.1.5. latency#