Precision Control

This page covers how TensorRT selects algorithms and how you control precision. It includes algorithm selection and reproducible builds, strongly typed networks, and control of computational precision.

Algorithm Selection and Reproducible Builds

The default behavior of TensorRT’s optimizer is to choose the algorithms that globally minimize the execution time of the engine. It does this by timing each implementation, and sometimes, when implementations have similar timings, system noise can determine which one is chosen on any particular run of the builder. Different implementations will typically use different orders of accumulation of floating point values, and two implementations can use different algorithms. As a result, different invocations of the builder will typically not result in engines that return bit-identical results.

When the engine is being built for the first time, you supply the BuilderFlag::kEDITABLE_TIMING_CACHE flag to TensorRT to enable the editable cache. At the same time, you enable and retain the logs and cache files. The logs will provide the name, key, available tactics, and the selected tactic for each model layer. The cache file will record the decisions made by TensorRT.

Next time the same engine is being built, you supply the same flags to TensorRT and use the interface ITimingCache::update to update the cache. Specifically, select tactics for some layers. Then, pass the cache to TensorRT. In the building process, TensorRT will use the newly assigned tactic. Unlike before, in the new version, only one tactic can be assigned to each layer.

Strongly Typed Networks

Since 11.0, TensorRT supports only Strongly Typed networks, meaning that the model must specify the precision for all tensors explicitly. This can be done for example using AutoCast or Quantization from Model-Optimizer, and TensorRT will adhere to the precision specifications. TensorRT will still autotune over different data layouts to find an optimal set of kernels for the network. Refer to the NVIDIA TensorRT Migration Guide if you are migrating from a Weakly Typed network.

TensorRT infers a type for each intermediate and output tensor using the rules in the operator type specification. TensorRT adheres to these inferred types while building the engine.

The builder flag kTF32 is permitted as it controls TF32 Tensor Core usage for FP32 types rather than controlling the use of TF32 data types.

Control of Computational Precision

TensorRT selects the computational precision for each layer based on tensor types in the network and performance considerations. In strongly typed networks (required since TensorRT 11.0), you control accumulation precision by how you define tensor types in the model graph.

Layers such as IConvolutionLayer and IMatrixMultiplyLayer accumulate sums of products. By default, the accumulation type is the input type or a wider type, at the whim of the optimizer. When you need to ensure wider accumulation (for example FP32 products from FP16 inputs), use ICastLayer to cast operands to the desired accumulation type before the operation, then cast the result back to the output precision. The builder normally fuses this cast pattern with the accumulating layer so you get FP32 accumulation without separate cast kernels at runtime, and TensorRT will recognize the pattern and consider efficient tactics that operate on the lower-precision types (FP16 in this example) for the convolution or matrix multiplication.

For quantized inputs, place an IDequantizeLayer before the accumulating layer and dequantize to FP32 when you need FP32 accumulation.

Matrix multiply and convolution. The cast-around-GEMM idiom above is the supported way to request FP32 accumulation for IMatrixMultiplyLayer and convolution-family layers in strongly typed networks. For a transformer attention example that applies the same pattern at scale, refer to Method 2: Construct an Attention Graph with Primitive INetwork Layers and the FP32 accumulation tip in Fused Attention.

Normalization. The Normalization operator supports a compute_precision attribute (see the Normalization operator documentation). When you need FP32 accumulation in a normalization layer, specify compute_precision in the imported model (for example via ONNX) or cast the normalization inputs to FP32 using ICastLayer before the layer.

For accuracy-oriented guidance on mixing precisions, refer to Accuracy Considerations.