Is this page helpful?

Work With Quantized Types#

TensorRT-RTX supports the reduced-precision data types INT4, INT8, FP4, and FP8 for improved performance at the cost of accuracy. For per-architecture precision and GEMM support, refer to the Support Matrix.

You must explicitly select reduced precision (or quantization) for each layer that should use it. To do this, insert IQuantizeLayer and IDequantizerLayer (Q/DQ) nodes in the graph. You can perform quantization during the training process (Quantization Aware Training = QAT) or in a separate postprocessing step (Post-Training Quantization = PTQ).

Several popular deep learning frameworks allow model quantization using either QAT or PTQ, such as:

You can encode information about which layers to quantize in an ONNX model file using the QuantizeLinear - ONNX 1.19.0 documentation and DequantizeLinear - ONNX 1.19.0 documentation operators, and import it using the TensorRT-RTX ONNX parser.

Strongly Typed Networks and Explicit Quantization#

TensorRT-RTX is optimized for 16-bit floating point types and delivers throughput very close to TensorRT for models using these data types. If your models use 32-bit floating point types, we recommend applying quantization to achieve better performance.

As mentioned in the Porting Guide for TensorRT Applications, weakly-typed networks are not supported by TensorRT-RTX (they are also removed in TensorRT as of version 11.0). The weakly-typed network feature in TensorRT allowed applications to enable particular data types at the model level. For example, passing a --fp16 flag to trtexec caused TensorRT to automatically consider FP16 kernels. While this feature made it simple to get higher throughput without quantization, it could compromise accuracy and didn’t provide adequate control to model authors.

Similarly, implicit quantization is not supported by TensorRT-RTX. To quantize a model, you must add quantize and dequantize layers to the network.

To achieve excellent performance and accuracy, strongly typed networks are the only supported approach for TensorRT-RTX and are also recommended for TensorRT.

The most straightforward way to get results similar to using the trtexec --fp16 flag or BuilderFlag::kFP16 is to use ModelOpt’s AutoCast feature on your model. Refer to the ModelOpt AutoCast documentation for more information; however, for a basic conversion from FP32 layers to FP16, run:

python -m modelopt.onnx.autocast --onnx_path model.onnx

For quantizing a network, run:

python -m modelopt.onnx.quantization --onnx_path model.onnx --calibration_data data.npz

For more control or other advanced quantization topics (such as quantization-aware training), refer to the ModelOpt Quantization documentation.

You may also choose to manually control a layer’s I/O types by using the INetworkDefinition::addCast API where applicable, or by adding Cast nodes to the ONNX graph.

Similarly, you may apply quantization to a layer in your network using the INetworkDefinition::addQuantize and INetworkDefinition::addDequantize APIs, or adding QuantizeLinear and DequantizeLinear nodes to the ONNX graph manually. This approach works, but figuring out which layers must be run at high precision, and calibrating the scales for quantization to get the best accuracy can be challenging. For most models, ModelOpt is the recommended approach to apply quantization effectively.

Finally, the NVIDIA TensorRT Model Optimizer (TensorRT-Model-Optimizer/examples/windows at main · NVIDIA/TensorRT-Model-Optimizer · GitHub) is an open source tool specifically developed to help TensorRT and TensorRT-RTX users add quantization to a pretrained model to achieve higher performance.