Is this page helpful?

Strongly Typed Networks for DLA#

Important

DLA support depends on your platform and package. TensorRT 11.4.1 supports DLA on NVIDIA Jetson AGX Orin when using NVIDIA JetPack 7.2.2 or later, and on Windows on ARM. DLA is not supported on NVIDIA Jetson Thor or for NVIDIA DriveOS deployments. Refer to Working with DLA for the platforms and packages that support DLA.

All networks are strongly typed, including DLA networks. DLA on NVIDIA Jetson AGX Orin is delivered through the Linux SBSA package installed under NVIDIA JetPack 7.2.2 or later, and those builds are strongly typed: precision is determined by network tensor types instead of the per-precision builder flags used before TensorRT 11.x.

The following builder flags were removed in TensorRT 11.0 and no longer exist:

  • BuilderFlag::kFP16

  • BuilderFlag::kINT8

  • BuilderFlag::kPREFER_PRECISION_CONSTRAINTS

  • BuilderFlag::kOBEY_PRECISION_CONSTRAINTS

Network tensor types, including types imported from an ONNX model, determine precision. An ONNX model that was never converted to FP16 keeps its original tensor types, so converting the model is what enables FP16 compute on DLA.

To migrate an existing DLA model, use the NVIDIA TensorRT Model Optimizer ONNX AutoCast tool to convert it to a strongly typed, FP16 representation:

python3 -m modelopt.onnx.autocast \
    --target_dla \
    --low_precision_type fp16 \
    --onnx_path model.onnx \
    --output_path model_fp16.onnx \
    --opset 20

Note

Models migrated from weakly typed networks or implicit quantization to strong typing or explicit quantization may show performance regressions. Try the AutoCast tool both with and without the --target_dla option and keep whichever gives better performance for your model.

If some operators do not convert to FP16 as expected, use --data_max to set the maximum absolute activation value allowed before an operator is kept in FP32. AutoCast also converts the network I/O tensor types to FP16 by default. If your deployment requires FP32 I/O, preserve the original I/O types during conversion using the keep_io_types option shown in Python Migration Patterns for TensorRT 11.x.

For general strong-typing behavior and migration guidance, refer to Strongly Typed Networks and Migrating from TensorRT 10.x to 11.x.

FP16 Compute#

For FP16 inference, DLA computes in FP16. TensorRT infers this precision from the network tensor types; no precision builder flag is required. Ensure that the network tensors intended for DLA are typed as FP16 before building.

When parsing from ONNX, kADJUST_FOR_DLA can modify layers and tensor shapes to make them more amenable to DLA execution:

parser.set_flag(trt.OnnxParserFlag.ADJUST_FOR_DLA)

Add explicit cast operations for FP32-to-FP16 conversions at DLA subgraph boundaries. These boundary operations require GPU fallback. Layout transformations, such as NCHW16 to NCHW, do not need to be included in the ONNX model.

Quantization#

TensorRT 11 does not support implicit quantization. For workflows that transition between FP16 and quantized tensors, add explicit IQuantizeLayer and IDequantizeLayer layers. On NVIDIA Jetson AGX Orin, the quantized compute type is INT8, so these layers convert between FP16 and INT8. On Windows on ARM, UINT8 quantization is restricted to the I/O boundaries, where IQuantizeLayer converts FP16 to UINT8 and IDequantizeLayer converts UINT8 back to FP16 or FP32; refer to DLA Supported Layers for Windows on ARM. In ONNX models, use QuantizeLinear and DequantizeLinear nodes.

To enable UINT8 and asymmetric quantization while parsing an ONNX model, set kENABLE_UINT8_AND_ASYMMETRIC_QUANTIZATION_DLA:

parser.set_flag(
    trt.OnnxParserFlag.ENABLE_UINT8_AND_ASYMMETRIC_QUANTIZATION_DLA
)

The flag is not required for symmetric INT8 quantization. DLA supports asymmetric quantization for UINT8 (nonzero zero-point values).

For validated Windows on ARM shapes, refer to IQuantizeLayer and IDequantizeLayer in Layer Details.

Models whose tensors are specified natively as INT8 do not require explicit quantization nodes throughout the network. Explicit quantization remains the recommended way to represent transitions between FP16 and INT8.

DLA Data Types#

DLA supports FP16 and INT8 compute. It does not support FP8, BF16, or other reduced-precision compute types.

Note

Pure INT8 compute is not supported on Windows on ARM, and that platform has no GPU fallback, so an INT8 network fails to build rather than degrading to FP16. Refer to DLA Supported Layers for Windows on ARM before targeting INT8 there.

NVIDIA TensorRT Model Optimizer AutoCast and quantization tools can prepare an ONNX model before parsing. DLA supports the resulting QuantizeLinear and DequantizeLinear nodes when kENABLE_UINT8_AND_ASYMMETRIC_QUANTIZATION_DLA is set for UINT8 or asymmetric quantization.