Is this page helpful?

Strongly Typed Networks for DLA#

Important

DLA is not supported in TensorRT 11.3.0 or in TensorRT 11.3.1 for DriveOS. TensorRT 10.7 was the last release that supported DLA. DLA was also not supported in TensorRT 11.0, 11.1, or 11.2.

The guidance in this section is reference material for supported earlier releases. Do not use it as a support claim for TensorRT 11.3.0 or for TensorRT 11.3.1 for DriveOS.

On TensorRT releases where DLA is supported, all networks are strongly typed, including DLA networks. DLA on NVIDIA Orin through the SBSA package is not available in TensorRT 11.3.0.

The following builder flags are not valid for DLA builds:

  • BuilderFlag::kFP16

  • BuilderFlag::kINT8

  • BuilderFlag::kPREFER_PRECISION_CONSTRAINTS

  • BuilderFlag::kOBEY_PRECISION_CONSTRAINTS

Network tensor types, including types imported from an ONNX model, determine precision. For general strong-typing behavior and migration guidance, refer to Strongly Typed Networks and Migrating from TensorRT 10.x to 11.x.

FP16 Compute#

For FP16 inference, DLA computes in FP16. TensorRT infers this precision from the network tensor types; no precision builder flag is required. Ensure that the network tensors intended for DLA are typed as FP16 before building.

When parsing from ONNX, kADJUST_FOR_DLA can modify layers and tensor shapes to make them more amenable to DLA execution:

parser.set_flag(trt.OnnxParserFlag.ADJUST_FOR_DLA)

Add explicit cast operations for FP32-to-FP16 conversions at DLA subgraph boundaries. These boundary operations require GPU fallback. Layout transformations, such as NCHW16 to NCHW, do not need to be included in the ONNX model.

Quantization#

TensorRT 11 does not support implicit quantization. For workflows that transition between FP16 and quantized tensors, add explicit IQuantizeLayer (FP16 to UINT8) and IDequantizeLayer (UINT8 to FP16) layers. In ONNX models, use QuantizeLinear and DequantizeLinear nodes.

To enable UINT8 and asymmetric quantization while parsing an ONNX model, set kENABLE_UINT8_AND_ASYMMETRIC_QUANTIZATION_DLA:

parser.set_flag(
    trt.OnnxParserFlag.ENABLE_UINT8_AND_ASYMMETRIC_QUANTIZATION_DLA
)

The flag is not required for symmetric INT8 quantization. DLA supports asymmetric quantization for UINT8 (nonzero zero-point values).

Models whose tensors are specified natively as INT8 do not require explicit quantization nodes throughout the network. Explicit quantization remains the recommended way to represent transitions between FP16 and INT8.

DLA Data Types#

DLA supports FP16 and INT8 compute. It does not support FP8, BF16, or other reduced-precision compute types. NVIDIA TensorRT Model Optimizer AutoCast and quantization tools can prepare an ONNX model before parsing. DLA supports the resulting QuantizeLinear and DequantizeLinear nodes when kENABLE_UINT8_AND_ASYMMETRIC_QUANTIZATION_DLA is set for UINT8 or asymmetric quantization.