DLA Standalone Mode

Important

DLA is not supported in TensorRT 11.3.0 or in TensorRT 11.3.1 for DriveOS. TensorRT 10.7 was the last release that supported DLA. DLA was also not supported in TensorRT 11.0, 11.1, or 11.2.

The guidance in this section is reference material for supported earlier releases. Do not use it as a support claim for TensorRT 11.3.0 or for TensorRT 11.3.1 for DriveOS.

If you need to run inference outside of TensorRT, you can use EngineCapability::kDLA_STANDALONE to generate a DLA loadable instead of a TensorRT engine. This loadable can then be used with the cuDLA API.

Building A DLA Loadable

1. Set the default device type and engine capability to DLA standalone mode.

   C++

   builderConfig->setDefaultDeviceType(DeviceType::kDLA);
   builderConfig->setEngineCapability(EngineCapability::kDLA_STANDALONE);
   

   Python

   builder_config.default_device_type = trt.DeviceType.DLA
   builder_config.engine_capability = trt.EngineCapability.DLA_STANDALONE
   

2. Specify the desired precision (FP16, INT8, or mixed) with network tensor types rather than builder flags. In TensorRT 11.0 and later, all networks are strongly typed and NetworkDefinitionCreationFlag::kSTRONGLY_TYPED is deprecated and ignored. Tensor types in the network or a pre-quantized ONNX import drive precision. Refer to Strongly Typed Networks for DLA for DLA-specific guidance.

3. DLA standalone mode disallows reformatting; therefore, BuilderFlag::kDIRECT_IO needs to be set.

   C++

   builderConfig->setFlag(BuilderFlag::kDIRECT_IO);
   

   Python

   builder_config.set_flag(trt.BuilderFlag.DIRECT_IO)
   

4. Set the allowed formats for I/O tensors to one or more of those that are DLA-supported.

5. Build as normal.

Using trtexec To Generate A DLA Loadable

The trtexec tool can generate a DLA loadable instead of a TensorRT engine. Specifying both --useDLACore and --safe parameters sets the builder capability to EngineCapability::kDLA_STANDALONE. Specifying --inputIOFormats and --outputIOFormats restricts I/O data type and memory layout. The DLA loadable is saved into a file by specifying --saveEngine parameter.

For example, to generate an FP16 DLA loadable for an ONNX model using trtexec, run:

./trtexec --onnx=model.onnx --saveEngine=model_loadable.bin --useDLACore=0 --fp16 --inputIOFormats=fp16:chw16 --outputIOFormats=fp16:chw16 --skipInference --safe