Is this page helpful?

DLA Runtime Configuration#

Important

DLA is not supported in TensorRT 11.3.0 or in TensorRT 11.3.1 for DriveOS. TensorRT 10.7 was the last release that supported DLA. DLA was also not supported in TensorRT 11.0, 11.1, or 11.2.

The guidance in this section is reference material for supported earlier releases. Do not use it as a support claim for TensorRT 11.3.0 or for TensorRT 11.3.1 for DriveOS.

This page covers GPU fallback. It covers DLA I/O formats. It also covers workspace allocation when multiple engines are loaded. It covers cuDLA tensor registration.

GPU Fallback Mode#

The GPUFallbackMode sets the builder to use GPU. This applies when a layer marked to run on DLA cannot run on DLA. A layer cannot run on DLA for the following reasons:

  • The layer operation is not supported on DLA.

  • The parameters specified are out of the supported range for DLA.

  • The given batch size exceeds the maximum permissible DLA batch size. For more information, refer to the DLA Supported Layers and Restrictions section.

  • A combination of layers in the network causes the internal state to exceed what the DLA can support.

  • There are no DLA engines available on the platform.

Important

When GPU fallback is enabled, layers that cannot run on DLA silently fall back to GPU execution without an error. When GPU fallback is disabled, an error is emitted instead. Verify DLA layer assignments in the engine inspector output to confirm expected behavior.

I/O Formats on DLA#

DLA supports formats that are unique to the device. These formats have constraints on their layout due to vector width byte requirements.

For DLA input tensors, kDLA_LINEAR(FP16, INT8), kDLA_HWC4(FP16, INT8), kCHW16(FP16), and kCHW32(INT8) are supported.

For DLA output tensors, only kDLA_LINEAR(FP16, INT8), kCHW16(FP16), and kCHW32(INT8) are supported.

For kCHW16 and kCHW32 formats, C must be an integer multiple. If it is not, pad it to the next 32-byte boundary.

For kDLA_LINEAR format, the stride along the W dimension must be padded up to 64 bytes. The memory format is equivalent to a C array with dimensions [N][C][H][roundUp(W, 64/elementSize)]. Here elementSize is 2 for FP16 and 1 for Int8. The tensor coordinates (n, c, h, w) map to array subscript [n][c][h][w].

For kDLA_HWC4 format, the stride along the W dimension must be a multiple of 32 bytes on Xavier. On NVIDIA Orin it must be a multiple of 64 bytes.

  • When C == 1, TensorRT maps the format to the native grayscale image format.

  • When C == 3 or C == 4, it maps to the native color image format. If C == 3, the stride for stepping along the W axis must be padded to 4 in elements.

    • In this case, the padded channel is located at the 4th index. Ideally, the padding value does not matter. The DLA compiler paddings the 4th channel in the weights to zero. It is safe for the application to allocate a zero-filled buffer of four channels and populate three valid channels.

  • When C is {1, 3, 4}, then padded C' is {1, 4, 4} respectively. The memory layout is equivalent to a C array with dimensions [N][H][roundUp(W, 32/C'/elementSize)][C']. Here elementSize is 2 for FP16 and 1 for Int8. The tensor coordinates (n, c, h, w) map to array subscript [n][h][w][c]. roundUp calculates the smallest multiple of 64/elementSize greater than or equal to W.

When using kDLA_HWC4 as the DLA input format, it has the following requirements:

  • C must be 1, 3, or 4.

  • The first layer must be convolution.

  • The convolution parameters must meet DLA requirements. For more information, refer to the DLA Supported Layers and Restrictions section.

When GPU fallback is enabled, TensorRT can insert reformatting layers to meet the DLA requirements. Otherwise, the input and output formats must be compatible with DLA. In all cases, query IExecutionContext::getStrides. That call returns the strides that TensorRT expects.

DLA Workspace Memory#

When multiple DLA engines are loaded concurrently, use IRuntime::setDLAWorkspaceAllocationStrategy() to control workspace memory usage. Call this method before deserializing each engine. Engines that have already been deserialized retain the strategy that was active when they were deserialized.

DLAWorkspaceAllocationStrategy provides the following strategies:

kDEFAULT

Each engine allocates its own workspace independently. This is the default.

kSHARED_STATIC

Engines deserialized by the same IRuntime for the same DLA core share a single static workspace pool. Use this strategy to reduce peak memory usage when loading multiple engines simultaneously. Each DLA core has a separate pool. If you do not select a core with IRuntime::setDLACore(), TensorRT uses DLA core 0.

Warning

Do not execute engines that share a workspace pool concurrently. Concurrent execution has undefined behavior. Serialize execution across these engines.

DLA Tensor Memory Registration#

When DLA is enabled, IExecutionContext::setTensorAddress() also controls registration of the tensor’s CUDA allocation with cuDLA.

  • setTensorAddress(name, ptr) defers cuDLA registration until immediately before the first execution that uses the binding.

  • setTensorAddress(name, nullptr) immediately unregisters the allocation from cuDLA.

Warning

Call setTensorAddress(name, nullptr) before freeing an allocation or binding the tensor to a new address. Rebinding a tensor without first unregistering its current allocation results in an error.

The following example unregisters ptrA before freeing it and binding ptrB:

1context->setTensorAddress(name, ptrA);
2context->enqueueV3(stream);
3
4context->setTensorAddress(name, nullptr);
5cudaFree(ptrA);
6context->setTensorAddress(name, ptrB);
7context->enqueueV3(stream);

1context.set_tensor_address(name, ptrA)
2context.execute_async_v3(stream)
3
4context.set_tensor_address(name, 0)  # Unregister. Equivalent to nullptr.
5cudaFree(ptrA)  # e.g. via your CUDA python binding
6context.set_tensor_address(name, ptrB)
7context.execute_async_v3(stream)