RT-DETR with TAO Deploy#

To generate an optimized TensorRT engine for RT-DETR, the gen_trt_engine action takes an ONNX file previously produced by the RT-DETR export action. For more information about training a RT-DETR model, refer to the RT-DETR training documentation.

Converting RT-DETR .onnx File into TensorRT Engine#

You can reuse the spec from the RT-DETR export action as a starting point.

gen_trt_engine#

The gen_trt_engine parameter defines TensorRT engine generation.

gen_trt_engine:
  onnx_file: /path/to/onnx_file
  trt_engine: /path/to/trt_engine
  input_channel: 3
  input_width: 640
  input_height: 640
  tensorrt:
    data_type: int8
    workspace_size: 1024
    min_batch_size: 1
    opt_batch_size: 10
    max_batch_size: 10
    calibration:
      cal_image_dir:
        - /path/to/cal/images
      cal_cache_file: /path/to/cal.bin
      cal_batch_size: 10
      cal_batches: 1000

Parameter	Datatype	Default	Description	Supported Values
`onnx_file`	string		The precision to be used for the TensorRT engine
`trt_engine`	string		The maximum workspace size for the TensorRT engine
`input_channel`	unsigned int	3	The input channel size. Only a value of 3 is supported.	3
`input_width`	unsigned int	960	The input width	>0
`input_height`	unsigned int	544	The input height	>0
`batch_size`	unsigned int	-1	The batch size of the ONNX model	>=-1

tensorrt#

The tensorrt parameter defines the TensorRT engine generation.

Parameter	Datatype	Default	Description	Supported Values
`data_type`	string	fp32	The precision to be used for the TensorRT engine	fp32/fp16/int8
`workspace_size`	unsigned int	1024	The maximum workspace size for the TensorRT engine	>1024
`min_batch_size`	unsigned int	1	The minimum batch size used for the optimization profile shape	>0
`opt_batch_size`	unsigned int	1	The optimal batch size used for the optimization profile shape	>0
`max_batch_size`	unsigned int	1	The maximum batch size used for the optimization profile shape	>0

calibration#

The calibration parameter defines the TensorRT engine generation with PTQ INT8 calibration.

Parameter	Datatype	Default	Description	Supported Values
`cal_image_dir`	string list		The list of paths that contain images used for calibration
`cal_cache_file`	string		The path to calibration cache file to be dumped
`cal_batch_size`	unsigned int	1	The batch size per batch during calibration	>0
`cal_batches`	unsigned int	1	The number of batches to calibrate	>0

Note

For RT-DETR, int8 calibration is only supported for the ResNet series of backbones.

Ask the agent to run the gen_trt_engine action against your spec. For example:

Build an FP16 TensorRT engine for RT-DETR from the exported ONNX at
``s3://my-bucket/rtdetr/model.onnx`` using ``trt-spec.yaml``. Write the
engine to ``s3://my-bucket/rtdetr/model.engine``. Run on the local Docker backend.

Running Evaluation through TensorRT Engine#

You can reuse the TAO evaluation specification file for evaluation through a TensorRT engine. The following is a sample specification file:

evaluate:
  trt_engine: /path/to/engine/file
  conf_threshold: 0.0
  input_width: 640
  input_height: 640
dataset:
  test_data_sources:
    image_dir: /data/raw-data/val2017/
    json_file: /data/raw-data/annotations/instances_val2017.json
  num_classes: 80
  batch_size: 8

Ask the agent to run the evaluate action against the engine you built. For example:

Evaluate the RT-DETR TensorRT engine at
``s3://my-bucket/rtdetr/model.engine`` against ``eval-spec.yaml``. Run on
local Docker.

Running Inference through TensorRT Engine#

You can reuse the TAO inference specification file for inference through a TensorRT engine. The following is a sample specification file:

inference:
  conf_threshold: 0.5
  input_width: 640
  input_height: 640
  trt_engine: /path/to/engine/file
  color_map:
    person: green
    car: red
    cat: blue
dataset:
  infer_data_sources:
    image_dir: ["/data/raw-data/val2017/"]
    classmap: /path/to/coco/annotations/coco_classmap.txt
  num_classes: 80
  batch_size: 8

Ask the agent to run the inference action against the engine you built. For example:

Run RT-DETR inference with the TensorRT engine at
``s3://my-bucket/rtdetr/model.engine`` using ``infer-spec.yaml``. Run on
your chosen backend.

Annotated visualizations are written to images_annotated under the configured results directory, and KITTI-format predictions are written to labels.