Deformable DETR with TAO Deploy#

To generate an optimized TensorRT engine for Deformable DETR, the gen_trt_engine action takes an ONNX file previously produced by the Deformable DETR export action. For more information about training a Deformable DETR model, refer to the Deformable DETR training documentation.

Converting .onnx File into TensorRT Engine#

The following is a sample specification file for Deformable DETR engine generation:

gen_trt_engine:
  onnx_file: /path/to/onnx_file
trt_engine: /path/to/trt_engine
input_channel: 3
input_width: 960
input_height: 544
tensorrt:
  data_type: int8
  workspace_size: 1024
  min_batch_size: 1
  opt_batch_size: 10
  max_batch_size: 10
  calibration:
    cal_image_dir:
      - /path/to/cal/images
    cal_cache_file: /path/to/cal.bin
    cal_batch_size: 10
    cal_batches: 1000

Parameter	Datatype	Default	Description	Supported Values
`onnx_file`	string		The precision to be used for the TensorRT engine
`trt_engine`	string		The maximum workspace size for the TensorRT engine
`input_channel`	unsigned int	3	The input channel size. Only a value of 3 is supported.	3
`input_width`	unsigned int	960	The input width	>0
`input_height`	unsigned int	544	The input height	>0
`batch_size`	unsigned int	-1	The batch size of the ONNX model	>=-1

tensorrt#

The tensorrt parameter defines the TensorRT engine generation.

Parameter	Datatype	Default	Description	Supported Values
`data_type`	string	fp32	The precision to be used for the TensorRT engine	fp32/fp16/int8
`workspace_size`	unsigned int	1024	The maximum workspace size for the TensorRT engine	>1024
`min_batch_size`	unsigned int	1	The minimum batch size used for the optimization profile shape	>0
`opt_batch_size`	unsigned int	1	The optimal batch size used for the optimization profile shape	>0
`max_batch_size`	unsigned int	1	The maximum batch size used for the optimization profile shape	>0

calibration#

The calibration parameter defines the TensorRT engine generation with PTQ INT8 calibration.

Parameter	Datatype	Default	Description	Supported Values
`cal_image_dir`	string list		The list of paths that contain images used for calibration
`cal_cache_file`	string		The path to calibration cache file to be dumped
`cal_batch_size`	unsigned int	1	The batch size per batch during calibration	>0
`cal_batches`	unsigned int	1	The number of batches to calibrate	>0

Ask the agent to run the gen_trt_engine action against your spec. For example:

Build an FP16 TensorRT engine for Deformable DETR from the exported ONNX
at ``s3://my-bucket/ddetr/model.onnx`` using ``trt-spec.yaml``. Write the
engine to ``s3://my-bucket/ddetr/model.engine``. Run on local Docker.

Running Evaluation through TensorRT Engine#

You can reuse the TAO evaluation specification file for evaluation through a TensorRT engine. The following is a sample specification file:

evaluate:
  trt_engine: /path/to/engine/file
  conf_threshold: 0.0
  input_width: 960
  input_height: 544
dataset:
  test_data_sources:
    image_dir: /data/raw-data/val2017/
    json_file: /data/raw-data/annotations/instances_val2017.json
  num_classes: 91
  batch_size: 8

Ask the agent to run the evaluate action against the engine you built. For example:

Evaluate the Deformable DETR TensorRT engine at
``s3://my-bucket/ddetr/model.engine`` against ``eval-spec.yaml``. Run on
local Docker.

Running Inference through TensorRT Engine#

You can reuse the TAO inference specification file for inference through a TensorRT engine. The following is a sample specification file:

inference:
  conf_threshold: 0.5
  input_width: 960
  input_height: 544
  trt_engine: /path/to/engine/file
  color_map:
    person: green
    car: red
    cat: blue
dataset:
  infer_data_sources:
    image_dir: /data/raw-data/val2017/
    classmap: /path/to/coco/annotations/coco_classmap.txt
  num_classes: 91
  batch_size: 8

Ask the agent to run the inference action against the engine you built. For example:

Run Deformable DETR inference with the TensorRT engine at
``s3://my-bucket/ddetr/model.engine`` using ``infer-spec.yaml``. Run on
your chosen backend.

Annotated visualizations are written to images_annotated under the configured results directory, and KITTI-format predictions are written to labels.