OCRNet with TAO Deploy#

To generate an optimized TensorRT engine for OCRNet, the gen_trt_engine action takes an ONNX file previously produced by the OCRNet export action. For more information about training the OCRNet, refer to the OCRNet training documentation.

Each task is explained in detail in the following sections.

Converting an .etlt or .onnx File into TensorRT Engine#

gen_trt_engine#

The gen_trt_engine parameter in the experiment specification file provides options to generate the TensorRT engine from .etlt` or .onnx.

gen_trt_engine:
    onnx_file: "??"
    results_dir: "${results_dir}/convert_dataset"

Parameter	Datatype	Default	Description	Supported Values
`onnx_file`	String	–	The absolute path to the exported `.etlt` or `.onnx` model	–
`trt_engine`	String	–	The absolute path to the generated TensorRT engine	–
`gpu_id`	Unsigned int	0	The GPU device index	Valid gpu index
`input_channel`	Unsigned int	1	The input channel of the TensorRT engine	>0
`input_width`	Unsigned int	100	The input width of the TensorRT engine	>0
`input_height`	Unsigned int	32	The input height of the TensorRT engine	>0
`opset_version`	Unsigned int	12	The ONNX opset version	Valid ONNX opset version
`batch_size`	Unsigned int	-1	The batch size of the TensorRT engine. Set it to `-1` to enable dynamic batch.	-1 or >0
`verbose`	Bool	False	A flag to enable verbose information output during TensorRT engine generation	True/False
`tensorrt`	Dict config	–	Other options for TensorRT-engine generation	–
`results_dir`	String	–	The absolute path to the `gen_trt_engine` log output	–

tensorrt#

The tensorrt parameter provides more options for TensorRT generation.

tensorrt:
    data_type: fp16
    workspace_size: 1024
    min_batch_size: 1
    opt_batch_size: 1
    max_batch_size: 1

Parameter	Datatype	Default	Description	Supported Values
`data_type`	String	fp16	The precision of the generated TensorRT engine	fp16,FP32
`workspace_size`	Unsigned int	1024	The workspace size of the generated TensorRT engine	>0
`min_batch_size`	Unsigned int	1	The minimum batch size of the generated TensorRT engine	>0
`opt_batch_size`	Unsigned int	1	The optimal batch size of the generated TensorRT engine	>0
`max_batch_size`	Unsigned int	1	The maximum batch size of the generated TensorRT engine	>0

Ask the agent to run the gen_trt_engine action against your spec. For example:

Build an FP16 TensorRT engine for OCRNet from the exported ONNX at
``s3://my-bucket/ocrnet/model.onnx`` using ``trt-spec.yaml``. Write the
engine to ``s3://my-bucket/ocrnet/model.engine``. Run on the local Docker backend.

Running Evaluation through the TensorRT Engine#

The evaluate parameter in the experiment specification file provides options to set evaluation with TensorRT engine:

evaluate:
    trt_engine: "??"
    test_dataset_dir: "/path/to/test_images_directory"
    test_dataset_gt_file: "/path/to/gt_file_list"
    input_width: 100
    input_height: 32

Parameter	Datatype	Default	Description	Supported Values
`trt_engine`	String	–	The absolute path to the TensorRT engine	–
`gput_id`	Unsigned int	0	The GPU device index	Valid gpu index
`test_dataset_dir`	String	–	The absolute path to the test images directory	–
`test_dataset_gt_file`	String	–	The absolute path to the ground truth file for `test_images`. The required format for the ground truth is described in the Preparing Data section.	>0
`input_width`	Unsigned int	100	The input width of the TensorRT engine	>0
`input_height`	Unsigned int	32	The input height of the TensorRT engine	>0
`batch_size`	Unsigned int	1	The batch size of the inference	>0
`results_dir`	String	–	The absolute path to the `gen_trt_engine` log output	–

Ask the agent to run the evaluate action against the engine you built. For example:

Evaluate the OCRNet TensorRT engine at
``s3://my-bucket/ocrnet/model.engine`` against ``eval-spec.yaml``. Run on
local Docker.

Running Inference through TensorRT Engine#

The inference parameter in the experiment specification file provides options to set evaluation with TensorRT engine:

inference:
    trt_engine: "??"
    inference_dataset_dir: "/path/to/test_images_directory"
    input_width: 100
    input_height: 32

Parameter	Datatype	Default	Description	Supported Values
`trt_engine`	String	–	The absolute path to the TensorRT engine	–
`gput_id`	Unsigned int	0	The GPU device index	Valid gpu index
`inference_dataset_dir`	String	–	The absolute path to the inference images directory	–
`input_width`	Unsigned int	100	The input width of the TensorRT engine	>0
`input_height`	Unsigned int	32	The input height of the TensorRT engine	>0
`batch_size`	Unsigned int	1	The batch size of the inference	>0
`results_dir`	String	–	The absolute path to the gen_trt_engine log output	–

Ask the agent to run the inference action against the engine you built. For example:

Run OCRNet inference with the TensorRT engine at
``s3://my-bucket/ocrnet/model.engine`` using ``infer-spec.yaml``. Run on
your chosen backend.