SiameseOI with TAO Deploy

To generate an optimized TensorRT engine for SiameseOI, the gen_trt_engine action takes an ONNX file previously produced by the SiameseOI (visual_changenet) export action. For more information about training a SiameseOI model, refer to the SiameseOI training documentation.

Converting an ETLT or ONNX File into TensorRT Engine

gen_trt_engine

The gen_trt_engine section in the experiment specification file provides options for generating a TensorRT engine from an ONNX file. The following is an example configuration:

gen_trt_engine:
  results_dir: "${results_dir}/gen_trt_engine"
  onnx_file: "${results_dir}/export/oi_model.onnx"
  trt_engine: "${results_dir}/gen_trt_engine/oi_model.trt.v100"
  input_channel: 3
  input_width: 400
  input_height: 100
  tensorrt:
    data_type: fp32
    workspace_size: int = 1024
    min_batch_size: int = 1
    opt_batch_size: int = 1
    max_batch_size: int = 1

Parameter	Datatype	Default	Description	Supported Values
results_dir	string	–	The path to the results directory	–
onnx_file	string	–	The path to the exported ETLT or ONNX model	–
trt_engine	string	–	The absolute path to the generated TensorRT engine	–
input_channel	unsigned int	3	The input channel size. Only a value of 3 is supported.	3
input_width	unsigned int	400	The input width	>0
input_height	unsigned int	100	The input height	>0
batch_size	unsigned int	-1	The batch size of the ONNX model	>=-1

tensorrt

The tensorrt parameter defines TensorRT engine generation.

Parameter	Datatype	Default	Description	Supported Values
data_type	string	fp32	The precision to be used for the TensorRT engine	fp32/fp16/int8
workspace_size	unsigned int	1024	The maximum workspace size for the TensorRT engine	>1024
min_batch_size	unsigned int	1	The minimum batch size used for the optimization profile shape	>0
opt_batch_size	unsigned int	1	The optimal batch size used for the optimization profile shape	>0
max_batch_size	unsigned int	1	The maximum batch size used for the optimization profile shape	>0

Ask the agent to run the gen_trt_engine action against your spec. For example:

Build an FP16 TensorRT engine for SiameseOI from the exported ONNX at
``s3://my-bucket/siamese_oi/oi_model.onnx`` using ``trt-spec.yaml``.
Write the engine to ``s3://my-bucket/siamese_oi/oi_model.engine``. Run on
the local Docker daemon.

Running Inference through TensorRT Engine

You can reuse the specification file that was specified for TAO inference. The following is an example inference spec:

inference:
  gpu_id: 0
  trt_engine: /path/to/engine/file
  results_dir: "${results_dir}/inference"

Ask the agent to run the inference action against the engine you built. For example:

Run SiameseOI inference with the TensorRT engine at
``s3://my-bucket/siamese_oi/oi_model.engine`` using ``infer-spec.yaml``.
Run on your chosen backend.