SiameseOI with TAO Deploy#

To generate an optimized TensorRT engine, a SiameseOI .etlt or .onnx file, which is first generated using tao model visual_changenet export, is taken as an input to tao deploy visual_changenet gen_trt_engine. For more information about training a SiameseOI model, refer to the SiameseOI training documentation.

Note

Throughout this documentation, you will see references to $EXPERIMENT_ID and $DATASET_ID in the FTMS Client sections.
- For instructions on creating a dataset using the remote client, see the Creating a dataset section in the Remote Client documentation.
- For instructions on creating an experiment using the remote client, see the Creating an experiment section in the Remote Client documentation.
The spec format is YAML for TAO Launcher and JSON for FTMS Client.
File-related parameters, such as dataset paths or pretrained model paths, are required only for TAO Launcher and not for FTMS Client.

Converting an ETLT or ONNX File into TensorRT Engine#

gen_trt_engine#

The gen_trt_engine section in the experiment specification file provides options for generating a TensorRT engine from an .etlt or .onnx file. The following is an example configuration:

Use the following command to get an experiment spec file for ReIdentificationNet:

SPECS=$(tao-client optical_inspection get-spec --action train --job_type experiment --id $EXPERIMENT_ID)

gen_trt_engine:
  results_dir: "${results_dir}/gen_trt_engine"
  onnx_file: "${results_dir}/export/oi_model.onnx"
  trt_engine: "${results_dir}/gen_trt_engine/oi_model.trt.v100"
  input_channel: 3
  input_width: 400
  input_height: 100
  tensorrt:
    data_type: fp32
    workspace_size: int = 1024
    min_batch_size: int = 1
    opt_batch_size: int = 1
    max_batch_size: int = 1

Parameter	Datatype	Default	Description	Supported Values
`results_dir`	string	–	The path to the results directory	–
`onnx_file`	string	–	The path to the exported ETLT or ONNX model	–
`trt_engine`	string	–	The absolute path to the generated TensorRT engine	–
`input_channel`	unsigned int	3	The input channel size. Only a value of 3 is supported.	3
`input_width`	unsigned int	400	The input width	>0
`input_height`	unsigned int	100	The input height	>0
`batch_size`	unsigned int	-1	The batch size of the ONNX model	>=-1

tensorrt#

The tensorrt parameter defines TensorRT engine generation.

Parameter	Datatype	Default	Description	Supported Values
`data_type`	string	fp32	The precision to be used for the TensorRT engine	fp32/fp16/int8
`workspace_size`	unsigned int	1024	The maximum workspace size for the TensorRT engine	>1024
`min_batch_size`	unsigned int	1	The minimum batch size used for the optimization profile shape	>0
`opt_batch_size`	unsigned int	1	The optimal batch size used for the optimization profile shape	>0
`max_batch_size`	unsigned int	1	The maximum batch size used for the optimization profile shape	>0

Use the following command to run SiameseOI engine generation:

GTE_JOB_ID=$(tao-client optical_inspection experiment-run-action --action gen_trt_engine --id $EXPERIMENT_ID --parent_job_id $EXPORT_JOB_ID --specs "$SPECS")

See also

The Export job ID is the job ID of the tao-client optical_inspection experiment-run-action --action export command.

tao deploy optical_inspection gen_trt_engine -e /path/to/spec.yaml \
          results_dir=/path/to/etlt/file \
          gen_trt_engine.onnx_file=/path/to/onnx/file \
          gen_trt_engine.trt_engine=/path/to/engine/file \
          gen_trt_engine.tensorrt.data_type=<data_type>

Required Arguments

-e, --experiment_spec_file: The path to the experiment spec file
results_dir: The global results directory. The engine generation log will be saved in the results_dir.
gen_trt_engine.onnx_file: The .onnx model to be converted
gen_trt_engine.trt_engine: The path where the generated engine will be stored
gen_trt_engine.tensorrt.data_type: The precision to be exported

Sample Usage

Here’s an example of using the gen_trt_engine command to generate an FP16 TensorRT engine:

tao deploy optical_inspection gen_trt_engine -e $DEFAULT_SPEC
            results_dir=$RESULTS_DIR
            gen_trt_engine.onnx_file=$ONNX_FILE \
            gen_trt_engine.trt_engine=$ENGINE_FILE \
            gen_trt_engine.tensorrt.data_type=FP16

Running Inference through TensorRT Engine#

You can reuse the spec file that was specified for TAO inference. The following is an example inference spec:

inference:
  gpu_id: 0
  trt_engine: /path/to/engine/file
  results_dir: "${results_dir}/inference"

Use the following command to run SiameseOI engine inference:

EVAL_JOB_ID=$(tao-client optical_inspection experiment-run-action --action inference --id $EXPERIMENT_ID --parent_job_id $GTE_JOB_ID --specs "$SPECS")

tao deploy optical_inspection inference -e /path/to/spec.yaml \
          results_dir=$RESULTS_DIR \

Required Arguments

-e, --experiment_spec_file: The path to the experiment spec file
results_dir: The global results directory. The engine generation log will be saved in the results_dir.

Sample Usage

Here’s an example of using the inference command to run inference with the TensorRT engine:

tao deploy optical_inspection inference -e $DEFAULT_SPEC
            results_dir=$RESULTS_DIR