TAO Deploy Overview

TAO Toolkit v5.2.0

NVIDIA TensorRT is an SDK for high-performance deep learning inference. It provides APIs and parsers to import trained models from all major deep learning frameworks; it then generates optimized runtime engines deployable in a data center, as well as in automotive and embedded environments. To understand TensorRT and its capabilities better, refer to the official TensorRT documentation.

Models trained in TAO Toolkit are deployed to NVIDIA inference SDKs, like DeepStream, via TensorRT. Computer vision models trained by TAO Toolkit can be consumed by TensorRT via tao deploy, which is included as part of the tao launcher. TAO Deploy parses the exported .onnx model file and generates an optimized TensorRT engine. These engines can be generated to support inference at low precision (e.g. FP16 or INT8). While most of the TAO models support direct integration of the .onnx files with DeepStream, DeepStream can also consume the optimized engine generated by tao deploy.

TAO Deploy separates the model training and optimization steps from deployment by parsing a .onnx file to generate an optimized TensorRT engine. TAO Deploy also provides tools to run evaluation and inference using the original TAO spec file. With TAO Deploy, you can perform the following tasks:

  • gen_trt_engine

  • evaluate

  • inference

Like other TAO commands, the TAO Deploy CLI follows a cascaded structure:


tao deploy <task> <sub-task> <args>

Currently, TAO Deploy only supports computer vision models. For example, DetectNet_v2 is a computer vision task for object detection in TAO Toolkit and supports the gen_trt_engine, evaluate, and inference subtasks. When you execute a command like tao deploy detectnet_v2 gen_trt_engine --help, the TAO Toolkit Launcher does the following:

  1. Pulls the TAO Deploy container with the entrypoint for detectnet_v2.

  2. Creates an instance of the container.

  3. Runs the detectnet_v2 entrypoint with the getn_trt_engine sub-task.

Once the TAO launcher has been installed, the workflow to run the launcher is as follows.

  1. List the tasks supported in the deploy docker.

    You can list the tao deploy tasks that are supported in the TAO Toolkit Launcher using the tao deploy --help` command:


    usage: tao deploy [-h] {list,stop,info,dataset,deploy,model} ... {classification_pyt,classification_tf1,classification_tf2,deformable_detr,detectnet_v2,dino,dssd,efficientdet_tf1,efficientdet_tf2,faster_rcnn,lprnet,mask_rcnn,ml_recog,multitask_classification,ocdnet,ocrnet,optical_inspection,retinanet,segformer,ssd,unet,yolo_v3,yolo_v4,yolo_v4_tiny} ... optional arguments: -h, --help show this help message and exit task_groups: {list,stop,info,dataset,deploy,model} task: {classification_pyt,classification_tf1,classification_tf2,deformable_detr,detectnet_v2,dino,dssd,efficientdet_tf1,efficientdet_tf2,faster_rcnn,lprnet,mask_rcnn,ml_recog,multitask_classification,ocdnet,ocrnet,optical_inspection,retinanet,segformer,ssd,unet,yolo_v3,yolo_v4,yolo_v4_tiny}

  2. Configure the launcher instance.

    This step is identical to the regular tao launcher steps. For more details, refer to TAO Launcher documentation.

  3. Run a task.

    You use the following command format to run tasks supported by TAO Toolkit:


    tao deploy <task> <sub-task> <cli_args>

    To view the sub-tasks supported by a certain task, you can use the help command. For example, the following command lists the tasks for detectnet_v2:


    $ tao deploy detectnet_v2 --help usage: detectnet_v2 [-h] [--gpu_index GPU_INDEX] [--log_file LOG_FILE] {evaluate,gen_trt_engine,inference} ... Transfer Learning Toolkit optional arguments: -h, --help show this help message and exit --gpu_index GPU_INDEX The index of the GPU to be used. --log_file LOG_FILE Path to the output log file. tasks: {evaluate,gen_trt_engine,inference}

Previous Deploying nvOCDR to DeepStream
Next TAO Deploy Installation
© Copyright 2024, NVIDIA. Last updated on Mar 18, 2024.