NVIDIA TAO Toolkit v4.0

Introduction

• Overview
  
  • TAO Toolkit Architecture
  • TAO Computer Vision Workflow Overview
  • TAO Conversational AI Workflow Overview
  • Model Pruning
  • Learning Resources
• TAO Toolkit Quick Start Guide
  
  • Requirements
  • Running TAO Toolkit
  • Run sample jupyter notebooks
  • Downloading the Models
• TAO Toolkit Launcher
  
  • Running the launcher
  • Handling launched processes
  • Useful Environment variables
• Migrating from older TLT to TAO Toolkit
• Migrating from TAO Toolkit 3.x to TAO Toolkit 4.0
  
  • Container Mapping
• TAO Model Export and INT8 Calibration Changes
• Working With the Containers
  
  • Invoking the Containers Directly
  • Running Multi-Node Training
  • Running without Elevated User Privileges

Model Zoo

• Overview
• Computer Vision Model Zoo
• Conversational AI Model Zoo

Running TAO Toolkit in the Cloud

• Running TAO Toolkit in the Cloud
• Running TAO Toolkit on an AWS VM
  
  • Pre-Requisites
  • Setting up an AWS EC2 instance
  • Installing the Pre-Requisites for TAO Toolkit in the VM
  • Download and run the test samples
• Running TAO Toolkit on Google Cloud Platform
  
  • Setting up a VM Linux VM Instance
  • Using the VM
  • Setting up the VM and Enabling GPUs
  • Installing the Pre-requisites for TAO Toolkit
  • Downloading and Running Test Samples
• Running TAO Toolkit on an Azure VM
  
  • Setting up an Azure VM
  • Installing the Pre-Requisites for TAO Toolkit in the VM
  • Downloading and Running the Test Samples
• Running TAO Toolkit on Google Colab
  
  • Pre-Requisites
  • Launching Notebooks with Google Colab
  • Utility scripts to obtain subset of data
  • Steps to Locate Files in a Colab Notebook
  • Notes
• Running TAO Toolkit on an EKS
• Running TAO Toolkit on an AKS

TAO Toolkit API

• Overview
• Setup
  
  • Bare-Metal Setup
  • AWS EKS Setup
  • Azure AKS Setup
• Deployment
• REST API
  
  • User Authentication
  • API Specs
  • Examples
• Remote Client
  
  • Installation
  • Storage Topology
  • CLI Specs
  • Examples
• API Reference
• Action Specs
  
  • classification
  • detectnet_v2
  • dssd
  • efficientdet
  • faster_rcnn
  • semantic_segmentation
  • instance_segmentation
  • lprnet
  • mask_rcnn
  • multitask_classification
  • object_detection
  • retinanet
  • spectro_gen
  • speech
  • ssd
  • unet
  • vocoder
  • yolo_v3
  • yolo_v4
  • yolo_v4_tinny

AutoML

• AutoML
  
  • AutoML Supported Applications and Models
  • Prerequisites
  • AutoML Notebooks
  • Getting Started
  • AutoML Outcomes
  • AutoML Algorithm Explanation
  • Hyperband Parameter Auto-adjustment Mechanism

CV Applications

• Offline Data Augmentation
  
  • Object Detection
• Optimizing the Training Pipeline
  
  • Quantization Aware Training
  • Automatic Mixed Precision
• Visualizing Training
  
  • Enabling Tensorboard during Training
  • Visualizing using TensorBoard
  • Additional Resources
• Data Annotation Format
  
  • Image Classification Format
  • Object Detection – KITTI Format
  • Object Detection – COCO Format
  • Instance Segmentation – COCO format
  • Semantic Segmentation - PNG Mask Format
  • Gesture Recognition – Custom Format
  • Heart Rate Estimation – Custom Format
  • EmotionNet, FPENET, GazeNet – JSON Label Data Format
  • BodyposeNet – COCO Format
• Image Classification (TF1)
  
  • Preparing the Input Data Structure
  • Creating an Experiment Spec File - Specification File for Classification
  • Training the model
  • Evaluating the Model
  • Running Inference on a Model
  • Pruning the Model
  • Re-training the Pruned Model
  • Exporting the model
  • TensorRT Engine Generation, Validation, and int8 Calibration
  • Deploying to DeepStream
• Image Classification (TF2)
  
  • Preparing the Input Data Structure
  • Creating an Experiment Spec File - Specification File for Classification
  • Training the model
  • Evaluating the Model
  • Running Inference on a Model
  • Pruning the Model
  • Re-training the Pruned Model
  • Exporting the model
  • TensorRT Engine Generation, Validation, and int8 Calibration
  • Deploying to DeepStream
• Object Detection
  
  • DetectNet_v2
  • FasterRCNN
  • YOLOv3
  • YOLOv4
  • YOLOv4-tiny
  • SSD
  • DSSD
  • RetinaNet
  • DeformableDETR
  • EfficientDet (TF1)
  • EfficientDet (TF2)
• Instance Segmentation
  
  • Data Input for Instance Segmentation
  • MaskRCNN
• Semantic Segmentation
  
  • UNET
  • SegFormer
• Gaze Estimation
  
  • Gaze Estimation
• Emotion Classification
  
  • Emotion Classification
• HeartRate Estimation
  
  • Heart Rate Estimation
• Facial Landmarks Estimation
  
  • Facial Landmarks Estimation
• Gesture Recognition
  
  • Gesture Recognition
• Body Pose Estimation
  
  • Body Pose Estimation
• Multitask Image Classification
  
  • Preparing the Input Data Structure
  • Creating an Experiment Spec File - Specification File for Multitask Classification
  • Training the model
  • Evaluating the Model
  • Generating Confusion Matrix
  • Running Inference on a Model
  • Pruning the Model
  • Re-training the Pruned Model
  • Exporting the model
  • TensorRT Engine Generation, Validation, and int8 Calibration
  • Deploying to DeepStream
• Character Recognition
  
  • LPRNet
• ActionRecognitionNet
  
  • Preparing the Dataset
  • Creating an Experiment Spec File
  • Training the Model
  • Evaluating the model
  • Running Inference on the Model
  • Exporting the Model
  • Deploying the Model
• Re-Identification
  
  • ReIdentificationNet
• Pose Classification
  
  • PoseClassificationNet

MLOPS integration

• TAO Toolkit MLOPS Integration
• TAO Toolkit WandB Integration
  
  • Quick Start
  • Configuring the wandb element in the training spec
  • Visualization output
• TAO Toolkit Clearml Integration
  
  • Quick Start
  • Configuring the clearml Element in the Training Spec
  • Visualization Output

Conversational AI Applications

• ASR
  
  • Speech Recognition
  • Speech Recognition With CitriNet
  • Speech Recognition With Conformer
• Natural Language Processing
  
  • Joint Intent and Slot Classification
  • Punctuation and Capitalization
  • Question Answering
  • Text Classification
  • Token Classification (Named Entity Recognition)
• Language Models
  
  • N-Gram Language Model
• Speech Synthesis or Text To Speech
  
  • Overview
  • Spectrogram Generator
  • Vocoder

Point Cloud Applications

• 3D Object Detection
  
  • PointPillars

Deploying to Inference SDKs

• Integrating TAO CV Models with Triton Inference Server
• Integrating Conversational AI Models into Riva
• TAO Converter
  
  • TAO Converter with Classification TF1/TF2
  • TAO Converter with Deformable DETR
  • TAO Converter with Deformable DETR
  • TAO Converter with DSSD
  • TAO Converter with EfficientDet
  • TAO Converter with FasterRCNN
  • TAO Converter with MaskRCNN
  • TAO Converter with Multitask Classification
  • TAO Converter with Retinanet
  • TAO Converter with SSD
  • TAO Converter with UNET
  • TAO Converter with YOLOv3
  • TAO Converter with YOLOv4
  • TAO Converter with YOLOv4-tiny
• Integrating TAO Models into DeepStream
  
  • Installation Prerequisites
  • Deployment Files
  • Sample Application
  • Image classification
  • Multitask Classification
  • Object Detection
  • Instance Segmentation
  • Semantic Segmentation

Deploying with TAO Deploy

• TAO Deploy Overview
  
  • Running TAO Deploy with the Launcher
• TAO Deploy Installation
  
  • Invoking the TAO Deploy Container Directly
  • Installing TAO Deploy through wheel
• Classification (TF1) with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• Classification (TF2) with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• Deformable DETR with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• DetectNet_v2 with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• DSSD with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• EfficientDet (TF1) with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• EfficientDet (TF2) with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• Faster RCNN with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• LPRNet with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• Mask RCNN with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• Multitask Image Classification with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• RetinaNet with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• SSD with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• Segformer with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• UNet with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• YOLOv3 with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• YOLOv4 with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• YOLOv4-tiny with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine

Bring Your Own Model (BYOM)

• BYOM Converter
  
  • Installing the TAO BYOM Converter for TF1 Classification and UNet
  • Installing the TAO BYOM Converter for TF2 Classification
  • Preparing the ONNX Model
  • Running the TAO BYOM Converter
  • Examples of Converting Open-Source Models through TAO BYOM
  • Supported ONNX nodes in TAO BYOM
• BYOM Image Classification
  
  • Preparing the Input Data Structure
  • Creating an Experiment Spec File - Specification File for Classification
  • Training the model
  • Evaluating the Model
  • Running Inference on a Model
  • Pruning the Model
  • Re-training the Pruned Model
  • Exporting the Model
  • Deploying to DeepStream
• BYOM UNET
  
  • Data Input for Semantic Segmentation
  • Creating a Configuration File
  • Training the Model
  • Pruning the Model
  • Re-training the Pruned Model
  • Evaluating the Model
  • Using Inference on the Model
  • Exporting the Model
  • Deploying to Deepstream

More Information

• Release Notes
  
  • Version list
  • TAO Toolkit 4.0.0
  • TAO Toolkit 3.0-22.05
  • TAO Toolkit 3.0-22.02
  • TAO Toolkit 3.0-21.11
  • TAO Toolkit 3.0-21.08
• Frequently Asked Questions
  
  • TLT to TAO Toolkit rename
  • Model support
  • Pruning
  • Model Export and Deployment
  • Training
• Troubleshooting Guide
  
  • NGC
  • TAO Launcher
  • MaskRCNN
  • DetectNet_v2
  • Natural Language Processing
• Support Information
• Acknowledgements
  
  • nitime
  • OpenSSL
  • JsonCpp
  • Python
  • libcurl
  • OpenCV
  • zlib
  • TensorFlow
  • Keras
  • PyTorch
  • ssd_keras
  • Yamale
  • PyCUDA
  • protobuf
  • onnx
  • PIL
  • PyYAML
  • addict
  • argcomplete
  • bto3
  • cryptography
  • docker
  • dockerpty
  • gRPC
  • h5py
  • jupyter
  • numba
  • numpy
  • pandas
  • posix_ipc
  • prettytable
  • arrow
  • PyJWT
  • requests
  • retrying
  • seaborn
  • scikit-image
  • scikit-learn
  • semver
  • Shapely
  • simplejson
  • six
  • python-tabulate
  • toposort
  • tqdm
  • uplink
  • xmltodict
  • recordclass
  • cocoapi
  • mpi4py
  • Open MPI
  • lazy_object_proxy
  • onnxruntime
  • pytorch-lightning
  • KenLM
  • Eigen
  • google/automl
  • open-mmlab/OpenPCDet
  • VainF/Torch-Pruning
  • gmalivenko/onnx2keras
  • open-mmlab/mmskeleton