NVIDIA TAO Toolkit v5.5.0

Introduction

• Overview
  
  • Pretrained models
  • Key Features
  • How to Get Started
  • TAO Architecture
  • Model Pruning
  • Learning Resources
  • Support Information
• TAO Launcher
  
  • Running the launcher
  • Handling launched processes
  • Useful Environment variables
• Migration Guides
  
  • Migrating to TAO 5.5
  • Migrating from TAO 4.0.x to TAO 5.0.0
  • Migrating from TAO 3.x to TAO 4.0
  • Migrating from Legacy TLT to TAO
  • Migrating to the TAO API from TAO 5.3.0
• Working With the Containers
  
  • Invoking the Containers Directly
  • Running Multi-Node Training
  • Running Training on a Multi-Instance GPU
  • Running without Elevated User Privileges

Getting Started

• Getting Started
• Beginners
• Intermediate Users
• Advanced Users

Model Zoo

• Overview

Running TAO in the Cloud

• Running TAO in the Cloud
• Running TAO on an AWS VM
  
  • Pre-Requisites
  • Setting up an AWS EC2 instance
  • Installing the Pre-Requisites for TAO in the VM
  • Download and run the test samples
• Running TAO on Google Cloud Platform
  
  • Setting up a VM Linux VM Instance
  • Using the VM
  • Setting up the VM and Enabling GPUs
  • Installing the Pre-requisites for TAO
  • Downloading and Running Test Samples
• Running TAO on an Azure VM
  
  • Setting up an Azure VM
  • Installing the Pre-Requisites for TAO in the VM
  • Downloading and Running the Test Samples
• Running TAO on Google Colab
  
  • Pre-Requisites
  • Launching Notebooks with Google Colab
  • Utility scripts to obtain subset of data
  • Steps to Locate Files in a Colab Notebook
  • Notes
• Running TAO on an EKS
• Running TAO on an AKS

TAO API

• Overview
• Setup
  
  • Bare-Metal Setup
  • AWS EKS Setup
  • Azure AKS
  • GCP GKS
• Deployment
• Train, Adapt, Optimize (TAO) REST API
  
  • Requirements
  • User Authentication
  • API Specs
  • Examples
• Remote Client
  
  • Installation
  • Storage Topology
  • CLI Specs
  • Examples
• API Reference

AutoML

• AutoML
  
  • Prerequisites
  • AutoML Notebooks
  • Getting Started
  • AutoML Outcomes
  • AutoML Algorithm Explanation
  • Hyperband Parameter Auto-adjustment Mechanism

CV Fine-tuning

• Optimizing the Training Pipeline
  
  • Quantization Aware Training
  • Automatic Mixed Precision
  • Knowledge Distillation
• Visualizing Training
  
  • Enabling Tensorboard during Training
  • Visualizing using TensorBoard
  • Additional Resources
• Data Annotation Format
  
  • Image Classification Format
  • Optical Inspection Format
  • Change Detection (Segmentation) Format
  • CenterPose Format
  • Image Classification Format PyTorch
  • Object Detection – KITTI Format
  • Object Detection – COCO Format
  • Open Vocabulary Object Detection/Segmentation – ODVG Format
  • Instance Segmentation – COCO format
  • Semantic Segmentation - PNG Mask Format
  • Gesture Recognition – Custom Format
  • Heart Rate Estimation – Custom Format
  • EmotionNet, FPENET, GazeNet – JSON Label Data Format
  • BodyposeNet – COCO Format
  • Re-Identification – Market-1501 Format
• Computer Vision Finetuning
  
  • PyTorch
  • TensorFlow 2.x
  • TensorFlow 1.x [Deprecated]
• Foundation Models
  
  • Object Detection with Foundational Model
  • Change Detection with Foundational Model

MLOPS integration

• TAO MLOPS Integration
• TAO WandB Integration
  
  • Quick Start
  • Configuring the wandb element in the training spec
  • Visualization output
• TAO Clearml Integration
  
  • Quick Start
  • Configuring the clearml Element in the Training Spec
  • Visualization Output

Deploying to Inference SDKs

• Integrating TAO CV Models with Triton Inference Server
• TAO Converter
  
  • TAO Converter with Classification TF1/TF2
  • TAO Converter with Deformable DETR
  • TAO Converter with Detectnet_v2
  • TAO Converter with DSSD
  • TAO Converter with EfficientDet
  • TAO Converter with FasterRCNN
  • TAO Converter with MaskRCNN
  • TAO Converter with Multitask Classification
  • TAO Converter with Retinanet
  • TAO Converter with SSD
  • TAO Converter with UNET
  • TAO Converter with YOLOv3
  • TAO Converter with YOLOv4
  • TAO Converter with YOLOv4-tiny
• Optimizing and Profiling with TensorRT
  
  • TRTEXEC with ActionRecognitionNet
  • TRTEXEC with BodyPoseNet
  • TRTEXEC with CenterPose
  • TRTEXEC with Classification TF1/TF2/PyT
  • TRTEXEC with Deformable-DETR
  • TRTEXEC with DetectNet-v2
  • TRTEXEC with DINO
  • TRTEXEC with DSSD
  • TRTEXEC with EfficientDet TF1/TF2
  • TRTEXEC with Facial Landmarks Estimation
  • TRTEXEC with Faster RCNN
  • TRTEXEC with Grounding DINO
  • TRTEXEC with LPRNet
  • TRTEXEC with Metric Learning Recognition
  • TRTEXEC with Mask RCNN
  • TRTEXEC with Multitask Classification
  • TRTEXEC with OCDNet
  • TRTEXEC with OCRNet
  • TRTEXEC with PointPillars
  • TRTEXEC with PoseClassificationNet
  • TRTEXEC with ReIdentificationNet
  • TRTEXEC with ReIdentificationNet Transformer
  • TRTEXEC with RetinaNet
  • TRTEXEC with Segformer
  • TRTEXEC with SiameseOI
  • TRTEXEC with SSD
  • TRTEXEC with UNet
  • TRTEXEC with YOLO_v3
  • TRTEXEC with YOLO_v4
  • TRTEXEC with YOLO_v4_tiny
  • TRTEXEC with VisualChangeNet
  • TRTEXEC with Mask2former
  • TRTEXEC with Mask Grounding DINO
• Integrating TAO Models into DeepStream
  
  • Installation Prerequisites
  • Deployment Files
  • Sample Application
  • Image classification
  • Multitask Classification
  • Object Detection
  • Instance Segmentation
  • Semantic Segmentation
  • Character Recognition Using nvOCDR

Deploying with TAO Deploy

• TAO Deploy Overview
  
  • Running TAO Deploy with the Launcher
• TAO Deploy Installation
  
  • Invoking the TAO Deploy Container Directly
  • Installing TAO Deploy through wheel
• CenterPose with TAO Deploy
  
  • Converting an ONNX File into TensorRT Engine
  • Running Evaluation Through a TensorRT Engine
  • Running Inference Through a TensorRT Engine
• Classification (PyTorch) with TAO Deploy
  
  • Converting .onnx File into TensorRT Engine
  • Running Evaluation through a TensorRT Engine
  • Running Inference through a TensorRT Engine
• Classification (TF1) with TAO Deploy
  
  • Converting .onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• Classification (TF2) with TAO Deploy
  
  • Converting ONNX File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• Deformable DETR with TAO Deploy
  
  • Converting .onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• DINO with TAO Deploy
  
  • Converting .onnx File into TensorRT Engine
  • Running Evaluation through a TensorRT Engine
  • Running Inference through a TensorRT Engine
• Grounding DINO with TAO Deploy
  
  • Converting ONNX File into TensorRT Engine
  • Running Evaluation through a TensorRT Engine
  • Running Inference through a TensorRT Engine
• DetectNet_v2 with TAO Deploy
  
  • Converting .onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• DSSD with TAO Deploy
  
  • Converting an .onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• EfficientDet (TF1) with TAO Deploy
  
  • Converting .onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• EfficientDet (TF2) with TAO Deploy
  
  • Converting ONNX File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• Faster RCNN with TAO Deploy
  
  • Converting .onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• LPRNet with TAO Deploy
  
  • Converting .onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• Mask RCNN with TAO Deploy
  
  • Converting .uff File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• Mask2former with TAO Deploy
  
  • Converting .onnx File into TensorRT Engine
  • Running Evaluation through a TensorRT Engine
  • Running Inference through a TensorRT Engine
• MLRecogNet with TAO Deploy
  
  • Converting ONNX File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• Multitask Image Classification with TAO Deploy
  
  • Converting .etlt File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• OCDNet with TAO Deploy
  
  • Converting onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• RetinaNet with TAO Deploy
  
  • Converting an .onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• SSD with TAO Deploy
  
  • Converting .onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• Segformer with TAO Deploy
  
  • Converting .onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• UNet with TAO Deploy
  
  • Converting .onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• YOLOv3 with TAO Deploy
  
  • Converting .onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• YOLOv4 with TAO Deploy
  
  • Converting an .onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• YOLOv4-tiny with TAO Deploy
  
  • Converting .onnx File into TensorRT Engine
  • Running Evaluation through TensorRT Engine
  • Running Inference through TensorRT Engine
• OCRNet with TAO Deploy
  
  • Converting an .etlt or .onnx File into TensorRT Engine
  • Running Evaluation through the TensorRT Engine
  • Running Inference through TensorRT Engine
• SiameseOI with TAO Deploy
  
  • Converting an ETLT or ONNX File into TensorRT Engine
  • Running Inference through TensorRT Engine
• VisualChangeNet-Classification with TAO Deploy
  
  • Converting an ONNX File into TensorRT Engine
  • Running Inference through TensorRT Engine
  • Running Evaluation through a TensorRT Engine
• VisualChangeNet-Segmentation with TAO Deploy
  
  • Converting an ONNX File into TensorRT Engine
  • Running Inference through TensorRT Engine
  • Running Evaluation through a TensorRT Engine
• Mask Grounding DINO with TAO Deploy
  
  • Converting ONNX File into TensorRT Engine
  • Running Evaluation through a TensorRT Engine
  • Running Inference through a TensorRT Engine

Bring Your Own Model (BYOM)

• Bring Your Own Model (BYOM)
  
  • BYOM Converter
  • BYOM Image Classification

Data Services

• Annotations
  
  • Supported Data Formats
  • Configuring a Spec File for Annotation Conversion
  • Running the Annotation Conversion
  • Configuring a Spec File for the Annotation Slicing Service
  • Running the Annotation Slicer
  • Configuring a Spec file for Annotation Merge
  • Running the Annotation Merge
• Offline Data Augmentation
  
  • Data Input for Augmentation
  • Configuring a Spec File for Augmentation
  • Running the augmentation service
• Auto-Label
  
  • Data Input for Auto-Label
  • Configuring Spec File for Auto-Label
  • Running the Auto-Label Tool
• Data Analytics
  
  • Data input for Data Analytics
  • Creating Experiment Spec File for the Analyze Task
  • Creating an Experiment Spec File for the Validate Task
  • Running Analyze task on the Data
  • Running Validate task on the Data
  • Creating an Experiment Spec File for the KPI Analyze Task
  • Running the KPI Analyze Task on the Data

More Information

• Release Notes
  
  • Version list
  • TAO 5.5.0
  • TAO 5.3.0
  • TAO 5.2.0
  • TAO 5.1.0
  • TAO 5.0.0
  • TAO 4.0.2
  • TAO 4.0.1
  • TAO 4.0.0
  • TAO 3.0-22.05
  • TAO 3.0-22.02
  • TAO 3.0-21.11
  • TAO 3.0-21.08
• Frequently Asked Questions
  
  • TLT to TAO rename
  • Model support
  • Pruning
  • Model Export and Deployment
  • Training
• Troubleshooting Guide
  
  • NGC
  • TAO Launcher
  • MaskRCNN
  • DetectNet_v2
• Support Information
• Acknowledgements
  
  • nitime
  • OpenSSL
  • JsonCpp
  • Python
  • libcurl
  • OpenCV
  • zlib
  • TensorFlow
  • Keras
  • PyTorch
  • ssd_keras
  • Yamale
  • PyCUDA
  • protobuf
  • onnx
  • PIL
  • PyYAML
  • addict
  • argcomplete
  • bto3
  • cryptography
  • docker
  • dockerpty
  • gRPC
  • h5py
  • jupyter
  • numba
  • numpy
  • pandas
  • posix_ipc
  • prettytable
  • arrow
  • PyJWT
  • requests
  • retrying
  • seaborn
  • scikit-image
  • scikit-learn
  • semver
  • Shapely
  • simplejson
  • six
  • python-tabulate
  • toposort
  • tqdm
  • uplink
  • xmltodict
  • recordclass
  • cocoapi
  • mpi4py
  • Open MPI
  • lazy_object_proxy
  • onnxruntime
  • pytorch-lightning
  • KenLM
  • Eigen
  • google/automl
  • open-mmlab/OpenPCDet
  • VainF/Torch-Pruning
  • gmalivenko/onnx2keras
  • open-mmlab/mmskeleton