Automatic Accuracy Tuning
Overview
PipeTuner is an automatic accuracy tuning tool designed to optimize the parameters of any video AI and analytics processing pipeline, including the multi-camera tracking workflows, such as Multi-Target Multi-Camera (MTMC) tracking and Real-Time Location System (RTLS). It efficiently explores the high-dimensional parameter space to find the optimal set of parameters that yield the highest accuracy on a given dataset, without requiring in-depth technical knowledge from the user.
Key Features:
Supports accuracy tuning for the multi-camera tracking workflows, including for Perception (DeepStream) microservice
Users provide a representative dataset and PipeTuner automatically searches for optimal parameters
Supports data augmentation to improve robustness of tuned parameters
Integrates multiple optimization algorithms to efficiently search the parameter space
Provides intuitive configuration files to define the search space for each parameter
PipeTuner works by iteratively executing the perception or multi-camera tracking pipeline with different parameter configurations, evaluating the accuracy metrics (such as MOTA, IDF1, or HOTA) on the provided dataset, and intelligently updating the parameter values using optimization algorithms. This process continues until the desired accuracy is achieved or a specified number of iterations is reached, resulting in an optimized set of parameters tailored to the specific use case and dataset.
Download
PipeTuner is hosted on NGC. Users need to download the following resources to start.
PipeTuner Collection: The collection of all Pipetuner resources, including introduction, user guide and setup instructions;
PipeTuner Container: PipeTuner docker container;
PipeTuner User Guide and Sample Data: PipeTuner user guide and the sample data to run as an example, including a sample dataset for multi-camera tracking, configuration files for tuning, and scripts to launch the pipeline.
Requirements
Here is a summary of PipeTuner’s requirements:
Dataset |
Users provide the typical dataset for their usecase. It should include a few sample videos, with relevant ground truth, such as bounding box, single-camera ID, multi-camera ID, annotated |
Container |
Users provide container(s) of the AI and/or analytics pipeline |
Accuracy KPI |
Users select one of the multi-object tracking KPIs: HOTA, MOTA or IDF1 |
Process
Overall steps are as below:
Download Container: Pull PipeTuner from NGC repository;
Download Sample Data: Download and extract sample data from NGC resource;
Data Preparation: Users create their own dataset with the same format as sample data, and update configuration files to match their usecase;
Launch Tuning: Launch the tuning pipeline using the desired configuration and data;
Retrieve Results: Retrieve the optimal parameters and visualize the tuning results;
Deploy: Deploy the optimal parameters into the desired use case.
PipeTuner searches the optimal parameters by iterating the following three steps until the accuracy KPI converges or up to the max number of iterations (i.e., epochs) specified:
ParamSearch: Given the accuracy KPI score in the previous iteration, make an educated guess on the set of parameters that would yield a higher accuracy KPI. For the very first iteration, a random sampling in the parameter space would be conducted;
PipeExec: Given the sampled/guessed parameter set, execute the pipeline with the params and generates metadata to allow accuracy evaluation;
PipeEval: Given the metadata outputs from the pipeline and the dataset, perform the accuracy evaluation based on the accuracy metric and generates accuracy KPI score.