Changelog
2.1 (DP) - 2024-06-24
Changed
[Perception]
Update the ReIdentificationNet_Transformer model to enhance association accuracy in multi-camera tracking
[Multi-Camera Fusion]
Refactor original scripts and utilities for multi-camera tracking to support MTMC mode
Change the original visualization of ground truth into visualization of ground-truth bounding boxes
Enhance the consumption of Kafka messages to leverage multiprocessing for improved efficiency
Change the protobuf input mechanism from a byte file to a text file format
Fine-tune the configuration settings for both MTMC and RTLS modes to optimize accuracy and performance
Update sequential metadata generation guide to support SV3DT logging
[Web API]
Enhance OpenAPI specification
Enhance calibration schema to support camera matrices and translation to global coordinates
Update Elasticsearch mapping of calibration to support large text values (Polygon Field Of View) and camera matrices
Update schema of MTMC config to support dynamic updates of newly added config params
[Web UI]
Upgrade dependencies
Replace text-only event cards with ones including thumbnails in Occupancy Analytics App
[Deployment]
Update directory structure of videos in sample input data tar to use the
nv_streamer_sync_playback
featureRefactor docker compose by moving nvstreamer from foundational to app specific yaml thereby supporting the
nv_streamer_sync_playback
feature and new sample input data’s directory structureUpdate Kafka topic creation script to make it configurable in docker compose
Enhance MTMC reference workflow to run both MTMC and RTLS microservices when enough resources are available in the system
Added
[Reference Workflows]
Introduce Multi-Camera Sim2Deploy workflow for end-to-end development using simulation and synthetic data
Add cloud quickstart option for Sim2Deploy, currently supporting AWS & Azure
Introduce Real-Time Location System (RTLS) reference workflow
Add architecture diagrams for the playback mode of MTMC, RTLS, & Occupancy Analytics reference workflows
[Perception]
Integrate PipeTuner support for automatic accuracy tuning of DeepStream pipeline parameters
[Multi-Camera Fusion]
Introduce RTLS (Real-Time Location System) mode to enable low-latency, continuous updates of global object locations across multiple cameras
Add main scripts to support both batch processing and stream processing in RTLS mode
Add configurations specific to RTLS mode
Develop a visualization notebook to showcase RTLS results
Create an evaluation notebook for analyzing RTLS results
Add configs and calibration for a sample synthetic dataset for RTLS testing
Add object type in the schema for MTMC objects
Enable data-driven end-to-end parameter tuning with PipeTuner
Enhance calibration with support for camera matrices and translation to global coordinates
Integrate a direction factor in the calculation of spatio-temporal distance
Provide visualization of ground-truth locations in MTMC mode
Incorporate debug information outputs for MTMC stream processing
[Web API]
Create a new endpoint
/tracker/unique-object-count-with-locations
to support RTLS mode & workflow
[Web UI]
Add Real-Time Location System workflow with documentation
Include match score in Query-by-Example Behavior Event cards in Multi-Camera Tracking workflow
Add Polygon Field-of-View Drawing alongside existing Field-of-View Arc for Camera Icons in Multi-Camera Tracking workflow
Add user input based option to show/hide camera icons in Multi-Camera Tracking workflow
[Data]
Add a new documentation page that outlines techniques for annotating ground truth data in MTMC and ReID finetuning workflows
Provide scripts to annotate ReID data based on semi-automatic labelling via behavior-based sampling
Add sample warehouse data which is used as part of the Multi-Camera Sim2Deploy workflow
Removed
[Multi-Camera Fusion]
Remove subsets within co-existing behavior groups to expedite the clustering process
Fixed
[Multi-Camera Fusion]
Address edge cases where anomalies were not identified in the notebook designed for anomaly inspection
Correct the mechanism for reading foot locations from DeepStream perception when SV3DT is enabled
Fix the issue due to integer input for batch ID in workflow configuration
Fix the evaluation of HOTA scores based on 3D distance metrics
[Web UI]
Resolve bug causing UI crash when switching from ‘Minutes’ to ‘Hours’ in Live Events in Multi Camera Tracking App
[Calibration Toolkit]
Resolve bug when importing project from sample assets
Correct mechanism when refreshing page would lead to blank page in hosted environments
Corrected imported calibration did not load properly on calibration page
Resolve bug where Cartesian calibration would display off-screen
Resolve bug when floorplan did not show in MTMC Calibration
Fix issue with sensor images not displaying properly in UI
Update calibration.json and imageMetadata.json to match expected outputs
2.0 (DP) - 2023-12-12
First Developer Preview (DP) version
Changed
[Perception]
Upgrade perception container operating system to Ubuntu 22.04 from 20.04
[Multi-Camera Fusion]
Breaking: group workflow configuration parameters by categories:
“io”
,“preprocessing”
,“localization”
,“clustering”
, and“streaming”
Breaking: group visualization configuration parameters by categories:
“setup”
,“io”
, and“plotting”
[Web UI]
Enhance trajectory visualization in Occupancy Analytics & Multi-Camera Tracking (MTMC) apps
[Calibration Toolkit]
Breaking: Change deploy command to accommodate Calibration as a Service
Restrict sensor discovery to VST sources only
[Deployment]
Breaking: update directory structure for config and script files in Docker Compose deployments for improved readability
Update Azure & AWS cloud-deployment scripts for Ubuntu 22.04 and K8s 1.27.xx support
Added
[Perception]
Add PeopleNet_Transformer model as a detector model
Add ReIdentificationNet_Transformer as a tracker ReID model
Add Single-View 3D Tracking (SV3DT) for improved tracking accuracy, visibility, foot point estimation, and object shape modeling
Add support for using
camera_name
assensorId
in messages from the Perception microservice over Kafka
[Multi-Camera Fusion]
Add support for SV3DT attributes from Perception to enhance foot point localization
Add option for bounding box rectification by calibration for better people height estimation
Add hierarchical clustering option using HDBSCAN along with Agglomerative Clustering
Add spatio-temporal matching using directed Hausdorff distance in addition to pairwise distance
Add accuracy evaluation using 3D distance to ground-truth locations on the ground plane, in addition to IoU with ground-truth bounding boxes
Add accuracy evaluation based on projected full-body bounding boxes
Add plotting of foot locations in visualization mode
Add multi-processing and memory sharing for accelerated behavior processing
[Web API]
Add support for dynamic config updates in Analytics & Tracker microservices
[Web UI]
Add Query-by-Example for recorded videos in MTMC app
Introduce seek functionality for WebRTC recorded videos
Implement continuous frame reception validation in WebRTC streaming with error display for interruptions
[Calibration Toolkit]
Add import of
calibration.json
,imageMetadata.json
, andImages.zip
from project exports for Cartesian, Multi-Camera Tracking, and Image Calibration modes
[Deployment]
Include edge-to-cloud connectivity for streaming & inferencing via WebRTC and OpenVPN
Add dynamic stream addition & removal in Docker Compose deployments
Add live K8s model updates in the Perception microservice
Expose Jupyter Notebook and Camera Calibration as K8s Services via Ingress
Removed
[Deployment]
Support for docker-compose (older version) has been removed. It is recommended to use
docker compose
Fixed
[Multi-Camera Fusion]
Handle cases with no cluster creation due to empty behavior lists
Address empty embedding and confidence issues in raw metadata JSON format
Correct attribute types in calibration JSON schema
Fix SDK document creation issues
[Web UI]
Fix memory leak causing unresponsiveness in Occupancy Analytics workflow during Sensor Live mode
Handle sensor IDs array as REST API query parameters for certain endpoints
[Calibration Toolkit]
Fix Calibration Export errors
Resolve minor bugs
1.1 (EA) - 2023-09-13
Changed
Breaking: Use TAO Re-ID model both for the single-camera tracker and to generate embedding, in the DeepStream pipeline, removing the need for a separate SGIE
Breaking: Replace the
[mars_tracker | default_tracker]
options with[e2e | playback]
for the--profile
argument of the Docker Composeup
commandsBreaking: Changes to Analytics API endpoints
/kpi
was changed to/metrics
/occupancy/reset
was included as part of/metrics
Calibration related endpoints have been renamed
/health-check
has changed to/livez
Upgrade to DeepStream 6.3 for the Perception microservice
Use
docker compose
by default for reference workflow Docker Compose deployment, switching from & deprecatingdocker-compose
Update reference workflow notebooks to use the new API endpoints
Patch various container vulnerabilities across microservices
Added
Support ROI-based filtering of object instances in MTMC processing
Support ROI-based occupancy counting in Web API endpoints
Support remote deployment of Camera Calibration Toolkit
Support H.265 IP cameras
Enable automatic fallback on software encoding in VST
Configure automatically added or removed camera streams in Perception (in K8s mode)
Facilitate quick K8s deployment to popular public clouds
Support a “playback” Docker Compose profile that replays saved perception metadata as inputs for downstream microservices
Include HOTA evaluation metrics
Include visualization of MTMC ground truth
Use shared memory during multiprocessing to efficiently pre-process the data and reduce the time taken per micro-batch in MTMC stream processing
Enable direct processing of raw protobuf objects and creation of behaviors, further reducing pre-processing time in MTMC stream processing
Document (sequential) perception metadata generation for MTMC batch processing
Document best practices for calibration drawing
Removed
Remove the
mars_tracker
&default_tracker
Docker Compose launch profiles (added in v1.0.1)
Fixed
Fix various bugs in the Camera Calibration Toolkit
Remove the limit of 100 behaviors per global ID in the GET /tracker/behavior-locations API endpoint
1.0.1 (EA) - 2023-06-29
Changed
Breaking: Need to specify the Docker Compose launch command with
--profile [mars_tracker | default_tracker]
argumentUpgrade to DeepStream 6.2 for the Perception microservice
Retrain the Re-Id model in the Perception microservice with more data
Improve the clustering & re-assignment algorithms in MTMC Analytics library to be more accurate & have fewer parameters
Optimize micro-batch Kafka consumer implementation, reducing the micro-batch processing time in MTMC stream processing
Added
Add a Docker Compose profile that doesn’t use any Re-ID model in the DeepStream tracker within the Perception microservice
Support sampling of locations to limit the memory usage
Document MTMC workflow parameter tuning
Document quantitative evaluation of end-to-end MTMC batch processing on custom data
Fixed
Stop the Multi-Camera Tracking microservice from failing with undefined tripwire & ROI during calibration
Reverse tracker width & height in Docker Compose DeepStream pipeline
Fix GPU memory leakage in re-ranking
1.0 (EA) - 2023-03-24
Initial Early Access (EA) release
Added
[Reference Workflows]
Introduce Multi-Camera Tracking workflow for tracking people across multiple cameras and providing unique person counts over time
Introduce Occupancy Analytics workflow for analyzing people movements, inter-person distances, and interactions with predefined feature maps
Introduce Retail Loss Prevention workflow with few-shot learning for self-checkout use cases
Introduce Occupancy Heatmap workflow for visualizing space occupancy and movement patterns
[Perception]
Implement unique object and people tracking across multiple cameras
Add pre-built analytics for object speed, distance, line-crossing, and region of interest (ROI)
Introduce pretrained models for person Re-ID, retail object detection, and retail object recognition
Enable model fine-tuning on custom datasets using NVIDIA TAO Toolkit
[Web API]
Introduce REST APIs for querying and extracting metadata
[Web UI]
Implement support for extracting live and recorded media data from VMS for UI playback
Enable visualization of raw and filtered data in Kibana
[Calibration Toolkit]
Introduce browser-based toolkit for camera calibration and ROI creation
[Deployment]
Implement fully modularized microservice architecture deployable via Kubernetes and Helm
Provide Helm charts for foundational systems, reference workflows, and core services
Include Docker images for all microservices and components
[Data]
Provide sample datasets for demonstrating reference workflows (subject to EULA terms)