Changelog

2.1 (DP) - 2024-06-24

Changed

[Perception]

  • Update the ReIdentificationNet_Transformer model to enhance association accuracy in multi-camera tracking

[Multi-Camera Fusion]

  • Refactor original scripts and utilities for multi-camera tracking to support MTMC mode

  • Change the original visualization of ground truth into visualization of ground-truth bounding boxes

  • Enhance the consumption of Kafka messages to leverage multiprocessing for improved efficiency

  • Change the protobuf input mechanism from a byte file to a text file format

  • Fine-tune the configuration settings for both MTMC and RTLS modes to optimize accuracy and performance

  • Update sequential metadata generation guide to support SV3DT logging

[Web API]

  • Enhance OpenAPI specification

  • Enhance calibration schema to support camera matrices and translation to global coordinates

  • Update Elasticsearch mapping of calibration to support large text values (Polygon Field Of View) and camera matrices

  • Update schema of MTMC config to support dynamic updates of newly added config params

[Web UI]

  • Upgrade dependencies

  • Replace text-only event cards with ones including thumbnails in Occupancy Analytics App

[Deployment]

  • Update directory structure of videos in sample input data tar to use the nv_streamer_sync_playback feature

  • Refactor docker compose by moving nvstreamer from foundational to app specific yaml thereby supporting the nv_streamer_sync_playback feature and new sample input data’s directory structure

  • Update Kafka topic creation script to make it configurable in docker compose

  • Enhance MTMC reference workflow to run both MTMC and RTLS microservices when enough resources are available in the system

Added

[Reference Workflows]

  • Introduce Multi-Camera Sim2Deploy workflow for end-to-end development using simulation and synthetic data

  • Add cloud quickstart option for Sim2Deploy, currently supporting AWS

  • Introduce Real-Time Location System (RTLS) reference workflow

  • Add architecture diagrams for the playback mode of MTMC, RTLS, & Occupancy Analytics reference workflows

[Perception]

  • Integrate PipeTuner support for automatic accuracy tuning of DeepStream pipeline parameters

[Multi-Camera Fusion]

  • Introduce RTLS (Real-Time Location System) mode to enable low-latency, continuous updates of global object locations across multiple cameras

  • Add main scripts to support both batch processing and stream processing in RTLS mode

  • Add configurations specific to RTLS mode

  • Develop a visualization notebook to showcase RTLS results

  • Create an evaluation notebook for analyzing RTLS results

  • Add configs and calibration for a sample synthetic dataset for RTLS testing

  • Add object type in the schema for MTMC objects

  • Enable data-driven end-to-end parameter tuning with PipeTuner

  • Enhance calibration with support for camera matrices and translation to global coordinates

  • Integrate a direction factor in the calculation of spatio-temporal distance

  • Provide visualization of ground-truth locations in MTMC mode

  • Incorporate debug information outputs for MTMC stream processing

[Web API]

  • Create a new endpoint /tracker/unique-object-count-with-locations to support RTLS mode & workflow

[Web UI]

  • Add Real-Time Location System workflow with documentation

  • Include match score in Query-by-Example Behavior Event cards in Multi-Camera Tracking workflow

  • Add Polygon Field-of-View Drawing alongside existing Field-of-View Arc for Camera Icons in Multi-Camera Tracking workflow

  • Add user input based option to show/hide camera icons in Multi-Camera Tracking workflow

[Data]

  • Add a new documentation page that outlines techniques for annotating ground truth data in MTMC and ReID finetuning workflows

  • Provide scripts to annotate ReID data based on semi-automatic labelling via behavior-based sampling

  • Add sample warehouse data which is used as part of the Multi-Camera Sim2Deploy workflow

Removed

[Multi-Camera Fusion]

  • Remove subsets within co-existing behavior groups to expedite the clustering process

Fixed

[Multi-Camera Fusion]

  • Address edge cases where anomalies were not identified in the notebook designed for anomaly inspection

  • Correct the mechanism for reading foot locations from DeepStream perception when SV3DT is enabled

  • Fix the issue due to integer input for batch ID in workflow configuration

  • Fix the evaluation of HOTA scores based on 3D distance metrics

[Web UI]

  • Resolve bug causing UI crash when switching from ‘Minutes’ to ‘Hours’ in Live Events in Multi Camera Tracking App

[Calibration Toolkit]

  • Resolve bug when importing project from sample assets

  • Correct mechanism when refreshing page would lead to blank page in hosted environments

  • Corrected imported calibration did not load properly on calibration page

  • Resolve bug where Cartesian calibration would display off-screen

  • Resolve bug when floorplan did not show in MTMC Calibration

  • Fix issue with sensor images not displaying properly in UI

  • Update calibration.json and imageMetadata.json to match expected outputs


2.0 (DP) - 2023-12-12

First Developer Preview (DP) version

Changed

[Perception]

  • Upgrade perception container operating system to Ubuntu 22.04 from 20.04

[Multi-Camera Fusion]

  • Breaking: group workflow configuration parameters by categories: “io”, “preprocessing”, “localization”, “clustering”, and “streaming”

  • Breaking: group visualization configuration parameters by categories: “setup”, “io”, and “plotting”

[Web UI]

  • Enhance trajectory visualization in Occupancy Analytics & Multi-Camera Tracking (MTMC) apps

[Calibration Toolkit]

  • Breaking: Change deploy command to accommodate Calibration as a Service

  • Restrict sensor discovery to VST sources only

[Deployment]

  • Breaking: update directory structure for config and script files in Docker Compose deployments for improved readability

  • Update Azure & AWS cloud-deployment scripts for Ubuntu 22.04 and K8s 1.27.xx support

Added

[Perception]

  • Add PeopleNet_Transformer model as a detector model

  • Add ReIdentificationNet_Transformer as a tracker ReID model

  • Add Single-View 3D Tracking (SV3DT) for improved tracking accuracy, visibility, foot point estimation, and object shape modeling

  • Add support for using camera_name as sensorId in messages from the Perception microservice over Kafka

[Multi-Camera Fusion]

  • Add support for SV3DT attributes from Perception to enhance foot point localization

  • Add option for bounding box rectification by calibration for better people height estimation

  • Add hierarchical clustering option using HDBSCAN along with Agglomerative Clustering

  • Add spatio-temporal matching using directed Hausdorff distance in addition to pairwise distance

  • Add accuracy evaluation using 3D distance to ground-truth locations on the ground plane, in addition to IoU with ground-truth bounding boxes

  • Add accuracy evaluation based on projected full-body bounding boxes

  • Add plotting of foot locations in visualization mode

  • Add multi-processing and memory sharing for accelerated behavior processing

[Web API]

  • Add support for dynamic config updates in Analytics & Tracker microservices

[Web UI]

  • Add Query-by-Example for recorded videos in MTMC app

  • Introduce seek functionality for WebRTC recorded videos

  • Implement continuous frame reception validation in WebRTC streaming with error display for interruptions

[Calibration Toolkit]

  • Add import of calibration.json, imageMetadata.json, and Images.zip from project exports for Cartesian, Multi-Camera Tracking, and Image Calibration modes

[Deployment]

  • Include edge-to-cloud connectivity for streaming & inferencing via WebRTC and OpenVPN

  • Add dynamic stream addition & removal in Docker Compose deployments

  • Add live K8s model updates in the Perception microservice

  • Expose Jupyter Notebook and Camera Calibration as K8s Services via Ingress

Removed

[Deployment]

  • Support for docker-compose (older version) has been removed. It is recommended to use docker compose

Fixed

[Multi-Camera Fusion]

  • Handle cases with no cluster creation due to empty behavior lists

  • Address empty embedding and confidence issues in raw metadata JSON format

  • Correct attribute types in calibration JSON schema

  • Fix SDK document creation issues

[Web UI]

  • Fix memory leak causing unresponsiveness in Occupancy Analytics workflow during Sensor Live mode

  • Handle sensor IDs array as REST API query parameters for certain endpoints

[Calibration Toolkit]

  • Fix Calibration Export errors

  • Resolve minor bugs


1.1 (EA) - 2023-09-13

Changed

  • Breaking: Use TAO Re-ID model both for the single-camera tracker and to generate embedding, in the DeepStream pipeline, removing the need for a separate SGIE

  • Breaking: Replace the [mars_tracker | default_tracker] options with [e2e | playback] for the --profile argument of the Docker Compose up commands

  • Breaking: Changes to Analytics API endpoints

    • /kpi was changed to /metrics

    • /occupancy/reset was included as part of /metrics

    • Calibration related endpoints have been renamed

    • /health-check has changed to /livez

  • Upgrade to DeepStream 6.3 for the Perception microservice

  • Use docker compose by default for reference workflow Docker Compose deployment, switching from & deprecating docker-compose

  • Update reference workflow notebooks to use the new API endpoints

  • Patch various container vulnerabilities across microservices

Added

  • Support ROI-based filtering of object instances in MTMC processing

  • Support ROI-based occupancy counting in Web API endpoints

  • Support remote deployment of Camera Calibration Toolkit

  • Support H.265 IP cameras

  • Enable automatic fallback on software encoding in VST

  • Configure automatically added or removed camera streams in Perception (in K8s mode)

  • Facilitate quick K8s deployment to popular public clouds

  • Support a “playback” Docker Compose profile that replays saved perception metadata as inputs for downstream microservices

  • Include HOTA evaluation metrics

  • Include visualization of MTMC ground truth

  • Use shared memory during multiprocessing to efficiently pre-process the data and reduce the time taken per micro-batch in MTMC stream processing

  • Enable direct processing of raw protobuf objects and creation of behaviors, further reducing pre-processing time in MTMC stream processing

  • Document (sequential) perception metadata generation for MTMC batch processing

  • Document best practices for calibration drawing

Removed

  • Remove the mars_tracker & default_tracker Docker Compose launch profiles (added in v1.0.1)

Fixed

  • Fix various bugs in the Camera Calibration Toolkit

  • Remove the limit of 100 behaviors per global ID in the GET /tracker/behavior-locations API endpoint


1.0.1 (EA) - 2023-06-29

Changed

  • Breaking: Need to specify the Docker Compose launch command with --profile [mars_tracker | default_tracker] argument

  • Upgrade to DeepStream 6.2 for the Perception microservice

  • Retrain the Re-Id model in the Perception microservice with more data

  • Improve the clustering & re-assignment algorithms in MTMC Analytics library to be more accurate & have fewer parameters

  • Optimize micro-batch Kafka consumer implementation, reducing the micro-batch processing time in MTMC stream processing

Added

  • Add a Docker Compose profile that doesn’t use any Re-ID model in the DeepStream tracker within the Perception microservice

  • Support sampling of locations to limit the memory usage

  • Document MTMC workflow parameter tuning

  • Document quantitative evaluation of end-to-end MTMC batch processing on custom data

Fixed

  • Stop the Multi-Camera Tracking microservice from failing with undefined tripwire & ROI during calibration

  • Reverse tracker width & height in Docker Compose DeepStream pipeline

  • Fix GPU memory leakage in re-ranking


1.0 (EA) - 2023-03-24

Initial Early Access (EA) release

Added

[Reference Workflows]

  • Introduce Multi-Camera Tracking workflow for tracking people across multiple cameras and providing unique person counts over time

  • Introduce Occupancy Analytics workflow for analyzing people movements, inter-person distances, and interactions with predefined feature maps

  • Introduce Retail Loss Prevention workflow with few-shot learning for self-checkout use cases

  • Introduce Occupancy Heatmap workflow for visualizing space occupancy and movement patterns

[Perception]

  • Implement unique object and people tracking across multiple cameras

  • Add pre-built analytics for object speed, distance, line-crossing, and region of interest (ROI)

  • Introduce pretrained models for person Re-ID, retail object detection, and retail object recognition

  • Enable model fine-tuning on custom datasets using NVIDIA TAO Toolkit

[Web API]

  • Introduce REST APIs for querying and extracting metadata

[Web UI]

  • Implement support for extracting live and recorded media data from VMS for UI playback

  • Enable visualization of raw and filtered data in Kibana

[Calibration Toolkit]

  • Introduce browser-based toolkit for camera calibration and ROI creation

[Deployment]

  • Implement fully modularized microservice architecture deployable via Kubernetes and Helm

  • Provide Helm charts for foundational systems, reference workflows, and core services

  • Include Docker images for all microservices and components

[Data]

  • Provide sample datasets for demonstrating reference workflows (subject to EULA terms)