Analytics and Tracking API
The Web API microservice is developed using Node.js and the Web Application Framework used is Express.js. It consists of various REST API endpoints.
The API is used to fetch tripwire events, compute KPIs, fetch tracked objects across cameras, get behavior clusters and find the time when the latest data was sent.
It also provides the functionality to insert and fetch configurations.
Overview
The architecture of the API can be explained using the following block diagram.
The data sent by sensor processing module is further processed in streaming pipelines to create behavior, track objects across cameras and detect event/anomalies. The processed data is inserted into Elasticsearch. Analytics & Tracking API comprises of REST API endpoints which queries Elasticsearch/Milvus to obtain relevant data based on the input of client requests. To obtain real time data, certain endpoints use Kafka as their data source. Kafka is also used to handle notification related messages.
The response is aggregated (if required) and formatted appropriately before responding to clients.
Note: Both Kafka and Milvus are optional. If Kafka or Milvus is not present in the server config and a request is made to an API endpoint which uses them, then appropriate error message will be returned as part of the response.
One can create a client with the programming language of their choice to connect with the HTTP endpoints. Python client examples are provided in Jupyter lab which is made available in docker-compose and K8s deployments.
Configurations
For more context on configuration
As a component of the Multi-Camera Tracking app, please refer to its Operation Parameters section.
As a component of the Real Time Location System app, refer to its Operation Parameters section.
As a component of the Occupancy Analytics app, please refer to its Operation Parameters section.
As a standalone microservice, refer to the
README.mdin its respective directory withinmetropolis-apps-standalone-deployment/modules/.
API Bootstrap Config
The Utils.Config class of the Web API library initializes the default bootstrap config. This config can be overridden by the user by passing either the entire config or a minimal config. The config passed by the user is read by API while starting the server. It remains constant as long as the server runs. If anything needs to be modified then the server needs to be restarted.
The bootstrap config’s attributes are as follows:
{
"server":{
"port": <Server will run on this port.>,
"configs":[
{
"name": "postBodySizeLimit",
"value": <Body size for post requests.>
},
{
"name":"amrRetentionInSec",
"value": <Rentention of AMR data in seconds>
}
]
},
"elasticsearch":{
"node": <Elasticsearch url>,
"indexPrefix": <Index Prefix>,
"rawIndex": <Raw Index>
},
"milvus":{
"url": <Milvus url or null - if it is not applicable>,
"collectionName": <name of collection>,
"partitioningStrategy": <strategy used for partitioning data. Valid values are 'day' or 'week'>,
"nprobeQbe": <nprobe config of Milvus for query by example>,
"partitionRetentionInDays": <partition retention in days>,
"searchQueryRetryPeriodInSec": <Milvus query retry time period in seconds>
},
"kafka":{
"brokers": <List of kafka brokers or (null or empty array) - if it is not applicable.>
}
}
Calibration Config
Calibration file is generated by the calibration toolkit. More details can be found over here.
Library
Documentation
Overview
The web-api-core library consists of four namespaces.
Metrics
Errors
Services
Utils
Each namespace consists of classes which helps to either compute metrics, get records from database or provide utilities with helper functions.
Metrics
The Metrics namespace consists of following classes:
TripwireEvent: Used to find the total number of events (eg: effective and actual tripwire crossing events) that have occurred. It is also used to compute Tripwire Histogram.LastProcessedTimestamp: Used to find the timestamp of last processed object.Occupancy: Used to find current occupancy of a place based on tripwire events. It also enables to reset occupancy. It helps in computing the average occupancy/histogram for objects in FOV and ROI.
Errors
The Errors namespace consists of following classes:
BadRequestError: Error to be used when the request sent by client is a Bad Request.IndexNotFoundError: Error to be used when Elasticsearch index is not found.InternalServerError: Error to be used when there is a server side error.InvalidInputError: Error to be used when the input is not valid.ResourceNotFoundError: Error to be used when a resource is not found.ServiceUnavailableError: Error to be used when a service is not available.
Services
The Services namespace consists of following classes:
Behavior: Used to fetch behavior, a behavior’s timestamp or to calculate a behavior’s start and end pts.Calibration: Used to upload/insert, update or delete sensors in calibration. It is also used to produce/consumer Kafka messages. It can also be used to upload/retrieve calibration related images.Clustering: Used to obtain behavior cluster. It is also used to add cluster labels.ConfigManager: Used to update the configs of other microservices. It is also used to obtain the current config of a microservice.Events: Used to fetch events data (eg: tripwire).Frames: Used to fetch clusters of objects which are in proximity of each other. Also used to fetch raw and enhanced frames. It is used to calculate pts and also fetch object with max confidence.MTMC: Used to obtain unique object and unique object counts. It can also be used for Query-by-Example (QBE) and to obtain locations of matched behaviors.NotificationManager: Used to produce and consume notification messages.
Utils
The Utils namespace consists of following classes:
Config: Provides the utility which initializes server config with default values. It also reads the bootstrap config provided by user which is used to override config values.Database: Wrapper class for various databases.Elasticsearch: Utilities related to Elasticsearch are provided in this class.FileUploadHandler: Provides helper functions related to file uploads.Histogram: Provides helper functions related to histogram.Kafka: Utilities related to Kafka are provided in this class.MessageBroker: Wrapper class for various message brokers.Milvus: Utilities related to Milvus are provided in this class.Utils: Provides helper functions used by Analytics & Tracking APIs.Validator: Utilities which help to validate inputs are provided in this class.
API
Documentation
The following sections describe the various API endpoints along with their respective sequence diagrams.
Multi Camera Tracking: Unique Objects
Library Usage
The web-api-core package has function which helps to retrieve unique objects tracked across sensors.
For example:
const mdx = require("@nvidia-mdx/web-api-core");
const elastic = new mdx.Utils.Elasticsearch({node: "elasticsearch-url"},databaseConfigMap);
let input = {sensorIds:["abc","def"],fromTimestamp: "2021-06-10T11:10:05.000Z", toTimestamp: "2021-06-10T11:10:10.000Z"}
let mtmc = new mdx.Services.MTMC();
let uniqueObjects = await mtmc.getUniqueObjects(elastic, input);
In the above example, getUniqueObjects is an async function which returns unique objects which were identified by MTMC microservice.
REST API
Sequence Diagram
The http request sent to the /tracker/unique-objects route is redirected to the controller function where the query input is validated and an Elasticsearch query object is generated. The Elasticsearch is queried as a promise (async call) which then responds with results. The results are formatted appropriately and sent back to the client.
Note
The query parameter ‘place’ sent as part of request is a hierarchy of places. For example, a room of a building located in a particular city will be in the following format: city=abc/building=pqr/room=xyz.
Tripwire Events
Library Usage
The web-api-core package enables the user to retrieve tripwire events. For example:
const mdx = require("@nvidia-mdx/web-api-core");
const elastic = new mdx.Utils.Elasticsearch({node: "elasticsearch-url"},databaseConfigMap);
let input = {sensorId:"abc",fromTimestamp: "2021-06-10T11:10:05.000Z", toTimestamp: "2021-06-10T11:10:10.000Z"}
let tripwireMetadata = new mdx.Services.Events();
let tripwireEvents = await tripwireMetadata.getTripwireEvents(elastic,input);
In the above example getTripwireEvents is an async function which returns tripwire events.
REST API
Sequence Diagram
The http request sent to the /events/tripwire route is redirected to the controller function where the query input is validated and an Elasticsearch query object is generated. The Elasticsearch is queried as a promise (async call) which then responds with results. The results are formatted appropriately and sent back to the client.
Note
The query parameter ‘place’ sent as part of request is a hierarchy of places. For example, a room of a building located in a particular city will be in the following format: city=abc/building=pqr/room=xyz.
Metrics: Tripwire Counts
Library Usage
The web-api-core package has helpful functions which can be used to calculate Tripwire related metrics.
For example:
const mdx = require("@nvidia-mdx/web-api-core");
const elastic = new mdx.Utils.Elasticsearch({node: "elasticsearch-url"},databaseConfigMap);
let input = {sensorId:"abc",fromTimestamp: "2021-06-10T11:10:05.000Z", toTimestamp: "2021-06-10T11:10:10.000Z"}
let tripwireMetricObject = new mdx.Metrics.TripwireEvent();
let tripwireCounts = await tripwireMetricObject.getTripwireCounts(elastic,input);
In the above example, getTripwireCounts is being used to get tripwire counts for each tripwire of sensor abc. It is also used to get effective tripwire counts for each tripwire of sensor abc.
Effective counts are used to count only the effective event generated by an object. An effective tripwire event can be defined with the help of following example: Suppose an object generates an IN tripwire event followed by an OUT and then finally an IN. The effective count in this case consideres only the final IN event. This is helpful to handle loitering conditions.
REST API
The http request sent to the /metrics/tripwire/counts route is redirected to the controller function where the query input is validated and an Elasticsearch query object is generated.
The Elasticsearch is queried as a promise (async call) to calculate detailed count and effective count for each tripwire of the sensor.
The API receives the result for both the queries which is formatted appropriately before sending the response to the client.
Example:
A client sends a request to obtain tripwire counts of sensorId abc which has 2 tripwires.
Let’s assume the query was sent for the time period t1 - t2 when tripwire-id-1 had IN, OUT and IN events produced by the same object and tripwire-id-2 had a single IN event.
Then the response in this scenario should look as follows:
{
"tripwireKpis":[
{
"id": "tripwire-id-1",
"events":[
{
"type": "IN",
"count": 1,
"actualCount":2
},
{
"type": "OUT",
"count": 0,
"actualCount":1
}
]
},
{
"id": "tripwire-id-2",
"events":[
{
"type": "IN",
"count": 1,
"actualCount": 1
},
{
"type": "OUT",
"count": 0,
"actualCount": 0
}
]
}
],
"aggregatedKpis":{
"events": [
{
"type": "IN",
"count": 2,
"actualCount": 3
},
{
"type": "OUT",
"count": 0,
"actualCount": 1
}
]
}
}
The count attribute in the above response is the effective count of events.
Note
The query parameter ‘place’ sent as part of request is a hierarchy of places. For example, a room of a building located in a particular city will be in the following format: city=abc/building=pqr/room=xyz.
Other API Endpoints
The endpoints listed over here have very similar sequence diagrams as tripwire count and tripwire events endpoints.
Last Processed Timestamp: This endpoint is used to find the latest timestamp of a place, successors of a place in a hierarchy or of a particular sensor.
Tripwire Histogram: This endpoint is used to create histogram for tripwire events. Each bucket of histogram will have effective counts and actual counts.
Tripwire Based Occupancy: This endpoint is used to compute occupancy of a place based on tripwire events.
Occupancy Reset: This endpoint is used to reset the occupancy of a place.
FOV Occupancy: This endpoint is used to calculate the occupancy of a sensor’s FOV by different objects during a given time period.
FOV Histogram: This endpoint is used to create histogram based on the occupancy of a sensor’s FOV by different objects.
ROI Occupancy: This endpoint is used to calculate the occupancy of sensor’s ROIs by different objects during a given time period.
ROI Histogram: This endpoint is used to create histogram based on the occupancy of sensor’s ROIs by different objects.
Occupancy based on Mutually Exclusive ROIs: This endpoint is used to calculate occupancy of a place based on mutually exclusive ROIs.
Behavior Clustering: This endpoint is used to retrieve sampled behaviors of each cluster belonging to a sensor for a given time range.
Add Cluster Labels: This endpoint is used to insert labels for a cluster.
Frames Metadata: This endpoint is used to retrieve raw frame data.
Enhanced Frames Metadata: This endpoint is used to retrieve enhanced frame data.
Frame Pts: This endpoint is used to get the current presentation time of a video source. This is mainly used for NVStreamer sources.
Proximity Detection: This endpoint is used to obtain clusters of objects that are in proximity with each other.
High Confidence Objects: This endpoint is used to obtain objects detected in a particular sensor with high confidence.
Behavior Metadata: This endpoint is used to retrieve behavior data.
Behavior Pts: This endpoint is used to get a behavior’s start and end presentation time based on the video source. This is mainly used for NVStreamer sources.
Unique Object Count: This endpoint is used to get unique object count across sensors.
Unique Object Count With Locations: This endpoint is used to get unique object count of different object types along with their locations.
Behavior Locations: This endpoint is used to get locations of behaviors that get matched to a global object.
Query By Example: This endpoint is used to obtain behaviors having embedding similar to the example provided in the input.
Note
This endpoint may fail if it is used at midnight (machine time). This is because a Cron job will be executing at that time to load/unload and drop Milvus partitions.
Upload Config Files: This endpoint is used to upload config files like calibration.json.
Update Config: This endpoint is used to dynamically update existing configurations of microservices. If a configuration doesn’t exist in database then it is fetched from that microservice.
Get Calibration Config: This endpoint is used to get config like calibration from database.
Upsert Calibration Config: This endpoint is used to insert new sensors into calibration config. It can also be used to update an existing sensor in the calibration config.
Delete Sensor Calibration: This endpoint is used to delete sensors present in calibration config.
Upload Calibration Images: This endpoint is used to upload calibration related images. The upload consists of images and imageMetadata JSON file.
Get Calibration Image: This endpoint is used to retrieve a calibration related image.
Get Metadata of Calibration Images: This endpoint is used to retrieve metadata of calibration related images.
Delete Calibration Images: This endpoint is used to delete calibration related images.
Health Check: This endpoint responds with
{ "isAlive": true }if the web-API server has started successfully.