Known Limitations#
This section aligns with PRD and current release constraints, providing transparency about known limitations.
Perception#
Truncated Bounding Box#
In the 2D Blueprint, if an object is partially visible due to occlusion or being outside the camera’s field of view, the detected bounding box from the perception model can be truncated. The midpoint on the lower edge of the bounding box is used to approximate the object’s location, which is then used for various downstream analytics computations such as proximity detection. If the bounding box is truncated, the accuracy of location-related analytics can be impacted.
For example, in the image below, the person on the right has a truncated bounding box because they are partially outside the camera’s field of view. This can affect the accuracy of proximity detection between the two individuals.
More advanced techniques will be included in future releases to address similar issues.
VLM Verification of Incidents#
When a person is inside a forklift, it may be classified as a proximity incident. This false positive will be addressed in a future release.
Agents#
Known issues with the Agents are listed below.
Video clips analyzed by VLM should be 5-300 seconds for optimal results
UI conversation storage is limited by browser storage limits.
Report generated by VLM may contain inaccurate timestamps sometimes.
On H100 and RTX PRO 6000 Blackwell, only the default NIMs (LLM and VLM) have been tested to run on shared GPUs. On L40S, shared GPU mode is not supported. Use Dedicated GPUs or Remote NIMs for other models.
On L4, RTX 6000 ADA, IGX-THOR and DGX-SPARK local NIMs are not supported. Use Remote NIMs for all models.
Alert Verification#
While running the Warehouse Blueprint 2D Vision AI Profile with Agents, you may sometimes see the following error in mdx-vlm-incidents:
VLM analysis error: cannot access local variable 'response_content' where it is not associated with a value
This can occur when there is a connectivity issue with the VLM service.
Using Real Cameras in 3D Blueprint#
The 3D Warehouse Blueprint is provided and tested with synthetic cameras. Benefit of using synthetic cameras is that the calibration related parameters are fully accurate and easily provided via Simulation-Based Calibration documentation. Potentially the 3D Blueprint can be used with real cameras but it has not been thoroughly tested and currently there is no tooling provided to acquire camera parameters required for 3D use. For users who have deep understanding and knowledge in camera calibration, you can refer to the manual calibration guide documentation in VSS Next to get the parameters required for 3D use from real cameras.
2D UI known limitations#
Dots in warped image and live stream go out of sync#
In reference Video Analytics UI, currently the dots representing detected objects in the warped image and the live stream video can go out of sync as each of these is rendered independently. Therefore, the sync between them is best effort. This can be improved by modifying the uiDelaySeconds and liveSyncVideoRefreshIntervalMin values under the UI config and making them lower.
Note
Decreasing the value of liveSyncVideoRefreshIntervalMin will cause the user interface to refresh more frequently.
Dots location accuracy in UI#
In reference Video Analytics UI, the cameras in the data package have noticeable lens distortion. As a result, the provided calibration file created using Calibration Toolkit may not provide the most accurate location projection in 2D UI, hence impacting the accuracy of dots corresponding to location of a person.
Object Dots always showing red outer circle in UI#
In reference Video Analytics UI, the outer red circle around object dots indicates a proximity violation. Currently, this red circle appears around all detected objects, even when no proximity violation has occurred. This is a known visual indicator issue that will be addressed in future releases.
Can I have multiple Buildings in single project in Calibration Toolkit ?#
Currently in Metropolis Calibration Toolkit, there is only one building per project. The user can manually adjust the calibration.json to add buildings to each sensor.
When adjusting the Warped Image Crop, the Load Defaults does not load anything.#
Currently in Metropolis Calibration Toolkit, the Load Default is inactive when trying to adjust Warped Image crop.
The tripwire and ROI coordinates in the output calibration.json file are inaccurate when using project type - Cartesian with more than 4 anchor points in Calibration Toolkit.#
Currently in Metropolis Calibration Toolkit, when using project type - Cartesian with more than 4 anchor points, the tripwire and ROI coordinates in the output calibration.json file appear to be inaccurate. If you need to define tripwire and ROI in project type - Cartesian, we recommend using just 4 anchor points for now, please refer to Cartesian Calibration Section for more details. This is a known limitation and will be addressed in future releases.
WebRTC based Video Playback in UI#
In reference Video Analytics UI, the video playback may not be supported. If the Video streaming service is running behind a private network where the WebRTC turn server is not accessible (or setup appropriately), the video playback may not be supported.
Camera Calibration with AutoMagicCalib (AMC)#
The following are known limitations when using the VSS Auto Calibration Microservice with AutoMagicCalib.
Floor Map and Scale Factor Requirement#
AMC currently requires the user to provide a floor map and a proper scaleFactor.
AMC uses these inputs to scale the global coordinate system to meters.
Behavior analytics assumes the global coordinate system is
in meters and computes distance-related properties, such as proximity violation, based on it.
If the scaleFactor is not provided or is improper with respect to the floor map,
the computed distance-related properties can be affected.
In future releases, AMC will remove the requirement to provide both a floor map and scaleFactor.
Calibration is based on detected persons#
AMC detects person movements from input videos and performs camera calibration based on them. The video should have people moving into various locations in camera view for AMC to work best. Currently AMC only supports person as the target and it will not work on video without enough person movements. Support for general object tracking will be added in future releases.
Global Tripwire and ROI for 3D Blueprint#
VSS warehouse blueprint 3D profile needs global tripwire / ROI to generate related events.
AMC currently supports per sensor tripwire / ROI only.
To make the AMC output calibration file compatible with 3D blueprint,
the user can first define the tripwire / ROI for one particular sensor then in the output calibration file
manually move the tripwire/ROI from under sensors section to top level of the calibration file.
Image mode calibration#
VSS warehouse blueprint has mainly been used for use cases where the locations of objects are tracked and analyzed in a global coordinate system.
It supports all functionalities in image pixel coordinate as well, which we call image mode.
AMC currently does not support image mode. For image mode, there is no camera parameters needed (intrinsics, extrinsics, homography, etc.) since there is no coordinate transformation involved.
The tripwires and ROIs need to be defined in pixel coordinates. Since AMC does not support image mode at the moment, the user needs to manually find the pixel coordinates for the defined tripwires and ROIs.
It can be done by using image editing software or tools like Paint, IrfanView, etc.
In addition, the user needs to set "calibrationType": "image" instead of "calibrationType": "cartesian" in the calibration file.
Below is an example of the calibration file for image mode:
{
"version": "1.0",
"osmURL": "",
"calibrationType": "image",
"sensors": [
{
"id": "Camera_01",
"type": "camera",
"imageCoordinates": [],
"globalCoordinates": [],
"coordinates": {
"x": 0,
"y": 0
},
"origin": {
"lat": 0,
"lng": 0
},
"geoLocation": {
"lat": 0,
"lng": 0
},
"scaleFactor": 1,
"attributes": [
{
"name": "fps",
"value": "0.0"
},
{
"name": "depth",
"value": ""
},
{
"name": "fieldOfView",
"value": ""
},
{
"name": "direction",
"value": ""
},
{
"name": "source",
"value": "vst"
},
{
"name": "frameWidth",
"value": "1920"
},
{
"name": "frameHeight",
"value": "1080"
}
],
"rois": [
{
"id": "roi-id-1",
"roiCoordinates": [
{
"x": 4.87747667,
"y": 18.9756077
},
{
"x": 4.87747667,
"y": 11.2385153
},
{
"x": 9.57440535,
"y": 11.2385153
},
{
"x": 9.57440535,
"y": 18.9756077
}
],
"restrictedObjectTypes":["Person"],
"confinedObjectTypes": ["Forklift"]
}
],
"place": [],
"tripwires": [
{
"id": "tripwire-id-1",
"wire": {
"p1": {
"x": 943,
"y": 606.0
},
"p2": {
"x": 1403.0,
"y": 309.0
}
},
"direction": {
"p1": {
"x": 1040.0,
"y": 395
},
"p2": {
"x": 1282.0,
"y": 566.0
}
}
}
]
}
]
}