Known Limitations#

This section aligns with PRD and current release constraints, providing transparency about known limitations.

Perception#

Truncated Bounding Box#

In the 2D Blueprint, if an object is partially visible due to occlusion or being outside the camera’s field of view, the detected bounding box from the perception model can be truncated. The midpoint on the lower edge of the bounding box is used to approximate the object’s location, which is then used for various downstream analytics computations such as proximity detection. If the bounding box is truncated, the accuracy of location-related analytics can be impacted.

For example, in the image below, the person on the right has a truncated bounding box because they are partially outside the camera’s field of view. This can affect the accuracy of proximity detection between the two individuals.

Truncated Bbox Proximity Detection

More advanced techniques will be included in future releases to address similar issues.

Agents#

Known issues with the Agents are listed below.

  • Video clips analyzed by VLM should be 5-300 seconds for optimal results

  • UI conversation storage is limited by browser storage limits.

  • Report generated by VLM may contain inaccurate timestamps sometimes.

  • On H100 and RTX PRO 6000 Blackwell, only the default NIMs (LLM and VLM) have been tested to run on shared GPUs. On L40S, shared GPU mode is not supported. Use Dedicated GPUs or Remote NIMs for other models.

  • On L4, RTX 6000 ADA, IGX-THOR and DGX-SPARK local NIMs are not supported. Use Remote NIMs for all models.

Alert Verification#

While running the Warehouse Blueprint 2D Vision AI Profile with Agents, you may sometimes see the following error in mdx-vlm-incidents:

VLM analysis error: cannot access local variable 'response_content' where it is not associated with a value

This can occur when there is a connectivity issue with the VLM service.

Performance#

Latency Issues on IGX Thor#

Latency issues are observed on NVIDIA IGX Thor. See Latency Measurements for detailed performance comparisons across different hardware platforms. Latency issues will be fixed in future release.

Using Real Cameras in 3D Blueprint#

The 3D Warehouse Blueprint is provided and tested with synthetic cameras. Benefit of using synthetic cameras is that the calibration related parameters are fully accurate and easily provided via Simulation-Based Calibration documentation. Potentially the 3D Blueprint can be used with real cameras but it has not been thoroughly tested and currently there is no tooling provided to acquire camera parameters required for 3D use. For users who have deep understanding and knowledge in camera calibration, you can refer to the manual calibration guide documentation in VSS Next to get the parameters required for 3D use from real cameras.

2D UI known limitations#

Dots in warped image and live stream go out of sync#

In reference Video Analytics UI, currently the dots representing detected objects in the warped image and the live stream video can go out of sync as each of these is rendered independently. Therefore, the sync between them is best effort. This can be improved by modifying the uiDelaySeconds and liveSyncVideoRefreshIntervalMin values under the UI config and making them lower.

Note

Decreasing the value of liveSyncVideoRefreshIntervalMin will cause the user interface to refresh more frequently.

Dots location accuracy in UI#

In reference Video Analytics UI, the cameras in the data package have noticeable lens distortion. As a result, the provided calibration file created using Calibration Toolkit may not provide the most accurate location projection in 2D UI, hence impacting the accuracy of dots corresponding to location of a person.

Can I have multiple Buildings in single project in Calibration Toolkit ?#

Currently in Metropolis Calibration Toolkit, there is only one building per project. The user can manually adjust the calibration.json to add buildings to each sensor.

When adjusting the Warped Image Crop, the Load Defaults does not load anything.#

Currently in Metropolis Calibration Toolkit, the Load Default is inactive when trying to adjust Warped Image crop.

The tripwire and ROI coordinates in the output calibration.json file are inaccurate when using project type - Cartesian with more than 4 anchor points in Calibration Toolkit.#

Currently in Metropolis Calibration Toolkit, when using project type - Cartesian with more than 4 anchor points, the tripwire and ROI coordinates in the output calibration.json file appear to be inaccurate. If you need to define tripwire and ROI in project type - Cartesian, we recommend using just 4 anchor points for now, please refer to Cartesian Calibration Section for more details. This is a known limitation and will be addressed in future releases.

WebRTC based Video Playback in UI#

In reference Video Analytics UI, the video playback may not be supported. If the Video streaming service is running behind a private network where the WebRTC turn server is not accessible (or setup appropriately), the video playback may not be supported.

Camera Calibration with AutoMagicCalib (AMC)#

The following are known limitations when using the VSS Auto Calibration Microservice with AutoMagicCalib (AMC).

Manual Input of Attribute Values Not Supported#

Currently, AMC does not support manually inputting attribute values during the calibration workflow. After the calibration file is generated, you may need to manually edit the output file to change attribute values from their defaults to your customized values. In v3.1, AMC will add the ability to modify attribute values directly within the tool before the calibration file is output.

Floor Map and Scale Factor Requirement#

AMC currently requires user to provide a floor map and a proper scaleFactor. AMC uses these inputs to scale the global coordinate system to meters. Behavior analytics assumes the global coordinate system is in meters and computes distance-related properties, such as proximity violation, based on it. If the scaleFactor is not provided or is improper with respect to the floor map, the computed distance-related properties can be affected. In future releases, AMC will remove the requirement to provide both a floor map and scaleFactor.