2D Vision AI Profile

Overview

Introduction

The VSS Warehouse 2D Blueprint is a comprehensive guide to building a 2D intelligent video analytics system. It provides a detailed overview of the system architecture, data flow, and key components.

Deployment Architecture

Components and Interactions

The diagram depicts VSS Warehouse 2D Blueprint, emphasizing 2D single-camera detection, tracking, and behavior analytics for safety events and metrics. Below is a breakdown of the components and their interactions.

1. Input Source

   • Videos: Raw video data stored in a filesystem, serving as input for processing.
   • NvStreamer (NVStreamer): A microservice that streams videos via RTSP (Real-Time Streaming Protocol) to the VIOS (Video IO & Storage). NvStreamer can be swapped with real-world cameras.

2. Video IO & Storage (VIOS) (VIOS Microservices)

   • VIOS ingests video streams from NvStreamer via RTSP.
   • It records the streams and forwards them (via RTSP) to the DeepStream microservice for further processing.

3. DeepStream (2D Single Camera Detection and Tracking (RT-DETR))

   • DeepStream processes RTSP streams for 2D single-camera detection and tracking, utilizing the RT-DETR (Real-Time Detection Transformer) model (RT-DETR (2D warehouse detection)) to generate precise 2D bounding boxes for diverse objects including people, humanoid robots, autonomous vehicles, and warehouse equipment.
   • RT-DETR features a ResNet-50 backbone, pretrained on warehouse scene datasets for accurate 2D object detection in industrial environments.
   • It sends frame data, including detected and tracked object IDs, in Protobuf format to the message broker via the mdx-raw topic.

4. Message Broker (Kafka or Redis)

   • The message broker serves as the central hub for data distribution, using Protobuf for all data exchanges.
   • Kafka (Kafka): High-throughput message broker optimized for datacenter deployments with robust persistence and scalability.
   • Redis Streams: Lightweight message broker ideal for edge deployments with minimal memory footprint and low-latency requirements.
   • It also functions as a control bus, managing notifications (in JSON, via mdx-notification) for calibration updates, such as new ROI or tripwire definitions.

5. Behavior Analytics (Behavior Analytics)

   • This microservice consumes mdx-raw data (Protobuf) from the message broker.
   • It processes the data to generate behavior analytics, safety insights, and metrics.
   • The resulting data, in Protobuf format, is sent back to the message broker for indexing into Elasticsearch.

6. Storage

   • ELK (Elasticsearch, Logstash, Kibana) (ELK): Logstash retrieves mdx-raw outputs and safety violation frames from the message broker, converts Protobuf to JSON, stores the data in Elasticsearch, and supports querying and visualization.

7. Visualization

   • Kibana UI (Kibana UI): A user interface for visualizing analytics data stored in Elasticsearch.
   • VIOS UI (VIOS UI): A separate interface for interacting with the VIOS system, receiving JSON notifications from the message broker.

8. External Interfaces

   • API Gateway and MCP (API Gateway / MCP): Enables external systems to interact with the events data through API calls.

Key Technologies

• Microservices: Components like NvStreamer, VIOS, DeepStream, and Behavior Analytics are modular microservices.
• RTSP: Facilitates real-time video streaming.
• Protobuf: Ensures efficient, compact data exchange.
• Message Broker: Manages data distribution and control messaging.
• ELK Stack: Supports storage, logging, and visualization.
• JSON: Used for notifications and calibration data.

Setup and Configuration

see Quickstart Guide

Testing and Validation

Kibana UI

Check for Events, Frames and Behavior Data Views in Kibana

1. Launch Chrome browser
2. In the address bar enter http://<IP_address>:7777/kibana

In the user interface, navigate to the Management -> Stack Management section and select Data Views under Kibana. If the data views are not visible, create a new data view (via “Create data view” button on the top right corner) for mdx-raw, mdx-frames and mdx-behavior.

Browse the Kibana UI, discover the data views and visualize the data.

VIOS UI

Please note: At this point the web-based application is only available for Chrome browser running on Linux, Windows or macOS, details can be found in VIOS docs.

1. Launch Chrome browser
2. In the address bar enter http://<IP_address>:30888/vst/

Configure and View

Show Object IDs

To display object IDs in VST for the 2D profile, open Analytics Overlay Settings and:

1. Enable Show Bounding Boxes under Bounding Box Configuration.
2. Enable Show Object ID under Object ID Display.
3. Click Save Settings.

Debug Mode is optional and adds diagnostic overlays; it is not required to display object IDs.

Events and Incidents

Events

The VSS Warehouse 2D Blueprint processes live camera feeds, detects and tracks objects of interest, and generates events based on configurable logic.

Two types of events are supported:

• ROI Events: Triggered when objects enter or exit a predefined region of interest (ROI).
• Tripwire Events: Triggered when objects cross a predefined tripwire.

Each event type is associated with a specific logic and configuration. For more details, see Event Detection.

All event data is stored in Elasticsearch, and can be queried and visualized in Kibana and the VIOS UI.

Incidents

In addition to instantaneous events, the 2D Blueprint detects violations in each enhanced frame and tracks them over time. When a violation persists beyond its configured threshold, it is promoted to a confirmed incident. The 2D Blueprint generates the following incident types:

• Proximity Violation: Objects come too close to each other (for example, a person near moving equipment).
• Restricted Area Violation: Objects enter a prohibited zone (ROI).
• Confined Area Violation: Objects leave a designated safe zone.

Incident timing (threshold and expiration window) is configurable per type. For the violation-to-incident framework, configuration parameters, and enhanced-frame data, see Incident Generation.

Use Cases

ROI events, tripwire events, and the proximity, restricted-area, and confined-area incidents together provide a powerful foundation for safety monitoring and operational intelligence in warehouse environments, enabling real-time situational awareness and proactive safety management across multiple critical use cases.

• ROI events continuously monitor whether designated safety zones remain human-free or equipment-free, ensuring that hazardous areas such as automated vehicle pathways, robotic work cells, or restricted maintenance zones are clear before operations commence. By tracking cumulative enter and exit counts for defined ROIs, the system maintains accurate occupancy counts and can alert when unauthorized access occurs or when occupancy thresholds are exceeded.

• Tripwire events enable precise directional counting at strategic locations such as doorways, loading dock entrances, and zone boundaries. This is particularly valuable for monitoring areas not directly visible to cameras: if a person crosses a tripwire entering a blind corridor but no corresponding exit event is detected within an expected timeframe, the system can infer continued presence and raise safety alerts or delay automated equipment operations in that zone.

• Incidents capture sustained unsafe conditions: proximity violations flag dangerous closeness between humans and moving equipment such as forklifts, autonomous mobile robots (AMRs), or industrial manipulators for collision avoidance; restricted-area violations enforce access control on prohibited zones; and confined-area violations detect when objects leave their designated safe zones.

Together, these events and incidents form a comprehensive safety net that supports critical warehouse operations including dynamic safe-zone enforcement, pedestrian-vehicle collision prevention, restricted-area access control, and loading-dock safety management—all while providing the rich data necessary for compliance reporting, incident investigation, and continuous safety process improvement.

Latency Measurements

Customization

The Blueprint supports several levels of customization:

• Data Level: Add, remove, or replace cameras while maintaining the existing workflow.
• Model Level: Fine-tune perception models to better suit your use cases.
• Application Level: Build new microservices or applications using the provided APIs and components.
• Microservice Level: Modify existing microservices from source code to extend functionality.

Adding New Cameras or Custom Datasets

Step 1: Prepare video sources

Choose one of the following options to provide video sources for the 2D profile:

1. Using recorded videos

   Use this option to run the VSS with local video files instead of live camera feeds. Place your videos in the videos directory under the path configured by VSS_DATA_DIR in $VSS_APPS_DIR/industry-profiles/warehouse-operations/overrides.env.

2. Using live RTSP streams

   Use this option when you want to use live RTSP endpoints. Define each camera in $VSS_APPS_DIR/industry-profiles/warehouse-operations/camera_configs/camera_info.json so the VSS Configurator can read the camera list and generate the required configuration. The following steps apply when using live RTSP streams.

   (i) Create a Sensor Info File (camera_info.json):

   {
     "sensors": [
       {
         "camera_name": "camera-01",
         "rtsp_url": "rtsp://<IP_address:port>/stream1"
       },
       {
         "camera_name": "camera-02",
         "rtsp_url": "rtsp://<IP_address:port>/stream2"
       }
     ]
   }

   The camera_name value should be unique for each camera, and rtsp_url should point to the camera stream.

   Required fields: camera_name, rtsp_url

(ii) Configure VSS Configurator environment variables: Set the following in $VSS_APPS_DIR/industry-profiles/warehouse-operations/overrides.env, which is loaded after warehouse .env and takes precedence:

SENSOR_INFO_SOURCE=file
SENSOR_FILE_PATH=<path_to_camera_info.json>

Step 2: Configure number of streams

Configuring Number of Streams

Number of streams to be processed can be configured in two ways:

1. Static Configuration: Set the NUM_STREAMS environment variable in warehouse overrides.env file to specify the desired number of streams for video dataset or RTSP streams.

   NUM_STREAMS=4

   The configured number of streams should be less than or equal to the maximum streams supported by your hardware profile and deployment mode. VSS Configurator can be used to automatically cap the stream count using the formula: final_stream_count = min(NUM_STREAMS, max_streams_supported). When SENSOR_INFO_SOURCE=nvstreamer, the same NUM_STREAMS value is the expected camera count the Sensor Configuration Manager waits for before VST registration. For more details, refer to the How to Count Files Dynamically (Prerequisites) section and NVStreamer Settings in VSS Configurator Documentation.

2. Dynamic Configuration: Use the VSS Configurator’s prerequisite operations to automatically count the number of video files in the recorded videos directory and use that count for configuration updates.

Example: Automatically determine stream count from video files in the dataset directory:

# In blueprint_config.yml
commons:
  # Step 1: Count video files BEFORE variable processing
  prerequisites:
    2d:
      - operation_type: "file_management"
        target_directories:
          - "${VSS_DATA_DIR}/videos/${SAMPLE_VIDEO_DATASET}"
        file_management:
          action: "file_count"
          parameters:
            pattern: "*.mp4"
          output_variable: "available_video_count"  # Stores count (e.g., 6)
  # Step 2: Use the count to compute final stream count
  variables:
    2d:
      # Cap to minimum of: available videos, GPU limit
      - final_stream_count: "min(${available_video_count}, ${max_streams_supported})"
  # Step 3: Use computed variable in config file updates
  file_operations:
    2d:
      - operation_type: "text_config_update"
        target_file: "${DS_CONFIG_DIR}/ds-main-config.txt"
        updates:
          max-batch-size: ${final_stream_count}
          batch-size: ${final_stream_count}

How it works: If your dataset directory has 6 video files and the GPU supports a maximum of 4 streams, the configurator computes: final_stream_count = min(6, 4) = 4. For more details, refer to the How to Count Files Dynamically (Prerequisites) section in VSS Configurator Documentation.

When using a custom recorded-video dataset, NUM_STREAMS depends on the number of video files available in that dataset. Make sure NUM_STREAMS is less than or equal to the number of videos you want to process.

To use a custom dataset:

• For a custom synthetic dataset, use a dataset name that ends in -synthetic (for example, my-warehouse-synthetic). Custom real datasets do not have a suffix requirement.

• Set SAMPLE_VIDEO_DATASET in $VSS_APPS_DIR/industry-profiles/warehouse-operations/overrides.env to the name of your dataset.

• Place the video files in $VSS_DATA_DIR/videos/$SAMPLE_VIDEO_DATASET/.

• Add the image metadata and floor map files under $VSS_APPS_DIR/industry-profiles/warehouse-operations/warehouse-2d-app/calibration/sample-data/${SAMPLE_VIDEO_DATASET}. If you already have a calibration, add calibration.json to the same directory. Otherwise, generate it in Step 3. You can refer to sample datasets provided in the repository for the expected file structure and sample contents.

Step 3: Choose a calibration option

2D supports three calibration states — pick one:

• Cartesian ("calibrationType": "cartesian"): full image-to-global calibration, generated using Auto Calibration’s AMC method. ROI/tripwire authoring is fully supported. Follow Cartesian coordinate calibration below.
• Image coordinates, with a calibration file ("calibrationType": "image"): a pixel-coordinate calibration file exported from Auto Calibration workflow, without an image-to-global transform. Auto Calibration supports ROI/tripwire authoring and image-mode JSON export. Follow Image coordinate calibration below.
• Running 2D without calibration: skip calibration entirely. Detection/ tracking still runs in image (pixel) coordinates; ROI/tripwire events are unavailable. See below.

Auto Calibration setup

Auto Calibration supports two types of calibrations in 2D profile: image-coordinate calibration and Cartesian calibration.

Before deploying the stack and generating a calibration, create the projects directory:

mkdir -p ${VSS_APPS_DIR}/services/auto-calibration/projects

Change the ownership of the Auto Calibration data and projects directories to UID 1000 and GID 1000:

sudo chown -R 1000:1000 ${VSS_DATA_DIR}/auto-calib
sudo chown -R 1000:1000 ${VSS_APPS_DIR}/services/auto-calibration/projects

Configure industry-profiles/warehouse-operations/overrides.env using the Auto-Calibration tab under Deployment Selection, then follow Deploy the Blueprint to start the Auto-Calibration profile.

Follow the Auto Calibration workflow steps using the warehouse dataset and VST stack. The workflow supports both calibration types; choose the appropriate export for your use case.

Image coordinate calibration

Use Export Image-Mode JSON to export ROIs and tripwires in pixel coordinates with "calibrationType": "image".

For image-coordinate calibration ("calibrationType": "image") only, add ROIs or tripwires to the sensors that require them. If only one sensor needs an ROI or tripwire, annotate that sensor and proceed with the export. Sensors that need neither ROIs nor tripwires do not need to be included in the image-coordinate calibration.json. This omission applies only to image-coordinate calibration, not Cartesian calibration.

Cartesian coordinate calibration

Export a Cartesian calibration with "calibrationType": "cartesian" to map camera image coordinates to a global coordinate system.

Save and apply the calibration

Bring the Docker Compose stack down after exporting the calibration. Save the exported file as $VSS_APPS_DIR/industry-profiles/warehouse-operations/warehouse-2d-app/calibration/sample-data/${SAMPLE_VIDEO_DATASET}/calibration.json, then proceed to Step 4 to redeploy with the new calibration.

Running 2D without calibration

Applies only when there’s no calibration.json file at all — not for either calibration type above. Detection/tracking still runs directly on the camera stream in image (pixel) coordinates; only ROI/tripwire events in behavior-analytics are unavailable. Calibration is enabled by default; to skip it entirely:

• Remove import-calibration-output-container-2d from whichever profile variable you’re deploying with — COMPOSE_PROFILES_WH_2D, COMPOSE_PROFILES_WH_KAFKA_2D, COMPOSE_PROFILES_WH_REDIS_2D, COMPOSE_PROFILES_PLAYBACK_KAFKA_2D, COMPOSE_PROFILES_PLAYBACK_REDIS_2D — edit the value directly in overrides.env.

• Remove --calibration /resources/calibration.json from the vss-behavior-analytics service’s command: in warehouse-2d-app/warehouse-2d-app.yml. This is a static line in the compose file, not an env var, so it needs a direct edit (or a compose override file layered on top).

• Also remove the vss-behavior-analytics-2d service’s calibration.json bind mount in the same file — the volumes: line ending .../calibration/sample-data/$SAMPLE_VIDEO_DATASET/calibration.json:/resources/calibration.json. This is independent of the profile and command changes above. If a custom dataset has no calibration.json on the host, Compose creates an empty directory at that path to satisfy the mount, which then blocks placing a real calibration.json there later — leave the mount in only if the file already exists.

• The bp-configurator-2d service’s /opt/data mount (.../calibration/sample-data/$SAMPLE_VIDEO_DATASET:/opt/data) is unused in 2D mode — good to remove.

Step 4: Redeploy the stack to apply the new calibration

Before redeploying, replace the Auto-Calibration settings in industry-profiles/warehouse-operations/overrides.env with the settings from the 2D tab under Deployment Selection. Then follow Deploy the Blueprint to start the 2D profile with the new calibration.

docker ps -a

Model Customization

Perception Model Fine-tuning

The Blueprint uses RT-DETR (Real-Time Detection Transformer) as its primary perception model. RT-DETR features a ResNet-50 backbone and is pretrained on warehouse scene datasets for precise 2D object detection in industrial environments. For details, see: 2D Single Camera Detection and Tracking.

For fine-tuning the RT-DETR model via TAO Toolkit on custom datasets, refer to: RT-DETR (TAO fine-tuning).

Application Customization

The Blueprint uses a modular microservices architecture with the following communication channels:

• Message Broker (Kafka or Redis)
• Elasticsearch database
• REST APIs

Users can build their own microservices by consuming data from the above channels.

For complete API documentation, see API Reference Page.

Available Service Ports

The following ports are used during deployment, and users can leverage them for any potential integration:

Analytics Microservices Customization (Advanced)

For detailed information about customizing specific analytics microservices, refer to:

• Behavior Analytics
• VSS Video Analytics API

Hardware Config Customization

The 2D Warehouse Blueprint requires several configuration files to be properly tuned based on your GPU hardware and deployment requirements. When changing hardware (e.g., switching from H100 to L4 GPU) or adjusting the number of video streams, multiple configuration files must be updated to ensure optimal performance and prevent GPU overload.

Configuration Files Requiring Hardware-Based Updates

The following table lists the configuration files that typically require updates when hardware changes:

There are two approaches to customize these configuration files:

Approach 1: Manual Configuration

Manually update all required configuration files before deploying the blueprint. This is time consuming, error prone and often not suitable for production deployments.

# Update DeepStream main config
vi <PATH_TO_DS_CONFIG_DIR>/ds-main-config.txt
# Set: num-source-bins=0, max-batch-size=<stream_count>, batch-size=<stream_count>
# Set msg-broker-config to ds-kafka-config.txt or ds-redis-config.txt depending on deployment mode
# Update NvStreamer config
vi <PATH_TO_NVSTREAMER_CONFIG_DIR>/vst-config.json
# Set: "nv_streamer_sync_file_count": <stream_count>
# Update VIOS (VST) config
vi <PATH_TO_VST_CONFIG_DIR>/vst_config.json
# Set: "onvif.max_devices_supported": <max_streams>, "data.always_recording": true,
#      "notifications.use_message_broker_consumer": "<deployment_mode>"

Approach 2: Automatic Config Management using VSS Configurator

The VSS Configurator provides a declarative approach to automatically update all required configuration files based on your hardware profile and deployment mode. This is the recommended approach for production deployments and simplifies the configuration management process.

For how the configurator updates configurations in this blueprint, see VSS Configurator for the Warehouse Blueprint. The guide describes configuration inputs, affected files, stream ceilings, and hardware-specific overrides.

To enable the VSS Configurator, in VSS Configurator’s environment variables you must set:

ENABLE_PROFILE_CONFIGURATOR=true

The VSS Configurator is enabled by default (ENABLE_PROFILE_CONFIGURATOR=true). When enabled, it runs before the Blueprint Deployment starts and adjusts configuration files based on the hardware profile and deployment mode as defined in the HARDWARE_PROFILE and MODE environment variables.

The VSS Configurator provides a comprehensive set of features for automated profile configuration management:

Execution Order: Environment Variable Validation → Prerequisite Operations → Variable Computations → Configuration File Updates

For detailed information on how to create custom hardware profiles and advanced configuration options, refer to the Profile Configuration Manager section in VSS Configurator Documentation.