2D Vision AI with Agents Profile

Overview

Introduction

The VSS Warehouse Blueprint’s 2D Vision AI with Agents Profile is a comprehensive guide to building a 2D intelligent video analytics system. It provides a detailed overview of the system architecture, data flow, and key components.

Deployment Architecture

Components and Interactions

The diagram depicts VSS Warehouse 2D Vision AI with Agents Profile, emphasizing 2D single-camera detection, tracking, and behavior analytics for safety events and metrics. Below is a breakdown of the components and their interactions.

1. Input Source

   • Videos: Raw video data stored in a filesystem, serving as input for processing.
   • NvStreamer (NVStreamer): A microservice that streams videos via RTSP (Real-Time Streaming Protocol) to the VIOS (Video IO & Storage). NvStreamer can be swapped with real-world cameras.

2. Video IO & Storage (VIOS) (VIOS Microservices):

   • VIOS ingests video streams from NvStreamer via RTSP.
   • It records the streams and forwards them (via RTSP) to the DeepStream microservice for further processing.

3. DeepStream (2D Single Camera Detection and Tracking (RT-DETR)):

   • DeepStream processes RTSP streams for 2D single-camera detection and tracking, utilizing the RT-DETR (Real-Time Detection Transformer) model (RT-DETR (2D warehouse detection)) to generate precise 2D bounding boxes for diverse objects including people, humanoid robots, autonomous vehicles, and warehouse equipment.
   • RT-DETR features a ResNet-50 backbone, pretrained on warehouse scene datasets for accurate 2D object detection in industrial environments.
   • It sends frame data, including detected and tracked object IDs, in Protobuf format to the Kafka messaging system via the mdx-raw topic.

4. Kafka (Messaging System) (Kafka):

   • Kafka serves as the central hub for data distribution, using Protobuf for all data exchanges.
   • It also functions as a control bus, managing notifications (in JSON, via mdx-notification) for calibration updates, such as new ROI or tripwire definitions.

5. Behavior Analytics (Behavior Analytics):

   • This microservice consumes mdx-raw data (Protobuf) from Kafka.
   • It processes the data to generate behavior analytics, safety insights, and metrics (ROI occupancy, tripwire crossings, proximity detections).
   • The resulting data, in Protobuf format, is sent back to Kafka (mdx-behavior, mdx-events, mdx-incidents) for indexing into Elasticsearch.
   • Candidate incidents (e.g., proximity violations) are forwarded to the Alerting Service for VLM-based verification.

6. RTVI VLM (Real-Time Vision-Language Model) (Real-Time VLM):

   • A real-time inference microservice that processes video through a Vision-Language Model (VLM).
   • Chunks live RTSP streams or stored video clips into short segments and sends them to the VLM NIM for analysis.
   • Serves both the Alert Verification workflow (re-evaluating candidate events) and the Real-Time Alerts workflow (continuous always-on monitoring).

7. Alerting Service (Alerting Service):

   • Operates in two modes:

     • Alert Verification: Consumes candidate events from Behavior Analytics (e.g., proximity violation on mdx-incidents), fetches the corresponding video clip from VIOS, sends it with a category-specific prompt to the VLM via RTVI VLM, and writes a verdict (confirmed / rejected / unverified) to Elasticsearch and Kafka.
     • Real-Time Alerts: Applies always-on rules (PPE, Load Quality, Pathway Obstruction, Spillover) to live RTSP streams via RTVI VLM, emitting an alert whenever the VLM classifies a chunk as a violation.

   • Alert type configurations are stored in Redis (RedisJSON) and can be updated at runtime.

   • For the full microservice reference, see Alerts Microservice.

8. LLM and VLM NIMs:

   • VLM NIM (e.g., NVIDIA Cosmos3 Nano Reasoner): The vision-language model backend that powers alert verification, real-time alerts, and video understanding for the agent. Receives video frames plus prompts and returns structured predictions.
   • LLM NIM (e.g., NVIDIA Nemotron 3.5 Lightning 30B A3B): The language model that powers the agent’s reasoning, query routing, and report generation.

9. Storage

   • Redis (Redis): Stores alert type configurations (RedisJSON), sensor data router state, and serves as an optional message broker.
   • ELK (Elasticsearch, Logstash, Kibana) (ELK): Logstash retrieves raw frames, behavior, incidents, and VLM alert outputs from Kafka, converts Protobuf to JSON, stores the data in Elasticsearch, and supports querying and visualization.

10. Visualization

    • Kibana UI (Kibana UI): A user interface for visualizing analytics data stored in Elasticsearch.
    • VIOS UI (VIOS UI): A separate interface for interacting with the VIOS system, receiving JSON notifications from Kafka.

11. External Interfaces

    • Video Analytics API (Video Analytics API): RESTful API providing programmatic access to analytics data, events, and incidents stored in Elasticsearch.
    • Video Analytics MCP Server (VSS Orchestrator MCP): Exposes Elasticsearch queries, RTVI VLM controls, and alert management as MCP (Model Context Protocol) tools for the agent.

12. Agents (Agents):

    • VSS Agent: An LLM-powered orchestration agent that interprets natural language queries, plans multi-step workflows, and invokes MCP tools. Capabilities include:

      • Querying warehouse safety incidents and occupancy data
      • Starting and stopping real-time VLM alert rules on specific cameras
      • Generating structured incident reports using VLM video analysis
      • Retrieving live camera snapshots and video clips

    • Agentic Reference UI (Reference Agentic UI): A browser-based interface with Chat, Alerts, and Dashboard modes for interacting with the agent system.

Key Technologies

• Microservices: Components like NvStreamer, VIOS, DeepStream, Behavior Analytics, RTVI VLM, and the Alerting Service are modular, independently deployable microservices.
• RTSP: Facilitates real-time video streaming between cameras, NvStreamer, VIOS, and RTVI VLM.
• Vision-Language Models (VLMs): NVIDIA Cosmos3 Nano Reasoner enables multi-step visual reasoning for alert verification, real-time anomaly detection, and video understanding.
• Large Language Models (LLMs): NVIDIA Nemotron 3.5 Lightning 30B A3B powers agent reasoning, query interpretation, and report generation.
• MCP (Model Context Protocol): A standardized tool interface that exposes microservice capabilities (analytics queries, alert management, video storage) to the LLM-powered agent.
• Protobuf: Ensures efficient, compact data exchange across the CV pipeline.
• Kafka: Manages data distribution and control messaging across all microservices.
• Redis: Stores alert configurations (RedisJSON) and provides optional message streaming.
• ELK Stack: Supports storage, logging, and visualization of events, incidents, and alert verdicts.
• JSON: Used for notifications, calibration data, and VLM alert outputs.

Setup and Configuration

For more information on setup and configuration, see the Quickstart.

Testing and Validation

Kibana UI

Check for Events, Frames and Behavior Data Views in Kibana

1. Launch Chrome browser
2. In the address bar enter http://<IP_address>:7777/kibana

In the user interface, navigate to the Management -> Stack Management section and select Data Views under Kibana. If the data views are not visible, create a new data view (via “Create data view” button on the top right corner) for mdx-raw, mdx-frames and mdx-behavior.

Browse the Kibana UI, discover the data views and visualize the data.

VIOS UI

Please note: At this point the web-based application is only available for Chrome browser running on Linux, Windows or macOS, details can be found in VIOS docs.

1. Launch Chrome browser
2. In the address bar enter http://<IP_address>:30888/vst/

Configure and View

Show Object IDs

To display object IDs in VST for the 2D profile, open Analytics Overlay Settings and:

1. Enable Show Bounding Boxes under Bounding Box Configuration.
2. Enable Show Object ID under Object ID Display.
3. Click Save Settings.

Debug Mode is optional and adds diagnostic overlays; it is not required to display object IDs.

Reference Agentic UI

The browser-based interface supports Chat, Alerts, and Dashboard modes. Details can be found in Reference Agentic UI.

1. Launch Chrome browser
2. In the address bar enter http://<IP_address>:7777

Configure and View

Events and Incidents

Events

The 2D Vision AI with Agents Profile runs the same Behavior Analytics 2D pipeline as the 2D Vision AI Profile, processing live camera feeds, detecting and tracking objects, and generating events based on configurable logic.

Two types of events are supported:

• ROI Events: Triggered when objects enter or exit a predefined region of interest (ROI).
• Tripwire Events: Triggered when objects cross a predefined tripwire.

Each event type is associated with a specific logic and configuration. For more details, see Event Detection.

All event data is stored in Elasticsearch, and can be queried and visualized in Kibana, the VIOS UI, and through the agent.

Incidents

In addition to instantaneous events, the profile detects violations in each enhanced frame and tracks them over time. When a violation persists beyond its configured threshold, it is promoted to a confirmed incident. The following incident types are generated:

• Proximity Violation: Objects come too close to each other (for example, a person near moving equipment).
• Restricted Area Violation: Objects enter a prohibited zone (ROI).
• Confined Area Violation: Objects leave a designated safe zone.

Incident timing (threshold and expiration window) is configurable per type. For the violation-to-incident framework, configuration parameters, and enhanced-frame data, see Incident Generation.

Use Cases

ROI events, tripwire events, and the proximity, restricted-area, and confined-area incidents provide the foundation for safety monitoring and operational intelligence in warehouse environments. With the agent, operators can query this data in natural language—surfacing ROI occupancy counts, tripwire crossings, recent violations, and incident histories—on top of the real-time situational awareness the 2D pipeline provides.

Latency Measurements

Customization

The Blueprint supports several levels of customization:

• Data Level: Add, remove, or replace cameras while maintaining the existing workflow
• Model Level: Fine-tune perception models or integrate different LLM models to better suit your use cases
• Application Level: Build new microservices or applications using the provided APIs and components
• Microservice Level: Modify existing microservices from source code to extend functionality

Adding New Cameras or Custom Datasets

Step 1: Prepare video sources

Choose one of the following options to provide video sources for the 2D profile with agents:

1. Using recorded videos

   Use this option to run the VSS with local video files instead of live camera feeds. Place your videos in the videos directory under the path configured by VSS_DATA_DIR in $VSS_APPS_DIR/industry-profiles/warehouse-operations/overrides.env.

2. Using live RTSP streams

   Use this option when you want to use live RTSP endpoints. Define each camera in $VSS_APPS_DIR/industry-profiles/warehouse-operations/camera_configs/camera_info.json so the VSS Configurator can read the camera list and generate the required configuration. The following steps apply when using live RTSP streams.

   (i) Create a Sensor Info File (camera_info.json):

   {
     "sensors": [
       {
         "camera_name": "camera-01",
         "rtsp_url": "rtsp://<IP_address:port>/stream1"
       },
       {
         "camera_name": "camera-02",
         "rtsp_url": "rtsp://<IP_address:port>/stream1"
       }
     ]
   }

   The camera_name value should be unique for each camera, and rtsp_url should point to the camera stream.

   Required fields: camera_name, rtsp_url

(ii) Configure VSS Configurator environment variables: Set the following in $VSS_APPS_DIR/industry-profiles/warehouse-operations/overrides.env, which is loaded after warehouse .env and takes precedence:

SENSOR_INFO_SOURCE=file
SENSOR_FILE_PATH=<path_to_camera_info.json>

Step 2: Configure number of streams

Configuring Number of Streams

Number of streams to be processed can be configured in two ways:

1. Static Configuration: Set the NUM_STREAMS environment variable in warehouse overrides.env to specify the desired number of streams for video dataset or RTSP streams.

   NUM_STREAMS=4

   The configured number of streams should be less than or equal to the maximum streams supported by your hardware profile and deployment mode. VSS Configurator can be used to automatically cap the stream count using the formula: final_stream_count = min(NUM_STREAMS, max_streams_supported). When SENSOR_INFO_SOURCE=nvstreamer, the same NUM_STREAMS value is the expected camera count the Sensor Configuration Manager waits for before VST registration. For more details, refer to the How to Count Files Dynamically (Prerequisites) section and NVStreamer Settings in VSS Configurator Documentation.

2. Dynamic Configuration: Use the VSS Configurator’s prerequisite operations to automatically count the number of video files in the recorded videos directory and use that count for configuration updates.

Example: Automatically determine stream count from video files in the dataset directory:

# In blueprint_config.yml
commons:
  # Step 1: Count video files BEFORE variable processing
  prerequisites:
    2d:
      - operation_type: "file_management"
        target_directories:
          - "${VSS_DATA_DIR}/videos/${SAMPLE_VIDEO_DATASET}"
        file_management:
          action: "file_count"
          parameters:
            pattern: "*.mp4"
          output_variable: "available_video_count"  # Stores count (e.g., 6)
  # Step 2: Use the count to compute final stream count
  variables:
    2d:
      # Cap to minimum of: available videos, GPU limit
      - final_stream_count: "min(${available_video_count}, ${max_streams_supported})"
  # Step 3: Use computed variable in config file updates
  file_operations:
    2d:
      - operation_type: "text_config_update"
        target_file: "${DS_CONFIG_DIR}/ds-main-config.txt"
        updates:
          max-batch-size: ${final_stream_count}
          batch-size: ${final_stream_count}

How it works: If your dataset directory has 6 video files and the GPU supports a maximum of 4 streams, the configurator computes: final_stream_count = min(6, 4) = 4. For more details, refer to the How to Count Files Dynamically (Prerequisites) section in VSS Configurator Documentation.

When using a custom recorded-video dataset, NUM_STREAMS depends on the number of video files available in that dataset. Make sure NUM_STREAMS is less than or equal to the number of videos you want to process.

To use a custom dataset:

• For a custom synthetic dataset, use a dataset name that ends in -synthetic (for example, my-warehouse-synthetic). Custom real datasets do not have a suffix requirement.

• Set SAMPLE_VIDEO_DATASET in $VSS_APPS_DIR/industry-profiles/warehouse-operations/overrides.env to the name of your dataset.

• Place the video files in $VSS_DATA_DIR/videos/$SAMPLE_VIDEO_DATASET/.

• Add the image metadata and floor map files under $VSS_APPS_DIR/industry-profiles/warehouse-operations/warehouse-2d-app/calibration/sample-data/${SAMPLE_VIDEO_DATASET}. If you already have a calibration, add calibration.json to the same directory. Otherwise, generate it in Step 3. You can refer to sample datasets provided in the repository for the expected file structure and sample contents.

Step 3: Choose a calibration option

2D with Agents supports three calibration states — pick one:

• Cartesian ("calibrationType": "cartesian"): full image-to-global calibration, generated using Auto Calibration’s AMC method. ROI/tripwire authoring is fully supported. Follow Cartesian coordinate calibration below.
• Image coordinates, with a calibration file ("calibrationType": "image"): a pixel-coordinate calibration file exported from Auto Calibration workflow, without an image-to-global transform. Auto Calibration supports ROI/tripwire authoring and image-mode JSON export. Follow Image coordinate calibration below.
• Running 2D without calibration: skip calibration entirely. Detection/ tracking still runs in image (pixel) coordinates; ROI/tripwire events are unavailable. See below.

Auto Calibration setup

Auto Calibration supports two types of calibrations in 2D profile with Agents: image-coordinate calibration and Cartesian calibration.

Before deploying the stack and generating a calibration, create the projects directory:

mkdir -p ${VSS_APPS_DIR}/services/auto-calibration/projects

Change the ownership of the Auto Calibration data and projects directories to UID 1000 and GID 1000:

sudo chown -R 1000:1000 ${VSS_DATA_DIR}/auto-calib
sudo chown -R 1000:1000 ${VSS_APPS_DIR}/services/auto-calibration/projects

Configure industry-profiles/warehouse-operations/overrides.env using the Auto-Calibration tab under Deployment Selection, then follow Deploy the Blueprint to start the Auto-Calibration profile.

Follow the Auto Calibration workflow steps using the warehouse dataset and VST stack. The workflow supports both calibration types; choose the appropriate export for your use case.

Image coordinate calibration

Use Export Image-Mode JSON to export ROIs and tripwires in pixel coordinates with "calibrationType": "image".

For image-coordinate calibration ("calibrationType": "image") only, add ROIs or tripwires to the sensors that require them. If only one sensor needs an ROI or tripwire, annotate that sensor and proceed with the export. Sensors that need neither ROIs nor tripwires do not need to be included in the image-coordinate calibration.json. This omission applies only to image-coordinate calibration, not Cartesian calibration.

Cartesian coordinate calibration

Export a Cartesian calibration with "calibrationType": "cartesian" to map camera image coordinates to a global coordinate system.

Save and apply the calibration

Bring the Docker Compose stack down after exporting the calibration. Save the exported file as $VSS_APPS_DIR/industry-profiles/warehouse-operations/warehouse-2d-app/calibration/sample-data/${SAMPLE_VIDEO_DATASET}/calibration.json, then proceed to Step 4 to redeploy with the new calibration.

Running 2D without calibration

Applies only when there’s no calibration.json file at all — not for either calibration type above. Detection/tracking still runs directly on the camera stream in image (pixel) coordinates; only ROI/tripwire events in behavior-analytics are unavailable. Calibration is enabled by default; to skip it entirely:

• Remove import-calibration-output-container-2d from whichever profile variable you’re deploying with — COMPOSE_PROFILES_WH_2D, COMPOSE_PROFILES_WH_KAFKA_2D, COMPOSE_PROFILES_WH_REDIS_2D, COMPOSE_PROFILES_PLAYBACK_KAFKA_2D, COMPOSE_PROFILES_PLAYBACK_REDIS_2D — edit the value directly in overrides.env.

• Remove --calibration /resources/calibration.json from the vss-behavior-analytics service’s command: in warehouse-2d-app/warehouse-2d-app.yml. This is a static line in the compose file, not an env var, so it needs a direct edit (or a compose override file layered on top).

• Also remove the vss-behavior-analytics-2d service’s calibration.json bind mount in the same file — the volumes: line ending .../calibration/sample-data/$SAMPLE_VIDEO_DATASET/calibration.json:/resources/calibration.json. This is independent of the profile and command changes above. If a custom dataset has no calibration.json on the host, Compose creates an empty directory at that path to satisfy the mount, which then blocks placing a real calibration.json there later — leave the mount in only if the file already exists.

• The bp-configurator-2d service’s /opt/data mount (.../calibration/sample-data/$SAMPLE_VIDEO_DATASET:/opt/data) is unused in 2D mode — good to remove.

Step 4: Redeploy the stack to apply the new calibration

Before redeploying, replace the Auto-Calibration settings in industry-profiles/warehouse-operations/overrides.env with the settings from the 2D + Agents tab under Deployment Selection. Then follow Deploy the Blueprint to start the 2D with Agents profile with the new calibration.

docker ps -a

Model Customization

Perception Model Fine-tuning

The Blueprint uses RT-DETR (Real-Time Detection Transformer) as its primary perception model. RT-DETR features a ResNet-50 backbone and is pretrained on warehouse scene datasets for precise 2D object detection in industrial environments. For details, see: 2D Single Camera Detection and Tracking.

For fine-tuning the RT-DETR model via TAO Toolkit on custom datasets, refer to: RT-DETR (TAO fine-tuning).

LLM Integration

Integrate different LLM services or locally deployed models in the agent.

Application Customization

The Blueprint uses a modular microservices architecture with the following communication channels:

• Kafka message broker
• Elasticsearch database
• REST APIs

Users can build their own microservices by consuming data from the above channels.

For complete API documentation, see API Reference Page.

A deployed 2D Vision AI with Agents Profile can also be driven from a coding agent (Claude Code or Codex) using Agent Skills. Skills such as vss-build-vision-ai, vss-deploy-detection-tracking-2d, vss-query-analytics, vss-setup-behavior-analytics, and vss-manage-video-io-storage cover deployment, operation, and analytics queries against this profile through natural language.

Available Service Ports

The following ports are used during deployment, and users can leverage them for any potential integration:

Analytics Microservices Customization (Advanced)

For detailed information about customizing specific analytics microservices, refer to:

• Behavior Analytics
• VSS Video Analytics API

Alert Microservices Customization (Advanced)

For detailed information about customizing specific alert microservices, refer to:

• Alerting Service — alert use cases (Near Miss, Load Quality, Pathway Obstruction, Spillover, PPE), Alert Bridge output formats, and prompt-tuning workflow.
• RTVI VLM for Alert Verification and Real-Time Alerts — the two operating modes that drive RTVI VLM. Alert Verification re-evaluates upstream candidate events (nv.Incident / nv.Behavior) with per-alert-type prompts in alert_type_config.json; Real-Time Alerts run continuous always-on rules from realtime-config.yml against live RTSP streams.
• Alert Bridge — supported VLM response formats (Cosmos Reason, JSON, custom parsers) and verdict vocabulary.

Hardware Config Customization

The 2D Warehouse Blueprint requires several configuration files to be properly tuned based on your GPU hardware and deployment requirements. When changing hardware (e.g., switching from H100 to L4 GPU) or adjusting the number of video streams, multiple configuration files must be updated to ensure optimal performance and prevent GPU overload.

Configuration Files Requiring Hardware-Based Updates

The following table lists the configuration files that typically require updates when hardware changes:

There are two approaches to customize these configuration files:

Approach 1: Manual Configuration

Manually update all required configuration files before deploying the blueprint. This is time consuming, error prone and often not suitable for production deployments.

# Update DeepStream main config
vi <PATH_TO_DS_CONFIG_DIR>/ds-main-config.txt
# Set: num-source-bins=0, max-batch-size=<stream_count>, batch-size=<stream_count>
# Set msg-broker-config to ds-kafka-config.txt or ds-redis-config.txt depending on STREAM_TYPE
# Update DeepStream message broker config (Kafka or Redis variant, selected by STREAM_TYPE)
vi <PATH_TO_DS_CONFIG_DIR>/ds-kafka-config.txt    # for STREAM_TYPE=kafka
vi <PATH_TO_DS_CONFIG_DIR>/ds-redis-config.txt    # for STREAM_TYPE=redis
# Update NvStreamer config
vi <PATH_TO_NVSTREAMER_CONFIG_DIR>/vst-config.json
# Set: "nv_streamer_sync_file_count": <stream_count>
# Update VIOS (VST) config
vi <PATH_TO_VST_CONFIG_DIR>/vst_config.json
# Set: "onvif.max_devices_supported": <max_streams>, "data.always_recording": true,
#      "notifications.use_message_broker_consumer": "<STREAM_TYPE>"

Approach 2: Automatic Config Management using VSS Configurator

The VSS Configurator provides a declarative approach to automatically update all required configuration files based on your hardware profile and deployment mode. This is the recommended approach for production deployments and simplifies the configuration management process.

For how the configurator updates configurations in this blueprint, see VSS Configurator for the Warehouse Blueprint. The guide describes configuration inputs, affected files, stream ceilings, and hardware-specific overrides.

To enable the VSS Configurator, in VSS Configurator’s environment variables you must set:

ENABLE_PROFILE_CONFIGURATOR=true

The VSS Configurator is enabled by default (ENABLE_PROFILE_CONFIGURATOR=true). When enabled, it runs before the Blueprint Deployment starts and adjusts configuration files based on the hardware profile and deployment mode as defined in the HARDWARE_PROFILE and MODE environment variables.

The VSS Configurator provides a comprehensive set of features for automated profile configuration management:

Execution Order: Environment Variable Validation → Prerequisite Operations → Variable Computations → Configuration File Updates

For detailed information on how to create custom hardware profiles and advanced configuration options, refer to the Profile Configuration Manager section in VSS Configurator Documentation.