Agent Profiles

The VSS Agent connects to a matching backend blueprint. Each profile configures the agent for a specific use case with tailored prompts, tools, and report templates.

Blueprint Profiles

Two production blueprint profiles are available. The agent configuration connects to the corresponding backend analytics pipeline.

Warehouse Blueprint

Industrial facility monitoring for safety incidents.

• Config: deploy/docker/industry-profiles/warehouse-operations/vss-agent/configs/config.yml
• Warehouse Blueprint Documentation

Smart Cities Blueprint

Traffic and urban monitoring at intersections.

• Config: deploy/docker/industry-profiles/smartcities/vss-agent/configs/config.yml
• Smart Cities Blueprint Documentation

Developer Profiles

For development and testing without a full blueprint deployment, four developer profiles are available. These use OOBE (Out-of-Box Experience) mode for standalone operation.

dev-profile-base

Basic video upload and analysis without VA-MCP or incident database.

• Config: deploy/docker/developer-profiles/dev-profile-base/vss-agent/configs/config.yml
• Use Case: Upload videos, generate reports, answer questions about video content

Capabilities:

• Upload videos via /api/v1/videos endpoint
• Generate video analysis reports with timestamped observations
• Answer follow-up questions about video content or report content
• Retrieve snapshots and video clips

Tools Available:

• video_understanding: Analyze video content using Cosmos VLM
• vst_video_clip: Get video playback URLs
• vst_snapshot: Get snapshot images at specific timestamps
• vst_video_list: List available video streams

dev-profile-lvs

Video Summarization and stream analysis with interactive Human-in-the-Loop (HITL) prompts.

• Config: deploy/docker/developer-profiles/dev-profile-lvs/vss-agent/configs/config.yml
• Use Case: Summarize uploaded videos, generate reports, and configure stream captioning.

Capabilities:

All dev-profile-base capabilities, plus:

• Video Summarization for uploaded videos.
• Report generation for one or more uploaded videos.
• Stream caption generation with configurable scenario, events, and objects of interest (experimental).
• Stream summarization over a requested timestamp range (experimental).
• Report generation for a live stream over a requested time range (experimental).
• RTVI-VLM as the VLM path used by the Video Summarization profile.

Additional Tools:

• lvs_video_understanding: Summarize uploaded videos with configurable Video Summarization parameters.
• lvs_config_media: Start caption generation for a live stream after collecting HITL parameters (experimental).
• lvs_stream_understanding: Summarize a live stream over a requested timestamp range (experimental).
• video_report_gen: Generate downloadable reports for uploaded videos or live streams.

HITL Prompts:

When using the Video Summarization profile, the agent prompts for:

1. Scenario: What type of monitoring (e.g., “warehouse monitoring”, “traffic monitoring”)
2. Events: Comma-separated list of events to detect (e.g., “accident, forklift stuck, person entering restricted area”)
3. Objects (optional): Objects to focus on (e.g., “forklifts, pallets, workers”)

Routing Examples:

Requirements:

• Video Summarization backend service.
• RTVI-VLM service.
• Elasticsearch for summaries, captions, and events.
• Kafka and Logstash for the live-stream caption path.

Stream Caption Limitations:

• Stream summarization and report generation require captions in Elasticsearch for the requested timestamp range.
• If multiple agent sessions or instances connect to the same backend, the caption generation prompt is overwritten by the latest query.
• The agent cannot see caption prompts set by other agent instances.

dev-profile-search

Video embedding search with semantic queries.

• Config: deploy/docker/developer-profiles/dev-profile-search/vss-agent/configs/config.yml
• Use Case: Semantic search across video content using Cosmos Embed

Capabilities:

All dev-profile-base capabilities, plus:

• Semantic video search using Cosmos Embed embeddings
• Real-time video ingest with embedding generation

Additional Tools:

• search: Semantic search across indexed video content
• embed_search: Low-level embedding search against Elasticsearch

Requirements:

• Cosmos Embed NIM endpoint for embedding generation
• Elasticsearch for storing and querying embeddings
• RTVI service for real-time video ingest

dev-profile-alerts

Real-time alert generation and verification with two deployment modes:

• Alert verification (MODE=2d_cv): RT-CV and Behavior Analytics generate candidate incidents, and RT-VLM verifies them.

• Real-time alerts (MODE=2d_vlm): RT-VLM continuously evaluates video chunks against an alert rule.

• Config: deploy/docker/developer-profiles/dev-profile-alerts/vss-agent/configs/config.yml

• Use Case: Verify CV-generated incidents or generate alerts directly from live video with a VLM.

Requirements:

• Alert Bridge and the Video Analytics API.
• Elasticsearch and Kafka for incident storage and delivery.
• RT-CV plus Behavior Analytics for 2d_cv, or RT-VLM for 2d_vlm.
• Hardware-specific GPU counts and placement from Development Profile GPU Requirements.

Developer Profile Comparison

Services

When deployed, the following services are available:

For deployment instructions, see the respective blueprint documentation linked above.