Real-Time Alert Workflow#

The Real-Time Alert Workflow monitors live video streams and generates alerts when the VLM detects anomalies or specified events.

Use Cases

Traffic collision detection
Unusual behavior detection
Equipment malfunction identification
Safety hazard detection

Estimated Deployment Time: 15-20 minutes

The following diagram illustrates the real-time alert workflow architecture:

Key Features of the Real-Time Alert Agent:

Continuous frame sampling and VLM-based anomaly detection from video streams using the RTVI VLM microservice
Natural language queries for detected alerts
Configurable alert prompts and invocation settings for custom detection scenarios
Report Generation

What’s being deployed#

VSS Agent: Agent service that uses a configured LLM endpoint to route requests and orchestrate tool calls to VSS microservices and model endpoints (LLM/VLM NIMs) to answer questions and generate outputs
VSS Agent UI: Web UI with chat, video upload, and different views
RTVI VLM: Real-Time VLM Microservice for vision-language model inference used by real-time alerting
Video IO & Storage (VIOS): Video ingestion, recording, and playback services used by the agent for video access and management
NVStreamer: Video streaming service for video playback
Nemotron LLM (NIM): LLM inference service used for reasoning, tool selection, and response generation
ELK: Elasticsearch, Logstash, and Kibana stack for log storage and analysis
Phoenix: Observability and telemetry service for agent workflow monitoring

Prerequisites#

Before you begin, ensure all of the prerequisites are met. See Prerequisites for more details.

Deploy#

Note

For instructions on downloading sample data and the deployment package, see Download Sample Data and Deployment Package in the Quickstart guide.

With Agent Skills#

Step 1: Deploy the Agent#

Use VSS Agent Skills from an agent such as Claude Code, Codex, or NemoClaw to deploy and manage the real-time alerts workflow. First, install the deploy skill as described in Agent Skills and make it accessible to your agent. The host must meet the same deployment requirements listed above (supported GPU/hardware for that profile) and must meet the Prerequisites.

The deploy skill will choose the <platform> and <mode> to match your system, as detailed in Development Profile GPU Requirements. Refer to the requirements table for valid platform and mode combinations compatible with your hardware.

Note

Optional: You can enable notifications by setting webhook.openclaw.enabled to true in the deploy/docker/developer-profiles/dev-profile-alerts/vlm-as-verifier/configs/config.yml config.

Deploy prompt structure#

Deploy the VSS alerts profile in real-time mode.

Step 2: Add a Video Stream#

To manage real-time alerts from your agent, install the vss-manage-alerts skill as described in Agent Skills. Your agent can then start, stop, and inspect alerts through VSS with natural-language prompts, without requiring manual interaction with the UI.

Add a stream with the vss-manage-video-io-storage skill.

Add an RTSP stream to VIOS#

Add RTSP stream <rtsp stream URL> to VIOS with the sensor name warehouse_sample.

Note

If you do not have an RTSP stream, you can use NVStreamer at http://<HOST_IP>:31000 to upload a video file and create an RTSP stream.

What the agent does:

Loads the vss-manage-video-io-storage skill and verifies that VIOS is reachable.
Extracts the RTSP URL and requested sensor name from the prompt.
Registers the stream with VIOS by calling POST /vst/api/v1/sensor/add with sensorUrl and name.
Confirms that VIOS lists the new sensor.
Returns the sensor ID or configured name to use in alert prompts.

Step 3: Manage Alert Rules#

Ask your agent to manage alert rules in natural language. The vss-manage-alerts skill routes these requests to the alert-specific workflow, checks that the alerts profile is running in VLM real-time mode, resolves the sensor name through VIOS/VST, and calls Alert Bridge to create, list, or delete persistent real-time rules. To change an existing rule, the agent uses the same rule-management workflow to replace the old rule with a new one.

For rule creation and replacement requests, the skill extracts the sensor and detection condition from your prompt, resolves the sensor ID and RTSP stream URL, derives a short alert_type tag, and submits the rule to Alert Bridge. For alert lookups, the skill routes the request through the agent incident-query workflow so that you can review notifications that have fired.

Add an Alert Rule#

Add an alert rule#

Detect anyone entering without personal protective equipment on sensor warehouse_sample.
Send me alerts for fallen boxes in camera warehouse_sample.
Watch warehouse_sample for anyone without a safety vest.

What the agent does:

Parses the sensor name and detection condition from the prompt.
Resolves the sensor ID and RTSP stream URL through VIOS/VST.
Derives an alert_type tag and submits a persistent real-time rule to Alert Bridge.

List Active Alert Rules#

List active alert rules#

What alert rules are currently active?

What the agent does:

Fetches active rules from Alert Bridge.
Resolves stream URLs back to readable sensor names when needed.
Applies any sensor or alert-type filter from your prompt and returns the current rule IDs.

Show Alert Notifications#

Show alert notifications#

List any alerts for alert ID <alert ID>.

What the agent does:

Routes the request to the incident-query workflow.
Posts the natural-language query to the agent endpoint, which invokes video_analytics_mcp.get_incidents against the stored alert incidents.
Summarizes matching notifications with status, sensor, timestamp, and available evidence.

Replace an Alert Rule#

Replace an alert rule#

Replace alert rule ID <alert ID> with a rule that watches warehouse_sample for fallen boxes.

What the agent does:

Uses the rule ID to identify the rule that needs to change.
Resolves the requested sensor and detection condition for the replacement rule.
Creates the replacement rule through Alert Bridge, then asks for confirmation before deleting the superseded rule.

Delete an Alert Rule#

Delete an alert rule#

Delete alert rule ID <alert ID>.

What the agent does:

Uses the rule ID to identify the rule to stop.
Asks you to confirm before stopping the rule.
After you confirm, calls Alert Bridge to delete the rule and reports whether it was stopped or was already inactive.

Note

The Alerts Microservice supports receiving alerts on the OpenClaw UI. Ensure that notifications are enabled as noted above.

To use this feature, send the following prompt after creating an alert rule:

Start the alert-notify relay (dashboard backend only). The relay runs INSIDE the openshell sandbox, so use these env values exactly — do not ask me, do not substitute:

NOTIFY_BACKENDS=dashboard

VST_ENDPOINT=host.openshell.internal:30888

VST_PUBLIC_URL_BASE=https://7777-xbrxpi7ia.brevlab.com

OPENCLAW_GATEWAY_URL=ws://127.0.0.1:18789

Infer OPENCLAW_GATEWAY_URL based on the deployment.

Be sure to replace the VST_PUBLIC_URL_BASE listed above with your own VST public URL.

Service Endpoints#

Once deployed, the following services are available:

Service Endpoints#
Service	URL
VSS UI	`http://<HOST_IP>:7777`
NVStreamer UI	`http://<HOST_IP>:31000/#/dashboard`
VST UI	`http://<HOST_IP>:30888/vst/#/dashboard`
Phoenix UI	`http://<HOST_IP>:7777/phoenix/projects`

Known Issues#

It is recommended to clear chat and refresh the VSS UI page after deployment to clear any data from a previous deployment.
VSS-RTVI-VLM logs may repeatedly show a “Generated text is empty” warning and result in incomplete/unreliable alert verdicts
VLM verdict verification can fail in the UI even when VLM requests are processed successfully. Alert Bridge response parsing accepts only raw verdicts such as A/B or Yes/No before serializing them to confirmed or rejected. As a result, a semantically valid rejected verdict from Cosmos Reason can fail schema validation and be persisted as verification-failed.
Report generation may produce inaccurate results. As a potential workaround, remove the use_base64: true line under video_understanding_iso in developer-profiles/dev-profile-alerts/vss-agent/configs/config.yml.
The VST UI is externally accessible on both port 30000 and 30888 because both ports are exposed on the host network. For security hardening, consider using a firewall to allowlist only the required VST ingress port.

Real-Time Alert Workflow#

What’s being deployed#

Prerequisites#

Deploy#

With Agent Skills#

Step 1: Deploy the Agent#

Step 2: Add a Video Stream#

Step 3: Manage Alert Rules#

Add an Alert Rule#

List Active Alert Rules#

Show Alert Notifications#

Replace an Alert Rule#

Delete an Alert Rule#

Manually#

Step 1: Deploy the Agent#

Step 2: Add a video stream#

Step 3: Start a real-time alert#

Troubleshooting#

Step 4: Generate a Report for the Alert#

Step 5: Teardown the Agent#

Service Endpoints#

Known Issues#