Real-Time Alerts

The Real-Time Alert Workflow monitors live video streams and generates alerts when the VLM detects anomalies or specified events.

Use Cases

• Traffic collision detection
• Unusual behavior detection
• Equipment malfunction identification
• Safety hazard detection

Estimated Deployment Time: 15-20 minutes

The following diagram illustrates the real-time alert workflow architecture:

Key Features of the Real-Time Alert Agent:

• Continuous frame sampling and VLM-based anomaly detection from video streams using the RTVI VLM microservice
• Natural language queries for detected alerts
• Configurable alert prompts and invocation settings for custom detection scenarios
• Report Generation

What’s being deployed

• VSS Agent: Agent service that uses a configured LLM endpoint to route requests and orchestrate tool calls to VSS microservices and model endpoints (LLM/VLM NIMs) to answer questions and generate outputs
• VSS Agent UI: Web UI with chat, video upload, and different views
• RTVI VLM: Real-Time VLM Microservice for vision-language model inference used by real-time alerting
• Video IO & Storage (VIOS): Video ingestion, recording, and playback services used by the agent for video access and management
• NVStreamer: Video streaming service for video playback
• Nemotron LLM (NIM): LLM inference service used for reasoning, tool selection, and response generation
• ELK: Elasticsearch, Logstash, and Kibana stack for log storage and analysis
• Phoenix: Observability and telemetry service for agent workflow monitoring

This list describes the stock profile used by the manual Compose deployment. Build Vision AI customizes that profile for the requested capabilities. With the default NemoClaw harness, or with no harness, it does not deploy the in-stack VSS Agent or Phoenix. It also omits the Agent’s local LLM unless another selected service needs it. Ask for the in-stack VSS Agent if you plan to use the Agent REST API or the manual Agent UI workflow below.

Prerequisites

Before you begin, ensure all of the prerequisites are met. See Prerequisites for more details.

Deploy

With Agent Skills

Step 1: Deploy with Agent Skills

Use VSS Agent Skills from an agent such as Claude Code, Codex, or NemoClaw to deploy and manage the real-time alerts workflow. First, install the vss-build-vision-ai skill as described in Agent Skills and make it accessible to your agent. The host must meet the same deployment requirements listed above (supported GPU/hardware for that profile) and must meet the Prerequisites.

The vss-build-vision-ai skill composes the stack from the capabilities you name and selects the <platform> and <mode> to match your system, as detailed in Development Profile GPU Requirements. Refer to the requirements table for valid platform and mode combinations compatible with your hardware.

Deploy prompt structure

Deploy the VSS alerts profile in real-time mode.

Name an image tag in the prompt to run a release’s images instead of the containers.env default. See Container image tag.

Deploy the VSS alerts profile in real-time mode with image tag 3.3.0.

The vss-build-vision-ai skill uses Cosmos3 Nano Reasoner BF16 as the default VLM. You can ask the agent which VLM models are available during the deployment process, or request a different model directly in your deployment prompt:

Which VLM models are available for the alerts profile?
Deploy the VSS alerts profile in real-time mode with the Cosmos3 Super Reasoner FP8 VLM.

The agent sets VLM_NAME and RTVI_VLM_MODEL_PATH in the generated _builds/<name>/override.env to the model you request. The same options and hardware constraints listed in Available VLM Models apply.

If you selected NemoClaw, follow the Quickstart’s chat connection steps and continue the operational prompts in that chat.

Step 2: Add a Video Stream

To manage real-time alerts from your agent, install the vss-manage-alerts skill as described in Agent Skills. Your agent can then start, stop, and inspect alerts through VSS with natural-language prompts, without requiring manual interaction with the UI.

Add a stream with the vss-manage-video-io-storage skill.

Add an RTSP stream to VIOS

Add RTSP stream <rtsp stream URL> to VIOS with the sensor name warehouse_sample.

What the agent does:

1. Loads the vss-manage-video-io-storage skill and verifies that VIOS is reachable.
2. Extracts the RTSP URL and requested sensor name from the prompt.
3. Registers the stream with VIOS by calling POST /vst/api/v1/sensor/add with sensorUrl and name.
4. Confirms that VIOS lists the new sensor.
5. Returns the sensor ID or configured name to use in alert prompts.

Step 3: Manage Alert Rules

Ask your agent to manage alert rules in natural language. The vss-manage-alerts skill routes these requests to the alert-specific workflow, checks that the alerts profile is running in VLM real-time mode, resolves the sensor name through VIOS/VST, and calls Alert Bridge to create, list, or delete persistent real-time rules. To change an existing rule, the agent uses the same rule-management workflow to replace the old rule with a new one.

For rule creation and replacement requests, the skill extracts the sensor and detection condition from your prompt, resolves the sensor ID and RTSP stream URL, derives a short alert_type tag, and submits the rule to Alert Bridge. For alert lookups, the skill routes the request through the agent incident-query workflow so that you can review notifications that have fired.

For direct API instructions, see Create a Rule with the REST API in the manual workflow.

Add an Alert Rule

Detect anyone entering without personal protective equipment on sensor warehouse_sample.
Send me alerts for fallen boxes in camera warehouse_sample.
Watch warehouse_sample for anyone without a safety vest.

What the agent does:

1. Parses the sensor name and detection condition from the prompt.
2. Resolves the sensor ID and RTSP stream URL through VIOS/VST.
3. Derives an alert_type tag and submits a persistent real-time rule to Alert Bridge.

List Active Alert Rules

What alert rules are currently active?

What the agent does:

1. Fetches active rules from Alert Bridge.
2. Resolves stream URLs back to readable sensor names when needed.
3. Applies any sensor or alert-type filter from your prompt and returns the current rule IDs.

Show Alert Notifications

List any alerts for alert ID <alert ID>.

What the agent does:

1. Routes the request to the incident-query workflow.
2. Queries the stored incidents from Alert Bridge through GET /api/v1/realtime/incidents.
3. Summarizes matching notifications with status, sensor, timestamp, and available evidence.

Replace an Alert Rule

Replace alert rule ID <alert ID> with a rule that watches warehouse_sample for fallen boxes.

What the agent does:

1. Uses the rule ID to identify the rule that needs to change.
2. Resolves the requested sensor and detection condition for the replacement rule.
3. Creates the replacement rule through Alert Bridge, then asks for confirmation before deleting the superseded rule.

Delete an Alert Rule

Delete alert rule ID <alert ID>.

What the agent does:

1. Uses the rule ID to identify the rule to stop.
2. Asks you to confirm before stopping the rule.
3. After you confirm, calls Alert Bridge to delete the rule and reports whether it was stopped or was already inactive.

Start the alert-notify relay (dashboard backend only).
NOTIFY_BACKENDS=dashboard
VST_ENDPOINT=<vst-host>:<vst-port>
VST_PUBLIC_URL_BASE=https://<your-public-vst-or-tunnel-base>
OPENCLAW_GATEWAY_URL=ws://127.0.0.1:18789
Infer OPENCLAW_GATEWAY_URL and VST_PUBLIC_URL_BASE based on the deployment.

Manually

Step 1: Deploy with Docker Compose

# Set NGC CLI API key
export NGC_CLI_API_KEY='your_ngc_api_key'
# Authenticate Docker to pull VSS release images, NIM, and other NGC images.
printf '%s' "$NGC_CLI_API_KEY" | \
  docker login nvcr.io --username '$oauthtoken' --password-stdin

See Configuration for common variables, valid values, defaults, and env-file precedence. The complete profile-specific template is deploy/docker/developer-profiles/dev-profile-alerts/overrides.env.

From the repository checkout root, enter the Compose deployment directory:

cd deploy/docker

overrides.env is the checked-in template. Copy it to any filename matching *-overrides.env before selecting a layout. Files matching that pattern are ignored by Git so local settings and credentials are not committed. The commands below use user-overrides.env as a concrete example; if you choose another name, use it consistently in every --env-file argument. Repeat the copy only when you want to discard prior layout changes:

PROFILE=alerts
cp "developer-profiles/dev-profile-${PROFILE}/overrides.env" \
  "developer-profiles/dev-profile-${PROFILE}/user-overrides.env"

Before starting Compose, complete the host settings and select one hardware layout below in deploy/docker/developer-profiles/dev-profile-alerts/user-overrides.env.

Environment-variable overrides: In deploy/docker/developer-profiles/dev-profile-alerts/user-overrides.env, update existing values in place; add a variable only when it is absent. COMPOSE_PROFILES is derived from values above it, so appending duplicate model-mode or model-slug variables can select the wrong services.

Direct Docker Compose Overrides

Apply all of the following settings in the user-overrides.env file created above:

MODE=2d_vlm
ALERT_AGENT_ALWAYS_ON=true
COMPOSE_PROFILES=${COMPOSE_PROFILES_VLM}
RTVI_VLM_MESSAGE_BUS=kafka
RTVI_VLM_KAFKA_ENABLED=true
NEXT_PUBLIC_ALERTS_TAB_MANAGE_ALERTS_SUB_TAB_ENABLE_REALTIME_ALERTS=true
NEXT_PUBLIC_ALERTS_TAB_MANAGE_ALERTS_SUB_TAB_ENABLE_CV_ALERTS_VERIFICATION=false

In RT-VLM 26.08.2, MESSAGE_BUS is the startup source of truth for generated-message publishing. The packaged Compose variable RTVI_VLM_MESSAGE_BUS=kafka sets it explicitly. RTVI_VLM_KAFKA_ENABLED=true is retained only as the legacy compatibility field while Compose still forwards it. Later --env-file arguments take precedence over earlier arguments; therefore, changes to overrides.env made after it has been copied do not affect a command that reads user-overrides.env. VSS_CONTAINER_REGISTRY (default nvcr.io/nvidia/vss-core) and VSS_CONTAINER_TAG (default 3.3.0) are defined in deploy/docker/containers.env.

Host-path and service-address overrides (all hardware profiles):

• VSS_APPS_DIR: Absolute path to this repository’s deploy/docker directory.
• VSS_DATA_DIR: Absolute writable host directory for downloaded models, videos, and service data.
• HOST_IP: IP address or hostname that clients use to reach services on this VSS host. Set it to an address reachable by the intended clients.
• EXTERNAL_IP: Address VSS puts in externally reachable service URLs. It defaults to ${HOST_IP}; on a cloud host or when clients connect from outside the host network, set it to the instance’s public address or external hostname.

Available VLM Models

Set these required placeholder values in the existing assignments:

VSS_APPS_DIR=/absolute/path/to/video-search-and-summarization/deploy/docker
VSS_DATA_DIR=/absolute/path/to/vss-apps-data
HOST_IP=<reachable-host-ip-or-name>

Prepare host data directories before starting Compose. Export VSS_DATA_DIR and VSS_APPS_DIR with the same absolute paths set above, then run this block. The guards stop before mkdir or chmod if either value is absent or is not absolute:

: "${VSS_DATA_DIR:?Export VSS_DATA_DIR with the value from user-overrides.env}"
: "${VSS_APPS_DIR:?Export VSS_APPS_DIR with the value from user-overrides.env}"
case "$VSS_DATA_DIR" in
  /*) ;;
  *) echo "VSS_DATA_DIR must be an absolute path" >&2; exit 1 ;;
esac
case "$VSS_APPS_DIR" in
  /*) ;;
  *) echo "VSS_APPS_DIR must be an absolute path" >&2; exit 1 ;;
esac
mkdir -p \
  "$VSS_DATA_DIR/data_log/analytics_cache" \
  "$VSS_DATA_DIR/data_log/calibration_toolkit" \
  "$VSS_DATA_DIR/data_log/elastic/data" \
  "$VSS_DATA_DIR/data_log/elastic/logs" \
  "$VSS_DATA_DIR/data_log/kafka" \
  "$VSS_DATA_DIR/data_log/nvstreamer/vst_data" \
  "$VSS_DATA_DIR/data_log/redis/data" \
  "$VSS_DATA_DIR/data_log/redis/log" \
  "$VSS_DATA_DIR/data_log/vss_video_analytics_api" \
  "$VSS_DATA_DIR/data_log/vst/clip_storage" \
  "$VSS_DATA_DIR/agent_eval/dataset" \
  "$VSS_DATA_DIR/agent_eval/results" \
  "$VSS_DATA_DIR/models" \
  "$VSS_DATA_DIR/videos/dev-profile-alerts"
mkdir -p "$VSS_APPS_DIR/engines/gdino" "$VSS_APPS_DIR/engines/rtdetr-its"
chmod -R a+rwx "$VSS_APPS_DIR/engines"
chmod -R a+rwx \
  "$VSS_DATA_DIR/data_log" \
  "$VSS_DATA_DIR/agent_eval" \
  "$VSS_DATA_DIR/models" \
  "$VSS_DATA_DIR/videos"

Default container image tag

containers.env uses VSS_CONTAINER_TAG as the default tag for all images. Export the tag selected for this documentation branch before choosing a hardware layout:

export VSS_CONTAINER_TAG="3.3.0"

Any service-specific image tag or tag suffix shown inside a hardware tab supersedes or augments this common default. Use the SBSA settings shown in the GB300 and DGX Spark tabs when deploying on those platforms.

HARDWARE_PROFILE=H100
MODE=2d_vlm
NEXT_PUBLIC_APP_SUBTITLE="Vision (Alerts - VLM)"
ALERT_AGENT_ALWAYS_ON=true
RTVI_VLLM_GPU_MEMORY_UTILIZATION=0.4
COMPOSE_PROFILES=${COMPOSE_PROFILES_VLM}

After replacing every placeholder in the selected layout, from deploy/docker validate the resolved configuration, then start the profile:

PROFILE=alerts
docker compose -f compose.yml \
  --env-file containers.env \
  --env-file "developer-profiles/dev-profile-${PROFILE}/.env" \
  --env-file "developer-profiles/dev-profile-${PROFILE}/user-overrides.env" \
  config --quiet
docker compose -f compose.yml \
  --env-file containers.env \
  --env-file "developer-profiles/dev-profile-${PROFILE}/.env" \
  --env-file "developer-profiles/dev-profile-${PROFILE}/user-overrides.env" \
  up --detach --pull always --force-recreate --build

On the first bring-up, Docker Compose pulls the required containers from the configured NGC registries and builds five support images locally: Elasticsearch, its initializer, the Kafka topic initializer, the broker health checker, and the alerts Kibana initializer. Those builds pull base images from Docker Hub and docker.elastic.co; the Kafka support-image builds also download jq from GitHub. Allow outbound access and disk space for both pulls and local builds. Depending on your network speed, this may take a few minutes.

This deployment uses the following defaults:

• LLM model: nvidia/nemotron-3.5-lightning-30b-a3b
• VLM model: cosmos-reason3 (served via RTVI VLM container)

Once the deployment is complete, check that all the containers are running and healthy:

docker ps

Once all the containers are running, you can access the Agent UI at http://<HOST_IP>:7777/.

Step 2: Add a video stream

Add an RTSP stream by clicking the + Add RTSP button on the Video Management tab in the agent UI. If you do not have an RTSP stream, you can use NVStreamer at http://<HOST_IP>:31000 to upload a video file and create an RTSP stream.

For this profile, use the warehouse_sample.mp4 stream. When prompting, ensure the sensor name matches exactly what you configured for the stream (for example, warehouse_sample).

Step 3: Start a real-time alert

Launch the Agent UI at http://<HOST_IP>:7777/.

Use the Global Chat Sidebar to interact with the system:

• Start an RTVI real-time alert for a stream by specifying the alert type.

  • Sample prompt: Start real-time alert for boxes dropped on sensor warehouse_sample

  

  

  

  

From the same Alerts tab, enable VLM Verified to view verified alerts for all active sensors.

Create a Rule with the REST API

For Docker Compose deployments, Alert Bridge listens on port 9080. Resolve the sensor UUID and main RTSP URL from VIOS, then create the rule through Alert Bridge—not directly through RT-VLM:

curl -sS -X POST "http://<HOST_IP>:9080/api/v1/realtime" \
  -H "Content-Type: application/json" \
  -d '{
    "live_stream_url": "<RTSP_URL>",
    "sensor_id": "<SENSOR_UUID>",
    "sensor_name": "warehouse_sample",
    "alert_type": "missing_ppe",
    "prompt": "Is anyone visible without required personal protective equipment?",
    "system_prompt": "Answer yes or no",
    "chunk_duration": 30,
    "chunk_overlap_duration": 5
  }'

A successful create returns HTTP 201 and a body like:

{
  "status": "success",
  "id": "496aebd1-16d0-4123-81cf-10603e047d02",
  "created_at": "2026-04-21T11:09:40.111515+00:00",
  "message": "Realtime alert rule created"
}

Keep the returned id; it identifies the rule for GET /api/v1/realtime/{alert_rule_id} and DELETE /api/v1/realtime/{alert_rule_id}. A real-time rule has no separate threshold, sensitivity, or cooldown field. Tune the detection condition in prompt and the evaluation cadence with the two chunk fields. The alert_agent.event_filters settings documented for the Alerts Microservice apply to upstream alert-verification event filtering, not to this real-time rule payload.

Troubleshooting

If alerts do not appear in the VSS UI after you create an alert rule, check the following item:

• View the RTVI VLM container logs with docker logs -f vss-rtvi-vlm and confirm that the alert rule was created and is being processed. If processing fails, confirm that the stream is valid and that the deployment has enough available GPU memory.

Step 4: Generate a Report for the Alert

You can use the chat interface to request creation of a report for the generated alerts. The report is currently generated in markdown format and displayed in the VSS UI.

As a first step, identify the alert ID for which the report needs to be generated. Use the chat interface to retrieve the ID as described earlier, or expand any alert in the Alerts tab to display the Id field along with associated metadata.

Now, use the ID to request generation of the report while also specifying the associated sensor, as shown in the sample image below.

Step 5: Teardown the Agent

To tear down the agent, run the following command:

cd deploy/docker
PROFILE=alerts
docker compose -f compose.yml \
  --env-file containers.env \
  --env-file "developer-profiles/dev-profile-${PROFILE}/.env" \
  --env-file "developer-profiles/dev-profile-${PROFILE}/user-overrides.env" \
  down -v --remove-orphans

This command stops the manually deployed real-time-alert containers and removes their Compose volumes.

Service Endpoints

The following endpoints apply to the stock manual Compose deployment. Build Vision AI reports the endpoints available in the deployment it creates; use that deployment summary instead of assuming every endpoint below is present.

The Jaeger UI is present only when the jaeger Compose profile is enabled; see OpenTelemetry Tracing in the Alerts Microservice page.

Known Issues

• The default sample prompt may cause false positives on some streams. When using a video other than the default sample stream, update and make the alert prompt as specific as possible about the conditions to detect.

• Only RTSP streams are supported as input when deploying the Alerts developer profile in real-time mode. Attempting to upload MP4 files directly to VIOS causes RTVI VLM errors.

• Removing a camera from VIOS with always-on alerts enabled can leave an orphaned RT-VLM stream record even though the camera_remove webhook reports success. Alert Bridge stops caption generation and deletes the stream concurrently; RT-VLM can reject the delete with HTTP 409 while shutdown is in progress. Alert Bridge removes the in-memory rule despite the delete failure, so subsequent VIOS retries return HTTP 200 without cleaning up the stale record. The camera UUID can remain in GET /v1/streams/get-stream-info after the pipeline stops. Workaround: Wait for caption shutdown to complete, then retry the RT-VLM stream delete for that camera UUID. This issue affects automatic camera removal teardown.

• Deleting a real-time alert rule for a camera that has multiple rules causes all other rules for that camera to stop producing alerts, even though they remain listed as active.

• It is recommended to clear chat and refresh the VSS UI page after deployment to clear any data from a previous deployment.

• VSS-RTVI-VLM logs may repeatedly show a “Generated text is empty” warning and result in incomplete/unreliable alert verdicts

• VLM verdict verification can fail in the UI even when VLM requests are processed successfully. Alert Bridge response parsing accepts only raw verdicts such as A/B or Yes/No before serializing them to confirmed or rejected. As a result, a semantically valid rejected verdict from Cosmos Reason can fail schema validation and be persisted as verification-failed.

• Report generation may produce inaccurate results.

• The VST UI is externally accessible on both port 30000 and 30888 because both ports are exposed on the host network. For security hardening, consider using a firewall to allowlist only the required VST ingress port.