Alert Verification

Two approaches for leveraging VLMs to generate alerts are showcased as part of the agent workflows:

• Alert Verification: The VLM analyzes video snippets corresponding to alerts generated upstream for verification; original alerts are generated through a combination of object detection/tracking and behavior analytics microservices that process video streams in real time. This approach invokes the VLM more sporadically and hence has lower GPU requirements, but depends on an upstream entity to generate “candidate” alerts for verification.

• Real-Time Alerts: The VLM continuously processes segments from a video source (for example, a camera) at periodic intervals based on a user-defined chunk duration. This approach leverages the generalizability of VLMs to trigger alerts for a broad set of cases (VLM fine-tuning or prompt tuning may be needed). However, it has higher GPU requirements due to more frequent VLM usage.

This section addresses the Alert Verification workflow; see Real-Time Alerts for the other approach.

Use Cases for Alert Verification

• PPE compliance verification (hard hats, safety vests)
• Restricted area monitoring
• Asset presence/absence detection
• Custom object detection scenarios

Estimated Deployment Time: 15-20 minutes

The following diagram illustrates the alert verification workflow architecture:

Key Features of the Alert Verification Agent:

• RTVI CV for real-time object detection using Grounding DINO (open-vocabulary detection)
• Behavior Analytics for rule-based and configurable alert generation from detection results
• Alert Verification for VLM-based alert clip review to reduce false positives
• Alert storage for querying and reporting
• Report Generation

What’s being deployed

• NVStreamer: Video streaming service for dataset video playback, thereby replicating live cameras in a real-world deployment
• Video IO & Storage (VIOS): Video ingestion (of NVStreamer video streams) supporting live streaming, recording, and playback features used by the agent
• RTVI CV: Real-Time Video Intelligence CV Microservice for object detection that processes VIOS live streams to output metadata to Kafka
• Behavior Analytics: Processes metadata from RTVI CV to generate alerts
• Alert Verification: Verification of alert video using VLM
• RTVI VLM: Real-Time VLM Microservice for vision-language model inference used by Alert Verification
• ELK: Elasticsearch, Logstash, and Kibana stack for log storage and analysis
• VSS Agent: Agent service that uses a configured LLM endpoint to route requests and orchestrate tool calls to VSS microservices and model endpoints (LLM/VLM NIMs) to answer questions and generate outputs
• Nemotron LLM (NIM): LLM inference service used for reasoning, tool selection, and response generation
• Phoenix: Observability and telemetry service for agent workflow monitoring

This list describes the stock profile used by the manual Compose deployment. Build Vision AI customizes that profile for the requested capabilities. With the default NemoClaw harness, or with no harness, it does not deploy the in-stack VSS Agent or Phoenix. It also omits the Agent’s local LLM unless another selected service needs it. Ask for the in-stack VSS Agent if you plan to use the Agent REST API or the manual Agent UI workflow below.

Prerequisites

Before you begin, ensure all of the prerequisites are met. See Prerequisites for more details.

Deploy

Step 1: Deploy the Agent

# Set NGC CLI API key
export NGC_CLI_API_KEY='your_ngc_api_key'
# Authenticate Docker to pull VSS release images, NIM, and other NGC images.
printf '%s' "$NGC_CLI_API_KEY" | \
  docker login nvcr.io --username '$oauthtoken' --password-stdin

See Configuration for common variables, valid values, defaults, and env-file precedence. The complete profile-specific template is deploy/docker/developer-profiles/dev-profile-alerts/overrides.env.

From the repository checkout root, enter the Compose deployment directory:

cd deploy/docker

overrides.env is the checked-in template. Copy it to any filename matching *-overrides.env before selecting a layout. Files matching that pattern are ignored by Git so local settings and credentials are not committed. The commands below use user-overrides.env as a concrete example; if you choose another name, use it consistently in every --env-file argument. Repeat the copy only when you want to discard prior layout changes:

PROFILE=alerts
cp "developer-profiles/dev-profile-${PROFILE}/overrides.env" \
  "developer-profiles/dev-profile-${PROFILE}/user-overrides.env"

Before starting Compose, complete the host settings and select one hardware layout below in deploy/docker/developer-profiles/dev-profile-alerts/user-overrides.env.

Environment-variable overrides: In deploy/docker/developer-profiles/dev-profile-alerts/user-overrides.env, update existing values in place; add a variable only when it is absent. COMPOSE_PROFILES is derived from values above it, so appending duplicate model-mode or model-slug variables can select the wrong services.

Host-path and service-address overrides (all hardware profiles):

• VSS_APPS_DIR: Absolute path to this repository’s deploy/docker directory.

• VSS_DATA_DIR: Absolute writable host directory for downloaded models, videos, and service data.

• HOST_IP: IP address or hostname that clients use to reach services on this VSS host. Set it to an address reachable by the intended clients.

• EXTERNAL_IP: Address VSS puts in externally reachable service URLs. It defaults to ${HOST_IP}; on a cloud host or when clients connect from outside the host network, set it to the instance’s public address or external hostname.

• VST_INGRESS_ENDPOINT: Host and path VIOS puts in the alert-clip URLs it hands to the VLM. The variable is absent from the profile files; add it when the VLM runs off this host (VLM_MODE=remote), on a line below the existing VST_INGRESS_HOST_PORT assignment:

  VST_INGRESS_ENDPOINT=${HOST_IP}:${VST_INGRESS_HOST_PORT}/vst

  Use host and path only, without a scheme. Substitute ${EXTERNAL_IP} for ${HOST_IP} when the VLM connects from outside the host network, and allow inbound traffic to port 30888 from wherever the VLM runs.

Available VLM Models

Enable Efficient Video Sampling

You can enable Adaptive Efficient Video Sampling (Adaptive EVS) to prune redundant video tokens and reduce VLM inference latency. Add the following settings to deploy/docker/developer-profiles/dev-profile-alerts/user-overrides.env before starting the profile:

VIA_EVS_SESSION=true
VLM_VIDEO_PRUNING_RATE=0.5
VLLM_EVS_SIMILARITY_THRESHOLD=0.015

In testing with a warehouse dataset and the Cosmos 3 Super Reasoner FP8 VLM on RTX PRO 6000, this configuration reduced VLM latency by approximately 20% with an approximately 1% decrease in accuracy.

Configure Required Deployment Values

Set these required placeholder values in the existing assignments:

VSS_APPS_DIR=/absolute/path/to/video-search-and-summarization/deploy/docker
VSS_DATA_DIR=/absolute/path/to/vss-apps-data
HOST_IP=<reachable-host-ip-or-name>

Prepare host data directories before starting Compose. Export VSS_DATA_DIR and VSS_APPS_DIR with the same absolute paths set above, then run this block. The guards stop before mkdir or chmod if either value is absent or is not absolute:

: "${VSS_DATA_DIR:?Export VSS_DATA_DIR with the value from user-overrides.env}"
: "${VSS_APPS_DIR:?Export VSS_APPS_DIR with the value from user-overrides.env}"
case "$VSS_DATA_DIR" in
  /*) ;;
  *) echo "VSS_DATA_DIR must be an absolute path" >&2; exit 1 ;;
esac
case "$VSS_APPS_DIR" in
  /*) ;;
  *) echo "VSS_APPS_DIR must be an absolute path" >&2; exit 1 ;;
esac
mkdir -p \
  "$VSS_DATA_DIR/data_log/analytics_cache" \
  "$VSS_DATA_DIR/data_log/calibration_toolkit" \
  "$VSS_DATA_DIR/data_log/elastic/data" \
  "$VSS_DATA_DIR/data_log/elastic/logs" \
  "$VSS_DATA_DIR/data_log/kafka" \
  "$VSS_DATA_DIR/data_log/nvstreamer/vst_data" \
  "$VSS_DATA_DIR/data_log/redis/data" \
  "$VSS_DATA_DIR/data_log/redis/log" \
  "$VSS_DATA_DIR/data_log/vss_video_analytics_api" \
  "$VSS_DATA_DIR/data_log/vst/clip_storage" \
  "$VSS_DATA_DIR/agent_eval/dataset" \
  "$VSS_DATA_DIR/agent_eval/results" \
  "$VSS_DATA_DIR/models" \
  "$VSS_DATA_DIR/videos/dev-profile-alerts"
mkdir -p "$VSS_APPS_DIR/engines/gdino" "$VSS_APPS_DIR/engines/rtdetr-its"
chmod -R a+rwx "$VSS_APPS_DIR/engines"
chmod -R a+rwx \
  "$VSS_DATA_DIR/data_log" \
  "$VSS_DATA_DIR/agent_eval" \
  "$VSS_DATA_DIR/models" \
  "$VSS_DATA_DIR/videos"

Direct Docker Compose Overrides

Add this common alert-verification block to developer-profiles/dev-profile-alerts/user-overrides.env before adding the hardware-specific values from the tabs below:

MODE=2d_cv
ALERT_AGENT_ALWAYS_ON=false
COMPOSE_PROFILES=${COMPOSE_PROFILES_CV}

Default container image tag

containers.env uses VSS_CONTAINER_TAG as the default tag for all images. Export the tag selected for this documentation branch before choosing a hardware layout:

export VSS_CONTAINER_TAG="3.3.0"

Any service-specific image tag or tag suffix shown inside a hardware tab supersedes or augments this common default. Use the SBSA settings shown in the GB300 and DGX Spark tabs when deploying on those platforms.

HARDWARE_PROFILE=H100
RTVI_VLLM_GPU_MEMORY_UTILIZATION=0.4

After replacing every placeholder in the selected layout, from deploy/docker validate the resolved configuration, then start the profile:

PROFILE=alerts
docker compose -f compose.yml \
  --env-file containers.env \
  --env-file "developer-profiles/dev-profile-${PROFILE}/.env" \
  --env-file "developer-profiles/dev-profile-${PROFILE}/user-overrides.env" \
  config --quiet
docker compose -f compose.yml \
  --env-file containers.env \
  --env-file "developer-profiles/dev-profile-${PROFILE}/.env" \
  --env-file "developer-profiles/dev-profile-${PROFILE}/user-overrides.env" \
  up --detach --pull always --force-recreate --build

On the first bring-up, Docker Compose pulls the required containers from the configured NGC registries and builds five support images locally: Elasticsearch, its initializer, the Kafka topic initializer, the broker health checker, and the alerts Kibana initializer. Those builds pull base images from Docker Hub and docker.elastic.co; the Kafka support-image builds also download jq from GitHub. Allow outbound access and disk space for both pulls and local builds. Depending on your network speed, this may take a few minutes.

This deployment uses the following defaults:

• LLM model: nvidia/nemotron-3.5-lightning-30b-a3b
• VLM model: cosmos-reason3 (served via RTVI VLM container)

Once the deployment is complete, check that all the containers are running and healthy:

docker ps

Once all the containers are running, you can access the Agent UI at http://<HOST_IP>:7777/.

Deploy with Agent Skills

As an alternative to running the deployment command manually, you can use VSS Agent Skills from a coding agent such as Claude Code, Codex, or NemoClaw. First, install the vss-build-vision-ai skill as described in Agent Skills and make it accessible to your coding agent. The host must meet the same deployment requirements listed above (supported GPU/hardware for that profile) and must meet the Prerequisites.

The vss-build-vision-ai skill composes the stack from the capabilities you name and selects the <platform> and <mode> to match your system, as detailed in Development Profile GPU Requirements. Refer to the requirements table for valid platform and mode combinations compatible with your hardware.

Deploy prompt structure

The remaining steps on this page use the in-stack Agent UI, so request the in-stack VSS Agent explicitly when you deploy with Build Vision AI.

Deploy the VSS alerts verification profile with the in-stack VSS Agent so I can use the Agent UI workflow.

Name an image tag in the prompt to run a release’s images instead of the containers.env default. See Container image tag.

Deploy the VSS alerts verification profile with the in-stack VSS Agent and image tag 3.3.0.

For edge platforms, ask the vss-build-vision-ai skill to use a remote LLM when possible. Local LLM/VLM sharing on the edge GPU is supported, but a remote LLM is recommended for better response time. See VSS Agent Customization - Configure LLM for remote LLM endpoint options.

Recommended edge deployment prompts

Deploy the VSS alerts verification profile on this DGX-SPARK machine with a remote LLM and the in-stack VSS Agent.
Deploy the VSS alerts verification profile on this IGX-THOR machine with a remote LLM and the in-stack VSS Agent.
Deploy the VSS alerts verification profile on this AGX-THOR machine with a remote LLM and the in-stack VSS Agent.

The vss-build-vision-ai skill uses Cosmos3 Nano Reasoner BF16 as the default VLM. You can ask the agent which VLM models are available during the deployment process, or request a different model directly in your deployment prompt:

Which VLM models are available for the alerts profile?
Deploy the VSS alerts verification profile with the Cosmos3 Super Reasoner FP8 VLM and the in-stack VSS Agent.

The agent sets VLM_NAME and RTVI_VLM_MODEL_PATH in the generated _builds/<name>/override.env to the model you request. The same options and hardware constraints listed in Available VLM Models apply.

If you would like to modify the workflow to work with other videos or use cases, you can update the following files:

• deploy/docker/developer-profiles/dev-profile-alerts/vlm-as-verifier/configs/alert_type_config.json

  • Modify the “output_category” and “user” fields here. “output_category” modifies the category type that shows up in the agent UI. “user” is the user prompt that is sent to the VLM for verification of each alert clip.

• deploy/docker/developer-profiles/dev-profile-alerts/deepstream/configs/config_triton_nvinferserver_gdino.txt

  • Update the “type_name” field under “postprocess” to change the objects that are detected by Grounding DINO. By default this is “person” with a threshold of 0.5. This can be modified to support multiple classes by using ” . ” as a delimiter.

• deploy/docker/developer-profiles/dev-profile-alerts/vss-behavior-analytics/configs/vss-behavior-analytics-config.json

  • Update the “value” field under "name": "fovCountViolationIncidentObjectType" to change the objects that Behavior Analytics creates alerts for.

• Custom response-parser directory (optional; user-created)

  • The repository does not ship or mount a parser directory. Create a host directory, add the read-only /app/parsers volume described in Pluggable Response Parser, and place a Python module exposing a parser class with a parse(self, raw_response: str) -> dict method in it. Then set vlm.response_parser in config.yml to its dotted path and recreate Alert Bridge. Use this to replace the built-in YES/NO verification parsing with classification, analytics, or enrichment outputs.

Scale Alert processing across CPU cores

A single Alerts Microservice process uses about one CPU core. Running the pipeline in several processes adds CPU capacity, and it helps when that process is consistently near one core while Kafka consumer lag keeps growing. A process below one core is limited by offered traffic or by a downstream service, and gains nothing from additional processes.

Edit the profile’s Alerts Microservice configuration and redeploy:

deploy/docker/developer-profiles/dev-profile-alerts/vlm-as-verifier/configs/config.yml

This setting is not available through user-overrides.env. A conservative four-process configuration that keeps the profile’s instance-wide VLM and VIOS load unchanged:

alert_agent:
  pipeline_mode: event_loop
  processes: 4
  startup_timeout_seconds: 60
  async_dispatch_max_in_flight: 4
  async_io:
    max_vlm_concurrent: 1
    max_vst_concurrent: 2
persistence:
  enabled: true

Every per-process limit is multiplied by the process count, which is why the three limits above are divided by four from the one-process profile values of 16, 4, and 8. max_vlm_concurrent caps what each process offers the VLM backend and does not raise the backend’s throughput. When the VLM is the limiting service, capacity comes from the VLM side, for example more replicas behind the same endpoint.

Before raising the process count, confirm the following:

• pipeline_mode is event_loop and persistence.enabled is true. Both are the profile defaults.
• Every Alert input topic has at least as many partitions as the process count, counted per topic. The profile creates eight partitions per topic, so one instance can run four processes as shipped. When several Alerts Microservice instances share the consumer group, for example as replicas on Kubernetes, the bound applies to instances × processes.
• At least as many active streams as processes. The Behavior Analytics pipeline keys every message of one stream by sensorId, so a stream lands on one partition and one process, and work is spread across processes at stream granularity. Fewer streams than processes leaves processes without work, and more streams spread the load more evenly.
• The host has about one CPU core per process, plus capacity for the parent and API processes. Each process also holds its own Kafka consumers, HTTP connection pools to the VLM and VIOS backends, Elasticsearch clients, and memory, so connection, file-descriptor, and memory use grow with the process count.
• The VLM and VIOS backends can serve processes × max_vlm_concurrent and processes × max_vst_concurrent requests concurrently.

Startup refuses a combination the deployment cannot honor and names the reason. /health returns HTTP 503 until every process holds a Kafka assignment. Allow for startup_timeout_seconds before the instance reports ready.

For pipeline modes, startup validation, throughput monitoring, and rollback, see Scaling and Performance Tuning in the Alerts Microservice guide.

Step 2: Add a video stream

Add an RTSP stream by clicking the + Add RTSP button on the Video Management tab in the agent UI. If you do not have an RTSP stream, you can use NVStreamer at http://<HOST_IP>:31000 to upload a video file and create an RTSP stream.

For this profile, use the sample-warehouse-ladder.mp4 stream.

Step 3: Verify pipeline components

Open the Kibana UI at http://<HOST_IP>:7777/kibana/app/home#/ and navigate to the Discover tab.

Verify the following data indices are populated. It may take a few minutes for data to start appearing after adding the stream:

• mdx-raw-* - Raw detection data
• mdx-incidents-* - Generated incidents
• mdx-vlm-incidents-* - VLM-verified alerts

If the indices do not show data, check the following items:

• Ensure that the Perception service is running and processing the stream. View the container logs with docker logs -f vss-rtvi-cv and confirm that the engine files are built and the added stream is being processed. Common causes of failure include model download errors during startup and insufficient GPU memory.
• Confirm that the labels emitted by the Perception model match the labels that Behavior Analytics expects. Behavior Analytics processes only the object type specified by the fovCountViolationIncidentObjectType field in deploy/docker/developer-profiles/dev-profile-alerts/vss-behavior-analytics/configs/vss-behavior-analytics-config.json.

Step 4: View alerts in the Agent UI

Launch the Agent UI at http://<HOST_IP>:7777/.

List streams to verify connectivity, then use the Alerts tab to list alerts. Select the verified alerts option to view VLM-verified alerts. To see the alerts, go to Agent UI > Alerts Tab > View Alerts.

You can then click on a video thumbnail to play the video and view the alert. When the bounding box overlay is enabled, playback includes bounding boxes over objects of interest. This overlay is off by default; to enable it, set vst_config.add_overlay: true on the alert verification service and NEXT_PUBLIC_ALERTS_TAB_MEDIA_WITH_OBJECTS_BBOX=true for the Agent UI.

Step 5: Generate a Report for the Alert

You can use the chat interface to request creation of a report for the generated alerts. The report is currently generated in markdown format and displayed in the VSS UI.

As a first step, identify the alert ID for which the report needs to be generated. Use the chat interface to retrieve the ID by listing recent alerts by count, or expand any alert in the Alerts tab to display the Id field along with associated metadata.

Now, use the ID to request generation of the report while also specifying the associated sensor, as shown in the sample image below.

Step 6: Teardown the Agent

To tear down the agent, run the following command:

cd deploy/docker
PROFILE=alerts
docker compose -f compose.yml \
  --env-file containers.env \
  --env-file "developer-profiles/dev-profile-${PROFILE}/.env" \
  --env-file "developer-profiles/dev-profile-${PROFILE}/user-overrides.env" \
  down -v --remove-orphans

This command stops the manually deployed alert-verification containers and removes their Compose volumes.

Service Endpoints

The following endpoints apply to the stock manual Compose deployment. Build Vision AI reports the endpoints available in the deployment it creates; use that deployment summary instead of assuming every endpoint below is present.

The Jaeger UI is present only when the jaeger Compose profile is enabled; see OpenTelemetry Tracing in the Alerts Microservice page.

Next Steps

Once you’ve familiarized yourself with the alert verification workflow, you can explore:

• Modifying the alert prompt in the Alerts Microservice configuration.
• Adjusting rate limit settings to control alert verification frequency.
• Configuring G-DINO prompting and class thresholds for custom detection scenarios.

Known Issues

• Some VLM inaccuracies might be observed depending on the model and configuration used with the RTVI VLM container.

• Video snippets generated for alerts may be short (for example, only a couple of seconds) depending on behavior analytics processing of the specific video, which could impact VLM accuracy. To address this issue, modify the fovCountViolationIncidentThreshold setting to the desired minimum alert clip duration in deploy/docker/developer-profiles/dev-profile-alerts/vss-behavior-analytics/configs/vss-behavior-analytics-config.json.

• Video playback duration for verified alerts may not exactly match the alert timestamps.

• Report generation may produce inaccurate results.

• If perception crashes and restarts, streams are not automatically re-added and alerts will not be generated.

• For remote VLM and LLM deployments, the alert verification timeout may need to be increased from the default value of 5 seconds. See Alert Verification VLM Configuration Options for specific details.

• VLM verdict verification can fail in the UI even when VLM requests are processed successfully. Alert Bridge response parsing accepts only raw verdicts such as A/B or Yes/No before serializing them to confirmed or rejected. As a result, a semantically valid rejected verdict from Cosmos Reason can fail schema validation and be persisted as verification-failed.

• The VST UI is externally accessible on both port 30000 and 30888 because both ports are exposed on the host network. For security hardening, consider using a firewall to allowlist only the required VST ingress port.

• Spurious records in the mdx-vlm-incidents-1970-01-01 index may be generated by RTVI VLM. Workaround: add this near the top of the Logstash filter { } block in deploy/docker/services/infra/elk/logstash/pipelines/kafka/mdx-logstash.conf:

  # Drop internal RT-VLM activity-detector noise before indexing.
  if [analyticsModule][id] == "vlm-activity-detector" and [analyticsModule][info][streamType] == "file" {
    drop { }
  }

  If streamType is unavailable, use the following fallback:

  if [analyticsModule][id] == "vlm-activity-detector" and [category] == "vlm-alert" {
    drop { }
  }