UI Application

A sample UI application based on Gradio is provided along with VSS. It supports file and live stream summarization, Q&A, and alerts.

The VSS container logs show the port on which the UI application server is running. The UI can be accessed by navigating to http://<VIA_HOST_IP:<FRONTEND_PORT>.

File Summarization, Q&A, and Alerts

Note

File types supported: mp4, mkv - with h264/h265 videos.

The following image shows the file summarization page of the UI. It also supports Q&A and alerts for video files.

To start file summarization, use the following:

0. Make sure the Video File Summarization tab is selected.

1. Either:

   • 1A. Upload a file.

   • 1B. Choose a preloaded example.

2. Select the chunk size from the drop down list.

3. (Optional) Set VLM prompt and summarization prompts for CA-RAG. Refer to Tuning Prompts for more details on the prompts.

4. (Optional) Set alerts in the format <alert_name>:<event1>,<event2>,....

5. Adjust the VLM parameters by clicking on Show Parameters and modifying the available parameters (Optional).

6. Select one of the following actions:

   • 6A. Click Summarize to start summarization only.

   • 6B. (Optional) Check Enable Chat for the file checkbox to enable Q&A after summarization.

   The button and checkbox are enabled after a video is selected or uploaded.

The summarization takes from a few seconds to minutes, depending on various factors including video length, chunk size, prompt, VLM model, and GPUs installed on the host.

After the processing is finished, the response shows the video summary. For example:

If any alerts were configured and detected, they are visible under the Alerts tab. For example:

To do Q&A on the file (if Chat was chosen):

0. Go back to Chat.

1. Enter a question in the text box.

2. Click Ask.

3. Verify the answer is visible in the Chat window after the request is processed.

Image Summarization, Q&A, and Alerts

Image summarization is similar to Video summarization except the Image Summarization and Q&A tab must be chosen.

Note

The default prompts in use are for warehouse videos. Refer to more examples here: Examples

File types supported: jpg and png.

The following image shows the image file summarization and Q&A page of the UI. To get started, follow these steps:

0. Make sure the Image File Summarization & Q&A tab is selected.

1. Upload images. Note that you can also upload multiple images at a time.

2. (Optional) Set alerts in the format <alert_name>:<event1>,<event2>,....

3. Set VLM prompt and summarization prompts for CA-RAG. Refer to Tuning Prompts for more details on the prompts.

4. Adjust the VLM parameters by clicking on Show Parameters and modifying the available parameters (Optional).

5. Select one of the following actions:

   • Click Summarize to start summarization only.

   • Click Chat to enable Q&A after summarization.

   The buttons are enabled after a video is selected or uploaded.

6. If chat was chosen, enter a question after the summary is generated

7. Click on Ask to send the query.

Live Stream Summarization, Alerts, and Q&A

The following image shows the live stream page of the UI. It also supports Q&A and alerts for live streams.

To start summarization, use the following:

0. Make sure the Live Stream Summarization tab is selected.

1. Enter a valid RTSP Live Stream URL. Optionally, enter credentials for the RTSP stream.

2. Select the Chunk Size and Summary Duration from the drop down list.

3. (Optional) Set VLM prompt and the summarization prompts for CA-RAG. Refer to Tuning Prompts for more details on the prompts.

4. (Optional) Set alerts in the format <alert_name>:<event1>,<event2>,....

5. (Optional) Adjust the VLM parameters by clicking on Show Parameters and modifying the available parameters.

6. Select one or both of the following:

   • Click Start streaming & summarize to start summarization. The button gets enabled after the RTSP URL is entered.

   • (Optional) Select Enable Chat to enable Q&A on the stream.

After the server starts processing the stream, the UI displays a message Waiting for first summary. The first summary is available after approximately summary duration plus a few seconds depending on the summary duration, model, and GPU being used.

Summaries for consecutive summary duration periods are appended one below the other. Newest one being at the bottom.

Live Stream Summarization can be stopped by clicking Stop Summarization & Delete Live Stream.

If any alerts were configured and detected, they are visible under the Alerts tab.

If you want to set alerts on multiple live streams, you can open a second browser tab to the VSS UI and follow the same steps to connect an additional stream and set new alerts. Both tabs will show alert output for their respective stream.

To do Q&A (if Enable Chat was chosen):

1. Go back to Chat.

2. Enter a question in the text box.

3. Click Ask.

4. Verify the answer is visible in the Chat window after the request is processed.

Reconnecting to Live Stream Summarization

For live streams, even if the client disconnects, summarization keeps happening in the VSS server. To reconnect to the live stream and get the latest updates after being disconnected, use the following steps:

0. Make sure the Live Stream Summarization tab is selected.

1. Click Refresh List.

2. After the list is refreshed, select the active stream from the drop down.

3. Click Reconnect. The button is enabled after a stream is selected from the drop-down.