UI Application

A sample UI application based on Gradio is provided along with VSS. It supports file and live stream summarization, Q&A, and alerts.

The VSS container logs show the port on which the UI application server is running. The UI can be accessed by navigating to http://<VIA_HOST_IP:<FRONTEND_PORT>.

File Summarization, Q&A, and Alerts

Note

File types supported: mp4, mkv - with h264/h265 video and Opus/Vorbis audio codecs.

The following image shows the file summarization page of the UI. It also supports Q&A and alerts for video files.

To start file summarization, use the following:

0. Make sure the Video File Summarization tab is selected.

1. Either:

   • 1A. Upload a file.

   • 1B. Choose a preloaded example from Samples Tab.

2. Select the chunk size from the drop down list.

3. (Optional) Set VLM prompt and summarization prompts for CA-RAG. Refer to Tuning Prompts for more details on the prompts.

4. (Optional) See Add Alerts to Video File Summarization.

5. Adjust the parameters by clicking on Show Parameters and modifying the available parameters (Optional).

6. Select one of the following actions:

   • 6A. Click Summarize to start summarization only.

   • 6B. (Optional) Check Enable CV Metadata checkbox to enable CV metadata generation. Modify the prompt to detect specific objects.

   • 6C. (Optional) Check Enable Audio checkbox to enable audio transcript generation.

   • 6D. (Optional) Check Enable Chat for the file checkbox to enable Q&A after summarization.

   • 6E. (Optional) Check Enable chat history checkbox to preserve the context of the Q&A session. Q&A will take into account the conversation history.

   Note

   The audio and cv related configurations are hidden by default. Audio / CV pipeline needs to be enabled for the options to show.

   The button and checkbox are enabled after a video is selected or uploaded.

The summarization takes from a few seconds to minutes, depending on various factors including video length, chunk size, prompt, VLM model, and GPUs installed on the host.

After the processing is finished, the response shows the video summary. For example:

If “Enable CV Metadata” was chosen, the response also shows the video overlaid with CV metadata (i.e. masks & tracking ids for detected objects) after the summary. (Optional) Modify the CV PIPELINE PROMPT to detect specific objects. For example:

Add Alerts to Video File Summarization

For adding alerts to Video File Summarization, use the following steps:

0. Make sure the Video File Summarization tab is selected.

1. Click Create Alerts.

2. Then click on Add Alert.

3. This will open a popup Add New Alert.

4. Enter the alert name and the events to alert on.

5. Click on Save to save the alert.

Note

The UI does not support adding/removing/modifying alerts for video file summarization once the summarization is started.

If any alerts were configured and detected, they are visible under the Alerts tab. For example:

To do Q&A on the file (if Chat was chosen):

0. Go back to Chat.

1. Enter a question in the text box.

2. Click Ask.

3. Verify the answer is visible in the Chat window after the request is processed.

Note

The summarization and chat requests will keep running on the server side till the request is completed even after the browser window is closed or refreshed.

To generate highlights from the video file:

After summarization is complete, click on Generate Highlight to generate highlights from the video file.

If you want to generate highlights of specific scenarios,

0. Go back to Chat.

1. Click on Generate Scenario Highlight.

2. You will see a popup with a text box to enter the scenario keywords separated by commas. e.g. forklift,worker,ppe

3. Enter the scenario keywords and click on Generate Scenario Highlights.

Note

The scenario highlight generation is an experimental feature. It may fail if the LLM fails to generate a valid JSON response. Please try again if it fails.

If you want to configure any parameters click on Show Parameters and modify the available parameters.

It will show the available parameters and their descriptions as shown below:

Configure any parameters that you want and click on close icon at top right of the pop up.

If you want to delete session and start fresh, click on Delete File button.

Note

The Stop/Delete File button is only activated once the summarization is complete.

Image Summarization, Q&A, and Alerts

Image summarization is similar to Video summarization except the Image Summarization and Q&A tab must be chosen.

Note

The default prompts in use are for warehouse videos. Refer to more examples here: Examples Please refer to Tuning Prompts for more info on tuning prompts

File types supported: jpg and png.

The following image shows the image file summarization and Q&A page of the UI. To get started, follow these steps:

0. Make sure the Image File Summarization & Q&A tab is selected.

1. Either:

   • 1A. Upload images. Note that you can also upload multiple images at a time.

   • 1B. Select image(s) from the Sample Tab.

Please click on multiple images to add them one-by-one to the list of input images for data processing.

2. (Optional) See Add Alerts to Image File Summarization.

3. Set VLM prompt and summarization prompts for CA-RAG. Refer to Tuning Prompts for more details on the prompts.

4. Adjust the VLM parameters by clicking on Show Parameters and modifying the available parameters (Optional).

5. Select one of the following actions:

   • 5A. Click Summarize to start summarization only.

   • 5B. Click Chat to enable Q&A after summarization.

   • 5C. (Optional) Check Enable chat history checkbox to preserve the context of the Q&A session. Q&A will take into account the conversation history.

   The buttons are enabled after a image is selected or uploaded.

6. If chat was chosen, enter a question after the summary is generated

7. Click on Ask to send the query.

Add Alerts to Image File Summarization

For adding alerts to Image File Summarization, use the following steps:

0. Make sure the Image File Summarization & Q&A tab is selected.

1. Click Create Alerts.

2. Then click on Add Alert.

3. This will open a popup Add New Alert.

4. Enter the alert name and the events to alert on.

5. Click on Save to save the alert.

Note

The UI does not support adding/removing/modifying alerts for image file summarization once the summarization is started.

Live Stream Summarization, Alerts, and Q&A

The following image shows the live stream page of the UI. It also supports Q&A and alerts for live streams.

Note

Support RTSP URLs with h264/h265 video and Opus/Vorbis audio codecs.

To start summarization, use the following:

0. Make sure the Live Stream Summarization tab is selected.

1. Enter a valid RTSP Live Stream URL. Optionally, enter credentials for the RTSP stream.

2. Select the Chunk Size and Summary Duration from the drop down list.

3. (Optional) Set VLM prompt and the summarization prompts for CA-RAG. Refer to Tuning Prompts for more details on the prompts.

4. (Optional) See Add Alerts to Live Stream Summarization.

5. (Optional) Adjust the parameters by clicking on Show Parameters and modifying the available parameters.

6. Select one or both of the following:

   • 6A. Click Start streaming & summarize to start summarization. The button gets enabled after the RTSP URL is entered.

   • 6B. (Optional) Check Enable CV Metadata checkbox to enable CV metadata generation. Modify the prompt to detect specific objects.

   • 6C. (Optional) Check Enable Audio checkbox to enable audio transcript generation.

   • 6D. (Optional) Select Enable Chat to enable Q&A on the stream.

   Note

   The audio and cv related configurations are hidden by default. Audio / CV pipeline needs to be enabled for the options to show.

After the server starts processing the stream, the UI displays a message Waiting for first summary. The first summary is available after approximately summary duration plus a few seconds depending on the summary duration, model, and GPU being used.

Summaries for consecutive summary duration periods are appended one below the other. Newest one being at the bottom.

If additional codecs are enabled, the UI will also show the live-stream preview as well as the Set-of-Marks preview (if CV pipeline is enabled). Only video is shown in the live-stream and Set-of-Marks preview.

The Set-Of-Marks preview only shows the frames sampled for VLM and hence will run at a much lower framerate. Additionally, it is delayed by 2 * chunk duration seconds.

Note

The live-stream preview and Set-Of-Marks preview are alpha features.

Live Stream Summarization can be stopped by clicking Stop/Delete Stream.

If any alerts were configured and detected, they are visible under the Alerts tab.

If you want to set alerts on multiple live streams, you can open a second browser tab to the VSS UI and follow the same steps to connect an additional stream and set new alerts. Both tabs will show alert output for their respective stream.

To do Q&A (if Enable Chat was chosen):

1. Go back to Chat.

2. Enter a question in the text box.

3. Click Ask.

4. Verify the answer is visible in the Chat window after the request is processed.

Note

The summarization requests will keep running on the server side till the live-stream reaches end-of-stream even if the browser window is closed or refreshed. Delete the live-stream to stop the summarization request.

Reconnecting to Live Stream Summarization

For live streams, even if the client disconnects, summarization keeps happening in the VSS server. To reconnect to the live stream and get the latest updates after being disconnected, use the following steps:

0. Make sure the Live Stream Summarization tab is selected.

1. Click Refresh List.

2. After the list is refreshed, select the active stream from the drop down.

3. Click Reconnect. The button is enabled after a stream is selected from the drop-down.

Add Alerts to Live Stream Summarization

For adding live stream alerts, use the following steps:

0. Make sure the Live Stream Summarization tab is selected.

1. Click Create Alerts.

2. Then click on Add Alert.

3. This will open a popup Add New Alert.

4. Enter the alert name and the events to alert on.

5. Click on Save to save the alert.

Note

The UI does not support adding/removing/modifying alerts for live streams once the summarization is started.

To View All Alerts of Live Stream

To view live stream alerts, click on All Alerts:

To View Set-of-Marks Preview Video for Live stream

To view set-of-marks preview video for live stream, follow below steps:

0. Make sure the Live Stream Summarization tab is selected.

1. Enter a valid RTSP Live Stream URL. Optionally, enter credentials for the RTSP stream.

2. Then click on Enable CV Metadata checkbox.

   • 2A. (Optional) Modify the CV PIPELINE PROMPT to detect specific objects.

3. Click on Start streaming & summarize and you will see Set-of-Marks preview video under Set-of-Marks Preview tab.

Note

The Set-Of-Marks preview only shows the frames sampled for VLM and hence will run at a much lower framerate. Additionally, it is delayed by 2 * chunk duration seconds.. This requires additional codecs to be installed additional codecs .

Multi-stream / Concurrent Requests

VSS supports multi-stream / concurrent requests. With Gradio UI, this can be achieved by opening multiple instances of the UI application in different browser tabs and running summarization / chat on different videos / images or live-streams.