Examples#

There are multiple example clips provided in the UI Application which can be explored to better understand how the agent works. For each example, we’ve provided a Prompt, Caption Summarization Prompt, and Summary Aggregation Prompt to get started. Refer to Tuning Prompts for more details on the prompts.

Traffic Camera Video#

This is a 2-minute 10-second video (its.mp4) consisting of a synthetically generated traffic scene.

Chunk Size: 10 sec

Prompt

You are an intelligent traffic system.
You must monitor and take note of all traffic related events.
Start and end each sentence with a time stamp.

Caption Summarization Prompt

You will be given captions from sequential clips of a video.
Aggregate captions in the format start_time:end_time:caption
based on whether captions are related to one another or create
a continuous scene.

Summary Aggregation Prompt

Based on the available information, generate a traffic report
that is organized chronologically and in logical sections.
This should be a concise, yet descriptive summary of all the important events.
The format should be intuitive and easy for a user to read and understand what happened.
Format the output in Markdown so it can be displayed nicely.

Sample questions

Did a car crash occur?
When do the police arrive at the crash?
What cars were involved in the crash?

Warehouse Video (short)#

This is a 3-minute 30-second video (warehouse.mp4) consisting of clips within a warehouse environment.

Chunk Size: 10 sec

Prompt

You are a warehouse monitoring system. Describe the events
in this warehouse and look for any anomalies.
Start and end each sentence with a time stamp.

Caption Summarization Prompt

Summarize similar captions that are sequential to one another,
while maintaining the details of each caption, in the format
start_time:end_time:caption. The output should be bullet points
in the format start_time:end_time: detailed_event_description.

Summary Aggregation Prompt

Aggregate captions in the format start_time:end_time:caption
based on whether captions are related to one another or create
a continuous scene. The output should only be bullet points in
the format start_time:end_time: detailed_event_description.

Sample questions

When did the forklift first arrive?
Did a worker drop any boxes?
What breaches of safety protocol took place?

Bridge Inspection Video#

This is a 3-minute video (bridge.mp4) consisting of drone footage inspecting a bridge.

Chunk Size: 20 sec

Prompt

You are a bridge inspection system. Describe the condition of
the bridge. Start and end each sentence with a time stamp.

Caption Summarization Prompt

You will be given captions from sequential clips of a video.
Aggregate captions in the format start_time:end_time:caption
based on whether captions are related to one another or create
a continuous scene.

Summary Aggregation Prompt

Based on the available information, generate a summary that
describes the condition of the bridge. The summary should be
organized chronologically and in logical sections. This should be a
concise, yet descriptive summary of all the important events.
The format should be intuitive and easy for a user to understand what happened.
Format the output in Markdown so it can be displayed nicely.

Sample questions

Where is graffiti located?
At what time do you see graffiti?
Are there people on the bridge?

Warehouse Video (long)#

This is an 82-minute video (warehouse_82min.mp4) consisting of clips within a warehouse environment.

Chunk Size: 60 sec

Prompt

Write a concise and clear dense caption for the provided warehouse video,
focusing on irregular or hazardous events such as boxes falling, workers
not wearing PPE, workers falling, workers taking photographs,
workers chitchatting, forklift stuck, etc. Start and end each sentence
with a time stamp.

Caption Summarization Prompt

You should summarize the following events of a warehouse in the format
start_time:end_time:caption. For start_time and end_time use . to
separate seconds, minutes, hours. If during a time segment only regular
activities happen, then ignore them, else note any irregular activities
in detail. The output should be bullet points in the format
start_time:end_time:detailed_event_description. Don't return anything
else except the bullet points.

Summary Aggregation Prompt

You are a warehouse monitoring system. Given the caption in the form
start_time:end_time:caption, Aggregate the following captions in the
format start_time:end_time:event_description. If the event_description
is the same as another event_description, aggregate the captions in
the format start_time1:end_time1,...,start_timek:end_timek:event_description.
If any two adjacent end_time1 and start_time2 is within a few tenths of a second,
merge the captions in the format start_time1:end_time2. The output should
only contain bullet points.  Cluster the output into Unsafe Behavior,
Operational Inefficiencies, Potential Equipment Damage and Unauthorized Personnel.

Sample questions

When did the red forklift first appear in the scene?
Was anyone not wearing personal protective equipment?
What were workers doing?

Traffic Camera Images#

This example is a sequence of images sampled from a traffic camera with a timestamp overlay. Multiple images can be provided under the “IMAGE FILE SUMMARIZATION & Q&A” tab of the UI.

Images

Select multiple images from the samples list sequentially until all images are loaded, as shown below.

Prompt

You are an intelligent traffic system. You will be given a set of images from a traffic intersection.
Write a detailed caption for each image to capture all traffic related events and details. For each caption,
include the timestamp from the image.

Caption Summarization Prompt

Combine the captions if needed. Do not lose any information.

Summary Aggregation Prompt

You will be given a set of captions describing several images from a traffic intersection.
Write a summary of the events from the captions and include the timestamp information.

Sample questions

What time did the car crash occur?
Did a police car respond to the crash?
When did the firetruck arrive?

Jensen Huang’s Special Address at AI Summit India 2024 (With Audio)#

This is a 4-minute video (Jensen_AI_Summit_India_2024_clip.mp4) clip from NVIDIA CEO Jensen Huang’s Special Address at AI Summit India 2024. The clip contains both video and audio streams. Please use below prompts with Enable Audio selected.

Chunk Size: 30 sec

Prompt

Write a concise and clear dense caption for the provided
NVIDIA AI Summit video, focusing on the technologies and products.

Caption Summarization Prompt

You should summarize the following events of a conference.
The output should be in bullet points with timestamps.
Do not return anything else except the bullet points.

Summary Aggregation Prompt

You are a video description service.  Given the video captions and
audio transcripts, aggregate them to a concise summary with timestamps.
The output should only contain bullet points.
Cluster the output into main themes in chronological order.

Sample questions

According to the presenter, how has the method of software development changed?
What are the different acceleration libraries mentioned in the presentation?
What examples of translating information from one modality to another were given by the speaker?

Traffic Camera Video (With “Enable CV Metadata” selected)#

Please replace the first prompt in Traffic Camera Video with the prompt below, so that VLM uses IDs in event descriptions. Also, set the following CV Pipeline Prompt.

Prompt
You are an intelligent traffic system. The provided video is
a processed clip where each vehicle is overlaid with an ID.
You must monitor and take note of all traffic related events.
Start each event description with a start and end time stamp
of the event, and use vehicle IDs in event description.
CV Pipeline Prompt
vehicle
If user needs to specify threshold (0-1.0) for object detection, they can do so by adding it after object prompts using semicolon as delimiter as shown in the following prompt:
vehicle;0.48

Sample questions

Do you see any abnormal events in the clip? If so, which cars are involved?
Did a police car respond to the collision? If so, please answer with the ID of the police car.
Did a firetruck respond to the collision? If so, please answer with the ID of the fire truck.

Warehouse Video Short (With “Enable CV Metadata” selected)#

Please replace the first prompt in Warehouse Video (short) with the prompt below, and set the following CV Pipeline Prompt.

Prompt

You are a warehouse monitoring system. The provided video is
a processed clip where each worker is overlaid with an ID.
Describe the events in this warehouse and look for any anomalies,
for example, box dropping, not waring PPE, unsafe forklift operations.
Start each sentence with start and end timestamp of the event,
and use worker IDs in event description.

CV Pipeline Prompt

person . forklift;0.5

Sample questions

Which worker dropped a box?
Which worker put on the caution tape?
Are there any workers who are not wearing a safety vest or helmet? If so, please identify them by their IDs.
Did anyone cross the restricted area?