Introduction

Advances in AI video understanding and interaction have the potential to revolutionize how we access, analyze, and interact with video content in various domains. These AI models are capable of:

• Video captioning Generating text descriptions or summary of videos.

• Question answering Answering questions about a video’s content.

• Video retrieval Finding specific videos or highlights based on text queries.

• Action recognition Identifying actions happening in the video.

Video Search and Summarization (VSS) Agent Blueprint demonstrates Video Summarization, Q&A, and alerts with accelerated performance on NVIDIA hardware.

Overview

• Release Notes
  • VSS 2.2.0
• Features
• VSS Blueprint Architecture
  • Ingestion Pipeline
  • Retrieval Pipeline
• API Documentation

Quickstart Guide

• Setup the Prerequisites
  • Supported Platforms
  • Prerequisites
  • Install the NVIDIA Driver
  • Installing a Kubernetes Cluster
  • Obtain NGC API Key
• Deploy the Blueprint
  • Create Required Secrets
  • Deploy the Helm Chart
  • Default Deployment Topology and Models in Use
  • Launch VSS UI
  • Configuration Options
  • Optional Deployment Topology with GPU sharing
• Summarize a Video

Reference Application

• Python CLI Client
  • Pre-requisites
  • Files Commands
  • Live Stream Commands
  • Models Commands
  • Summarization Command
  • Chat Command
  • Alerts Commands
  • Server Health and Metrics Commands
• UI Application
  • File Summarization, Q&A, and Alerts
  • Image Summarization, Q&A, and Alerts
  • Live Stream Summarization, Alerts, and Q&A
  • Reconnecting to Live Stream Summarization
• Examples
  • Traffic Camera Video
  • Warehouse Video (short)
  • Bridge Inspection Video
  • Warehouse Video (long)
  • Traffic Camera Images
• Context-Aware RAG
  • Summarization
  • Question and Answer
  • Alerts

Plug-and-Play Guide

• Plug-and-Play Overview
• Configure the VLM
  • Configuring for GPT-4o
  • Configuring for Fine-tuned VILA 1.5 (LoRA)
  • Configuring for NVILA research model
  • OpenAI Compatible REST API
  • Other Custom Models
• Configure the NIMs
• Using External Endpoints
  • Remote LLM Endpoint
  • Remote NeMo Rerank and Embedding Endpoint
  • Using NIMs from build.nvidia.com
• Configure Storage Directories
  • Configure the Assets Directory
  • Configure the Input Video Streams Directory
• Custom Deployment
  • Deployment Components
  • Deployment Examples

Cloud

• Amazon Web Services (AWS)
  • Prerequisites
  • 1. Download OneClick deployment package
  • 2. Prepare env variables
  • 3. Prepare config file
  • 4. Run OneClick script to deploy on AWS
  • 5. Access the deployment
  • 6. Teardown
  • Common Issues
• NVIDIA Brev Launchable
  • Prerequisites
  • Steps to launch the Blueprint

Configuration and Tuning

• VSS Customization
  • Configurable Parameters
  • Tuning Prompts
  • Accessing Milvus Vector DB
  • Custom Post-Processing Functions
  • CA-RAG Configuration
  • Tuning Guardrails
• Model Details
  • VSS VLM Models
  • VSS CA-RAG Models:
• VSS Observability
  • VSS Log Files
  • VSS Health Evaluation Reports
• Performance
• Latency Calculator

Support

• Known Issues
• FAQ
  • Overview
  • Prerequisite FAQs
  • Deployment FAQs