RT-VLM Performance | NVIDIA Blueprint for Video Search and Summarization

Overview

The Real-Time VLM (RTVI-VLM) microservice provides real-time video understanding capabilities using Vision Language Models (VLM) for live RTSP streams and pre-recorded video files, generating captions and detecting incidents. It uses the Cosmos3 Nano Reasoner model served via vLLM.

Video is processed in 10-second chunks: the microservice accumulates 80 frames per chunk at 448×448 resolution, producing 7,840 vision tokens per inference call. Benchmarks cover two operating modes and two output sequence lengths (OSL):

Streaming mode — the microservice reads live RTSP streams continuously. Latency is measured from when a chunk is ready until the VLM response is received.
File processing mode — pre-recorded video files are processed as fast as possible since all frames are immediately available.

Output sequence length (OSL) determines the use case:

OSL = 1 (Alerting) — the VLM outputs a single Yes/No token in response to a binary alert prompt. Lower latency and higher throughput.
OSL = 100 (Captioning) — the VLM generates a descriptive caption of the video chunk. Higher output token count leads to longer latency and lower throughput.

RTVI-VLM max concurrent streams per GPU for alerting and captioning use cases — Max concurrent live RTSP streams per GPU. Alerting (OSL=1) consistently supports more concurrent streams than captioning (OSL=100) due to its single-token output. H100 and RTX Pro 6000 SE are the recommended platforms for multi-stream deployments.

Test Configuration

Parameter	Value
VSS Release	3.2
Model	Cosmos3 Nano Reasoner
Model precision	FP8
Inference engine	vLLM
Chunk duration	10 seconds
Frames per chunk	80
Image resolution	448×448
Vision tokens per chunk	7,840
Input sequence length (ISL)	56 text tokens
GPUs tested	H100, RTX Pro 6000 SE, DGX Spark, AGX Thor

Performance by GPU

H100

RTX Pro 6000 SE

DGX Spark

AGX Thor

Streaming Mode

Use Case	Max Streams	Chunk E2E Avg (s)	p90 (s)	p95 (s)	GPU Core (%)	GPU Mem (%)
Alerting (OSL=1)	51	3.6	4.86	4.95	94.8	86.8
Captioning (OSL=100)	33	4.78	6.31	6.44	83.8	83.9

Use Case	Concurrent Streams	Chunk E2E Avg (s)	GPU Core Avg (%)
Alerting (OSL=1)	1	0.58	0.4
Alerting (OSL=1)	10	1.10	21.0
Alerting (OSL=1)	20	1.33	30.8
Captioning (OSL=100)	1	1.23	8.2
Captioning (OSL=100)	10	2.21	31.2
Captioning (OSL=100)	20	3.72	52.2

File Processing Mode

Use Case	Video Length	E2E Latency (s)
Alerting (OSL=1)	10 s	0.67
Alerting (OSL=1)	10 min	18.1
Alerting (OSL=1)	60 min	108.0
Captioning (OSL=100)	10 s	1.31
Captioning (OSL=100)	10 min	22.0
Captioning (OSL=100)	60 min	131.1

Concurrency	E2E Latency Avg (s)	Throughput (req/s)	p90 (s)	p95 (s)
1	0.61	1.64	0.74	0.74
2	0.60	3.33	0.75	0.75
4	0.67	5.97	0.80	0.81
8	0.72	11.11	1.07	1.14
16	1.12	14.29	1.75	1.76
32	1.58	20.25	2.38	2.44

Concurrency	E2E Latency Avg (s)	Throughput (req/s)	p90 (s)	p95 (s)
1	1.13	0.88	1.35	1.35
2	1.19	1.68	1.43	1.43
4	1.25	3.20	1.53	1.54
8	1.69	4.73	1.67	1.68
16	1.92	8.33	2.33	2.34
32	2.65	12.08	3.69	3.71

All benchmarks use Cosmos3 Nano Reasoner, FP8, vLLM, 10-second chunks, 80 frames per chunk, 448×448 resolution, 7,840 vision tokens, and ISL=56 text tokens. For streaming deployments, plan for 10–15% headroom below the maximum concurrent stream counts. GPU Memory Utilization is not available for DGX Spark and AGX Thor. p90/p95 latency is not available for DGX Spark and AGX Thor in file processing mode.