Video Summarization

Overview

Video Summarization is a microservice that uses Vision-Language Models (VLMs) and Large Language Models (LLMs) to extract insights from uploaded videos and live streams. The system provides REST API and Model Context Protocol (MCP) interfaces for video processing and summarization.

The Video Summarization microservice segments long-form media, generates VLM captions or structured events, stores the results, and produces concise summaries that preserve timestamped evidence.

Key Features

The Video Summarization microservice operates by analyzing media and generating structured, machine-readable output. The primary output includes timestamped events and a high-level summary, allowing users to identify when specific actions, object appearances, or scene changes occurred.

High Customizability and Model Flexibility

A defining feature of the Video Summarization microservice is its high degree of customization, particularly regarding the underlying language and vision models it employs.

• OpenAI Compatibility: The service is designed for maximum interoperability, allowing users to select and integrate any OpenAI compatible Vision-Language Model (VLM) or Large Language Model (LLM). This flexibility ensures that users can leverage the latest and most suitable models for their specific summarization needs and quality requirements.
• RTVI-VLM video inferencing: Per-chunk and live-stream video inferencing is delegated to the Real-Time VLM (RTVI-VLM) microservice. RTVI-VLM supports Cosmos Reason3, Nemotron Nano Omni, and Qwen 3.5 base models through its proxy and openai-compat modes, so operators can pick the best-fit VLM without rebuilding the Video Summarization microservice.

Data Persistence and Management

The integrity and accessibility of the processed data are managed through a configurable database layer.

• Configurable Database: Processed summaries, extracted events, captions, and associated metadata are stored in a dedicated database. Four backends are selectable — Elasticsearch (the default), Milvus, ArangoDB, and Neo4j. Only Elasticsearch is provisioned inside the compose stack; the others are selected by pointing the microservice at an instance you run. See Database Backends for the selectors and connection variables.

Access and Integration Methods

To ensure broad applicability and seamless integration into diverse workflows, the Video Summarization microservice exposes multiple methods for access and interaction.

• REST API: The service provides a standard RESTful Application Programming Interface (API). This allows users to connect to the Video Summarization microservice using programmatic scripts and traditional software integrations, making it ideal for back-end systems and custom applications.
• Model Context Protocol (MCP): In addition to the REST API, the Video Summarization microservice exposes an MCP interface for AI agents and orchestration systems.

Architecture

The Video Summarization microservice supports two deployment profiles: Summarization Base Profile and Summarization with Streaming and Message Bus. The base profile handles file summarization in-process; the streaming profile adds a Kafka message queue and a Logstash consumer service to decouple raw VLM events from the orchestrator and enables RTSP live-stream summarization.

Summarization Base Profile

Scenario 1 — File summarization, base profile. The client uploads a video file via REST or MCP. The Video Summarization microservice segments the video into chunks and forwards each chunk to the RTVI-VLM service for inference. RTVI-VLM returns per-chunk events synchronously over Server-Sent Events (SSE); the Video Summarization microservice calls the CA-RAG pipeline in-process to write events to Elasticsearch and aggregate them with the summarization LLM. The structured summary and timestamped events are returned in the HTTP response. The Kafka message queue is disabled in this profile (KAFKA_ENABLED=false).

Summarization with Streaming and Message Bus

In this profile a Kafka message queue carries raw VLM events and aggregated summaries between RTVI-VLM, the Video Summarization microservice, and Elasticsearch. A Logstash consumer service subscribes to the Kafka topics, decodes the protobuf payloads, normalizes their fields, and indexes the documents into Elasticsearch (default_<asset_id>). RTSP live-stream summarization is supported only in this profile.

Scenario 2 — File summarization with streaming and message bus. The client request is identical to Scenario 1, but RTVI-VLM publishes per-chunk raw events to the Kafka topic mdx-vlm-captions instead of returning them in-process. The Logstash consumer service indexes the events into Elasticsearch. The LVS_CAPTION_SOURCE environment variable controls where the Video Summarization microservice reads captions for aggregation: sse (default) uses the captions received in-process via the RTVI SSE response, while db retrieves captions from the Elasticsearch index populated by the Kafka → Logstash → ES pipeline. In both modes, after the RTVI-VLM SSE stream completes, the Video Summarization microservice waits kafka_consumer_settle_secs (default 5 seconds) so Logstash can flush the last chunk, then aggregates through CA-RAG and publishes the structured events and aggregated summary to the mdx-structured-events-summary topic for Logstash to index.

Scenario 3 — RTSP live-stream summarization with streaming and message bus. Live-stream summarization uses two dedicated APIs instead of the file-only POST /v1/summarize endpoint. The client first calls POST /v1/stream/add on RTVI-VLM with the RTSP URL (no inference yet). In phase 1, the client issues POST /v1/generate_captions on the Video Summarization microservice with the stream ID, model, and prompt parameters (scenario, events, objects_of_interest). The Video Summarization microservice builds the VLM prompt, calls RTVI-VLM to start captioning, and returns immediately with {status: "accepted"}. RTVI-VLM begins captioning frames continuously and publishes raw events to the Kafka message queue, which the Logstash consumer service indexes into Elasticsearch. In phase 2, the client calls POST /v1/stream_summarize with the stream ID and an optional start_time/end_time window. The Video Summarization microservice reads the captions from Elasticsearch, aggregates them via CA-RAG, publishes the structured events and aggregated summary to the mdx-structured-events-summary Kafka topic, and returns the summary synchronously.

Core Components

The Video Summarization microservice consists of the following core components:

• Video Summarization REST Server: FastAPI-based REST API server.
• Video Summarization MCP Server: Model Context Protocol server for AI agent integration.
• RTVI-VLM: Real-Time VLM service used for per-chunk and live-stream video inferencing.
• CA-RAG Pipeline: Context-aware retrieval and generation system that orchestrates event aggregation and summarization.
• LLM NIM: NVIDIA Inference Microservice for the summarization LLM.
• Data Store: Elasticsearch by default for storing extracted events, captions, and aggregated summaries, with Milvus, ArangoDB, and Neo4j available as alternative backends.
• Kafka Message Queue (Summarization with Streaming and Message Bus profile only): Carries raw VLM events on the mdx-vlm-captions topic and aggregated summaries on the mdx-structured-events-summary topic.
• Logstash Consumer Service (Summarization with Streaming and Message Bus profile only): Subscribes to the Kafka topics, decodes the nv.VisionLLM protobuf payloads, normalizes field types, and indexes the documents into Elasticsearch.

Getting Started

Prerequisites

• Docker and Docker Compose
• NVIDIA GPU(s) with appropriate drivers
• NGC API Key (for NVIDIA NIMs)

Migrating from 3.2.x to 3.3.0

Review the following changes before you reuse a 3.2.x deployment configuration with Video Summarization 3.3.0.

Update Required

• Replace KAFKA_TOPIC with MESSAGE_BUS_TOPIC in your .env file when you use the streaming profile. The default topic remains mdx-vlm-captions. MESSAGE_BUS_TOPIC configures the Logstash and RTVI-VLM services; the Video Summarization server does not read either variable.
• Update LVS_LLM_MODEL_NAME from the removed openai/gpt-oss-20b model to nvidia/nemotron-3.5-lightning-30b-a3b, or to another LLM that your deployment provides. The 3.3.0 deployment rejects the removed model name.
• If you use services/video-summarization/docker/deploy/compose.yaml with the rtvi profile, set VLM_MODEL_TO_USE explicitly. For a local model, also set MODEL_PATH. For an OpenAI-compatible endpoint, set VIA_VLM_ENDPOINT and its required API key. This standalone service Compose profile now rejects missing RTVI-VLM configuration at startup. The top-level blueprint profile under deploy/docker retains its openai-compat default.
• Replace LVS_ENABLE_LLM_MERGE with LVS_ENABLE_LLM_MERGING. The former name appeared in earlier documentation but is not read by the service.

New Environment Variables

• LVS_AGGREGATION_EMPTY_RETRIES sets the number of additional CA-RAG aggregation attempts when a response contains neither events nor a video summary. The default is 2. Set it to 0 to retain the 3.2.x single-attempt behavior.
• MESSAGE_BUS selects the RTVI-VLM message bus, and ERROR_BUS selects its error bus. Both default to kafka in the streaming profile.
• MESSAGE_BUS_TOPIC replaces KAFKA_TOPIC for the raw VLM event topic and defaults to mdx-vlm-captions.

The in-stack RTVI-VLM profile also forwards the new Adaptive EVS settings VIA_EVS_SESSION, VLM_VIDEO_PRUNING_RATE, VLLM_EVS_SIMILARITY_THRESHOLD, VIA_EVS_TOKEN_BUDGET, and VIA_EVS_MAX_SESSIONS. For supported models, defaults, and behavior, refer to Efficient Video Sampling (EVS and Adaptive EVS) in the Real-Time VLM documentation.

Removed Environment Variables

• KAFKA_TOPIC is no longer read by the 3.3.0 stack. Use MESSAGE_BUS_TOPIC.
• VSS_DISABLE_DECODER_REUSE is no longer read by the Video Summarization service. Remove it from carried-over configurations.

API Changes

• POST /v1/chat/completions adds question answering for assets that were summarized with enable_qa=true.
• DELETE /assets/{asset_id}/index adds an idempotent way to reclaim a retained per-asset Elasticsearch index. For automatic and manual cleanup behavior, refer to the Asset Management API.
• POST /summarize and POST /v1/summarize now return HTTP 422 when media_info.start_offset is greater than or equal to media_info.end_offset, or when min_tokens is greater than max_tokens. In 3.2.x, these requests could return HTTP 200 with an empty result.
• The development-only GET /files endpoint now returns HTTP 422 for an unsupported purpose value instead of HTTP 200 with an empty list.
• Responses omit media_info.start_offset and media_info.end_offset when either value is unavailable, nonnumeric, or not a strictly increasing range. Update clients that assume both keys are always present.
• POST /summarize and POST /v1/stream_summarize accept an inbound traceparent header and link the request to the caller’s distributed trace.

Functionality Changes

• Video Summarization now uses CA-RAG 3.1.0 instead of CA-RAG 3.0.1.
• Aggregation retries only when CA-RAG returns neither events nor a video summary. A response that has no events but has a summary is not retried.
• The service always forwards scenario and events to CA-RAG, including when enable_vlm_structured_output is enabled.
• The 3.3.0 RTVI-VLM profile supports optional Adaptive EVS video token pruning to reduce redundant visual input and per-chunk inference time.
• The Elasticsearch index retention default is unchanged. The Compose configuration retains per-asset indices by default, and the service drops them after file summarization when LVS_DISABLE_DB_RESET_ON_REQUEST_DONE=false. The new asset-management API provides explicit cleanup for retained indices.
• The RTVI-VLM client closes the caption-stream HTTP response after processing to prevent a long-running file descriptor and memory leak.

Deployment

Clone the Repository

git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git
cd video-search-and-summarization
git checkout "v3.3.0"
cd services/video-summarization

Build the Container Image

The Video Summarization compose stack uses a locally built image (via-engine-<username>). Build the image before bringing up the stack.

Use the ARM_PLATFORM variable to select the ARM64 build variant.

x86 (amd64) build:

The following command is for x86 (amd64) architecture only:

make -C docker build

DGX Spark / SBSA / Grace (server-class ARM64) build:

Set ARM_PLATFORM=sbsa to build the SBSA image variant. This is the variant required on DGX Spark (GB10):

ARM_PLATFORM=sbsa make -C docker build

Jetson / IGX-AGX Thor / Tegra (ARM64 iGPU) build:

Set ARM_PLATFORM=igpu to build the integrated-GPU ARM64 image variant:

ARM_PLATFORM=igpu make -C docker build

For the full developer workflow, deploy the Video Summarization profile as described in Video Summarization agent workflow. Video Summarization microservice is deployed via the docker-compose stack at docker/deploy/compose.yaml. Two deployment profiles are supported:

• Summarization Base Profile — file summarization without a Kafka message queue.
• Summarization with Streaming and Message Bus — file and RTSP live-stream summarization with a Kafka message queue and Logstash consumer service.

Use standalone deployment when you need to run only the Video Summarization server against existing Elasticsearch, LLM, and VLM services. A slimmer single-container option is documented under Standalone Container (Advanced) for operators who already run RTVI-VLM, the LLM NIM, and Elasticsearch externally.

Summarization Base Profile

This profile deploys Video Summarization microservice, RTVI-VLM, and Elasticsearch with the Kafka message queue disabled. Use it for file summarization workloads.

Prerequisites

• Docker Engine 28.3.3+ and Docker Compose v2.39.1+
• NVIDIA Container Toolkit
• NVIDIA GPU(s) with appropriate drivers
• External LLM NIM service (for summarization) — configured via LVS_LLM_HOST and LVS_LLM_PORT

Environment Setup

Create a .env file from the variables below. The standalone Compose stack does not ship a committed .env. For an integrated VSS deployment, use the lvs developer profile and the native Docker Compose procedure in Long Video Summarization.

Edit .env and set the following key variables:

# API Keys
NGC_API_KEY=<your-ngc-api-key>
NGC_CLI_API_KEY=<your-ngc-api-key>
NVIDIA_API_KEY=<your-nvidia-api-key>
OPENAI_API_KEY=<your-openai-api-key>   # required when RTVI uses openai-compat
# Host REST API port (container port is fixed at 38111)
LVS_BACKEND_PORT=38111
# The standalone Compose stack publishes MCP on fixed host/container port 38112.
# Database (in-stack Elasticsearch)
ES_HOST=elasticsearch
ES_PORT=9200
# LLM (external NIM)
LVS_LLM_HOST=<llm-nim-host-ip>
LVS_LLM_PORT=8002
# Query the configured NIM's /v1/models endpoint and copy its exact id here.
# Model IDs can differ between NIM versions and serving configurations.
LVS_LLM_MODEL_NAME=<id-returned-by-llm-nim-v1-models>
LVS_DATABASE_BACKEND=elasticsearch_db
# RTVI-VLM Integration
COMPOSE_PROFILES=rtvi
# Multi-arch tag, covering x86 and Jetson Thor (iGPU) platforms:
RTVI_VLM_IMAGE=nvcr.io/nvidia/vss-core/vss-rt-vlm:3.3.0
# For DGX Spark / SBSA / Grace (server-class ARM) platforms, use the -sbsa tag:
#RTVI_VLM_IMAGE=nvcr.io/nvidia/vss-core/vss-rt-vlm:3.3.0-sbsa
RTVI_VLM_PORT=8420
RTVI_VLM_GPU=1
# RTVI_VLM_URL=http://<external-rtvi-host>:8083  # set only when pointing at an external RTVI-VLM
VLM_MODEL_TO_USE=openai-compat
VIA_VLM_ENDPOINT=<openai-compatible-vlm-endpoint>
# Kafka message queue disabled for the base profile
KAFKA_ENABLED=false
# Feature Flags
LVS_ENABLE_MCP=true
ENABLE_VIA_HEALTH_EVAL=false
# Logging
VSS_LOG_LEVEL=DEBUG

Bring Up the Stack

Before starting the Video Summarization MS, query the external LLM NIM and copy its exact model ID into LVS_LLM_MODEL_NAME in .env. Replace the URL placeholders with the LVS_LLM_HOST and LVS_LLM_PORT values configured above:

curl -fsS "http://<llm-nim-host-ip>:<llm-nim-port>/v1/models" | python3 -m json.tool

The Compose --env-file option does not export variables into your shell. Restart the Video Summarization MS after changing its configured model ID.

docker compose -f docker/deploy/compose.yaml --profile rtvi --env-file .env up -d

Verify Service Health

curl http://localhost:38111/v1/ready

Before submitting a request, retrieve the deployed RTVI-VLM model ID for the request’s model field. Replace the port placeholder with the RTVI_VLM_PORT value configured in .env (8420 in this example):

curl -fsS "http://localhost:<rtvi-vlm-host-port>/v1/models" | python3 -m json.tool

Test the API

Submit a sample file summarization request:

curl --location 'http://localhost:38111/v1/summarize' \
  --header 'Content-Type: application/json' \
  --data '{
    "url": "http://<video-server-ip>:<port>/your-video.mp4",
    "model": "<id-returned-by-rtvi-vlm-v1-models>",
    "scenario": "law enforcement",
    "chunk_duration": 15,
    "events": ["pulling over", "arrest", "chasing"]
  }' | python3 -m json.tool

Replace <video-server-ip> and <port> with your video server address.

Tear Down the Stack

docker compose -f docker/deploy/compose.yaml --profile rtvi down

Summarization with Streaming and Message Bus

This profile adds a Kafka message queue and a Logstash consumer service to the base profile. It is required for RTSP live-stream summarization, and is also recommended for file summarization when end-to-end Kafka publish-back is desired (for example, to feed downstream consumers off the mdx-structured-events-summary topic).

Prerequisites

• Same as the base profile, plus the host ports for the in-stack Kafka broker (KAFKA_PORT and KAFKA_EXTERNAL_PORT) must be free.

Environment Setup

Source the values from .env for this profile. The diff from the base profile is:

# Enable the Kafka message queue
KAFKA_ENABLED=true
KAFKA_BOOTSTRAP_SERVERS=kafka:9092
MESSAGE_BUS_TOPIC=mdx-vlm-captions
KAFKA_STRUCTURED_SUMMARY_TOPIC=mdx-structured-events-summary
# Host-side broker ports — the compose defaults are 9092/9094.
# Override these to avoid collisions with other Kafka stacks on the same host.
KAFKA_PORT=9192               # default: 9092
KAFKA_EXTERNAL_PORT=9194      # default: 9094
KAFKA_ADVERTISED_HOST=localhost
# Recommended for multi-call live-stream sessions: keeps the per-stream
# context manager state intact across consecutive /v1/stream_summarize calls.
LVS_DISABLE_DB_RESET_ON_REQUEST_DONE=true

The checked-in services/video-summarization/config/config.yaml already reads KAFKA_ENABLED from the environment and sets kafka_consumer_settle_secs: 5.0. Compose mounts that file at /opt/nvidia/via/config/default_config.yaml in the container, so the .env settings above are sufficient for the standard Kafka profile. Edit the YAML only when you need advanced tuning; for example, the relevant defaults are:

tools:
  elasticsearch_db:
    type: elasticsearch
    params:
      host: !ENV ${ES_HOST}
      port: !ENV ${ES_PORT}
      kafka_consumer_settle_secs: 5.0
functions:
  summarization:
    type:
      vlm_structured_summarization
    params:
      kafka_enabled: !ENV ${KAFKA_ENABLED:false}
  summarization_online:
    type:
      vlm_structured_summarization_online
    params:
      kafka_enabled: !ENV ${KAFKA_ENABLED:false}

Bring Up the Stack

docker compose -f docker/deploy/compose.yaml --profile rtvi --profile kafka up -d

Live-Stream Operator Workflow

1. Register an RTSP stream on RTVI-VLM (no inference yet):

ASSET_ID=$(curl -fsS -X POST http://localhost:8420/v1/stream/add \
  -H 'Content-Type: application/json' \
  -d '{
    "key": "sensor",
    "value": {
      "camera_id": "",
      "camera_url": "rtsp://<rtsp-source>/stream",
      "change": "camera_add"
    }
  }' \
  | python3 -c "import sys,json; print(json.load(sys.stdin).get('asset_id'))")

2. Start VLM captioning on the stream (fire-and-forget). This is where prompt-related parameters such as scenario, events, and objects_of_interest are passed:

curl -fsS -X POST http://localhost:38111/v1/generate_captions \
  -H 'Content-Type: application/json' \
  -d "{
    \"id\": \"$ASSET_ID\",
    \"model\": \"<id-returned-by-rtvi-vlm-v1-models>\",
    \"scenario\": \"warehouse safety monitoring\",
    \"events\": [\"box dropping\", \"unsafe forklift operations\", \"normal activity\"],
    \"chunk_duration\": 10
  }"

3. Summarize the stream over a time window. No prompt-related parameters are needed — the Video Summarization microservice reads the captions already stored by the captioning phase:

curl -fsS -X POST http://localhost:38111/v1/stream_summarize \
  -H 'Content-Type: application/json' \
  -d "{
    \"id\": \"$ASSET_ID\",
    \"model\": \"<id-returned-by-rtvi-vlm-v1-models>\",
    \"start_time\": 0,
    \"end_time\": 0
  }"

4. Repeat the /v1/stream_summarize call as needed with progressively larger or narrower time windows to get the latest aggregated events and summary.

Tear Down the Stack

docker compose -f docker/deploy/compose.yaml --profile rtvi --profile kafka down --remove-orphans

This preserves existing named cache and plugin volumes. The standalone Compose stack does not configure persistent data volumes for Elasticsearch or Kafka; removing their containers can lose their data even without -v. Configure persistent storage and back up any required data before teardown.

Add -v only as an optional destructive cleanup step: it also removes the Compose-managed named volumes and attached anonymous volumes. Anonymous volumes are not automatically reattached when the stack is brought up again.

Standalone Container (Advanced)

This deployment runs the Video Summarization MS server as a single Docker container, assuming external services (RTVI-VLM, Elasticsearch, LLM NIM) are already running. The compose-based profiles above are recommended for most operators; use this only when you already run all dependencies separately.

Prerequisites

1. Copy the example environment file to .env:

touch .env

2. Edit .env and fill in your configuration values:

# Environment variables for standalone Video Summarization Server docker run
# Copy this file to .env and fill in the values
# Container Configuration
# Multi-arch tag, covering x86 and Jetson Thor (iGPU) platforms:
CONTAINER_IMAGE=nvcr.io/nvidia/vss-core/vss-video-summarization:3.3.0
# For DGX Spark / SBSA / Grace (server-class ARM) platforms, use the -sbsa tag:
#CONTAINER_IMAGE=nvcr.io/nvidia/vss-core/vss-video-summarization:3.3.0-sbsa
# API Keys and Authentication
NGC_API_KEY=<your-ngc-api-key>
NVIDIA_API_KEY=<your-nvidia-api-key>
# OPENAI_API_KEY=<your-openai-api-key>
# S3 Configuration (required for S3 URLs)
# AWS_ACCESS_KEY_ID=<your-aws-access-key-id>
# AWS_SECRET_ACCESS_KEY=<your-aws-secret-access-key>
# AWS_ENDPOINT_URL_S3=<your-s3-endpoint-url>
# Port Configuration
BACKEND_PORT=38111
BACKEND_HOST_PORT=38111
LVS_MCP_PORT=38112
LVS_MCP_HOST_PORT=38112
# CA RAG Configuration
# This will be set via mount path - do not override here
# CA_RAG_CONFIG=/opt/nvidia/via/config/default_config.yaml
# Feature Flags
ENABLE_VIA_HEALTH_EVAL=false
LVS_ENABLE_MCP=true
# Database Configuration - Elasticsearch
# Update these to point to your external Elasticsearch service
ES_HOST=<elasticsearch-host-ip>
ES_PORT=9202
ES_TRANSPORT_PORT=9302
# RTVI-VLM Integration (point at an externally running RTVI-VLM)
RTVI_VLM_URL=http://<rtvi-vlm-host>:8000
VLM_MODEL_TO_USE=openai-compat
# OpenTelemetry Configuration (optional)
# VIA_ENABLE_OTEL=false
# VIA_OTEL_ENDPOINT=http://localhost:4318
# VIA_OTEL_EXPORTER=console
# VIA_CTX_RAG_ENABLE_OTEL=false
# VIA_CTX_RAG_EXPORTER=console
# VIA_CTX_RAG_OTEL_ENDPOINT=http://localhost:4318
# Logging and Debug
VSS_LOG_LEVEL=DEBUG
# LLM Configuration
# Update these to point to your external LLM NIM services
# Query LVS_LLM_BASE_URL/models and use the exact returned id.
LVS_LLM_MODEL_NAME=<id-returned-by-llm-nim-v1-models>
LVS_LLM_BASE_URL=http://<llm-nim-host-ip>:8002/v1
# Database Selection
LVS_DATABASE_BACKEND=elasticsearch_db
# Kafka message queue (disabled in standalone mode)
KAFKA_ENABLED=false

Run Script

Create a file named run-video-summarization-server.sh with the following content:

#!/bin/bash
# Standalone Docker run command for Video Summarization Server
# This script runs only the Video Summarization server container, assuming other services are running separately
# Configuration
CONTAINER_NAME="video-summarization-server"
ENV_FILE="${ENV_FILE:-.env}"
# Check if .env file exists
if [ ! -f "$ENV_FILE" ]; then
    echo "Error: Environment file '$ENV_FILE' not found!"
    echo "Please copy .env to $ENV_FILE and fill in the values."
    exit 1
fi
# Load CONTAINER_IMAGE from env file
IMAGE=$(grep "^CONTAINER_IMAGE=" "$ENV_FILE" | cut -d'=' -f2)
if [ -z "$IMAGE" ]; then
    echo "Error: CONTAINER_IMAGE not found in $ENV_FILE"
    echo "Set CONTAINER_IMAGE (e.g., via-engine-\$USER after running make -C docker build)."
    exit 1
fi
# Load MODEL_ROOT_DIR from env file (optional)
MODEL_ROOT_DIR=$(grep "^MODEL_ROOT_DIR=" "$ENV_FILE" | cut -d'=' -f2)
# Load port values from env file
BACKEND_PORT=$(grep "^BACKEND_PORT=" "$ENV_FILE" | cut -d'=' -f2)
BACKEND_HOST_PORT=$(grep "^BACKEND_HOST_PORT=" "$ENV_FILE" | cut -d'=' -f2)
LVS_MCP_PORT=$(grep "^LVS_MCP_PORT=" "$ENV_FILE" | cut -d'=' -f2)
LVS_MCP_HOST_PORT=$(grep "^LVS_MCP_HOST_PORT=" "$ENV_FILE" | cut -d'=' -f2)
# Set defaults if not found
BACKEND_PORT=${BACKEND_PORT:-38111}
LVS_MCP_PORT=${LVS_MCP_PORT:-38112}
# Build port mapping arguments. Override the host-side ports independently
# when the defaults are already in use.
BACKEND_HOST_PORT=${BACKEND_HOST_PORT:-$BACKEND_PORT}
LVS_MCP_HOST_PORT=${LVS_MCP_HOST_PORT:-$LVS_MCP_PORT}
PORT_ARGS=(
    -p "${BACKEND_HOST_PORT}:${BACKEND_PORT}"
    -p "${LVS_MCP_HOST_PORT}:${LVS_MCP_PORT}"
)
# Build volume mount for MODEL_ROOT_DIR if set
MODEL_VOLUME_ARG=""
if [ -n "$MODEL_ROOT_DIR" ]; then
    # Expand tilde and get absolute path if directory exists or can be created
    MODEL_ROOT_DIR_EXPANDED="${MODEL_ROOT_DIR/#\~/$HOME}"
    if [ -d "$MODEL_ROOT_DIR_EXPANDED" ] || mkdir -p "$MODEL_ROOT_DIR_EXPANDED" 2>/dev/null; then
        MODEL_ROOT_DIR_ABS="$(cd "$MODEL_ROOT_DIR_EXPANDED" && pwd)"
        MODEL_VOLUME_ARG="-v ${MODEL_ROOT_DIR_ABS}:${MODEL_ROOT_DIR_ABS}"
        echo "MODEL_ROOT_DIR will be mounted: $MODEL_ROOT_DIR_ABS"
    else
        echo "Warning: MODEL_ROOT_DIR '$MODEL_ROOT_DIR' could not be accessed or created, skipping mount"
    fi
else
    echo "MODEL_ROOT_DIR not set in $ENV_FILE, skipping model cache mount"
fi
# Docker run command. Use addresses reachable from this container for
# Elasticsearch, RTVI-VLM, and the LLM NIM. Compose-only service names such as
# `elasticsearch` and `rtvi-vlm` do not resolve from this standalone container;
# use a routable host/IP, `host.docker.internal` for host-published services, or
# attach the container to the dependencies' Docker network explicitly.
docker run -d \
    --name "$CONTAINER_NAME" \
    --add-host host.docker.internal:host-gateway \
    "${PORT_ARGS[@]}" \
    --env-file "$ENV_FILE" \
    $MODEL_VOLUME_ARG \
    --restart unless-stopped \
    "$IMAGE"
echo "Video Summarization Server container started!"
echo "Container name: $CONTAINER_NAME"
echo "Container image: $IMAGE"
echo "Backend port: $BACKEND_PORT"
echo "MCP port: $LVS_MCP_PORT"
if [ -n "$MODEL_ROOT_DIR_ABS" ]; then
    echo "Model cache mounted from: $MODEL_ROOT_DIR_ABS"
fi
echo ""
echo "To view logs: docker logs -f $CONTAINER_NAME"
echo "To stop: docker stop $CONTAINER_NAME"
echo "To remove: docker rm $CONTAINER_NAME"

Running the Container

1. Make the script executable:

chmod +x run-video-summarization-server.sh

2. Run the script:

./run-video-summarization-server.sh

3. Check the container status:

docker logs -f video-summarization-server

4. Verify service health:

curl http://localhost:38111/v1/ready

Testing the API

Test video summarization with a sample request:

curl --location 'http://localhost:38111/v1/summarize' \
  --header 'Content-Type: application/json' \
  --data '{
  "id": null,
  "url": "http://<video-server-ip>:<port>/your-video.mp4",
  "model": "<id-returned-by-rtvi-vlm-v1-models>",
  "scenario": "law enforcement",
  "events": ["pulling over", "arrest", "chasing"]
}' | python3 -m json.tool

Note: Replace <video-server-ip> and <port> with your actual video server address and port.

Stopping the Container

# Stop the container
docker stop video-summarization-server
# Remove the container
docker rm video-summarization-server

Configuration Updates

To update environment variables:

1. Stop and remove the container:

docker stop video-summarization-server && docker rm video-summarization-server

2. Edit your .env file with the new values

3. Restart the container:

./run-video-summarization-server.sh

Network Configuration

The script runs the container on Docker’s default bridge network and publishes the REST API and MCP ports. Compose service names such as elasticsearch and rtvi-vlm do not resolve from this container, so set ES_HOST, RTVI_VLM_URL, and LVS_LLM_BASE_URL in .env to addresses it can reach: a routable host or IP, or host.docker.internal for services published on this host. To use service names instead, attach the container to the dependencies’ Docker network.

Troubleshooting

• Container fails to start: Check docker logs video-summarization-server for error messages
• Cannot connect to external services: Verify RTVI_VLM_URL, ES_HOST/ES_PORT, and LVS_LLM_BASE_URL in your .env file and confirm network connectivity
• Port conflicts: Change port numbers in .env file

Configuration

Video Summarization uses YAML configuration files to customize behavior. The main configuration file is mounted in the Video Summarization container and referenced by CA_RAG_CONFIG.

Configuration Structure

The configuration file (services/video-summarization/config/config.yaml, mounted at /opt/nvidia/via/config/default_config.yaml) defines:

1. Tools - External service connections:

tools:
  elasticsearch_db:
    type: elasticsearch
    params:
      host: !ENV ${ES_HOST}
      port: !ENV ${ES_PORT}
    tools:
      embedding: nvidia_embedding
  summarization_llm:
    type: llm
    params:
      model: !ENV ${LVS_LLM_MODEL_NAME}
      base_url: !ENV ${LVS_LLM_BASE_URL}
      max_tokens: 10240
      temperature: 0.2
      top_p: 0.7
      api_key: !ENV ${NVIDIA_API_KEY}
  nvidia_embedding:
    type: embedding
    params:
      enable: !ENV ${LVS_EMB_ENABLE:false}
      model: !ENV ${LVS_EMB_MODEL_NAME}
      base_url: !ENV ${LVS_EMB_BASE_URL}
      api_key: !ENV ${NVIDIA_API_KEY}

2. Functions - Processing pipelines:

functions:
  summarization:
    type: vlm_structured_summarization_online
    params:
      time_overlap_threshold: 0.1
      max_events_per_batch: 50
      kafka_enabled: true
    tools:
      db: !ENV ${LVS_DATABASE_BACKEND:elasticsearch_db}
      llm: summarization_llm
  summarization_online:
    type: vlm_structured_summarization_online
    params:
      time_overlap_threshold: 0.1
      max_events_per_batch: 50
      kafka_enabled: true
    tools:
      db: !ENV ${LVS_DATABASE_BACKEND:elasticsearch_db}
      llm: summarization_llm

3. Context Manager - Active functions:

context_manager:
  functions:
    - summarization
    - summarization_online

Key Configuration Options

Database

• type: elasticsearch (default), milvus, arango, or neo4j — see Database Backends
• params.host, params.port: Connection parameters
• params.username, params.password: Credentials, for the arango and neo4j types
• tools.embedding: Embedding model to use (reference to tool)

Summarization LLM

• type: llm (default)
• params.model: LLM model name (e.g., from ${LVS_LLM_MODEL_NAME})
• params.base_url: API endpoint (e.g., from ${LVS_LLM_BASE_URL})
• params.max_tokens: Maximum output tokens (default: 10240)
• params.temperature: Sampling temperature (default: 0.2)
• params.top_p: Top-p sampling (default: 0.7)
• params.api_key: Authentication key

Nvidia Embedding

• type: embedding (default)
• params.enable: Enable/disable embedding service
• params.model: Embedding model name
• params.base_url: Embedding service endpoint
• params.api_key: NVIDIA API key

VLM Structured Summarization

Used by the file summarization path (POST /v1/summarize). Configured as the summarization function in the context manager.

• type: Function type (vlm_structured_summarization (default))
• params.time_overlap_threshold: Threshold for overlapping time events (default: 0.1)
• params.max_events_per_batch: Maximum events per batch (default: 50)
• params.kafka_enabled: When true, the file path reads raw events from Elasticsearch (populated by the Kafka → Logstash pipeline) instead of using the in-process accumulation flow. Must be paired with KAFKA_ENABLED=true on the Video Summarization MS server. Default: false (reads from ${KAFKA_ENABLED})
• tools.db: Reference to database backend (from ${LVS_DATABASE_BACKEND:elasticsearch_db})
• tools.llm: Reference to LLM tool

VLM Structured Summarization Online

Used by the live-stream summarization path (POST /v1/stream_summarize). Configured as the summarization_online function in the context manager and must be registered alongside summarization for live-stream workflows to dispatch correctly.

• type: Function type (vlm_structured_summarization_online)
• params.time_overlap_threshold: Threshold for overlapping time events (default: 0.1)
• params.max_events_per_batch: Maximum events per batch (default: 50)
• params.kafka_enabled: When true, the aggregator reads raw events back from Elasticsearch (populated by the Logstash consumer) rather than expecting in-process accumulation. Required for the live-stream path. Must be paired with KAFKA_ENABLED=true on the Video Summarization MS server. Default: false (reads from ${KAFKA_ENABLED})
• tools.db: Reference to database backend (from ${LVS_DATABASE_BACKEND:elasticsearch_db})
• tools.llm: Reference to LLM tool

Example: Event Detection

The default configuration is optimized for event detection in videos:

functions:
  summarization:
    type: vlm_structured_summarization
    params:
      time_overlap_threshold: 0.1
      max_events_per_batch: 50
      kafka_enabled: !ENV ${KAFKA_ENABLED:false}
    tools:
      db: !ENV ${LVS_DATABASE_BACKEND:elasticsearch_db}
      llm: summarization_llm
  summarization_online:
    type: vlm_structured_summarization_online
    params:
      time_overlap_threshold: 0.1
      max_events_per_batch: 50
      kafka_enabled: !ENV ${KAFKA_ENABLED:false}
    tools:
      db: !ENV ${LVS_DATABASE_BACKEND:elasticsearch_db}
      llm: summarization_llm

This extracts structured events with timestamps. You can customize this for specific use cases:

Event Merging

When the same real-world event spans multiple video chunks, it can appear as several adjacent entries of the same type with overlapping or contiguous timestamps. The LVS_ENABLE_LLM_MERGING flag controls how these are combined.

Without LLM merge (default)

Adjacent events of the same type are concatenated with a | separator. Each chunk’s description is preserved verbatim:

{
  "id": 4,
  "type": "ladder use",
  "start": 7.0,
  "end": 29.33,
  "description": "The individual approaches the ladder, suggesting intent to use it for accessing higher shelves; however, no climbing action occurs within the observed timeframe. | A worker wearing a yellow hard hat and high-visibility vest approaches a green rolling step ladder positioned between two aisles labeled D and E. The worker climbs the ladder cautiously while holding a cardboard box, ascends to the top platform, places the box securely on the upper shelf of aisle E, and then begins to descend back down the ladder. | A worker wearing a yellow hard hat and reflective vest stands on a green rolling step ladder positioned between two tall warehouse aisles (D and E). The worker reaches up to adjust or interact with items stored high on shelf D, then lowers their hands while maintaining balance."
}

With LLM merge (LVS_ENABLE_LLM_MERGING=true)

The LLM reads all adjacent descriptions and produces a single, coherent natural-language summary:

{
  "id": 2,
  "type": "ladder use",
  "start": 10.0,
  "end": 40.0,
  "description": "A worker wearing a yellow hard hat and high-visibility vest climbs a green metal rolling step ladder positioned between shelves D and E in a well-lit industrial warehouse, ascending to the second-level platform to retrieve a white box labeled \"A110\" and prepare it for placement on the top shelf above."
}

Use LLM merge when clean, human-readable event descriptions are more important than preserving the raw per-chunk output. Note that it adds one LLM inference call per merged group, which slightly increases end-to-end latency.

Database Backends

The configuration defines four database tools. Any of them can be selected as the store for captions, events, and aggregated summaries.

Two independent selectors choose between them:

• LVS_DATABASE_BACKEND (default elasticsearch_db) selects the store used by the summarization and summarization_online functions — the file and live-stream summarization paths.
• LVS_QA_DATABASE_BACKEND (default graph_db) selects the store used by ingestion_function for the question-answering path, together with LVS_INGESTION_FUNCTION (default graph_ingestion) and LVS_RETRIEVER_FUNCTION.

Every database tool references the nvidia_embedding tool, which is disabled by default (LVS_EMB_ENABLE=false). The Milvus, ArangoDB, and Neo4j backends depend on embeddings, so enable LVS_EMB_ENABLE and set LVS_EMB_MODEL_NAME and LVS_EMB_BASE_URL when you select one of them.

Environment Variables

Configuration supports environment variable substitution using !ENV ${VAR_NAME}. For the complete list, see the Environment Variables Reference in the appendix.

Kafka Integration

When deployed under the Summarization with Streaming and Message Bus profile, the Video Summarization microservice publishes aggregated summarization results to a Kafka message queue and reads per-chunk raw VLM events that RTVI-VLM publishes to the same broker. A Logstash consumer service decodes the protobuf payloads from Kafka and indexes them into Elasticsearch (default_<asset_id>). This enables decoupling of the summarization pipeline from downstream consumers, analytics dashboards, and real-time alerting systems.

Kafka Topics

The streaming and message bus profile uses two Kafka topics:

• Raw VLM Events (default: mdx-vlm-captions): Contains nv.VisionLLM protobuf messages produced by RTVI-VLM. Each message carries one chunk’s caption result with frame metadata, sensor info, and the info["doc_type"]="raw_events" marker. After the Logstash consumer service decodes and indexes these messages into Elasticsearch, the doc_type value is stored at metadata.content_metadata.doc_type in the ES document.
• Structured Summary (default: mdx-structured-events-summary): Contains nv.VisionLLM protobuf messages produced by the Video Summarization microservice. Each call to POST /v1/summarize (file path) or POST /v1/stream_summarize (live-stream path) publishes one info["doc_type"]="structured_events" message per batch (up to max_events_per_batch events) plus one info["doc_type"]="aggregated_summary" message carrying the narrative summary. In Elasticsearch, the Logstash-mapped field path is metadata.content_metadata.doc_type.

Configuration:

Kafka integration is controlled by the following environment variables across the Video Summarization stack:

• KAFKA_ENABLED: Enable or disable Kafka integration (true / false). Default: false
• KAFKA_BOOTSTRAP_SERVERS: Comma-separated list of Kafka broker addresses (e.g., localhost:9092 or kafka:9092 for the in-stack broker)
• MESSAGE_BUS_TOPIC: Topic for raw VLM events produced by RTVI-VLM and consumed by Logstash. Default: mdx-vlm-captions
• KAFKA_STRUCTURED_SUMMARY_TOPIC: Topic for structured events and aggregated summaries. Default: mdx-structured-events-summary
• KAFKA_PORT: Host port mapping for the in-stack Kafka broker’s internal listener. Default: 9092
• KAFKA_EXTERNAL_PORT: Host port mapping for the broker’s external listener; also embedded in KAFKA_ADVERTISED_LISTENERS. Default: 9094
• KAFKA_ADVERTISED_HOST: Hostname or IP that host-side or cross-host clients use to reconnect. Default: localhost

RTVI-VLM uses MESSAGE_BUS, not KAFKA_ENABLED, to select its generated-message output bus. Set MESSAGE_BUS=kafka and KAFKA_BOOTSTRAP_SERVERS on the RTVI-VLM service so it publishes raw events to the same broker. In the packaged VSS Compose deployment, set RTVI_VLM_MESSAGE_BUS=kafka; keep RTVI_VLM_KAFKA_ENABLED=true only as the legacy compatibility field while Compose still forwards it. See RTVI-VLM Kafka Integration for the producer-side configuration.

ctx-rag function configuration:

The standard Compose workflow mounts services/video-summarization/config/config.yaml at /opt/nvidia/via/config/default_config.yaml. That configuration already reads KAFKA_ENABLED for both summarization functions, so prefer the .env setting above. The following YAML is shown only for advanced tuning:

tools:
  elasticsearch_db:
    type: elasticsearch
    params:
      host: !ENV ${ES_HOST}
      port: !ENV ${ES_PORT}
      kafka_consumer_settle_secs: 5.0   # seconds the Video Summarization MS waits after the
                                         # RTVI SSE [DONE] event to let
                                         # Logstash flush the last chunk
functions:
  summarization:
    params:
      kafka_enabled: !ENV ${KAFKA_ENABLED:false}
  summarization_online:
    params:
      kafka_enabled: !ENV ${KAFKA_ENABLED:false}

How Summaries Are Sent to Kafka

Summarization responses do not report Kafka delivery or Elasticsearch indexing status. HTTP 200 indicates successful summary generation, not confirmation that Kafka accepted the messages or that Logstash indexed them. Kafka producer initialization failures can disable publishing, and send or delivery failures can be logged without failing the summarization request. A successful SSE response likewise does not confirm downstream delivery.

Inspect the Video Summarization MS producer logs, verify the expected messages in Kafka, and check Logstash errors and indexed Elasticsearch documents before treating the archive workflow as complete.

Under the streaming and message bus profile, the publishing flow depends on the API path:

1. File path (POST /v1/summarize): RTVI-VLM publishes per-chunk raw events to MESSAGE_BUS_TOPIC as it processes the file. The LVS_CAPTION_SOURCE environment variable controls where the Video Summarization microservice reads captions for aggregation: sse (default) uses captions received in-process via the RTVI SSE response, while db retrieves them from the Elasticsearch index populated by the Kafka → Logstash → ES pipeline. In db mode, the Video Summarization microservice waits kafka_consumer_settle_secs (default 5 seconds) so the Logstash consumer service can flush the last chunk. In both modes, the Video Summarization microservice aggregates captions via CA-RAG and publishes the structured events and aggregated summary to KAFKA_STRUCTURED_SUMMARY_TOPIC.
2. Live-stream path (POST /v1/generate_captions + POST /v1/stream_summarize): The client first calls /v1/generate_captions to start VLM captioning (fire-and-forget). RTVI-VLM publishes raw events to the Kafka topic, and Logstash indexes them into Elasticsearch. The client then calls /v1/stream_summarize with an optional [start_time, end_time] window. The Video Summarization microservice reads captions from Elasticsearch, aggregates them via CA-RAG, and publishes the structured events and summary to KAFKA_STRUCTURED_SUMMARY_TOPIC with a per-call upsert ID so repeated calls overwrite the latest result for that stream.
3. Message keys: All messages are published with the key {request_id}:{chunk_idx|batch_i|doc_i} (UTF-8 encoded) for partitioning and ordering.
4. Requirement: Set KAFKA_ENABLED=true on the Video Summarization MS. On RTVI-VLM, set MESSAGE_BUS=kafka; for the packaged VSS Compose deployment, the corresponding variable is RTVI_VLM_MESSAGE_BUS=kafka alongside the legacy RTVI_VLM_KAFKA_ENABLED=true compatibility field. Configure KAFKA_BOOTSTRAP_SERVERS for both services. Without the Video Summarization setting or RT-VLM message-bus publishing, /v1/summarize silently falls back to the legacy in-process aggregation flow. POST /v1/summarize now rejects source_type=stream with HTTP 422, directing clients to use the dedicated live-stream APIs instead.

Video Summarization MS to RTVI Sticky Routing (x-stream-id)

Video Summarization MS to RTVI Sticky Routing (x-stream-id)

The Video Summarization microservice attaches an x-stream-id HTTP header on every outbound request to RTVI-VLM. The header value is the asset or stream ID for the request (for example, the UUID returned by POST /v1/stream/add). Load balancers or service meshes sitting in front of RTVI-VLM can use this header to route all requests that belong to the same stream to the same RTVI-VLM instance, preventing split-brain state in multi-instance deployments.

The header is sent on the following Video Summarization MS to RTVI call paths:

• start_captions — POST /v1/generate_captions (live-stream captioning trigger)
• generate_captions_stream — POST /v1/generate_captions with stream=True (file-path captioning)

Message Formats

Both topics carry nv.VisionLLM protobuf messages. The info map (map<string, string>) is used to distinguish doc types and carry per-stream metadata for the Logstash consumer service.

Raw VLM Events

Raw events are produced by RTVI-VLM, one per video chunk. See RTVI-VLM VisionLLM Messages for the complete VisionLLM protobuf schema. Video Summarization MS-relevant info fields injected by RTVI-VLM include:

• doc_type: "raw_events"
• collection_name: Target Elasticsearch index (default_<asset_id>)
• uuid / streamId: Asset identifier for the source video or live stream
• chunkIdx: Zero-indexed chunk number within the request
• start_pts / end_pts: Chunk start and end timestamps in milliseconds
• start_ntp / end_ntp: Chunk start and end NTP timestamps (ISO 8601)
• is_first / is_last: Boolean markers for chunk boundary in the SSE stream

Structured Events and Aggregated Summary

Structured-summary messages are produced by the Video Summarization microservice after CA-RAG aggregation. The same nv.VisionLLM protobuf carries two distinct doc_type values:

• doc_type=structured_events: One message per batch of up to max_events_per_batch aggregated events. The llm.queries[0].response field carries a JSON-encoded array of event objects (id, start_time, end_time, type, description).
• doc_type=aggregated_summary: A single message per summarization call (/v1/summarize for files, /v1/stream_summarize for live streams) carrying the narrative video_summary text in llm.queries[0].response.

Video Summarization MS-injected info fields on every published message:

• doc_type: "structured_events" or "aggregated_summary"
• collection_name: default_<sanitized stream_id> — drives the Elasticsearch _index Logstash writes to
• uuid: Source asset or stream identifier
• camera_id: When set on the live stream, copied from the operator’s /v1/stream/add payload
• batch_i: Zero-indexed batch number for structured_events messages * event_count / total_events: Per-batch and per-call event counts ID for upsert: Logstash builds the Elasticsearch document _id deterministically as <collection>:<uuid>:<doc_type>:<chunkIdx|batch_i|doc_i>. Repeated /v1/summarize calls on the same stream therefore overwrite the same two ES documents (one structured_events per batch, one aggregated_summary) — this is intentional for at-least-once Kafka redelivery safety.

MCP Server Integration

Video Summarization includes a Model Context Protocol (MCP) server that exposes the same functionality as the REST API in a format consumable by AI agents and tools like Claude Desktop, Cursor, and other MCP-compatible clients.

Overview

The MCP server (lvs_mcp.py) provides:

• Stdio Transport: Default mode for direct integration with MCP clients
• SSE Transport: HTTP-based Server-Sent Events transport for network access
• Tool-based Interface: All REST endpoints exposed as MCP tools

Configuration

The MCP server can be enabled/disabled and configured via environment variables:

• LVS_ENABLE_MCP: Enable/disable MCP server (default: true)
• LVS_MCP_PORT: Port for SSE transport (if not set, uses stdio)

Example: Enable SSE transport on port 38112:

export LVS_ENABLE_MCP=true
export LVS_MCP_PORT=38112

Available MCP Tools

The MCP server exposes the following tools:

Health & Status

• health_ready: Check server readiness
• health_live: Check server liveness
• get_metrics: Get Prometheus metrics

File Management

• add_file: Upload media file
• list_files: List uploaded files
• get_file_info: Get file metadata
• delete_file: Delete a file

Video Processing

• list_models: List available VLM models
• summarize_video: Generate video file summary
• generate_captions: Start VLM captioning on a live stream (fire-and-forget)
• stream_summarize: Summarize a live stream over a time window
• generate_vlm_captions: Generate timestamped VLM captions
• get_recommended_config: Get recommended configuration

Using with MCP Clients

Claude Desktop Configuration

Add to your Claude Desktop configuration:

{
  "mcpServers": {
    "via-engine": {
      "command": "docker",
      "args": [
        "exec",
        "-i",
        "video-summarization-server",
        "python",
        "/opt/nvidia/via/src/lvs_mcp.py"
      ]
    }
  }
}

Direct SSE Connection

When running with LVS_MCP_PORT set, connect to:

• SSE Endpoint: http://<host>:38112/sse
• Messages Endpoint: http://<host>:38112/messages

Example Tool Call:

{
  "name": "summarize_video",
  "arguments": {
    "id": "<file_id>",
    "model": "nvidia/cosmos3-nano-reasoner",
    "scenario": "police body camera",
    "events": ["pulling over", "arrest", "chasing"]
  }
}

Best Practices

Performance Optimization

1. Chunk Duration: Use 30-60 second chunks for optimal GPU utilization
2. RTVI-VLM Tuning: Configure batch size, frame counts, and per-process GPU allocation on the RTVI-VLM service. See Real-Time VLM for guidance.
3. Live-Stream Sessions: Set LVS_DISABLE_DB_RESET_ON_REQUEST_DONE=true to keep context-manager state intact across consecutive /v1/stream_summarize calls on the same stream.
4. Logstash Settle Window: Tune tools.elasticsearch_db.params.kafka_consumer_settle_secs (default 5 seconds) to balance ingest latency against the file-path summarize wall time when using the streaming profile.
5. Video Token Pruning: Enable Adaptive EVS on the RTVI-VLM service to prune redundant video tokens and reduce per-chunk inference time on long videos. See Efficient Video Sampling (EVS and Adaptive EVS) in the Real-Time VLM documentation.

Error Handling

• Enable stream: true for long videos to receive progressive updates
• Set appropriate max_tokens to avoid truncation
• Use chunk_overlap_duration to avoid missing events at boundaries

Troubleshooting

Common Issues

API Returns 503 Service Unavailable

• Another video is being processed (the Video Summarization microservice processes one video or live-stream request at a time)
• Wait for current processing to complete

Out of Memory Errors

• Tune RTVI-VLM batch size and frame inputs (see Real-Time VLM)
• Decrease chunk_duration
• Switch to a smaller VLM model on RTVI-VLM

Slow Processing

• Add GPUs to the RTVI-VLM service
• Use a smaller or faster VLM model on RTVI-VLM
• Reduce num_frames_per_chunk on RTVI-VLM
• When using the streaming profile, lower kafka_consumer_settle_secs if the Kafka and Logstash pipeline is keeping up

VLM Returns Incomplete or Invalid JSON Events (Missing or Empty Fields)

• The VLM may sometimes produce events that are missing required fields (like type, description) or contain fields that are present but empty (e.g., "type": ""). This will lead to warnings such as:

WARNING Skipping invalid event {'start_time': 75.01, 'end_time': 79.5}: 2 validation errors for Event
type
  Field required [type=missing, ...]
description
  Field required [type=missing, ...]
WARNING Chunk 1: vlm_pipeline_ctx is None - chunk span will be ***MASKED***
WARNING No events found in document 1
WARNING Skipping invalid event {'id': 2, 'start_time': 29.53, 'end_time': 29.53, 'type': '', 'description': 'There are no visible anomalies or artifacts in the frame.'}: 1 validation error for Event
type
  Field cannot be empty [type=value_error, ...]

• Cause: The VLM may not always strictly follow the JSON output schema. This can result in missing fields, or required fields present with empty values (such as an empty string for type), especially in difficult scenes or with certain models. When this happens, those events fail validation and are skipped.

• Impact: Some chunks may report zero extracted events, and the final summary may be incomplete for those time ranges.

• Workarounds:

  • Simplify the event list to reduce ambiguity for the model
  • Use override_vlm_prompt with a more explicit prompt that reinforces the required output fields (start_time, end_time, description, type), and clarifies that fields must not be empty
  • Increase the number of frames per chunk on RTVI-VLM to give the model more visual context
  • Retry the summarization request — VLM outputs can vary between runs

MCP Server Not Connecting

• Check LVS_ENABLE_MCP=true
• Verify LVS_MCP_PORT is accessible
• Check container logs: docker logs video-summarization-server

Logs and Debugging

# Set log level
export VSS_LOG_LEVEL=DEBUG
# View logs
docker-compose logs -f lvs-server
# Check specific component
grep "MCP" /var/log/via/via-server.log

API Error Codes

• 400 Bad Request: Invalid input syntax
• 401 Unauthorized: Missing or invalid authentication token
• 409 Conflict: File is in use and cannot be deleted
• 422 Unprocessable Entity: Failed to process request (validation error). POST /v1/summarize returns this status when called with source_type=stream — use POST /v1/generate_captions and POST /v1/stream_summarize for live-stream workflows instead. It also returns 422 when the media offsets are reversed (start later than end) or when min_tokens is greater than max_tokens.
• 429 Rate Limit Exceeded: Too many requests
• 500 Internal Server Error: Server-side error
• 503 Service Unavailable: Server is busy or a dependency failure is detected, such as RTVI-VLM being unavailable. Not all dependency failures produce this status; Kafka delivery failures can be logged while the request succeeds.

/v1/ready checks RTVI-VLM health and returns 503 if the stream handler is not initialized or RTVI-VLM is unavailable. It does not check the summarization LLM or run inference. The standalone Compose health check uses /v1/live, which returns 200 without checking dependencies. Run a summarization request to verify end-to-end inference; a healthy container alone does not verify it.

Appendix

Environment Variables Reference

Complete list of available environment variables:

API Configuration

• LVS_BACKEND_PORT: Host REST API port for the standalone Compose stack (default: 38111); maps to fixed container port 38111.
• BACKEND_PORT: REST API container port for the standalone docker run script (default: 38111). The standalone Compose stack fixes this value at 38111.
• BACKEND_HOST_PORT: Host REST API port for the standalone docker run script (defaults to BACKEND_PORT). Not read by the standalone Compose stack.
• LVS_MCP_PORT: MCP container port for the standalone docker run script (default: 38112). The standalone Compose stack fixes this value at 38112.
• LVS_MCP_HOST_PORT: Host MCP port for the standalone docker run script (defaults to LVS_MCP_PORT). Not read by the standalone Compose stack, which publishes MCP on fixed host port 38112.
• VSS_API_ENABLE_VERSIONING: Enable /v1 prefix

Client CLI

Read by the via_client_cli.py client rather than by the server.

• VIA_BACKEND: Default server address for the client’s --backend option (default: http://localhost:8000)

API Keys

• NGC_API_KEY: NVIDIA NGC API key
• NVIDIA_API_KEY: NVIDIA AI API key
• OPENAI_API_KEY: OpenAI API key

S3 Configuration

Media download runs inside RTVI-VLM, so set these on the RTVI-VLM container rather than on the Video Summarization MS container. They are read by the AWS SDK when a request supplies an s3:// URL.

• AWS_ACCESS_KEY_ID: AWS access key ID (required for S3 URL support)
• AWS_SECRET_ACCESS_KEY: AWS secret access key (required for S3 URL support)
• AWS_ENDPOINT_URL_S3: AWS S3 endpoint URL (required for S3 URL support)
• AZURE_OPENAI_API_KEY: Azure OpenAI API key, used by the RTVI-VLM openai-compat backend

VLM Configuration

• VLM_MODEL_TO_USE: VLM backend selector — set to openai-compat when delegating inference to RTVI-VLM in proxy mode
• VLM_SYSTEM_PROMPT: Default VLM system prompt applied when a request does not supply one (default: empty)

RTVI-VLM Configuration

• RTVI_VLM_URL: URL of the RTVI-VLM service (e.g., http://rtvi-vlm:8000 for the in-stack container, or an external host)
• RTVI_VLM_IMAGE: Container image for the in-stack RTVI-VLM service
• RTVI_VLM_PORT: Host port for the in-stack RTVI-VLM service (default: 8420; the container always listens on 8000)
• RTVI_VLM_GPU: GPU device ID assigned to the RTVI-VLM container. The Video Summarization service itself does not request a GPU.
• RTVI_VLM_URL_PASSTHROUGH: Route VLM calls through RTVI-VLM when set to true
• RTVI_HEALTH_RETRIES: Attempts to make while waiting for RTVI-VLM to report healthy at startup (default: 30)
• RTVI_HEALTH_RETRY_INTERVAL: Seconds between those attempts (default: 5)

Adaptive EVS video token pruning is a feature of RTVI-VLM, so it is configured on that service rather than on the Video Summarization MS server. Compose passes VIA_EVS_SESSION, VLM_VIDEO_PRUNING_RATE, VLLM_EVS_SIMILARITY_THRESHOLD, VIA_EVS_TOKEN_BUDGET, and VIA_EVS_MAX_SESSIONS from your .env file through to the in-stack rtvi-vlm service. For descriptions, defaults, and supported models, see Efficient Video Sampling (EVS and Adaptive EVS) in the Real-Time VLM documentation.

Kafka Configuration

• KAFKA_ENABLED: true enables the streaming profile end to end on the Video Summarization MS server; false for the base profile (default: false)
• KAFKA_BOOTSTRAP_SERVERS: Kafka broker bootstrap address used by the Video Summarization MS and RTVI-VLM (default: kafka:9092 inside the docker network)
• MESSAGE_BUS: Message bus used by RTVI-VLM for raw VLM events (default: kafka)
• MESSAGE_BUS_TOPIC: Topic for raw VLM events produced by RTVI-VLM and consumed by Logstash (default: mdx-vlm-captions)
• ERROR_BUS: Message bus used by RTVI-VLM for error events (default: kafka)
• KAFKA_STRUCTURED_SUMMARY_TOPIC: Kafka topic for structured events and aggregated summaries (default: mdx-structured-events-summary)
• KAFKA_PORT: Host port mapping for the in-stack Kafka broker’s internal listener (default: 9092)
• KAFKA_EXTERNAL_PORT: Host port mapping for the broker’s external listener; also embedded in KAFKA_ADVERTISED_LISTENERS (default: 9094)
• KAFKA_ADVERTISED_HOST: Hostname or IP that host-side or cross-host clients use to reconnect (default: localhost)
• LVS_CAPTION_SOURCE: Controls where file-path Kafka aggregation reads captions: sse (default) uses captions received in-process via the RTVI SSE response, db retrieves captions from Elasticsearch populated by the Kafka → Logstash → ES pipeline
• LVS_KAFKA_CONSUMER_SETTLE_SECS: Seconds to wait after the RTVI-VLM SSE stream completes so Logstash can flush the final chunk, used when the YAML key is absent (default: 5.0)

Database Configuration

Backend selection is covered in Database Backends.

• LVS_DATABASE_BACKEND: Store used by the summarization functions — elasticsearch_db (default), vector_db, graph_db_arango, or graph_db
• LVS_QA_DATABASE_BACKEND: Store used by the question-answering ingestion function (default: graph_db)
• LVS_INGESTION_FUNCTION: Ingestion function type (default: graph_ingestion)
• LVS_RETRIEVER_FUNCTION: Retriever function type
• LVS_DISABLE_DB_RESET_ON_REQUEST_DONE: true preserves request data and the per-asset Elasticsearch index after processing; false resets request data and automatically drops the index after file summarization completes. The Compose configuration defaults to true, while the service code defaults to false when the variable is unset. Live-stream summarization retains its index with either value because consecutive /v1/stream_summarize calls reuse the same asset ID.
• ES_HOST, ES_PORT: Elasticsearch connection
• ES_TRANSPORT_PORT: Elasticsearch transport port (default: 9302)
• ES_MAX_SHARDS_PER_NODE: Maximum number of shards allowed per Elasticsearch node (default: 2000). Each POST /v1/summarize creates one default_<asset_id> index with one shard; raise this cap for long-retention or high-volume deployments, or reclaim indices with DELETE /assets/{asset_id}/index.
• ES_JAVA_OPTS: JVM heap flags for the in-stack Elasticsearch container (default: -Xms4g -Xmx4g). Elasticsearch recommends no more than 20 active shards per GB of heap — the 4 GB default supports roughly 80 active shards in steady state. Tune ES_MAX_SHARDS_PER_NODE and ES_JAVA_OPTS together when increasing retention.
• MILVUS_DB_HOST, MILVUS_DB_GRPC_PORT: Milvus connection, used when vector_db is selected
• ARANGO_DB_HOST, ARANGO_DB_PORT, ARANGO_DB_USERNAME, ARANGO_DB_PASSWORD: ArangoDB connection, used when graph_db_arango is selected
• GRAPH_DB_HOST, GRAPH_DB_BOLT_PORT, GRAPH_DB_USERNAME, GRAPH_DB_PASSWORD: Neo4j connection, used when graph_db is selected

LLM and Embedding Configuration

• LVS_LLM_HOST, LVS_LLM_PORT: LLM NIM host and port
• LVS_LLM_MODEL_NAME: LLM model name (e.g., nvidia/nemotron-3.5-lightning-30b-a3b)
• LVS_LLM_BASE_URL: LLM API base URL
• LVS_ENABLE_LLM_MERGING: Use the LLM to merge adjacent duplicate events into natural text (default: false). See Event Merging for details and examples.
• LVS_EMB_ENABLE: Enable the embedding tool (default: false). Required by the Milvus, ArangoDB, and Neo4j backends.
• LVS_EMB_MODEL_NAME: Embedding model name
• LVS_EMB_BASE_URL: Embedding service endpoint

Prompt Overrides

These replace individual sections of the generated VLM prompt. Each is empty by default, which keeps the built-in text.

• LVS_PROMPT_VLM_ROLE: Override the AI role and persona description
• LVS_PROMPT_VLM_INSTRUCTION: Override the instruction section
• LVS_PROMPT_VLM_CONSTRAINTS: Append additional constraints
• LVS_PROMPT_VLM_STRUCTURED_OUTPUT: Override the output format and JSON template

The CA-RAG summarization function additionally accepts aggregation_prompt and description_merge_prompt as configuration keys rather than environment variables. description_merge_prompt applies only when LVS_ENABLE_LLM_MERGING is enabled.

Feature Flags

• LVS_ENABLE_MCP: Enable MCP server (default: true)
• ENABLE_VIA_HEALTH_EVAL: Enable health evaluation (default: false)
• ENABLE_DENSE_CAPTION: Request dense captions in addition to structured events (default: false)
• LVS_DROP_EMPTY_EVENT_FIELDS: Empty event fields are dropped from responses by default; set to false to keep them
• LVS_AGGREGATION_EMPTY_RETRIES: Retry budget for aggregation calls that return no events and no summary (default: 2; 0 disables retrying)
• VSS_POST_PROCESS_ON_EACH_DOC_ADD: Run post-processing after each document is added rather than at the end of the request (default: false)
• LVS_CHUNK_DEBUG: Debugging aid — when true, chunks are processed but summarization is skipped (default: false)

Logging and Monitoring

• VSS_LOG_LEVEL: Log level (DEBUG, INFO, WARNING, ERROR)
• VIA_LOG_DIR: Directory for VIA logs
• VIA_ENABLE_OTEL: Enable OpenTelemetry
• VIA_OTEL_ENDPOINT: OpenTelemetry endpoint (e.g., http://localhost:4318)
• VIA_OTEL_EXPORTER: OpenTelemetry exporter type (e.g., console)
• VIA_CTX_RAG_ENABLE_OTEL: Enable OpenTelemetry for context RAG
• VIA_CTX_RAG_EXPORTER: Context RAG exporter type
• VIA_CTX_RAG_OTEL_ENDPOINT: Context RAG OpenTelemetry endpoint (e.g., http://localhost:4318)

Performance

Published E2E latency and sizing guidance are in Video Summarization Performance.

Glossary

• CA-RAG: Caption-Augmented Retrieval-Augmented Generation
• VLM: Vision-Language Model
• NIM: NVIDIA Inference Microservice
• RTVI-VLM: Real-Time Vision-Language Model microservice — see Real-Time VLM
• RTSP: Real-Time Streaming Protocol, the network protocol used for live camera streams
• MCP: Model Context Protocol
• RAG: Retrieval-Augmented Generation
• CV: Computer Vision
• PPE: Personal Protective Equipment