> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/vss/dev/system-components/vision-microservices/video-summarization/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/vss/_mcp/server.

# Video Summarization

## Overview

Video Summarization is a microservice that uses Vision-Language Models (VLMs) and Large Language Models (LLMs) to extract insights from uploaded videos and live streams. The system provides REST API and Model Context Protocol (MCP) interfaces for video processing and summarization.

The Video Summarization microservice segments long-form media, generates VLM captions or structured events, stores the results, and produces concise summaries that preserve timestamped evidence.

### Key Features

The Video Summarization microservice operates by analyzing media and generating structured, machine-readable output. The primary output includes timestamped events and a high-level summary, allowing users to identify when specific actions, object appearances, or scene changes occurred.

### High Customizability and Model Flexibility

A defining feature of the Video Summarization microservice is its high degree of customization, particularly regarding the underlying language and vision models it employs.

* **OpenAI Compatibility**: The service is designed for maximum interoperability, allowing users to select and integrate any OpenAI compatible Vision-Language Model (VLM) or Large Language Model (LLM). This flexibility ensures that users can leverage the latest and most suitable models for their specific summarization needs and quality requirements.
* **RTVI-VLM video inferencing**: Per-chunk and live-stream video inferencing is delegated to the [Real-Time VLM (RTVI-VLM)](/system-components/vision-microservices/real-time-vlm) microservice. RTVI-VLM supports Cosmos Reason3, Nemotron Nano Omni, and Qwen 3.5 base models through its proxy and openai-compat modes, so operators can pick the best-fit VLM without rebuilding the Video Summarization microservice.

> **Note**
>
> * Nemotron Nano Omni model support in RTVI-VLM and the Video Summarization microservice is limited to integration testing and sanity coverage in this release. It has not undergone full production-level validation. Operators should evaluate model quality on their own workloads before deploying to production.
> * For the full list of supported VLM checkpoints, precision options, and configuration details, see [Models Supported](/system-components/vision-microservices/real-time-vlm#supported-models) in the [Real-Time VLM](/system-components/vision-microservices/real-time-vlm) documentation.

### Data Persistence and Management

The integrity and accessibility of the processed data are managed through a configurable database layer.

* **Configurable Database**: Processed summaries, extracted events, captions, and associated metadata are stored in a dedicated database. Four backends are selectable — Elasticsearch (the default), Milvus, ArangoDB, and Neo4j. Only Elasticsearch is provisioned inside the compose stack; the others are selected by pointing the microservice at an instance you run. See [Database Backends](/system-components/vision-microservices/video-summarization#database-backends) for the selectors and connection variables.

### Access and Integration Methods

To ensure broad applicability and seamless integration into diverse workflows, the Video Summarization microservice exposes multiple methods for access and interaction.

* **REST API**: The service provides a standard **RESTful Application Programming Interface (API)**. This allows users to connect to the Video Summarization microservice using programmatic scripts and traditional software integrations, making it ideal for back-end systems and custom applications.
* **Model Context Protocol (MCP)**: In addition to the REST API, the Video Summarization microservice exposes an MCP interface for AI agents and orchestration systems.

## Architecture

The Video Summarization microservice supports two deployment profiles: *Summarization Base Profile* and *Summarization with Streaming and Message Bus*. The base profile handles file summarization in-process; the streaming profile adds a Kafka message queue and a Logstash consumer service to decouple raw VLM events from the orchestrator and enables RTSP live-stream summarization.

### Summarization Base Profile

![Video Summarization Base Profile architecture](/vss/_fern-img/6bb9edc3e4cd755a499d9dfb3bd00f777d5bc3e06186d02b1f2909770f025206.webp)

**Scenario 1 — File summarization, base profile.** The client uploads a video file via REST or MCP. The Video Summarization microservice segments the video into chunks and forwards each chunk to the [RTVI-VLM](/system-components/vision-microservices/real-time-vlm) service for inference. RTVI-VLM returns per-chunk events synchronously over Server-Sent Events (SSE); the Video Summarization microservice calls the CA-RAG pipeline in-process to write events to Elasticsearch and aggregate them with the summarization LLM. The structured summary and timestamped events are returned in the HTTP response. The Kafka message queue is disabled in this profile (`KAFKA_ENABLED=false`).

### Summarization with Streaming and Message Bus

![Video Summarization with Streaming and Message Bus architecture](/vss/_fern-img/f614e10445c28a4884d280ff0a638fd4d7cc79d26559b882a1199dec9e43dda3.webp)

In this profile a Kafka message queue carries raw VLM events and aggregated summaries between RTVI-VLM, the Video Summarization microservice, and Elasticsearch. A Logstash consumer service subscribes to the Kafka topics, decodes the protobuf payloads, normalizes their fields, and indexes the documents into Elasticsearch (`default_<asset_id>`). RTSP live-stream summarization is supported only in this profile.

**Scenario 2 — File summarization with streaming and message bus.** The client request is identical to Scenario 1, but RTVI-VLM publishes per-chunk raw events to the Kafka topic `mdx-vlm-captions` instead of returning them in-process. The Logstash consumer service indexes the events into Elasticsearch. The `LVS_CAPTION_SOURCE` environment variable controls where the Video Summarization microservice reads captions for aggregation: `sse` (default) uses the captions received in-process via the RTVI SSE response, while `db` retrieves captions from the Elasticsearch index populated by the Kafka → Logstash → ES pipeline. In both modes, after the RTVI-VLM SSE stream completes, the Video Summarization microservice waits `kafka_consumer_settle_secs` (default 5 seconds) so Logstash can flush the last chunk, then aggregates through CA-RAG and publishes the structured events and aggregated summary to the `mdx-structured-events-summary` topic for Logstash to index.

**Scenario 3 — RTSP live-stream summarization with streaming and message bus.** Live-stream summarization uses two dedicated APIs instead of the file-only `POST /v1/summarize` endpoint. The client first calls `POST /v1/stream/add` on RTVI-VLM with the RTSP URL (no inference yet). In phase 1, the client issues `POST /v1/generate_captions` on the Video Summarization microservice with the stream ID, model, and prompt parameters (`scenario`, `events`, `objects_of_interest`). The Video Summarization microservice builds the VLM prompt, calls RTVI-VLM to start captioning, and returns immediately with `{status: "accepted"}`. RTVI-VLM begins captioning frames continuously and publishes raw events to the Kafka message queue, which the Logstash consumer service indexes into Elasticsearch. In phase 2, the client calls `POST /v1/stream_summarize` with the stream ID and an optional `start_time`/`end_time` window. The Video Summarization microservice reads the captions from Elasticsearch, aggregates them via CA-RAG, publishes the structured events and aggregated summary to the `mdx-structured-events-summary` Kafka topic, and returns the summary synchronously.

### Core Components

The Video Summarization microservice consists of the following core components:

* **Video Summarization REST Server**: FastAPI-based REST API server.
* **Video Summarization MCP Server**: Model Context Protocol server for AI agent integration.
* **RTVI-VLM**: [Real-Time VLM](/system-components/vision-microservices/real-time-vlm) service used for per-chunk and live-stream video inferencing.
* **CA-RAG Pipeline**: Context-aware retrieval and generation system that orchestrates event aggregation and summarization.
* **LLM NIM**: NVIDIA Inference Microservice for the summarization LLM.
* **Data Store**: Elasticsearch by default for storing extracted events, captions, and aggregated summaries, with Milvus, ArangoDB, and Neo4j available as alternative backends.
* **Kafka Message Queue** (Summarization with Streaming and Message Bus profile only): Carries raw VLM events on the `mdx-vlm-captions` topic and aggregated summaries on the `mdx-structured-events-summary` topic.
* **Logstash Consumer Service** (Summarization with Streaming and Message Bus profile only): Subscribes to the Kafka topics, decodes the `nv.VisionLLM` protobuf payloads, normalizes field types, and indexes the documents into Elasticsearch.

## Getting Started

### Prerequisites

* Docker and Docker Compose
* NVIDIA GPU(s) with appropriate drivers
* NGC API Key (for NVIDIA NIMs)

## Migrating from 3.2.x to 3.3.0

Review the following changes before you reuse a 3.2.x deployment configuration
with Video Summarization 3.3.0.

### Update Required

* Replace `KAFKA_TOPIC` with `MESSAGE_BUS_TOPIC` in your `.env` file when you use
  the streaming profile. The default topic remains `mdx-vlm-captions`.
  `MESSAGE_BUS_TOPIC` configures the Logstash and RTVI-VLM services; the Video
  Summarization server does not read either variable.
* Update `LVS_LLM_MODEL_NAME` from the removed `openai/gpt-oss-20b` model to
  `nvidia/nemotron-3.5-lightning-30b-a3b`, or to another LLM that your
  deployment provides. The 3.3.0 deployment rejects the removed model name.
* If you use `services/video-summarization/docker/deploy/compose.yaml` with the
  `rtvi` profile, set `VLM_MODEL_TO_USE` explicitly. For a local model, also
  set `MODEL_PATH`. For an OpenAI-compatible endpoint, set `VIA_VLM_ENDPOINT`
  and its required API key. This standalone service Compose profile now rejects
  missing RTVI-VLM configuration at startup. The top-level blueprint profile
  under `deploy/docker` retains its `openai-compat` default.
* Replace `LVS_ENABLE_LLM_MERGE` with `LVS_ENABLE_LLM_MERGING`. The former name
  appeared in earlier documentation but is not read by the service.

### New Environment Variables

* `LVS_AGGREGATION_EMPTY_RETRIES` sets the number of additional CA-RAG
  aggregation attempts when a response contains neither events nor a video
  summary. The default is `2`. Set it to `0` to retain the 3.2.x
  single-attempt behavior.
* `MESSAGE_BUS` selects the RTVI-VLM message bus, and `ERROR_BUS` selects its
  error bus. Both default to `kafka` in the streaming profile.
* `MESSAGE_BUS_TOPIC` replaces `KAFKA_TOPIC` for the raw VLM event topic and
  defaults to `mdx-vlm-captions`.

The in-stack RTVI-VLM profile also forwards the new EVS++ settings
`VIA_EVS_SESSION`, `VLM_VIDEO_PRUNING_RATE`,
`VLLM_EVS_SIMILARITY_THRESHOLD`, `VIA_EVS_TOKEN_BUDGET`, and
`VIA_EVS_MAX_SESSIONS`. For supported models, defaults, and behavior, refer to
*Efficient Video Sampling (EVS and EVS++)* in the
[Real-Time VLM](/system-components/vision-microservices/real-time-vlm)
documentation.

### Removed Environment Variables

* `KAFKA_TOPIC` is no longer read by the 3.3.0 stack. Use
  `MESSAGE_BUS_TOPIC`.
* `VSS_DISABLE_DECODER_REUSE` is no longer read by the Video Summarization
  service. Remove it from carried-over configurations.

### API Changes

* `POST /v1/chat/completions` adds question answering for assets that were
  summarized with `enable_qa=true`.
* `DELETE /assets/{asset_id}/index` adds an idempotent way to reclaim a
  retained per-asset Elasticsearch index. For automatic and manual cleanup
  behavior, refer to the
  [Asset Management API](/system-components/vision-microservices/video-summarization/api-reference#asset-management).
* `POST /summarize` and `POST /v1/summarize` now return HTTP 422 when
  `media_info.start_offset` is greater than or equal to
  `media_info.end_offset`, or when `min_tokens` is greater than `max_tokens`.
  In 3.2.x, these requests could return HTTP 200 with an empty result.
* The development-only `GET /files` endpoint now returns HTTP 422 for an
  unsupported `purpose` value instead of HTTP 200 with an empty list.
* Responses omit `media_info.start_offset` and `media_info.end_offset` when
  either value is unavailable, nonnumeric, or not a strictly increasing
  range. Update clients that assume both keys are always present.
* `POST /summarize` and `POST /v1/stream_summarize` accept an inbound
  `traceparent` header and link the request to the caller's distributed trace.

### Functionality Changes

* Video Summarization now uses CA-RAG 3.1.0 instead of CA-RAG 3.0.1.
* Aggregation retries only when CA-RAG returns neither events nor a video
  summary. A response that has no events but has a summary is not retried.
* The service always forwards `scenario` and `events` to CA-RAG, including when
  `enable_vlm_structured_output` is enabled.
* The 3.3.0 RTVI-VLM profile supports optional EVS++ video token pruning to
  reduce redundant visual input and per-chunk inference time.
* The Elasticsearch index retention default is unchanged. The Compose
  configuration retains per-asset indices by default, and the service drops
  them after file summarization when
  `LVS_DISABLE_DB_RESET_ON_REQUEST_DONE=false`. The new asset-management API
  provides explicit cleanup for retained indices.
* The RTVI-VLM client closes the caption-stream HTTP response after processing
  to prevent a long-running file descriptor and memory leak.

## Deployment

### Clone the Repository

```bash
git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git
cd video-search-and-summarization
git checkout "develop"
cd services/video-summarization
```

### Build the Container Image

The Video Summarization compose stack uses a locally built image
(`via-engine-<username>`). Build the image before bringing up the stack.

Use the `ARM_PLATFORM` variable to select the ARM64 build variant.

**x86 (amd64) build:**

The following command is for x86 (amd64) architecture only:

```bash
make -C docker build
```

**DGX Spark / SBSA / Grace (server-class ARM64) build:**

Set `ARM_PLATFORM=sbsa` to build the SBSA image variant. This is the variant
required on DGX Spark (GB10):

```bash
ARM_PLATFORM=sbsa make -C docker build
```

**Jetson / IGX-AGX Thor / Tegra (ARM64 iGPU) build:**

Set `ARM_PLATFORM=igpu` to build the integrated-GPU ARM64 image variant:

```bash
ARM_PLATFORM=igpu make -C docker build
```

> **Note**
>
> The container image tag must match the host CPU platform. Pre-built images
> are published as `ghcr.io/nvidia-ai-blueprints/vss/vss-video-summarization:3.3.0`
> for x86 and Jetson Thor, and
> `ghcr.io/nvidia-ai-blueprints/vss/vss-video-summarization:3.3.0-sbsa` for DGX Spark
> / SBSA / Grace. When building from source, select the matching
> `ARM_PLATFORM` value: `sbsa` for DGX Spark / Grace, `igpu` for Jetson /
> Thor.

For the full developer workflow, deploy the Video Summarization profile as described in [Video Summarization agent workflow](/vision-agent/agent-workflows/video-summarization).
Video Summarization microservice is deployed via the docker-compose stack at `docker/deploy/compose.yaml`. Two deployment profiles are supported:

* [Summarization Base Profile](/system-components/vision-microservices/video-summarization#summarization-base-profile) — file summarization without a Kafka message queue.
* [Summarization with Streaming and Message Bus](/system-components/vision-microservices/video-summarization#summarization-with-streaming-and-message-bus) — file and RTSP live-stream summarization with a Kafka message queue and Logstash consumer service.

Use standalone deployment when you need to run only the Video Summarization server against existing Elasticsearch, LLM, and VLM services.
A slimmer single-container option is documented under [Standalone Container (Advanced)](/system-components/vision-microservices/video-summarization#standalone-container-advanced) for operators who already run RTVI-VLM, the LLM NIM, and Elasticsearch externally.

> **Note**
>
> Build the container image as described in the **Build the Container Image** section above before bringing up the stack. The RTVI-VLM image is pulled from NGC automatically.

### Summarization Base Profile

This profile deploys Video Summarization microservice, RTVI-VLM, and Elasticsearch with the Kafka message queue disabled. Use it for file summarization workloads.

**Prerequisites**

* Docker Engine 28.3.3+ and Docker Compose v2.39.1+
* NVIDIA Container Toolkit
* NVIDIA GPU(s) with appropriate drivers
* External LLM NIM service (for summarization) — configured via `LVS_LLM_HOST` and `LVS_LLM_PORT`

**Environment Setup**

Create a `.env` file from the variables below. The standalone Compose stack does not ship a committed `.env`. For an integrated VSS deployment, use the `lvs` developer profile and the native Docker Compose procedure in [Long Video Summarization](/vision-agent/agent-workflows/video-summarization).

Edit `.env` and set the following key variables:

```bash
# API Keys
NGC_API_KEY=<your-ngc-api-key>
NGC_CLI_API_KEY=<your-ngc-api-key>
NVIDIA_API_KEY=<your-nvidia-api-key>
OPENAI_API_KEY=<your-openai-api-key>   # required when RTVI uses openai-compat

# Host REST API port (container port is fixed at 38111)
LVS_BACKEND_PORT=38111
# The standalone Compose stack publishes MCP on fixed host/container port 38112.

# Database (in-stack Elasticsearch)
ES_HOST=elasticsearch
ES_PORT=9200

# LLM (external NIM)
LVS_LLM_HOST=<llm-nim-host-ip>
LVS_LLM_PORT=8002
# Query the configured NIM's /v1/models endpoint and copy its exact id here.
# Model IDs can differ between NIM versions and serving configurations.
LVS_LLM_MODEL_NAME=<id-returned-by-llm-nim-v1-models>
LVS_DATABASE_BACKEND=elasticsearch_db

# RTVI-VLM Integration
COMPOSE_PROFILES=rtvi
# Multi-arch tag, covering x86 and Jetson Thor (iGPU) platforms:
RTVI_VLM_IMAGE=ghcr.io/nvidia-ai-blueprints/vss/vss-rt-vlm:3.3.0
# For DGX Spark / SBSA / Grace (server-class ARM) platforms, use the -sbsa tag:
#RTVI_VLM_IMAGE=ghcr.io/nvidia-ai-blueprints/vss/vss-rt-vlm:3.3.0-sbsa
RTVI_VLM_PORT=8420
RTVI_VLM_GPU=1
# RTVI_VLM_URL=http://<external-rtvi-host>:8083  # set only when pointing at an external RTVI-VLM
VLM_MODEL_TO_USE=openai-compat
VIA_VLM_ENDPOINT=<openai-compatible-vlm-endpoint>

# Kafka message queue disabled for the base profile
KAFKA_ENABLED=false

# Feature Flags
LVS_ENABLE_MCP=true
ENABLE_VIA_HEALTH_EVAL=false

# Logging
VSS_LOG_LEVEL=DEBUG
```

**Bring Up the Stack**

Before starting the Video Summarization MS, query the external LLM NIM and
copy its exact model ID into `LVS_LLM_MODEL_NAME` in `.env`. Replace the URL
placeholders with the `LVS_LLM_HOST` and `LVS_LLM_PORT` values configured above:

```bash
curl -fsS "http://<llm-nim-host-ip>:<llm-nim-port>/v1/models" | python3 -m json.tool
```

The Compose `--env-file` option does not export variables into your shell.
Restart the Video Summarization MS after changing its configured model ID.

```bash
docker compose -f docker/deploy/compose.yaml --profile rtvi --env-file .env up -d
```

**Verify Service Health**

```bash
curl http://localhost:38111/v1/ready
```

Before submitting a request, retrieve the deployed RTVI-VLM model ID for the
request's `model` field. Replace the port placeholder with the `RTVI_VLM_PORT`
value configured in `.env` (8420 in this example):

```bash
curl -fsS "http://localhost:<rtvi-vlm-host-port>/v1/models" | python3 -m json.tool
```

**Test the API**

Submit a sample file summarization request:

```bash
curl --location 'http://localhost:38111/v1/summarize' \
  --header 'Content-Type: application/json' \
  --data '{
    "url": "http://<video-server-ip>:<port>/your-video.mp4",
    "model": "<id-returned-by-rtvi-vlm-v1-models>",
    "scenario": "law enforcement",
    "chunk_duration": 15,
    "events": ["pulling over", "arrest", "chasing"]
  }' | python3 -m json.tool
```

Replace `<video-server-ip>` and `<port>` with your video server address.

**Tear Down the Stack**

```bash
docker compose -f docker/deploy/compose.yaml --profile rtvi down
```

### Summarization with Streaming and Message Bus

This profile adds a Kafka message queue and a Logstash consumer service to the base profile. It is required for RTSP live-stream summarization, and is also recommended for file summarization when end-to-end Kafka publish-back is desired (for example, to feed downstream consumers off the `mdx-structured-events-summary` topic).

**Prerequisites**

* Same as the base profile, plus the host ports for the in-stack Kafka broker (`KAFKA_PORT` and `KAFKA_EXTERNAL_PORT`) must be free.

**Environment Setup**

Source the values from `.env` for this profile. The diff from the base profile is:

```bash
# Enable the Kafka message queue
KAFKA_ENABLED=true
KAFKA_BOOTSTRAP_SERVERS=kafka:9092
MESSAGE_BUS_TOPIC=mdx-vlm-captions
KAFKA_STRUCTURED_SUMMARY_TOPIC=mdx-structured-events-summary

# Host-side broker ports — the compose defaults are 9092/9094.
# Override these to avoid collisions with other Kafka stacks on the same host.
KAFKA_PORT=9192               # default: 9092
KAFKA_EXTERNAL_PORT=9194      # default: 9094
KAFKA_ADVERTISED_HOST=localhost

# Recommended for multi-call live-stream sessions: keeps the per-stream
# context manager state intact across consecutive /v1/stream_summarize calls.
LVS_DISABLE_DB_RESET_ON_REQUEST_DONE=true
```

The checked-in `services/video-summarization/config/config.yaml` already reads
`KAFKA_ENABLED` from the environment and sets
`kafka_consumer_settle_secs: 5.0`. Compose mounts that file at
`/opt/nvidia/via/config/default_config.yaml` in the container, so the `.env`
settings above are sufficient for the standard Kafka profile. Edit the YAML
only when you need advanced tuning; for example, the relevant defaults are:

```yaml
tools:
  elasticsearch_db:
    type: elasticsearch
    params:
      host: !ENV ${ES_HOST}
      port: !ENV ${ES_PORT}
      kafka_consumer_settle_secs: 5.0

functions:
  summarization:
    type:
      vlm_structured_summarization
    params:
      kafka_enabled: !ENV ${KAFKA_ENABLED:false}
  summarization_online:
    type:
      vlm_structured_summarization_online
    params:
      kafka_enabled: !ENV ${KAFKA_ENABLED:false}
```

**Bring Up the Stack**

```bash
docker compose -f docker/deploy/compose.yaml --profile rtvi --profile kafka up -d
```

**Live-Stream Operator Workflow**

1. Register an RTSP stream on RTVI-VLM (no inference yet):

```bash
ASSET_ID=$(curl -fsS -X POST http://localhost:8420/v1/stream/add \
  -H 'Content-Type: application/json' \
  -d '{
    "key": "sensor",
    "value": {
      "camera_id": "",
      "camera_url": "rtsp://<rtsp-source>/stream",
      "change": "camera_add"
    }
  }' \
  | python3 -c "import sys,json; print(json.load(sys.stdin).get('asset_id'))")
```

2. Start VLM captioning on the stream (fire-and-forget). This is where prompt-related parameters such as `scenario`, `events`, and `objects_of_interest` are passed:

```bash
curl -fsS -X POST http://localhost:38111/v1/generate_captions \
  -H 'Content-Type: application/json' \
  -d "{
    \"id\": \"$ASSET_ID\",
    \"model\": \"<id-returned-by-rtvi-vlm-v1-models>\",
    \"scenario\": \"warehouse safety monitoring\",
    \"events\": [\"box dropping\", \"unsafe forklift operations\", \"normal activity\"],
    \"chunk_duration\": 10
  }"
```

3. Summarize the stream over a time window. No prompt-related parameters are needed — the Video Summarization microservice reads the captions already stored by the captioning phase:

```bash
curl -fsS -X POST http://localhost:38111/v1/stream_summarize \
  -H 'Content-Type: application/json' \
  -d "{
    \"id\": \"$ASSET_ID\",
    \"model\": \"<id-returned-by-rtvi-vlm-v1-models>\",
    \"start_time\": 0,
    \"end_time\": 0
  }"
```

4. Repeat the `/v1/stream_summarize` call as needed with progressively larger or narrower time windows to get the latest aggregated events and summary.

**Tear Down the Stack**

```bash
docker compose -f docker/deploy/compose.yaml --profile rtvi --profile kafka down --remove-orphans
```

This preserves existing named cache and plugin volumes. The standalone Compose
stack does not configure persistent data volumes for Elasticsearch or Kafka;
removing their containers can lose their data even without `-v`. Configure
persistent storage and back up any required data before teardown.

Add `-v` only as an optional destructive cleanup step: it also removes the
Compose-managed named volumes and attached anonymous volumes. Anonymous volumes
are not automatically reattached when the stack is brought up again.

### Standalone Container (Advanced)

This deployment runs the Video Summarization MS server as a single Docker container, assuming external services (RTVI-VLM, Elasticsearch, LLM NIM) are already running. The compose-based profiles above are recommended for most operators; use this only when you already run all dependencies separately.

**Prerequisites**

1. Copy the example environment file to `.env`:

```bash
touch .env
```

2. Edit `.env` and fill in your configuration values:

```bash
# Environment variables for standalone Video Summarization Server docker run
# Copy this file to .env and fill in the values

# Container Configuration
# Multi-arch tag, covering x86 and Jetson Thor (iGPU) platforms:
CONTAINER_IMAGE=ghcr.io/nvidia-ai-blueprints/vss/vss-video-summarization:3.3.0
# For DGX Spark / SBSA / Grace (server-class ARM) platforms, use the -sbsa tag:
#CONTAINER_IMAGE=ghcr.io/nvidia-ai-blueprints/vss/vss-video-summarization:3.3.0-sbsa

# API Keys and Authentication
NGC_API_KEY=<your-ngc-api-key>
NVIDIA_API_KEY=<your-nvidia-api-key>
# OPENAI_API_KEY=<your-openai-api-key>

# S3 Configuration (required for S3 URLs)
# AWS_ACCESS_KEY_ID=<your-aws-access-key-id>
# AWS_SECRET_ACCESS_KEY=<your-aws-secret-access-key>
# AWS_ENDPOINT_URL_S3=<your-s3-endpoint-url>

# Port Configuration
BACKEND_PORT=38111
BACKEND_HOST_PORT=38111
LVS_MCP_PORT=38112
LVS_MCP_HOST_PORT=38112

# CA RAG Configuration
# This will be set via mount path - do not override here
# CA_RAG_CONFIG=/opt/nvidia/via/config/default_config.yaml

# Feature Flags
ENABLE_VIA_HEALTH_EVAL=false
LVS_ENABLE_MCP=true

# Database Configuration - Elasticsearch
# Update these to point to your external Elasticsearch service
ES_HOST=<elasticsearch-host-ip>
ES_PORT=9202
ES_TRANSPORT_PORT=9302

# RTVI-VLM Integration (point at an externally running RTVI-VLM)
RTVI_VLM_URL=http://<rtvi-vlm-host>:8000
VLM_MODEL_TO_USE=openai-compat

# OpenTelemetry Configuration (optional)
# VIA_ENABLE_OTEL=false
# VIA_OTEL_ENDPOINT=http://localhost:4318
# VIA_OTEL_EXPORTER=console
# VIA_CTX_RAG_ENABLE_OTEL=false
# VIA_CTX_RAG_EXPORTER=console
# VIA_CTX_RAG_OTEL_ENDPOINT=http://localhost:4318

# Logging and Debug
VSS_LOG_LEVEL=DEBUG

# LLM Configuration
# Update these to point to your external LLM NIM services
# Query LVS_LLM_BASE_URL/models and use the exact returned id.
LVS_LLM_MODEL_NAME=<id-returned-by-llm-nim-v1-models>
LVS_LLM_BASE_URL=http://<llm-nim-host-ip>:8002/v1

# Database Selection
LVS_DATABASE_BACKEND=elasticsearch_db

# Kafka message queue (disabled in standalone mode)
KAFKA_ENABLED=false
```

**Run Script**

Create a file named `run-video-summarization-server.sh` with the following content:

```bash
#!/bin/bash

# Standalone Docker run command for Video Summarization Server
# This script runs only the Video Summarization server container, assuming other services are running separately

# Configuration
CONTAINER_NAME="video-summarization-server"
ENV_FILE="${ENV_FILE:-.env}"

# Check if .env file exists
if [ ! -f "$ENV_FILE" ]; then
    echo "Error: Environment file '$ENV_FILE' not found!"
    echo "Please copy .env to $ENV_FILE and fill in the values."
    exit 1
fi

# Load CONTAINER_IMAGE from env file
IMAGE=$(grep "^CONTAINER_IMAGE=" "$ENV_FILE" | cut -d'=' -f2)
if [ -z "$IMAGE" ]; then
    echo "Error: CONTAINER_IMAGE not found in $ENV_FILE"
    echo "Set CONTAINER_IMAGE (e.g., via-engine-\$USER after running make -C docker build)."
    exit 1
fi

# Load MODEL_ROOT_DIR from env file (optional)
MODEL_ROOT_DIR=$(grep "^MODEL_ROOT_DIR=" "$ENV_FILE" | cut -d'=' -f2)

# Load port values from env file
BACKEND_PORT=$(grep "^BACKEND_PORT=" "$ENV_FILE" | cut -d'=' -f2)
BACKEND_HOST_PORT=$(grep "^BACKEND_HOST_PORT=" "$ENV_FILE" | cut -d'=' -f2)
LVS_MCP_PORT=$(grep "^LVS_MCP_PORT=" "$ENV_FILE" | cut -d'=' -f2)
LVS_MCP_HOST_PORT=$(grep "^LVS_MCP_HOST_PORT=" "$ENV_FILE" | cut -d'=' -f2)

# Set defaults if not found
BACKEND_PORT=${BACKEND_PORT:-38111}
LVS_MCP_PORT=${LVS_MCP_PORT:-38112}

# Build port mapping arguments. Override the host-side ports independently
# when the defaults are already in use.
BACKEND_HOST_PORT=${BACKEND_HOST_PORT:-$BACKEND_PORT}
LVS_MCP_HOST_PORT=${LVS_MCP_HOST_PORT:-$LVS_MCP_PORT}
PORT_ARGS=(
    -p "${BACKEND_HOST_PORT}:${BACKEND_PORT}"
    -p "${LVS_MCP_HOST_PORT}:${LVS_MCP_PORT}"
)

# Build volume mount for MODEL_ROOT_DIR if set
MODEL_VOLUME_ARG=""
if [ -n "$MODEL_ROOT_DIR" ]; then
    # Expand tilde and get absolute path if directory exists or can be created
    MODEL_ROOT_DIR_EXPANDED="${MODEL_ROOT_DIR/#\~/$HOME}"
    if [ -d "$MODEL_ROOT_DIR_EXPANDED" ] || mkdir -p "$MODEL_ROOT_DIR_EXPANDED" 2>/dev/null; then
        MODEL_ROOT_DIR_ABS="$(cd "$MODEL_ROOT_DIR_EXPANDED" && pwd)"
        MODEL_VOLUME_ARG="-v ${MODEL_ROOT_DIR_ABS}:${MODEL_ROOT_DIR_ABS}"
        echo "MODEL_ROOT_DIR will be mounted: $MODEL_ROOT_DIR_ABS"
    else
        echo "Warning: MODEL_ROOT_DIR '$MODEL_ROOT_DIR' could not be accessed or created, skipping mount"
    fi
else
    echo "MODEL_ROOT_DIR not set in $ENV_FILE, skipping model cache mount"
fi

# Docker run command. Use addresses reachable from this container for
# Elasticsearch, RTVI-VLM, and the LLM NIM. Compose-only service names such as
# `elasticsearch` and `rtvi-vlm` do not resolve from this standalone container;
# use a routable host/IP, `host.docker.internal` for host-published services, or
# attach the container to the dependencies' Docker network explicitly.
docker run -d \
    --name "$CONTAINER_NAME" \
    --add-host host.docker.internal:host-gateway \
    "${PORT_ARGS[@]}" \
    --env-file "$ENV_FILE" \
    $MODEL_VOLUME_ARG \
    --restart unless-stopped \
    "$IMAGE"

echo "Video Summarization Server container started!"
echo "Container name: $CONTAINER_NAME"
echo "Container image: $IMAGE"
echo "Backend port: $BACKEND_PORT"
echo "MCP port: $LVS_MCP_PORT"
if [ -n "$MODEL_ROOT_DIR_ABS" ]; then
    echo "Model cache mounted from: $MODEL_ROOT_DIR_ABS"
fi
echo ""
echo "To view logs: docker logs -f $CONTAINER_NAME"
echo "To stop: docker stop $CONTAINER_NAME"
echo "To remove: docker rm $CONTAINER_NAME"
```

**Running the Container**

1. Make the script executable:

```bash
chmod +x run-video-summarization-server.sh
```

2. Run the script:

```bash
./run-video-summarization-server.sh
```

3. Check the container status:

```bash
docker logs -f video-summarization-server
```

4. Verify service health:

```bash
curl http://localhost:38111/v1/ready
```

**Testing the API**

Test video summarization with a sample request:

```bash
curl --location 'http://localhost:38111/v1/summarize' \
  --header 'Content-Type: application/json' \
  --data '{
  "id": null,
  "url": "http://<video-server-ip>:<port>/your-video.mp4",
  "model": "<id-returned-by-rtvi-vlm-v1-models>",
  "scenario": "law enforcement",
  "events": ["pulling over", "arrest", "chasing"]
}' | python3 -m json.tool
```

**Note**: Replace `<video-server-ip>` and `<port>` with your actual video server address and port.

**Stopping the Container**

```bash
# Stop the container
docker stop video-summarization-server

# Remove the container
docker rm video-summarization-server
```

**Configuration Updates**

To update environment variables:

1. Stop and remove the container:

```bash
docker stop video-summarization-server && docker rm video-summarization-server
```

2. Edit your `.env` file with the new values

3. Restart the container:

```bash
./run-video-summarization-server.sh
```

**Network Configuration**

The script runs the container on Docker's default bridge network and publishes
the REST API and MCP ports. Compose service names such as `elasticsearch` and
`rtvi-vlm` do not resolve from this container, so set `ES_HOST`, `RTVI_VLM_URL`,
and `LVS_LLM_BASE_URL` in `.env` to addresses it can reach: a routable host or
IP, or `host.docker.internal` for services published on this host. To use
service names instead, attach the container to the dependencies' Docker network.

**Troubleshooting**

* **Container fails to start**: Check `docker logs video-summarization-server` for error messages
* **Cannot connect to external services**: Verify `RTVI_VLM_URL`, `ES_HOST`/`ES_PORT`, and `LVS_LLM_BASE_URL` in your `.env` file and confirm network connectivity
* **Port conflicts**: Change port numbers in `.env` file

## Configuration

Video Summarization uses YAML configuration files to customize behavior. The main configuration file is mounted in the Video Summarization container and referenced by `CA_RAG_CONFIG`.

### Configuration Structure

The configuration file (`services/video-summarization/config/config.yaml`,
mounted at `/opt/nvidia/via/config/default_config.yaml`) defines:

**1. Tools** - External service connections:

```yaml
tools:

  elasticsearch_db:
    type: elasticsearch
    params:
      host: !ENV ${ES_HOST}
      port: !ENV ${ES_PORT}
    tools:
      embedding: nvidia_embedding

  summarization_llm:
    type: llm
    params:
      model: !ENV ${LVS_LLM_MODEL_NAME}
      base_url: !ENV ${LVS_LLM_BASE_URL}
      max_tokens: 10240
      temperature: 0.2
      top_p: 0.7
      api_key: !ENV ${NVIDIA_API_KEY}

  nvidia_embedding:
    type: embedding
    params:
      enable: !ENV ${LVS_EMB_ENABLE:false}
      model: !ENV ${LVS_EMB_MODEL_NAME}
      base_url: !ENV ${LVS_EMB_BASE_URL}
      api_key: !ENV ${NVIDIA_API_KEY}
```

**2. Functions** - Processing pipelines:

```yaml
functions:
  summarization:
    type: vlm_structured_summarization_online
    params:
      time_overlap_threshold: 0.1
      max_events_per_batch: 50
      kafka_enabled: true
    tools:
      db: !ENV ${LVS_DATABASE_BACKEND:elasticsearch_db}
      llm: summarization_llm
  summarization_online:
    type: vlm_structured_summarization_online
    params:
      time_overlap_threshold: 0.1
      max_events_per_batch: 50
      kafka_enabled: true
    tools:
      db: !ENV ${LVS_DATABASE_BACKEND:elasticsearch_db}
      llm: summarization_llm
```

**3. Context Manager** - Active functions:

```yaml
context_manager:
  functions:
    - summarization
    - summarization_online
```

### Key Configuration Options

**Database**

* `type`: `elasticsearch` (default), `milvus`, `arango`, or `neo4j` — see [Database Backends](/system-components/vision-microservices/video-summarization#database-backends)
* `params.host`, `params.port`: Connection parameters
* `params.username`, `params.password`: Credentials, for the `arango` and `neo4j` types
* `tools.embedding`: Embedding model to use (reference to tool)

**Summarization LLM**

* `type`: `llm` (default)
* `params.model`: LLM model name (e.g., from `${LVS_LLM_MODEL_NAME}`)
* `params.base_url`: API endpoint (e.g., from `${LVS_LLM_BASE_URL}`)
* `params.max_tokens`: Maximum output tokens (default: 10240)
* `params.temperature`: Sampling temperature (default: 0.2)
* `params.top_p`: Top-p sampling (default: 0.7)
* `params.api_key`: Authentication key

**Nvidia Embedding**

* `type`: `embedding` (default)
* `params.enable`: Enable/disable embedding service
* `params.model`: Embedding model name
* `params.base_url`: Embedding service endpoint
* `params.api_key`: NVIDIA API key

**VLM Structured Summarization**

Used by the file summarization path (`POST /v1/summarize`). Configured as the `summarization` function in the context manager.

* `type`: Function type (`vlm_structured_summarization` (default))
* `params.time_overlap_threshold`: Threshold for overlapping time events (default: 0.1)
* `params.max_events_per_batch`: Maximum events per batch (default: 50)
* `params.kafka_enabled`: When `true`, the file path reads raw events from Elasticsearch (populated by the Kafka → Logstash pipeline) instead of using the in-process accumulation flow. Must be paired with `KAFKA_ENABLED=true` on the Video Summarization MS server. Default: `false` (reads from `${KAFKA_ENABLED}`)
* `tools.db`: Reference to database backend (from `${LVS_DATABASE_BACKEND:elasticsearch_db}`)
* `tools.llm`: Reference to LLM tool

**VLM Structured Summarization Online**

Used by the live-stream summarization path (`POST /v1/stream_summarize`). Configured as the `summarization_online` function in the context manager and must be registered alongside `summarization` for live-stream workflows to dispatch correctly.

* `type`: Function type (`vlm_structured_summarization_online`)
* `params.time_overlap_threshold`: Threshold for overlapping time events (default: 0.1)
* `params.max_events_per_batch`: Maximum events per batch (default: 50)
* `params.kafka_enabled`: When `true`, the aggregator reads raw events back from Elasticsearch (populated by the Logstash consumer) rather than expecting in-process accumulation. Required for the live-stream path. Must be paired with `KAFKA_ENABLED=true` on the Video Summarization MS server. Default: `false` (reads from `${KAFKA_ENABLED}`)
* `tools.db`: Reference to database backend (from `${LVS_DATABASE_BACKEND:elasticsearch_db}`)
* `tools.llm`: Reference to LLM tool

### Example: Event Detection

The default configuration is optimized for event detection in videos:

```yaml
functions:
  summarization:
    type: vlm_structured_summarization
    params:
      time_overlap_threshold: 0.1
      max_events_per_batch: 50
      kafka_enabled: !ENV ${KAFKA_ENABLED:false}
    tools:
      db: !ENV ${LVS_DATABASE_BACKEND:elasticsearch_db}
      llm: summarization_llm

  summarization_online:
    type: vlm_structured_summarization_online
    params:
      time_overlap_threshold: 0.1
      max_events_per_batch: 50
      kafka_enabled: !ENV ${KAFKA_ENABLED:false}
    tools:
      db: !ENV ${LVS_DATABASE_BACKEND:elasticsearch_db}
      llm: summarization_llm
```

This extracts structured events with timestamps. You can customize this for specific use cases:

### Event Merging

When the same real-world event spans multiple video chunks, it can appear as several adjacent entries of the same type with overlapping or contiguous timestamps. The `LVS_ENABLE_LLM_MERGING` flag controls how these are combined.

**Without LLM merge (default)**

Adjacent events of the same type are concatenated with a `|` separator. Each chunk's description is preserved verbatim:

```json
{
  "id": 4,
  "type": "ladder use",
  "start": 7.0,
  "end": 29.33,
  "description": "The individual approaches the ladder, suggesting intent to use it for accessing higher shelves; however, no climbing action occurs within the observed timeframe. | A worker wearing a yellow hard hat and high-visibility vest approaches a green rolling step ladder positioned between two aisles labeled D and E. The worker climbs the ladder cautiously while holding a cardboard box, ascends to the top platform, places the box securely on the upper shelf of aisle E, and then begins to descend back down the ladder. | A worker wearing a yellow hard hat and reflective vest stands on a green rolling step ladder positioned between two tall warehouse aisles (D and E). The worker reaches up to adjust or interact with items stored high on shelf D, then lowers their hands while maintaining balance."
}
```

**With LLM merge** (`LVS_ENABLE_LLM_MERGING=true`)

The LLM reads all adjacent descriptions and produces a single, coherent natural-language summary:

```json
{
  "id": 2,
  "type": "ladder use",
  "start": 10.0,
  "end": 40.0,
  "description": "A worker wearing a yellow hard hat and high-visibility vest climbs a green metal rolling step ladder positioned between shelves D and E in a well-lit industrial warehouse, ascending to the second-level platform to retrieve a white box labeled \"A110\" and prepare it for placement on the top shelf above."
}
```

Use LLM merge when clean, human-readable event descriptions are more important than preserving the raw per-chunk output. Note that it adds one LLM inference call per merged group, which slightly increases end-to-end latency.

### Database Backends

The configuration defines four database tools. Any of them can be selected as
the store for captions, events, and aggregated summaries.

| Tool name          | Database                | Connection variables                                                            |
| ------------------ | ----------------------- | ------------------------------------------------------------------------------- |
| `elasticsearch_db` | Elasticsearch (default) | `ES_HOST`, `ES_PORT`                                                            |
| `vector_db`        | Milvus                  | `MILVUS_DB_HOST`, `MILVUS_DB_GRPC_PORT`                                         |
| `graph_db_arango`  | ArangoDB                | `ARANGO_DB_HOST`, `ARANGO_DB_PORT`, `ARANGO_DB_USERNAME`, `ARANGO_DB_PASSWORD`  |
| `graph_db`         | Neo4j                   | `GRAPH_DB_HOST`, `GRAPH_DB_BOLT_PORT`, `GRAPH_DB_USERNAME`, `GRAPH_DB_PASSWORD` |

Two independent selectors choose between them:

* `LVS_DATABASE_BACKEND` (default `elasticsearch_db`) selects the store used by
  the `summarization` and `summarization_online` functions — the file and
  live-stream summarization paths.
* `LVS_QA_DATABASE_BACKEND` (default `graph_db`) selects the store used by
  `ingestion_function` for the question-answering path, together with
  `LVS_INGESTION_FUNCTION` (default `graph_ingestion`) and
  `LVS_RETRIEVER_FUNCTION`.

> **Note**
>
> Only Elasticsearch is provisioned inside the compose stack. Milvus, ArangoDB,
> and Neo4j are selected by pointing the microservice at an instance you deploy
> and operate yourself, so set the matching connection variables before switching
> `LVS_DATABASE_BACKEND` or `LVS_QA_DATABASE_BACKEND` away from their defaults.

Every database tool references the `nvidia_embedding` tool, which is disabled by
default (`LVS_EMB_ENABLE=false`). The Milvus, ArangoDB, and Neo4j backends
depend on embeddings, so enable `LVS_EMB_ENABLE` and set `LVS_EMB_MODEL_NAME`
and `LVS_EMB_BASE_URL` when you select one of them.

### Environment Variables

Configuration supports environment variable substitution using `!ENV ${VAR_NAME}`.
For the complete list, see the
[Environment Variables Reference](/system-components/vision-microservices/video-summarization#environment-variables-reference)
in the appendix.

## Kafka Integration

When deployed under the [Summarization with Streaming and Message Bus](/system-components/vision-microservices/video-summarization#summarization-with-streaming-and-message-bus) profile, the Video Summarization microservice publishes aggregated summarization results to a Kafka message queue and reads per-chunk raw VLM events that RTVI-VLM publishes to the same broker. A Logstash consumer service decodes the protobuf payloads from Kafka and indexes them into Elasticsearch (`default_<asset_id>`). This enables decoupling of the summarization pipeline from downstream consumers, analytics dashboards, and real-time alerting systems.

### Kafka Topics

The streaming and message bus profile uses two Kafka topics:

* **Raw VLM Events** (default: `mdx-vlm-captions`): Contains `nv.VisionLLM` protobuf messages produced by RTVI-VLM. Each message carries one chunk's caption result with frame metadata, sensor info, and the `info["doc_type"]="raw_events"` marker. After the Logstash consumer service decodes and indexes these messages into Elasticsearch, the `doc_type` value is stored at `metadata.content_metadata.doc_type` in the ES document.
* **Structured Summary** (default: `mdx-structured-events-summary`): Contains `nv.VisionLLM` protobuf messages produced by the Video Summarization microservice. Each call to `POST /v1/summarize` (file path) or `POST /v1/stream_summarize` (live-stream path) publishes one `info["doc_type"]="structured_events"` message per batch (up to `max_events_per_batch` events) plus one `info["doc_type"]="aggregated_summary"` message carrying the narrative summary. In Elasticsearch, the Logstash-mapped field path is `metadata.content_metadata.doc_type`.

**Configuration:**

Kafka integration is controlled by the following environment variables across
the Video Summarization stack:

* **KAFKA_ENABLED**: Enable or disable Kafka integration (`true` / `false`). Default: `false`
* **KAFKA_BOOTSTRAP_SERVERS**: Comma-separated list of Kafka broker addresses (e.g., `localhost:9092` or `kafka:9092` for the in-stack broker)
* **MESSAGE_BUS_TOPIC**: Topic for raw VLM events produced by RTVI-VLM and consumed by Logstash. Default: `mdx-vlm-captions`
* **KAFKA_STRUCTURED_SUMMARY_TOPIC**: Topic for structured events and aggregated summaries. Default: `mdx-structured-events-summary`
* **KAFKA_PORT**: Host port mapping for the in-stack Kafka broker's internal listener. Default: `9092`
* **KAFKA_EXTERNAL_PORT**: Host port mapping for the broker's external listener; also embedded in `KAFKA_ADVERTISED_LISTENERS`. Default: `9094`
* **KAFKA_ADVERTISED_HOST**: Hostname or IP that host-side or cross-host clients use to reconnect. Default: `localhost`

RTVI-VLM uses `MESSAGE_BUS`, not `KAFKA_ENABLED`, to select its generated-message
output bus. Set `MESSAGE_BUS=kafka` and `KAFKA_BOOTSTRAP_SERVERS` on the
RTVI-VLM service so it publishes raw events to the same broker. In the packaged
VSS Compose deployment, set `RTVI_VLM_MESSAGE_BUS=kafka`; keep
`RTVI_VLM_KAFKA_ENABLED=true` only as the legacy compatibility field while
Compose still forwards it. See [RTVI-VLM Kafka Integration](/system-components/vision-microservices/real-time-vlm#kafka-integration) for the producer-side configuration.

> **Note**
>
> `MESSAGE_BUS_TOPIC` configures RTVI-VLM and the Logstash consumer service.
> `KAFKA_STRUCTURED_SUMMARY_TOPIC` configures the Video Summarization MS and
> Logstash. Override both variables in `.env` when the default topic names
> conflict with an existing Kafka deployment.

**ctx-rag function configuration:**

The standard Compose workflow mounts
`services/video-summarization/config/config.yaml` at
`/opt/nvidia/via/config/default_config.yaml`. That configuration already reads
`KAFKA_ENABLED` for both summarization functions, so prefer the `.env` setting
above. The following YAML is shown only for advanced tuning:

```yaml
tools:
  elasticsearch_db:
    type: elasticsearch
    params:
      host: !ENV ${ES_HOST}
      port: !ENV ${ES_PORT}
      kafka_consumer_settle_secs: 5.0   # seconds the Video Summarization MS waits after the
                                         # RTVI SSE [DONE] event to let
                                         # Logstash flush the last chunk

functions:
  summarization:
    params:
      kafka_enabled: !ENV ${KAFKA_ENABLED:false}
  summarization_online:
    params:
      kafka_enabled: !ENV ${KAFKA_ENABLED:false}
```

### How Summaries Are Sent to Kafka

Summarization responses do not report Kafka delivery or Elasticsearch indexing
status. HTTP 200 indicates successful summary generation, not confirmation
that Kafka accepted the messages or that Logstash indexed them. Kafka producer
initialization failures can disable publishing, and send or delivery failures
can be logged without failing the summarization request. A successful SSE
response likewise does not confirm downstream delivery.

Inspect the Video Summarization MS producer logs, verify the expected messages
in Kafka, and check Logstash errors and indexed Elasticsearch documents before
treating the archive workflow as complete.

Under the streaming and message bus profile, the publishing flow depends on the API path:

1. **File path** (`POST /v1/summarize`): RTVI-VLM publishes per-chunk raw events to `MESSAGE_BUS_TOPIC` as it processes the file. The `LVS_CAPTION_SOURCE` environment variable controls where the Video Summarization microservice reads captions for aggregation: `sse` (default) uses captions received in-process via the RTVI SSE response, while `db` retrieves them from the Elasticsearch index populated by the Kafka → Logstash → ES pipeline. In `db` mode, the Video Summarization microservice waits `kafka_consumer_settle_secs` (default 5 seconds) so the Logstash consumer service can flush the last chunk. In both modes, the Video Summarization microservice aggregates captions via CA-RAG and publishes the structured events and aggregated summary to `KAFKA_STRUCTURED_SUMMARY_TOPIC`.
2. **Live-stream path** (`POST /v1/generate_captions` + `POST /v1/stream_summarize`): The client first calls `/v1/generate_captions` to start VLM captioning (fire-and-forget). RTVI-VLM publishes raw events to the Kafka topic, and Logstash indexes them into Elasticsearch. The client then calls `/v1/stream_summarize` with an optional `[start_time, end_time]` window. The Video Summarization microservice reads captions from Elasticsearch, aggregates them via CA-RAG, and publishes the structured events and summary to `KAFKA_STRUCTURED_SUMMARY_TOPIC` with a per-call upsert ID so repeated calls overwrite the latest result for that stream.
3. **Message keys**: All messages are published with the key `{request_id}:{chunk_idx|batch_i|doc_i}` (UTF-8 encoded) for partitioning and ordering.
4. **Requirement**: Set `KAFKA_ENABLED=true` on the Video Summarization MS.
   On RTVI-VLM, set `MESSAGE_BUS=kafka`; for the packaged VSS Compose
   deployment, the corresponding variable is `RTVI_VLM_MESSAGE_BUS=kafka`
   alongside the legacy `RTVI_VLM_KAFKA_ENABLED=true` compatibility field.
   Configure `KAFKA_BOOTSTRAP_SERVERS` for both services. Without the Video
   Summarization setting or RT-VLM message-bus publishing,
   `/v1/summarize` silently falls back to the legacy in-process aggregation
   flow. `POST /v1/summarize` now rejects `source_type=stream` with HTTP
   422, directing clients to use the dedicated live-stream APIs instead.

Video Summarization MS to RTVI Sticky Routing (`x-stream-id`)

### Video Summarization MS to RTVI Sticky Routing (`x-stream-id`)

The Video Summarization microservice attaches an `x-stream-id` HTTP header on every outbound request to RTVI-VLM. The header value is the asset or stream ID for the request (for example, the UUID returned by `POST /v1/stream/add`). Load balancers or service meshes sitting in front of RTVI-VLM can use this header to route all requests that belong to the same stream to the same RTVI-VLM instance, preventing split-brain state in multi-instance deployments.

The header is sent on the following Video Summarization MS to RTVI call paths:

* `start_captions` — `POST /v1/generate_captions` (live-stream captioning trigger)
* `generate_captions_stream` — `POST /v1/generate_captions` with `stream=True` (file-path captioning)

### Message Formats

Both topics carry `nv.VisionLLM` protobuf messages. The `info` map (`map<string, string>`) is used to distinguish doc types and carry per-stream metadata for the Logstash consumer service.

#### Raw VLM Events

Raw events are produced by RTVI-VLM, one per video chunk. See [RTVI-VLM VisionLLM Messages](/system-components/vision-microservices/real-time-vlm#kafka-integration) for the complete `VisionLLM` protobuf schema. Video Summarization MS-relevant `info` fields injected by RTVI-VLM include:

* **doc_type**: `"raw_events"`
* **collection_name**: Target Elasticsearch index (`default_<asset_id>`)
* **uuid** / **streamId**: Asset identifier for the source video or live stream
* **chunkIdx**: Zero-indexed chunk number within the request
* **start_pts** / **end_pts**: Chunk start and end timestamps in milliseconds
* **start_ntp** / **end_ntp**: Chunk start and end NTP timestamps (ISO 8601)
* **is_first** / **is_last**: Boolean markers for chunk boundary in the SSE stream

#### Structured Events and Aggregated Summary

Structured-summary messages are produced by the Video Summarization microservice after CA-RAG aggregation. The same `nv.VisionLLM` protobuf carries two distinct `doc_type` values:

* **doc_type=structured_events**: One message per batch of up to `max_events_per_batch` aggregated events. The `llm.queries[0].response` field carries a JSON-encoded array of event objects (`id`, `start_time`, `end_time`, `type`, `description`).
* **doc_type=aggregated_summary**: A single message per summarization call (`/v1/summarize` for files, `/v1/stream_summarize` for live streams) carrying the narrative `video_summary` text in `llm.queries[0].response`.

Video Summarization MS-injected `info` fields on every published message:

* **doc_type**: `"structured_events"` or `"aggregated_summary"`
* **collection_name**: `default_<sanitized stream_id>` — drives the Elasticsearch `_index` Logstash writes to
* **uuid**: Source asset or stream identifier
* **camera_id**: When set on the live stream, copied from the operator's `/v1/stream/add` payload
* **batch_i**: Zero-indexed batch number for `structured_events` messages \* **event_count** / **total_events**: Per-batch and per-call event counts ID for upsert: Logstash builds the Elasticsearch document `_id` deterministically as `<collection>:<uuid>:<doc_type>:<chunkIdx|batch_i|doc_i>`. Repeated `/v1/summarize` calls on the same stream therefore overwrite the same two ES documents (one `structured_events` per batch, one `aggregated_summary`) — this is intentional for at-least-once Kafka redelivery safety.

## MCP Server Integration

Video Summarization includes a Model Context Protocol (MCP) server that exposes the same functionality as the REST API in a format consumable by AI agents and tools like Claude Desktop, Cursor, and other MCP-compatible clients.

### Overview

The MCP server (`lvs_mcp.py`) provides:

* **Stdio Transport**: Default mode for direct integration with MCP clients
* **SSE Transport**: HTTP-based Server-Sent Events transport for network access
* **Tool-based Interface**: All REST endpoints exposed as MCP tools

### Configuration

The MCP server can be enabled/disabled and configured via environment variables:

* `LVS_ENABLE_MCP`: Enable/disable MCP server (default: `true`)
* `LVS_MCP_PORT`: Port for SSE transport (if not set, uses stdio)

**Example**: Enable SSE transport on port 38112:

```bash
export LVS_ENABLE_MCP=true
export LVS_MCP_PORT=38112
```

### Available MCP Tools

The MCP server exposes the following tools:

**Health & Status**

* `health_ready`: Check server readiness
* `health_live`: Check server liveness
* `get_metrics`: Get Prometheus metrics

**File Management**

* `add_file`: Upload media file
* `list_files`: List uploaded files
* `get_file_info`: Get file metadata
* `delete_file`: Delete a file

**Video Processing**

* `list_models`: List available VLM models
* `summarize_video`: Generate video file summary
* `generate_captions`: Start VLM captioning on a live stream (fire-and-forget)
* `stream_summarize`: Summarize a live stream over a time window
* `generate_vlm_captions`: Generate timestamped VLM captions
* `get_recommended_config`: Get recommended configuration

### Using with MCP Clients

**Claude Desktop Configuration**

Add to your Claude Desktop configuration:

```json
{
  "mcpServers": {
    "via-engine": {
      "command": "docker",
      "args": [
        "exec",
        "-i",
        "video-summarization-server",
        "python",
        "/opt/nvidia/via/src/lvs_mcp.py"
      ]
    }
  }
}
```

**Direct SSE Connection**

When running with `LVS_MCP_PORT` set, connect to:

* **SSE Endpoint**: `http://<host>:38112/sse`
* **Messages Endpoint**: `http://<host>:38112/messages`

**Example Tool Call**:

```json
{
  "name": "summarize_video",
  "arguments": {
    "id": "<file_id>",
    "model": "nvidia/cosmos3-nano-reasoner",
    "scenario": "police body camera",
    "events": ["pulling over", "arrest", "chasing"]
  }
}
```

## Best Practices

### Performance Optimization

1. **Chunk Duration**: Use 30-60 second chunks for optimal GPU utilization
2. **RTVI-VLM Tuning**: Configure batch size, frame counts, and per-process GPU allocation on the RTVI-VLM service. See [Real-Time VLM](/system-components/vision-microservices/real-time-vlm) for guidance.
3. **Live-Stream Sessions**: Set `LVS_DISABLE_DB_RESET_ON_REQUEST_DONE=true` to keep context-manager state intact across consecutive `/v1/stream_summarize` calls on the same stream.
4. **Logstash Settle Window**: Tune `tools.elasticsearch_db.params.kafka_consumer_settle_secs` (default 5 seconds) to balance ingest latency against the file-path summarize wall time when using the streaming profile.
5. **Video Token Pruning**: Enable EVS++ on the RTVI-VLM service to prune redundant video tokens and reduce per-chunk inference time on long videos. See *Efficient Video Sampling (EVS and EVS++)* in the [Real-Time VLM](/system-components/vision-microservices/real-time-vlm) documentation.

### Error Handling

* Enable `stream: true` for long videos to receive progressive updates
* Set appropriate `max_tokens` to avoid truncation
* Use `chunk_overlap_duration` to avoid missing events at boundaries

## Troubleshooting

### Common Issues

**API Returns 503 Service Unavailable**

* Another video is being processed (the Video Summarization microservice processes one video or live-stream request at a time)
* Wait for current processing to complete

**Out of Memory Errors**

* Tune RTVI-VLM batch size and frame inputs (see [Real-Time VLM](/system-components/vision-microservices/real-time-vlm))
* Decrease `chunk_duration`
* Switch to a smaller VLM model on RTVI-VLM

**Slow Processing**

* Add GPUs to the RTVI-VLM service
* Use a smaller or faster VLM model on RTVI-VLM
* Reduce `num_frames_per_chunk` on RTVI-VLM
* When using the streaming profile, lower `kafka_consumer_settle_secs` if the Kafka and Logstash pipeline is keeping up

**VLM Returns Incomplete or Invalid JSON Events (Missing or Empty Fields)**

* The VLM may sometimes produce events that are missing required fields (like `type`, `description`) or contain fields that are present but empty (e.g., `"type": ""`). This will lead to warnings such as:

```text
WARNING Skipping invalid event {'start_time': 75.01, 'end_time': 79.5}: 2 validation errors for Event
type
  Field required [type=missing, ...]
description
  Field required [type=missing, ...]

WARNING Chunk 1: vlm_pipeline_ctx is None - chunk span will be ***MASKED***
WARNING No events found in document 1

WARNING Skipping invalid event {'id': 2, 'start_time': 29.53, 'end_time': 29.53, 'type': '', 'description': 'There are no visible anomalies or artifacts in the frame.'}: 1 validation error for Event
type
  Field cannot be empty [type=value_error, ...]
```

* **Cause**: The VLM may not always strictly follow the JSON output schema. This can result in missing fields, or required fields present with empty values (such as an empty string for `type`), especially in difficult scenes or with certain models. When this happens, those events fail validation and are skipped.
* **Impact**: Some chunks may report zero extracted events, and the final summary may be incomplete for those time ranges.
* **Workarounds**:

  * Simplify the event list to reduce ambiguity for the model
  * Use `override_vlm_prompt` with a more explicit prompt that reinforces the required output fields (`start_time`, `end_time`, `description`, `type`), and clarifies that fields must not be empty
  * Increase the number of frames per chunk on RTVI-VLM to give the model more visual context
  * Retry the summarization request — VLM outputs can vary between runs

**MCP Server Not Connecting**

* Check `LVS_ENABLE_MCP=true`
* Verify `LVS_MCP_PORT` is accessible
* Check container logs: `docker logs video-summarization-server`

### Logs and Debugging

```bash
# Set log level
export VSS_LOG_LEVEL=DEBUG

# View logs
docker-compose logs -f lvs-server

# Check specific component
grep "MCP" /var/log/via/via-server.log
```

## API Error Codes

* **400 Bad Request**: Invalid input syntax
* **401 Unauthorized**: Missing or invalid authentication token
* **409 Conflict**: File is in use and cannot be deleted
* **422 Unprocessable Entity**: Failed to process request (validation error). `POST /v1/summarize` returns this status when called with `source_type=stream` — use `POST /v1/generate_captions` and `POST /v1/stream_summarize` for live-stream workflows instead. It also returns 422 when the media offsets are reversed (`start` later than `end`) or when `min_tokens` is greater than `max_tokens`.
* **429 Rate Limit Exceeded**: Too many requests
* **500 Internal Server Error**: Server-side error
* **503 Service Unavailable**: Server is busy or a dependency failure is detected, such as RTVI-VLM being unavailable. Not all dependency failures produce this status; Kafka delivery failures can be logged while the request succeeds.

`/v1/ready` checks RTVI-VLM health and returns 503 if the stream handler is
not initialized or RTVI-VLM is unavailable. It does not check the summarization
LLM or run inference. The standalone Compose health check uses `/v1/live`,
which returns 200 without checking dependencies. Run a summarization request
to verify end-to-end inference; a healthy container alone does not verify it.

## Appendix

### Environment Variables Reference

Complete list of available environment variables:

**API Configuration**

* `LVS_BACKEND_PORT`: Host REST API port for the standalone Compose stack (default: 38111); maps to fixed container port 38111.
* `BACKEND_PORT`: REST API container port for the standalone `docker run` script (default: 38111). The standalone Compose stack fixes this value at 38111.
* `BACKEND_HOST_PORT`: Host REST API port for the standalone `docker run` script (defaults to `BACKEND_PORT`). Not read by the standalone Compose stack.
* `LVS_MCP_PORT`: MCP container port for the standalone `docker run` script (default: 38112). The standalone Compose stack fixes this value at 38112.
* `LVS_MCP_HOST_PORT`: Host MCP port for the standalone `docker run` script (defaults to `LVS_MCP_PORT`). Not read by the standalone Compose stack, which publishes MCP on fixed host port 38112.
* `VSS_API_ENABLE_VERSIONING`: Enable /v1 prefix

**Client CLI**

Read by the `via_client_cli.py` client rather than by the server.

* `VIA_BACKEND`: Default server address for the client's `--backend` option (default: `http://localhost:8000`)

**API Keys**

* `NGC_API_KEY`: NVIDIA NGC API key
* `NVIDIA_API_KEY`: NVIDIA AI API key
* `OPENAI_API_KEY`: OpenAI API key

**S3 Configuration**

Media download runs inside RTVI-VLM, so set these on the RTVI-VLM container
rather than on the Video Summarization MS container. They are read by the AWS
SDK when a request supplies an `s3://` URL.

* `AWS_ACCESS_KEY_ID`: AWS access key ID (required for S3 URL support)
* `AWS_SECRET_ACCESS_KEY`: AWS secret access key (required for S3 URL support)
* `AWS_ENDPOINT_URL_S3`: AWS S3 endpoint URL (required for S3 URL support)
* `AZURE_OPENAI_API_KEY`: Azure OpenAI API key, used by the RTVI-VLM `openai-compat` backend

**VLM Configuration**

* `VLM_MODEL_TO_USE`: VLM backend selector — set to `openai-compat` when delegating inference to RTVI-VLM in proxy mode
* `VLM_SYSTEM_PROMPT`: Default VLM system prompt applied when a request does not supply one (default: empty)

**RTVI-VLM Configuration**

* `RTVI_VLM_URL`: URL of the RTVI-VLM service (e.g., `http://rtvi-vlm:8000` for the in-stack container, or an external host)
* `RTVI_VLM_IMAGE`: Container image for the in-stack RTVI-VLM service
* `RTVI_VLM_PORT`: Host port for the in-stack RTVI-VLM service (default: 8420; the container always listens on 8000)
* `RTVI_VLM_GPU`: GPU device ID assigned to the RTVI-VLM container. The Video Summarization service itself does not request a GPU.
* `RTVI_VLM_URL_PASSTHROUGH`: Route VLM calls through RTVI-VLM when set to `true`
* `RTVI_HEALTH_RETRIES`: Attempts to make while waiting for RTVI-VLM to report healthy at startup (default: 30)
* `RTVI_HEALTH_RETRY_INTERVAL`: Seconds between those attempts (default: 5)

EVS++ video token pruning is a feature of RTVI-VLM, so it is configured on that
service rather than on the Video Summarization MS server. Compose passes
`VIA_EVS_SESSION`, `VLM_VIDEO_PRUNING_RATE`, `VLLM_EVS_SIMILARITY_THRESHOLD`,
`VIA_EVS_TOKEN_BUDGET`, and `VIA_EVS_MAX_SESSIONS` from your `.env` file through
to the in-stack `rtvi-vlm` service. For descriptions, defaults, and supported
models, see *Efficient Video Sampling (EVS and EVS++)* in the
[Real-Time VLM](/system-components/vision-microservices/real-time-vlm)
documentation.

**Kafka Configuration**

* `KAFKA_ENABLED`: `true` enables the streaming profile end to end on the Video Summarization MS server; `false` for the base profile (default: `false`)
* `KAFKA_BOOTSTRAP_SERVERS`: Kafka broker bootstrap address used by the Video Summarization MS and RTVI-VLM (default: `kafka:9092` inside the docker network)
* `MESSAGE_BUS`: Message bus used by RTVI-VLM for raw VLM events (default: `kafka`)
* `MESSAGE_BUS_TOPIC`: Topic for raw VLM events produced by RTVI-VLM and consumed by Logstash (default: `mdx-vlm-captions`)
* `ERROR_BUS`: Message bus used by RTVI-VLM for error events (default: `kafka`)
* `KAFKA_STRUCTURED_SUMMARY_TOPIC`: Kafka topic for structured events and aggregated summaries (default: `mdx-structured-events-summary`)
* `KAFKA_PORT`: Host port mapping for the in-stack Kafka broker's internal listener (default: `9092`)
* `KAFKA_EXTERNAL_PORT`: Host port mapping for the broker's external listener; also embedded in `KAFKA_ADVERTISED_LISTENERS` (default: `9094`)
* `KAFKA_ADVERTISED_HOST`: Hostname or IP that host-side or cross-host clients use to reconnect (default: `localhost`)
* `LVS_CAPTION_SOURCE`: Controls where file-path Kafka aggregation reads captions: `sse` (default) uses captions received in-process via the RTVI SSE response, `db` retrieves captions from Elasticsearch populated by the Kafka → Logstash → ES pipeline
* `LVS_KAFKA_CONSUMER_SETTLE_SECS`: Seconds to wait after the RTVI-VLM SSE stream completes so Logstash can flush the final chunk, used when the YAML key is absent (default: `5.0`)

**Database Configuration**

Backend selection is covered in [Database Backends](/system-components/vision-microservices/video-summarization#database-backends).

* `LVS_DATABASE_BACKEND`: Store used by the summarization functions — `elasticsearch_db` (default), `vector_db`, `graph_db_arango`, or `graph_db`
* `LVS_QA_DATABASE_BACKEND`: Store used by the question-answering ingestion function (default: `graph_db`)
* `LVS_INGESTION_FUNCTION`: Ingestion function type (default: `graph_ingestion`)
* `LVS_RETRIEVER_FUNCTION`: Retriever function type
* `LVS_DISABLE_DB_RESET_ON_REQUEST_DONE`: `true` preserves request data and the per-asset Elasticsearch index after processing; `false` resets request data and automatically drops the index after file summarization completes. The Compose configuration defaults to `true`, while the service code defaults to `false` when the variable is unset. Live-stream summarization retains its index with either value because consecutive `/v1/stream_summarize` calls reuse the same asset ID.
* `ES_HOST`, `ES_PORT`: Elasticsearch connection
* `ES_TRANSPORT_PORT`: Elasticsearch transport port (default: 9302)
* `ES_MAX_SHARDS_PER_NODE`: Maximum number of shards allowed per Elasticsearch node (default: `2000`). Each `POST /v1/summarize` creates one `default_<asset_id>` index with one shard; raise this cap for long-retention or high-volume deployments, or reclaim indices with `DELETE /assets/{asset_id}/index`.
* `ES_JAVA_OPTS`: JVM heap flags for the in-stack Elasticsearch container (default: `-Xms4g -Xmx4g`). Elasticsearch recommends no more than 20 active shards per GB of heap — the 4 GB default supports roughly 80 active shards in steady state. Tune `ES_MAX_SHARDS_PER_NODE` and `ES_JAVA_OPTS` together when increasing retention.
* `MILVUS_DB_HOST`, `MILVUS_DB_GRPC_PORT`: Milvus connection, used when `vector_db` is selected
* `ARANGO_DB_HOST`, `ARANGO_DB_PORT`, `ARANGO_DB_USERNAME`, `ARANGO_DB_PASSWORD`: ArangoDB connection, used when `graph_db_arango` is selected
* `GRAPH_DB_HOST`, `GRAPH_DB_BOLT_PORT`, `GRAPH_DB_USERNAME`, `GRAPH_DB_PASSWORD`: Neo4j connection, used when `graph_db` is selected

**LLM and Embedding Configuration**

* `LVS_LLM_HOST`, `LVS_LLM_PORT`: LLM NIM host and port
* `LVS_LLM_MODEL_NAME`: LLM model name (e.g., `nvidia/nemotron-3.5-lightning-30b-a3b`)
* `LVS_LLM_BASE_URL`: LLM API base URL
* `LVS_ENABLE_LLM_MERGING`: Use the LLM to merge adjacent duplicate events into natural text (default: false). See [Event Merging](/system-components/vision-microservices/video-summarization#event-merging) for details and examples.
* `LVS_EMB_ENABLE`: Enable the embedding tool (default: `false`). Required by the Milvus, ArangoDB, and Neo4j backends.
* `LVS_EMB_MODEL_NAME`: Embedding model name
* `LVS_EMB_BASE_URL`: Embedding service endpoint

**Prompt Overrides**

These replace individual sections of the generated VLM prompt. Each is empty by
default, which keeps the built-in text.

* `LVS_PROMPT_VLM_ROLE`: Override the AI role and persona description
* `LVS_PROMPT_VLM_INSTRUCTION`: Override the instruction section
* `LVS_PROMPT_VLM_CONSTRAINTS`: Append additional constraints
* `LVS_PROMPT_VLM_STRUCTURED_OUTPUT`: Override the output format and JSON template

The CA-RAG summarization function additionally accepts `aggregation_prompt` and
`description_merge_prompt` as configuration keys rather than environment
variables. `description_merge_prompt` applies only when `LVS_ENABLE_LLM_MERGING`
is enabled.

**Feature Flags**

* `LVS_ENABLE_MCP`: Enable MCP server (default: true)
* `ENABLE_VIA_HEALTH_EVAL`: Enable health evaluation (default: false)
* `ENABLE_DENSE_CAPTION`: Request dense captions in addition to structured events (default: false)
* `LVS_DROP_EMPTY_EVENT_FIELDS`: Empty event fields are dropped from responses by default; set to `false` to keep them
* `LVS_AGGREGATION_EMPTY_RETRIES`: Retry budget for aggregation calls that return no events and no summary (default: 2; `0` disables retrying)
* `VSS_POST_PROCESS_ON_EACH_DOC_ADD`: Run post-processing after each document is added rather than at the end of the request (default: false)
* `LVS_CHUNK_DEBUG`: Debugging aid — when `true`, chunks are processed but summarization is skipped (default: false)

**Logging and Monitoring**

* `VSS_LOG_LEVEL`: Log level (`DEBUG`, `INFO`, `WARNING`, `ERROR`)
* `VIA_LOG_DIR`: Directory for VIA logs
* `VIA_ENABLE_OTEL`: Enable OpenTelemetry
* `VIA_OTEL_ENDPOINT`: OpenTelemetry endpoint (e.g., `http://localhost:4318`)
* `VIA_OTEL_EXPORTER`: OpenTelemetry exporter type (e.g., `console`)
* `VIA_CTX_RAG_ENABLE_OTEL`: Enable OpenTelemetry for context RAG
* `VIA_CTX_RAG_EXPORTER`: Context RAG exporter type
* `VIA_CTX_RAG_OTEL_ENDPOINT`: Context RAG OpenTelemetry endpoint (e.g., `http://localhost:4318`)

### Performance

Published E2E latency and sizing guidance are in [Video Summarization Performance](/appendix/performance/agent-workflows/video-summarization-performance).

### Glossary

* **CA-RAG**: Caption-Augmented Retrieval-Augmented Generation
* **VLM**: Vision-Language Model
* **NIM**: NVIDIA Inference Microservice
* **RTVI-VLM**: Real-Time Vision-Language Model microservice — see [Real-Time VLM](/system-components/vision-microservices/real-time-vlm)
* **RTSP**: Real-Time Streaming Protocol, the network protocol used for live camera streams
* **MCP**: Model Context Protocol
* **RAG**: Retrieval-Augmented Generation
* **CV**: Computer Vision
* **PPE**: Personal Protective Equipment

## Docs

- [API Reference](https://docs.nvidia.com/vss/dev/system-components/vision-microservices/video-summarization/api-reference.md):