Search

The Search Workflow enables natural language queries across video archives to locate specific events, objects, or actions.

Use Cases

• Event retrieval from large video archives
• Cross-video search for specific objects or actions
• Forensic analysis of recorded footage

Estimated Deployment Time: 15-20 minutes

The following diagram illustrates the search workflow architecture:

Key Features of the Vision Agent with Search:

• Upload videos to the agent for search.
• Semantic search of videos for key actions, events, and object attributes using embedding-based video indexing.
• Natural language query support (e.g., “find all instances of forklifts”).
• Automatic query decomposition for chat requests. The search agent extracts refined query text, source names, time windows, visual attributes, action intent, image-search context, and result-count hints.
• Four search routes: embed search for actions/events, attribute search for visual descriptors, fusion search for queries that combine actions and visual descriptors, and Search by Image for finding visually similar objects from selected bounding boxes on paused video frames.
• Filter and retrieve timestamped results using similarity scores and video/stream names.

What’s being deployed

• VSS Agent: Agent service that uses a configured LLM endpoint to route requests and orchestrate tool calls to VSS microservices and model endpoints (LLM/VLM NIMs) to answer questions and generate outputs
• VSS Agent UI: Web UI with chat, video upload, and different views
• VSS Video IO & Storage (VIOS): Video ingestion, recording, and playback services used by the agent for video access and management
• Nemotron LLM (NIM): LLM inference service used for reasoning, tool selection, and response generation
• RTVI-VLM: Real Time Video Intelligence VLM Microservice serving Cosmos3 Nano Reasoner. It backs the critic agent that verifies search results and the video understanding tool. There is no separate VLM NIM container in this profile
• Video Analytics API: REST query and browse surface over the Elasticsearch indices, used by the Search tab
• Phoenix: Observability and telemetry service for agent workflow monitoring
• ELK: Elasticsearch, Logstash and Kibana stack to index and search embeddings of video clips
• Kafka: A real-time message bus to publish embeddings, to be consumed and indexed by ELK for search
• RTVI-Embed: Real Time Video Intelligence Embed Microservice to generate action/event embeddings for videos and text, based on Cosmos-Embed1-448p-Anomaly-Detection
• RTVI-CV: Real Time Video Intelligence Computer Vision Microservice to generate object attribute embeddings for videos
• Behavior Analytics: Behavior Analytics microservice to perform sequential frame analysis for object detection and tracking in videos/streams.

This list describes the stock profile. The manual Compose deployment below uses it as it stands, and so does the Agent Skills deployment that names the in-stack Agent. The other two Agent Skills deployments do not: both drop the in-stack VSS Agent, Phoenix, and the Agent’s LLM. The NemoClaw deployment keeps the Web UI; a capability-composed build drops it. Ask for the in-stack VSS Agent if you plan to use the Agent REST API or the manual Agent UI workflow.

Search Workflow Data Flow

The search workflow has two related paths: ingestion and query execution.

During ingestion, uploaded videos and RTSP streams are registered with VSS Video IO & Storage (VIOS). VIOS notification webhooks then fan each source out to RTVI-Embed, which generates video/action embeddings, and to RTVI-CV, whose output Behavior Analytics turns into object-level behavior embeddings. Elasticsearch stores these embeddings in separate indices:

• mdx-embed-filtered-* for video/action embeddings used by embed search.
• mdx-behavior-* for object behavior embeddings used by attribute search and Search by Image.
• mdx-raw-* for raw frame/object data used when frame lookup is enabled for attribute search.

During query execution, the Agent UI Search Tab calls the Vision Agent chat interface. Queries are entered in the Chat Sidebar, which streams search_agent through /v1/chat/stream. An LLM decomposes the natural language query and selects the best search route. Matching clips populate the Search Tab results grid. Filter controls on the Search Tab refine those agent results; they do not run a separate search.

The Vision Agent chat path can run:

• Embed search: Generates a text embedding with RTVI-Embed/Cosmos-Embed1-448p-Anomaly-Detection and searches video/action embeddings in Elasticsearch.
• Attribute search: Generates text embeddings with RTVI-CV and searches behavior embeddings for visual descriptors such as clothing, color, or carried objects.
• Fusion search: Runs embed search first, then runs attribute search over each candidate window and reranks with a fusion score.
• Search by Image: When the user starts from a selected bounding box on a paused video frame, the workflow uses the selected object’s embedding from the behavior index to search for visually similar objects.

Prerequisites

Before you begin, ensure all of the prerequisites are met. See Prerequisites for more details.

Deploy

With Agent Skills

As an alternative to running the workflow manually, you can use VSS Agent Skills from a coding agent such as Claude Code, Codex, or NemoClaw.

First install the skills as described in Installing Skills and make it accessible to your coding agent.

Your coding agent drives the workflow: it deploys the stack, provisions sources, and runs retrieval. Step 1 offers three deployments, and the one you pick decides how you operate it afterwards. All three assume the same supported hardware as the manual workflow.

Step 1: Deploy the Search Stack

Deployment is driven by the vss-build-vision-ai skill. It can deploy the Search developer profile as it ships, or compose a leaner stack from the capabilities you name. Any prompt below can also name an image tag to run a release’s images instead of the containers.env default, for example “…with image tag 3.3.0.” See Container image tag. The skill presents the composed architecture for your confirmation before it generates or deploys anything.

Stock Search profile with the in-stack Agent

This deploys the profile unchanged, including the in-stack VSS Agent, the Agent UI, and Phoenix — the service set under What’s being deployed. Naming the Agent is what keeps it in the build, so this is the deployment to choose if you want the Agent UI or the Agent REST API.

Deploy prompt structure

Deploy the VSS search profile with the in-stack VSS Agent so I can use the Agent UI workflow.

What the agent does:

1. Loads the vss-build-vision-ai skill and detects the repository, GPU hardware, and host networking values.
2. Validates required credentials (such as NGC_CLI_API_KEY) and prompts you if any are missing.
3. Selects the search profile and a hardware profile that fits your system, then builds the deployment configuration.
4. Reviews the configuration with you, then deploys the containers.
5. Waits until all services are healthy and returns the endpoints available for the deployment.

Operate it either way: with the prompts in Steps 2 to 7 below, or through the UI steps under Manually.

Stock Search profile driven by NemoClaw

The same profile with NemoClaw as the harness. When the prompt does not name an agent, the skill asks whether to deploy NemoClaw and takes yes as the default. NemoClaw runs as a sandbox on the host and drives the deployment over its public routes, so the in-stack VSS Agent, Phoenix, and the Agent’s LLM are left out.

Deploy prompt structure

Deploy the VSS search profile.

The agent follows the same steps as above, then brings up the NemoClaw sandbox against the deployed origin and reports its chat URL. You reach the deployment through that chat, and the Web UI’s chat sidebar and Chat tab answer through it as well. The /api, /chat, and /websocket ingress routes are unavailable here. Operate this deployment with the prompts in Steps 2 to 7 below.

Custom search build

The skill can also compose a stack from the capabilities you name rather than from a fixed profile. State the capabilities you want and how you intend to reach them. Do not name services or list what to exclude; the skill selects a Foundation and keeps only what your stated capabilities reach.

This build carries no chat interface and no in-stack agent, so the VSS UI and Phoenix entries under Service Endpoints do not apply to it. Kibana and the Video Analytics API are retained as browse surfaces over stored data, and retrieval runs through the vss CLI.

Deploy prompt structure

Provide ingestion for streaming and on-demand videos with object
detection/tracking and video embeddings. Make the browse and operate surfaces
accessible via service-native REST API/CLI and exposed through single
ingress.

This prompt builds a retrieval-only stack. Search returns matches with similarity scores and time ranges, but no VLM is deployed, so results are not verified and no tags are produced to search. To include both, add a sentence naming them:

Provide ingestion for streaming and on-demand videos with object
detection/tracking and video embeddings. Verify search results and support
search over VLM-generated tags of the ingested video. Make the browse and
operate surfaces accessible via service-native REST API/CLI and exposed
through single ingress.

That added sentence does two things. Asking for verification puts RTVI-VLM in the build and routes /rtvi-vlm through the ingress, which also gives you visual follow-up questions about a clip: verification and those questions are both per-request calls to the same service, so asking for one gives you both. Asking for tag search additionally makes RTVI-VLM a VIOS fan-out consumer, so each newly registered source is tagged as it arrives — deploying the service alone never does that. Ask for neither and the service is left out, because retrieval itself never calls it.

What the agent does:

1. Reads the skill and its search references to determine the Foundation, its authoritative service set, the GPU layout, and sizing.
2. Composes the build as a delta on that Foundation. The VSS Agent, VSS Agent UI, Phoenix, and the LLM are dropped, because retrieval is driven through the CLI rather than a chat surface. RTVI-VLM is dropped as well unless you asked for verification or tag search.
3. Runs pre-flight checks on the repo path, GPU and Docker runtime, and NGC credentials.
4. Prepares the environment and the data directory tree, then writes the build artifacts under _builds/<name>/, including the resolved Compose file resolved.yml.
5. Deploys the resolved Compose file and waits until every service reports healthy, then reports the build’s public origin.

Step 2: Video Management

Replace <sample-data-dir> with the absolute extraction location from the Quickstart’s download step. For a coding agent running on the deployment host, provide the host path:

Provision <sample-data-dir>/dev-profile-sample-data/warehouse_sample.mp4 as an
on-demand VIOS source. Confirm the source appears in VIOS's sensor list and
that the configured consumers finish processing it.

If you selected NemoClaw, follow the Quickstart’s handoff, using warehouse_sample.mp4 as the filename in the transfer command. Then provide the sandbox path:

Provision /sandbox/warehouse_sample.mp4 as an on-demand VIOS source. Confirm
the source appears in VIOS's sensor list and that the configured consumers
finish processing it.

What the agent does on Agent-backed and headless deployments:

1. Reads the vss-manage-video-io-storage skill and its source provisioning reference.
2. Uses the recorded deployment configuration and registers the source once with vss vios add.
3. Lets VIOS’s mounted notification configuration fan the source out asynchronously to the configured consumers, such as RTVI-CV, RTVI-Embed, and optional RTVI-VLM tagging. Their outputs feed the indices queried by search.
4. Verifies the sensor is registered and online, then waits for the configured consumers to finish processing it.

The in-stack Agent deployment uses the same registration path. VIOS notification receivers perform the detection, embedding, and optional tagging fan-out; no Agent /complete call is required. The search skill queries the resulting archive and hands source registration to vss-manage-video-io-storage.

It may take up to a few minutes depending on the size of the video(s).

To ingest a live stream instead, have NVStreamer serve a file as a synthetic RTSP camera and provision that:

Provision a live RTSP source

Serve <sample-data-dir>/dev-profile-sample-data/warehouse_sample.mp4 as a synthetic
RTSP camera and register it once in VIOS. Confirm the live stream is visible
and the configured consumers have received it before reporting done.

For NemoClaw, use the file copied into the sandbox above:

Serve /sandbox/warehouse_sample.mp4 as a synthetic RTSP camera and register it
once in VIOS. Confirm the live stream is visible and the configured consumers
have received it before reporting done.

Before the CLI can search, it has to be pointed at the build’s ingress, but this is not a step you need to run. The agent does it on the first query: it resolves the deployment’s origin, runs vss configure against it, and repeats that after each ingestion, because the recorded index inventory is a snapshot and a freshly deployed stack records none.

For optional configuration from the host terminal, see With the VSS CLI.

Step 3: Search with a simple query

Your agent can search videos given natural-language queries, without requiring a chat UI/interface. Use its search capability to find all instances of forklifts in the uploaded video warehouse_sample.mp4 by using the following prompt:

Simple search prompt

Find all instances of forklifts in the uploaded videos.
Report the query results with source, timestamps, similarity, and the
clip/screenshot URL.
Don't fetch screenshot pixels unless I opt in.

What the agent does:

1. Reads the vss-search-archive skill to determine the retrieval path.
2. Confirms the source is registered in VIOS; if it is not, reports that and asks before ingesting.
3. Chooses a retrieval route and runs it through vss search run, which takes no endpoint because the CLI resolves everything from what vss configure recorded. The four routes match the ones the chat interface selects between: embed for actions and events, attribute for visual descriptors, fusion for queries combining both, and object for identity lookup by tracked object id.
4. Verifies the top results with the critic if the build includes RTVI-VLM. The CLI attempts this by default and skips it when the service is absent.
5. Presents results as a table with time ranges, similarity scores, verification verdicts, and the per-criteria breakdown.

Each result carries the source name, a description, start_time and end_time, the sensor id, a screenshot URL, the similarity score, any tracked object ids, and a verification verdict. The verdict is confirmed, rejected, or unverified, and the criteria breakdown lists the individual conditions extracted from the query and whether each was satisfied. See Critic Agent Overview for how the verdicts are reached.

Step 4: Search with additional filters

To search with additional filters:

I have a video sample-warehouse-ladder.mp4. I need to find instances of a person climbing a ladder with source type video_file, video source sample-warehouse-ladder.mp4, and top 5 results.

• VLM tag search: search the tags RTVI-VLM produced during ingestion with a prompt like Find clips tagged "forklift loading a pallet". Fusion search leaves those tags out of its ranking unless its tag leg is enabled with --w-tag on vss search run fusion, which is off by default.

Both stock deployments tag by default, because the profile’s VIOS notification configuration ships the RTVI-VLM fan-out enabled. A custom build tags only if you asked for tag search in Step 1.

Step 5: Ask a Follow-up Question

Follow-up questions related to the search results can also be asked in the same coding-agent conversation. For example:

What are the durations of the top 5 results?

Questions like this one are answered from the results already returned, so they work on any deployment. Questions about what is visible in a clip are different: they re-examine the video through RTVI-VLM. Both stock deployments include it; a custom build needs the verification prompt from Step 1. Without RTVI-VLM the agent reports the missing service rather than deploying one.

Step 6: Delete Videos or Streams

To remove an uploaded video or RTSP stream, issue the appropriate prompt:

Delete an uploaded video

Delete the video sample-warehouse-ladder.mp4.

Delete an RTSP stream

Delete the RTSP stream warehouse_sample.

Step 7: Teardown the deployment

To teardown the deployment, issue the following prompt:

Tear down the search deployment.

The default teardown removes the project’s containers, network, and named volumes, including the multi-GB NIM and RTVI model caches. To keep the caches for an immediate redeploy, ask for that explicitly:

Tear down the search deployment and preserve the model caches.

Bind-mounted data under VSS_DATA_DIR, such as Elasticsearch and Kafka data and VIOS recordings, survives either path and would carry into the next deployment. Clearing it is a separate step that needs root, so the agent will surface the command and confirm with you before running it.

On the NemoClaw deployment the sandbox is a separate lifecycle from the containers, so tearing down one leaves the other running. Ask for the sandbox to be destroyed as well when you are finished with it.

With the VSS CLI

Use the CLI directly to see what was recorded, or to point it at a different deployment. It is a host process invoked project-locally, never a globally installed vss and never inside a container:

VSS_REPO_ROOT="${VSS_REPO_ROOT:-$HOME/video-search-and-summarization}"
vss() { uv run --project "${VSS_REPO_ROOT}/libs/vss" vss "$@"; }
vss configure --base-url "<VSS_PUBLIC_URL>"
vss configure show

Replace <VSS_PUBLIC_URL> with the ingress origin returned for your deployment.

libs/vss is the CLI’s own workspace, so no extra is required. configure probes each route and records the ones that answer, along with the Elasticsearch index list, and configure show prints what it recorded.

Manually

Step 1: Deploy the Search Agent

The release Search configuration loads search_prompt_policy.SearchPromptWorker from the deployment’s mounted search_prompt_policy.py. Keep that module on PYTHONPATH when customizing the agent container; the supplied Compose and Helm deployments configure the mount and import path. Its prompt policy infers time filters only when the query explicitly requests a date or time. Explicit API start/end parameters still apply. For a custom Helm agent configuration, enable searchPromptPolicy when selecting this worker; the stock Search profile enables it automatically.

GPU requirements:

The search profile deploys on 2 GPUs, and both are shared. GPU 0 runs RTVI-CV alongside RTVI-VLM, which serves the critic from a Cosmos3 Nano Reasoner FP8 checkpoint at a memory fraction of 0.4 so that RTVI-CV fits beside it. GPU 1 runs RTVI-Embed alongside the LLM. Both LLM_MODE and VLM_MODE default to local_shared, and the profile marks GPU 0 and GPU 1 as shared rather than reserving either for a single service.

Offloading a model to a remote endpoint frees capacity on a shared GPU without changing the count, because RTVI-CV and RTVI-Embed stay on their pinned devices. Dedicating a model to its own GPU raises the count:

• Default: 2 GPUs. GPU 0 runs RTVI-CV and RTVI-VLM; GPU 1 runs RTVI-Embed and the LLM.
• Dedicated LLM + VLM: 4 GPUs. GPU 0 and GPU 1 keep RTVI-CV and RTVI-Embed; the LLM and the VLM each take a GPU of their own.
• Remote LLM: 2 GPUs. GPU 1 runs RTVI-Embed alone; the LLM is offloaded.
• Remote VLM: 2 GPUs. GPU 0 runs RTVI-CV alone; the VLM is offloaded.
• Remote LLM + VLM: 2 GPUs, running the retrieval services only.

Single-GPU hosts are the exception. On GB300, DGX Spark and AGX Thor the profile collapses every local service onto the one GPU. The two edge boards go further and require both models to be remote, since their GPU is already carrying the perception pipeline.

From the repository checkout root, enter the Compose deployment directory:

cd deploy/docker

overrides.env is the checked-in template. Copy it to any filename matching *-overrides.env before selecting a layout. Files matching that pattern are ignored by Git so local settings and credentials are not committed. The commands below use user-overrides.env as a concrete example; if you choose another name, use it consistently in every --env-file argument. Repeat the copy only when you want to discard prior layout changes:

PROFILE=search
cp "developer-profiles/dev-profile-${PROFILE}/overrides.env" \
  "developer-profiles/dev-profile-${PROFILE}/user-overrides.env"

Before starting Compose, complete the host settings and select one hardware layout below in deploy/docker/developer-profiles/dev-profile-search/user-overrides.env.

Environment-variable overrides: In deploy/docker/developer-profiles/dev-profile-search/user-overrides.env, update existing values in place; add a variable only when it is absent. COMPOSE_PROFILES is derived from values above it, so appending duplicate model-mode or model-slug variables can select the wrong services.

Host-path and service-address overrides (all hardware profiles):

• VSS_APPS_DIR: Absolute path to this repository’s deploy/docker directory.
• VSS_DATA_DIR: Absolute writable host directory for downloaded models, videos, and service data.
• HOST_IP: IP address or hostname that clients use to reach services on this VSS host. Set it to an address reachable by the intended clients.
• EXTERNAL_IP: Address VSS puts in externally reachable service URLs. It defaults to ${HOST_IP}; on a cloud host or when clients connect from outside the host network, set it to the instance’s public address or external hostname.

Set these required placeholder values in the existing assignments:

VSS_APPS_DIR=/absolute/path/to/video-search-and-summarization/deploy/docker
VSS_DATA_DIR=/absolute/path/to/vss-apps-data
HOST_IP=<reachable-host-ip-or-name>

Prepare host data directories before starting Compose. Export VSS_DATA_DIR with the same absolute path set above, then run this block. The guard stops before mkdir or chmod if the value is absent or is not absolute:

: "${VSS_DATA_DIR:?Export VSS_DATA_DIR with the value from user-overrides.env}"
case "$VSS_DATA_DIR" in
  /*) ;;
  *) echo "VSS_DATA_DIR must be an absolute path" >&2; exit 1 ;;
esac
mkdir -p \
  "$VSS_DATA_DIR/data_log/analytics_cache" \
  "$VSS_DATA_DIR/data_log/calibration_toolkit" \
  "$VSS_DATA_DIR/data_log/elastic/data" \
  "$VSS_DATA_DIR/data_log/elastic/logs" \
  "$VSS_DATA_DIR/data_log/kafka" \
  "$VSS_DATA_DIR/data_log/nvstreamer/streamer_videos" \
  "$VSS_DATA_DIR/data_log/nvstreamer/vst_data" \
  "$VSS_DATA_DIR/data_log/redis/data" \
  "$VSS_DATA_DIR/data_log/redis/log" \
  "$VSS_DATA_DIR/data_log/vss_video_analytics_api" \
  "$VSS_DATA_DIR/data_log/vst/clip_storage" \
  "$VSS_DATA_DIR/agent_eval/dataset" \
  "$VSS_DATA_DIR/agent_eval/results" \
  "$VSS_DATA_DIR/models"
chmod -R a+rwx \
  "$VSS_DATA_DIR/data_log" \
  "$VSS_DATA_DIR/agent_eval" \
  "$VSS_DATA_DIR/models"

Default container image tag

containers.env uses VSS_CONTAINER_TAG as the default tag for all images. Export the tag selected for this documentation branch before choosing a hardware layout:

export VSS_CONTAINER_TAG="3.3.0"

Any service-specific image tag or tag suffix shown inside a hardware tab supersedes or augments this common default. Use the SBSA settings shown in the GB300 and DGX Spark tabs when deploying on those platforms.

HARDWARE_PROFILE=H100
RTVI_VLLM_GPU_MEMORY_UTILIZATION=0.4

After replacing every placeholder in the selected layout, from deploy/docker validate the resolved configuration, then start the profile:

PROFILE=search
docker compose -f compose.yml \
  --env-file containers.env \
  --env-file "developer-profiles/dev-profile-${PROFILE}/.env" \
  --env-file "developer-profiles/dev-profile-${PROFILE}/user-overrides.env" \
  config --quiet
docker compose -f compose.yml \
  --env-file containers.env \
  --env-file "developer-profiles/dev-profile-${PROFILE}/.env" \
  --env-file "developer-profiles/dev-profile-${PROFILE}/user-overrides.env" \
  up --detach --pull always --force-recreate --build

On the first bring-up, Docker Compose pulls the required containers from the configured NGC registries and builds five support images locally: Elasticsearch, its initializer, the Kafka topic initializer, the broker health checker, and the search Kibana initializer. Those builds pull base images from Docker Hub and docker.elastic.co; the Kafka support-image builds also download jq from GitHub. Allow outbound access and disk space for both pulls and local builds. Depending on your network speed, this may take a few minutes.

Once the deployment is complete, check that all the containers are running and healthy:

docker ps

You can now access the Agent UI at http://<HOST_IP>:7777/.

This deployment uses the following defaults:

• LLM model: nvidia/nemotron-3.5-lightning-30b-a3b, served locally and sharing GPU 1 with RTVI-Embed
• VLM model: nvidia/cosmos3-nano-reasoner, FP8 (ngc:nim/nvidia/cosmos3-nano-reasoner:modelopt-fp8-final_format_fix), served by RTVI-VLM on GPU 0 at a memory fraction of 0.4
• Critic agent: chosen per request through the use_critic flag, on by default. There is no deploy-time flag

Available VLM Models

Super’s weights are roughly three times larger than Nano’s, so raise the VLM’s share of the GPU when you select the FP8 checkpoint:

RTVI_VLLM_GPU_MEMORY_UTILIZATION=0.55

VLM_NAME must match the id RT-VLM advertises at /v1/models for the selected checkpoint; a mismatch fails with 400 BadParameters: No such model.

Embedding Models

The search profile deploys with Cosmos-Embed1-448p-anomaly-detection for RT-Embed and SigLIP2 for RTVI-CV by default. To use different embedding models, see Changing Embedding Models below.

Step 2: Video Management

Click on the Video Management tab in the left sidebar to access video management features. You can add video sources by uploading video files or adding RTSP streams.

Upload a Video

From the UI’s Video Management tab, upload the video file warehouse_sample.mp4 from your local machine and wait for the upload to complete. It may take a few minutes depending on the size of the video. Once the video is uploaded, it appears in the video list.

The upload streams the file to VIOS, whose notification webhooks fan it out to RTVI-CV, RTVI-Embed, and RTVI-VLM for detection, embedding, and tagging. Uploaded files support video/mp4 and video/x-matroska content types.

Add an RTSP Stream

From the UI’s Video Management tab, add a live RTSP stream as a video source by providing:

• RTSP URL (required): The RTSP stream URL (e.g., rtsp://<HOST_IP>:<PORT>/nvstream/tmp/nv_streamer/videos/warehouse_sample.mp4)
• Sensor Name (required): A descriptive name for the camera (e.g., warehouse_sample.mp4). This field is automatically populated from the RTSP URL but can be edited.

Once added, the RTSP stream appears in the video list alongside uploaded videos and can be used for search queries.

When an RTSP stream is added, VIOS registers it and its notification webhooks fan it out to RTVI-CV, RTVI-Embed, and RTVI-VLM, as for uploaded videos.

Create an RTSP Stream from a Video Using NVStreamer

If you have a video file and want to stream it as an RTSP source, you can use the NVStreamer service that is deployed alongside other services.

Access NVStreamer

The NVStreamer UI is available at http://<HOST_IP>:31000, where <HOST_IP> is the same IP address used for the agent UI.

Upload a Video to NVStreamer

1. Open the NVStreamer UI at http://<HOST_IP>:31000.
2. Click on the Media Upload tab in the left sidebar.
3. Click or drag video files to the upload area.
4. Wait for the upload to complete.

Get the RTSP URL

1. After the upload completes, click on the Dashboard tab.
2. Locate your video in the table and copy the RTSP URL using the copy button.

Add the RTSP URL to the Agent

Add the copied RTSP URL (e.g., rtsp://<HOST_IP>:31556/nvstream/tmp/nv_streamer/videos/warehouse_sample.mp4) as an RTSP stream in the UI’s Video Management tab (see above) to use it as a video source for search queries.

Step 3: Search with the Vision Agent chat

The Search tab includes a Vision Agent chat interface on the right side. All Search Tab results come from this chat path. The Vision Agent automatically selects the best search method based on your query. You can also specify filters directly within your query instead of using the filter panel.

Using the Chat Interface

1. Navigate to the Search tab.

2. Click the round Chat button at the bottom right to open the chat sidebar.

3. Open the chat settings popover next to the chat input and configure:

   • Search media source type: Select from the dropdown (for example, video_file or rtsp) to specify which source index to search.
   • Enable Critic: Toggle the critic agent on or off for this query. The toggle is on by default; turn it off to skip VLM verification of the results for this request (e.g., for faster responses or when exploring broad results).

4. Enter a natural language query in the chat input at the bottom (e.g., a person with green jacket).

5. Press Enter or click the send button to submit your query.

Result Returned by the Agent

The Vision Agent processes your query, selects the best search method (Embed, Attribute, Fusion, or Search by Image), and returns matching video clips in the left panel.

Each response in the chat window includes:

• Intermediate Steps — Expandable section showing the sequence of function calls and tool invocations the agent makes to process your query.

• Reasoning Trace — Step-by-step breakdown of the agent’s decision-making: query decomposition, search method selection, and result summary. Expand each step for full details.

• Search Results Summary — Number of matching videos found (e.g., “Found 10 matching videos”).

• Search API result (JSON) — Raw JSON response with detailed metadata for each result:

  • video_name: Name of the source video

  • description: Match description

  • start_time / end_time: Clip timestamps in ISO 8601 format

  • similarity: Cosine similarity score

  • sensor_id: Source sensor identifier

  • screenshot_url: URL to the clip thumbnail

  • object_ids: Detected object identifiers

  • critic_result (only present when the critic agent is enabled for the request): Verification details for the clip:

    • result: "confirmed" or "rejected".
    • criteria_met: A map of each criterion extracted from the query to a boolean indicating whether the clip satisfies it (e.g., {"subject:person": true, "carrying boxes": false, "wearing a yellow safety vest": true}).

• Download — Download the JSON response as a file.

• Regenerate response — Re-run the query for updated results.

search_agent:
  fusion_method: rrf
  rrf_k: 60           # RRF constant k
  rrf_w: 0.5          # RRF weight for attribute component

search_agent:
  fusion_method: weighted_linear
  w_embed: 0.35        # Weight for embed score
  w_attribute: 0.55    # Weight for attribute score

search_agent:
  fusion_method: rrf_with_attribute_rank
  rrf_k: 60
  rrf_w: 0.5

search_agent:
  embed_confidence_threshold: 0.1  # Minimum embed score to proceed with fusion

search_agent:
  top_percent_filter: 0.9

Step 4: Narrow results with filters

To try additional filters, upload the video file sample-warehouse-ladder.mp4 (following the steps in Step 2: Upload a video). Run a query as in Step 3, then collapse the chat sidebar with the chevron button (Collapse Chat sidebar) on its left edge. The controls below are in the left sidebar and are disabled while the chat sidebar is open or a response is streaming. They narrow the results the query returned:

• Expand Filters in the left sidebar. Changes apply as you edit the fields:
  • Video sources: Show results only from the selected videos
  • Min Cosine Similarity: Set a minimum similarity threshold (-1.00 to 1.00) to filter results based on how closely they match your query. Set a lower threshold for broader results, or raise it for high-confidence matches. Optimal values vary depending on the video content.
  • Show top K Results: Set the maximum number of returned results to display (default 10)

Configure the desired filters. They refine the clips already returned by the agent and do not issue a new search.

For example, in the filter panel:

• Select specific Video sources to show results only from those videos
• Set Min Cosine Similarity to 0.2 to only show results with a similarity score of 0.2 or higher
• Set Show top K Results to 5 to display maximum of top 5 results

After configuring the filters, the search results will be refined based on the applied criteria.

Critic Agent Overview

The Critic Agent is a specialized agent that reviews search results with a VLM and annotates each evaluated result as confirmed or rejected.

When configured, the critic agent reviews search results and annotates evaluated clips before the final response is assembled. If additional search iterations are enabled, rejected results can cause the backend to fetch more candidates. The default search profile sets search_max_iterations: 1, so the critic verifies the initial result set and the result JSON includes the verdict in critic_result.

Enabling/Disabling the Critic Agent

The critic agent runs by default on every query, and whether it runs is decided per request rather than at deploy time. In the Vision Agent chat, open the chat settings popover next to the chat input and use the Enable Critic toggle to turn the critic agent on or off for each query. The toggle is on by default; turn it off when you want raw search results without VLM verification, for example for faster responses or when you are exploring broad results. This toggle corresponds to the use_critic flag in the search request payload.

Because the choice is per request, there is no deployment-wide switch, and the search profile always deploys RTVI-VLM since it also serves the video understanding tool. What the server config controls is how much the critic evaluates, through num_videos_to_evaluate below.

Partial Verification (num_videos_to_evaluate)

The num_videos_to_evaluate configuration parameter controls how many of the top search results are sent to the VLM for verification. Results are ordered by relevance score, and only the first N results (determined by this parameter) are evaluated by the critic. The remaining results are returned as-is without verification.

For example, with the default configuration of num_videos_to_evaluate: 5 and a search that returns 10 results:

• Results 1—5 (highest relevance) are sent to the VLM and receive a verification status (confirmed, rejected, or unverified) along with a criteria_met breakdown.
• Results 6—10 are included in the final output with a verification status of --, indicating they were not evaluated.

This means the final response contains a mix of verified and unverified results. Verified results have an explicit pass/fail status, while unverified results have no critic metadata. This trade-off reduces VLM inference cost and latency at the expense of full coverage.

To verify all results, set num_videos_to_evaluate to a value equal to or greater than the expected result count, or remove the parameter entirely (the code default evaluates all results when unset).

# In config.yml under the critic_agent function
critic_agent:
  num_videos_to_evaluate: 5          # Only verify the top 5 results (default in deployment config)

How the Critic Agent Appears in the Reasoning Trace

When the critic agent is enabled for a request, the Vision Agent’s reasoning trace shows the verification flow:

1. Verifying N results with critic agent — The critic agent is invoked to evaluate the candidate set (for example, “Verifying 9 results with critic agent”).
2. Critic verification complete — The trace reports how many results were verified and rejected (for example, “Critic verification complete: 2/5 results verified, 3/5 results rejected”).
3. Found N results — The final count reflects the result set returned after any configured search iteration. Each evaluated result includes a critic_result field in the JSON output.

Critic Agent Input and Output

The critic agent uses a VLM (Vision Language Model) to verify each search result clip against the user query. It receives the query and metadata for multiple video clips, fetches and analyzes each clip via the VLM, then classifies each clip as confirmed or rejected.

Input — For each search result clip, the critic receives:

• The user query (e.g., “a person carrying boxes”).
• Video metadata for that clip: sensor_id, start_timestamp, and end_timestamp.

Verification — For each clip:

• The query is turned into a verification prompt that asks the VLM to break the query into criteria and judge each as true or false for that video.
• A playable URL for the clip is obtained and the clip plus the prompt is sent to the VLM.
• The VLM returns a small JSON object per clip, e.g., {"subject:person": true, "carrying boxes": false}.

Decision — Each clip is classified as follows:

• CONFIRMED — Every criterion is true → keep the result.
• REJECTED — Any criterion is false. The result is annotated as rejected, and the system can increment top_k and run another search iteration if configured.

For each verified clip, the critic output is returned as a critic_result object on that result, with the following fields:

• result — One of "confirmed" or "rejected".
• criteria_met — A map of each criterion extracted from the query to a boolean indicating whether the clip satisfies it (e.g., {"subject:person": true, "carrying boxes": false, "wearing a yellow safety vest": true}).

How the Critic Result Is Displayed in the UI

When the critic agent has run for a request, the search results panel and chat response display the verification details alongside each clip:

• Each evaluated result card shows a colored badge that reflects the critic verdict:

  • ✓ Confirmed — green badge.
  • ✗ Rejected — red badge.

• The result card border is colored to match the verdict (green for confirmed, red for rejected). Cards from a search that did not run the critic agent retain the default border.

• Below the badge, each criterion extracted from the query is listed as a small chip with a green check (✓) when satisfied or a red cross (✗) when not satisfied.

• Search results are sorted by critic verdict before similarity: Confirmed clips appear first, then Rejected. Within each verdict group, clips are ordered by descending similarity score. When the critic did not run for the request, results retain their original similarity ordering.

• In the chat response, the same per-result outcome is summarized in a table with a Critic column (e.g., confirmed (subject:person: ✓, carrying boxes: ✓, walking through the warehouse: ✓)), and the full critic_result object is included in the Search API result (JSON) payload.

Step 5: Delete Videos or Streams

You can delete uploaded videos or RTSP streams from the UI’s Video Management tab. Once deleted, the selected videos or streams are removed from the video list and no longer appear in search results.

Step 6: Teardown the agent

To teardown the agent, run the following command:

cd deploy/docker
PROFILE=search
docker compose -f compose.yml \
  --env-file containers.env \
  --env-file "developer-profiles/dev-profile-${PROFILE}/.env" \
  --env-file "developer-profiles/dev-profile-${PROFILE}/user-overrides.env" \
  down -v --remove-orphans

This command stops the manually deployed search-profile containers and removes their Compose volumes.

Service Endpoints

Once deployed, the following services are available:

Changing Embedding Models

Real-Time Embedding

The Real-Time Embedding microservice (RT-Embed) supports the Cosmos-Embed1 model, a joint video-text embedder. Cosmos-Embed1-448p-anomaly-detection is the default variant deployed with the VSS search profile.

To use a different model, for example, Cosmos-Embed1-448p set the environment RTVI_EMBED_MODEL and MODEL_PATH variables as follows:

RTVI_EMBED_MODEL=cosmos-embed1-448p
MODEL_PATH=git:https://huggingface.co/nvidia/Cosmos-Embed1-448p

• RTVI_EMBED_MODEL — Identifier for the embedding model used by RT-Embed
• MODEL_PATH — HuggingFace repository URL from which RT-Embed downloads the model at startup

Real-Time Object Embedding

The Real-Time Video Intelligence CV microservice (RTVI-CV) supports CLIP-style image (vision encoder) and text embedder models. SigLIP2 is the default model deployed with the VSS search profile.

Output Embedding Dimension

The object embeddings are stored in elasticsearch as vector embeddings, using a fixed dimension of 1152 for SigLIP2.

Known Issues

• A race condition between RTVI-embed and LLM NIM during deployment can result in an unhealthy state for the RTVI-embed container. To resolve this:

  1. Stop the LLM NIM.
  2. Wait for RTVI-embed to become healthy.
  3. Restart LLM NIM.

• Queries with negative intent (e.g., “people without a yellow hat”) may return the same results as positive intent queries (e.g., “people with a yellow hat”).

• Sometimes, the agent may also return false positive results (i.e., results that are not relevant to the query).

• Queries with a single word (e.g., “person”) may return no results.

• The duration of video clips in search results may be longer than the displayed duration.

• ‘Description’ is empty in the response generated by the Vision Agent chat interface.

• Uploading a video through the Agent chat sidebar in the Search profile returns an Internal Server Error. During upload through chat sidebar, query is sent to the agent, and the top agent for search currently does not have the necessary support to handle this.

• By default, the timestamps for uploaded videos start from 2025-01-01 00:00:00.

• When an RTSP stream is added, an extra mdx-raw-1970-01-01 index may be created in Kibana for the initial frames. This index can be safely ignored; subsequent frames are written to the current-date mdx-raw-* index, and search results are unaffected.

• Renaming an uploaded RTSP stream after it has been added is not supported. We would need to delete the existing entry from the Video Management tab and re-add the RTSP stream with the new name.

• Deleting an RTSP stream that has ended, may subsequently fail new stream addition or a new video upload.

• When a source video is removed from NVStreamer, the corresponding RTSP stream continues to appear as a live input in the VSS UI Video Management tab.

• When the critic agent is enabled and VLM service is not available, the search results may not appear in the main window.

• For RTSP streams with H265 encoding that have been removed, thumbnail may not be visible in the VSS UI. See Image capture failure for more details.

• An ‘Index not found’ error may occur, when there are no videos corresponding to the source type selected.

• When uploading a video to VIOS, if the video is larger than the maximum upload size, the upload may fail. See Why do large file uploads to VIOS fail? for more details.