Search Performance
Overview
The Search Workflow lets you query indexed video archives in
natural language. During ingestion, uploaded clips are embedded through
RTVI-Embed (Cosmos-Embed1-448p) and
RTVI-CV/Behavior Analytics. At query time,
each request is decomposed by the reasoning LLM into a retrieval action — embed, attribute,
or fusion search — and its arguments, and run through the VSS CLI (vss search run <action>).
The CLI retrieves candidates and verifies them with the critic agent (Cosmos3 Nano Reasoner
VLM), which is on whenever RTVI-VLM is reachable, before returning timestamped results.
The tables below benchmark that search performance. Query
latency and throughput measurements use the vad-r1-v2 / benchmark query set (121 natural-language queries
over 17 indexed clips) at concurrent query levels 1, 10, 20, and 50, issued through
vss search run <action> with the critic agent verifying every returned hit. Each run reports
client-side end-to-end (E2E) average and P90 latency, successful query throughput (QPS),
and per-stage latency. A separate
section reports end-to-end video ingestion latency and chunk counts on both platforms for files ranging from short clips to 2-hour sources. Use these numbers to size GPU choice and
expected query latency under concurrent search load on VSS 3.3.
Test Configuration
Search Performance
vss search run takes an action — embed, attribute, tag, fusion or object — so
there is no bare vss search run "<query>" form. Each benchmark query ran as a single
invocation carrying the decomposed arguments:
The CLI does not decompose. The Query decomposition (LLM) row in the stage tables is a
separate call made before each invocation, against the reasoning LLM at temperature 0 with
thinking disabled, reusing the prompt the Vision Agent’s search tool uses
(services/agent/packages/vss_agents/src/vss_agents/tools/search.py).
That call does not return an action. It returns a rewritten query, an attributes list, a
has_action flag, object_ids, and any source or time bounds the question mentioned. The
action is derived from those: object_ids present selects object; otherwise empty
attributes selects embed, and attributes with has_action true selects fusion or
false selects attribute. --original-query carries the user’s untouched question through
so the critic verifies against what was actually asked.
Reproducing the decomposition row therefore needs both the LLM call and this derivation in
front of the CLI — running vss search run alone reproduces every other row but not this
one. The Routed actions counts above are the actions invoked at 1 concurrent query, and
on this path they are also the retrieval that ran: the CLI passes the action straight through
to the search library, which executes the named mode without re-routing. Decomposition is
redone in every run and is not fully deterministic even at temperature 0, so the split
shifts by a few queries between concurrency levels (for example, H100 at 20 concurrent
queries routed 56 / 53 / 12).
Search (RTVI-CV) Concurrent Streams
On both platforms, RTVI-CV sustains up to 8 concurrent 1920x1080, 30 FPS streams without FPS
drops, using the profile’s default vision encoder settings (skip-interval=0,
smart-infer=0, ofa-predict=0). To support more concurrent streams without FPS drops,
use a combination of skip-interval, smart-infer, and ofa-predict.
For details on these parameters, see
Smart Embedding Inference
and Smart embedding properties
in the RTVI-CV documentation.
Video Ingestion Performance
E2E Performance
Avg and P90 client-side E2E latency and successful query throughput for the 121-query
vad-r1-v2 / benchmark run under increasing concurrent query load. E2E covers the
vss search run invocation: retrieval and critic verification of every returned hit. Query
decomposition runs before it and is not included; it is reported as its own row in the
Latency Stage Breakdown. Successful QPS is measured over the
whole run, so it does include decomposition.
Throughput is bounded by RTVI-VLM, which serves the critic agent. Each query fans out to up to five concurrent verification requests (one per returned hit), so the deployment reaches its VLM inference ceiling at around 10 concurrent queries. Beyond that, throughput does not rise and additional concurrency only adds queue time — see Reading these numbers.
H100
NVIDIA RTX PRO 6000 SE
Reading these numbers
Successful QPS peaks at 10 concurrent queries (1.003 on H100, 1.091 on RTX) and does not rise after that, because the deployment is already at its VLM inference ceiling by that point. It eases off slightly as load grows further, to 0.910 on H100 and 0.866 on RTX at 50. Latency, by contrast, scales roughly linearly with concurrency — 2.5s, 9.5s, 19.1s, 43.6s on H100 — so concurrency past the knee buys queue depth, not work. RTX is about 26% slower than H100 at one concurrent query (3.2s against 2.5s) but 8–12% faster on average from 10 onward; at 50 its P90 is the higher of the two (61.1s against 54.2s).
Critic verification is the largest single cost, and its share grows with load rather than
staying constant. Per-call VLM inference is 65% of E2E at one concurrent query on H100 and
88–90% at 20 and 50; RTX follows the same curve, from 62% to 92%. RTVI-VLM is therefore the
first thing to size: give it a dedicated GPU rather than sharing one with RTVI-CV, raise
RTVI_VLM_BATCH_SIZE within the available GPU memory, or scale
it out. Passing --critic-eval-count to vss search run reduces the per-query fan-out at the cost of
leaving lower-ranked hits unverified.
On H100 it is not the only thing to size past roughly 20 concurrent queries. Attribute search holds near its single-query cost through 20 — 0.384s to 0.468s — and then degrades at 50, to 1.624s (P90 4.097s). Nearly all of that is the attribute path’s text-embedding call, which climbs from 0.273s to 0.359s through 20 and then to 1.511s at 50. That call goes to RTVI-CV, not RTVI-Embed — the attribute leg embeds its text through RTVI-CV, while RTVI-Embed serves the embed-search path only. On RTX, attribute search stays between 0.340s and 0.495s at every level. Embed search, Elasticsearch query execution and VST clip resolution stay flat throughout on both platforms. If you expect sustained load above about 20 concurrent queries on H100, give RTVI-CV headroom as well as RTVI-VLM.
Latency Stage Breakdown
Per-stage average and P90 latency breakdown.
Calls/query is how many times a stage runs within one query. Stages that run more than once report a per-call duration, not the sum, so the figure is comparable with the end-to-end row. The critic verifies up to five hits per query (fewer when retrieval returns fewer; about 4.5 on average in these runs) and issues those verifications concurrently, so its per-call time is also very close to the wall-clock time the critic phase adds to a query.
H100
NVIDIA RTX PRO 6000 SE
Latency Stage Breakdown — 1 Concurrent Query
Latency Stage Breakdown — 10 Concurrent Queries
Latency Stage Breakdown — 20 Concurrent Queries
Latency Stage Breakdown — 50 Concurrent Queries
Video Ingestion Latency
This table shows average and P90 per-video ingestion latency, with chunk counts. Each row is one test: either a single video or a batch of multiple videos uploaded together. The Duration column shows how long each video is (or the range of video durations in a batch).
Ingestion latency runs from the start of a video’s upload until every index it feeds has stabilized: all of its chunk embeddings are written, RTVI-CV has finished the stream and stopped writing raw frames, and its behavior documents have stopped changing. The upload call itself returns within seconds; the rest is perception processing the video, which is why a long video takes about as long to ingest as it does to play.