InternVideo2-CLIP with TAO Deploy#

To generate an optimized NVIDIA^®TensorRT™ engine for InternVideo2-CLIP, the gen_trt_engine action takes an Open Neural Network Exchange (ONNX) file that the video_clip export action produces. For more information about training and exporting an InternVideo2-CLIP model, refer to Training and Exporting InternVideo2-CLIP.

Important

TAO Deploy consumes only the combined ONNX file, the single graph that holds both the vision tower and the text tower, and it needs a dynamic batch axis so that the engine can serve the batch size you configure for evaluate and inference. The shipped video_clip reference specifications set export.encoder_type: separate and export.batch_size: 1, so you must override both keys before you export a model that you intend to deploy.

Set the two export keys like this:

export:
  encoder_type: combined
  batch_size: -1

A separate-mode ONNX file still builds an engine, but evaluate and inference fail afterwards because the engine does not expose the expected three inputs and the image_embedding and text_embedding outputs. A statically batched ONNX file pins the optimization profile to that one batch size, so a larger evaluate.batch_size or inference.batch_size fails at run time.

TAO Deploy supports three actions for InternVideo2-CLIP: gen_trt_engine, evaluate, and inference.

gen_trt_engine#

The gen_trt_engine action builds the TensorRT engine from the combined ONNX file. The shipped deploy specification configures the action as follows, with example paths substituted for the required ??? values:

gen_trt_engine:
  onnx_file: /results/video_clip_experiment/export/video_clip_combined.onnx
  trt_engine: /results/video_clip_experiment/deploy/video_clip.engine
  batch_size: -1               # -1 for dynamic batch
  tensorrt:
    workspace_size: 8192       # MB
    data_type: fp32            # fp32 is the recommended precision
    min_batch_size: 1
    opt_batch_size: 8
    max_batch_size: 16

The shipped specification raises workspace_size to 8192 MB, which is larger than the 1024 MB default.

TensorRT Engine Generation Configuration#

The following table lists the parameters of the gen_trt_engine block.

Parameter	Type	Description	Default
`results_dir`	string	Path to where all the assets generated from a task are stored.	`null`
`gpu_id`	int	The index of the GPU to build the TensorRT engine.	`0`
`onnx_file`	string	Path to the ONNX model file.	Required
`trt_engine`	string	Path to the TensorRT engine generated should be stored. This only works with `tao-deploy`.	Required
`timing_cache`	string	Path to a TensorRT timing cache that speeds up engine generation. This will be created/read/updated.	`null`
`batch_size`	int	The batch size of the input Tensor for the engine. A value of `-1` implies dynamic tensor shapes.	`-1`
`verbose`	bool	Flag to enable verbose TensorRT logging.	`false`
`tensorrt`	collection		TensorRT Builder Configuration

TensorRT Builder Configuration#

The nested tensorrt block controls the builder precision and the optimization profile.

Parameter	Type	Description	Default
`workspace_size`	int	The size (in MB) of the workspace TensorRT has to run its optimization tactics and generate the TensorRT engine.	`1024`
`min_batch_size`	int	The minimum batch size in the optimization profile for the input tensor of the TensorRT engine.	`1`
`opt_batch_size`	int	The optimum batch size in the optimization profile for the input tensor of the TensorRT engine.	`1`
`max_batch_size`	int	Maximum batch size in the TensorRT optimization profile.	`16`
`layers_precision`	list	The list to specify layer precision.	`[]`
`data_type`	categorical	TensorRT precision: FP32 or FP16. FP16 automatically keeps numerically sensitive InternVideo2 vision block normalization reductions in FP32. Valid options: `fp32`, `fp16`.	`fp32`
`strongly_typed`	bool	Build a strongly-typed TensorRT engine that honors the explicit precision of a mixed-precision ONNX (for example, from ModelOpt AutoCast, which keeps the vision RMSNorm in FP32). Required for a pin- free FP16 engine: a weakly-typed FP16 build re-lowers the RMSNorm reduction and collapses accuracy. Use with `data_type: fp32` (graph carries the precision).	`false`

Engine-Build Notes#

Keep the following behaviors in mind when you build an InternVideo2-CLIP engine:

Precision: fp32 is the recommended precision, and the shipped specification sets it. An fp16 build is supported. Left unprotected, a weakly typed FP16 build re-lowers the vision tower’s root-mean-square normalization reduction and loses accuracy, so TAO automatically pins those numerically sensitive vision-block reductions back to FP32 and logs how many layers it pinned. The alternative is a strongly typed build, which you enable with tensorrt.strongly_typed: true and drive from a mixed-precision ONNX file that already carries the precision in the graph. Leave data_type at fp32 for that build, because the graph carries the precision.
Dynamic shapes: only the batch axis is dynamic. The frame count, channel count, height, width, and text sequence length are static in the ONNX file and pass through to the engine unchanged. TAO reads the batch dimension from the ONNX file itself. When that dimension is dynamic, the optimization profile uses tensorrt.min_batch_size, tensorrt.opt_batch_size, and tensorrt.max_batch_size. When it is static, the profile is pinned to that value and the three bounds are ignored. The gen_trt_engine.batch_size key does not override the batch dimension recorded in the ONNX file, which is why you set export.batch_size: -1 at export time. An ONNX file that carries a dynamic non-batch axis fails the build with an explicit error.
Export artifacts: after the build, TAO copies the *_config.yaml file and the *_tokenizer/ directory from the ONNX directory to the engine directory. The copy is a no-op when the two directories are the same, and TAO skips any destination that already exists.

Important

The evaluate and inference actions require the tokenizer directory. TAO looks for it beside the engine, then in a sibling export directory, then in the parent directory, and fails with an explicit error when it finds none. A missing configuration file is not fatal, but TAO then falls back to assumed defaults for model.type and model.canonicalize_text, which can degrade retrieval accuracy silently.

Ask the agent to run the gen_trt_engine action against your specification. For example:

Build an FP32 TensorRT engine for InternVideo2-CLIP from the combined ONNX file at
``s3://my-bucket/video_clip/video_clip_combined.onnx`` using ``trt-spec.yaml``. Write
the engine to ``s3://my-bucket/video_clip/video_clip.engine``. Run on the local Docker
backend.

Dataset Configuration for Deployment#

The deploy evaluate and inference actions score an explicit-relevance retrieval set. dataset.val.gt_queries names a JSON file that holds a gallery of video chunks and a list of text queries with their relevant chunks. dataset.val.metadata names the nested per-video chunk metadata that resolves each gallery chunk identifier to a video file and a time window.

Evaluation Dataset Configuration#

The following table lists the parameters of the dataset.val block.

Parameter	Type	Description	Default
`gt_queries`	string	Explicit-relevance evaluation file (for example, `domain_test_*.json`) with `gallery` and `queries` (each query carries a text prompt, a slice label, and a list of relevant chunk identifiers).	Required
`metadata`	string	Nested per-video chunk metadata JSON used to resolve each gallery `chunk_id` to its video path and time window.	Required
`video_root`	string	Optional root prepended to relative video paths.	`null`
`path_prefix_mapping`	collection	Optional mapping from original path prefixes to local prefixes, applied before `video_root`.	`{}`

Query File Format#

The query file names the gallery and the text queries. A minimal example looks like this:

{
  "gallery": [
    {"chunk_id": "CHAD/1_086_1#0"},
    {"chunk_id": "CHAD/1_086_1#1"}
  ],
  "queries": [
    {
      "query": "a person falls down in the parking lot",
      "slice": "anomaly",
      "relevant_clip_ids": ["CHAD/1_086_1#0", "CHAD/1_086_1#1"]
    },
    {
      "query": "a white sedan drives past the camera",
      "slice": "specific",
      "relevant_clip_ids": ["CHAD/1_086_1#1"]
    }
  ]
}

The chunk metadata file is a list of video records, each with its own chunk list. TAO also accepts the same list wrapped in a data key:

[
  {
    "dataset": "CHAD",
    "video_id": "1_086_1",
    "video_path": "/data/CHAD/1_086_1.mp4",
    "chunks": [
      {"chunk_index": 0, "start_time_sec": 0.0, "end_time_sec": 4.0,
       "start_frame": 0, "end_frame": 120},
      {"chunk_index": 1, "start_time_sec": 4.0, "end_time_sec": 8.0,
       "start_frame": 120, "end_frame": 240}
    ]
  }
]

TAO reads only the following keys from the query file:

gallery: a list whose entries supply chunk_id.
queries: a list of query objects, each of which supplies the following keys.
- query: the prompt text that TAO tokenizes and embeds.
- relevant_clip_ids: a list of gallery chunk identifiers, so a query can name any number of relevant chunks. The field name retains clip for historical reasons.
- slice: a label that groups queries in the per-slice metric breakdown. It defaults to all.

TAO forms the chunk identifier as <dataset>/<video_id>#<chunk_index> from the metadata record. TAO drops relevant identifiers that are absent from the gallery and logs a warning, and it skips any query that has no remaining relevant chunks. A gallery chunk_id that is absent from the chunk metadata is fatal instead, so make sure the two files agree.

Path Resolution#

TAO checks path_prefix_mapping first. The first matching prefix is rewritten, and TAO uses the resulting path as it is. Otherwise TAO uses absolute paths unchanged, and only relative paths receive video_root. The video_root setting does not rewrite absolute paths, which is what path_prefix_mapping is for.

Running Evaluation with a TensorRT Engine#

The evaluate action embeds the whole gallery of video chunks, embeds every text query, scores each query against the entire gallery by cosine similarity, and grades the ranking against that query’s explicit relevant chunk identifiers. TAO reports overall numbers and a breakdown per slice.

Evaluation and Inference Configuration#

The same configuration block backs both the evaluate action and the inference action.

Parameter	Type	Description	Default
`trt_engine`	string	Path to the combined TensorRT engine.	`null`
`batch_size`	int	Batch size for embedding extraction.	`8`
`results_dir`	string	Directory to save results.	`null`
`text_file`	string	Optional text file (one prompt per line) for text-embedding inference.	`null`

TAO reports the following metrics:

mAP: mean average precision over the full ranked gallery.
recall@1, recall@5, and recall@10: averaged over queries, the fraction of a query’s relevant chunks found in the top k.
hit@1, hit@5, and hit@10: the fraction of queries with at least one relevant chunk in the top k.
ndcg@1, ndcg@5, and ndcg@10: normalized discounted cumulative gain (nDCG) at k.
median_rank and mean_rank: statistics over the one-based rank of the first relevant chunk.
auc: the mean per-query area under the receiver operating characteristic (ROC) curve over the whole gallery.
num_queries and gallery_size: the number of scored queries and the number of gallery chunks.

TAO writes the results to <results_dir>/results.json as an object keyed by overall and by each slice name.

Important

This protocol is not the same as the PyTorch-side evaluate protocol. The PyTorch path runs bidirectional N-to-N retrieval or category-as-query classification over the validation set, while the deploy path runs multi-relevance text-to-video retrieval against a fixed gallery with explicit per-query relevance. The two sets of numbers are therefore not directly comparable. Expect agreement at the embedding level instead: the deploy preprocessing and metric code are ported from the training path, so an FP32 engine follows the same numerical contract. For the PyTorch protocol, refer to Training and Exporting InternVideo2-CLIP.

The metrics in this section assume an FP32 engine. Refer to Engine-Build Notes.

Ask the agent to run the evaluate action against the engine you built. For example:

Evaluate the InternVideo2-CLIP TensorRT engine at
``s3://my-bucket/video_clip/video_clip.engine`` against ``eval-spec.yaml`` and write the
retrieval metrics to my results directory. Run on the local Docker backend.

Running Inference with a TensorRT Engine#

The inference action extracts text embeddings, gallery-video embeddings, or both, and writes them as Hierarchical Data Format version 5 (HDF5) files under results_dir:

text_embeddings.h5: written when inference.text_file is set. That file is plain text with one prompt per line, and TAO skips empty lines.
video_embeddings.h5: written when both dataset.val.gt_queries and dataset.val.metadata are set.

Setting neither input fails with an explicit error.

Each HDF5 file holds a float32 embeddings dataset of shape N by D, an index-aligned variable-length string dataset named texts or chunk_ids, and the attributes num_texts or num_videos, embedding_dim, and embedding_type. TAO L2-normalizes the embeddings before it writes them.

The inference.trt_engine field accepts either an engine file or a directory that holds exactly one. Point it at the engine file, so that TAO finds the tokenizer and configuration artifacts beside it.

Note

A text-only inference run still executes the vision tower on a zero-filled dummy video for each batch, because the combined engine has a single fused graph.

Ask the agent to run the inference action against the engine you built. For example:

Run InternVideo2-CLIP inference with the TensorRT engine at
``s3://my-bucket/video_clip/video_clip.engine`` using ``infer-spec.yaml`` and the prompts
in ``s3://my-bucket/video_clip/prompts.txt``. Run on the local Docker backend.