InternVideo2-CLIP Introduction#

InternVideo2-CLIP is a dual-encoder video-text embedding model that aligns short video clips and text in a shared 512-dimensional space. You can use it for text-to-video retrieval, zero-shot video classification, and video-anomaly search.

TAO Toolkit supports fine-tuning InternVideo2-CLIP on custom video-text data and zero-shot evaluation of the pretrained model.

The video_clip task supports the following subtasks:

  • train

  • evaluate

  • inference

  • export

  • default_specs

The gen_trt_engine action belongs to TAO Deploy. It is not a video_clip subtask.

For full training, evaluation, inference, and export commands, refer to InternVideo2-CLIP Training and Deployment.

For downstream application examples that use InternVideo2-CLIP embeddings, refer to Using InternVideo2-CLIP Embeddings.

For engine generation and NVIDIA®TensorRT™ usage notes, refer to Deploying InternVideo2-CLIP.

Supported Models#

The video_clip task supports a single model.type value, internvideo2-clip-l14. It pairs an InternVideo2 L14 vision tower, which uses a patch size of 14, 24 transformer blocks, and an embedding width of 1024, with a MobileCLIP mobileclip_b text tower, which uses 12 transformer blocks and a 77-token context length. Both towers project into a shared 512-dimensional embedding space. The default input resolution is 224, and the default model.num_frames value is 8.

Model type

Image size

Frames

Embedding dimension

Vision tower

Text tower

internvideo2-clip-l14

224

8

512

InternVideo2 L14

MobileCLIP mobileclip_b

The clip task handles the image-only C-RADIO, SigLIP2, and OpenCLIP families. The video_clip task rejects any other model.type at build time, because the video dataloader emits a five-dimensional [B, T, C, H, W] batch that only the InternVideo2-CLIP adapter consumes.

Note

The model.canonicalize_text and dataset.augmentation settings have no effect on this task. The InternVideo2-CLIP adapter applies the same fixed frame transform for training and evaluation.

Model Weights#

InternVideo2-CLIP accepts two initialization forms: a set of component weight sources, or one complete checkpoint. Each weight-source field defaults to null, so you must choose one of the two forms in your experiment specification.

In the component form, model.internvideo2clip_hf_id supplies the vision encoder and the CLIP alignment head, and model.text_encoder supplies the MobileCLIP text tower. The model.text_encoder field is required in this form. The model.vision_encoder and model.clip_head fields are optional explicit overrides for the two components that model.internvideo2clip_hf_id otherwise provides. Each of these fields accepts either a local file path or a HuggingFace repository identifier. TAO resolves repository identifiers through the ambient HuggingFace cache, so set HF_HOME before you run any subtask.

In the complete form, model.pretrained_ckpt points at a local .pth file that holds the complete model state dictionary. When you set this field, it overrides everything else and TAO ignores the component sources.

Within the component form, TAO loads the vision keys first, then the text keys, and applies the alignment head last. The alignment head overwrites any overlapping vision and text keys.

Setting every weight-source field to null constructs the architecture with random initialization. Training from random initialization is not a validated path in this release.

The suggested HuggingFace repository for the vision encoder and the alignment head is OpenGVLab/InternVideo2_distillation_models. TAO pins a specific revision of that repository. A repository identifier that you supply yourself resolves at its own default branch instead.

Set model.text_encoder to the MobileCLIP text-tower weights. The expected file is mobileclip_blt.pt, which you obtain from HuggingFace.

Note

The model.text_encoder field is required whenever you set model.pretrained_ckpt to null and set any other weight source, because the MobileCLIP weights are not part of the InternVideo2-CLIP repository. Omitting it in that case raises an error at model build time. The one exception is the all-null configuration described above, which builds the architecture without pretrained weights.

Limitations#

Keep the following constraints in mind when you plan an InternVideo2-CLIP experiment.

  • The task supports only the internvideo2-clip-l14 model type.

  • The dataset.<split>.type field accepts only video_text. This task has no image-text dataset type.

  • The model configuration fixes model.num_frames and bakes it into the temporal position embedding at build time, so you cannot change it for an existing checkpoint.

  • The model.use_flash_attn, model.use_fused_rmsnorm, and model.use_fused_mlp toggles default to false and require additional flash-attn CUDA extensions in the environment.

  • The dataset.train.video_text.caption_mode values all and one_per_field apply to training only and require train.loss_type: internvideo2_vtc.

Warning

TAO Deploy consumes only the combined Open Neural Network Exchange (ONNX) export, and 32-bit floating point (FP32) is the validated TensorRT precision. FP16 is available but is not the recommended path. Refer to Deploying InternVideo2-CLIP.

Data Input for InternVideo2-CLIP#

Your video files can live anywhere on disk. A metadata JSON or JSONL file references them, and you point TAO at that file with dataset.<split>.video_text.metadata. You can also supply a list of metadata files, in which case TAO concatenates their records. This input format is metadata plus video files, not WebDataset tar shards.

Path Resolution#

Metadata files often store video paths that were recorded on a different machine. Two fields reconcile those stored paths with your local layout: path_prefix_mapping remaps a stored path prefix to a local one, and data_root supplies a root directory for relative paths.

dataset:
  train:
    type: video_text
    video_text:
      metadata: /data/vad/metadata.json
      data_root: /data/vad/videos
      path_prefix_mapping:
        /mnt/original/videos: /data/vad/videos

Metadata Schemas#

TAO detects the metadata schema automatically from the file contents, so you do not declare it. TAO recognizes three layouts.

Nested Per-Video Chunks#

This is the canonical schema. The file holds a JSON array of video records. Each record carries video_path, split, anomaly_type, video_id, dataset, total_frames, and a chunks list. Each entry in chunks carries chunk_index, is_anomaly, start_time_sec, end_time_sec, start_frame, end_frame, and the caption fields.

[
  {
    "video_path": "/data/vad/videos/store_0142.mp4",
    "split": "train",
    "anomaly_type": "Shoplifting",
    "video_id": "store_0142",
    "dataset": "vad",
    "total_frames": 4500,
    "chunks": [
      {
        "chunk_index": 0,
        "is_anomaly": false,
        "start_time_sec": 0.0,
        "end_time_sec": 8.0,
        "start_frame": 0,
        "end_frame": 240,
        "caption": "A customer walks along the aisle and looks at the shelf."
      },
      {
        "chunk_index": 1,
        "is_anomaly": true,
        "start_time_sec": 8.0,
        "end_time_sec": 16.0,
        "start_frame": 240,
        "end_frame": 480,
        "caption": "A customer conceals an item inside a jacket and leaves."
      }
    ]
  }
]

Each chunk becomes one training sample with its own temporal bounds, and the sample identifier is <dataset>/<video_id>#<chunk_index>. The caption_fields setting selects which record or chunk keys become captions. The caption_fields default is [caption], which applies whenever you omit the key. Set caption_fields to null or to an empty list to fall back to [anomaly_type] instead, which is the useful setting for this schema. A chunk that is not an anomaly and carries no caption falls back to the literal string Normal.

MSR-VTT Style#

This schema is a JSON object with a videos section and a sentences section. Use it for standard retrieval benchmarks.

{
  "videos": [
    {"video_id": "video7010", "split": "test", "video_path": "/data/msrvtt/video7010.mp4"}
  ],
  "sentences": [
    {"video_id": "video7010", "caption": "A man is riding a motorcycle down a road."},
    {"video_id": "video7010", "caption": "Someone rides a motorbike along a highway."}
  ]
}

Flat Rows#

This schema is a JSON array or a JSONL file with one row per sample. Each row carries a video path key and a caption key, plus optional category, split, and temporal-bound keys.

[
  {
    "video_path": "/data/clips/clip_00001.mp4",
    "caption": "A dog runs across a beach toward the water.",
    "category": "animal",
    "split": "train",
    "start_time_sec": 2.5,
    "end_time_sec": 10.5
  }
]

TAO checks the layouts in order. A JSON array whose first record carries a chunks list is read as nested per-video chunks. A JSON object carrying both videos and sentences is read as MSR-VTT. A file with a .jsonl extension is read as flat rows, as is any remaining JSON array. A top-level JSON object that matches none of these raises an error rather than degrading silently.

Training and Evaluation Splits#

The same metadata file can serve both training and evaluation. The split field filters the records that each dataset section reads. The shipped specifications define the shared video_text block once as a YAML anchor and reuse it for validation and inference, overriding only the fields that differ:

dataset:
  train:
    type: video_text
    video_text: &vt
      metadata: /data/vad/metadata.json
      split: train
      caption_fields: [caption]
      num_frames: ${model.num_frames}
  val:
    type: video_text
    video_text:
      <<: *vt
      split: test

Video Decoding and Frame Sampling#

TAO decodes each clip through a chain of three backends. It tries PyAV first, then the FFmpeg command-line tool, and then OpenCV. The first backend that succeeds wins, and TAO logs the winning backend once per process. A silent fallback from PyAV to the command-line tool is roughly 10 times slower, which is why TAO logs the active backend.

TAO samples num_frames indices uniformly across the temporal bounds of the clip. A clip that is shorter than num_frames pads by repeating its last frame.

During training, TAO skips a clip that fails to decode and advances the loader to the next sample. During evaluation and inference, TAO disables substitution so that the gallery and query indices stay aligned.

Inference Text Input#

The inference.query block defines what you embed or search with. Set input_texts for inline prompts, input_videos for inline video paths, and text_file for a plain-text file that holds one prompt per line. TAO merges the prompts from text_file with input_texts and removes duplicates.

An example prompt file looks like this:

A person throws a bag over a fence.
A car drives the wrong way down a one-way street.
A crowd runs away from a storefront.

Set inference.query.text_file to the path of this file in your experiment specification.

End-to-End Workflow#

The following two paths illustrate common InternVideo2-CLIP workflows. For full command details, refer to InternVideo2-CLIP Training and Deployment.

Fine-Tuning on Custom Video Data#

Use this path when you have domain-specific video-text pairs and want to fine-tune the pretrained model on your data.

  1. Prepare your metadata

    Write a metadata JSON or JSONL file in one of the recognized schemas and confirm that the video paths resolve on your machine.

  2. Obtain the text-encoder weights

    Set model.text_encoder to the MobileCLIP mobileclip_blt.pt weights.

  3. Train

    Run full fine-tuning, or enable Low-Rank Adaptation (LoRA) with the LoRA experiment specification.

  4. Evaluate

  5. Export

    Set export.encoder_type to combined and export.batch_size to -1 so that TAO Deploy can consume the result.

  6. Generate a TensorRT Engine

  7. Evaluate or Infer with the Engine

Zero-Shot Evaluation of the Pretrained Model#

Use this path to measure the pretrained model on your own data without any training.

  1. Configure the experiment specification

    Point dataset.val.video_text at your evaluation metadata and set the weight sources described in Model Weights.

  2. Evaluate

    The evaluate.checkpoint field is required in the shipped specification.