Using InternVideo2-CLIP Embeddings#

A trained or pretrained InternVideo2-CLIP model produces video embeddings, text embeddings, a logit_scale value, and a logit_bias value. The video and text embeddings share a single 512-dimensional space, so you can compare any clip against any caption with a dot product.

The examples below load a combined Open Neural Network Exchange (ONNX) model with ONNX Runtime and run the graph directly. The same tensor names and shapes apply to an NVIDIA® TensorRT™ engine that TAO Deploy builds from that ONNX file, so you can swap the runtime without changing the surrounding code. Refer to Deploying the Model for the engine build. If you prefer not to run the graph yourself, the video_clip inference subtask writes the same embeddings to Hierarchical Data Format version 5 (HDF5) files. Refer to Training and Inference for that path.

Preprocessing Video Clips#

Important

The export subtask defaults to export.opset_version: 23, which current ONNX Runtime releases cannot load. To run the ONNX Runtime examples on this page, re-export with export.opset_version=20, or build a TensorRT engine and use the engine path instead. Refer to InternVideo2-CLIP with TAO Deploy for the engine route.

Every snippet on this page starts from the same video tensor. The model consumes a float32 array of shape (B, T, 3, 224, 224), where T is model.num_frames and defaults to eight. To build it, sample num_frames frame indices uniformly across the clip, resize each frame to 224 by 224 with bicubic interpolation, scale the pixels into the range 0 to 1, and normalize with the ImageNet per-channel mean and standard deviation. Clips that hold fewer frames than num_frames repeat their last frame until the tensor is full.

import numpy as np
from PIL import Image

IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
IMAGENET_STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)

def preprocess_clip(video_path, num_frames=8, size=224):
    """Return one clip as a float32 array of shape (1, T, 3, 224, 224)."""
    # Decode with your preferred reader (PyAV, OpenCV, or decord).
    frames = decode_rgb_frames(video_path)  # list of RGB PIL images
    total = len(frames)

    if total >= num_frames:
        indices = np.linspace(0, total - 1, num_frames).astype(np.int64)
    else:
        # Short clips pad by repeating the last frame.
        indices = np.full(num_frames, total - 1, dtype=np.int64)
        indices[:total] = np.arange(total)

    sampled = []
    for index in indices:
        frame = frames[int(index)].resize((size, size), Image.BICUBIC)
        array = np.asarray(frame, dtype=np.float32) / 255.0   # (224, 224, 3) in [0, 1]
        array = (array - IMAGENET_MEAN) / IMAGENET_STD
        sampled.append(array.transpose(2, 0, 1))              # (3, 224, 224)

    clip = np.stack(sampled).astype(np.float32)               # (T, 3, 224, 224)
    return clip[None, ...]                                    # (1, T, 3, 224, 224)

Important

The bicubic interpolation and the ImageNet normalization statistics form a silent accuracy contract with training. InternVideo2-CLIP uses ImageNet statistics, not CLIP statistics, and applies the same deterministic transform during training and validation. Substituting a different resize filter or a different mean and standard deviation degrades retrieval quality without raising any error.

Text-to-Video Retrieval#

To search a video corpus with natural language, precompute an embedding for every clip in the gallery, embed the text query, and rank the gallery by cosine similarity against the query. The graph already L2-normalizes both embeddings, so the explicit l2_normalize calls below are defensive no-ops.

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

session = ort.InferenceSession("video_clip_model.onnx")
tokenizer = AutoTokenizer.from_pretrained("video_clip_model_tokenizer/")

NUM_FRAMES = 8

def l2_normalize(embeddings):
    return embeddings / np.linalg.norm(embeddings, axis=-1, keepdims=True)

# The combined graph shares one dynamic batch axis across the video and the text
# inputs, so feed a matching placeholder for the tower you do not need.
def text_placeholder(batch):
    ids = np.zeros((batch, 77), dtype=np.int64)
    return {"input_ids": ids, "attention_mask": np.ones_like(ids)}

# Embed the gallery clip by clip.
gallery_embeddings = []
for path in video_paths:
    inputs = {"image": preprocess_clip(path, num_frames=NUM_FRAMES)}
    inputs.update(text_placeholder(1))
    gallery_embeddings.append(session.run(["image_embedding"], inputs)[0])  # (1, 512)
gallery_embeddings = l2_normalize(np.concatenate(gallery_embeddings))       # (N, 512)

# Embed the text query.
query = "a delivery van blocking the loading bay"
tokens = tokenizer(
    [query], padding="max_length", truncation=True,
    max_length=77, return_tensors="np"
)
text_emb = session.run(["text_embedding"], {
    "image": np.zeros((1, NUM_FRAMES, 3, 224, 224), dtype=np.float32),
    "input_ids": tokens["input_ids"].astype(np.int64),
    # attention_mask is a required graph input, but the model ignores its values
    # and always substitutes an all-ones mask internally.
    "attention_mask": np.ones_like(tokens["input_ids"], dtype=np.int64),
})[0]                                          # (1, 512)
text_emb = l2_normalize(text_emb)

scores = (text_emb @ gallery_embeddings.T)[0]  # (N,)
top_k = scores.argsort()[::-1][:5]
for rank, index in enumerate(top_k, start=1):
    print(f"{rank}. {video_paths[index]}  {scores[index]:.4f}")

TAO runs the same protocol for you when you set inference.mode to retrieval. The inference.search.search_metric field selects cosine similarity or knn Euclidean distance, inference.search.normalize controls the L2 normalization step, and inference.search.top_k sets the number of clips returned per query. The subtask writes the ranked matches to retrieval_results.json under the results directory. Refer to Training and Inference for the full configuration.

Zero-Shot Video Classification#

To classify a clip without any task-specific training data, turn each class name into a text prompt, embed the clip and the prompts into the shared space, and take the highest-scoring prompt. Score each prompt as logit_scale * similarity + logit_bias. Because logit_bias is a single scalar, it shifts every class score equally and cancels under the softmax, so it cannot change the ranking or the predicted class. It matters only for sigmoid-style pairwise scoring, where you score each video-text pair on its own. Subtract the maximum before exponentiating: logit_scale is approximately 100, and the graph returns float32, so the unshifted exponential overflows.

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

session = ort.InferenceSession("video_clip_model.onnx")
tokenizer = AutoTokenizer.from_pretrained("video_clip_model_tokenizer/")

class_names = ["a person falling down", "a road accident", "normal traffic flow"]
prompts = [f"a video of {name}" for name in class_names]
tokens = tokenizer(
    prompts, padding="max_length", truncation=True,
    max_length=77, return_tensors="np"
)

clip = preprocess_clip("street_camera.mp4", num_frames=8)  # (1, 8, 3, 224, 224)
# One dynamic batch axis is shared, so repeat the clip to match the prompt count.
video = np.repeat(clip, len(prompts), axis=0)              # (C, 8, 3, 224, 224)

# attention_mask is a required graph input, but the model ignores its values.
outputs = session.run(None, {
    "image": video,
    "input_ids": tokens["input_ids"].astype(np.int64),
    "attention_mask": np.ones_like(tokens["input_ids"], dtype=np.int64),
})
image_emb, text_emb, logit_scale, logit_bias = outputs

# The graph already L2-normalizes its outputs; this is a defensive no-op.
video_emb = image_emb[:1] / np.linalg.norm(image_emb[:1], axis=-1, keepdims=True)
text_emb = text_emb / np.linalg.norm(text_emb, axis=-1, keepdims=True)

# logit_bias is a scalar, so it cancels under the softmax and cannot change the
# prediction. Keep it for the general form and for sigmoid-style scoring.
logits = float(logit_scale) * (video_emb @ text_emb.T) + float(logit_bias)  # (1, C)

# Shift by the maximum before exponentiating. logit_scale is around 100 and these
# are float32, so np.exp on the raw logits overflows to inf and yields nan.
shifted = logits[0] - logits[0].max()
probabilities = np.exp(shifted) / np.exp(shifted).sum()
print(f"Predicted: {class_names[int(probabilities.argmax())]}")

The evaluate subtask automates this protocol when dataset.metrics.mode and dataset.val.video_text.task_type are both classification. Each category name becomes a query, every clip becomes a document, and TAO reports mean average precision (mAP), mean reciprocal rank (MRR), macro precision, macro recall, macro F1, and the top-1, top-5, and top-10 hit rates. Refer to Training and Inference for the metric configuration.

Video Anomaly Detection#

Video anomaly detection is the use case the canonical metadata schema is built around. That schema splits each video into fixed temporal chunks. Every chunk carries an is_anomaly flag and an anomaly_type label, and each chunk becomes one sample with its own start and end bounds. Because the bounds travel with the sample, the dataloader decodes only the frames that belong to the chunk.

The chunk category is the anomaly_type value when the chunk is flagged as anomalous and Normal otherwise. Two configuration fields shape how you use these labels:

  • dataset.<split>.video_text.anomaly_only keeps only the anomaly chunks. Set it when you fine-tune on anomaly captions alone.

  • dataset.metrics.exclude_categories defaults to [Normal, Abnormal]. Classification evaluation drops those two umbrella labels from the mean average precision, the mean reciprocal rank, and the macro averages so that they do not distort the per-class numbers.

The following excerpt sets both fields:

dataset:
  train:
    video_text:
      anomaly_only: true
  metrics:
    mode: classification
    exclude_categories: ["Normal", "Abnormal"]

Operationally, build a prompt set that describes the anomalies you care about, embed every chunk of an incoming stream, and score each chunk against that prompt set with the zero-shot classification snippet above. A chunk whose top prompt score crosses the threshold you choose becomes a candidate detection for review. Calibrate that threshold on your own labeled chunks, because the right value depends on the camera, the scene, and the cost of a false alarm.

Direct ONNX Runtime and TensorRT Inference#

The following snippet shows the complete input and output structure of the combined graph in a single call. Use it as the reference when you integrate the model into a custom pipeline.

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

session = ort.InferenceSession("video_clip_model.onnx")
tokenizer = AutoTokenizer.from_pretrained("video_clip_model_tokenizer/")

tokens = tokenizer(
    ["a forklift entering the aisle"],
    padding="max_length", truncation=True,
    max_length=77, return_tensors="np"
)
video = preprocess_clip("clip.mp4", num_frames=8)   # (1, 8, 3, 224, 224), float32

# attention_mask is a required graph input, but the model ignores its values and
# always substitutes an all-ones mask internally, so an all-ones array is equivalent.
outputs = session.run(None, {
    "image": video,
    "input_ids": tokens["input_ids"].astype(np.int64),        # (1, 77)
    "attention_mask": np.ones((1, 77), dtype=np.int64),       # (1, 77)
})
image_emb, text_emb, logit_scale, logit_bias = outputs
# image_emb: (1, 512), text_emb: (1, 512), logit_scale and logit_bias: scalars.

A TensorRT engine that the TAO Deploy gen_trt_engine action builds from this ONNX file exposes the same three inputs and the same four output names in the same order. Only the batch axis is dynamic, and 32-bit floating point (FP32) is the validated precision. Refer to Deploying the Model for the engine build and the batch-size settings.

Choosing Between InternVideo2-CLIP and Cosmos-Embed1#

InternVideo2-CLIP and Cosmos-Embed1 are both dual-encoder video-text embedding models in the TAO embedding section, and both produce video and text embeddings in a shared space for retrieval and zero-shot classification. InternVideo2-CLIP pairs an InternVideo2 L14 vision tower with a MobileCLIP text tower. It can export both encoders as a single combined ONNX graph with export.encoder_type: combined. It also supports Low-Rank Adaptation (LoRA) and frozen-teacher preservation regularization for parameter-efficient domain adaptation. Choose between the two models by running your own evaluation on your own footage, because the right fit depends on your domain, your label set, and your deployment constraints.