InternVideo2-CLIP Training, Evaluation, Inference, and Export

The following sections cover the experiment specification parameters, the training and evaluation protocols, inference, Open Neural Network Exchange (ONNX) export, and deployment notes for InternVideo2-CLIP.

InternVideo2-CLIP aligns an InternVideo2 video tower with a MobileCLIP text tower, so every subtask on this page operates on video clips rather than single images. For an overview of the architecture, the supported weight sources, the metadata format, and the end-to-end workflow, refer to InternVideo2-CLIP Overview.

Creating an Experiment Specification File

TAO ships two reference specifications inside the container: experiment_spec.yaml for full fine-tuning and experiment_spec_lora.yaml for Low-Rank Adaptation (LoRA). Every subtask uses experiment_spec.yaml as its default configuration. Select the LoRA specification explicitly with -e.

The video_clip default_specs subtask writes a specification generated from the configuration dataclass defaults to <results_dir>/experiment.yaml, or prints it to standard output when results_dir is unset. This output is the dataclass default set, not one of the curated reference specifications described above.

Both shipped specifications leave eight fields as ???, which marks them as required: results_dir, model.text_encoder, dataset.train.video_text.metadata, train.optim.warmup_steps, evaluate.checkpoint, inference.checkpoint, export.checkpoint, and export.onnx_file. Fill each of these fields before you run any subtask.

Note

The shipped video_text block is a YAML anchor, so dataset.val.video_text.metadata and dataset.inference.video_text.metadata resolve to the same ??? value. Editing the anchor fills all three. A command-line override of dataset.train.video_text.metadata alone leaves the validation and inference copies unset, and evaluate then fails.

You can override any field on the command line using dot notation, for example train.num_gpus=4.

model

The model group selects the architecture, the input geometry, and the weight sources.

model:
  type: internvideo2-clip-l14
  image_size: 224
  num_frames: 8
  freeze_vision_encoder: true
  freeze_text_encoder: true
  init_logit_scale: 4.605170185988092
  init_logit_bias: 0.0
  internvideo2clip_hf_id: OpenGVLab/InternVideo2_distillation_models
  text_encoder: ???
  vision_encoder: null
  clip_head: null
  pretrained_ckpt: null

The weight-source fields support two forms of base-model initialization. Set internvideo2clip_hf_id together with text_encoder to assemble the model from component weights, or set pretrained_ckpt to load a complete InternVideo2-CLIP state dictionary that overrides every component source. For the full description of each source and the recommended values, refer to InternVideo2-CLIP Overview.

model.num_frames is the single source of truth for clip length. The shipped specifications interpolate ${model.num_frames} into the dataset blocks and into inference.query. TAO does not cross-validate the two values, so keep them in step whenever you change the clip length.

Note

model.canonicalize_text has no effect on the video_clip task. It applies only to the image-only CLIP families handled by the clip task.

Parameter	Type	Description	Default
type	string	CLIP model type. Only internvideo2-clip-l14 is supported: the video dataloader emits [B, T, C, H, W] and the InternVideo2-CLIP adapter is the only one that consumes 5D input. Image-only CLIP families (C-RADIO, SigLIP2, OpenCLIP) belong to the clip task.	internvideo2-clip-l14
freeze_vision_encoder	bool	When True, freeze vision encoder weights during training.	false
freeze_text_encoder	bool	When True, freeze text encoder weights during training.	false
image_size	int	Input image resolution for training transforms. Common values: 224 (InternVideo2-CLIP L14 / RADIO / OpenCLIP), 256 (SigLIP2-so400m), 384 (SigLIP2-g). Must be a multiple of the model’s patch size (typically 14 or 16).	224
init_logit_scale	float	Override for the initial logit scale (log-space). When null, TAO sets it from train.loss_type: 4.605170185988092 (the natural logarithm of 100) for internvideo2_vtc, 2.6592 for clip, and 2.3026 for siglip. Set manually only with caution, because incorrect values can destabilize training.	null
init_logit_bias	float	Override for the initial logit bias. When null, TAO sets it from train.loss_type: -10.0 for siglip and 0.0 for every other loss, including the default internvideo2_vtc. Set manually only with caution, because incorrect values can destabilize training.	null
canonicalize_text	bool	Lowercase the text and strip punctuation before tokenization. This field has no effect on the video_clip training, evaluation, or inference path, which never applies it. TAO records it in the exported *_config.yaml, and TAO Deploy does apply it when tokenizing queries, so leave it at false unless you need that deploy-side behavior.	false
pretrained_ckpt	string	Optional local .pth holding the COMPLETE InternVideo2-CLIP model state_dict. If set, it is loaded last and overrides all component weights (vision/text/clip_head). Must also be null for the all-null train-from-scratch configuration.	null
internvideo2clip_hf_id	string	HuggingFace repo id for InternVideo2-CLIP assets (provides the vision encoder and CLIP head). Resolved using the ambient HF cache (set HF_HOME). Suggested value: OpenGVLab/InternVideo2_distillation_models. Leave null together with vision_encoder, text_encoder, clip_head, and pretrained_ckpt (all null) to train from scratch / random initialization.	null
vision_encoder	string	Vision encoder source: a local file OR an HF repo id. If unset, resolved from internvideo2clip_hf_id (suggested: OpenGVLab/InternVideo2_distillation_models). Null with all other weight-source fields null results in train from scratch.	null
text_encoder	string	Text encoder (MobileCLIP) source: a local file OR an HF repo id. Required when pretrained_ckpt is not set (MobileCLIP is not in the IV2CLIP repo). Suggested: MobileCLIP mobileclip_blt.pt (local path or HF id). Leave null only for the all-null train-from-scratch configuration.	null
clip_head	string	CLIP alignment head source: a local file OR an HF repo id. If unset, resolved from internvideo2clip_hf_id (suggested: OpenGVLab/InternVideo2_distillation_models). Applied last and overwrites overlapping vision/text keys. Null with all other weight- source fields null results in train from scratch.	null
num_frames	int	Number of video frames for InternVideo2 video-text inputs.	8
use_flash_attn	bool	Enable upstream InternVideo2 FlashAttention in the vision encoder (use_flash_attn). Requires flash-attn in the env.	false
use_fused_rmsnorm	bool	Enable upstream InternVideo2 fused RMSNorm (use_fused_rmsnorm). Requires the flash-attn dropout_layer_norm CUDA extension.	false
use_fused_mlp	bool	Enable upstream InternVideo2 fused MLP (use_fused_mlp). Requires the flash-attn fused_dense_lib CUDA extension.	false

dataset

The dataset group holds three independent data sources, the evaluation metric settings, and the augmentation settings.

dataset:
  seed: 42
  pin_memory: true
  train:
    type: video_text
    batch_size: 4
    num_workers: 12
  val:
    type: video_text
    batch_size: 4
    num_workers: 12
  inference:
    type: video_text
    batch_size: 4
    num_workers: 12

dataset.inference owns the inference video corpus directly, so TAO builds the inference dataloader from that block rather than from dataset.val.

Parameter	Type	Description	Default
train	collection	Training dataset configuration.	Training Dataset Configuration
val	collection	Validation dataset configuration.	Validation Dataset Configuration
metrics	collection	Evaluation metric configuration (metric mode, excluded categories).	Metrics Configuration
inference	collection	Inference corpus configuration (gallery to embed/search).	Inference Dataset Configuration
augmentation	collection	Data augmentation configuration.	Augmentation Configuration
pin_memory	bool	Pin memory in DataLoader for faster GPU transfer.	true
seed	int	Random seed for data loading and shuffling.	42

video_text

Each data source carries a video_text block that points at the metadata files and controls caption selection, positive-pair grouping, and clip length. The shipped specifications define this block a single time on dataset.train with a YAML anchor and reuse it for dataset.val and dataset.inference, overriding only the split.

dataset:
  train:
    video_text:
      metadata: ???
      data_root: null
      split: train
      path_prefix_mapping: {}
      caption_fields: [caption]
      caption_mode: first
      idx_mode: sample_id
      idx_field: null
      anomaly_only: false
      num_frames: ${model.num_frames}

The caption_mode values all and one_per_field are train-only and require train.loss_type: internvideo2_vtc. TAO ignores them for validation and evaluation.

Parameter	Type	Description	Default
metadata	union	List of video-text JSON/JSONL metadata file paths whose records are concatenated (each file must be a JSON array / JSONL). A single file may be given as a one-element list; a bare string path is still accepted for backward compatibility.	null
data_root	string	Root directory used to resolve relative/remapped video paths.	null
split	string	Optional split filter, such as train or test.	null
path_prefix_mapping	collection	Optional mapping from original path prefixes to local prefixes.	{}
caption_fields	list	Metadata fields used as caption candidates.	['caption']
caption_mode	categorical	Caption selection strategy. first: always use the first caption (captions[0]). random: sample one caption per epoch from the flat pool. all (train-only): explode each chunk into one entry per caption so every caption is trained each epoch. one_per_field (train-only): explode one entry per caption field, sampling one caption from within that field each epoch (equal weight per field). Exploded entries share the chunk’s idx (multi-positive); all/one_per_field require train.loss_type=internvideo2_vtc and are ignored for val/eval. Valid options: first, random, all, one_per_field.	first
idx_mode	categorical	How to build InternVideo2 VTC positive ids. Valid options: sample_id, video_id, category, field.	sample_id
idx_field	string	Metadata field used when idx_mode=``field``.	null
task_type	categorical	Dataset task type. classification groups samples by their category label (for example, Vad-R1 anomaly_type) for supervised-contrastive training; retrieval uses instance/video grouping. Governs how the dataset class derives the category and default idx grouping. Valid options: retrieval, classification.	retrieval
anomaly_only	bool	For Vad-R1 chunks, keep only anomaly chunks.	false
num_frames	int	Number of frames to sample from each video clip.	8
relevance_file	string	Optional path to an explicit-relevance eval-query file (for example, a frozen domain_test.json with a queries list of {query, chunk_id, slice, relevant_clip_ids}). When set on the val source, evaluation scores those text queries against this dataset as the shared gallery, per slice, using the per-query relevant_clip_ids. When null, evaluation uses the default idx-grouped retrieval.	null

dataset.train

The training data source supplies the clips and captions that the contrastive loss consumes. batch_size and num_workers apply per GPU.

dataset:
  train:
    type: video_text
    batch_size: 4
    num_workers: 12

Parameter	Type	Description	Default
batch_size	int	Training batch size per GPU.	16
num_workers	int	Number of data loading worker processes.	8
type	categorical	Dataset type. Only video_text (video-text retrieval metadata) is supported; the legacy image-text custom and wds types were removed. Valid options: video_text.	video_text
video_text	collection	Video-text configuration (used when type=``video_text``).	Video-Text Source Configuration

dataset.val

The validation data source drives both the in-training validation loop and the evaluate subtask. Set dataset.val.video_text.metadata before you run either one.

dataset:
  val:
    type: video_text
    video_text:
      split: test
    batch_size: 4
    num_workers: 12

Parameter	Type	Description	Default
batch_size	int	Batch size per GPU.	16
num_workers	int	Number of data loading worker processes.	8
type	categorical	Validation dataset type. Only video_text is supported; the legacy image-text custom type was removed. Valid options: video_text.	video_text
video_text	collection	Video-text validation configuration.	Video-Text Source Configuration

dataset.inference

The inference data source defines the corpus that TAO embeds and, in retrieval mode, ranks against your queries.

dataset:
  inference:
    type: video_text
    video_text:
      split: test
    batch_size: 4
    num_workers: 12

Parameter	Type	Description	Default
batch_size	int	Batch size per GPU.	16
num_workers	int	Number of data loading worker processes.	8
type	categorical	Inference corpus type (video_text metadata). Valid options: video_text.	video_text
video_text	collection	Video-text corpus configuration for inference.	Video-Text Source Configuration

metrics

The metrics block selects the evaluation protocol and the categories that TAO excludes from the classification averages.

dataset:
  metrics:
    mode: retrieval
    exclude_categories: [Normal, Abnormal]

For the meaning of each protocol and the exact set of numbers that exclude_categories affects, refer to Evaluating the Model.

Parameter	Type	Description	Default
mode	categorical	Evaluation metric mode. retrieval reports N-to-N multi-relevant mAP / recall@k / hit@k; classification reports category-level MAP / MRR / macro P-R-F1 / top-K (cosmos-embed1 parity). Valid options: retrieval, classification.	retrieval
exclude_categories	list	Category names excluded from classification MAP/MRR and macro averages (case-insensitive).	['Normal', 'Abnormal']

augmentation

The augmentation block describes random resized cropping, color jitter, and grayscale conversion.

dataset:
  augmentation:
    scale: [1.0, 1.0]
    color_jitter: []
    grayscale: 0.0

Note

The InternVideo2 path does not consume dataset.augmentation. Both shipped specifications disable all three settings, and TAO applies a deterministic frame transform to training and validation alike: a bicubic resize to model.image_size, a tensor conversion, and ImageNet per-channel normalization.

Parameter	Type	Description	Default
scale	list	Scale range [min, max] for random resized crop. Set to [1.0, 1.0] to disable.	[0.4, 1.0]
color_jitter	list	Color jitter [prob, brightness, contrast, saturation, hue]. Set to [] to disable.	[0.8, 0.32, 0.32, 0.32, 0.08]
grayscale	float	Probability of grayscale conversion. Set to 0.0 to disable.	0.2

train

The train group controls the schedule, the loss formulation, the precision, and the distribution strategy.

train:
  num_epochs: 3
  checkpoint_interval: 1
  checkpoint_interval_unit: epoch
  resume_training_checkpoint_path: null
  grad_checkpointing: false
  grad_clip_norm: null
  precision: bf16
  pretrained_model_path: null
  loss_type: internvideo2_vtc
  seed: 42
  num_nodes: 1
  num_gpus: 4
  gpu_ids: [0, 1, 2, 3]
  distributed_strategy: ddp
  validation_interval: 1
  val_check_interval: null

train.loss_type selects the contrastive objective. The default, internvideo2_vtc, is the InternVideo2 video-text contrastive loss with identifier-based positives, which lets several clips that share an identifier act as positives for the same caption. The clip value selects the softmax-based formulation, and siglip selects the sigmoid-based formulation.

When you leave model.init_logit_scale and model.init_logit_bias at null, TAO sets both from train.loss_type. The internvideo2_vtc value gives a logit scale of 4.605170185988092, which is the natural logarithm of 100, and a logit bias of 0.0. The clip value gives 2.6592 and 0.0. The siglip value gives 2.3026 and -10.0. Both shipped specifications set the InternVideo2 values explicitly.

Parameter	Type	Description	Default
num_gpus	int	The number of GPUs to run the train job.	1
gpu_ids	list	List of GPU IDs to run the training on. The length of this list must be equal to the number of gpus in train.num_gpus.	[0]
num_nodes	int	Number of nodes to run the training on. If > 1, then multi-node is enabled.	1
seed	int	The seed for the initializer in PyTorch. If < 0, disable fixed seed.	1234
num_epochs	int	Number of epochs to run the training.	10
checkpoint_interval	int	The interval (in epochs) at which a checkpoint will be saved. Helps resume training.	1
checkpoint_interval_unit	categorical	The unit of the checkpoint interval. Valid options: epoch, step.	epoch
validation_interval	int	The interval (in epochs) at which a evaluation will be triggered on the validation dataset.	1
resume_training_checkpoint_path	string	Path to the checkpoint to resume training from.	null
optim	collection	Optimizer configuration with per-tower learning rates.	Optimizer Configuration
loss_type	categorical	Contrastive loss function: siglip (sigmoid), clip (softmax), or internvideo2_vtc (InternVideo2 video-text contrastive loss with idx positives). Valid options: siglip, clip, internvideo2_vtc.	internvideo2_vtc
precision	categorical	Training precision: fp16 (mixed), fp32 (full), or bf16 (bfloat16). Valid options: fp16, fp32, bf16.	fp16
grad_clip_norm	float	Maximum gradient norm for clipping. Set to None to disable.	null
grad_checkpointing	bool	Enable gradient checkpointing to reduce memory at cost of speed.	false
distributed_strategy	categorical	Distributed training strategy: ddp or fsdp (fully sharded). Valid options: ddp, fsdp.	ddp
pretrained_model_path	string	Path to pretrained model checkpoint for fine-tuning.	null
val_check_interval	int	Run validation every N training steps. If None, validates at end of epoch.	null

optim

The optim block sets the optimizer, the per-tower learning rates, and the warmup schedule. TAO builds three parameter groups: vision, text, and logit. The logit group follows text_lr.

train:
  optim:
    optimizer_type: adamw
    vision_lr: 0.0004
    text_lr: 0.0004
    betas: [0.9, 0.98]
    eps: 1e-06
    weight_decay: 0.2
    warmup_steps: ???
    scheduler: cosine

train.optim.warmup_steps is required in both shipped specifications, even though the dataclass carries a default.

Note

The upstream OpenGVLab L14 recipe uses 0.6 warmup epochs. Convert that figure to a step count for your dataset size and global batch size before you start a run.

Parameter	Type	Description	Default
optimizer_type	categorical	Optimizer type: adamw (AdamW) or lamb (LAMB). Valid options: adamw, lamb.	adamw
vision_lr	float	Learning rate for the vision encoder.	0.0001
text_lr	float	Learning rate for the text encoder.	0.0001
weight_decay	float	Weight decay (L2 regularization) coefficient.	0.0001
betas	list	Adam/LAMB beta parameters [beta1, beta2] for momentum.	[0.9, 0.95]
eps	float	Epsilon for numerical stability.	1e-06
warmup_steps	int	Number of linear warmup steps for learning rate.	100
scheduler	categorical	LR schedule after warmup: cosine (cosine decay to 0), constant (hold at base LR), linear (linear decay to 0). Valid options: cosine, constant, linear.	cosine

peft

The peft group enables Low-Rank Adaptation (LoRA), which trains a small set of injected adapters instead of the full backbone.

peft:
  enabled: true
  method: lora
  vision:
    enabled: true
    target_modules: [qkv, proj]
    num_last_blocks: 3
    rank: 8
    alpha: 16
    dropout: 0.05
  text:
    enabled: false
    target_modules: [qkv_proj, out_proj]
    num_last_blocks: 3
    rank: 8
    alpha: 16
    dropout: 0.05

Enabling LoRA freezes the entire backbone regardless of model.freeze_vision_encoder and model.freeze_text_encoder. Only the injected adapters, the logit scale, and the logit bias remain trainable, so the freeze flags carry no meaning under LoRA.

TAO matches target modules by leaf module name. The InternVideo2 vision tower has 24 transformer blocks with fused qkv and proj projections. The MobileCLIP text tower has 12 transformer blocks with fused qkv_proj and out_proj projections. Setting num_last_blocks: 0 adapts all blocks in that tower. The effective adapter scale is alpha divided by rank.

Export folds the adapters back into the base weights, so the exported ONNX graph carries zero LoRA overhead and needs no adapter-aware runtime. Refer to Exporting the Model.

Parameter	Type	Description	Default
enabled	bool	Enable PEFT mode. When False, training uses standard full fine-tuning (the standard behavior).	false
method	categorical	PEFT method. Currently only lora is supported. Valid options: lora.	lora
vision	collection	LoRA configuration for the vision encoder (InternVideo2: target_modules qkv, proj).	LoRA Tower Configuration
text	collection	LoRA configuration for the text encoder (MobileCLIP: target_modules qkv_proj, out_proj).	LoRA Tower Configuration

vision and text

peft.vision and peft.text take the same set of fields, so one table covers both towers. The only difference is the default value of target_modules, which matches the projection names of each tower.

Parameter	Type	Description	Default
enabled	bool	Enable LoRA adaptation for this encoder tower.	false
target_modules	list	Attention-projection leaf module names to wrap with LoRA within each adapted block. Defaults are for the InternVideo2-CLIP vision tower (fused qkv + proj); the MobileCLIP text tower uses qkv_proj + out_proj (set on the peft.text default). Matching is by the leaf module name.	['qkv', 'proj']
num_last_blocks	int	Number of final transformer blocks to adapt. 0 means adapt all blocks.	3
rank	int	LoRA rank (low-rank dimension).	8
alpha	int	LoRA alpha scaling factor. Effective scale = alpha / rank.	16
dropout	float	Dropout applied to LoRA input.	0.05

regularization

The regularization group adds geometry-preserving losses that constrain how far the student embeddings drift from the pretrained model during fine-tuning.

regularization:
  enabled: true
  embedding_mse_weight: 0.05
  cosine_weight: 0.05
  similarity_weight: 0.10

When you enable regularization, TAO deep-copies the pretrained model into a frozen teacher before adapter injection and adds three weighted terms to the contrastive loss: a mean-squared-error term between the student and teacher embeddings, a cosine term between the same pairs, and a term that matches the student and teacher video-text similarity matrices. The total training loss is the contrastive loss plus the weighted sum of these three terms.

We recommend enabling regularization alongside LoRA for domain adaptation that must retain zero-shot generality. The guidance is advisory: tune the three weights against your own validation set.

TAO excludes the frozen teacher from checkpoints, and export disables regularization before it restores the checkpoint, so no second backbone copy is loaded at export time.

Parameter	Type	Description	Default
enabled	bool	Enable preservation regularization. When False, only the contrastive loss is used (the standard behavior).	false
embedding_mse_weight	float	Weight for MSE loss between student and teacher embeddings.	0.05
cosine_weight	float	Weight for cosine preservation loss between student and teacher embeddings.	0.05
similarity_weight	float	Weight for similarity matrix preservation loss (MSE between student and teacher image-text similarity matrices).	0.1

evaluate

The evaluate group supplies the checkpoint and the device selection for the evaluate subtask.

evaluate:
  checkpoint: ???
  num_gpus: 4
  gpu_ids: [0, 1, 2, 3]

Note

On the PyTorch path, the evaluation data, the batch size, and the worker count come from dataset.val rather than from evaluate. The evaluate group contributes the checkpoint and the device selection. Evaluation precision follows train.precision.

Parameter	Type	Description	Default
checkpoint	string	Required path to a trained model checkpoint (.ckpt or .pth) for evaluation or inference.	Required
results_dir	string	Directory to save inference/evaluation results.	null
num_gpus	int	Number of GPUs to use.	1
gpu_ids	list	List of GPU device IDs to use.	[0]

inference

The inference group selects the inference mode, the checkpoint, the query sources, and the ranking behavior.

inference:
  checkpoint: ???
  results_dir: null
  num_gpus: 4
  gpu_ids: [0, 1, 2, 3]
  batch_size: 4
  num_workers: 12
  mode: embeddings
  overwrite_embeddings: false
  video_embeddings_file: null
  text_embeddings_file: null

inference.mode selects what the subtask produces. The embeddings value extracts embeddings for whatever sources are present, which covers the corpus, the inline text queries, and the inline video queries. The retrieval value embeds the queries and the dataset.inference corpus and writes ranked matches.

Parameter	Type	Description	Default
checkpoint	string	Required path to a trained model checkpoint (.ckpt or .pth) for evaluation or inference.	Required
results_dir	string	Directory to save inference/evaluation results.	null
num_gpus	int	Number of GPUs to use.	1
gpu_ids	list	List of GPU device IDs to use.	[0]
batch_size	int	Batch size per GPU.	16
num_workers	int	Number of data loading worker processes.	8
video_embeddings_file	string	Explicit path for the video embeddings .h5. If it exists it is reused (generation skipped); if set but missing it is generated there. Defaults to <results_dir>/video_embeddings.h5.	null
text_embeddings_file	string	Explicit path for the text embeddings .h5 (reusable query/label cache). Reused if present, generated if missing. Defaults to <results_dir>/text_embeddings.h5.	null
overwrite_embeddings	bool	Regenerate embeddings even if a cached file is present.	false
search	collection	Search / retrieval ranking over the embeddings.	Search Configuration
mode	categorical	embeddings extracts embeddings; retrieval ranks the corpus against the queries and writes top-k matches (see the search sub- config). Valid options: embeddings, retrieval.	embeddings
query	collection	Inline ad-hoc text/video queries.	Query Configuration

search

The search block controls how TAO ranks the corpus against your queries. These parameters apply when inference.mode is retrieval. TAO names the ranking metric field search_metric.

inference:
  search:
    search_metric: cosine
    top_k: 5
    normalize: true

Parameter	Type	Description	Default
search_metric	categorical	Ranking metric: cosine (cosine similarity, higher is closer) or knn (Euclidean/L2 distance, smaller is closer). Valid options: cosine, knn.	cosine
normalize	bool	L2-normalize embeddings before scoring. On normalized vectors cosine and knn give the same ranking.	true
top_k	int	Number of video clips returned per text query.	10

query

The query block carries inline queries. You can mix inline text strings, a text file with one prompt per line, and inline video file paths in the same run.

inference:
  query:
    input_texts: []
    input_videos: []
    text_file: null
    num_frames: ${model.num_frames}

TAO merges text_file into input_texts, so you can use the file for large query lists and the inline list for one-off prompts.

Parameter	Type	Description	Default
input_texts	list	Inline text queries to embed / search with.	[]
input_videos	list	Inline video file paths to embed / search with.	[]
text_file	string	Optional file with one text prompt per line (convenience for large query lists; merged with input_texts).	null
num_frames	int	Frames sampled per inline video query.	8

export

The export group controls the ONNX output path, the export mode, the input geometry, and the opset version.

export:
  checkpoint: ???
  onnx_file: ???
  encoder_type: separate
  input_height: 224
  input_width: 224
  input_channel: 3
  batch_size: 1
  opset_version: 23
  gpu_id: 0
  on_cpu: false
  verbose: false

The excerpt above reproduces the shipped specification. For deployment you must change encoder_type and batch_size. Refer to Exporting the Model for the required overrides and the exported graph interfaces.

Parameter	Type	Description	Default
checkpoint	string	Required path to a trained TAO model checkpoint (.ckpt or .pth).	Required
onnx_file	string	Output ONNX file path (without extension for separate encoder_type).	null
encoder_type	categorical	Export mode: combined (single ONNX with both encoders), separate (two ONNX files: vision and text). Valid options: combined, separate.	combined
opset_version	int	ONNX opset version for export.	23
batch_size	int	ONNX batch mode: -1 exports a symbolic dynamic batch; a positive value exports a fixed batch. The internal dynamic tracing sample does not set a runtime maximum.	-1
input_height	int	Input image height for vision encoder export.	224
input_width	int	Input image width for vision encoder export.	224
gpu_id	int	GPU device ID to use for export.	0
on_cpu	bool	When True, export on CPU instead of GPU.	false
input_channel	int	Number of channels in the input image.	3
verbose	bool	Enable verbose ONNX export logging.	false

Training the Model

The train subtask fine-tunes InternVideo2-CLIP on your video-text metadata. Fill every required field in the specification before you start a run.

Required arguments:

• -e: Path to the experiment specification file.

• results_dir: Directory in which TAO writes checkpoints, logs, and status files.

• model.text_encoder: MobileCLIP text tower source, either a local file or a HuggingFace repository identifier.

• dataset.train.video_text.metadata: Video-text metadata file or list of files for the training split.

• train.optim.warmup_steps: Number of linear warmup steps at the start of training.

Optional arguments:

• train.num_gpus: Number of GPUs per node; defaults to 1.

• train.gpu_ids: List of GPU device identifiers; defaults to [0].

• train.num_epochs: Total number of training epochs; defaults to 10.

• train.precision: Training precision, one of fp16, fp32, or bf16; defaults to fp16.

• train.optim.vision_lr and train.optim.text_lr: Per-tower learning rates; both default to 1e-4.

• dataset.train.batch_size: Batch size per GPU; defaults to 16.

• train.resume_training_checkpoint_path: Checkpoint from which to resume an interrupted run; defaults to null.

• train.pretrained_model_path: TAO checkpoint to use as the starting point for fine-tuning; defaults to null.

Note

To run multi-GPU training, set train.num_gpus and train.gpu_ids. For multi-node training, set train.num_nodes in addition. The train.distributed_strategy field accepts ddp for Distributed Data Parallel and fsdp for Fully Sharded Data Parallel. Selecting fsdp forces mixed 16-bit precision regardless of train.precision, so use it only for configurations that exceed single-node GPU memory.

Training Outputs

TAO writes checkpoints and logs under <results_dir>/train/. The checkpoint cadence follows train.checkpoint_interval and train.checkpoint_interval_unit.

TAO logs the following scalars during training: train/vision_lr, train/text_lr, train/lr, train_samples, train_loss, and train/logit_scale. When you enable regularization, TAO additionally logs train/contrastive_loss, train/embedding_mse_loss, train/cosine_loss, and train/similarity_loss.

Evaluating the Model

The evaluate subtask scores a trained checkpoint against dataset.val. dataset.metrics.mode selects one of two protocols: retrieval or classification. Setting dataset.val.video_text.task_type to classification also selects classification mode, even when dataset.metrics.mode remains retrieval. TAO evaluates at cutoffs of 1, 5, and 10 in both protocols.

Important

For classification mode, set dataset.val.video_text.task_type to classification as well as dataset.metrics.mode. The task_type field is what switches idx_mode to category and groups the samples by label. Setting dataset.metrics.mode alone leaves idx_mode at sample_id, which makes every clip its own category and produces meaningless numbers without raising an error.

Retrieval Mode

Retrieval mode runs bidirectional N-to-N video-to-text and text-to-video retrieval over the validation set. TAO forms relevance groups from the shared identifier chosen by dataset.val.video_text.idx_mode, so several clips that share an identifier count as relevant for the same caption. When the metadata carries no identifier, the ground truth falls back to a one-to-one diagonal. TAO scores similarity as cosine similarity over L2-normalized embeddings.

TAO reports the following metrics:

• mean average precision (mAP)

• R@1, R@5, and R@10

• Hit@1, Hit@5, and Hit@10

• normalized discounted cumulative gain (nDCG) at 1, 5, and 10

• median rank

• mean rank

• area under the curve (AUC)

Recall at k is the fraction of a query’s relevant items that appear in the top k results. Hit at k is one when any relevant item appears in the top k and zero otherwise. Median rank and mean rank both use the one-based rank of the first correct match, so lower values are better.

TAO reports every metric for two directions: image_to_text, which retrieves captions for a given clip, and text_to_image, which retrieves clips for a given caption.

Setting dataset.val.video_text.relevance_file switches evaluation to explicit per-query relevance. TAO then scores the listed text queries against the validation set as a shared gallery, reporting results per slice instead of using the identifier-grouped ground truth.

Classification Mode

Classification mode treats each category name as a query and every video as a document. For a given category, the videos whose label matches that category are relevant and all other videos are non-relevant. TAO builds the category prototypes by encoding the category-name strings with the text tower.

TAO reports the following metrics:

• mean average precision (mAP)

• mean reciprocal rank (MRR)

• macro precision

• macro recall

• macro F1

• top-1, top-5, and top-10 hit rate

TAO also writes a per-category breakdown that carries precision, recall, F1, support, and average precision for every category.

dataset.metrics.exclude_categories defaults to [Normal, Abnormal] and matches category names case-insensitively. It excludes those categories from mAP, from MRR, and from the macro precision, recall, and F1 averages only. It does not remove them from the top-k hit rates, from the prediction space, or from the per-category breakdown, so an excluded category can still absorb a prediction and still appears in the breakdown table.

Running Inference

The inference subtask extracts embeddings and, in retrieval mode, ranks a corpus against your queries. inference.mode selects the behavior. In embeddings mode TAO extracts embeddings for whatever sources you supply. In retrieval mode TAO embeds the queries and the dataset.inference corpus and writes ranked matches.

You supply queries through inference.query. The input_texts list holds inline text prompts, text_file points at a file with one prompt per line, and input_videos holds inline video file paths. Retrieval mode requires dataset.inference.video_text.metadata and at least one query source. Embeddings mode requires at least one of the corpus, the text queries, or the video queries.

TAO writes the following files under results_dir, unless an explicit video_embeddings_file or text_embeddings_file path overrides the location:

• video_embeddings.h5: Corpus embeddings, extracted across all GPUs and de-duplicated by sample identifier.

• text_embeddings.h5: Text query embeddings.

• query_video_embeddings.h5: Inline video query embeddings.

• retrieval_results.json: Ranked matches, written in retrieval mode only.

• similarity_stats.json: Summary statistics for the score matrix and the embedding sets, written in retrieval mode only.

Each Hierarchical Data Format version 5 (HDF5) file uses the same layout: a float32 embeddings dataset of shape N by D, and a variable-length string identifier dataset named for the embedding type, which is video_ids, texts, or image_paths. Attributes carry the item count, the embedding dimension, the embedding type, and provenance values in model_type, checkpoint, checkpoint_sha, text_encoder, and normalized.

TAO reuses an existing embeddings file only when the model type, the checkpoint fingerprint, and the text encoder all match the current run. A mismatch raises an error that directs you to set inference.overwrite_embeddings to true, which forces regeneration.

The retrieval_results.json payload carries the keys metric, normalize, top_k, num_corpus, num_queries, and queries. Each record in queries carries query, metric, query_type, and a results list. Each entry in results carries rank, video_id, and a score field, which is score for the cosine metric and distance for the k-nearest-neighbor metric.

For examples of how to consume these embeddings in downstream applications, refer to Using InternVideo2-CLIP Embeddings.

Exporting the Model

TAO exports InternVideo2-CLIP to ONNX. You can export a single combined graph or two separate graphs, depending on how you intend to serve the model.

Combined encoder (encoder_type: combined): TAO writes a single ONNX file at exactly the path you give in export.onnx_file.

Direction	Details
Inputs	image (B by T by 3 by H by W, float32, where T is model.num_frames), input_ids (B by 77, int64), attention_mask (B by 77, int64)
Outputs	image_embedding (B by 512), text_embedding (B by 512), logit_scale (scalar), logit_bias (scalar)

Use the combined graph when you encode video and text together at inference time, such as in a real-time retrieval or classification pipeline.

Separate encoders (encoder_type: separate): TAO writes two ONNX files next to each other, <base>_vision<ext> and <base>_text<ext>, where <base> and <ext> come from export.onnx_file.

Engine	Details
Vision	Input: image (B by T by 3 by H by W, float32). Outputs: image_embedding (B by 512), logit_scale (scalar), logit_bias (scalar)
Text	Inputs: input_ids (B by 77, int64), attention_mask (B by 77, int64). Outputs: text_embedding (B by 512), logit_scale (scalar), logit_bias (scalar)

Use the separate graphs when you want to precompute text embeddings offline, for example to index a fixed set of category names ahead of time and then run only the vision graph at query time.

Important

The shipped reference specifications set export.encoder_type: separate and export.batch_size: 1, but TAO Deploy consumes only the combined ONNX graph with a dynamic batch axis. Set both fields explicitly before you export a model that you intend to deploy with NVIDIA® TensorRT™:

export:
  encoder_type: combined
  batch_size: -1

Refer to Deploying InternVideo2-CLIP for the deployment workflow. A separate-mode ONNX graph still builds a TensorRT engine successfully, but the engine fails later during evaluate or inference, so the mismatch is not caught at build time.

Note

attention_mask is a required graph input whose values TAO ignores. The model always substitutes an all-ones mask internally, because the InternVideo2 text tower is causal and pooled at the end-of-text token. Passing the tokenizer mask or an all-ones array produces identical results.

Note

The exported graph L2-normalizes both embeddings, so the dot product image_embedding @ text_embedding.T is already a cosine similarity. Normalizing again is a harmless no-op.

Export writes two side artifacts next to the ONNX file:

• <base>_config.yaml: The saved configuration, which carries the trained logit scale and logit bias.

• <base>_tokenizer/: The saved tokenizer directory.

Important

TAO Deploy requires <base>_tokenizer/ for the evaluate and inference actions, and its absence is a hard error. The <base>_config.yaml file is optional, but without it TAO Deploy assumes default model settings, which can change results silently. Keep both files next to the ONNX file.

Important

Models larger than 1.9 GB write an external weights file alongside the ONNX file. Both files must travel together. If you move the ONNX file, move the external weights file with it, or the engine build cannot succeed.

Note

Export runs an adapter merge when the checkpoint contains LoRA modules, folding the adapters into the base weights. The exported graph therefore carries zero LoRA overhead and needs no adapter-aware runtime. The merge is a no-op for a standard full fine-tuning checkpoint.

export.opset_version defaults to 23, with a minimum of 11 and a maximum of 23. Opset 23 fuses the decomposed root-mean-square normalization into the ONNX RMSNormalization operator, which reduces the node count of the exported graph.

Usage Notes for ONNX and TensorRT Deployment

The following notes apply when you load the exported ONNX graph or the TensorRT engine directly.

Attention Mask Behavior

attention_mask is present as an ONNX graph input, but the model ignores its values and always substitutes an all-ones mask internally. The InternVideo2 text tower is causal and pooled at the end-of-text token, so a padding mask changes nothing. You can pass the tokenizer mask or an all-ones array of the same shape and get identical results.

Sequence Length

The MobileCLIP tokenizer uses a context length of 77 tokens, so input_ids and attention_mask must both have a second dimension of 77. Passing a different length causes a shape mismatch at runtime.

Video Input Layout and Preprocessing

The vision input is five-dimensional with the layout [B, T, C, H, W], where T is model.num_frames. Preprocess each frame exactly as training does: sample num_frames frames uniformly across the clip, resize each frame to 224 by 224 with bicubic interpolation, convert it to a tensor, and normalize it with the ImageNet per-channel mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225).

Warning

These are ImageNet statistics, not CLIP statistics. Substituting the CLIP mean and standard deviation degrades accuracy silently, because the graph still runs and still returns embeddings.

Dynamic Batch and TensorRT Shape Profiles

When you set export.batch_size: -1, only the batch axis is dynamic. The frame count, the channel count, the height, the width, and the sequence length are all static in the graph, so every shape profile must repeat them exactly. For an eight-frame clip at 224 by 224 with 77-token sequences, build the engine like this:

trtexec --onnx=video_clip_model.onnx \
  --minShapes=image:1x8x3x224x224,input_ids:1x77,attention_mask:1x77 \
  --optShapes=image:8x8x3x224x224,input_ids:8x77,attention_mask:8x77 \
  --maxShapes=image:16x8x3x224x224,input_ids:16x77,attention_mask:16x77

The attention_mask profile must match the input_ids profile exactly. The TAO Deploy gen_trt_engine action is the supported route to an engine and handles these profiles for you. Refer to Deploying InternVideo2-CLIP.

Logit Scale and Logit Bias

TAO exports logit_scale and logit_bias as scalar outputs. The logit_scale output is the exponentiated temperature, which is exp(model.init_logit_scale) and therefore approximately 100 with the default initialization. It is ready to multiply the similarity matrix directly. The <base>_config.yaml file stores the log-space value instead.

Compute match scores as logit_scale * similarity + logit_bias. Because logit_bias is a single scalar, it cancels under a softmax over classes and cannot change the ranking or the predicted class. It matters for sigmoid-style pairwise scoring, where you score each video-text pair on its own. When you take a softmax, subtract the maximum logit first: logit_scale is approximately 100 and the graph returns float32, so the unshifted exponential overflows. When you use the separate graphs, both scalars are available from either graph, and you need only one copy.