Important

NeMo 2.0 is an experimental feature and currently released in the dev container only: nvcr.io/nvidia/nemo:dev. Please refer to NeMo 2.0 overview for information on getting started.

Evaluation

For CLIP models, our evaluation script calculates zero-shot ImageNet 1K validation accuracy.

To enable the evaluation stage with a CLIP model, configure the configuration files:

1. In the defaults section of conf/config.yaml, update the evaluation field to point to the desired CLIP configuration file. For example, if you want to use the clip/imagenet_zeroshot configuration, change the evaluation field to clip/imagenet_zeroshot.

   defaults:
     - evaluation: clip/imagenet_zeroshot
     ...
   

2. In the stages field of conf/config.yaml, make sure the evaluation stage is included. For example,

   stages:
     - evaluation
     ...
   

3. Configure imagenet_val field of conf/evaluation/clip/imagenet_zeroshot.yaml to be the ImageNet 1K validation folder.

Remarks:

1. To load a pretrained checkpoint for inference, set the restore_from_path field in the model section to the path of the pretrained checkpoint in .nemo format in conf/evaluation/clip/imagenet_zeroshot.yaml. By default, this field links to the .nemo format checkpoint located in the CLIP trainning checkpoints folder.