Important

You are viewing the NeMo 2.0 documentation. This release introduces significant changes to the API and a new library, NeMo Run. We are currently porting all features from NeMo 1.0 to 2.0. For documentation on previous versions or features not yet available in 2.0, please refer to the NeMo 24.07 documentation.

Evaluation#

For CLIP models, our evaluation script calculates zero-shot ImageNet 1K validation accuracy.

To enable the evaluation stage with a CLIP model, configure the configuration files:

In the defaults section of conf/config.yaml, update the evaluation field to point to the desired CLIP configuration file. For example, if you want to use the clip/imagenet_zeroshot configuration, change the evaluation field to clip/imagenet_zeroshot.
```
defaults:
  - evaluation: clip/imagenet_zeroshot
  ...
```
In the stages field of conf/config.yaml, make sure the evaluation stage is included. For example,
```
stages:
  - evaluation
  ...
```
Configure imagenet_val field of conf/evaluation/clip/imagenet_zeroshot.yaml to be the ImageNet 1K validation folder.

Remarks:

To load a pretrained checkpoint for inference, set the restore_from_path field in the model section to the path of the pretrained checkpoint in .nemo format in conf/evaluation/clip/imagenet_zeroshot.yaml. By default, this field links to the .nemo format checkpoint located in the CLIP trainning checkpoints folder.