Evaluation

User Guide (Latest Version)

For CLIP models, our evaluation script calculates zero-shot ImageNet 1K validation accuracy.

To enable the evaluation stage with a CLIP model, configure the configuration files:

  1. In the defaults section of conf/config.yaml, update the evaluation field to point to the desired CLIP configuration file. For example, if you want to use the clip/imagenet_zeroshot configuration, change the evaluation field to clip/imagenet_zeroshot.

    Copy
    Copied!
                

    defaults: - evaluation: clip/imagenet_zeroshot ...

  2. In the stages field of conf/config.yaml, make sure the evaluation stage is included. For example,

    Copy
    Copied!
                

    stages: - evaluation ...

  3. Configure imagenet_val field of conf/evaluation/clip/imagenet_zeroshot.yaml to be the ImageNet 1K validation folder.

Remarks:

  1. To load a pretrained checkpoint for inference, set the restore_from_path field in the model section to the path of the pretrained checkpoint in .nemo format in conf/evaluation/clip/imagenet_zeroshot.yaml. By default, this field links to the .nemo format checkpoint located in the CLIP trainning checkpoints folder.

Previous Training with Predefined Configurations
Next Framework Inference
© | | | | | | |. Last updated on Jun 14, 2024.