Evaluation

For the Vision Transformer, our evaluation script processes the ImageNet 1K validation folder and computes the final validation accuracy.

To enable the evaluation stage with a ViT model, configure the configuration files:

1. In the defaults section of conf/config.yaml, update the evaluation field to point to the desired ViT configuration file. For example, if you want to use the vit/imagenet_val configuration, change the evaluation field to vit/imagenet_val.

   Copy

   Copied!

               
   
               
   defaults:
     - evaluation: vit/imagenet_val
     ...
           

2. In the stages field of conf/config.yaml, make sure the evaluation stage is included. For example,

   Copy

   Copied!

               
   
               
   stages:
     - evaluation
     ...
           

3. Configure imagenet_val field of conf/evaluation/vit/imagenet_val.yaml to be the ImageNet 1K validation folder.

4. Execute launcher pipeline: python3 main.py

Remarks:

1. To load a pretrained checkpoint for inference, set the restore_from_path field in the model section to the path of the pretrained checkpoint in .nemo format in conf/evaluation/vit/imagenet_val.yaml. By default, this field links to the .nemo format checkpoint located in the ImageNet 1K fine-tuning checkpoints folder.

2. We highly recommend users to use the same precision (i.e., trainer.precision) for evaluation as was used during training.