Important

NeMo 2.0 is an experimental feature and currently released in the dev container only: nvcr.io/nvidia/nemo:dev. Please refer to NeMo 2.0 overview for information on getting started.

Evaluation

For the Vision Transformer, our evaluation script processes the ImageNet 1K validation folder and computes the final validation accuracy.

To enable the evaluation stage with a ViT model, configure the configuration files:

1. In the defaults section of conf/config.yaml, update the evaluation field to point to the desired ViT configuration file. For example, if you want to use the vit/imagenet_val configuration, change the evaluation field to vit/imagenet_val.

   defaults:
     - evaluation: vit/imagenet_val
     ...
   

2. In the stages field of conf/config.yaml, make sure the evaluation stage is included. For example,

   stages:
     - evaluation
     ...
   

3. Configure imagenet_val field of conf/evaluation/vit/imagenet_val.yaml to be the ImageNet 1K validation folder.

4. Execute the launcher pipeline: python3 main.py.

Remarks:

1. To load a pretrained checkpoint for inference, set the restore_from_path field in the model section to the path of the pretrained checkpoint in .nemo format in conf/evaluation/vit/imagenet_val.yaml. By default, this field links to the .nemo format checkpoint located in the ImageNet 1K fine-tuning checkpoints folder.

2. We highly recommend users to use the same precision (i.e., trainer.precision) for evaluation as was used during training.