Evaluation

For the Vision Transformer, our evaluation script processes the ImageNet 1K validation folder and computes the final validation accuracy.

To enable the evaluation stage with a ViT model, configure the configuration files:

  1. In the defaults section of conf/config.yaml, update the evaluation field to point to the desired ViT configuration file. For example, if you want to use the vit/imagenet_val configuration, change the evaluation field to vit/imagenet_val.

    Copy
    Copied!
                

    defaults: - evaluation: vit/imagenet_val ...

  2. In the stages field of conf/config.yaml, make sure the evaluation stage is included. For example,

    Copy
    Copied!
                

    stages: - evaluation ...

  3. Configure imagenet_val field of conf/evaluation/vit/imagenet_val.yaml to be the ImageNet 1K validation folder.

  4. Execute launcher pipeline: python3 main.py

Remarks:

  1. To load a pretrained checkpoint for inference, set the restore_from_path field in the model section to the path of the pretrained checkpoint in .nemo format in conf/evaluation/vit/imagenet_val.yaml. By default, this field links to the .nemo format checkpoint located in the ImageNet 1K fine-tuning checkpoints folder.

  2. We highly recommend users to use the same precision (i.e., trainer.precision) for evaluation as was used during training.

Previous Fine-tuning
Next Framework Inference
© Copyright 2023-2024, NVIDIA. Last updated on May 17, 2024.