Performance

NVIDIA DGX SuperPOD (4 x 8 x A100 80GB for ViT B/16 Model)

We pretrained a ViT B/16 model on the ImageNet 1K dataset and fine-tuned it on the same dataset at a higher resolution, following the recipe outlined in the ViT paper. As a result, we achieved a Top-1 accuracy of 79.47%, which is 1.56% higher than the reported accuracy of 77.91% in the paper. Below are the highlights of the training and fine-tuning recipe we used:

• Model: ViT B/16

• Dataset: ImageNet 1K

• Pretraining:

  • Epochs: 300

  • Batch Size: 4096

  • Training Resolution: 224

  • Optimizer: Adam (0.9, 0.999)

  • Base Learning Rate: 3.00E-03

  • Learning Rate Decay: Cosine

  • Weight Decay: 0.3

  • Dropout: 0.1

• Fine-tuning:

  • Steps: 20,000

  • Batch Size: 512

  • Fine-tuning Resolution: 512

  • Optimizer: SGD (0.9)

  • Base Learning Rate: 0.003 - 0.06

  • Learning Rate Decay: Cosine

  • Weight Decay: 0

We measured the throughput of training Vision Transformer models on different numbers of DGX A100 nodes and DGX H100 nodes, and we achieved near-linear scaling on both platforms.

We are comparing the out-of-box performance on DGX H100 machines with the same configuration from DGX A100 machines. This comparison is an apple-to-apple assessment, ensuring that we evaluate the relative performance of the two machine types under equivalent conditions and configurations.

The tables and charts below show the performance results.

• NVIDIA DGX SuperPODs (16 x 8 x A100 80GB for ViT g/14 model)

• NVIDIA DGX SuperPODs (16 x 8 x H100 80GB for ViT g/14 model)

• DGX A100 vs. DGX H100: A Comparative Analysis of Vision Transformer Training

Latency times are taken as starting with an image on CPU and stopped on output. For framework we use the Torch Automated Mixed Precision (AMP) for FP16 computation. For TRT, we export the various models with the FP16 acceleration. We use the optimized TRT engine setup present in the deployment directory to get the numbers in the same environment as the framework.

GPU: NVIDIA DGX A100 (1x A100 80 GB) Batch Size: Number of Images in a Batch