GPT Results

Training accuracy, NVIDIA DGX SuperPOD™:

• 8 × 8 × A100 80GB for 126M GPT Model

• 16 × 8 × A100 80GB for 5B GPT Model

NVIDIA evaluated the 126M parameter and 5B parameter models on eight different language tasks. The results are shown in the table below. All of the tasks are provided as part of the evaluation harness, so you can evaluate any .nemo checkpoint file on all of them.

Training the 5B GPT model to convergence takes 6.5 days, and the loss curve is shown in the figure below.

5B GPT Training Loss

The table below shows the converged training loss, the throughput, and the total time to train for the 5B GPT model, using a given number of GPUs and a given Global Batch Size (GBS).

Training performance:

• NVIDIA DGX SuperPOD (16 × 8 × A100 80GB for 5B GPT model)

• NVIDIA DGX SuperPODs (128 × 8 × A100 80GB for 175B GPT model)

NVIDIA measured the throughput of training 5B and 175B parameter GPT models on different numbers of DGX nodes and achieved near-linear scaling. For example, scaling from 1 node to 32 nodes with a 5B model yielded a 28.73x speed-up. Scaling from 8 nodes to 128 nodes (16 × more) with a 175B model yielded a 14.62 × speed-up. The tables and charts below show the performance results.

5B GPT NeMo Framework Throughput

175B GPT NeMo Framework Throughput

Inference performance was measured for - (1- 8 × A100 80GB SXM4) - (1- 8 × H100 80GB HBM3)

Configuration 1: Chatbot Conversation use case

• batch size: 1 - 8

  • input tokens length: 128

  • output tokens length: 20

Average Latency, Average Throughput, and Model Size

Configuration 2: Translation / Style Transfer use case

• batch size: 1 - 8

  • input tokens length: 200

  • output tokens length: 200

Average Latency, Average Throughput, and Model Size