Performance | NVIDIA NIM Benchmarking Documentation

This section shows the latency and throughput numbers for Llama models powered by NVIDIA NIM. Please see Using AIPerf to Benchmark for the benchmark process.

For specifications of the hardware on which these measurements were collected, see the Hardware Specifications section.

Llama-3.3-70b-instruct Results

NIM Container: Llama-3.3-70b-Instruct

Version: 1.8.0

2 x H100 80G

2 x H200 141GB

4 x H100 80GB

8 x A100 80GB

8 x H100 80GB

fp8 TP2

5000 Input Tokens / 500 Output Tokens

500 Input Tokens / 2000 Output Tokens

200 Input Tokens / 200 Output Tokens

1000 Input Tokens / 1000 Output Tokens

20000 Input Tokens / 2000 Output Tokens

Concurrency	TTFT (ms)	ITL (ms)	Throughput(Tokens/s)
1	406.77	18.58	51.67
5	546.7	21.83	218.48
25	688.69	38.75	623.4
50	834.51	59.73	814.37
100	7996.25	92.91	917.26
150	34794.22	92.86	920.9
200	61680.22	93.06	920.54
250	88605.32	93.2	920.12

Version: 1.5.0

4 x H100 80G

fp16 TP4

5000 Input Tokens / 500 Output Tokens

500 Input Tokens / 2000 Output Tokens

200 Input Tokens / 200 Output Tokens

1000 Input Tokens / 1000 Output Tokens

2500 Input Tokens / 300 Output Tokens

2000 Input Tokens / 2000 Output Tokens

Concurrency	TTFT (ms)	ITL (ms)	Throughput(Tokens/s)
1	368.11	19.94	48.44
5	446.68	23.89	199.17
25	751.74	42.12	564.5
50	1083.13	63.72	738.78
100	17754.93	87.55	777.31
150	46381.93	87.39	777.94
200	73326.6	87.64	777.02

Llama-3.1-8b-instruct Results

NIM Container: Llama-3.1-8b-Instruct

Version: 1.8.0

1 x H100 80G

1 x H200 141GB

1 x L40s 48GB

fp8 TP1

bf16 TP1

200 Input Tokens / 200 Output Tokens

500 Input Tokens / 2000 Output Tokens

1000 Input Tokens / 1000 Output Tokens

5000 Input Tokens / 500 Output Tokens

20000 Input Tokens / 2000 Output Tokens

Concurrency	TTFT (ms)	ITL (ms)	Throughput(Tokens/s)
1	10.26	4.63	214.56
5	16.81	4.58	1076.88
25	36.34	5.19	4678.88
50	67.41	5.91	8018.23
100	119.97	7.57	12214.97
150	346.53	9.81	12988.89
200	465.18	12.65	13348.4
250	605.6	16.14	12964.72

Version: 1.3.0

1 x H100 80G

fp8 TP1

fp16 TP1

200 Input Tokens / 200 Output Tokens

500 Input Tokens / 2000 Output Tokens

1000 Input Tokens / 1000 Output Tokens

5000 Input Tokens / 500 Output Tokens

20000 Input Tokens / 2000 Output Tokens

Concurrency	TTFT (ms)	ITL (ms)	Throughput(Tokens/s)
1	8.98	4.37	227.44
5	15.31	5.1	971.34
25	21.08	6.02	4100.33
50	50.68	7.19	6745.78
100	209.66	8.62	10369.56
150	398.54	12.49	10383.16
200	501.44	17.76	9884.77

Llama-3.1-70b-instruct Results

NIM Container: llama-3.1-70b-instruct

Version: 1.3.0

4 x H100 80G

fp8 TP4

200 Input Tokens / 200 Output Tokens

500 Input Tokens / 2000 Output Tokens

1000 Input Tokens / 1000 Output Tokens

5000 Input Tokens / 500 Output Tokens

20000 Input Tokens / 2000 Output Tokens

Concurrency	TTFT (ms)	ITL (ms)	Throughput(Tokens/s)
1	25.05	13.21	75.36
5	64.82	12.64	387.55
25	146.11	15.31	1564.9
50	169.61	17.42	2743.6
100	235.17	23.28	4092.34
150	511.21	27.73	4962.78
200	582.07	31.22	5855.9
250	718.98	35.25	6429.95

Hardware Specifications

NVIDIA H100

Property	Value
Motherboard Model	NVIDIA DGX H100
Server Model	NVIDIA DGX H100
Number of Nodes	1
CPU Information	Platinum 8480CL @ 3.8GHz Turbo (Sapphire Rapids) HT On
Number of CPU sockets enabled	2
Number of CPU threads enabled	224
GPU Information	H100 80GB HBM3(GH100)
Driver Information	570.124.06 (r570_00)
GPU Core Clock (MHz)	1980
GPU Boost Clock (MHz)	1980
GPU Memory Clock (MHz)	2619

NVIDIA H200

Property	Value
Motherboard Model	NVIDIA DGX H200
Server Model	NVIDIA DGX H200
Number of Nodes	1
CPU Information	Platinum 8480CL @ 3.8GHz Turbo (Sapphire Rapids) HT On
Number of CPU sockets enabled	2
Number of CPU threads enabled	224
GPU Information	H200 141GB HBM3e(GH100)
Driver Information	570.124.06 (r570_00)
GPU Core Clock (MHz)	1980
GPU Boost Clock (MHz)	1980
GPU Memory Clock (MHz)	2619

NVIDIA A100

Property	Value
Motherboard Model	NVIDIA DGX A100
Server Model	NVIDIA DGX A100
Number of Nodes	1
CPU Information	AMD EPYC 7742 @ 2.2GHz 3.4GHz Turbo (Rome) HT On
Number of CPU sockets enabled	2
Number of CPU threads enabled	256
GPU Information	A100 SXM4 80GB(GA100)
Driver Information	570.124.06 (r570_00)
GPU Core Clock (MHz)	1155
GPU Boost Clock (MHz)	1401
GPU Memory Clock (MHz)	1593

NVIDIA L40s

Property	Value
Motherboard Model	Supermicro X13DEG-OA
Server Model	Supermicro SYS-521GE-TNRT
Number of Nodes	1
CPU Information	Platinum 8570 @ 2.1GHz 4.0GHz Turbo (Emerald Rapids) HT On
Number of CPU sockets enabled	2
Number of CPU threads enabled	224
GPU Information	L40S(AD102)
Driver Information	570.124.06 (r570_00)
GPU Core Clock (MHz)	2520
GPU Boost Clock (MHz)	2520
GPU Memory Clock (MHz)	9001