Performance for NVIDIA NeMo Retriever Reranking NIM

To benchmark the performance of NVIDIA NeMo Retriever Reranking NIM, you can use the AIPerf tool.

To run a performance benchmark with representative input data, create a JSONL file where each line represents a ranking request with a query and one or more passages.

Example

Create a file named rankings.jsonl that contains the following content.

{"texts": [{"name":"query","contents":["What was the first car ever driven?"]},{"name":"passages","contents":["The first car was driven in the late 19th century.","A steam-powered vehicle was demonstrated before gasoline automobiles became common."]}]}
{"texts": [{"name":"query","contents":["Who served as the 5th President of the United States of America?"]},{"name":"passages","contents":["James Monroe served as the fifth president of the United States.","George Washington was the first president of the United States."]}]}
{"texts": [{"name":"query","contents":["Is the Sydney Opera House located in Australia?"]},{"name":"passages","contents":["The Sydney Opera House is located in Sydney, Australia.","The Eiffel Tower is located in Paris, France."]}]}
{"texts": [{"name":"query","contents":["In what state did they film Shrek 2?"]},{"name":"passages","contents":["Principal voice recording for Shrek 2 took place in California.","DreamWorks Animation produced Shrek 2."]}]}

Install the verified AIPerf version in a Python environment that can reach the NIM endpoint.

pip install aiperf==0.9.0

Run the following command to benchmark the ranking endpoint by using AIPerf.

aiperf profile \
    --model nvidia/llama-nemotron-rerank-vl-1b-v2 \
    --endpoint-type nim_rankings \
    --url localhost:8000 \
    --input-file rankings.jsonl \
    --custom-dataset-type single_turn \
    --extra-inputs truncate:END \
    --request-count 20 \
    --concurrency 5

You can see the full set of command line options in the AIPerf command line options documentation.

Benchmarks

All latency measurements are reported in milliseconds.

The previous AIPerf example benchmarks ranking requests using custom queries and passages. The benchmark tables use a separate synthetic AIPerf benchmark matrix for consistent SKU comparisons. The benchmark tables report measurements for the supported GPU SKUs and precisions that have benchmark data. For text rows, batch size is text passages per request and throughput is text passages per second. For image rows, batch size is image passages per request, image size is the generated JPEG resolution, and throughput is image passages per second.

Llama Nemotron Rerank VL 1B v2

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	512	10	1	28.6	28.6	28.7	28.7	344.1
passage	512	10	3	72.2	55.0	91.7	92.2	409.5
passage	512	10	5	122.5	131.1	149.5	150.2	399.1
passage	512	20	1	57.6	57.5	58.6	58.8	343.7
passage	512	20	3	149.1	115.7	187.5	187.7	397.9
passage	512	20	5	252.8	239.8	350.2	358.0	390.8
passage	512	40	1	112.5	112.3	114.0	114.7	352.9
passage	512	40	3	325.3	321.2	353.4	360.3	365.2
passage	512	40	5	588.1	599.3	635.0	651.0	334.4
image passage (336x336 JPEG)		10	1	9.9	9.9	9.9	10.0	953.0
image passage (336x336 JPEG)		10	3	18.8	13.9	24.2	24.3	1539.5
image passage (336x336 JPEG)		10	5	29.8	28.4	36.2	36.4	1626.3
image passage (560x560 JPEG)		10	1	9.9	9.9	10.0	10.0	937.1
image passage (560x560 JPEG)		10	3	18.4	15.7	22.8	23.2	1554.4
image passage (560x560 JPEG)		10	5	29.3	27.4	35.8	35.9	1637.2
image passage (1024x1024 JPEG)		10	1	29.3	29.3	29.6	29.7	326.9
image passage (1024x1024 JPEG)		10	3	76.1	54.0	100.6	100.8	384.3
image passage (1024x1024 JPEG)		10	5	125.7	114.4	153.3	153.5	387.6

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	512	10	1	40.2	40.2	40.6	40.7	243.2
passage	512	10	3	103.4	85.9	132.2	133.0	285.8
passage	512	10	5	179.7	184.3	219.6	221.1	273.1
passage	512	20	1	83.1	83.0	84.1	84.4	237.8
passage	512	20	3	219.0	171.4	273.2	274.2	270.9
passage	512	20	5	370.2	345.0	522.9	525.3	266.1
passage	512	40	1	161.6	161.4	163.4	164.0	246.0
passage	512	40	3	430.9	433.3	436.0	437.9	275.9
passage	512	40	5	722.8	695.7	896.3	952.4	274.6
image passage (336x336 JPEG)		10	1	11.2	11.1	11.4	11.5	836.1
image passage (336x336 JPEG)		10	3	20.2	15.8	25.7	25.8	1413.0
image passage (336x336 JPEG)		10	5	32.3	31.0	39.3	39.4	1485.1
image passage (560x560 JPEG)		10	1	11.3	11.3	11.6	11.6	807.4
image passage (560x560 JPEG)		10	3	19.9	16.5	24.7	24.9	1415.9
image passage (560x560 JPEG)		10	5	32.1	30.1	39.2	39.4	1487.7
image passage (1024x1024 JPEG)		10	1	34.6	34.3	34.9	36.3	275.0
image passage (1024x1024 JPEG)		10	3	81.7	69.0	105.9	106.4	356.4
image passage (1024x1024 JPEG)		10	5	135.3	123.6	164.8	165.1	359.4

NVIDIA-B200

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	512	10	1	19.8	19.7	19.9	20.0	470.0
passage	512	10	3	37.5	36.2	39.6	39.7	764.3
passage	512	10	5	61.8	62.2	63.3	63.5	780.7
passage	512	20	1	33.0	33.0	33.4	33.5	578.7
passage	512	20	3	67.9	68.1	70.2	70.8	858.0
passage	512	20	5	113.0	105.6	159.2	160.9	861.9
passage	512	40	1	64.1	64.1	64.7	65.1	608.6
passage	512	40	3	131.9	132.0	135.0	137.1	891.8
passage	512	40	5	221.9	217.7	274.2	281.0	885.8
image passage (336x336 JPEG)		10	1	6.9	6.9	7.0	7.0	1192.8
image passage (336x336 JPEG)		10	3	8.7	8.6	9.0	9.2	2910.8
image passage (336x336 JPEG)		10	5	13.8	13.5	16.7	17.3	3208.3
image passage (560x560 JPEG)		10	1	7.0	7.0	7.1	7.1	1146.3
image passage (560x560 JPEG)		10	3	8.6	8.5	9.0	9.1	2887.6
image passage (560x560 JPEG)		10	5	13.9	13.3	16.6	18.5	3152.6
image passage (1024x1024 JPEG)		10	1	16.8	16.8	17.0	17.2	518.9
image passage (1024x1024 JPEG)		10	3	28.4	25.4	34.9	35.3	967.0
image passage (1024x1024 JPEG)		10	5	48.0	48.2	58.4	59.1	981.0

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	512	10	1	30.3	30.4	30.7	30.8	314.7
passage	512	10	3	68.7	59.8	79.7	80.0	424.7
passage	512	10	5	101.3	102.2	105.1	105.4	481.4
passage	512	20	1	41.0	41.0	41.4	41.5	470.3
passage	512	20	3	93.4	87.8	104.3	105.9	627.3
passage	512	20	5	157.3	146.6	221.7	223.0	621.6
passage	512	40	1	83.1	83.0	84.3	84.8	472.1
passage	512	40	3	192.4	193.1	195.6	197.4	614.0
passage	512	40	5	313.8	321.3	330.2	378.4	626.7
image passage (336x336 JPEG)		10	1	7.3	7.3	7.4	7.5	1113.2
image passage (336x336 JPEG)		10	3	10.3	10.2	10.6	10.8	2514.3
image passage (336x336 JPEG)		10	5	16.1	15.8	19.9	20.1	2789.1
image passage (560x560 JPEG)		10	1	7.4	7.4	7.7	7.8	1113.6
image passage (560x560 JPEG)		10	3	10.0	9.9	10.4	10.5	2527.5
image passage (560x560 JPEG)		10	5	15.9	15.6	19.6	19.8	2778.9
image passage (1024x1024 JPEG)		10	1	19.4	19.7	20.9	21.0	458.2
image passage (1024x1024 JPEG)		10	3	35.6	32.8	39.7	40.0	784.3
image passage (1024x1024 JPEG)		10	5	61.2	57.4	76.3	76.8	771.6

NVIDIA-H100-80GB-HBM3

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	512	10	1	26.6	26.5	26.8	26.9	367.5
passage	512	10	3	59.1	51.8	69.8	70.2	499.4
passage	512	10	5	99.0	100.2	104.1	107.1	495.3
passage	512	20	1	49.6	49.7	50.1	50.2	397.9
passage	512	20	3	112.3	95.7	132.6	133.2	527.7
passage	512	20	5	191.3	179.6	268.9	271.8	514.3
passage	512	40	1	96.0	96.0	97.0	97.3	413.1
passage	512	40	3	218.2	219.0	221.3	222.5	544.1
passage	512	40	5	362.0	366.8	371.0	389.0	543.7
image passage (336x336 JPEG)		10	1	7.7	7.7	7.8	7.8	1171.4
image passage (336x336 JPEG)		10	3	13.8	12.2	17.1	17.2	2038.7
image passage (336x336 JPEG)		10	5	21.1	20.6	25.4	25.5	2250.7
image passage (560x560 JPEG)		10	1	7.9	7.9	8.0	8.0	1125.9
image passage (560x560 JPEG)		10	3	13.2	13.2	14.2	14.3	2090.3
image passage (560x560 JPEG)		10	5	20.6	20.2	25.2	25.3	2287.2
image passage (1024x1024 JPEG)		10	1	24.3	24.2	24.5	24.7	383.3
image passage (1024x1024 JPEG)		10	3	51.5	38.7	68.4	68.8	559.3
image passage (1024x1024 JPEG)		10	5	83.9	83.9	102.1	103.3	576.7

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	512	10	1	36.5	36.5	36.9	37.1	269.4
passage	512	10	3	91.7	72.9	114.1	114.7	322.9
passage	512	10	5	140.9	151.3	169.9	170.8	347.2
passage	512	20	1	63.2	63.2	63.8	64.1	313.2
passage	512	20	3	156.6	127.7	193.0	193.5	378.8
passage	512	20	5	256.4	255.1	363.9	367.3	384.4
passage	512	40	1	123.8	123.7	125.4	125.6	321.1
passage	512	40	3	308.4	309.8	314.4	317.1	385.5
passage	512	40	5	507.2	495.3	591.4	642.4	390.9
image passage (336x336 JPEG)		10	1	8.3	8.3	8.4	8.4	1096.1
image passage (336x336 JPEG)		10	3	15.3	15.2	15.8	16.0	1846.6
image passage (336x336 JPEG)		10	5	23.8	23.5	29.0	29.2	2001.3
image passage (560x560 JPEG)		10	1	8.5	8.5	8.5	8.5	1053.8
image passage (560x560 JPEG)		10	3	15.0	14.9	15.6	15.7	1857.9
image passage (560x560 JPEG)		10	5	23.4	23.2	28.7	28.9	2021.1
image passage (1024x1024 JPEG)		10	1	26.2	26.1	26.4	26.5	358.2
image passage (1024x1024 JPEG)		10	3	58.9	58.3	64.7	68.0	490.8
image passage (1024x1024 JPEG)		10	5	97.9	103.4	120.1	121.2	495.5

NVIDIA-H100-NVL

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	512	10	1	33.2	33.1	33.9	34.2	292.7
passage	512	10	3	83.7	67.9	102.1	102.4	351.5
passage	512	10	5	139.6	144.6	166.5	167.4	349.8
passage	512	20	1	64.0	63.8	64.9	65.0	306.5
passage	512	20	3	161.2	131.2	197.0	197.4	366.9
passage	512	20	5	272.9	255.2	386.2	387.9	359.8
passage	512	40	1	124.9	124.8	126.3	126.7	316.5
passage	512	40	3	314.6	315.9	321.6	326.4	377.2
passage	512	40	5	522.5	515.9	622.2	657.8	378.8
image passage (336x336 JPEG)		10	1	8.4	8.4	8.5	8.6	1072.5
image passage (336x336 JPEG)		10	3	16.1	14.5	20.0	20.2	1748.3
image passage (336x336 JPEG)		10	5	25.8	25.1	31.5	31.7	1834.6
image passage (560x560 JPEG)		10	1	8.5	8.4	8.6	8.9	1039.8
image passage (560x560 JPEG)		10	3	16.0	13.5	19.5	19.7	1747.3
image passage (560x560 JPEG)		10	5	25.6	24.3	31.4	31.6	1835.4
image passage (1024x1024 JPEG)		10	1	28.0	27.9	28.6	28.9	333.1
image passage (1024x1024 JPEG)		10	3	70.8	51.2	93.9	94.7	408.6
image passage (1024x1024 JPEG)		10	5	116.7	114.6	142.2	143.6	414.5

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	512	10	1	47.2	47.3	48.2	48.5	206.5
passage	512	10	3	125.1	100.6	153.4	153.9	235.6
passage	512	10	5	194.5	197.6	235.0	235.4	251.2
passage	512	20	1	83.4	83.3	84.6	85.0	235.9
passage	512	20	3	219.7	172.5	273.3	274.0	269.8
passage	512	20	5	370.0	348.0	523.2	526.1	265.9
passage	512	40	1	163.9	164.0	166.4	167.3	241.4
passage	512	40	3	432.4	435.2	440.6	442.0	274.8
passage	512	40	5	715.1	719.3	800.8	905.9	277.2
image passage (336x336 JPEG)		10	1	9.4	9.4	9.5	9.5	958.3
image passage (336x336 JPEG)		10	3	19.6	19.5	20.8	21.1	1443.6
image passage (336x336 JPEG)		10	5	30.9	30.6	37.9	38.1	1538.2
image passage (560x560 JPEG)		10	1	9.5	9.5	9.7	9.7	926.4
image passage (560x560 JPEG)		10	3	19.3	19.2	20.6	20.7	1446.4
image passage (560x560 JPEG)		10	5	30.5	30.1	37.4	37.6	1550.3
image passage (1024x1024 JPEG)		10	1	31.6	31.5	32.5	32.6	295.3
image passage (1024x1024 JPEG)		10	3	80.4	76.9	92.5	96.9	359.1
image passage (1024x1024 JPEG)		10	5	133.7	132.6	164.1	166.0	362.1

NVIDIA-A100-SXM4-80GB

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	512	10	1	71.1	71.5	72.2	72.3	138.9
passage	512	10	3	189.4	140.6	243.9	244.6	156.6
passage	512	10	5	321.1	375.9	389.6	390.8	153.4
passage	512	20	1	140.6	140.7	141.8	142.2	141.3
passage	512	20	3	384.8	273.4	507.9	508.7	154.5
passage	512	20	5	651.4	610.7	919.3	922.6	151.4
passage	512	40	1	280.4	280.4	283.8	284.6	142.1
passage	512	40	3	765.0	764.2	776.0	786.3	156.4
passage	512	40	5	1263.2	1279.1	1292.4	1297.6	156.0
image passage (336x336 JPEG)		10	1	14.3	14.3	14.4	14.5	652.9
image passage (336x336 JPEG)		10	3	31.3	28.8	39.3	39.6	919.9
image passage (336x336 JPEG)		10	5	49.4	47.9	60.1	60.4	979.3
image passage (560x560 JPEG)		10	1	14.5	14.4	14.7	14.8	638.0
image passage (560x560 JPEG)		10	3	31.1	28.8	37.0	38.0	922.8
image passage (560x560 JPEG)		10	5	49.1	47.2	59.8	60.1	981.9
image passage (1024x1024 JPEG)		10	1	57.4	57.5	58.0	58.2	168.5
image passage (1024x1024 JPEG)		10	3	145.7	103.3	193.9	194.6	201.8
image passage (1024x1024 JPEG)		10	5	240.9	222.9	293.8	294.5	203.3

Llama Nemotron Rerank 500m

H100-HBM3-80GB

FP8

Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	27.1	27.0	30.0	31.3	367.0
512	10	3	33.2	33.0	36.0	37.2	878.0
512	10	5	54.5	55.0	60.0	62.9	896.0
512	20	1	38.7	38.0	42.0	43.1	513.0
512	20	3	60.5	61.0	67.0	67.7	967.0
512	20	5	100.9	101.0	110.0	111.2	971.0
512	40	1	66.6	67.0	72.0	72.6	598.0
512	40	3	123.8	124.0	133.0	135.7	958.0
512	40	5	205.0	208.0	220.0	224.3	956.0

FP16

Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	29.7	29.0	33.0	34.3	334.0
512	10	3	41.1	41.0	44.0	45.6	717.0
512	10	5	68.1	68.0	73.0	77.7	718.0
512	20	1	44.2	43.0	48.0	48.7	450.0
512	20	3	78.1	78.0	84.0	86.4	759.0
512	20	5	129.0	130.0	137.0	137.7	759.0
512	40	1	77.1	77.0	82.0	82.7	517.0
512	40	3	157.6	158.0	166.0	169.9	753.0
512	40	5	260.7	264.0	272.0	277.6	751.0

H100-NVL

FP8

Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	28.2	28.0	32.0	32.6	352.0
512	10	3	43.3	43.0	47.0	55.3	675.0
512	10	5	71.3	72.0	80.0	88.3	685.0
512	20	1	43.0	42.0	47.0	47.5	462.0
512	20	3	83.2	84.0	91.0	92.3	710.0
512	20	5	138.4	141.0	148.0	148.5	707.0
512	40	1	76.1	76.0	81.0	81.9	524.0
512	40	3	166.9	168.0	179.0	179.5	707.0
512	40	5	278.2	283.0	291.0	292.2	703.0

FP16

Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	32.7	33.0	35.0	37.0	304.0
512	10	3	58.0	58.0	62.0	68.5	511.0
512	10	5	95.6	97.0	104.0	113.9	510.0
512	20	1	53.7	54.0	58.0	58.7	371.0
512	20	3	114.1	116.0	124.0	125.9	515.0
512	20	5	191.0	195.0	204.0	204.9	513.0
512	40	1	96.1	96.0	103.0	104.6	415.0
512	40	3	232.1	236.0	246.0	247.9	506.0
512	40	5	387.5	396.0	409.0	413.3	505.0

L40S

FP8

Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	39.7	38.0	52.0	53.9	251.0
512	10	3	67.2	67.0	70.0	74.3	441.0
512	10	5	111.0	112.0	117.0	120.1	440.0
512	20	1	70.8	71.0	75.0	78.2	282.0
512	20	3	146.8	148.0	153.0	153.9	404.0
512	20	5	242.8	246.0	251.0	253.1	403.0
512	40	1	134.2	133.0	144.0	145.2	297.0
512	40	3	305.4	307.0	314.0	316.2	389.0
512	40	5	504.5	513.0	522.0	527.0	388.0

FP16

Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	53.7	51.0	66.0	68.1	186.0
512	10	3	105.4	106.0	109.0	110.8	280.0
512	10	5	175.5	178.0	184.0	186.3	279.0
512	20	1	94.5	95.0	99.0	100.1	211.0
512	20	3	218.6	222.0	228.0	229.5	270.0
512	20	5	364.1	370.0	377.0	379.9	269.0
512	40	1	186.6	185.0	195.0	198.8	214.0
512	40	3	459.2	462.0	470.0	472.0	258.0
512	40	5	759.3	773.0	785.0	787.0	258.0

A100-SXM4-80GB

FP16

Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	47.8	47.0	53.0	58.0	209.0
512	10	3	96.3	96.0	102.0	109.3	305.0
512	10	5	160.0	161.0	167.0	181.6	306.0
512	20	1	83.0	84.0	88.0	89.3	240.0
512	20	3	192.4	194.0	201.0	204.7	309.0
512	20	5	317.6	323.0	331.0	334.2	308.0
512	40	1	156.4	156.0	164.0	165.4	255.0
512	40	3	380.9	384.0	401.0	405.1	309.0
512	40	5	632.9	641.0	658.0	671.0	310.0

A10G

FP16

Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	124.7	125.0	131.0	132.9	80.0
512	10	3	326.5	330.0	338.0	339.4	91.0
512	10	5	542.0	552.0	562.0	565.1	90.0
512	20	1	238.0	238.0	245.0	246.1	84.0
512	20	3	656.1	662.0	672.0	675.0	91.0
512	20	5	1084.7	1105.0	1119.0	1120.6	90.0
512	40	1	469.6	470.0	476.0	478.6	85.0
512	40	3	1308.2	1330.0	1345.0	1346.8	90.0
512	40	5	2173.5	2219.0	2235.0	2240.1	90.0

L4

FP16

Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	136.5	137.0	140.0	143.2	73.0
512	10	3	379.0	384.0	392.0	393.7	78.0
512	10	5	627.5	640.0	649.0	651.9	78.0
512	20	1	275.9	276.0	282.0	285.2	72.0
512	20	3	786.4	795.0	806.0	809.3	76.0
512	20	5	1298.2	1324.0	1339.0	1342.5	75.0
512	40	1	549.2	549.0	558.0	559.7	73.0
512	40	3	1567.2	1594.0	1609.0	1610.5	75.0
512	40	5	2603.4	2654.0	2677.0	2684.0	75.0

Llama Nemotron Rerank 1B

H100-HBM3-80GB

FP8

Input tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	33.0	31.0	37.0	37.0	307.3
512	10	3	62.0	63.0	72.0	74.0	474.9
512	10	5	103.0	104.0	113.0	116.0	471.7
512	20	1	57.0	59.0	64.0	65.0	351.1
512	20	3	124.0	123.0	139.0	140.0	475.7
512	20	5	206.0	207.0	227.0	230.0	477.7
512	40	1	99.0	99.0	109.0	110.0	402.8
512	40	3	244.0	248.0	261.0	267.0	483.4
512	40	5	405.0	414.0	429.0	434.0	483.7

FP16

Input tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	40.0	39.0	43.0	44.0	250.4
512	10	3	88.0	89.0	96.0	104.0	337.7
512	10	5	145.0	148.0	153.0	154.0	337.1
512	20	1	71.0	74.0	78.0	79.0	280.1
512	20	3	171.0	172.0	184.0	187.0	345.0
512	20	5	284.0	288.0	305.0	310.0	345.3
512	40	1	129.0	129.0	139.0	140.0	309.5
512	40	3	341.0	346.0	358.0	361.0	347.5
512	40	5	565.0	575.0	592.0	601.0	347.2

L40S

FP8

Input tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	58.0	58.0	61.0	61.0	170.8
512	10	3	149.0	150.0	159.0	204.0	197.8
512	10	5	246.0	250.0	257.0	260.0	198.8
512	20	1	119.0	117.0	123.0	124.0	168.5
512	20	3	315.0	319.0	325.0	326.0	187.9
512	20	5	523.0	532.0	540.0	540.0	187.5
512	40	1	234.0	234.0	242.0	243.0	171.1
512	40	3	652.0	661.0	670.0	674.0	181.6
512	40	5	1080.0	1101.0	1118.0	1120.0	181.4

FP16

Input tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	78.0	78.0	80.0	80.0	128.2
512	10	3	205.0	207.0	210.0	212.0	144.8
512	10	5	340.0	346.0	354.0	357.0	144.0
512	20	1	158.0	157.0	162.0	163.0	126.8
512	20	3	430.0	436.0	443.0	446.0	137.2
512	20	5	716.0	728.0	739.0	744.0	136.9
512	40	1	312.0	312.0	320.0	321.0	128.0
512	40	3	886.0	896.0	907.0	910.0	134.0
512	40	5	1463.0	1492.0	1512.0	1515.0	134.0
512	40	5	1463.0	1492.0	1512.0	1515.0	134.0

A100-SXM4-80GB

FP16

Input tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	74.0	74.0	79.0	80.0	134.8
512	10	3	185.0	188.0	202.0	244.0	157.3
512	10	5	311.0	315.0	325.0	329.0	157.6
512	20	1	139.0	137.0	149.0	150.0	143.5
512	20	3	371.0	373.0	394.0	398.0	159.8
512	20	5	615.0	622.0	644.0	648.0	159.5
512	40	1	267.0	266.0	286.0	290.0	149.8
512	40	3	744.0	752.0	788.0	799.0	159.1
512	40	5	1231.0	1252.0	1296.0	1316.0	159.0

A100-PCIe-80GB

FP16

Input tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	79.0	79.0	84.0	85.0	126.4
512	10	3	203.0	206.0	217.0	271.0	144.5
512	10	5	339.0	346.0	354.0	358.0	144.4
512	20	1	151.0	151.0	160.0	161.0	132.0
512	20	3	405.0	411.0	428.0	435.0	145.4
512	20	5	672.0	685.0	710.0	717.0	145.5
512	40	1	289.0	287.0	307.0	310.0	138.2
512	40	3	811.0	817.0	842.0	851.0	146.3
512	40	5	1340.0	1357.0	1405.0	1412.0	146.2

A10G

FP16

Input tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	230.0	229.0	240.0	244.0	43.5
512	10	3	639.0	644.0	667.0	675.0	46.5
512	10	5	1055.0	1076.0	1101.0	1107.0	46.4
512	20	1	447.0	442.0	464.0	468.0	44.7
512	20	3	1257.0	1276.0	1310.0	1320.0	46.9
512	20	5	2088.0	2126.0	2172.0	2185.0	46.9
512	40	1	877.0	879.0	902.0	906.0	45.6
512	40	3	2534.0	2566.0	2605.0	2617.0	46.7
512	40	5	4194.0	4273.0	4321.0	4339.0	46.7

L4

FP8

Input tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	190.0	187.0	197.0	199.0	52.7
512	10	3	534.0	543.0	554.0	557.0	55.2
512	10	5	888.0	907.0	924.0	925.0	55.2
512	20	1	383.0	384.0	395.0	397.0	52.1
512	20	3	1117.0	1134.0	1150.0	1153.0	53.0
512	20	5	1850.0	1885.0	1908.0	1916.0	53.0
512	40	1	768.0	768.0	785.0	787.0	52.1
512	40	3	2254.0	2279.0	2302.0	2308.0	52.7
512	40	5	3712.0	3795.0	3834.0	3839.0	52.7

FP16

Input tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
512	10	1	188.0	188.0	196.0	197.0	53.2
512	10	3	533.0	541.0	554.0	559.0	55.4
512	10	5	884.0	902.0	917.0	922.0	55.4
512	20	1	383.0	382.0	396.0	399.0	52.2
512	20	3	1119.0	1134.0	1147.0	1152.0	53.1
512	20	5	1848.0	1884.0	1910.0	1916.0	53.0
512	40	1	767.0	767.0	784.0	788.0	52.2
512	40	3	2245.0	2276.0	2301.0	2308.0	52.8
512	40	5	3714.0	3790.0	3825.0	3831.0	52.8