Performance for NVIDIA NeMo Retriever Embedding NIM

To benchmark the performance of NVIDIA NeMo Retriever Embedding NIM, you can use the AIPerf tool.

To run a performance benchmark with representative input data, create a JSONL file where each line contains a {"texts": [...]} object, as shown in the following example.

Example

Create a file named embeddings.jsonl that contains the following content.

{"texts": ["What was the first car ever driven?"]}
{"texts": ["Who served as the 5th President of the United States of America?"]}
{"texts": ["Is the Sydney Opera House located in Australia?"]}
{"texts": ["In what state did they film Shrek 2?"]}

Install the verified AIPerf version in a Python environment that can reach the NIM endpoint.

pip install aiperf==0.9.0

Run the following command to benchmark the embedding endpoint by using AIPerf.

aiperf profile \
    --model nvidia/llama-nemotron-embed-vl-1b-v2 \
    --endpoint-type nim_embeddings \
    --url localhost:8000 \
    --input-file embeddings.jsonl \
    --custom-dataset-type single_turn \
    --extra-inputs input_type:query \
    --extra-inputs truncate:END \
    --request-count 20 \
    --concurrency 5

You can see the full set of command line options in the AIPerf command line options documentation.

Benchmarks

All latency measurements are reported in milliseconds.

The previous AIPerf example benchmarks a custom JSONL workload. The benchmark tables use a separate synthetic AIPerf benchmark matrix for consistent SKU comparisons. The VLM Embed benchmark tables report measurements for the supported GPU SKUs. For text rows, batch size is embedding inputs per request and throughput is embedding inputs per second. For image rows, batch size is images per request, image size is the generated JPEG resolution, and throughput is images per second.

Llama Nemotron Embed VL 1B v2

NVIDIA-RTX-PRO-6000-Blackwell-Workstation-Edition

FP8

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	2.3	2.3	2.4	2.4	352.8
query	20		1	3	4.1	4.3	4.4	4.4	628.0
query	20		1	5	4.1	4.3	4.4	4.5	1041.4
query	20		1	7	4.4	4.6	4.9	5.2	1316.6
query	20		1	9	4.4	4.4	4.6	4.6	1668.9
query	20		1	11	5.4	5.7	5.8	5.9	1699.0
query	20		1	13	4.6	4.7	4.9	5.1	2267.7
query	20		1	15	4.9	5.0	5.3	5.4	2421.9
passage	300		64	1	123.1	121.1	131.8	133.3	493.5
passage	300		64	3	349.6	353.4	361.5	364.8	534.4
passage	300		64	5	589.8	604.0	614.4	615.7	526.2
passage	512		64	1	195.7	195.6	198.0	198.7	317.9
passage	512		64	3	589.6	595.4	601.8	602.2	319.0
passage	512		64	5	1007.9	1024.3	1055.6	1058.9	309.1
image passage		336x336 JPEG	1	1	17.5	17.5	17.6	17.6	57.0
image passage		336x336 JPEG	1	2	29.5	29.5	29.7	29.8	34.1
image passage		336x336 JPEG	1	3	44.3	44.6	44.8	44.8	22.8
image passage		336x336 JPEG	1	5	73.4	74.8	74.9	75.0	14.1
image passage		336x336 JPEG	1	8	115.9	119.9	120.1	120.1	9.3
image passage		336x336 JPEG	4	1	63.3	63.3	63.5	63.6	63.2
image passage		336x336 JPEG	4	2	119.2	119.8	119.9	120.0	33.7
image passage		336x336 JPEG	4	3	178.5	180.2	180.4	180.4	22.7
image passage		336x336 JPEG	4	5	295.2	301.0	301.5	301.6	14.1
image passage		336x336 JPEG	4	8	466.1	482.9	483.3	483.4	9.4
image passage		336x336 JPEG	8	1	124.7	124.7	124.8	124.9	64.2
image passage		336x336 JPEG	8	2	239.9	241.0	241.3	241.4	33.5
image passage		336x336 JPEG	8	3	358.6	362.1	362.4	362.6	22.6
image passage		336x336 JPEG	8	5	592.7	604.4	605.4	607.0	14.0
image passage		336x336 JPEG	8	8	934.9	968.0	970.9	971.4	9.4
image passage		512x512 JPEG	1	1	17.7	17.7	17.9	17.9	56.4
image passage		512x512 JPEG	1	2	29.6	29.7	29.9	29.9	33.9
image passage		512x512 JPEG	1	3	44.4	44.8	45.0	45.0	22.7
image passage		512x512 JPEG	1	5	73.7	75.1	75.3	75.3	14.0
image passage		512x512 JPEG	1	8	116.5	120.4	120.7	120.8	9.3
image passage		512x512 JPEG	4	1	63.9	63.9	64.2	64.2	62.6
image passage		512x512 JPEG	4	2	119.6	120.1	120.4	120.4	33.6
image passage		512x512 JPEG	4	3	179.1	180.8	181.1	181.1	22.6
image passage		512x512 JPEG	4	5	296.3	302.1	302.4	302.4	14.0
image passage		512x512 JPEG	4	8	467.3	484.0	484.3	484.4	9.3
image passage		512x512 JPEG	8	1	126.0	125.9	126.5	126.6	63.5
image passage		512x512 JPEG	8	2	240.2	241.3	241.7	241.8	33.5
image passage		512x512 JPEG	8	3	359.3	362.7	363.1	363.2	22.6
image passage		512x512 JPEG	8	5	594.0	605.7	607.1	607.5	14.0
image passage		512x512 JPEG	8	8	936.8	970.1	971.4	972.0	9.3
image passage		1024x1024 JPEG	1	1	69.0	68.9	69.3	69.4	14.5
image passage		1024x1024 JPEG	1	2	130.6	131.2	131.5	131.6	7.7
image passage		1024x1024 JPEG	1	3	195.4	197.2	197.6	197.7	5.2
image passage		1024x1024 JPEG	1	5	323.1	329.4	329.9	330.5	3.2
image passage		1024x1024 JPEG	1	8	509.4	527.5	528.1	528.2	2.1
image passage		1024x1024 JPEG	4	1	272.3	271.2	277.1	279.1	14.7
image passage		1024x1024 JPEG	4	2	533.7	537.0	543.8	546.7	7.5
image passage		1024x1024 JPEG	4	3	798.0	809.6	818.3	819.6	5.1
image passage		1024x1024 JPEG	4	5	1322.1	1354.5	1371.0	1375.4	3.2
image passage		1024x1024 JPEG	4	8	2079.5	2159.9	2183.7	2186.3	2.1
image passage		1024x1024 JPEG	8	1	554.1	554.6	564.7	565.9	14.4
image passage		1024x1024 JPEG	8	2	1088.1	1096.3	1112.3	1113.3	7.4
image passage		1024x1024 JPEG	8	3	1628.1	1651.7	1664.9	1670.7	5.0
image passage		1024x1024 JPEG	8	5	2679.2	2740.7	2767.2	2776.2	3.1
image passage		1024x1024 JPEG	8	8	4225.0	4389.5	4426.5	4429.8	2.1
image passage		2048x2048 JPEG	1	1	70.5	70.5	71.1	71.2	14.2
image passage		2048x2048 JPEG	1	2	131.2	131.8	132.1	132.2	7.7
image passage		2048x2048 JPEG	1	3	196.4	198.3	198.7	198.8	5.2
image passage		2048x2048 JPEG	1	5	325.1	331.5	332.2	332.2	3.2
image passage		2048x2048 JPEG	1	8	512.0	530.3	530.8	530.9	2.1
image passage		2048x2048 JPEG	4	1	273.8	273.5	275.0	275.5	14.6
image passage		2048x2048 JPEG	4	2	524.3	526.3	530.9	531.6	7.7
image passage		2048x2048 JPEG	4	3	786.2	790.6	805.7	810.8	5.2
image passage		2048x2048 JPEG	4	5	1297.0	1319.6	1335.3	1337.6	3.2
image passage		2048x2048 JPEG	4	8	2045.1	2113.8	2135.7	2138.1	2.1
image passage		2048x2048 JPEG	8	1	550.1	548.1	559.5	563.8	14.5
image passage		2048x2048 JPEG	8	2	1060.5	1061.0	1087.1	1090.1	7.6
image passage		2048x2048 JPEG	8	3	1585.0	1592.0	1630.7	1633.7	5.1
image passage		2048x2048 JPEG	8	5	2617.5	2655.0	2724.0	2728.7	3.2
image passage		2048x2048 JPEG	8	8	4123.5	4248.4	4350.1	4357.3	2.1

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

FP8

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	2.6	2.6	2.6	2.6	312.1
query	20		1	3	4.6	4.6	4.7	4.8	558.7
query	20		1	5	4.7	4.7	4.9	5.0	903.9
query	20		1	7	4.7	4.8	4.9	5.1	1241.8
query	20		1	9	5.2	5.2	5.6	5.7	1420.7
query	20		1	11	5.1	5.1	5.5	5.9	1747.8
query	20		1	13	5.6	5.3	6.5	6.7	1922.1
query	20		1	15	6.8	6.9	7.2	8.6	1800.2
passage	300		64	1	117.9	117.9	118.4	118.6	507.5
passage	300		64	3	323.5	326.4	327.7	328.1	573.5
passage	300		64	5	538.5	548.9	550.5	550.8	573.5
passage	512		64	1	200.0	199.8	202.0	202.3	307.5
passage	512		64	3	567.7	573.0	575.8	576.7	329.9
passage	512		64	5	943.8	962.3	966.4	967.6	329.4
image passage		336x336 JPEG	1	1	20.1	20.1	20.2	20.3	49.8
image passage		336x336 JPEG	1	2	33.7	33.6	34.4	35.2	29.8
image passage		336x336 JPEG	1	3	50.7	51.0	51.2	51.8	19.9
image passage		336x336 JPEG	1	5	83.9	85.4	86.0	86.7	12.3
image passage		336x336 JPEG	1	8	132.8	137.4	137.5	137.5	8.1
image passage		336x336 JPEG	4	1	72.8	72.7	73.0	73.6	54.9
image passage		336x336 JPEG	4	2	136.3	136.8	137.3	138.4	29.5
image passage		336x336 JPEG	4	3	204.1	205.9	206.7	207.1	19.9
image passage		336x336 JPEG	4	5	337.8	344.3	345.1	346.0	12.3
image passage		336x336 JPEG	4	8	532.7	551.6	552.8	553.1	8.2
image passage		336x336 JPEG	8	1	143.0	142.8	143.3	143.9	55.9
image passage		336x336 JPEG	8	2	273.5	274.3	275.9	276.9	29.4
image passage		336x336 JPEG	8	3	409.3	412.3	413.4	418.7	19.8
image passage		336x336 JPEG	8	5	676.9	689.0	690.1	697.7	12.3
image passage		336x336 JPEG	8	8	1067.6	1103.6	1108.5	1125.1	8.2
image passage		512x512 JPEG	1	1	20.3	20.3	20.4	20.5	49.3
image passage		512x512 JPEG	1	2	33.6	33.7	34.0	34.1	29.9
image passage		512x512 JPEG	1	3	50.8	51.1	51.3	51.9	19.9
image passage		512x512 JPEG	1	5	84.3	85.8	85.9	86.5	12.3
image passage		512x512 JPEG	1	8	132.9	137.4	137.8	138.1	8.1
image passage		512x512 JPEG	4	1	73.7	73.6	74.2	75.5	54.3
image passage		512x512 JPEG	4	2	137.0	137.0	138.6	142.2	29.3
image passage		512x512 JPEG	4	3	204.5	206.2	207.6	208.4	19.8
image passage		512x512 JPEG	4	5	338.0	344.5	344.9	346.1	12.3
image passage		512x512 JPEG	4	8	533.6	552.3	554.2	554.9	8.2
image passage		512x512 JPEG	8	1	144.7	144.5	145.2	145.7	55.3
image passage		512x512 JPEG	8	2	274.1	274.7	276.3	277.1	29.3
image passage		512x512 JPEG	8	3	410.2	413.1	414.7	421.4	19.8
image passage		512x512 JPEG	8	5	677.5	689.9	692.2	698.3	12.3
image passage		512x512 JPEG	8	8	1069.3	1105.2	1109.3	1126.3	8.2
image passage		1024x1024 JPEG	1	1	77.1	77.1	77.7	77.7	13.0
image passage		1024x1024 JPEG	1	2	145.6	145.9	146.3	148.5	6.9
image passage		1024x1024 JPEG	1	3	217.3	219.3	219.6	219.9	4.7
image passage		1024x1024 JPEG	1	5	359.1	366.2	366.5	366.6	2.9
image passage		1024x1024 JPEG	1	8	566.5	586.7	587.2	587.3	1.9
image passage		1024x1024 JPEG	4	1	302.2	301.8	303.8	305.6	13.2
image passage		1024x1024 JPEG	4	2	584.0	586.0	588.7	591.5	6.9
image passage		1024x1024 JPEG	4	3	872.6	880.3	883.0	890.7	4.6
image passage		1024x1024 JPEG	4	5	1441.6	1468.0	1477.5	1486.2	2.9
image passage		1024x1024 JPEG	4	8	2271.5	2351.6	2362.9	2364.7	1.9
image passage		1024x1024 JPEG	8	1	601.6	600.7	603.5	607.0	13.3
image passage		1024x1024 JPEG	8	2	1166.9	1171.7	1176.8	1178.0	6.9
image passage		1024x1024 JPEG	8	3	1743.4	1759.6	1762.9	1769.8	4.7
image passage		1024x1024 JPEG	8	5	2881.6	2936.8	2952.5	2953.7	2.9
image passage		1024x1024 JPEG	8	8	4538.7	4699.2	4712.7	4714.2	1.9
image passage		2048x2048 JPEG	1	1	79.4	79.5	80.4	80.6	12.6
image passage		2048x2048 JPEG	1	2	146.4	146.5	147.7	151.6	6.9
image passage		2048x2048 JPEG	1	3	218.2	220.2	220.7	220.9	4.6
image passage		2048x2048 JPEG	1	5	361.4	368.4	369.0	369.3	2.9
image passage		2048x2048 JPEG	1	8	571.2	590.5	591.6	601.0	1.9
image passage		2048x2048 JPEG	4	1	310.7	310.4	312.7	313.7	12.9
image passage		2048x2048 JPEG	4	2	584.8	586.8	589.1	590.4	6.9
image passage		2048x2048 JPEG	4	3	875.5	883.1	885.8	888.2	4.6
image passage		2048x2048 JPEG	4	5	1447.2	1475.0	1476.9	1491.8	2.9
image passage		2048x2048 JPEG	4	8	2282.4	2363.7	2366.4	2367.1	1.9
image passage		2048x2048 JPEG	8	1	619.6	618.6	625.5	626.1	12.9
image passage		2048x2048 JPEG	8	2	1170.0	1174.1	1179.5	1181.6	6.9
image passage		2048x2048 JPEG	8	3	1749.5	1765.4	1770.5	1773.4	4.6
image passage		2048x2048 JPEG	8	5	2897.2	2952.6	2963.6	2966.9	2.9
image passage		2048x2048 JPEG	8	8	4576.9	4738.8	4749.0	4752.8	1.9

NVIDIA-B200

FP8

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	2.6	2.5	2.7	2.7	249.4
query	20		1	3	4.1	4.4	4.8	5.0	524.8
query	20		1	5	4.4	4.5	6.3	7.0	833.4
query	20		1	7	4.7	4.6	6.2	9.0	1093.6
query	20		1	9	5.5	5.2	8.6	8.9	1284.3
query	20		1	11	5.5	5.2	10.1	11.5	1478.0
query	20		1	13	5.6	5.3	7.9	8.1	1679.9
query	20		1	15	6.2	6.1	8.9	10.6	1873.0
passage	300		64	1	149.6	149.6	150.1	150.5	408.9
passage	300		64	3	422.2	426.1	427.1	427.6	443.0
passage	300		64	5	701.0	714.9	716.4	716.5	443.0
passage	512		64	1	361.2	360.7	365.6	367.3	173.8
passage	512		64	3	1049.8	1059.5	1067.8	1068.7	179.8
passage	512		64	5	1736.6	1771.6	1782.3	1784.5	179.8
image passage		336x336 JPEG	1	1	18.5	17.4	17.5	25.3	55.7
image passage		336x336 JPEG	1	2	27.9	26.9	27.4	34.0	36.5
image passage		336x336 JPEG	1	3	42.0	41.3	45.5	50.2	24.1
image passage		336x336 JPEG	1	5	69.9	70.0	70.9	77.0	14.8
image passage		336x336 JPEG	1	8	114.4	113.1	139.6	142.4	9.5
image passage		336x336 JPEG	4	1	62.5	60.9	63.3	71.7	64.4
image passage		336x336 JPEG	4	2	113.4	111.9	124.6	127.2	35.5
image passage		336x336 JPEG	4	3	171.2	169.0	185.6	188.4	23.7
image passage		336x336 JPEG	4	5	285.5	283.4	305.0	307.6	14.6
image passage		336x336 JPEG	4	8	454.3	469.6	481.9	482.3	9.4
image passage		336x336 JPEG	8	1	121.1	118.6	130.7	135.2	66.2
image passage		336x336 JPEG	8	2	231.4	225.8	251.6	257.7	34.8
image passage		336x336 JPEG	8	3	347.6	343.0	372.0	378.9	23.4
image passage		336x336 JPEG	8	5	574.6	583.0	603.7	614.8	14.5
image passage		336x336 JPEG	8	8	901.2	926.7	955.7	960.4	9.7
image passage		512x512 JPEG	1	1	18.0	17.5	17.7	18.6	56.1
image passage		512x512 JPEG	1	2	27.6	27.2	27.5	28.5	36.5
image passage		512x512 JPEG	1	3	41.9	41.3	41.7	52.4	24.2
image passage		512x512 JPEG	1	5	73.6	69.9	85.6	95.4	14.2
image passage		512x512 JPEG	1	8	110.6	112.7	113.7	131.6	9.8
image passage		512x512 JPEG	4	1	63.0	61.4	70.5	73.5	63.8
image passage		512x512 JPEG	4	2	114.6	112.2	126.1	127.5	35.2
image passage		512x512 JPEG	4	3	173.0	169.3	190.2	193.3	23.5
image passage		512x512 JPEG	4	5	283.4	283.6	298.0	304.7	14.5
image passage		512x512 JPEG	4	8	452.2	459.4	490.2	518.2	9.7
image passage		512x512 JPEG	8	1	123.3	120.2	133.5	138.6	65.1
image passage		512x512 JPEG	8	2	231.1	226.0	242.0	246.2	34.8
image passage		512x512 JPEG	8	3	344.3	341.2	359.2	362.7	23.6
image passage		512x512 JPEG	8	5	571.6	582.6	601.6	605.3	14.6
image passage		512x512 JPEG	8	8	903.2	933.5	941.6	948.8	9.7
image passage		1024x1024 JPEG	1	1	89.0	87.0	87.8	104.4	11.3
image passage		1024x1024 JPEG	1	2	165.9	164.8	166.7	175.2	6.1
image passage		1024x1024 JPEG	1	3	247.7	247.7	251.0	270.0	4.1
image passage		1024x1024 JPEG	1	5	409.8	414.4	432.3	438.5	2.5
image passage		1024x1024 JPEG	1	8	642.3	663.7	667.1	676.7	1.7
image passage		1024x1024 JPEG	4	1	343.2	339.6	356.0	363.0	11.7
image passage		1024x1024 JPEG	4	2	662.6	663.2	670.1	680.4	6.1
image passage		1024x1024 JPEG	4	3	997.8	996.5	1033.5	1041.3	4.1
image passage		1024x1024 JPEG	4	5	1642.2	1668.1	1701.8	1714.4	2.5
image passage		1024x1024 JPEG	4	8	2591.9	2671.7	2707.8	2771.5	1.7
image passage		1024x1024 JPEG	8	1	683.2	677.5	698.8	704.3	11.7
image passage		1024x1024 JPEG	8	2	1333.9	1331.0	1369.9	1381.8	6.0
image passage		1024x1024 JPEG	8	3	1987.5	1998.3	2044.4	2058.4	4.1
image passage		1024x1024 JPEG	8	5	3288.4	3342.0	3391.1	3423.1	2.5
image passage		1024x1024 JPEG	8	8	5184.3	5368.9	5411.5	5448.4	1.7
image passage		2048x2048 JPEG	1	1	88.6	87.6	89.4	100.6	11.3
image passage		2048x2048 JPEG	1	2	163.4	161.7	171.3	179.5	6.2
image passage		2048x2048 JPEG	1	3	245.1	243.6	260.1	260.6	4.1
image passage		2048x2048 JPEG	1	5	406.1	407.6	425.4	426.5	2.6
image passage		2048x2048 JPEG	1	8	639.7	660.4	669.9	686.7	1.7
image passage		2048x2048 JPEG	4	1	343.4	340.5	352.8	362.2	11.7
image passage		2048x2048 JPEG	4	2	660.5	665.7	673.2	688.2	6.1
image passage		2048x2048 JPEG	4	3	988.0	995.9	1005.8	1020.3	4.1
image passage		2048x2048 JPEG	4	5	1629.1	1663.9	1677.2	1694.1	2.6
image passage		2048x2048 JPEG	4	8	2567.7	2662.8	2672.1	2684.2	1.7
image passage		2048x2048 JPEG	8	1	684.9	678.8	700.5	706.7	11.7
image passage		2048x2048 JPEG	8	2	1321.7	1322.9	1346.3	1359.7	6.1
image passage		2048x2048 JPEG	8	3	1972.1	1991.7	2004.8	2022.1	4.1
image passage		2048x2048 JPEG	8	5	3261.1	3320.0	3352.2	3384.5	2.6
image passage		2048x2048 JPEG	8	8	5137.7	5319.9	5338.2	5344.2	1.7

NVIDIA-GB200

FP8

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	1.8	1.8	1.9	1.9	426.9
query	20		1	3	2.6	2.3	3.8	3.8	935.5
query	20		1	5	3.6	3.8	4.4	4.5	1159.1
query	20		1	7	4.1	4.0	5.6	5.8	1438.4
query	20		1	9	4.3	4.4	5.7	5.9	1747.4
query	20		1	11	5.2	4.6	7.1	7.2	1786.2
query	20		1	13	4.6	4.5	6.0	6.1	2320.9
query	20		1	15	4.7	5.0	6.0	6.4	2596.1
passage	300		64	1	139.9	139.9	140.5	140.8	439.8
passage	300		64	3	399.1	402.8	404.3	404.6	469.5
passage	300		64	5	662.2	675.4	677.2	677.5	469.5
passage	512		64	1	334.0	333.6	337.4	338.3	188.4
passage	512		64	3	975.0	984.4	990.0	991.3	193.8
passage	512		64	5	1612.3	1645.1	1652.2	1653.5	193.8
image passage		336x336 JPEG	1	1	22.3	21.7	21.8	22.5	45.4
image passage		336x336 JPEG	1	2	37.9	38.0	38.1	38.6	26.5
image passage		336x336 JPEG	1	3	56.7	57.2	57.2	57.3	17.8
image passage		336x336 JPEG	1	5	93.8	95.6	95.7	95.8	11.0
image passage		336x336 JPEG	1	8	149.4	153.1	156.5	170.9	7.3
image passage		336x336 JPEG	4	1	81.2	79.8	80.2	89.0	49.5
image passage		336x336 JPEG	4	2	153.7	153.1	153.4	166.3	26.2
image passage		336x336 JPEG	4	3	231.0	229.9	247.2	248.6	17.6
image passage		336x336 JPEG	4	5	378.8	383.7	389.9	400.1	11.0
image passage		336x336 JPEG	4	8	604.4	615.6	642.8	657.9	7.2
image passage		336x336 JPEG	8	1	160.1	158.0	162.0	174.7	50.1
image passage		336x336 JPEG	8	2	310.4	307.7	322.6	336.0	25.9
image passage		336x336 JPEG	8	3	464.4	461.9	483.5	512.2	17.5
image passage		336x336 JPEG	8	5	767.4	772.0	823.1	830.1	10.9
image passage		336x336 JPEG	8	8	1208.0	1235.9	1298.0	1307.0	7.3
image passage		512x512 JPEG	1	1	22.4	21.7	21.9	24.3	45.1
image passage		512x512 JPEG	1	2	39.1	37.9	38.4	50.6	25.9
image passage		512x512 JPEG	1	3	57.0	57.4	57.6	58.2	17.7
image passage		512x512 JPEG	1	5	94.1	95.7	95.9	97.2	11.0
image passage		512x512 JPEG	1	8	148.3	153.4	153.7	153.9	7.3
image passage		512x512 JPEG	4	1	81.8	80.4	82.7	94.1	49.0
image passage		512x512 JPEG	4	2	155.9	153.2	166.4	177.9	25.8
image passage		512x512 JPEG	4	3	230.4	230.5	231.4	248.9	17.6
image passage		512x512 JPEG	4	5	380.7	385.0	390.2	408.5	10.9
image passage		512x512 JPEG	4	8	605.2	616.2	657.1	677.5	7.3
image passage		512x512 JPEG	8	1	161.8	159.4	163.2	174.3	49.6
image passage		512x512 JPEG	8	2	310.7	307.4	322.0	348.5	25.9
image passage		512x512 JPEG	8	3	464.8	462.5	484.0	510.1	17.5
image passage		512x512 JPEG	8	5	766.0	772.5	810.1	821.0	10.9
image passage		512x512 JPEG	8	8	1210.6	1238.3	1299.0	1304.6	7.2
image passage		1024x1024 JPEG	1	1	109.8	108.4	108.9	113.4	9.1
image passage		1024x1024 JPEG	1	2	211.1	210.0	210.9	216.3	4.8
image passage		1024x1024 JPEG	1	3	314.1	315.3	322.3	329.1	3.2
image passage		1024x1024 JPEG	1	5	520.3	525.7	549.1	554.9	2.0
image passage		1024x1024 JPEG	1	8	823.4	842.5	892.1	897.7	1.3
image passage		1024x1024 JPEG	4	1	433.7	428.0	452.5	474.8	9.2
image passage		1024x1024 JPEG	4	2	846.5	846.1	868.6	871.0	4.7
image passage		1024x1024 JPEG	4	3	1265.0	1268.4	1305.0	1309.4	3.2
image passage		1024x1024 JPEG	4	5	2085.6	2117.9	2151.5	2157.8	2.0
image passage		1024x1024 JPEG	4	8	3292.2	3410.4	3433.6	3437.2	1.3
image passage		1024x1024 JPEG	8	1	864.3	855.6	890.3	897.4	9.3
image passage		1024x1024 JPEG	8	2	1693.5	1695.3	1723.9	1727.6	4.7
image passage		1024x1024 JPEG	8	3	2531.1	2555.8	2579.4	2583.6	3.2
image passage		1024x1024 JPEG	8	5	4184.7	4270.2	4287.7	4296.9	2.0
image passage		1024x1024 JPEG	8	8	6578.8	6813.0	6838.2	6845.4	1.3
image passage		2048x2048 JPEG	1	1	110.9	109.9	110.7	116.1	9.0
image passage		2048x2048 JPEG	1	2	211.0	210.3	214.5	226.4	4.8
image passage		2048x2048 JPEG	1	3	316.6	315.9	327.4	343.1	3.2
image passage		2048x2048 JPEG	1	5	520.8	527.1	547.1	548.6	2.0
image passage		2048x2048 JPEG	1	8	822.8	845.6	872.5	875.1	1.3
image passage		2048x2048 JPEG	4	1	437.5	434.4	449.5	453.0	9.1
image passage		2048x2048 JPEG	4	2	850.9	848.4	874.0	877.0	4.7
image passage		2048x2048 JPEG	4	3	1265.7	1272.8	1288.3	1293.9	3.2
image passage		2048x2048 JPEG	4	5	2091.5	2132.9	2155.0	2158.8	2.0
image passage		2048x2048 JPEG	4	8	3299.6	3419.2	3439.6	3444.4	1.3
image passage		2048x2048 JPEG	8	1	876.0	870.1	891.7	898.0	9.1
image passage		2048x2048 JPEG	8	2	1699.3	1706.7	1722.6	1727.6	4.7
image passage		2048x2048 JPEG	8	3	2535.9	2558.4	2580.8	2586.6	3.2
image passage		2048x2048 JPEG	8	5	4176.6	4253.3	4296.1	4305.1	2.0
image passage		2048x2048 JPEG	8	8	6584.1	6816.2	6852.1	6862.0	1.3

NVIDIA-H200

FP8

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	3.0	3.0	3.1	3.1	225.7
query	20		1	3	4.5	4.8	5.6	5.7	489.2
query	20		1	5	5.6	4.8	9.4	11.1	690.3
query	20		1	7	7.7	5.5	14.5	15.5	742.8
query	20		1	9	6.7	5.7	10.4	11.9	1057.9
query	20		1	11	7.2	5.6	16.3	17.0	1236.9
query	20		1	13	9.4	9.1	17.2	17.5	1137.7
query	20		1	15	9.9	8.3	15.2	15.4	1261.1
passage	300		64	1	71.0	70.8	72.3	72.6	816.9
passage	300		64	3	189.9	191.6	193.1	194.2	962.6
passage	300		64	5	318.3	324.5	326.5	326.8	962.1
passage	512		64	1	121.0	121.1	122.1	122.4	497.5
passage	512		64	3	338.7	341.6	344.4	344.7	548.7
passage	512		64	5	564.9	575.9	581.0	581.7	547.7
image passage		336x336 JPEG	1	1	17.2	17.0	17.1	17.2	58.2
image passage		336x336 JPEG	1	2	26.4	26.3	26.6	26.7	38.0
image passage		336x336 JPEG	1	3	40.7	40.1	44.1	47.7	24.9
image passage		336x336 JPEG	1	5	68.6	68.0	77.0	78.9	15.1
image passage		336x336 JPEG	1	8	107.7	109.7	119.1	119.3	10.0
image passage		336x336 JPEG	4	1	60.0	59.4	59.6	64.7	66.8
image passage		336x336 JPEG	4	2	112.0	109.3	120.8	124.1	36.0
image passage		336x336 JPEG	4	3	165.0	164.8	175.4	176.8	24.6
image passage		336x336 JPEG	4	5	276.1	276.9	289.4	290.8	15.0
image passage		336x336 JPEG	4	8	434.0	445.0	459.4	460.5	10.1
image passage		336x336 JPEG	8	1	117.8	115.9	126.0	130.3	68.0
image passage		336x336 JPEG	8	2	222.3	220.1	231.5	232.8	36.2
image passage		336x336 JPEG	8	3	334.0	332.9	343.2	348.1	24.3
image passage		336x336 JPEG	8	5	550.8	562.5	567.7	571.5	15.1
image passage		336x336 JPEG	8	8	868.7	899.8	904.9	907.0	10.1
image passage		512x512 JPEG	1	1	17.4	17.2	17.4	17.4	57.7
image passage		512x512 JPEG	1	2	26.8	26.3	26.7	31.7	37.6
image passage		512x512 JPEG	1	3	40.6	40.2	41.1	46.8	24.9
image passage		512x512 JPEG	1	5	68.5	68.1	76.8	78.8	15.1
image passage		512x512 JPEG	1	8	109.0	110.2	121.5	129.9	9.9
image passage		512x512 JPEG	4	1	60.9	60.1	60.7	65.9	65.9
image passage		512x512 JPEG	4	2	110.3	109.4	114.6	120.3	36.4
image passage		512x512 JPEG	4	3	165.8	165.3	173.8	174.1	24.4
image passage		512x512 JPEG	4	5	274.2	276.0	287.6	287.8	15.0
image passage		512x512 JPEG	4	8	440.3	453.0	468.1	496.9	9.9
image passage		512x512 JPEG	8	1	119.1	117.4	123.9	129.1	67.3
image passage		512x512 JPEG	8	2	222.1	220.5	231.0	232.2	36.2
image passage		512x512 JPEG	8	3	333.5	332.4	345.0	347.5	24.3
image passage		512x512 JPEG	8	5	551.5	563.6	570.0	574.1	15.0
image passage		512x512 JPEG	8	8	870.9	899.7	902.0	941.1	10.1
image passage		1024x1024 JPEG	1	1	63.0	62.4	62.8	64.8	15.9
image passage		1024x1024 JPEG	1	2	115.3	115.3	119.2	120.5	8.7
image passage		1024x1024 JPEG	1	3	172.5	173.5	178.4	178.7	5.9
image passage		1024x1024 JPEG	1	5	285.7	290.6	293.6	294.7	3.6
image passage		1024x1024 JPEG	1	8	450.3	466.2	466.9	466.9	2.4
image passage		1024x1024 JPEG	4	1	242.6	240.7	242.7	253.8	16.5
image passage		1024x1024 JPEG	4	2	466.3	465.5	471.9	472.6	8.6
image passage		1024x1024 JPEG	4	3	697.4	702.8	705.3	705.8	5.8
image passage		1024x1024 JPEG	4	5	1152.7	1171.4	1182.0	1188.1	3.6
image passage		1024x1024 JPEG	4	8	1818.0	1879.6	1885.2	1940.0	2.4
image passage		1024x1024 JPEG	8	1	482.0	479.4	488.5	490.7	16.6
image passage		1024x1024 JPEG	8	2	933.9	936.5	938.7	955.2	8.6
image passage		1024x1024 JPEG	8	3	1394.4	1405.3	1411.6	1422.5	5.8
image passage		1024x1024 JPEG	8	5	2309.2	2346.5	2395.0	2413.7	3.6
image passage		1024x1024 JPEG	8	8	3631.9	3760.2	3768.9	3817.7	2.4
image passage		2048x2048 JPEG	1	1	64.8	64.5	65.2	65.3	15.5
image passage		2048x2048 JPEG	1	2	115.7	115.6	116.1	117.1	8.7
image passage		2048x2048 JPEG	1	3	173.0	174.5	175.0	175.8	5.8
image passage		2048x2048 JPEG	1	5	287.8	292.3	295.9	299.7	3.6
image passage		2048x2048 JPEG	1	8	455.1	469.1	471.8	490.2	2.4
image passage		2048x2048 JPEG	4	1	250.5	248.7	252.9	261.8	16.0
image passage		2048x2048 JPEG	4	2	467.3	466.4	473.3	481.7	8.6
image passage		2048x2048 JPEG	4	3	700.9	703.0	723.5	732.6	5.8
image passage		2048x2048 JPEG	4	5	1162.2	1174.6	1215.4	1221.7	3.6
image passage		2048x2048 JPEG	4	8	1825.3	1882.2	1902.1	1970.5	2.4
image passage		2048x2048 JPEG	8	1	498.5	495.8	510.4	514.5	16.1
image passage		2048x2048 JPEG	8	2	935.0	933.8	949.8	975.7	8.6
image passage		2048x2048 JPEG	8	3	1395.8	1402.2	1433.3	1467.3	5.8
image passage		2048x2048 JPEG	8	5	2307.1	2348.7	2368.9	2382.3	3.6
image passage		2048x2048 JPEG	8	8	3647.7	3759.5	3838.6	3852.8	2.4

NVIDIA-A100-SXM4-80GB

FP16

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	3.3	3.3	3.6	3.7	245.3
query	20		1	3	5.4	5.4	5.7	5.7	474.4
query	20		1	5	6.5	6.4	6.7	6.8	669.8
query	20		1	7	7.2	7.1	7.8	8.5	815.5
query	20		1	9	7.4	7.3	8.3	8.4	1026.8
query	20		1	11	10.4	10.7	11.0	11.1	917.6
query	20		1	13	8.7	8.8	9.3	10.4	1253.7
query	20		1	15	9.3	9.3	11.1	11.2	1352.2
passage	300		64	1	381.9	382.0	383.4	383.8	164.0
passage	300		64	3	1105.8	1116.4	1119.7	1120.8	170.6
passage	300		64	5	1832.3	1869.4	1873.3	1873.8	170.4
passage	512		64	1	838.4	837.7	846.2	848.1	75.5
passage	512		64	3	2463.4	2488.3	2500.3	2503.0	76.9
passage	512		64	5	4071.6	4153.5	4170.1	4175.1	76.9
image passage		336x336 JPEG	1	1	37.7	34.3	43.9	47.2	26.9
image passage		336x336 JPEG	1	2	69.9	66.9	79.1	82.2	14.5
image passage		336x336 JPEG	1	3	105.8	103.3	119.7	124.9	9.7
image passage		336x336 JPEG	1	5	172.0	172.1	191.0	195.3	6.0
image passage		336x336 JPEG	1	8	272.7	279.7	301.9	305.0	4.0
image passage		336x336 JPEG	4	1	143.7	140.2	158.7	162.4	28.0
image passage		336x336 JPEG	4	2	280.6	280.9	303.9	307.0	14.3
image passage		336x336 JPEG	4	3	420.6	422.8	439.0	443.9	9.6
image passage		336x336 JPEG	4	5	694.6	706.6	727.9	730.9	5.9
image passage		336x336 JPEG	4	8	1093.9	1128.7	1148.4	1150.3	3.9
image passage		336x336 JPEG	8	1	288.0	288.5	307.3	309.1	27.8
image passage		336x336 JPEG	8	2	562.2	564.9	574.3	584.1	14.3
image passage		336x336 JPEG	8	3	840.2	848.4	871.9	873.1	9.7
image passage		336x336 JPEG	8	5	1390.4	1419.2	1439.5	1447.7	6.0
image passage		336x336 JPEG	8	8	2184.5	2262.3	2278.7	2285.0	4.0
image passage		512x512 JPEG	1	1	37.3	34.5	42.1	43.7	27.1
image passage		512x512 JPEG	1	2	69.8	65.7	79.6	85.0	14.5
image passage		512x512 JPEG	1	3	105.0	101.2	118.4	123.9	9.8
image passage		512x512 JPEG	1	5	171.9	169.8	195.5	197.8	6.1
image passage		512x512 JPEG	1	8	276.9	283.3	309.8	316.2	3.9
image passage		512x512 JPEG	4	1	142.7	139.3	156.5	162.2	28.2
image passage		512x512 JPEG	4	2	281.0	284.0	302.7	306.0	14.3
image passage		512x512 JPEG	4	3	420.2	424.7	437.6	440.9	9.6
image passage		512x512 JPEG	4	5	693.8	701.3	728.8	731.7	6.0
image passage		512x512 JPEG	4	8	1093.0	1131.7	1140.0	1145.5	4.0
image passage		512x512 JPEG	8	1	290.0	291.5	312.0	314.0	27.7
image passage		512x512 JPEG	8	2	562.7	565.3	575.4	581.1	14.3
image passage		512x512 JPEG	8	3	839.6	850.2	866.6	868.1	9.7
image passage		512x512 JPEG	8	5	1386.4	1414.2	1437.8	1441.3	6.0
image passage		512x512 JPEG	8	8	2182.3	2261.9	2282.9	2290.4	4.0
image passage		1024x1024 JPEG	1	1	215.2	194.8	206.4	224.3	5.1
image passage		1024x1024 JPEG	1	2	376.7	376.2	394.7	399.2	2.7
image passage		1024x1024 JPEG	1	3	563.2	569.3	580.9	589.7	1.8
image passage		1024x1024 JPEG	1	5	932.8	950.6	967.0	973.3	1.1
image passage		1024x1024 JPEG	1	8	1472.2	1521.6	1548.8	1557.3	0.7
image passage		1024x1024 JPEG	4	1	764.6	765.3	784.2	788.2	5.2
image passage		1024x1024 JPEG	4	2	1512.8	1519.0	1546.4	1550.2	2.7
image passage		1024x1024 JPEG	4	3	2259.7	2280.7	2308.4	2313.8	1.8
image passage		1024x1024 JPEG	4	5	3735.3	3807.6	3836.4	3846.5	1.1
image passage		1024x1024 JPEG	4	8	5882.5	6088.6	6136.9	6145.3	0.7
image passage		1024x1024 JPEG	8	1	1532.2	1531.4	1548.9	1562.8	5.2
image passage		1024x1024 JPEG	8	2	3026.4	3044.2	3070.3	3076.5	2.7
image passage		1024x1024 JPEG	8	3	4518.2	4562.2	4588.5	4603.9	1.8
image passage		1024x1024 JPEG	8	5	7448.2	7595.0	7640.3	7649.3	1.1
image passage		1024x1024 JPEG	8	8	11746.7	12167.5	12202.4	12208.9	0.7
image passage		2048x2048 JPEG	1	1	190.9	192.7	202.4	205.9	5.2
image passage		2048x2048 JPEG	1	2	377.5	377.8	392.9	394.8	2.7
image passage		2048x2048 JPEG	1	3	561.5	563.8	578.8	593.8	1.8
image passage		2048x2048 JPEG	1	5	931.6	949.6	969.1	975.7	1.1
image passage		2048x2048 JPEG	1	8	1466.4	1521.4	1538.8	1540.7	0.8
image passage		2048x2048 JPEG	4	1	770.8	773.5	783.6	787.2	5.2
image passage		2048x2048 JPEG	4	2	1512.0	1520.9	1539.9	1544.3	2.7
image passage		2048x2048 JPEG	4	3	2261.7	2282.7	2305.3	2312.0	1.8
image passage		2048x2048 JPEG	4	5	3732.2	3804.0	3837.9	3845.6	1.1
image passage		2048x2048 JPEG	4	8	5895.7	6102.5	6145.4	6153.3	0.7
image passage		2048x2048 JPEG	8	1	1541.6	1541.3	1563.5	1570.8	5.2
image passage		2048x2048 JPEG	8	2	3017.7	3034.2	3061.5	3067.3	2.7
image passage		2048x2048 JPEG	8	3	4507.2	4550.4	4581.2	4589.9	1.8
image passage		2048x2048 JPEG	8	5	7445.9	7591.7	7622.3	7632.9	1.1
image passage		2048x2048 JPEG	8	8	11960.2	12184.1	12272.2	14723.2	0.7

NVIDIA-H100-NVL

FP8

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	2.4	2.4	2.5	2.5	322.8
query	20		1	3	4.2	4.3	5.2	8.4	616.4
query	20		1	5	5.1	4.5	5.5	6.7	856.9
query	20		1	7	6.8	5.3	13.9	15.2	899.9
query	20		1	9	7.1	5.4	12.2	17.6	1087.3
query	20		1	11	5.9	5.2	8.7	8.9	1514.1
query	20		1	13	7.2	5.9	15.6	16.0	1540.9
query	20		1	15	6.3	6.7	8.3	8.4	1958.3
passage	300		64	1	97.7	97.4	99.4	99.9	609.7
passage	300		64	3	272.8	275.5	277.3	278.9	677.8
passage	300		64	5	455.4	464.7	467.1	467.6	677.9
passage	512		64	1	168.1	168.0	169.3	169.7	364.9
passage	512		64	3	483.8	488.4	490.9	491.3	387.0
passage	512		64	5	804.7	820.9	825.2	825.9	386.1
image passage		336x336 JPEG	1	1	18.0	18.0	18.1	18.1	55.6
image passage		336x336 JPEG	1	2	29.6	29.7	29.8	29.8	33.9
image passage		336x336 JPEG	1	3	44.4	44.8	44.8	44.9	22.7
image passage		336x336 JPEG	1	5	74.2	75.5	75.6	76.2	13.9
image passage		336x336 JPEG	1	8	117.2	121.1	121.2	122.3	9.2
image passage		336x336 JPEG	4	1	64.8	64.7	64.9	65.0	61.8
image passage		336x336 JPEG	4	2	120.3	120.7	121.0	121.2	33.4
image passage		336x336 JPEG	4	3	180.2	181.9	182.1	182.2	22.5
image passage		336x336 JPEG	4	5	298.8	304.1	306.0	306.6	13.9
image passage		336x336 JPEG	4	8	511.4	488.3	505.2	981.6	8.9
image passage		336x336 JPEG	8	1	127.2	127.3	127.5	127.6	62.9
image passage		336x336 JPEG	8	2	242.3	243.5	243.7	243.7	33.2
image passage		336x336 JPEG	8	3	387.0	366.5	367.7	368.8	21.8
image passage		336x336 JPEG	8	5	639.7	612.6	615.5	707.0	13.5
image passage		336x336 JPEG	8	8	1010.8	982.1	985.0	1762.6	8.9
image passage		512x512 JPEG	1	1	18.2	18.2	18.3	18.3	55.0
image passage		512x512 JPEG	1	2	29.6	29.6	29.7	29.8	33.9
image passage		512x512 JPEG	1	3	44.4	44.8	44.8	44.9	22.7
image passage		512x512 JPEG	1	5	74.2	75.5	75.7	75.7	13.9
image passage		512x512 JPEG	1	8	117.7	121.6	121.9	122.6	9.2
image passage		512x512 JPEG	4	1	65.4	65.4	65.6	65.7	61.2
image passage		512x512 JPEG	4	2	135.9	120.8	121.3	123.4	32.6
image passage		512x512 JPEG	4	3	181.0	182.1	183.0	187.9	22.4
image passage		512x512 JPEG	4	5	299.2	304.8	306.1	306.2	13.9
image passage		512x512 JPEG	4	8	506.6	489.6	493.2	906.2	8.9
image passage		512x512 JPEG	8	1	128.7	128.6	129.0	129.4	62.2
image passage		512x512 JPEG	8	2	251.1	244.0	245.6	248.4	32.7
image passage		512x512 JPEG	8	3	364.0	367.0	367.7	371.4	22.3
image passage		512x512 JPEG	8	5	624.2	613.4	632.1	668.1	13.5
image passage		512x512 JPEG	8	8	950.1	983.4	987.0	994.3	9.2
image passage		1024x1024 JPEG	1	1	70.4	70.2	70.9	71.9	14.2
image passage		1024x1024 JPEG	1	2	139.9	131.5	132.6	134.7	7.5
image passage		1024x1024 JPEG	1	3	196.4	198.2	198.5	199.1	5.2
image passage		1024x1024 JPEG	1	5	324.9	331.1	332.3	333.0	3.2
image passage		1024x1024 JPEG	1	8	512.7	530.9	531.4	531.5	2.1
image passage		1024x1024 JPEG	4	1	281.1	272.8	273.5	274.7	14.5
image passage		1024x1024 JPEG	4	2	552.8	530.1	532.5	539.5	7.5
image passage		1024x1024 JPEG	4	3	827.0	796.5	806.4	1140.9	5.0
image passage		1024x1024 JPEG	4	5	1347.0	1329.7	1332.0	1382.4	3.1
image passage		1024x1024 JPEG	4	8	2122.7	2128.9	2134.3	2989.9	2.1
image passage		1024x1024 JPEG	8	1	563.1	542.8	544.8	546.7	14.5
image passage		1024x1024 JPEG	8	2	1088.2	1064.9	1077.0	1101.7	7.5
image passage		1024x1024 JPEG	8	3	1644.9	1599.6	1613.2	2350.2	5.0
image passage		1024x1024 JPEG	8	5	2722.2	2669.0	3152.7	3508.9	3.1
image passage		1024x1024 JPEG	8	8	4283.4	4263.1	5069.3	5133.6	2.1
image passage		2048x2048 JPEG	1	1	80.5	72.3	73.3	73.9	13.6
image passage		2048x2048 JPEG	1	2	131.5	131.8	133.4	134.0	7.6
image passage		2048x2048 JPEG	1	3	197.1	198.9	199.5	199.8	5.1
image passage		2048x2048 JPEG	1	5	391.4	333.5	682.2	762.1	3.0
image passage		2048x2048 JPEG	1	8	516.3	534.4	535.4	535.9	2.1
image passage		2048x2048 JPEG	4	1	292.6	280.0	281.9	283.1	14.1
image passage		2048x2048 JPEG	4	2	545.0	530.1	532.4	533.1	7.5
image passage		2048x2048 JPEG	4	3	818.1	799.3	802.3	815.1	5.0
image passage		2048x2048 JPEG	4	5	1352.5	1336.8	1342.1	1394.2	3.1
image passage		2048x2048 JPEG	4	8	2137.4	2144.8	2151.3	2986.1	2.1
image passage		2048x2048 JPEG	8	1	579.0	558.2	562.1	567.6	14.1
image passage		2048x2048 JPEG	8	2	1101.7	1063.5	1069.2	1484.1	7.4
image passage		2048x2048 JPEG	8	3	1650.1	1600.1	1661.8	2153.5	5.0
image passage		2048x2048 JPEG	8	5	2727.4	2676.4	3111.1	3512.9	3.1
image passage		2048x2048 JPEG	8	8	4316.8	4293.0	5152.2	5171.9	2.0

NVIDIA-H100-80GB-HBM3

FP8

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	2.5	2.4	2.5	2.5	316.9
query	20		1	3	4.6	3.6	5.4	14.6	551.8
query	20		1	5	6.1	4.5	11.6	14.2	718.2
query	20		1	7	7.4	6.3	14.6	16.8	839.0
query	20		1	9	5.1	5.3	5.6	7.1	1444.0
query	20		1	11	6.9	5.4	14.2	14.4	1358.0
query	20		1	13	6.5	5.5	13.2	14.7	1672.4
query	20		1	15	7.5	7.1	11.6	11.7	1693.4
passage	300		64	1	72.4	72.2	73.6	74.2	796.8
passage	300		64	3	195.1	196.7	198.6	199.9	936.1
passage	300		64	5	328.3	333.6	337.9	343.7	932.3
passage	512		64	1	123.4	123.5	124.7	124.7	487.2
passage	512		64	3	347.6	351.2	352.8	354.2	534.3
passage	512		64	5	578.9	590.3	594.2	595.2	534.3
image passage		336x336 JPEG	1	1	17.0	17.0	17.1	17.1	58.9
image passage		336x336 JPEG	1	2	28.2	28.2	28.3	28.3	35.6
image passage		336x336 JPEG	1	3	42.3	42.7	42.7	42.7	23.9
image passage		336x336 JPEG	1	5	70.3	71.6	71.9	72.4	14.7
image passage		336x336 JPEG	1	8	111.2	115.0	115.3	115.5	9.7
image passage		336x336 JPEG	4	1	61.4	61.2	61.4	63.6	65.2
image passage		336x336 JPEG	4	2	114.4	114.5	116.2	118.5	35.1
image passage		336x336 JPEG	4	3	171.9	172.4	175.2	176.9	23.6
image passage		336x336 JPEG	4	5	283.4	288.2	293.0	294.7	14.7
image passage		336x336 JPEG	4	8	449.6	462.4	468.4	491.4	9.7
image passage		336x336 JPEG	8	1	120.8	120.3	121.0	124.0	66.2
image passage		336x336 JPEG	8	2	229.5	229.8	233.8	235.4	35.0
image passage		336x336 JPEG	8	3	343.5	345.4	349.5	351.6	23.6
image passage		336x336 JPEG	8	5	567.0	577.4	581.6	583.7	14.7
image passage		336x336 JPEG	8	8	896.8	924.4	931.6	962.6	9.8
image passage		512x512 JPEG	1	1	17.2	17.1	17.3	17.3	58.2
image passage		512x512 JPEG	1	2	28.2	28.2	28.3	28.3	35.5
image passage		512x512 JPEG	1	3	42.8	42.8	42.9	44.9	23.6
image passage		512x512 JPEG	1	5	70.6	71.8	71.9	74.7	14.6
image passage		512x512 JPEG	1	8	112.0	115.8	115.9	116.0	9.6
image passage		512x512 JPEG	4	1	62.6	62.1	62.5	66.6	64.0
image passage		512x512 JPEG	4	2	114.7	114.8	115.2	117.9	35.0
image passage		512x512 JPEG	4	3	171.3	172.7	173.2	174.0	23.6
image passage		512x512 JPEG	4	5	284.0	288.7	292.8	294.8	14.6
image passage		512x512 JPEG	4	8	447.5	462.7	464.2	468.5	9.8
image passage		512x512 JPEG	8	1	122.2	121.9	123.1	124.7	65.5
image passage		512x512 JPEG	8	2	229.8	230.1	233.8	234.9	35.0
image passage		512x512 JPEG	8	3	343.7	345.9	349.6	352.7	23.6
image passage		512x512 JPEG	8	5	568.2	578.1	582.0	585.9	14.6
image passage		512x512 JPEG	8	8	898.7	927.7	933.9	940.4	9.7
image passage		1024x1024 JPEG	1	1	63.8	63.7	64.2	64.3	15.7
image passage		1024x1024 JPEG	1	2	119.0	119.4	119.7	119.9	8.4
image passage		1024x1024 JPEG	1	3	178.1	179.6	180.2	180.6	5.7
image passage		1024x1024 JPEG	1	5	295.6	301.3	302.0	304.6	3.5
image passage		1024x1024 JPEG	1	8	483.0	481.1	496.3	687.1	2.3
image passage		1024x1024 JPEG	4	1	251.3	248.1	250.4	252.7	16.0
image passage		1024x1024 JPEG	4	2	483.4	479.6	482.4	484.3	8.3
image passage		1024x1024 JPEG	4	3	720.2	720.1	738.3	786.4	5.6
image passage		1024x1024 JPEG	4	5	1178.8	1201.5	1205.6	1206.6	3.5
image passage		1024x1024 JPEG	4	8	1861.6	1926.5	1937.7	1939.9	2.3
image passage		1024x1024 JPEG	8	1	494.1	493.7	495.4	498.2	16.2
image passage		1024x1024 JPEG	8	2	956.0	958.9	966.6	969.6	8.4
image passage		1024x1024 JPEG	8	3	1430.4	1442.1	1452.7	1457.8	5.7
image passage		1024x1024 JPEG	8	5	2359.9	2404.8	2414.0	2420.4	3.5
image passage		1024x1024 JPEG	8	8	3724.2	3854.9	3870.6	3872.7	2.4
image passage		2048x2048 JPEG	1	1	66.9	66.8	68.5	68.6	15.0
image passage		2048x2048 JPEG	1	2	119.7	120.3	120.5	120.6	8.4
image passage		2048x2048 JPEG	1	3	180.0	180.6	186.5	188.7	5.6
image passage		2048x2048 JPEG	1	5	296.4	302.0	303.9	304.2	3.5
image passage		2048x2048 JPEG	1	8	467.5	484.0	484.6	484.7	2.3
image passage		2048x2048 JPEG	4	1	257.8	256.9	259.5	261.4	15.5
image passage		2048x2048 JPEG	4	2	479.7	480.6	484.0	485.3	8.4
image passage		2048x2048 JPEG	4	3	718.2	723.3	730.0	732.6	5.6
image passage		2048x2048 JPEG	4	5	1197.2	1209.2	1230.5	1250.0	3.5
image passage		2048x2048 JPEG	4	8	1875.2	1939.4	1944.1	1972.7	2.3
image passage		2048x2048 JPEG	8	1	512.8	512.4	516.1	517.9	15.6
image passage		2048x2048 JPEG	8	2	976.2	965.6	995.9	1042.0	8.3
image passage		2048x2048 JPEG	8	3	1435.7	1448.2	1452.3	1457.4	5.6
image passage		2048x2048 JPEG	8	5	2388.4	2420.7	2455.6	2491.0	3.5
image passage		2048x2048 JPEG	8	8	3762.5	3883.6	3927.7	3985.5	2.3

NVIDIA-L4

FP16

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	10.0	10.0	10.1	10.1	92.6
query	20		1	3	18.5	18.5	18.7	18.7	153.4
query	20		1	5	19.9	19.7	19.8	22.0	236.8
query	20		1	7	28.6	29.5	29.7	29.7	227.6
query	20		1	9	22.1	22.6	23.3	23.5	367.8
query	20		1	11	23.2	23.3	24.2	24.4	433.7
query	20		1	13	31.4	33.1	33.6	33.7	379.4
query	20		1	15	32.8	34.2	34.7	35.1	406.8
passage	300		64	1	1104.3	1105.0	1114.6	1116.9	57.5
passage	300		64	3	3271.2	3303.0	3323.7	3332.1	58.0
passage	300		64	5	5406.1	5515.4	5546.1	5552.9	57.9
passage	512		64	1	2188.5	2186.1	2213.1	2221.7	29.1
passage	512		64	3	6509.7	6573.1	6608.2	6616.6	29.2
passage	512		64	5	10746.7	10966.7	11010.6	11025.5	29.2
image passage		336x336 JPEG	1	1	87.0	87.0	89.3	89.9	11.5
image passage		336x336 JPEG	1	2	169.2	169.9	172.6	173.3	5.9
image passage		336x336 JPEG	1	3	253.9	256.4	259.0	259.3	4.0
image passage		336x336 JPEG	1	5	419.6	428.1	431.5	432.1	2.5
image passage		336x336 JPEG	1	8	662.1	685.8	689.2	690.9	1.7
image passage		336x336 JPEG	4	1	346.3	346.2	349.7	350.5	11.6
image passage		336x336 JPEG	4	2	685.8	689.1	692.3	693.2	5.9
image passage		336x336 JPEG	4	3	1026.6	1037.2	1040.7	1041.5	4.0
image passage		336x336 JPEG	4	5	1696.5	1731.2	1735.5	1736.8	2.5
image passage		336x336 JPEG	4	8	2674.5	2771.0	2776.2	2777.0	1.6
image passage		336x336 JPEG	8	1	696.4	696.0	699.9	701.0	11.5
image passage		336x336 JPEG	8	2	1378.7	1385.2	1390.5	1391.7	5.8
image passage		336x336 JPEG	8	3	2060.2	2080.3	2086.4	2088.0	3.9
image passage		336x336 JPEG	8	5	3400.5	3469.7	3477.2	3478.7	2.5
image passage		336x336 JPEG	8	8	5358.8	5552.8	5560.9	5563.9	1.6
image passage		512x512 JPEG	1	1	87.8	87.9	89.5	89.8	11.4
image passage		512x512 JPEG	1	2	170.4	171.3	173.8	174.4	5.9
image passage		512x512 JPEG	1	3	254.5	256.9	259.7	260.3	4.0
image passage		512x512 JPEG	1	5	421.5	429.7	432.5	434.1	2.5
image passage		512x512 JPEG	1	8	663.6	687.3	690.3	691.5	1.7
image passage		512x512 JPEG	4	1	347.5	347.1	350.8	351.6	11.5
image passage		512x512 JPEG	4	2	685.9	689.4	692.6	693.4	5.9
image passage		512x512 JPEG	4	3	1027.0	1037.1	1041.0	1042.9	4.0
image passage		512x512 JPEG	4	5	1694.8	1729.1	1733.5	1734.7	2.5
image passage		512x512 JPEG	4	8	2672.8	2769.4	2773.3	2778.1	1.6
image passage		512x512 JPEG	8	1	696.5	696.7	699.6	701.1	11.5
image passage		512x512 JPEG	8	2	1379.2	1386.1	1391.2	1391.6	5.8
image passage		512x512 JPEG	8	3	2058.7	2079.3	2083.8	2086.6	3.9
image passage		512x512 JPEG	8	5	3399.9	3468.4	3476.6	3480.0	2.5
image passage		512x512 JPEG	8	8	5357.3	5550.7	5561.2	5566.0	1.6
image passage		1024x1024 JPEG	1	1	465.9	462.6	476.9	480.6	2.1
image passage		1024x1024 JPEG	1	2	921.7	920.3	946.6	951.2	1.1
image passage		1024x1024 JPEG	1	3	1378.9	1390.7	1412.4	1431.6	0.7
image passage		1024x1024 JPEG	1	5	2267.7	2313.3	2328.9	2332.3	0.5
image passage		1024x1024 JPEG	1	8	3580.6	3704.8	3734.6	3748.2	0.3
image passage		1024x1024 JPEG	4	1	1864.2	1864.9	1877.4	1882.0	2.1
image passage		1024x1024 JPEG	4	2	3695.1	3708.9	3735.8	3746.2	1.1
image passage		1024x1024 JPEG	4	3	5509.4	5566.5	5583.7	5585.1	0.7
image passage		1024x1024 JPEG	4	5	9098.7	9280.1	9306.9	9323.1	0.5
image passage		1024x1024 JPEG	4	8	14327.0	14843.6	14886.3	14897.9	0.3
image passage		1024x1024 JPEG	8	1	3726.8	3723.1	3748.4	3757.9	2.1
image passage		1024x1024 JPEG	8	2	7395.5	7428.5	7459.6	7483.6	1.1
image passage		1024x1024 JPEG	8	3	11044.6	11154.7	11192.7	11200.7	0.7
image passage		1024x1024 JPEG	8	5	18205.8	18577.5	18614.3	18618.1	0.5
image passage		1024x1024 JPEG	8	8	28702.8	29714.8	29850.5	29966.5	0.3
image passage		2048x2048 JPEG	1	1	468.4	464.4	481.6	483.0	2.1
image passage		2048x2048 JPEG	1	2	920.0	920.0	941.3	946.8	1.1
image passage		2048x2048 JPEG	1	3	1373.0	1384.2	1407.0	1409.6	0.7
image passage		2048x2048 JPEG	1	5	2268.4	2313.8	2326.4	2327.6	0.5
image passage		2048x2048 JPEG	1	8	3576.4	3703.5	3732.0	3740.5	0.3
image passage		2048x2048 JPEG	4	1	1863.3	1862.4	1879.9	1883.3	2.1
image passage		2048x2048 JPEG	4	2	3695.4	3711.7	3735.0	3739.0	1.1
image passage		2048x2048 JPEG	4	3	5510.1	5567.9	5582.6	5586.7	0.7
image passage		2048x2048 JPEG	4	5	9087.8	9270.3	9300.1	9312.2	0.5
image passage		2048x2048 JPEG	4	8	14332.3	14851.5	14876.4	14885.6	0.3
image passage		2048x2048 JPEG	8	1	3731.9	3729.7	3751.4	3759.3	2.1
image passage		2048x2048 JPEG	8	2	7393.5	7427.6	7460.4	7477.7	1.1
image passage		2048x2048 JPEG	8	3	11035.5	11149.6	11177.5	11183.9	0.7
image passage		2048x2048 JPEG	8	5	18226.0	18573.6	18637.3	18684.3	0.5
image passage		2048x2048 JPEG	8	8	28696.2	29727.8	29790.0	29803.4	0.3

NVIDIA-L40S

FP16

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	4.2	4.2	4.2	4.2	207.2
query	20		1	3	6.6	6.9	7.0	7.0	403.9
query	20		1	5	7.1	7.1	7.3	7.3	624.6
query	20		1	7	7.3	7.3	7.5	7.5	843.0
query	20		1	9	10.8	11.2	11.3	11.4	729.9
query	20		1	11	7.9	8.0	8.2	9.2	1186.2
query	20		1	13	8.2	8.3	8.6	9.7	1347.5
query	20		1	15	8.4	8.4	10.0	10.1	1502.9
passage	300		64	1	336.2	336.3	337.5	337.8	185.7
passage	300		64	3	977.5	987.6	989.9	990.3	192.8
passage	300		64	5	1618.4	1652.1	1654.9	1655.9	192.8
passage	512		64	1	647.5	648.6	652.7	657.9	97.6
passage	512		64	3	1938.7	1958.4	1974.7	1979.3	97.6
passage	512		64	5	3233.8	3301.3	3313.4	3316.3	96.7
image passage		336x336 JPEG	1	1	28.9	28.5	30.2	32.3	34.6
image passage		336x336 JPEG	1	2	52.5	51.9	55.1	58.0	19.1
image passage		336x336 JPEG	1	3	78.9	78.4	84.4	86.2	12.8
image passage		336x336 JPEG	1	5	130.6	131.1	139.0	140.0	8.0
image passage		336x336 JPEG	1	8	206.4	210.2	219.1	219.7	5.3
image passage		336x336 JPEG	4	1	110.4	108.8	116.3	117.4	36.3
image passage		336x336 JPEG	4	2	212.9	211.4	219.8	220.6	18.9
image passage		336x336 JPEG	4	3	319.1	323.5	326.8	327.3	12.7
image passage		336x336 JPEG	4	5	527.5	537.9	540.4	540.9	7.9
image passage		336x336 JPEG	4	8	833.2	862.1	868.9	869.3	5.3
image passage		336x336 JPEG	8	1	220.3	218.4	225.7	226.4	36.3
image passage		336x336 JPEG	8	2	430.3	433.2	435.7	436.4	18.7
image passage		336x336 JPEG	8	3	643.7	649.7	656.6	658.0	12.6
image passage		336x336 JPEG	8	5	1064.9	1086.6	1089.3	1091.0	7.8
image passage		336x336 JPEG	8	8	1683.4	1742.4	1750.7	1751.6	5.2
image passage		512x512 JPEG	1	1	29.4	29.0	30.4	32.4	34.1
image passage		512x512 JPEG	1	2	53.1	52.5	55.6	59.0	18.9
image passage		512x512 JPEG	1	3	79.6	79.1	85.2	87.2	12.7
image passage		512x512 JPEG	1	5	131.7	132.2	140.7	142.6	7.9
image passage		512x512 JPEG	1	8	207.7	211.8	221.5	221.9	5.3
image passage		512x512 JPEG	4	1	111.5	109.9	117.5	118.4	35.9
image passage		512x512 JPEG	4	2	213.8	212.4	220.7	221.4	18.8
image passage		512x512 JPEG	4	3	319.8	323.6	327.6	328.0	12.7
image passage		512x512 JPEG	4	5	529.1	539.6	543.6	544.4	7.9
image passage		512x512 JPEG	4	8	835.2	864.7	870.5	872.0	5.2
image passage		512x512 JPEG	8	1	221.7	219.7	227.6	227.9	36.1
image passage		512x512 JPEG	8	2	429.9	432.7	435.5	436.2	18.7
image passage		512x512 JPEG	8	3	644.6	649.8	657.1	659.3	12.6
image passage		512x512 JPEG	8	5	1066.3	1088.1	1092.0	1093.3	7.8
image passage		512x512 JPEG	8	8	1679.4	1739.0	1747.9	1749.4	5.2
image passage		1024x1024 JPEG	1	1	149.5	147.3	157.5	158.5	6.7
image passage		1024x1024 JPEG	1	2	288.9	287.1	300.6	301.9	3.5
image passage		1024x1024 JPEG	1	3	432.0	431.3	448.2	451.8	2.3
image passage		1024x1024 JPEG	1	5	713.0	725.5	737.6	742.5	1.5
image passage		1024x1024 JPEG	1	8	1122.4	1164.3	1171.6	1171.8	1.0
image passage		1024x1024 JPEG	4	1	590.6	584.6	604.3	605.9	6.8
image passage		1024x1024 JPEG	4	2	1165.0	1172.2	1181.3	1182.8	3.4
image passage		1024x1024 JPEG	4	3	1739.6	1756.2	1768.0	1771.9	2.3
image passage		1024x1024 JPEG	4	5	2870.5	2929.2	2938.8	2946.4	1.5
image passage		1024x1024 JPEG	4	8	4524.7	4687.8	4701.4	4702.9	1.0
image passage		1024x1024 JPEG	8	1	1185.2	1186.0	1194.4	1196.2	6.8
image passage		1024x1024 JPEG	8	2	2335.2	2348.2	2359.3	2361.2	3.4
image passage		1024x1024 JPEG	8	3	3478.4	3511.3	3529.7	3533.6	2.3
image passage		1024x1024 JPEG	8	5	5755.0	5872.1	5887.7	5897.0	1.4
image passage		1024x1024 JPEG	8	8	9119.4	9396.7	9429.2	10020.8	1.0
image passage		2048x2048 JPEG	1	1	151.1	150.1	159.2	161.0	6.6
image passage		2048x2048 JPEG	1	2	289.5	287.5	301.9	306.0	3.5
image passage		2048x2048 JPEG	1	3	431.6	431.6	445.9	449.0	2.4
image passage		2048x2048 JPEG	1	5	712.0	725.5	739.4	740.7	1.5
image passage		2048x2048 JPEG	1	8	1123.8	1163.0	1173.5	1174.9	1.0
image passage		2048x2048 JPEG	4	1	597.8	596.1	611.9	613.6	6.7
image passage		2048x2048 JPEG	4	2	1159.1	1166.7	1174.1	1175.9	3.5
image passage		2048x2048 JPEG	4	3	1732.7	1749.6	1762.3	1766.1	2.3
image passage		2048x2048 JPEG	4	5	2861.1	2915.5	2935.8	2939.2	1.5
image passage		2048x2048 JPEG	4	8	4507.9	4671.0	4692.5	4694.9	1.0
image passage		2048x2048 JPEG	8	1	1197.7	1198.8	1208.7	1211.2	6.7
image passage		2048x2048 JPEG	8	2	2320.4	2332.1	2350.3	2354.3	3.5
image passage		2048x2048 JPEG	8	3	3467.4	3501.4	3525.5	3530.1	2.3
image passage		2048x2048 JPEG	8	5	5734.6	5854.6	5876.2	5879.3	1.5
image passage		2048x2048 JPEG	8	8	9030.5	9361.0	9400.7	9411.1	1.0

NVIDIA-A10G

FP16

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	6.3	6.2	6.3	6.3	143.7
query	20		1	3	10.4	11.0	11.2	11.2	263.4
query	20		1	5	11.7	11.8	12.0	12.0	384.0
query	20		1	7	12.9	13.0	13.2	13.3	488.6
query	20		1	9	14.4	14.5	14.9	16.0	559.7
query	20		1	11	20.9	21.3	21.6	22.7	478.1
query	20		1	13	20.9	21.2	22.9	23.1	556.9
query	20		1	15	21.1	21.4	25.6	25.7	635.1
passage	300		64	1	1171.3	1171.2	1178.0	1180.2	54.3
passage	300		64	3	3463.7	3498.8	3507.1	3508.2	54.8
passage	300		64	5	5720.4	5839.0	5849.4	5855.6	54.7
passage	512		64	1	2137.1	2140.7	2180.9	2208.9	29.8
passage	512		64	3	6409.9	6474.2	6517.6	6531.5	29.6
passage	512		64	5	10634.0	10818.2	10997.6	11028.1	29.5
image passage		336x336 JPEG	1	1	77.0	77.0	79.0	80.1	13.0
image passage		336x336 JPEG	1	2	151.4	151.7	156.1	157.6	6.6
image passage		336x336 JPEG	1	3	228.3	230.0	237.5	239.1	4.4
image passage		336x336 JPEG	1	5	376.9	386.0	388.8	389.7	2.8
image passage		336x336 JPEG	1	8	595.3	618.2	629.1	630.1	1.8
image passage		336x336 JPEG	4	1	318.2	317.4	326.1	326.7	12.6
image passage		336x336 JPEG	4	2	631.7	635.6	643.5	648.3	6.4
image passage		336x336 JPEG	4	3	948.5	955.9	973.0	975.1	4.3
image passage		336x336 JPEG	4	5	1565.0	1595.8	1607.9	1615.5	2.7
image passage		336x336 JPEG	4	8	2472.0	2555.7	2582.2	2597.4	1.8
image passage		336x336 JPEG	8	1	645.4	645.9	650.6	653.4	12.4
image passage		336x336 JPEG	8	2	1278.3	1285.2	1294.7	1299.9	6.3
image passage		336x336 JPEG	8	3	1911.8	1931.6	1945.2	1947.8	4.2
image passage		336x336 JPEG	8	5	3154.5	3216.9	3238.7	3247.3	2.6
image passage		336x336 JPEG	8	8	4966.3	5147.1	5178.2	5184.0	1.8
image passage		512x512 JPEG	1	1	81.5	80.2	85.9	91.9	12.3
image passage		512x512 JPEG	1	2	155.5	155.7	160.8	162.6	6.5
image passage		512x512 JPEG	1	3	232.0	233.8	239.1	240.8	4.4
image passage		512x512 JPEG	1	5	384.1	390.8	399.5	401.3	2.7
image passage		512x512 JPEG	1	8	601.9	623.0	638.8	640.3	1.8
image passage		512x512 JPEG	4	1	319.9	319.2	327.5	329.1	12.5
image passage		512x512 JPEG	4	2	633.1	636.7	640.9	643.6	6.4
image passage		512x512 JPEG	4	3	947.7	957.4	966.6	969.1	4.3
image passage		512x512 JPEG	4	5	1565.0	1596.5	1611.9	1614.5	2.7
image passage		512x512 JPEG	4	8	2466.5	2556.6	2575.5	2578.4	1.8
image passage		512x512 JPEG	8	1	645.4	645.1	654.4	656.8	12.4
image passage		512x512 JPEG	8	2	1278.3	1283.7	1298.9	1301.8	6.3
image passage		512x512 JPEG	8	3	1907.1	1928.4	1943.2	1947.5	4.3
image passage		512x512 JPEG	8	5	3149.6	3214.2	3238.3	3242.0	2.7
image passage		512x512 JPEG	8	8	4970.0	5153.5	5174.4	5180.6	1.8
image passage		1024x1024 JPEG	1	1	418.3	416.6	428.1	433.1	2.4
image passage		1024x1024 JPEG	1	2	824.2	826.9	839.9	846.3	1.2
image passage		1024x1024 JPEG	1	3	1232.5	1241.7	1261.7	1266.2	0.8
image passage		1024x1024 JPEG	1	5	2031.5	2071.0	2092.5	2102.3	0.5
image passage		1024x1024 JPEG	1	8	3200.8	3317.1	3341.4	3346.4	0.3
image passage		1024x1024 JPEG	4	1	1673.2	1672.9	1687.6	1691.2	2.4
image passage		1024x1024 JPEG	4	2	3326.9	3339.5	3356.2	3364.5	1.2
image passage		1024x1024 JPEG	4	3	4954.8	5004.8	5027.8	5031.4	0.8
image passage		1024x1024 JPEG	4	5	8175.0	8343.3	8364.3	8368.3	0.5
image passage		1024x1024 JPEG	4	8	12880.9	13353.1	13379.7	13387.6	0.3
image passage		1024x1024 JPEG	8	1	3349.5	3350.6	3366.1	3369.4	2.4
image passage		1024x1024 JPEG	8	2	6645.6	6679.9	6702.3	6706.0	1.2
image passage		1024x1024 JPEG	8	3	9926.7	10025.7	10060.1	10066.6	0.8
image passage		1024x1024 JPEG	8	5	16384.2	16720.9	16761.6	16774.9	0.5
image passage		1024x1024 JPEG	8	8	25793.0	26726.9	26766.8	26785.6	0.3
image passage		2048x2048 JPEG	1	1	418.1	416.2	427.3	434.6	2.4
image passage		2048x2048 JPEG	1	2	821.4	822.1	841.6	848.2	1.2
image passage		2048x2048 JPEG	1	3	1225.7	1236.4	1256.1	1260.2	0.8
image passage		2048x2048 JPEG	1	5	2023.9	2064.9	2088.0	2091.2	0.5
image passage		2048x2048 JPEG	1	8	3185.8	3304.6	3329.7	3336.3	0.3
image passage		2048x2048 JPEG	4	1	1675.1	1676.4	1696.7	1702.0	2.4
image passage		2048x2048 JPEG	4	2	3312.6	3332.9	3352.7	3357.7	1.2
image passage		2048x2048 JPEG	4	3	4949.6	5001.9	5028.8	5037.4	0.8
image passage		2048x2048 JPEG	4	5	8157.4	8330.8	8358.3	8368.9	0.5
image passage		2048x2048 JPEG	4	8	12861.0	13340.3	13401.5	13411.9	0.3
image passage		2048x2048 JPEG	8	1	3359.5	3360.5	3379.8	3385.4	2.4
image passage		2048x2048 JPEG	8	2	6645.9	6681.6	6705.8	6710.8	1.2
image passage		2048x2048 JPEG	8	3	9929.8	10031.0	10060.4	10068.0	0.8
image passage		2048x2048 JPEG	8	5	16388.4	16735.1	16765.1	16774.6	0.5
image passage		2048x2048 JPEG	8	8	25765.3	26693.5	26773.7	26778.5	0.3

Llama Nemotron Embed 300m

H100-HBM3-80GB

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	7.8	8.0	8.0	8.3	126.0
passage	20	1	3	8.3	8.0	9.0	8.8	348.0
passage	20	1	5	8.8	8.0	9.0	10.3	547.0
passage	20	1	7	10.8	11.0	12.0	11.9	585.0
passage	20	1	9	13.2	13.0	15.0	15.1	643.0
passage	20	1	11	15.5	16.0	18.0	18.4	652.0
passage	20	1	13	17.5	17.0	21.0	21.5	673.0
passage	20	1	15	20.6	21.0	24.0	24.0	662.0
passage	300	64	1	69.5	69.0	71.0	72.3	896.0
passage	300	64	3	103.5	104.0	107.0	108.3	1779.0
passage	300	64	5	172.3	173.0	178.0	179.9	1799.0
passage	512	64	1	103.6	101.0	116.0	120.1	606.0
passage	512	64	3	180.6	182.0	192.0	193.6	1040.0
passage	512	64	5	298.6	300.0	318.0	325.1	1041.0

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	8.0	8.0	9.0	8.9	122.0
passage	20	1	3	8.6	8.0	9.0	9.2	340.0
passage	20	1	5	8.8	9.0	9.0	9.2	548.0
passage	20	1	7	10.9	11.0	12.0	12.6	575.0
passage	20	1	9	13.3	14.0	15.0	15.2	638.0
passage	20	1	11	15.4	16.0	17.0	17.9	668.0
passage	20	1	13	17.6	17.0	21.0	21.3	688.0
passage	20	1	15	21.2	22.0	24.0	24.9	645.0
passage	300	64	1	70.9	71.0	73.0	74.3	880.0
passage	300	64	3	112.8	113.0	117.0	118.4	1652.0
passage	300	64	5	187.7	189.0	194.0	194.9	1653.0
passage	512	64	1	109.5	108.0	128.0	128.9	574.0
passage	512	64	3	202.5	203.0	216.0	219.4	919.0
passage	512	64	5	336.4	338.0	352.0	356.9	926.0

H100-NVL

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	8.2	8.0	9.0	8.6	120.0
passage	20	1	3	8.5	8.0	9.0	9.2	345.0
passage	20	1	5	9.4	9.0	10.0	10.8	508.0
passage	20	1	7	11.4	11.0	13.0	13.4	576.0
passage	20	1	9	14.4	15.0	17.0	16.9	584.0
passage	20	1	11	16.6	16.0	20.0	20.2	602.0
passage	20	1	13	20.2	21.0	24.0	24.2	590.0
passage	20	1	15	22.6	23.0	26.0	26.7	600.0
passage	300	64	1	68.8	68.0	72.0	73.1	907.0
passage	300	64	3	119.3	120.0	125.0	126.4	1552.0
passage	300	64	5	199.8	202.0	208.0	210.3	1555.0
passage	512	64	1	112.7	112.0	125.0	131.9	559.0
passage	512	64	3	229.5	232.0	242.0	246.6	814.0
passage	512	64	5	383.5	391.0	401.0	407.0	813.0

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	8.2	8.0	8.0	8.7	120.0
passage	20	1	3	8.3	8.0	9.0	8.8	339.0
passage	20	1	5	8.7	9.0	9.0	9.3	535.0
passage	20	1	7	10.7	11.0	12.0	11.8	597.0
passage	20	1	9	13.2	13.0	15.0	15.2	642.0
passage	20	1	11	15.5	16.0	18.0	18.3	654.0
passage	20	1	13	18.4	19.0	21.0	21.4	650.0
passage	20	1	15	20.8	21.0	24.0	24.5	668.0
passage	300	64	1	78.7	79.0	82.0	82.8	795.0
passage	300	64	3	149.7	151.0	156.0	157.8	1250.0
passage	300	64	5	249.8	254.0	260.0	261.9	1246.0
passage	512	64	1	129.6	127.0	142.0	150.2	487.0
passage	512	64	3	283.1	286.0	299.0	302.8	660.0
passage	512	64	5	474.0	482.0	499.0	502.4	658.0

L40S

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	8.3	8.0	8.0	8.7	118.0
passage	20	1	3	9.0	9.0	9.0	9.5	320.0
passage	20	1	5	9.9	10.0	11.0	11.4	476.0
passage	20	1	7	11.5	12.0	14.0	13.7	573.0
passage	20	1	9	14.4	14.0	16.0	16.5	583.0
passage	20	1	11	17.8	18.0	21.0	21.7	518.0
passage	20	1	13	20.2	20.0	24.0	23.7	582.0
passage	20	1	15	20.5	21.0	24.0	24.7	664.0
passage	300	64	1	97.3	97.0	99.0	100.2	646.0
passage	300	64	3	200.0	202.0	204.0	204.9	940.0
passage	300	64	5	332.7	336.0	340.0	340.5	938.0
passage	512	64	1	199.9	189.0	261.0	268.1	317.0
passage	512	64	3	415.9	419.0	427.0	428.2	455.0
passage	512	64	5	690.1	701.0	711.0	712.8	453.0

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	8.6	9.0	9.0	9.1	114.0
passage	20	1	3	9.3	9.0	10.0	10.2	312.0
passage	20	1	5	10.5	10.0	11.0	11.2	459.0
passage	20	1	7	14.4	14.0	15.0	21.8	453.0
passage	20	1	9	17.6	18.0	20.0	20.2	476.0
passage	20	1	11	20.2	21.0	25.0	24.9	499.0
passage	20	1	13	24.5	24.0	27.0	28.4	480.0
passage	20	1	15	28.7	30.0	33.0	33.3	471.0
passage	300	64	1	124.1	124.0	126.0	127.2	508.0
passage	300	64	3	273.1	275.0	278.0	279.2	692.0
passage	300	64	5	451.2	458.0	463.0	463.7	692.0
passage	512	64	1	239.6	230.0	297.0	301.0	265.0
passage	512	64	3	529.9	534.0	541.0	542.7	357.0
passage	512	64	5	876.4	892.0	900.0	901.8	357.0

A100-SXM4-80GB

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	5.5	5.0	7.0	7.0	180.0
passage	20	1	3	7.0	7.0	7.0	7.1	421.0
passage	20	1	5	10.5	11.0	12.0	11.8	455.0
passage	20	1	7	12.6	12.0	16.0	16.4	529.0
passage	20	1	9	18.5	20.0	21.0	21.3	466.0
passage	20	1	11	19.6	19.0	23.0	25.2	524.0
passage	20	1	13	24.6	25.0	28.0	30.1	481.0
passage	20	1	15	26.4	28.0	30.0	30.3	509.0
passage	300	64	1	115.7	115.0	120.0	120.4	544.0
passage	300	64	3	229.7	231.0	236.0	239.2	817.0
passage	300	64	5	381.6	385.0	393.0	397.6	817.0
passage	512	64	1	199.6	195.0	237.0	242.4	317.0
passage	512	64	3	431.6	433.0	447.0	448.9	438.0
passage	512	64	5	715.0	724.0	740.0	746.6	437.0

A10G

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	9.6	9.0	10.0	11.0	103.0
passage	20	1	3	12.6	13.0	14.0	14.2	232.0
passage	20	1	5	20.6	22.0	23.0	23.1	237.0
passage	20	1	7	28.3	31.0	32.0	31.7	234.0
passage	20	1	9	35.1	36.0	41.0	41.8	244.0
passage	20	1	11	43.3	45.0	50.0	50.4	236.0
passage	20	1	13	52.6	54.0	59.0	58.8	234.0
passage	20	1	15	58.0	60.0	68.0	68.7	234.0
passage	300	64	1	304.9	305.0	309.0	310.3	209.0
passage	300	64	3	780.6	791.0	799.0	800.0	241.0
passage	300	64	5	1296.8	1320.0	1330.0	1331.8	241.0
passage	512	64	1	520.7	519.0	533.0	538.6	122.0
passage	512	64	3	1385.5	1404.0	1424.0	1428.4	137.0
passage	512	64	5	2294.3	2341.0	2362.0	2368.3	137.0

L4

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	8.8	8.0	10.0	10.0	113.0
passage	20	1	3	11.1	11.0	12.0	11.8	259.0
passage	20	1	5	16.4	16.0	19.0	18.9	295.0
passage	20	1	7	23.3	22.0	26.0	26.8	290.0
passage	20	1	9	29.5	30.0	33.0	33.9	289.0
passage	20	1	11	34.2	34.0	40.0	40.6	302.0
passage	20	1	13	41.6	44.0	47.0	48.1	288.0
passage	20	1	15	45.7	48.0	55.0	55.2	300.0
passage	300	64	1	339.6	339.0	343.0	345.5	187.0
passage	300	64	3	918.3	927.0	933.0	934.6	207.0
passage	300	64	5	1517.3	1547.0	1554.0	1555.5	206.0
passage	512	64	1	642.3	643.0	657.0	661.9	99.0
passage	512	64	3	1795.3	1818.0	1835.0	1841.2	105.0
passage	512	64	5	2976.6	3034.0	3066.0	3075.5	105.0

Llama Nemotron Embed 1B

H100-HBM3-80GB

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	7.0	7.0	8.0	8.0	140.7
query	20	1	3	10.0	10.0	10.0	10.0	291.7
query	20	1	5	16.0	17.0	17.0	17.0	307.0
query	20	1	7	22.0	23.0	24.0	24.0	310.9
query	20	1	9	25.0	26.0	31.0	31.0	329.7
query	20	1	11	30.0	32.0	38.0	38.0	332.8
query	20	1	13	36.0	37.0	43.0	44.0	340.0
query	20	1	15	42.0	43.0	48.0	50.0	327.5
passage	300	64	1	159.0	159.0	162.0	163.0	401.8
passage	300	64	3	267.0	269.0	275.0	277.0	709.4
passage	300	64	5	392.0	390.0	401.0	411.0	814.5
passage	512	64	1	222.0	218.0	233.0	235.0	286.3
passage	512	64	3	431.0	430.0	450.0	455.0	440.1
passage	512	64	5	615.0	617.0	694.0	701.0	504.3

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	8.0	8.0	8.0	9.0	125.6
query	20	1	3	12.0	12.0	13.0	13.0	246.1
query	20	1	5	20.0	21.0	22.0	22.0	247.9
query	20	1	7	25.0	26.0	30.0	30.0	267.5
query	20	1	9	34.0	34.0	38.0	39.0	251.5
query	20	1	11	43.0	46.0	47.0	48.0	237.7
query	20	1	13	46.0	49.0	53.0	54.0	260.9
query	20	1	15	55.0	58.0	65.0	65.0	248.4
passage	300	64	1	186.0	186.0	190.0	192.0	342.7
passage	300	64	3	345.0	346.0	354.0	356.0	550.7
passage	300	64	5	525.0	524.0	535.0	544.0	608.3
passage	512	64	1	269.0	265.0	280.0	284.0	237.2
passage	512	64	3	563.0	564.0	581.0	584.0	337.3
passage	512	64	5	846.0	861.0	946.0	956.0	366.0

L40S

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	10.0	10.0	10.0	11.0	98.0
query	20	1	3	15.0	17.0	17.0	17.0	192.5
query	20	1	5	24.0	23.0	28.0	28.0	200.8
query	20	1	7	34.0	34.0	39.0	40.0	197.0
query	20	1	9	44.0	45.0	51.0	51.0	197.3
query	20	1	11	53.0	56.0	62.0	62.0	196.8
query	20	1	13	63.0	67.0	73.0	73.0	190.6
query	20	1	15	73.0	79.0	84.0	84.0	188.3
passage	300	64	1	277.0	278.0	280.0	281.0	230.3
passage	300	64	3	615.0	617.0	629.0	632.0	309.4
passage	300	64	5	976.0	975.0	983.0	987.0	327.4
passage	512	64	1	443.0	441.0	448.0	454.0	144.3
passage	512	64	3	1071.0	1077.0	1089.0	1090.0	177.6
passage	512	64	5	1736.0	1735.0	1752.0	1758.0	184.1

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	12.0	12.0	12.0	12.0	85.7
query	20	1	3	19.0	21.0	22.0	22.0	150.3
query	20	1	5	32.0	29.0	36.0	36.0	152.8
query	20	1	7	46.0	50.0	50.0	50.0	148.0
query	20	1	9	55.0	57.0	65.0	65.0	156.3
query	20	1	11	68.0	72.0	79.0	79.0	154.5
query	20	1	13	79.0	86.0	93.0	93.0	154.4
query	20	1	15	84.0	86.0	95.0	101.0	165.0
passage	300	64	1	350.0	350.0	353.0	354.0	182.8
passage	300	64	3	821.0	823.0	831.0	842.0	231.7
passage	300	64	5	1320.0	1317.0	1330.0	1346.0	242.1
passage	512	64	1	567.0	566.0	573.0	575.0	112.7
passage	512	64	3	1440.0	1448.0	1464.0	1475.0	132.2
passage	512	64	5	2370.0	2371.0	2391.0	2396.0	134.8

A100-SXM4-80GB

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	8.0	8.0	9.0	10.0	122.3
query	20	1	3	11.0	12.0	14.0	14.0	252.9
query	20	1	5	20.0	19.0	23.0	24.0	250.9
query	20	1	7	27.0	27.0	31.0	32.0	251.6
query	20	1	9	30.0	30.0	37.0	42.0	280.1
query	20	1	11	41.0	42.0	47.0	51.0	250.9
query	20	1	13	46.0	49.0	53.0	57.0	262.0
query	20	1	15	50.0	51.0	60.0	63.0	269.7
passage	300	64	1	325.0	324.0	331.0	333.0	196.9
passage	300	64	3	699.0	699.0	717.0	723.0	272.1
passage	300	64	5	1095.0	1093.0	1109.0	1113.0	291.8
passage	512	64	1	496.0	497.0	502.0	518.0	128.8
passage	512	64	3	1165.0	1169.0	1199.0	1209.0	163.2
passage	512	64	5	1872.0	1870.0	1904.0	1916.0	170.5

A100-PCIe-80GB

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	10.0	9.0	10.0	10.0	104.7
query	20	1	3	12.0	12.0	12.0	12.0	246.7
query	20	1	5	18.0	19.0	20.0	20.0	276.7
query	20	1	7	25.0	24.0	28.0	28.0	268.9
query	20	1	9	30.0	31.0	36.0	36.0	291.4
query	20	1	11	38.0	39.0	44.0	44.0	270.3
query	20	1	13	43.0	44.0	52.0	52.0	272.4
query	20	1	15	49.0	52.0	56.0	57.0	276.4
passage	300	64	1	362.0	362.0	367.0	369.0	176.8
passage	300	64	3	786.0	789.0	803.0	812.0	241.9
passage	300	64	5	1240.0	1239.0	1258.0	1262.0	257.6
passage	512	64	1	550.0	548.0	565.0	570.0	116.2
passage	512	64	3	1327.0	1335.0	1354.0	1359.0	143.4
passage	512	64	5	2145.0	2145.0	2180.0	2187.0	149.0

A10G

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	21.0	21.0	21.0	21.0	48.6
query	20	1	3	39.0	44.0	45.0	45.0	75.8
query	20	1	5	61.0	60.0	75.0	75.0	80.2
query	20	1	7	85.0	89.0	104.0	105.0	79.2
query	20	1	9	109.0	119.0	134.0	134.0	79.0
query	20	1	11	144.0	149.0	164.0	164.0	73.2
query	20	1	13	159.0	164.0	194.0	194.0	76.5
query	20	1	15	175.0	179.0	209.0	209.0	79.4
passage	300	64	1	888.0	888.0	899.0	899.0	72.1
passage	300	64	3	2272.0	2280.0	2329.0	2341.0	83.9
passage	300	64	5	3795.0	3801.0	3828.0	3840.0	84.2
passage	512	64	1	1451.0	1451.0	1471.0	1473.0	44.1
passage	512	64	3	3926.0	3947.0	3988.0	4009.0	48.6
passage	512	64	5	6577.0	6571.0	6632.0	6657.0	48.6

L4

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	13.0	13.0	14.0	15.0	74.2
query	20	1	3	21.0	23.0	23.0	23.0	138.0
query	20	1	5	32.0	31.0	39.0	39.0	152.2
query	20	1	7	46.0	46.0	53.0	54.0	145.8
query	20	1	9	56.0	61.0	69.0	69.0	150.3
query	20	1	11	67.0	69.0	76.0	77.0	154.0
query	20	1	13	86.0	91.0	98.0	99.0	141.2
query	20	1	15	98.0	105.0	113.0	114.0	141.0
passage	300	64	1	724.0	725.0	730.0	733.0	88.3
passage	300	64	3	1865.0	1876.0	1891.0	1892.0	102.1
passage	300	64	5	3117.0	3127.0	3147.0	3150.0	102.1
passage	512	64	1	1300.0	1300.0	1314.0	1318.0	49.2
passage	512	64	3	3551.0	3577.0	3606.0	3618.0	53.6
passage	512	64	5	5940.0	5968.0	6000.0	6007.0	53.6

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	18.0	18.0	18.0	19.0	55.8
query	20	1	3	33.0	37.0	38.0	38.0	87.7
query	20	1	5	55.0	62.0	63.0	63.0	87.4
query	20	1	7	71.0	75.0	88.0	88.0	94.6
query	20	1	9	96.0	101.0	114.0	114.0	90.3
query	20	1	11	115.0	126.0	139.0	139.0	89.8
query	20	1	13	131.0	138.0	164.0	164.0	91.4
query	20	1	15	154.0	157.0	189.0	189.0	88.4
passage	300	64	1	1011.0	1011.0	1019.0	1019.0	63.3
passage	300	64	3	2702.0	2716.0	2734.0	2754.0	70.6
passage	300	64	5	4527.0	4527.0	4550.0	4571.0	70.6
passage	512	64	1	1705.0	1710.0	1733.0	1740.0	37.5
passage	512	64	3	4770.0	4799.0	4860.0	4872.0	39.9
passage	512	64	5	8001.0	8025.0	8079.0	8085.0	39.9

NVIDIA QA E5 Embedding

H100-HBM3-80GB

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	300	64	1	99.8	100.9	107.9	108.6	639.0
passage	300	64	3	143.8	143.3	156.6	159.0	1330.0
passage	300	64	5	239.7	239.7	259.1	265.0	1331.0
passage	512	64	1	114.6	114.4	115.9	117.0	556.5
passage	512	64	3	170.2	169.9	171.2	171.8	1124.2
passage	512	64	5	284.6	284.5	285.6	286.1	1121.4
query	20	1	1	5.1	5.1	5.4	5.4	196.3
query	20	1	3	6.0	5.5	7.4	7.6	498.5
query	20	1	5	11.9	12.3	12.8	12.9	418.3
query	20	1	7	16.5	17.2	18.0	18.1	422.0
query	20	1	9	21.4	22.3	23.3	23.6	418.3
query	20	1	11	26.0	26.0	28.4	28.6	421.3
query	20	1	13	30.7	30.9	33.1	33.6	422.2
query	20	1	15	37.3	37.9	39.1	39.3	401.4

L40S

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	300	64	1	2554.3	2563.9	2678.1	2698.3	25.0
passage	300	64	3	7349.2	7502.1	7889.3	7968.1	25.5
passage	300	64	5	11913.2	12461.9	12893.4	12969.4	25.6
passage	512	64	1	3701.9	3701.6	3703.1	3703.4	17.3
passage	512	64	3	10730.2	10985.2	10987.0	11029.2	17.5
passage	512	64	5	17355.4	14691.3	22035.4	22035.7	17.4
query	20	1	1	32.4	32.4	32.7	32.8	30.7
query	20	1	3	82.5	85.6	85.9	86.0	36.3
query	20	1	5	135.5	142.9	143.3	143.3	36.8
query	20	1	7	191.7	200.2	200.5	200.6	36.5
query	20	1	9	246.9	257.4	257.8	257.9	36.4
query	20	1	11	301.7	314.6	315.1	315.2	36.4
query	20	1	13	356.6	371.6	372.2	372.4	36.4
query	20	1	15	409.5	401.4	429.8	429.9	36.5

A100-SXM4-80GB

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	300	64	1	176.5	177.1	188.6	190.4	362.2
passage	300	64	3	336.1	337.0	359.1	365.8	570.2
passage	300	64	5	560.2	562.9	592.3	634.6	569.8
passage	512	64	1	205.3	204.7	208.2	210.8	311.4
passage	512	64	3	410.9	411.1	412.5	412.7	466.4
passage	512	64	5	681.5	682.0	683.6	684.1	468.7
query	20	1	1	5.3	5.3	5.6	5.7	186.3
query	20	1	3	7.4	7.4	7.5	7.7	403.8
query	20	1	5	11.9	12.4	12.6	12.8	419.2
query	20	1	7	16.6	17.3	17.5	17.6	421.5
query	20	1	9	21.2	22.1	22.5	22.6	423.9
query	20	1	11	26.1	27.2	27.7	27.8	420.5
query	20	1	13	30.8	31.2	32.6	32.7	422.3
query	20	1	15	36.4	37.3	37.9	38.0	411.7

A100-PCIe-80GB

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	300	64	1	188.4	191.7	197.7	198.8	338.7
passage	300	64	3	371.7	372.7	393.8	471.0	515.3
passage	300	64	5	619.5	621.7	648.5	728.8	515.1
passage	512	64	1	222.7	222.3	226.0	227.4	286.5
passage	512	64	3	447.0	447.0	448.7	449.4	428.4
passage	512	64	5	742.3	742.8	745.0	745.5	430.1
query	20	1	1	6.6	6.6	7.0	7.1	149.3
query	20	1	3	7.4	7.3	7.6	7.7	404.8
query	20	1	5	11.8	12.2	12.5	12.6	421.5
query	20	1	7	16.4	17.1	17.4	17.5	426.4
query	20	1	9	20.9	21.9	22.3	22.4	429.9
query	20	1	11	25.7	26.8	27.4	27.7	427.2
query	20	1	13	30.4	31.5	32.1	32.2	427.4
query	20	1	15	35.6	36.4	37.9	38.0	420.6

A10G

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	300	64	1	377.9	376.4	396.5	404.5	169.1
passage	300	64	3	929.9	932.3	972.0	979.2	205.8
passage	300	64	5	1545.5	1555.2	1597.3	1610.3	205.9
passage	512	64	1	469.5	468.6	473.9	475.1	136.2
passage	512	64	3	1178.9	1182.4	1183.0	1183.2	162.2
passage	512	64	5	1958.1	1970.9	1971.6	1971.8	162.2
query	20	1	1	11.1	11.1	11.5	11.6	89.8
query	20	1	3	19.3	20.3	20.8	21.0	154.9
query	20	1	5	32.1	34.0	34.6	34.8	155.5
query	20	1	7	44.8	47.4	48.1	48.2	156.0
query	20	1	9	57.7	60.9	61.8	62.0	155.8
query	20	1	11	70.6	74.0	75.5	75.7	155.5
query	20	1	13	83.8	82.8	89.2	89.6	154.9
query	20	1	15	97.5	96.6	103.1	103.4	153.7

L4

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	300	64	1	483.6	483.6	505.2	509.5	132.3
passage	300	64	3	1328.0	1334.4	1367.9	1379.6	143.9
passage	300	64	5	2181.8	2203.7	2241.9	2250.4	145.5
passage	512	64	1	633.8	633.8	638.6	639.4	100.9
passage	512	64	3	1744.9	1755.3	1761.5	1763.1	109.4
passage	512	64	5	2892.2	2923.9	2934.8	2936.8	109.4
query	20	1	1	8.0	8.0	8.3	8.3	124.1
query	20	1	3	11.2	12.2	12.6	12.8	266.1
query	20	1	5	19.9	20.6	21.1	21.2	250.3
query	20	1	7	27.6	28.9	29.4	29.6	253.0
query	20	1	9	35.1	36.7	37.3	37.5	256.1
query	20	1	11	42.7	44.6	45.5	45.7	256.9
query	20	1	13	50.7	50.3	54.0	54.2	255.9
query	20	1	15	57.4	57.9	62.2	62.5	261.0