Performance for NVIDIA NeMo Retriever Embedding NIM

To benchmark the performance of NVIDIA NeMo Retriever Embedding NIM, you can use the genai-perf tool. genai-perf is pre-installed in the Triton Server SDK container. For the remainder of this section, we will use genai-perf==0.0.11 that comes packaged with the Triton Server SDK 25.02.

To run a performance benchmark, first create a dataset of text examples that genai-perf can use when making requests to the embedding service. These examples should be representative of the type of data that you expect to receive in a production setting. The dataset should be formatted as a JSONL file where each line contains a {"text": ...} object, as shown in the following example.

Example

Create a file named embeddings.jsonl that contains the following content.

{"text": "What was the first car ever driven?"}
{"text": "Who served as the 5th President of the United States of America?"}
{"text": "Is the Sydney Opera House located in Australia?"}
{"text": "In what state did they film Shrek 2?"}

Use the following example to run the Triton Inference Server SDK docker container. Mount the directory where you created your JSONL file, which appears as datasets/ in the following example.

export RELEASE="25.02"

docker run -it --rm \
  --gpus=all \
  --network="host" \
  --mount type=bind,source=${PWD}/datasets,target=/datasets \
  nvcr.io/nvidia/tritonserver:${RELEASE}-py3-sdk

In the container, run the following command to run a performance benchmark by using the genai-perf command line tool.

genai-perf profile \
    -m nvidia/llama-nemotron-embed-vl-1b-v2 \
    --service-kind openai \
    --endpoint-type embeddings \
    --batch-size 2 \
    --input-file /datasets/embeddings.jsonl \
    --extra-inputs input_type:query \
    --extra-inputs truncate:END \
    --concurrency 5 \
    --url http://localhost:8000

You can see the full set of command line options for genai-perf in the Command Line Options section of the GenAI-Perf documentation.

Benchmarks

All latency measurements are reported in milliseconds.

The VLM Embed benchmark tables report measurements for the supported GPU SKUs. For text rows, batch size is embedding inputs per request and throughput is embedding inputs per second. For image rows, batch size is images per request, image size is the generated JPEG resolution, and throughput is images per second.

Llama Nemotron Embed VL 1B v2

NVIDIA-RTX-PRO-6000-Blackwell-Workstation-Edition

FP8

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	2.3	2.3	2.4	2.4	352.8
query	20		1	3	4.1	4.3	4.4	4.4	628.0
query	20		1	5	4.1	4.3	4.4	4.5	1041.4
query	20		1	7	4.4	4.6	4.9	5.2	1316.6
query	20		1	9	4.4	4.4	4.6	4.6	1668.9
query	20		1	11	5.4	5.7	5.8	5.9	1699.0
query	20		1	13	4.6	4.7	4.9	5.1	2267.7
query	20		1	15	4.9	5.0	5.3	5.4	2421.9
passage	300		64	1	123.1	121.1	131.8	133.3	493.5
passage	300		64	3	349.6	353.4	361.5	364.8	534.4
passage	300		64	5	589.8	604.0	614.4	615.7	526.2
passage	512		64	1	195.7	195.6	198.0	198.7	317.9
passage	512		64	3	589.6	595.4	601.8	602.2	319.0
passage	512		64	5	1007.9	1024.3	1055.6	1058.9	309.1
image passage		336x336 JPEG	1	1	17.5	17.5	17.6	17.6	57.0
image passage		336x336 JPEG	1	2	29.5	29.5	29.7	29.8	34.1
image passage		336x336 JPEG	1	3	44.3	44.6	44.8	44.8	22.8
image passage		336x336 JPEG	1	5	73.4	74.8	74.9	75.0	14.1
image passage		336x336 JPEG	1	8	115.9	119.9	120.1	120.1	9.3
image passage		336x336 JPEG	4	1	63.3	63.3	63.5	63.6	63.2
image passage		336x336 JPEG	4	2	119.2	119.8	119.9	120.0	33.7
image passage		336x336 JPEG	4	3	178.5	180.2	180.4	180.4	22.7
image passage		336x336 JPEG	4	5	295.2	301.0	301.5	301.6	14.1
image passage		336x336 JPEG	4	8	466.1	482.9	483.3	483.4	9.4
image passage		336x336 JPEG	8	1	124.7	124.7	124.8	124.9	64.2
image passage		336x336 JPEG	8	2	239.9	241.0	241.3	241.4	33.5
image passage		336x336 JPEG	8	3	358.6	362.1	362.4	362.6	22.6
image passage		336x336 JPEG	8	5	592.7	604.4	605.4	607.0	14.0
image passage		336x336 JPEG	8	8	934.9	968.0	970.9	971.4	9.4
image passage		512x512 JPEG	1	1	17.7	17.7	17.9	17.9	56.4
image passage		512x512 JPEG	1	2	29.6	29.7	29.9	29.9	33.9
image passage		512x512 JPEG	1	3	44.4	44.8	45.0	45.0	22.7
image passage		512x512 JPEG	1	5	73.7	75.1	75.3	75.3	14.0
image passage		512x512 JPEG	1	8	116.5	120.4	120.7	120.8	9.3
image passage		512x512 JPEG	4	1	63.9	63.9	64.2	64.2	62.6
image passage		512x512 JPEG	4	2	119.6	120.1	120.4	120.4	33.6
image passage		512x512 JPEG	4	3	179.1	180.8	181.1	181.1	22.6
image passage		512x512 JPEG	4	5	296.3	302.1	302.4	302.4	14.0
image passage		512x512 JPEG	4	8	467.3	484.0	484.3	484.4	9.3
image passage		512x512 JPEG	8	1	126.0	125.9	126.5	126.6	63.5
image passage		512x512 JPEG	8	2	240.2	241.3	241.7	241.8	33.5
image passage		512x512 JPEG	8	3	359.3	362.7	363.1	363.2	22.6
image passage		512x512 JPEG	8	5	594.0	605.7	607.1	607.5	14.0
image passage		512x512 JPEG	8	8	936.8	970.1	971.4	972.0	9.3
image passage		1024x1024 JPEG	1	1	69.0	68.9	69.3	69.4	14.5
image passage		1024x1024 JPEG	1	2	130.6	131.2	131.5	131.6	7.7
image passage		1024x1024 JPEG	1	3	195.4	197.2	197.6	197.7	5.2
image passage		1024x1024 JPEG	1	5	323.1	329.4	329.9	330.5	3.2
image passage		1024x1024 JPEG	1	8	509.4	527.5	528.1	528.2	2.1
image passage		1024x1024 JPEG	4	1	272.3	271.2	277.1	279.1	14.7
image passage		1024x1024 JPEG	4	2	533.7	537.0	543.8	546.7	7.5
image passage		1024x1024 JPEG	4	3	798.0	809.6	818.3	819.6	5.1
image passage		1024x1024 JPEG	4	5	1322.1	1354.5	1371.0	1375.4	3.2
image passage		1024x1024 JPEG	4	8	2079.5	2159.9	2183.7	2186.3	2.1
image passage		1024x1024 JPEG	8	1	554.1	554.6	564.7	565.9	14.4
image passage		1024x1024 JPEG	8	2	1088.1	1096.3	1112.3	1113.3	7.4
image passage		1024x1024 JPEG	8	3	1628.1	1651.7	1664.9	1670.7	5.0
image passage		1024x1024 JPEG	8	5	2679.2	2740.7	2767.2	2776.2	3.1
image passage		1024x1024 JPEG	8	8	4225.0	4389.5	4426.5	4429.8	2.1
image passage		2048x2048 JPEG	1	1	70.5	70.5	71.1	71.2	14.2
image passage		2048x2048 JPEG	1	2	131.2	131.8	132.1	132.2	7.7
image passage		2048x2048 JPEG	1	3	196.4	198.3	198.7	198.8	5.2
image passage		2048x2048 JPEG	1	5	325.1	331.5	332.2	332.2	3.2
image passage		2048x2048 JPEG	1	8	512.0	530.3	530.8	530.9	2.1
image passage		2048x2048 JPEG	4	1	273.8	273.5	275.0	275.5	14.6
image passage		2048x2048 JPEG	4	2	524.3	526.3	530.9	531.6	7.7
image passage		2048x2048 JPEG	4	3	786.2	790.6	805.7	810.8	5.2
image passage		2048x2048 JPEG	4	5	1297.0	1319.6	1335.3	1337.6	3.2
image passage		2048x2048 JPEG	4	8	2045.1	2113.8	2135.7	2138.1	2.1
image passage		2048x2048 JPEG	8	1	550.1	548.1	559.5	563.8	14.5
image passage		2048x2048 JPEG	8	2	1060.5	1061.0	1087.1	1090.1	7.6
image passage		2048x2048 JPEG	8	3	1585.0	1592.0	1630.7	1633.7	5.1
image passage		2048x2048 JPEG	8	5	2617.5	2655.0	2724.0	2728.7	3.2
image passage		2048x2048 JPEG	8	8	4123.5	4248.4	4350.1	4357.3	2.1

NVIDIA-RTX-PRO-6000-Blackwell-Server-Edition

FP8

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	2.6	2.6	2.6	2.6	312.1
query	20		1	3	4.6	4.6	4.7	4.8	558.7
query	20		1	5	4.7	4.7	4.9	5.0	903.9
query	20		1	7	4.7	4.8	4.9	5.1	1241.8
query	20		1	9	5.2	5.2	5.6	5.7	1420.7
query	20		1	11	5.1	5.1	5.5	5.9	1747.8
query	20		1	13	5.6	5.3	6.5	6.7	1922.1
query	20		1	15	6.8	6.9	7.2	8.6	1800.2
passage	300		64	1	117.9	117.9	118.4	118.6	507.5
passage	300		64	3	323.5	326.4	327.7	328.1	573.5
passage	300		64	5	538.5	548.9	550.5	550.8	573.5
passage	512		64	1	200.0	199.8	202.0	202.3	307.5
passage	512		64	3	567.7	573.0	575.8	576.7	329.9
passage	512		64	5	943.8	962.3	966.4	967.6	329.4
image passage		336x336 JPEG	1	1	20.1	20.1	20.2	20.3	49.8
image passage		336x336 JPEG	1	2	33.7	33.6	34.4	35.2	29.8
image passage		336x336 JPEG	1	3	50.7	51.0	51.2	51.8	19.9
image passage		336x336 JPEG	1	5	83.9	85.4	86.0	86.7	12.3
image passage		336x336 JPEG	1	8	132.8	137.4	137.5	137.5	8.1
image passage		336x336 JPEG	4	1	72.8	72.7	73.0	73.6	54.9
image passage		336x336 JPEG	4	2	136.3	136.8	137.3	138.4	29.5
image passage		336x336 JPEG	4	3	204.1	205.9	206.7	207.1	19.9
image passage		336x336 JPEG	4	5	337.8	344.3	345.1	346.0	12.3
image passage		336x336 JPEG	4	8	532.7	551.6	552.8	553.1	8.2
image passage		336x336 JPEG	8	1	143.0	142.8	143.3	143.9	55.9
image passage		336x336 JPEG	8	2	273.5	274.3	275.9	276.9	29.4
image passage		336x336 JPEG	8	3	409.3	412.3	413.4	418.7	19.8
image passage		336x336 JPEG	8	5	676.9	689.0	690.1	697.7	12.3
image passage		336x336 JPEG	8	8	1067.6	1103.6	1108.5	1125.1	8.2
image passage		512x512 JPEG	1	1	20.3	20.3	20.4	20.5	49.3
image passage		512x512 JPEG	1	2	33.6	33.7	34.0	34.1	29.9
image passage		512x512 JPEG	1	3	50.8	51.1	51.3	51.9	19.9
image passage		512x512 JPEG	1	5	84.3	85.8	85.9	86.5	12.3
image passage		512x512 JPEG	1	8	132.9	137.4	137.8	138.1	8.1
image passage		512x512 JPEG	4	1	73.7	73.6	74.2	75.5	54.3
image passage		512x512 JPEG	4	2	137.0	137.0	138.6	142.2	29.3
image passage		512x512 JPEG	4	3	204.5	206.2	207.6	208.4	19.8
image passage		512x512 JPEG	4	5	338.0	344.5	344.9	346.1	12.3
image passage		512x512 JPEG	4	8	533.6	552.3	554.2	554.9	8.2
image passage		512x512 JPEG	8	1	144.7	144.5	145.2	145.7	55.3
image passage		512x512 JPEG	8	2	274.1	274.7	276.3	277.1	29.3
image passage		512x512 JPEG	8	3	410.2	413.1	414.7	421.4	19.8
image passage		512x512 JPEG	8	5	677.5	689.9	692.2	698.3	12.3
image passage		512x512 JPEG	8	8	1069.3	1105.2	1109.3	1126.3	8.2
image passage		1024x1024 JPEG	1	1	77.1	77.1	77.7	77.7	13.0
image passage		1024x1024 JPEG	1	2	145.6	145.9	146.3	148.5	6.9
image passage		1024x1024 JPEG	1	3	217.3	219.3	219.6	219.9	4.7
image passage		1024x1024 JPEG	1	5	359.1	366.2	366.5	366.6	2.9
image passage		1024x1024 JPEG	1	8	566.5	586.7	587.2	587.3	1.9
image passage		1024x1024 JPEG	4	1	302.2	301.8	303.8	305.6	13.2
image passage		1024x1024 JPEG	4	2	584.0	586.0	588.7	591.5	6.9
image passage		1024x1024 JPEG	4	3	872.6	880.3	883.0	890.7	4.6
image passage		1024x1024 JPEG	4	5	1441.6	1468.0	1477.5	1486.2	2.9
image passage		1024x1024 JPEG	4	8	2271.5	2351.6	2362.9	2364.7	1.9
image passage		1024x1024 JPEG	8	1	601.6	600.7	603.5	607.0	13.3
image passage		1024x1024 JPEG	8	2	1166.9	1171.7	1176.8	1178.0	6.9
image passage		1024x1024 JPEG	8	3	1743.4	1759.6	1762.9	1769.8	4.7
image passage		1024x1024 JPEG	8	5	2881.6	2936.8	2952.5	2953.7	2.9
image passage		1024x1024 JPEG	8	8	4538.7	4699.2	4712.7	4714.2	1.9
image passage		2048x2048 JPEG	1	1	79.4	79.5	80.4	80.6	12.6
image passage		2048x2048 JPEG	1	2	146.4	146.5	147.7	151.6	6.9
image passage		2048x2048 JPEG	1	3	218.2	220.2	220.7	220.9	4.6
image passage		2048x2048 JPEG	1	5	361.4	368.4	369.0	369.3	2.9
image passage		2048x2048 JPEG	1	8	571.2	590.5	591.6	601.0	1.9
image passage		2048x2048 JPEG	4	1	310.7	310.4	312.7	313.7	12.9
image passage		2048x2048 JPEG	4	2	584.8	586.8	589.1	590.4	6.9
image passage		2048x2048 JPEG	4	3	875.5	883.1	885.8	888.2	4.6
image passage		2048x2048 JPEG	4	5	1447.2	1475.0	1476.9	1491.8	2.9
image passage		2048x2048 JPEG	4	8	2282.4	2363.7	2366.4	2367.1	1.9
image passage		2048x2048 JPEG	8	1	619.6	618.6	625.5	626.1	12.9
image passage		2048x2048 JPEG	8	2	1170.0	1174.1	1179.5	1181.6	6.9
image passage		2048x2048 JPEG	8	3	1749.5	1765.4	1770.5	1773.4	4.6
image passage		2048x2048 JPEG	8	5	2897.2	2952.6	2963.6	2966.9	2.9
image passage		2048x2048 JPEG	8	8	4576.9	4738.8	4749.0	4752.8	1.9

NVIDIA-B200

FP8

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	2.6	2.5	2.7	2.7	249.4
query	20		1	3	4.1	4.4	4.8	5.0	524.8
query	20		1	5	4.4	4.5	6.3	7.0	833.4
query	20		1	7	4.7	4.6	6.2	9.0	1093.6
query	20		1	9	5.5	5.2	8.6	8.9	1284.3
query	20		1	11	5.5	5.2	10.1	11.5	1478.0
query	20		1	13	5.6	5.3	7.9	8.1	1679.9
query	20		1	15	6.2	6.1	8.9	10.6	1873.0
passage	300		64	1	149.6	149.6	150.1	150.5	408.9
passage	300		64	3	422.2	426.1	427.1	427.6	443.0
passage	300		64	5	701.0	714.9	716.4	716.5	443.0
passage	512		64	1	361.2	360.7	365.6	367.3	173.8
passage	512		64	3	1049.8	1059.5	1067.8	1068.7	179.8
passage	512		64	5	1736.6	1771.6	1782.3	1784.5	179.8
image passage		336x336 JPEG	1	1	18.5	17.4	17.5	25.3	55.7
image passage		336x336 JPEG	1	2	27.9	26.9	27.4	34.0	36.5
image passage		336x336 JPEG	1	3	42.0	41.3	45.5	50.2	24.1
image passage		336x336 JPEG	1	5	69.9	70.0	70.9	77.0	14.8
image passage		336x336 JPEG	1	8	114.4	113.1	139.6	142.4	9.5
image passage		336x336 JPEG	4	1	62.5	60.9	63.3	71.7	64.4
image passage		336x336 JPEG	4	2	113.4	111.9	124.6	127.2	35.5
image passage		336x336 JPEG	4	3	171.2	169.0	185.6	188.4	23.7
image passage		336x336 JPEG	4	5	285.5	283.4	305.0	307.6	14.6
image passage		336x336 JPEG	4	8	454.3	469.6	481.9	482.3	9.4
image passage		336x336 JPEG	8	1	121.1	118.6	130.7	135.2	66.2
image passage		336x336 JPEG	8	2	231.4	225.8	251.6	257.7	34.8
image passage		336x336 JPEG	8	3	347.6	343.0	372.0	378.9	23.4
image passage		336x336 JPEG	8	5	574.6	583.0	603.7	614.8	14.5
image passage		336x336 JPEG	8	8	901.2	926.7	955.7	960.4	9.7
image passage		512x512 JPEG	1	1	18.0	17.5	17.7	18.6	56.1
image passage		512x512 JPEG	1	2	27.6	27.2	27.5	28.5	36.5
image passage		512x512 JPEG	1	3	41.9	41.3	41.7	52.4	24.2
image passage		512x512 JPEG	1	5	73.6	69.9	85.6	95.4	14.2
image passage		512x512 JPEG	1	8	110.6	112.7	113.7	131.6	9.8
image passage		512x512 JPEG	4	1	63.0	61.4	70.5	73.5	63.8
image passage		512x512 JPEG	4	2	114.6	112.2	126.1	127.5	35.2
image passage		512x512 JPEG	4	3	173.0	169.3	190.2	193.3	23.5
image passage		512x512 JPEG	4	5	283.4	283.6	298.0	304.7	14.5
image passage		512x512 JPEG	4	8	452.2	459.4	490.2	518.2	9.7
image passage		512x512 JPEG	8	1	123.3	120.2	133.5	138.6	65.1
image passage		512x512 JPEG	8	2	231.1	226.0	242.0	246.2	34.8
image passage		512x512 JPEG	8	3	344.3	341.2	359.2	362.7	23.6
image passage		512x512 JPEG	8	5	571.6	582.6	601.6	605.3	14.6
image passage		512x512 JPEG	8	8	903.2	933.5	941.6	948.8	9.7
image passage		1024x1024 JPEG	1	1	89.0	87.0	87.8	104.4	11.3
image passage		1024x1024 JPEG	1	2	165.9	164.8	166.7	175.2	6.1
image passage		1024x1024 JPEG	1	3	247.7	247.7	251.0	270.0	4.1
image passage		1024x1024 JPEG	1	5	409.8	414.4	432.3	438.5	2.5
image passage		1024x1024 JPEG	1	8	642.3	663.7	667.1	676.7	1.7
image passage		1024x1024 JPEG	4	1	343.2	339.6	356.0	363.0	11.7
image passage		1024x1024 JPEG	4	2	662.6	663.2	670.1	680.4	6.1
image passage		1024x1024 JPEG	4	3	997.8	996.5	1033.5	1041.3	4.1
image passage		1024x1024 JPEG	4	5	1642.2	1668.1	1701.8	1714.4	2.5
image passage		1024x1024 JPEG	4	8	2591.9	2671.7	2707.8	2771.5	1.7
image passage		1024x1024 JPEG	8	1	683.2	677.5	698.8	704.3	11.7
image passage		1024x1024 JPEG	8	2	1333.9	1331.0	1369.9	1381.8	6.0
image passage		1024x1024 JPEG	8	3	1987.5	1998.3	2044.4	2058.4	4.1
image passage		1024x1024 JPEG	8	5	3288.4	3342.0	3391.1	3423.1	2.5
image passage		1024x1024 JPEG	8	8	5184.3	5368.9	5411.5	5448.4	1.7
image passage		2048x2048 JPEG	1	1	88.6	87.6	89.4	100.6	11.3
image passage		2048x2048 JPEG	1	2	163.4	161.7	171.3	179.5	6.2
image passage		2048x2048 JPEG	1	3	245.1	243.6	260.1	260.6	4.1
image passage		2048x2048 JPEG	1	5	406.1	407.6	425.4	426.5	2.6
image passage		2048x2048 JPEG	1	8	639.7	660.4	669.9	686.7	1.7
image passage		2048x2048 JPEG	4	1	343.4	340.5	352.8	362.2	11.7
image passage		2048x2048 JPEG	4	2	660.5	665.7	673.2	688.2	6.1
image passage		2048x2048 JPEG	4	3	988.0	995.9	1005.8	1020.3	4.1
image passage		2048x2048 JPEG	4	5	1629.1	1663.9	1677.2	1694.1	2.6
image passage		2048x2048 JPEG	4	8	2567.7	2662.8	2672.1	2684.2	1.7
image passage		2048x2048 JPEG	8	1	684.9	678.8	700.5	706.7	11.7
image passage		2048x2048 JPEG	8	2	1321.7	1322.9	1346.3	1359.7	6.1
image passage		2048x2048 JPEG	8	3	1972.1	1991.7	2004.8	2022.1	4.1
image passage		2048x2048 JPEG	8	5	3261.1	3320.0	3352.2	3384.5	2.6
image passage		2048x2048 JPEG	8	8	5137.7	5319.9	5338.2	5344.2	1.7

NVIDIA-GB200

FP8

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	1.8	1.8	1.9	1.9	426.9
query	20		1	3	2.6	2.3	3.8	3.8	935.5
query	20		1	5	3.6	3.8	4.4	4.5	1159.1
query	20		1	7	4.1	4.0	5.6	5.8	1438.4
query	20		1	9	4.3	4.4	5.7	5.9	1747.4
query	20		1	11	5.2	4.6	7.1	7.2	1786.2
query	20		1	13	4.6	4.5	6.0	6.1	2320.9
query	20		1	15	4.7	5.0	6.0	6.4	2596.1
passage	300		64	1	139.9	139.9	140.5	140.8	439.8
passage	300		64	3	399.1	402.8	404.3	404.6	469.5
passage	300		64	5	662.2	675.4	677.2	677.5	469.5
passage	512		64	1	334.0	333.6	337.4	338.3	188.4
passage	512		64	3	975.0	984.4	990.0	991.3	193.8
passage	512		64	5	1612.3	1645.1	1652.2	1653.5	193.8
image passage		336x336 JPEG	1	1	22.3	21.7	21.8	22.5	45.4
image passage		336x336 JPEG	1	2	37.9	38.0	38.1	38.6	26.5
image passage		336x336 JPEG	1	3	56.7	57.2	57.2	57.3	17.8
image passage		336x336 JPEG	1	5	93.8	95.6	95.7	95.8	11.0
image passage		336x336 JPEG	1	8	149.4	153.1	156.5	170.9	7.3
image passage		336x336 JPEG	4	1	81.2	79.8	80.2	89.0	49.5
image passage		336x336 JPEG	4	2	153.7	153.1	153.4	166.3	26.2
image passage		336x336 JPEG	4	3	231.0	229.9	247.2	248.6	17.6
image passage		336x336 JPEG	4	5	378.8	383.7	389.9	400.1	11.0
image passage		336x336 JPEG	4	8	604.4	615.6	642.8	657.9	7.2
image passage		336x336 JPEG	8	1	160.1	158.0	162.0	174.7	50.1
image passage		336x336 JPEG	8	2	310.4	307.7	322.6	336.0	25.9
image passage		336x336 JPEG	8	3	464.4	461.9	483.5	512.2	17.5
image passage		336x336 JPEG	8	5	767.4	772.0	823.1	830.1	10.9
image passage		336x336 JPEG	8	8	1208.0	1235.9	1298.0	1307.0	7.3
image passage		512x512 JPEG	1	1	22.4	21.7	21.9	24.3	45.1
image passage		512x512 JPEG	1	2	39.1	37.9	38.4	50.6	25.9
image passage		512x512 JPEG	1	3	57.0	57.4	57.6	58.2	17.7
image passage		512x512 JPEG	1	5	94.1	95.7	95.9	97.2	11.0
image passage		512x512 JPEG	1	8	148.3	153.4	153.7	153.9	7.3
image passage		512x512 JPEG	4	1	81.8	80.4	82.7	94.1	49.0
image passage		512x512 JPEG	4	2	155.9	153.2	166.4	177.9	25.8
image passage		512x512 JPEG	4	3	230.4	230.5	231.4	248.9	17.6
image passage		512x512 JPEG	4	5	380.7	385.0	390.2	408.5	10.9
image passage		512x512 JPEG	4	8	605.2	616.2	657.1	677.5	7.3
image passage		512x512 JPEG	8	1	161.8	159.4	163.2	174.3	49.6
image passage		512x512 JPEG	8	2	310.7	307.4	322.0	348.5	25.9
image passage		512x512 JPEG	8	3	464.8	462.5	484.0	510.1	17.5
image passage		512x512 JPEG	8	5	766.0	772.5	810.1	821.0	10.9
image passage		512x512 JPEG	8	8	1210.6	1238.3	1299.0	1304.6	7.2
image passage		1024x1024 JPEG	1	1	109.8	108.4	108.9	113.4	9.1
image passage		1024x1024 JPEG	1	2	211.1	210.0	210.9	216.3	4.8
image passage		1024x1024 JPEG	1	3	314.1	315.3	322.3	329.1	3.2
image passage		1024x1024 JPEG	1	5	520.3	525.7	549.1	554.9	2.0
image passage		1024x1024 JPEG	1	8	823.4	842.5	892.1	897.7	1.3
image passage		1024x1024 JPEG	4	1	433.7	428.0	452.5	474.8	9.2
image passage		1024x1024 JPEG	4	2	846.5	846.1	868.6	871.0	4.7
image passage		1024x1024 JPEG	4	3	1265.0	1268.4	1305.0	1309.4	3.2
image passage		1024x1024 JPEG	4	5	2085.6	2117.9	2151.5	2157.8	2.0
image passage		1024x1024 JPEG	4	8	3292.2	3410.4	3433.6	3437.2	1.3
image passage		1024x1024 JPEG	8	1	864.3	855.6	890.3	897.4	9.3
image passage		1024x1024 JPEG	8	2	1693.5	1695.3	1723.9	1727.6	4.7
image passage		1024x1024 JPEG	8	3	2531.1	2555.8	2579.4	2583.6	3.2
image passage		1024x1024 JPEG	8	5	4184.7	4270.2	4287.7	4296.9	2.0
image passage		1024x1024 JPEG	8	8	6578.8	6813.0	6838.2	6845.4	1.3
image passage		2048x2048 JPEG	1	1	110.9	109.9	110.7	116.1	9.0
image passage		2048x2048 JPEG	1	2	211.0	210.3	214.5	226.4	4.8
image passage		2048x2048 JPEG	1	3	316.6	315.9	327.4	343.1	3.2
image passage		2048x2048 JPEG	1	5	520.8	527.1	547.1	548.6	2.0
image passage		2048x2048 JPEG	1	8	822.8	845.6	872.5	875.1	1.3
image passage		2048x2048 JPEG	4	1	437.5	434.4	449.5	453.0	9.1
image passage		2048x2048 JPEG	4	2	850.9	848.4	874.0	877.0	4.7
image passage		2048x2048 JPEG	4	3	1265.7	1272.8	1288.3	1293.9	3.2
image passage		2048x2048 JPEG	4	5	2091.5	2132.9	2155.0	2158.8	2.0
image passage		2048x2048 JPEG	4	8	3299.6	3419.2	3439.6	3444.4	1.3
image passage		2048x2048 JPEG	8	1	876.0	870.1	891.7	898.0	9.1
image passage		2048x2048 JPEG	8	2	1699.3	1706.7	1722.6	1727.6	4.7
image passage		2048x2048 JPEG	8	3	2535.9	2558.4	2580.8	2586.6	3.2
image passage		2048x2048 JPEG	8	5	4176.6	4253.3	4296.1	4305.1	2.0
image passage		2048x2048 JPEG	8	8	6584.1	6816.2	6852.1	6862.0	1.3

NVIDIA-H200

FP8

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	3.0	3.0	3.1	3.1	225.7
query	20		1	3	4.5	4.8	5.6	5.7	489.2
query	20		1	5	5.6	4.8	9.4	11.1	690.3
query	20		1	7	7.7	5.5	14.5	15.5	742.8
query	20		1	9	6.7	5.7	10.4	11.9	1057.9
query	20		1	11	7.2	5.6	16.3	17.0	1236.9
query	20		1	13	9.4	9.1	17.2	17.5	1137.7
query	20		1	15	9.9	8.3	15.2	15.4	1261.1
passage	300		64	1	71.0	70.8	72.3	72.6	816.9
passage	300		64	3	189.9	191.6	193.1	194.2	962.6
passage	300		64	5	318.3	324.5	326.5	326.8	962.1
passage	512		64	1	121.0	121.1	122.1	122.4	497.5
passage	512		64	3	338.7	341.6	344.4	344.7	548.7
passage	512		64	5	564.9	575.9	581.0	581.7	547.7
image passage		336x336 JPEG	1	1	17.2	17.0	17.1	17.2	58.2
image passage		336x336 JPEG	1	2	26.4	26.3	26.6	26.7	38.0
image passage		336x336 JPEG	1	3	40.7	40.1	44.1	47.7	24.9
image passage		336x336 JPEG	1	5	68.6	68.0	77.0	78.9	15.1
image passage		336x336 JPEG	1	8	107.7	109.7	119.1	119.3	10.0
image passage		336x336 JPEG	4	1	60.0	59.4	59.6	64.7	66.8
image passage		336x336 JPEG	4	2	112.0	109.3	120.8	124.1	36.0
image passage		336x336 JPEG	4	3	165.0	164.8	175.4	176.8	24.6
image passage		336x336 JPEG	4	5	276.1	276.9	289.4	290.8	15.0
image passage		336x336 JPEG	4	8	434.0	445.0	459.4	460.5	10.1
image passage		336x336 JPEG	8	1	117.8	115.9	126.0	130.3	68.0
image passage		336x336 JPEG	8	2	222.3	220.1	231.5	232.8	36.2
image passage		336x336 JPEG	8	3	334.0	332.9	343.2	348.1	24.3
image passage		336x336 JPEG	8	5	550.8	562.5	567.7	571.5	15.1
image passage		336x336 JPEG	8	8	868.7	899.8	904.9	907.0	10.1
image passage		512x512 JPEG	1	1	17.4	17.2	17.4	17.4	57.7
image passage		512x512 JPEG	1	2	26.8	26.3	26.7	31.7	37.6
image passage		512x512 JPEG	1	3	40.6	40.2	41.1	46.8	24.9
image passage		512x512 JPEG	1	5	68.5	68.1	76.8	78.8	15.1
image passage		512x512 JPEG	1	8	109.0	110.2	121.5	129.9	9.9
image passage		512x512 JPEG	4	1	60.9	60.1	60.7	65.9	65.9
image passage		512x512 JPEG	4	2	110.3	109.4	114.6	120.3	36.4
image passage		512x512 JPEG	4	3	165.8	165.3	173.8	174.1	24.4
image passage		512x512 JPEG	4	5	274.2	276.0	287.6	287.8	15.0
image passage		512x512 JPEG	4	8	440.3	453.0	468.1	496.9	9.9
image passage		512x512 JPEG	8	1	119.1	117.4	123.9	129.1	67.3
image passage		512x512 JPEG	8	2	222.1	220.5	231.0	232.2	36.2
image passage		512x512 JPEG	8	3	333.5	332.4	345.0	347.5	24.3
image passage		512x512 JPEG	8	5	551.5	563.6	570.0	574.1	15.0
image passage		512x512 JPEG	8	8	870.9	899.7	902.0	941.1	10.1
image passage		1024x1024 JPEG	1	1	63.0	62.4	62.8	64.8	15.9
image passage		1024x1024 JPEG	1	2	115.3	115.3	119.2	120.5	8.7
image passage		1024x1024 JPEG	1	3	172.5	173.5	178.4	178.7	5.9
image passage		1024x1024 JPEG	1	5	285.7	290.6	293.6	294.7	3.6
image passage		1024x1024 JPEG	1	8	450.3	466.2	466.9	466.9	2.4
image passage		1024x1024 JPEG	4	1	242.6	240.7	242.7	253.8	16.5
image passage		1024x1024 JPEG	4	2	466.3	465.5	471.9	472.6	8.6
image passage		1024x1024 JPEG	4	3	697.4	702.8	705.3	705.8	5.8
image passage		1024x1024 JPEG	4	5	1152.7	1171.4	1182.0	1188.1	3.6
image passage		1024x1024 JPEG	4	8	1818.0	1879.6	1885.2	1940.0	2.4
image passage		1024x1024 JPEG	8	1	482.0	479.4	488.5	490.7	16.6
image passage		1024x1024 JPEG	8	2	933.9	936.5	938.7	955.2	8.6
image passage		1024x1024 JPEG	8	3	1394.4	1405.3	1411.6	1422.5	5.8
image passage		1024x1024 JPEG	8	5	2309.2	2346.5	2395.0	2413.7	3.6
image passage		1024x1024 JPEG	8	8	3631.9	3760.2	3768.9	3817.7	2.4
image passage		2048x2048 JPEG	1	1	64.8	64.5	65.2	65.3	15.5
image passage		2048x2048 JPEG	1	2	115.7	115.6	116.1	117.1	8.7
image passage		2048x2048 JPEG	1	3	173.0	174.5	175.0	175.8	5.8
image passage		2048x2048 JPEG	1	5	287.8	292.3	295.9	299.7	3.6
image passage		2048x2048 JPEG	1	8	455.1	469.1	471.8	490.2	2.4
image passage		2048x2048 JPEG	4	1	250.5	248.7	252.9	261.8	16.0
image passage		2048x2048 JPEG	4	2	467.3	466.4	473.3	481.7	8.6
image passage		2048x2048 JPEG	4	3	700.9	703.0	723.5	732.6	5.8
image passage		2048x2048 JPEG	4	5	1162.2	1174.6	1215.4	1221.7	3.6
image passage		2048x2048 JPEG	4	8	1825.3	1882.2	1902.1	1970.5	2.4
image passage		2048x2048 JPEG	8	1	498.5	495.8	510.4	514.5	16.1
image passage		2048x2048 JPEG	8	2	935.0	933.8	949.8	975.7	8.6
image passage		2048x2048 JPEG	8	3	1395.8	1402.2	1433.3	1467.3	5.8
image passage		2048x2048 JPEG	8	5	2307.1	2348.7	2368.9	2382.3	3.6
image passage		2048x2048 JPEG	8	8	3647.7	3759.5	3838.6	3852.8	2.4

NVIDIA-A100-SXM4-80GB

FP16

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	3.3	3.3	3.6	3.7	245.3
query	20		1	3	5.4	5.4	5.7	5.7	474.4
query	20		1	5	6.5	6.4	6.7	6.8	669.8
query	20		1	7	7.2	7.1	7.8	8.5	815.5
query	20		1	9	7.4	7.3	8.3	8.4	1026.8
query	20		1	11	10.4	10.7	11.0	11.1	917.6
query	20		1	13	8.7	8.8	9.3	10.4	1253.7
query	20		1	15	9.3	9.3	11.1	11.2	1352.2
passage	300		64	1	381.9	382.0	383.4	383.8	164.0
passage	300		64	3	1105.8	1116.4	1119.7	1120.8	170.6
passage	300		64	5	1832.3	1869.4	1873.3	1873.8	170.4
passage	512		64	1	838.4	837.7	846.2	848.1	75.5
passage	512		64	3	2463.4	2488.3	2500.3	2503.0	76.9
passage	512		64	5	4071.6	4153.5	4170.1	4175.1	76.9
image passage		336x336 JPEG	1	1	37.7	34.3	43.9	47.2	26.9
image passage		336x336 JPEG	1	2	69.9	66.9	79.1	82.2	14.5
image passage		336x336 JPEG	1	3	105.8	103.3	119.7	124.9	9.7
image passage		336x336 JPEG	1	5	172.0	172.1	191.0	195.3	6.0
image passage		336x336 JPEG	1	8	272.7	279.7	301.9	305.0	4.0
image passage		336x336 JPEG	4	1	143.7	140.2	158.7	162.4	28.0
image passage		336x336 JPEG	4	2	280.6	280.9	303.9	307.0	14.3
image passage		336x336 JPEG	4	3	420.6	422.8	439.0	443.9	9.6
image passage		336x336 JPEG	4	5	694.6	706.6	727.9	730.9	5.9
image passage		336x336 JPEG	4	8	1093.9	1128.7	1148.4	1150.3	3.9
image passage		336x336 JPEG	8	1	288.0	288.5	307.3	309.1	27.8
image passage		336x336 JPEG	8	2	562.2	564.9	574.3	584.1	14.3
image passage		336x336 JPEG	8	3	840.2	848.4	871.9	873.1	9.7
image passage		336x336 JPEG	8	5	1390.4	1419.2	1439.5	1447.7	6.0
image passage		336x336 JPEG	8	8	2184.5	2262.3	2278.7	2285.0	4.0
image passage		512x512 JPEG	1	1	37.3	34.5	42.1	43.7	27.1
image passage		512x512 JPEG	1	2	69.8	65.7	79.6	85.0	14.5
image passage		512x512 JPEG	1	3	105.0	101.2	118.4	123.9	9.8
image passage		512x512 JPEG	1	5	171.9	169.8	195.5	197.8	6.1
image passage		512x512 JPEG	1	8	276.9	283.3	309.8	316.2	3.9
image passage		512x512 JPEG	4	1	142.7	139.3	156.5	162.2	28.2
image passage		512x512 JPEG	4	2	281.0	284.0	302.7	306.0	14.3
image passage		512x512 JPEG	4	3	420.2	424.7	437.6	440.9	9.6
image passage		512x512 JPEG	4	5	693.8	701.3	728.8	731.7	6.0
image passage		512x512 JPEG	4	8	1093.0	1131.7	1140.0	1145.5	4.0
image passage		512x512 JPEG	8	1	290.0	291.5	312.0	314.0	27.7
image passage		512x512 JPEG	8	2	562.7	565.3	575.4	581.1	14.3
image passage		512x512 JPEG	8	3	839.6	850.2	866.6	868.1	9.7
image passage		512x512 JPEG	8	5	1386.4	1414.2	1437.8	1441.3	6.0
image passage		512x512 JPEG	8	8	2182.3	2261.9	2282.9	2290.4	4.0
image passage		1024x1024 JPEG	1	1	215.2	194.8	206.4	224.3	5.1
image passage		1024x1024 JPEG	1	2	376.7	376.2	394.7	399.2	2.7
image passage		1024x1024 JPEG	1	3	563.2	569.3	580.9	589.7	1.8
image passage		1024x1024 JPEG	1	5	932.8	950.6	967.0	973.3	1.1
image passage		1024x1024 JPEG	1	8	1472.2	1521.6	1548.8	1557.3	0.7
image passage		1024x1024 JPEG	4	1	764.6	765.3	784.2	788.2	5.2
image passage		1024x1024 JPEG	4	2	1512.8	1519.0	1546.4	1550.2	2.7
image passage		1024x1024 JPEG	4	3	2259.7	2280.7	2308.4	2313.8	1.8
image passage		1024x1024 JPEG	4	5	3735.3	3807.6	3836.4	3846.5	1.1
image passage		1024x1024 JPEG	4	8	5882.5	6088.6	6136.9	6145.3	0.7
image passage		1024x1024 JPEG	8	1	1532.2	1531.4	1548.9	1562.8	5.2
image passage		1024x1024 JPEG	8	2	3026.4	3044.2	3070.3	3076.5	2.7
image passage		1024x1024 JPEG	8	3	4518.2	4562.2	4588.5	4603.9	1.8
image passage		1024x1024 JPEG	8	5	7448.2	7595.0	7640.3	7649.3	1.1
image passage		1024x1024 JPEG	8	8	11746.7	12167.5	12202.4	12208.9	0.7
image passage		2048x2048 JPEG	1	1	190.9	192.7	202.4	205.9	5.2
image passage		2048x2048 JPEG	1	2	377.5	377.8	392.9	394.8	2.7
image passage		2048x2048 JPEG	1	3	561.5	563.8	578.8	593.8	1.8
image passage		2048x2048 JPEG	1	5	931.6	949.6	969.1	975.7	1.1
image passage		2048x2048 JPEG	1	8	1466.4	1521.4	1538.8	1540.7	0.8
image passage		2048x2048 JPEG	4	1	770.8	773.5	783.6	787.2	5.2
image passage		2048x2048 JPEG	4	2	1512.0	1520.9	1539.9	1544.3	2.7
image passage		2048x2048 JPEG	4	3	2261.7	2282.7	2305.3	2312.0	1.8
image passage		2048x2048 JPEG	4	5	3732.2	3804.0	3837.9	3845.6	1.1
image passage		2048x2048 JPEG	4	8	5895.7	6102.5	6145.4	6153.3	0.7
image passage		2048x2048 JPEG	8	1	1541.6	1541.3	1563.5	1570.8	5.2
image passage		2048x2048 JPEG	8	2	3017.7	3034.2	3061.5	3067.3	2.7
image passage		2048x2048 JPEG	8	3	4507.2	4550.4	4581.2	4589.9	1.8
image passage		2048x2048 JPEG	8	5	7445.9	7591.7	7622.3	7632.9	1.1
image passage		2048x2048 JPEG	8	8	11960.2	12184.1	12272.2	14723.2	0.7

NVIDIA-H100-NVL

FP8

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	2.4	2.4	2.5	2.5	322.8
query	20		1	3	4.2	4.3	5.2	8.4	616.4
query	20		1	5	5.1	4.5	5.5	6.7	856.9
query	20		1	7	6.8	5.3	13.9	15.2	899.9
query	20		1	9	7.1	5.4	12.2	17.6	1087.3
query	20		1	11	5.9	5.2	8.7	8.9	1514.1
query	20		1	13	7.2	5.9	15.6	16.0	1540.9
query	20		1	15	6.3	6.7	8.3	8.4	1958.3
passage	300		64	1	97.7	97.4	99.4	99.9	609.7
passage	300		64	3	272.8	275.5	277.3	278.9	677.8
passage	300		64	5	455.4	464.7	467.1	467.6	677.9
passage	512		64	1	168.1	168.0	169.3	169.7	364.9
passage	512		64	3	483.8	488.4	490.9	491.3	387.0
passage	512		64	5	804.7	820.9	825.2	825.9	386.1
image passage		336x336 JPEG	1	1	18.0	18.0	18.1	18.1	55.6
image passage		336x336 JPEG	1	2	29.6	29.7	29.8	29.8	33.9
image passage		336x336 JPEG	1	3	44.4	44.8	44.8	44.9	22.7
image passage		336x336 JPEG	1	5	74.2	75.5	75.6	76.2	13.9
image passage		336x336 JPEG	1	8	117.2	121.1	121.2	122.3	9.2
image passage		336x336 JPEG	4	1	64.8	64.7	64.9	65.0	61.8
image passage		336x336 JPEG	4	2	120.3	120.7	121.0	121.2	33.4
image passage		336x336 JPEG	4	3	180.2	181.9	182.1	182.2	22.5
image passage		336x336 JPEG	4	5	298.8	304.1	306.0	306.6	13.9
image passage		336x336 JPEG	4	8	511.4	488.3	505.2	981.6	8.9
image passage		336x336 JPEG	8	1	127.2	127.3	127.5	127.6	62.9
image passage		336x336 JPEG	8	2	242.3	243.5	243.7	243.7	33.2
image passage		336x336 JPEG	8	3	387.0	366.5	367.7	368.8	21.8
image passage		336x336 JPEG	8	5	639.7	612.6	615.5	707.0	13.5
image passage		336x336 JPEG	8	8	1010.8	982.1	985.0	1762.6	8.9
image passage		512x512 JPEG	1	1	18.2	18.2	18.3	18.3	55.0
image passage		512x512 JPEG	1	2	29.6	29.6	29.7	29.8	33.9
image passage		512x512 JPEG	1	3	44.4	44.8	44.8	44.9	22.7
image passage		512x512 JPEG	1	5	74.2	75.5	75.7	75.7	13.9
image passage		512x512 JPEG	1	8	117.7	121.6	121.9	122.6	9.2
image passage		512x512 JPEG	4	1	65.4	65.4	65.6	65.7	61.2
image passage		512x512 JPEG	4	2	135.9	120.8	121.3	123.4	32.6
image passage		512x512 JPEG	4	3	181.0	182.1	183.0	187.9	22.4
image passage		512x512 JPEG	4	5	299.2	304.8	306.1	306.2	13.9
image passage		512x512 JPEG	4	8	506.6	489.6	493.2	906.2	8.9
image passage		512x512 JPEG	8	1	128.7	128.6	129.0	129.4	62.2
image passage		512x512 JPEG	8	2	251.1	244.0	245.6	248.4	32.7
image passage		512x512 JPEG	8	3	364.0	367.0	367.7	371.4	22.3
image passage		512x512 JPEG	8	5	624.2	613.4	632.1	668.1	13.5
image passage		512x512 JPEG	8	8	950.1	983.4	987.0	994.3	9.2
image passage		1024x1024 JPEG	1	1	70.4	70.2	70.9	71.9	14.2
image passage		1024x1024 JPEG	1	2	139.9	131.5	132.6	134.7	7.5
image passage		1024x1024 JPEG	1	3	196.4	198.2	198.5	199.1	5.2
image passage		1024x1024 JPEG	1	5	324.9	331.1	332.3	333.0	3.2
image passage		1024x1024 JPEG	1	8	512.7	530.9	531.4	531.5	2.1
image passage		1024x1024 JPEG	4	1	281.1	272.8	273.5	274.7	14.5
image passage		1024x1024 JPEG	4	2	552.8	530.1	532.5	539.5	7.5
image passage		1024x1024 JPEG	4	3	827.0	796.5	806.4	1140.9	5.0
image passage		1024x1024 JPEG	4	5	1347.0	1329.7	1332.0	1382.4	3.1
image passage		1024x1024 JPEG	4	8	2122.7	2128.9	2134.3	2989.9	2.1
image passage		1024x1024 JPEG	8	1	563.1	542.8	544.8	546.7	14.5
image passage		1024x1024 JPEG	8	2	1088.2	1064.9	1077.0	1101.7	7.5
image passage		1024x1024 JPEG	8	3	1644.9	1599.6	1613.2	2350.2	5.0
image passage		1024x1024 JPEG	8	5	2722.2	2669.0	3152.7	3508.9	3.1
image passage		1024x1024 JPEG	8	8	4283.4	4263.1	5069.3	5133.6	2.1
image passage		2048x2048 JPEG	1	1	80.5	72.3	73.3	73.9	13.6
image passage		2048x2048 JPEG	1	2	131.5	131.8	133.4	134.0	7.6
image passage		2048x2048 JPEG	1	3	197.1	198.9	199.5	199.8	5.1
image passage		2048x2048 JPEG	1	5	391.4	333.5	682.2	762.1	3.0
image passage		2048x2048 JPEG	1	8	516.3	534.4	535.4	535.9	2.1
image passage		2048x2048 JPEG	4	1	292.6	280.0	281.9	283.1	14.1
image passage		2048x2048 JPEG	4	2	545.0	530.1	532.4	533.1	7.5
image passage		2048x2048 JPEG	4	3	818.1	799.3	802.3	815.1	5.0
image passage		2048x2048 JPEG	4	5	1352.5	1336.8	1342.1	1394.2	3.1
image passage		2048x2048 JPEG	4	8	2137.4	2144.8	2151.3	2986.1	2.1
image passage		2048x2048 JPEG	8	1	579.0	558.2	562.1	567.6	14.1
image passage		2048x2048 JPEG	8	2	1101.7	1063.5	1069.2	1484.1	7.4
image passage		2048x2048 JPEG	8	3	1650.1	1600.1	1661.8	2153.5	5.0
image passage		2048x2048 JPEG	8	5	2727.4	2676.4	3111.1	3512.9	3.1
image passage		2048x2048 JPEG	8	8	4316.8	4293.0	5152.2	5171.9	2.0

NVIDIA-H100-80GB-HBM3

FP8

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	2.5	2.4	2.5	2.5	316.9
query	20		1	3	4.6	3.6	5.4	14.6	551.8
query	20		1	5	6.1	4.5	11.6	14.2	718.2
query	20		1	7	7.4	6.3	14.6	16.8	839.0
query	20		1	9	5.1	5.3	5.6	7.1	1444.0
query	20		1	11	6.9	5.4	14.2	14.4	1358.0
query	20		1	13	6.5	5.5	13.2	14.7	1672.4
query	20		1	15	7.5	7.1	11.6	11.7	1693.4
passage	300		64	1	72.4	72.2	73.6	74.2	796.8
passage	300		64	3	195.1	196.7	198.6	199.9	936.1
passage	300		64	5	328.3	333.6	337.9	343.7	932.3
passage	512		64	1	123.4	123.5	124.7	124.7	487.2
passage	512		64	3	347.6	351.2	352.8	354.2	534.3
passage	512		64	5	578.9	590.3	594.2	595.2	534.3
image passage		336x336 JPEG	1	1	17.0	17.0	17.1	17.1	58.9
image passage		336x336 JPEG	1	2	28.2	28.2	28.3	28.3	35.6
image passage		336x336 JPEG	1	3	42.3	42.7	42.7	42.7	23.9
image passage		336x336 JPEG	1	5	70.3	71.6	71.9	72.4	14.7
image passage		336x336 JPEG	1	8	111.2	115.0	115.3	115.5	9.7
image passage		336x336 JPEG	4	1	61.4	61.2	61.4	63.6	65.2
image passage		336x336 JPEG	4	2	114.4	114.5	116.2	118.5	35.1
image passage		336x336 JPEG	4	3	171.9	172.4	175.2	176.9	23.6
image passage		336x336 JPEG	4	5	283.4	288.2	293.0	294.7	14.7
image passage		336x336 JPEG	4	8	449.6	462.4	468.4	491.4	9.7
image passage		336x336 JPEG	8	1	120.8	120.3	121.0	124.0	66.2
image passage		336x336 JPEG	8	2	229.5	229.8	233.8	235.4	35.0
image passage		336x336 JPEG	8	3	343.5	345.4	349.5	351.6	23.6
image passage		336x336 JPEG	8	5	567.0	577.4	581.6	583.7	14.7
image passage		336x336 JPEG	8	8	896.8	924.4	931.6	962.6	9.8
image passage		512x512 JPEG	1	1	17.2	17.1	17.3	17.3	58.2
image passage		512x512 JPEG	1	2	28.2	28.2	28.3	28.3	35.5
image passage		512x512 JPEG	1	3	42.8	42.8	42.9	44.9	23.6
image passage		512x512 JPEG	1	5	70.6	71.8	71.9	74.7	14.6
image passage		512x512 JPEG	1	8	112.0	115.8	115.9	116.0	9.6
image passage		512x512 JPEG	4	1	62.6	62.1	62.5	66.6	64.0
image passage		512x512 JPEG	4	2	114.7	114.8	115.2	117.9	35.0
image passage		512x512 JPEG	4	3	171.3	172.7	173.2	174.0	23.6
image passage		512x512 JPEG	4	5	284.0	288.7	292.8	294.8	14.6
image passage		512x512 JPEG	4	8	447.5	462.7	464.2	468.5	9.8
image passage		512x512 JPEG	8	1	122.2	121.9	123.1	124.7	65.5
image passage		512x512 JPEG	8	2	229.8	230.1	233.8	234.9	35.0
image passage		512x512 JPEG	8	3	343.7	345.9	349.6	352.7	23.6
image passage		512x512 JPEG	8	5	568.2	578.1	582.0	585.9	14.6
image passage		512x512 JPEG	8	8	898.7	927.7	933.9	940.4	9.7
image passage		1024x1024 JPEG	1	1	63.8	63.7	64.2	64.3	15.7
image passage		1024x1024 JPEG	1	2	119.0	119.4	119.7	119.9	8.4
image passage		1024x1024 JPEG	1	3	178.1	179.6	180.2	180.6	5.7
image passage		1024x1024 JPEG	1	5	295.6	301.3	302.0	304.6	3.5
image passage		1024x1024 JPEG	1	8	483.0	481.1	496.3	687.1	2.3
image passage		1024x1024 JPEG	4	1	251.3	248.1	250.4	252.7	16.0
image passage		1024x1024 JPEG	4	2	483.4	479.6	482.4	484.3	8.3
image passage		1024x1024 JPEG	4	3	720.2	720.1	738.3	786.4	5.6
image passage		1024x1024 JPEG	4	5	1178.8	1201.5	1205.6	1206.6	3.5
image passage		1024x1024 JPEG	4	8	1861.6	1926.5	1937.7	1939.9	2.3
image passage		1024x1024 JPEG	8	1	494.1	493.7	495.4	498.2	16.2
image passage		1024x1024 JPEG	8	2	956.0	958.9	966.6	969.6	8.4
image passage		1024x1024 JPEG	8	3	1430.4	1442.1	1452.7	1457.8	5.7
image passage		1024x1024 JPEG	8	5	2359.9	2404.8	2414.0	2420.4	3.5
image passage		1024x1024 JPEG	8	8	3724.2	3854.9	3870.6	3872.7	2.4
image passage		2048x2048 JPEG	1	1	66.9	66.8	68.5	68.6	15.0
image passage		2048x2048 JPEG	1	2	119.7	120.3	120.5	120.6	8.4
image passage		2048x2048 JPEG	1	3	180.0	180.6	186.5	188.7	5.6
image passage		2048x2048 JPEG	1	5	296.4	302.0	303.9	304.2	3.5
image passage		2048x2048 JPEG	1	8	467.5	484.0	484.6	484.7	2.3
image passage		2048x2048 JPEG	4	1	257.8	256.9	259.5	261.4	15.5
image passage		2048x2048 JPEG	4	2	479.7	480.6	484.0	485.3	8.4
image passage		2048x2048 JPEG	4	3	718.2	723.3	730.0	732.6	5.6
image passage		2048x2048 JPEG	4	5	1197.2	1209.2	1230.5	1250.0	3.5
image passage		2048x2048 JPEG	4	8	1875.2	1939.4	1944.1	1972.7	2.3
image passage		2048x2048 JPEG	8	1	512.8	512.4	516.1	517.9	15.6
image passage		2048x2048 JPEG	8	2	976.2	965.6	995.9	1042.0	8.3
image passage		2048x2048 JPEG	8	3	1435.7	1448.2	1452.3	1457.4	5.6
image passage		2048x2048 JPEG	8	5	2388.4	2420.7	2455.6	2491.0	3.5
image passage		2048x2048 JPEG	8	8	3762.5	3883.6	3927.7	3985.5	2.3

NVIDIA-L4

FP16

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	10.0	10.0	10.1	10.1	92.6
query	20		1	3	18.5	18.5	18.7	18.7	153.4
query	20		1	5	19.9	19.7	19.8	22.0	236.8
query	20		1	7	28.6	29.5	29.7	29.7	227.6
query	20		1	9	22.1	22.6	23.3	23.5	367.8
query	20		1	11	23.2	23.3	24.2	24.4	433.7
query	20		1	13	31.4	33.1	33.6	33.7	379.4
query	20		1	15	32.8	34.2	34.7	35.1	406.8
passage	300		64	1	1104.3	1105.0	1114.6	1116.9	57.5
passage	300		64	3	3271.2	3303.0	3323.7	3332.1	58.0
passage	300		64	5	5406.1	5515.4	5546.1	5552.9	57.9
passage	512		64	1	2188.5	2186.1	2213.1	2221.7	29.1
passage	512		64	3	6509.7	6573.1	6608.2	6616.6	29.2
passage	512		64	5	10746.7	10966.7	11010.6	11025.5	29.2
image passage		336x336 JPEG	1	1	87.0	87.0	89.3	89.9	11.5
image passage		336x336 JPEG	1	2	169.2	169.9	172.6	173.3	5.9
image passage		336x336 JPEG	1	3	253.9	256.4	259.0	259.3	4.0
image passage		336x336 JPEG	1	5	419.6	428.1	431.5	432.1	2.5
image passage		336x336 JPEG	1	8	662.1	685.8	689.2	690.9	1.7
image passage		336x336 JPEG	4	1	346.3	346.2	349.7	350.5	11.6
image passage		336x336 JPEG	4	2	685.8	689.1	692.3	693.2	5.9
image passage		336x336 JPEG	4	3	1026.6	1037.2	1040.7	1041.5	4.0
image passage		336x336 JPEG	4	5	1696.5	1731.2	1735.5	1736.8	2.5
image passage		336x336 JPEG	4	8	2674.5	2771.0	2776.2	2777.0	1.6
image passage		336x336 JPEG	8	1	696.4	696.0	699.9	701.0	11.5
image passage		336x336 JPEG	8	2	1378.7	1385.2	1390.5	1391.7	5.8
image passage		336x336 JPEG	8	3	2060.2	2080.3	2086.4	2088.0	3.9
image passage		336x336 JPEG	8	5	3400.5	3469.7	3477.2	3478.7	2.5
image passage		336x336 JPEG	8	8	5358.8	5552.8	5560.9	5563.9	1.6
image passage		512x512 JPEG	1	1	87.8	87.9	89.5	89.8	11.4
image passage		512x512 JPEG	1	2	170.4	171.3	173.8	174.4	5.9
image passage		512x512 JPEG	1	3	254.5	256.9	259.7	260.3	4.0
image passage		512x512 JPEG	1	5	421.5	429.7	432.5	434.1	2.5
image passage		512x512 JPEG	1	8	663.6	687.3	690.3	691.5	1.7
image passage		512x512 JPEG	4	1	347.5	347.1	350.8	351.6	11.5
image passage		512x512 JPEG	4	2	685.9	689.4	692.6	693.4	5.9
image passage		512x512 JPEG	4	3	1027.0	1037.1	1041.0	1042.9	4.0
image passage		512x512 JPEG	4	5	1694.8	1729.1	1733.5	1734.7	2.5
image passage		512x512 JPEG	4	8	2672.8	2769.4	2773.3	2778.1	1.6
image passage		512x512 JPEG	8	1	696.5	696.7	699.6	701.1	11.5
image passage		512x512 JPEG	8	2	1379.2	1386.1	1391.2	1391.6	5.8
image passage		512x512 JPEG	8	3	2058.7	2079.3	2083.8	2086.6	3.9
image passage		512x512 JPEG	8	5	3399.9	3468.4	3476.6	3480.0	2.5
image passage		512x512 JPEG	8	8	5357.3	5550.7	5561.2	5566.0	1.6
image passage		1024x1024 JPEG	1	1	465.9	462.6	476.9	480.6	2.1
image passage		1024x1024 JPEG	1	2	921.7	920.3	946.6	951.2	1.1
image passage		1024x1024 JPEG	1	3	1378.9	1390.7	1412.4	1431.6	0.7
image passage		1024x1024 JPEG	1	5	2267.7	2313.3	2328.9	2332.3	0.5
image passage		1024x1024 JPEG	1	8	3580.6	3704.8	3734.6	3748.2	0.3
image passage		1024x1024 JPEG	4	1	1864.2	1864.9	1877.4	1882.0	2.1
image passage		1024x1024 JPEG	4	2	3695.1	3708.9	3735.8	3746.2	1.1
image passage		1024x1024 JPEG	4	3	5509.4	5566.5	5583.7	5585.1	0.7
image passage		1024x1024 JPEG	4	5	9098.7	9280.1	9306.9	9323.1	0.5
image passage		1024x1024 JPEG	4	8	14327.0	14843.6	14886.3	14897.9	0.3
image passage		1024x1024 JPEG	8	1	3726.8	3723.1	3748.4	3757.9	2.1
image passage		1024x1024 JPEG	8	2	7395.5	7428.5	7459.6	7483.6	1.1
image passage		1024x1024 JPEG	8	3	11044.6	11154.7	11192.7	11200.7	0.7
image passage		1024x1024 JPEG	8	5	18205.8	18577.5	18614.3	18618.1	0.5
image passage		1024x1024 JPEG	8	8	28702.8	29714.8	29850.5	29966.5	0.3
image passage		2048x2048 JPEG	1	1	468.4	464.4	481.6	483.0	2.1
image passage		2048x2048 JPEG	1	2	920.0	920.0	941.3	946.8	1.1
image passage		2048x2048 JPEG	1	3	1373.0	1384.2	1407.0	1409.6	0.7
image passage		2048x2048 JPEG	1	5	2268.4	2313.8	2326.4	2327.6	0.5
image passage		2048x2048 JPEG	1	8	3576.4	3703.5	3732.0	3740.5	0.3
image passage		2048x2048 JPEG	4	1	1863.3	1862.4	1879.9	1883.3	2.1
image passage		2048x2048 JPEG	4	2	3695.4	3711.7	3735.0	3739.0	1.1
image passage		2048x2048 JPEG	4	3	5510.1	5567.9	5582.6	5586.7	0.7
image passage		2048x2048 JPEG	4	5	9087.8	9270.3	9300.1	9312.2	0.5
image passage		2048x2048 JPEG	4	8	14332.3	14851.5	14876.4	14885.6	0.3
image passage		2048x2048 JPEG	8	1	3731.9	3729.7	3751.4	3759.3	2.1
image passage		2048x2048 JPEG	8	2	7393.5	7427.6	7460.4	7477.7	1.1
image passage		2048x2048 JPEG	8	3	11035.5	11149.6	11177.5	11183.9	0.7
image passage		2048x2048 JPEG	8	5	18226.0	18573.6	18637.3	18684.3	0.5
image passage		2048x2048 JPEG	8	8	28696.2	29727.8	29790.0	29803.4	0.3

NVIDIA-L40S

FP16

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	4.2	4.2	4.2	4.2	207.2
query	20		1	3	6.6	6.9	7.0	7.0	403.9
query	20		1	5	7.1	7.1	7.3	7.3	624.6
query	20		1	7	7.3	7.3	7.5	7.5	843.0
query	20		1	9	10.8	11.2	11.3	11.4	729.9
query	20		1	11	7.9	8.0	8.2	9.2	1186.2
query	20		1	13	8.2	8.3	8.6	9.7	1347.5
query	20		1	15	8.4	8.4	10.0	10.1	1502.9
passage	300		64	1	336.2	336.3	337.5	337.8	185.7
passage	300		64	3	977.5	987.6	989.9	990.3	192.8
passage	300		64	5	1618.4	1652.1	1654.9	1655.9	192.8
passage	512		64	1	647.5	648.6	652.7	657.9	97.6
passage	512		64	3	1938.7	1958.4	1974.7	1979.3	97.6
passage	512		64	5	3233.8	3301.3	3313.4	3316.3	96.7
image passage		336x336 JPEG	1	1	28.9	28.5	30.2	32.3	34.6
image passage		336x336 JPEG	1	2	52.5	51.9	55.1	58.0	19.1
image passage		336x336 JPEG	1	3	78.9	78.4	84.4	86.2	12.8
image passage		336x336 JPEG	1	5	130.6	131.1	139.0	140.0	8.0
image passage		336x336 JPEG	1	8	206.4	210.2	219.1	219.7	5.3
image passage		336x336 JPEG	4	1	110.4	108.8	116.3	117.4	36.3
image passage		336x336 JPEG	4	2	212.9	211.4	219.8	220.6	18.9
image passage		336x336 JPEG	4	3	319.1	323.5	326.8	327.3	12.7
image passage		336x336 JPEG	4	5	527.5	537.9	540.4	540.9	7.9
image passage		336x336 JPEG	4	8	833.2	862.1	868.9	869.3	5.3
image passage		336x336 JPEG	8	1	220.3	218.4	225.7	226.4	36.3
image passage		336x336 JPEG	8	2	430.3	433.2	435.7	436.4	18.7
image passage		336x336 JPEG	8	3	643.7	649.7	656.6	658.0	12.6
image passage		336x336 JPEG	8	5	1064.9	1086.6	1089.3	1091.0	7.8
image passage		336x336 JPEG	8	8	1683.4	1742.4	1750.7	1751.6	5.2
image passage		512x512 JPEG	1	1	29.4	29.0	30.4	32.4	34.1
image passage		512x512 JPEG	1	2	53.1	52.5	55.6	59.0	18.9
image passage		512x512 JPEG	1	3	79.6	79.1	85.2	87.2	12.7
image passage		512x512 JPEG	1	5	131.7	132.2	140.7	142.6	7.9
image passage		512x512 JPEG	1	8	207.7	211.8	221.5	221.9	5.3
image passage		512x512 JPEG	4	1	111.5	109.9	117.5	118.4	35.9
image passage		512x512 JPEG	4	2	213.8	212.4	220.7	221.4	18.8
image passage		512x512 JPEG	4	3	319.8	323.6	327.6	328.0	12.7
image passage		512x512 JPEG	4	5	529.1	539.6	543.6	544.4	7.9
image passage		512x512 JPEG	4	8	835.2	864.7	870.5	872.0	5.2
image passage		512x512 JPEG	8	1	221.7	219.7	227.6	227.9	36.1
image passage		512x512 JPEG	8	2	429.9	432.7	435.5	436.2	18.7
image passage		512x512 JPEG	8	3	644.6	649.8	657.1	659.3	12.6
image passage		512x512 JPEG	8	5	1066.3	1088.1	1092.0	1093.3	7.8
image passage		512x512 JPEG	8	8	1679.4	1739.0	1747.9	1749.4	5.2
image passage		1024x1024 JPEG	1	1	149.5	147.3	157.5	158.5	6.7
image passage		1024x1024 JPEG	1	2	288.9	287.1	300.6	301.9	3.5
image passage		1024x1024 JPEG	1	3	432.0	431.3	448.2	451.8	2.3
image passage		1024x1024 JPEG	1	5	713.0	725.5	737.6	742.5	1.5
image passage		1024x1024 JPEG	1	8	1122.4	1164.3	1171.6	1171.8	1.0
image passage		1024x1024 JPEG	4	1	590.6	584.6	604.3	605.9	6.8
image passage		1024x1024 JPEG	4	2	1165.0	1172.2	1181.3	1182.8	3.4
image passage		1024x1024 JPEG	4	3	1739.6	1756.2	1768.0	1771.9	2.3
image passage		1024x1024 JPEG	4	5	2870.5	2929.2	2938.8	2946.4	1.5
image passage		1024x1024 JPEG	4	8	4524.7	4687.8	4701.4	4702.9	1.0
image passage		1024x1024 JPEG	8	1	1185.2	1186.0	1194.4	1196.2	6.8
image passage		1024x1024 JPEG	8	2	2335.2	2348.2	2359.3	2361.2	3.4
image passage		1024x1024 JPEG	8	3	3478.4	3511.3	3529.7	3533.6	2.3
image passage		1024x1024 JPEG	8	5	5755.0	5872.1	5887.7	5897.0	1.4
image passage		1024x1024 JPEG	8	8	9119.4	9396.7	9429.2	10020.8	1.0
image passage		2048x2048 JPEG	1	1	151.1	150.1	159.2	161.0	6.6
image passage		2048x2048 JPEG	1	2	289.5	287.5	301.9	306.0	3.5
image passage		2048x2048 JPEG	1	3	431.6	431.6	445.9	449.0	2.4
image passage		2048x2048 JPEG	1	5	712.0	725.5	739.4	740.7	1.5
image passage		2048x2048 JPEG	1	8	1123.8	1163.0	1173.5	1174.9	1.0
image passage		2048x2048 JPEG	4	1	597.8	596.1	611.9	613.6	6.7
image passage		2048x2048 JPEG	4	2	1159.1	1166.7	1174.1	1175.9	3.5
image passage		2048x2048 JPEG	4	3	1732.7	1749.6	1762.3	1766.1	2.3
image passage		2048x2048 JPEG	4	5	2861.1	2915.5	2935.8	2939.2	1.5
image passage		2048x2048 JPEG	4	8	4507.9	4671.0	4692.5	4694.9	1.0
image passage		2048x2048 JPEG	8	1	1197.7	1198.8	1208.7	1211.2	6.7
image passage		2048x2048 JPEG	8	2	2320.4	2332.1	2350.3	2354.3	3.5
image passage		2048x2048 JPEG	8	3	3467.4	3501.4	3525.5	3530.1	2.3
image passage		2048x2048 JPEG	8	5	5734.6	5854.6	5876.2	5879.3	1.5
image passage		2048x2048 JPEG	8	8	9030.5	9361.0	9400.7	9411.1	1.0

NVIDIA-A10G

FP16

Input Type	Input Tokens	Image Size	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20		1	1	6.3	6.2	6.3	6.3	143.7
query	20		1	3	10.4	11.0	11.2	11.2	263.4
query	20		1	5	11.7	11.8	12.0	12.0	384.0
query	20		1	7	12.9	13.0	13.2	13.3	488.6
query	20		1	9	14.4	14.5	14.9	16.0	559.7
query	20		1	11	20.9	21.3	21.6	22.7	478.1
query	20		1	13	20.9	21.2	22.9	23.1	556.9
query	20		1	15	21.1	21.4	25.6	25.7	635.1
passage	300		64	1	1171.3	1171.2	1178.0	1180.2	54.3
passage	300		64	3	3463.7	3498.8	3507.1	3508.2	54.8
passage	300		64	5	5720.4	5839.0	5849.4	5855.6	54.7
passage	512		64	1	2137.1	2140.7	2180.9	2208.9	29.8
passage	512		64	3	6409.9	6474.2	6517.6	6531.5	29.6
passage	512		64	5	10634.0	10818.2	10997.6	11028.1	29.5
image passage		336x336 JPEG	1	1	77.0	77.0	79.0	80.1	13.0
image passage		336x336 JPEG	1	2	151.4	151.7	156.1	157.6	6.6
image passage		336x336 JPEG	1	3	228.3	230.0	237.5	239.1	4.4
image passage		336x336 JPEG	1	5	376.9	386.0	388.8	389.7	2.8
image passage		336x336 JPEG	1	8	595.3	618.2	629.1	630.1	1.8
image passage		336x336 JPEG	4	1	318.2	317.4	326.1	326.7	12.6
image passage		336x336 JPEG	4	2	631.7	635.6	643.5	648.3	6.4
image passage		336x336 JPEG	4	3	948.5	955.9	973.0	975.1	4.3
image passage		336x336 JPEG	4	5	1565.0	1595.8	1607.9	1615.5	2.7
image passage		336x336 JPEG	4	8	2472.0	2555.7	2582.2	2597.4	1.8
image passage		336x336 JPEG	8	1	645.4	645.9	650.6	653.4	12.4
image passage		336x336 JPEG	8	2	1278.3	1285.2	1294.7	1299.9	6.3
image passage		336x336 JPEG	8	3	1911.8	1931.6	1945.2	1947.8	4.2
image passage		336x336 JPEG	8	5	3154.5	3216.9	3238.7	3247.3	2.6
image passage		336x336 JPEG	8	8	4966.3	5147.1	5178.2	5184.0	1.8
image passage		512x512 JPEG	1	1	81.5	80.2	85.9	91.9	12.3
image passage		512x512 JPEG	1	2	155.5	155.7	160.8	162.6	6.5
image passage		512x512 JPEG	1	3	232.0	233.8	239.1	240.8	4.4
image passage		512x512 JPEG	1	5	384.1	390.8	399.5	401.3	2.7
image passage		512x512 JPEG	1	8	601.9	623.0	638.8	640.3	1.8
image passage		512x512 JPEG	4	1	319.9	319.2	327.5	329.1	12.5
image passage		512x512 JPEG	4	2	633.1	636.7	640.9	643.6	6.4
image passage		512x512 JPEG	4	3	947.7	957.4	966.6	969.1	4.3
image passage		512x512 JPEG	4	5	1565.0	1596.5	1611.9	1614.5	2.7
image passage		512x512 JPEG	4	8	2466.5	2556.6	2575.5	2578.4	1.8
image passage		512x512 JPEG	8	1	645.4	645.1	654.4	656.8	12.4
image passage		512x512 JPEG	8	2	1278.3	1283.7	1298.9	1301.8	6.3
image passage		512x512 JPEG	8	3	1907.1	1928.4	1943.2	1947.5	4.3
image passage		512x512 JPEG	8	5	3149.6	3214.2	3238.3	3242.0	2.7
image passage		512x512 JPEG	8	8	4970.0	5153.5	5174.4	5180.6	1.8
image passage		1024x1024 JPEG	1	1	418.3	416.6	428.1	433.1	2.4
image passage		1024x1024 JPEG	1	2	824.2	826.9	839.9	846.3	1.2
image passage		1024x1024 JPEG	1	3	1232.5	1241.7	1261.7	1266.2	0.8
image passage		1024x1024 JPEG	1	5	2031.5	2071.0	2092.5	2102.3	0.5
image passage		1024x1024 JPEG	1	8	3200.8	3317.1	3341.4	3346.4	0.3
image passage		1024x1024 JPEG	4	1	1673.2	1672.9	1687.6	1691.2	2.4
image passage		1024x1024 JPEG	4	2	3326.9	3339.5	3356.2	3364.5	1.2
image passage		1024x1024 JPEG	4	3	4954.8	5004.8	5027.8	5031.4	0.8
image passage		1024x1024 JPEG	4	5	8175.0	8343.3	8364.3	8368.3	0.5
image passage		1024x1024 JPEG	4	8	12880.9	13353.1	13379.7	13387.6	0.3
image passage		1024x1024 JPEG	8	1	3349.5	3350.6	3366.1	3369.4	2.4
image passage		1024x1024 JPEG	8	2	6645.6	6679.9	6702.3	6706.0	1.2
image passage		1024x1024 JPEG	8	3	9926.7	10025.7	10060.1	10066.6	0.8
image passage		1024x1024 JPEG	8	5	16384.2	16720.9	16761.6	16774.9	0.5
image passage		1024x1024 JPEG	8	8	25793.0	26726.9	26766.8	26785.6	0.3
image passage		2048x2048 JPEG	1	1	418.1	416.2	427.3	434.6	2.4
image passage		2048x2048 JPEG	1	2	821.4	822.1	841.6	848.2	1.2
image passage		2048x2048 JPEG	1	3	1225.7	1236.4	1256.1	1260.2	0.8
image passage		2048x2048 JPEG	1	5	2023.9	2064.9	2088.0	2091.2	0.5
image passage		2048x2048 JPEG	1	8	3185.8	3304.6	3329.7	3336.3	0.3
image passage		2048x2048 JPEG	4	1	1675.1	1676.4	1696.7	1702.0	2.4
image passage		2048x2048 JPEG	4	2	3312.6	3332.9	3352.7	3357.7	1.2
image passage		2048x2048 JPEG	4	3	4949.6	5001.9	5028.8	5037.4	0.8
image passage		2048x2048 JPEG	4	5	8157.4	8330.8	8358.3	8368.9	0.5
image passage		2048x2048 JPEG	4	8	12861.0	13340.3	13401.5	13411.9	0.3
image passage		2048x2048 JPEG	8	1	3359.5	3360.5	3379.8	3385.4	2.4
image passage		2048x2048 JPEG	8	2	6645.9	6681.6	6705.8	6710.8	1.2
image passage		2048x2048 JPEG	8	3	9929.8	10031.0	10060.4	10068.0	0.8
image passage		2048x2048 JPEG	8	5	16388.4	16735.1	16765.1	16774.6	0.5
image passage		2048x2048 JPEG	8	8	25765.3	26693.5	26773.7	26778.5	0.3

Llama Nemotron Embed 300m

H100-HBM3-80GB

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	7.8	8.0	8.0	8.3	126.0
passage	20	1	3	8.3	8.0	9.0	8.8	348.0
passage	20	1	5	8.8	8.0	9.0	10.3	547.0
passage	20	1	7	10.8	11.0	12.0	11.9	585.0
passage	20	1	9	13.2	13.0	15.0	15.1	643.0
passage	20	1	11	15.5	16.0	18.0	18.4	652.0
passage	20	1	13	17.5	17.0	21.0	21.5	673.0
passage	20	1	15	20.6	21.0	24.0	24.0	662.0
passage	300	64	1	69.5	69.0	71.0	72.3	896.0
passage	300	64	3	103.5	104.0	107.0	108.3	1779.0
passage	300	64	5	172.3	173.0	178.0	179.9	1799.0
passage	512	64	1	103.6	101.0	116.0	120.1	606.0
passage	512	64	3	180.6	182.0	192.0	193.6	1040.0
passage	512	64	5	298.6	300.0	318.0	325.1	1041.0

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	8.0	8.0	9.0	8.9	122.0
passage	20	1	3	8.6	8.0	9.0	9.2	340.0
passage	20	1	5	8.8	9.0	9.0	9.2	548.0
passage	20	1	7	10.9	11.0	12.0	12.6	575.0
passage	20	1	9	13.3	14.0	15.0	15.2	638.0
passage	20	1	11	15.4	16.0	17.0	17.9	668.0
passage	20	1	13	17.6	17.0	21.0	21.3	688.0
passage	20	1	15	21.2	22.0	24.0	24.9	645.0
passage	300	64	1	70.9	71.0	73.0	74.3	880.0
passage	300	64	3	112.8	113.0	117.0	118.4	1652.0
passage	300	64	5	187.7	189.0	194.0	194.9	1653.0
passage	512	64	1	109.5	108.0	128.0	128.9	574.0
passage	512	64	3	202.5	203.0	216.0	219.4	919.0
passage	512	64	5	336.4	338.0	352.0	356.9	926.0

H100-NVL

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	8.2	8.0	9.0	8.6	120.0
passage	20	1	3	8.5	8.0	9.0	9.2	345.0
passage	20	1	5	9.4	9.0	10.0	10.8	508.0
passage	20	1	7	11.4	11.0	13.0	13.4	576.0
passage	20	1	9	14.4	15.0	17.0	16.9	584.0
passage	20	1	11	16.6	16.0	20.0	20.2	602.0
passage	20	1	13	20.2	21.0	24.0	24.2	590.0
passage	20	1	15	22.6	23.0	26.0	26.7	600.0
passage	300	64	1	68.8	68.0	72.0	73.1	907.0
passage	300	64	3	119.3	120.0	125.0	126.4	1552.0
passage	300	64	5	199.8	202.0	208.0	210.3	1555.0
passage	512	64	1	112.7	112.0	125.0	131.9	559.0
passage	512	64	3	229.5	232.0	242.0	246.6	814.0
passage	512	64	5	383.5	391.0	401.0	407.0	813.0

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	8.2	8.0	8.0	8.7	120.0
passage	20	1	3	8.3	8.0	9.0	8.8	339.0
passage	20	1	5	8.7	9.0	9.0	9.3	535.0
passage	20	1	7	10.7	11.0	12.0	11.8	597.0
passage	20	1	9	13.2	13.0	15.0	15.2	642.0
passage	20	1	11	15.5	16.0	18.0	18.3	654.0
passage	20	1	13	18.4	19.0	21.0	21.4	650.0
passage	20	1	15	20.8	21.0	24.0	24.5	668.0
passage	300	64	1	78.7	79.0	82.0	82.8	795.0
passage	300	64	3	149.7	151.0	156.0	157.8	1250.0
passage	300	64	5	249.8	254.0	260.0	261.9	1246.0
passage	512	64	1	129.6	127.0	142.0	150.2	487.0
passage	512	64	3	283.1	286.0	299.0	302.8	660.0
passage	512	64	5	474.0	482.0	499.0	502.4	658.0

L40S

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	8.3	8.0	8.0	8.7	118.0
passage	20	1	3	9.0	9.0	9.0	9.5	320.0
passage	20	1	5	9.9	10.0	11.0	11.4	476.0
passage	20	1	7	11.5	12.0	14.0	13.7	573.0
passage	20	1	9	14.4	14.0	16.0	16.5	583.0
passage	20	1	11	17.8	18.0	21.0	21.7	518.0
passage	20	1	13	20.2	20.0	24.0	23.7	582.0
passage	20	1	15	20.5	21.0	24.0	24.7	664.0
passage	300	64	1	97.3	97.0	99.0	100.2	646.0
passage	300	64	3	200.0	202.0	204.0	204.9	940.0
passage	300	64	5	332.7	336.0	340.0	340.5	938.0
passage	512	64	1	199.9	189.0	261.0	268.1	317.0
passage	512	64	3	415.9	419.0	427.0	428.2	455.0
passage	512	64	5	690.1	701.0	711.0	712.8	453.0

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	8.6	9.0	9.0	9.1	114.0
passage	20	1	3	9.3	9.0	10.0	10.2	312.0
passage	20	1	5	10.5	10.0	11.0	11.2	459.0
passage	20	1	7	14.4	14.0	15.0	21.8	453.0
passage	20	1	9	17.6	18.0	20.0	20.2	476.0
passage	20	1	11	20.2	21.0	25.0	24.9	499.0
passage	20	1	13	24.5	24.0	27.0	28.4	480.0
passage	20	1	15	28.7	30.0	33.0	33.3	471.0
passage	300	64	1	124.1	124.0	126.0	127.2	508.0
passage	300	64	3	273.1	275.0	278.0	279.2	692.0
passage	300	64	5	451.2	458.0	463.0	463.7	692.0
passage	512	64	1	239.6	230.0	297.0	301.0	265.0
passage	512	64	3	529.9	534.0	541.0	542.7	357.0
passage	512	64	5	876.4	892.0	900.0	901.8	357.0

A100-SXM4-80GB

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	5.5	5.0	7.0	7.0	180.0
passage	20	1	3	7.0	7.0	7.0	7.1	421.0
passage	20	1	5	10.5	11.0	12.0	11.8	455.0
passage	20	1	7	12.6	12.0	16.0	16.4	529.0
passage	20	1	9	18.5	20.0	21.0	21.3	466.0
passage	20	1	11	19.6	19.0	23.0	25.2	524.0
passage	20	1	13	24.6	25.0	28.0	30.1	481.0
passage	20	1	15	26.4	28.0	30.0	30.3	509.0
passage	300	64	1	115.7	115.0	120.0	120.4	544.0
passage	300	64	3	229.7	231.0	236.0	239.2	817.0
passage	300	64	5	381.6	385.0	393.0	397.6	817.0
passage	512	64	1	199.6	195.0	237.0	242.4	317.0
passage	512	64	3	431.6	433.0	447.0	448.9	438.0
passage	512	64	5	715.0	724.0	740.0	746.6	437.0

A10G

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	9.6	9.0	10.0	11.0	103.0
passage	20	1	3	12.6	13.0	14.0	14.2	232.0
passage	20	1	5	20.6	22.0	23.0	23.1	237.0
passage	20	1	7	28.3	31.0	32.0	31.7	234.0
passage	20	1	9	35.1	36.0	41.0	41.8	244.0
passage	20	1	11	43.3	45.0	50.0	50.4	236.0
passage	20	1	13	52.6	54.0	59.0	58.8	234.0
passage	20	1	15	58.0	60.0	68.0	68.7	234.0
passage	300	64	1	304.9	305.0	309.0	310.3	209.0
passage	300	64	3	780.6	791.0	799.0	800.0	241.0
passage	300	64	5	1296.8	1320.0	1330.0	1331.8	241.0
passage	512	64	1	520.7	519.0	533.0	538.6	122.0
passage	512	64	3	1385.5	1404.0	1424.0	1428.4	137.0
passage	512	64	5	2294.3	2341.0	2362.0	2368.3	137.0

L4

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	20	1	1	8.8	8.0	10.0	10.0	113.0
passage	20	1	3	11.1	11.0	12.0	11.8	259.0
passage	20	1	5	16.4	16.0	19.0	18.9	295.0
passage	20	1	7	23.3	22.0	26.0	26.8	290.0
passage	20	1	9	29.5	30.0	33.0	33.9	289.0
passage	20	1	11	34.2	34.0	40.0	40.6	302.0
passage	20	1	13	41.6	44.0	47.0	48.1	288.0
passage	20	1	15	45.7	48.0	55.0	55.2	300.0
passage	300	64	1	339.6	339.0	343.0	345.5	187.0
passage	300	64	3	918.3	927.0	933.0	934.6	207.0
passage	300	64	5	1517.3	1547.0	1554.0	1555.5	206.0
passage	512	64	1	642.3	643.0	657.0	661.9	99.0
passage	512	64	3	1795.3	1818.0	1835.0	1841.2	105.0
passage	512	64	5	2976.6	3034.0	3066.0	3075.5	105.0

Llama Nemotron Embed 1B

H100-HBM3-80GB

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	7.0	7.0	8.0	8.0	140.7
query	20	1	3	10.0	10.0	10.0	10.0	291.7
query	20	1	5	16.0	17.0	17.0	17.0	307.0
query	20	1	7	22.0	23.0	24.0	24.0	310.9
query	20	1	9	25.0	26.0	31.0	31.0	329.7
query	20	1	11	30.0	32.0	38.0	38.0	332.8
query	20	1	13	36.0	37.0	43.0	44.0	340.0
query	20	1	15	42.0	43.0	48.0	50.0	327.5
passage	300	64	1	159.0	159.0	162.0	163.0	401.8
passage	300	64	3	267.0	269.0	275.0	277.0	709.4
passage	300	64	5	392.0	390.0	401.0	411.0	814.5
passage	512	64	1	222.0	218.0	233.0	235.0	286.3
passage	512	64	3	431.0	430.0	450.0	455.0	440.1
passage	512	64	5	615.0	617.0	694.0	701.0	504.3

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	8.0	8.0	8.0	9.0	125.6
query	20	1	3	12.0	12.0	13.0	13.0	246.1
query	20	1	5	20.0	21.0	22.0	22.0	247.9
query	20	1	7	25.0	26.0	30.0	30.0	267.5
query	20	1	9	34.0	34.0	38.0	39.0	251.5
query	20	1	11	43.0	46.0	47.0	48.0	237.7
query	20	1	13	46.0	49.0	53.0	54.0	260.9
query	20	1	15	55.0	58.0	65.0	65.0	248.4
passage	300	64	1	186.0	186.0	190.0	192.0	342.7
passage	300	64	3	345.0	346.0	354.0	356.0	550.7
passage	300	64	5	525.0	524.0	535.0	544.0	608.3
passage	512	64	1	269.0	265.0	280.0	284.0	237.2
passage	512	64	3	563.0	564.0	581.0	584.0	337.3
passage	512	64	5	846.0	861.0	946.0	956.0	366.0

L40S

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	10.0	10.0	10.0	11.0	98.0
query	20	1	3	15.0	17.0	17.0	17.0	192.5
query	20	1	5	24.0	23.0	28.0	28.0	200.8
query	20	1	7	34.0	34.0	39.0	40.0	197.0
query	20	1	9	44.0	45.0	51.0	51.0	197.3
query	20	1	11	53.0	56.0	62.0	62.0	196.8
query	20	1	13	63.0	67.0	73.0	73.0	190.6
query	20	1	15	73.0	79.0	84.0	84.0	188.3
passage	300	64	1	277.0	278.0	280.0	281.0	230.3
passage	300	64	3	615.0	617.0	629.0	632.0	309.4
passage	300	64	5	976.0	975.0	983.0	987.0	327.4
passage	512	64	1	443.0	441.0	448.0	454.0	144.3
passage	512	64	3	1071.0	1077.0	1089.0	1090.0	177.6
passage	512	64	5	1736.0	1735.0	1752.0	1758.0	184.1

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	12.0	12.0	12.0	12.0	85.7
query	20	1	3	19.0	21.0	22.0	22.0	150.3
query	20	1	5	32.0	29.0	36.0	36.0	152.8
query	20	1	7	46.0	50.0	50.0	50.0	148.0
query	20	1	9	55.0	57.0	65.0	65.0	156.3
query	20	1	11	68.0	72.0	79.0	79.0	154.5
query	20	1	13	79.0	86.0	93.0	93.0	154.4
query	20	1	15	84.0	86.0	95.0	101.0	165.0
passage	300	64	1	350.0	350.0	353.0	354.0	182.8
passage	300	64	3	821.0	823.0	831.0	842.0	231.7
passage	300	64	5	1320.0	1317.0	1330.0	1346.0	242.1
passage	512	64	1	567.0	566.0	573.0	575.0	112.7
passage	512	64	3	1440.0	1448.0	1464.0	1475.0	132.2
passage	512	64	5	2370.0	2371.0	2391.0	2396.0	134.8

A100-SXM4-80GB

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	8.0	8.0	9.0	10.0	122.3
query	20	1	3	11.0	12.0	14.0	14.0	252.9
query	20	1	5	20.0	19.0	23.0	24.0	250.9
query	20	1	7	27.0	27.0	31.0	32.0	251.6
query	20	1	9	30.0	30.0	37.0	42.0	280.1
query	20	1	11	41.0	42.0	47.0	51.0	250.9
query	20	1	13	46.0	49.0	53.0	57.0	262.0
query	20	1	15	50.0	51.0	60.0	63.0	269.7
passage	300	64	1	325.0	324.0	331.0	333.0	196.9
passage	300	64	3	699.0	699.0	717.0	723.0	272.1
passage	300	64	5	1095.0	1093.0	1109.0	1113.0	291.8
passage	512	64	1	496.0	497.0	502.0	518.0	128.8
passage	512	64	3	1165.0	1169.0	1199.0	1209.0	163.2
passage	512	64	5	1872.0	1870.0	1904.0	1916.0	170.5

A100-PCIe-80GB

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	10.0	9.0	10.0	10.0	104.7
query	20	1	3	12.0	12.0	12.0	12.0	246.7
query	20	1	5	18.0	19.0	20.0	20.0	276.7
query	20	1	7	25.0	24.0	28.0	28.0	268.9
query	20	1	9	30.0	31.0	36.0	36.0	291.4
query	20	1	11	38.0	39.0	44.0	44.0	270.3
query	20	1	13	43.0	44.0	52.0	52.0	272.4
query	20	1	15	49.0	52.0	56.0	57.0	276.4
passage	300	64	1	362.0	362.0	367.0	369.0	176.8
passage	300	64	3	786.0	789.0	803.0	812.0	241.9
passage	300	64	5	1240.0	1239.0	1258.0	1262.0	257.6
passage	512	64	1	550.0	548.0	565.0	570.0	116.2
passage	512	64	3	1327.0	1335.0	1354.0	1359.0	143.4
passage	512	64	5	2145.0	2145.0	2180.0	2187.0	149.0

A10G

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	21.0	21.0	21.0	21.0	48.6
query	20	1	3	39.0	44.0	45.0	45.0	75.8
query	20	1	5	61.0	60.0	75.0	75.0	80.2
query	20	1	7	85.0	89.0	104.0	105.0	79.2
query	20	1	9	109.0	119.0	134.0	134.0	79.0
query	20	1	11	144.0	149.0	164.0	164.0	73.2
query	20	1	13	159.0	164.0	194.0	194.0	76.5
query	20	1	15	175.0	179.0	209.0	209.0	79.4
passage	300	64	1	888.0	888.0	899.0	899.0	72.1
passage	300	64	3	2272.0	2280.0	2329.0	2341.0	83.9
passage	300	64	5	3795.0	3801.0	3828.0	3840.0	84.2
passage	512	64	1	1451.0	1451.0	1471.0	1473.0	44.1
passage	512	64	3	3926.0	3947.0	3988.0	4009.0	48.6
passage	512	64	5	6577.0	6571.0	6632.0	6657.0	48.6

L4

FP8

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	13.0	13.0	14.0	15.0	74.2
query	20	1	3	21.0	23.0	23.0	23.0	138.0
query	20	1	5	32.0	31.0	39.0	39.0	152.2
query	20	1	7	46.0	46.0	53.0	54.0	145.8
query	20	1	9	56.0	61.0	69.0	69.0	150.3
query	20	1	11	67.0	69.0	76.0	77.0	154.0
query	20	1	13	86.0	91.0	98.0	99.0	141.2
query	20	1	15	98.0	105.0	113.0	114.0	141.0
passage	300	64	1	724.0	725.0	730.0	733.0	88.3
passage	300	64	3	1865.0	1876.0	1891.0	1892.0	102.1
passage	300	64	5	3117.0	3127.0	3147.0	3150.0	102.1
passage	512	64	1	1300.0	1300.0	1314.0	1318.0	49.2
passage	512	64	3	3551.0	3577.0	3606.0	3618.0	53.6
passage	512	64	5	5940.0	5968.0	6000.0	6007.0	53.6

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
query	20	1	1	18.0	18.0	18.0	19.0	55.8
query	20	1	3	33.0	37.0	38.0	38.0	87.7
query	20	1	5	55.0	62.0	63.0	63.0	87.4
query	20	1	7	71.0	75.0	88.0	88.0	94.6
query	20	1	9	96.0	101.0	114.0	114.0	90.3
query	20	1	11	115.0	126.0	139.0	139.0	89.8
query	20	1	13	131.0	138.0	164.0	164.0	91.4
query	20	1	15	154.0	157.0	189.0	189.0	88.4
passage	300	64	1	1011.0	1011.0	1019.0	1019.0	63.3
passage	300	64	3	2702.0	2716.0	2734.0	2754.0	70.6
passage	300	64	5	4527.0	4527.0	4550.0	4571.0	70.6
passage	512	64	1	1705.0	1710.0	1733.0	1740.0	37.5
passage	512	64	3	4770.0	4799.0	4860.0	4872.0	39.9
passage	512	64	5	8001.0	8025.0	8079.0	8085.0	39.9

NVIDIA QA E5 Embedding

H100-HBM3-80GB

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	300	64	1	99.8	100.9	107.9	108.6	639.0
passage	300	64	3	143.8	143.3	156.6	159.0	1330.0
passage	300	64	5	239.7	239.7	259.1	265.0	1331.0
passage	512	64	1	114.6	114.4	115.9	117.0	556.5
passage	512	64	3	170.2	169.9	171.2	171.8	1124.2
passage	512	64	5	284.6	284.5	285.6	286.1	1121.4
query	20	1	1	5.1	5.1	5.4	5.4	196.3
query	20	1	3	6.0	5.5	7.4	7.6	498.5
query	20	1	5	11.9	12.3	12.8	12.9	418.3
query	20	1	7	16.5	17.2	18.0	18.1	422.0
query	20	1	9	21.4	22.3	23.3	23.6	418.3
query	20	1	11	26.0	26.0	28.4	28.6	421.3
query	20	1	13	30.7	30.9	33.1	33.6	422.2
query	20	1	15	37.3	37.9	39.1	39.3	401.4

L40S

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	300	64	1	2554.3	2563.9	2678.1	2698.3	25.0
passage	300	64	3	7349.2	7502.1	7889.3	7968.1	25.5
passage	300	64	5	11913.2	12461.9	12893.4	12969.4	25.6
passage	512	64	1	3701.9	3701.6	3703.1	3703.4	17.3
passage	512	64	3	10730.2	10985.2	10987.0	11029.2	17.5
passage	512	64	5	17355.4	14691.3	22035.4	22035.7	17.4
query	20	1	1	32.4	32.4	32.7	32.8	30.7
query	20	1	3	82.5	85.6	85.9	86.0	36.3
query	20	1	5	135.5	142.9	143.3	143.3	36.8
query	20	1	7	191.7	200.2	200.5	200.6	36.5
query	20	1	9	246.9	257.4	257.8	257.9	36.4
query	20	1	11	301.7	314.6	315.1	315.2	36.4
query	20	1	13	356.6	371.6	372.2	372.4	36.4
query	20	1	15	409.5	401.4	429.8	429.9	36.5

A100-SXM4-80GB

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	300	64	1	176.5	177.1	188.6	190.4	362.2
passage	300	64	3	336.1	337.0	359.1	365.8	570.2
passage	300	64	5	560.2	562.9	592.3	634.6	569.8
passage	512	64	1	205.3	204.7	208.2	210.8	311.4
passage	512	64	3	410.9	411.1	412.5	412.7	466.4
passage	512	64	5	681.5	682.0	683.6	684.1	468.7
query	20	1	1	5.3	5.3	5.6	5.7	186.3
query	20	1	3	7.4	7.4	7.5	7.7	403.8
query	20	1	5	11.9	12.4	12.6	12.8	419.2
query	20	1	7	16.6	17.3	17.5	17.6	421.5
query	20	1	9	21.2	22.1	22.5	22.6	423.9
query	20	1	11	26.1	27.2	27.7	27.8	420.5
query	20	1	13	30.8	31.2	32.6	32.7	422.3
query	20	1	15	36.4	37.3	37.9	38.0	411.7

A100-PCIe-80GB

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	300	64	1	188.4	191.7	197.7	198.8	338.7
passage	300	64	3	371.7	372.7	393.8	471.0	515.3
passage	300	64	5	619.5	621.7	648.5	728.8	515.1
passage	512	64	1	222.7	222.3	226.0	227.4	286.5
passage	512	64	3	447.0	447.0	448.7	449.4	428.4
passage	512	64	5	742.3	742.8	745.0	745.5	430.1
query	20	1	1	6.6	6.6	7.0	7.1	149.3
query	20	1	3	7.4	7.3	7.6	7.7	404.8
query	20	1	5	11.8	12.2	12.5	12.6	421.5
query	20	1	7	16.4	17.1	17.4	17.5	426.4
query	20	1	9	20.9	21.9	22.3	22.4	429.9
query	20	1	11	25.7	26.8	27.4	27.7	427.2
query	20	1	13	30.4	31.5	32.1	32.2	427.4
query	20	1	15	35.6	36.4	37.9	38.0	420.6

A10G

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	300	64	1	377.9	376.4	396.5	404.5	169.1
passage	300	64	3	929.9	932.3	972.0	979.2	205.8
passage	300	64	5	1545.5	1555.2	1597.3	1610.3	205.9
passage	512	64	1	469.5	468.6	473.9	475.1	136.2
passage	512	64	3	1178.9	1182.4	1183.0	1183.2	162.2
passage	512	64	5	1958.1	1970.9	1971.6	1971.8	162.2
query	20	1	1	11.1	11.1	11.5	11.6	89.8
query	20	1	3	19.3	20.3	20.8	21.0	154.9
query	20	1	5	32.1	34.0	34.6	34.8	155.5
query	20	1	7	44.8	47.4	48.1	48.2	156.0
query	20	1	9	57.7	60.9	61.8	62.0	155.8
query	20	1	11	70.6	74.0	75.5	75.7	155.5
query	20	1	13	83.8	82.8	89.2	89.6	154.9
query	20	1	15	97.5	96.6	103.1	103.4	153.7

L4

FP16

Input Type	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
passage	300	64	1	483.6	483.6	505.2	509.5	132.3
passage	300	64	3	1328.0	1334.4	1367.9	1379.6	143.9
passage	300	64	5	2181.8	2203.7	2241.9	2250.4	145.5
passage	512	64	1	633.8	633.8	638.6	639.4	100.9
passage	512	64	3	1744.9	1755.3	1761.5	1763.1	109.4
passage	512	64	5	2892.2	2923.9	2934.8	2936.8	109.4
query	20	1	1	8.0	8.0	8.3	8.3	124.1
query	20	1	3	11.2	12.2	12.6	12.8	266.1
query	20	1	5	19.9	20.6	21.1	21.2	250.3
query	20	1	7	27.6	28.9	29.4	29.6	253.0
query	20	1	9	35.1	36.7	37.3	37.5	256.1
query	20	1	11	42.7	44.6	45.5	45.7	256.9
query	20	1	13	50.7	50.3	54.0	54.2	255.9
query	20	1	15	57.4	57.9	62.2	62.5	261.0