Performance and Memory for NVIDIA NeMo Retriever Embedding NIM#
To benchmark the performance of NVIDIA NeMo Retriever Embedding NIM, you can use the AIPerf tool.
To run a performance benchmark with representative input data, create a JSONL file where each line contains a {"texts": [...]} object, as shown in the following example.
Example#
Create a file named embeddings.jsonl that contains the following content.
{"texts": ["What was the first car ever driven?"]}
{"texts": ["Who served as the 5th President of the United States of America?"]}
{"texts": ["Is the Sydney Opera House located in Australia?"]}
{"texts": ["In what state did they film Shrek 2?"]}
Install the verified AIPerf version in a Python environment that can reach the NIM endpoint.
pip install aiperf==0.9.0
Run the following command to benchmark the embedding endpoint by using AIPerf.
aiperf profile \
--model nvidia/nemotron-3-embed-1b \
--endpoint-type nim_embeddings \
--url localhost:8000 \
--input-file embeddings.jsonl \
--custom-dataset-type single_turn \
--extra-inputs input_type:query \
--extra-inputs truncate:END \
--request-count 20 \
--concurrency 5
For the full set of command line options, refer to the AIPerf command line options documentation.
About the Measurements#
The following sections report the memory footprint and the benchmark results for each model. All latency measurements are reported in milliseconds.
The previous AIPerf example benchmarks a custom JSONL workload. The benchmark tables use a separate synthetic AIPerf benchmark matrix for consistent GPU comparisons. For text rows, batch size is embedding inputs per request and throughput is embedding inputs per second. For image rows, batch size is images per request, image size is the generated JPEG resolution, and throughput is images per second.
Nemotron 3 Embed 1B#
Memory Footprint#
The following table contains the memory footprint data for Nemotron 3 Embed 1B.
Precision |
Approximate GPU Memory Size (GiB) |
|
|---|---|---|
12.0 |
nvfp4 |
2.34 |
10.0 |
nvfp4 |
2.4 |
9.0 |
bf16 |
3.9 |
8.9 |
bf16 |
3.72 |
8.6 |
bf16 |
3.55 |
8.0 |
bf16 |
3.72 |
NVIDIA RTX PRO 6000 Blackwell Server Edition#
The following table contains the performance data for Nemotron 3 Embed 1B on NVIDIA RTX PRO 6000 Blackwell Server Edition.
Precision |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
NVFP4 |
20 |
1 |
1 |
2.9 |
2.9 |
2.9 |
2.9 |
222.9 |
NVFP4 |
20 |
1 |
3 |
3.5 |
3.6 |
4.0 |
4.5 |
570.2 |
NVFP4 |
20 |
1 |
5 |
5.1 |
4.4 |
5.8 |
13.0 |
742.9 |
NVFP4 |
20 |
1 |
7 |
4.3 |
4.5 |
5.0 |
5.1 |
1123.8 |
NVFP4 |
20 |
1 |
9 |
5.6 |
5.1 |
5.5 |
13.5 |
1206.0 |
NVFP4 |
20 |
1 |
11 |
6.2 |
4.8 |
13.6 |
13.8 |
1345.3 |
NVFP4 |
20 |
1 |
13 |
11.2 |
13.7 |
14.3 |
14.3 |
936.1 |
NVFP4 |
20 |
1 |
15 |
9.6 |
12.4 |
14.4 |
14.5 |
1185.6 |
NVFP4 |
300 |
64 |
1 |
63.6 |
63.7 |
63.9 |
64.0 |
906.6 |
NVFP4 |
300 |
64 |
3 |
156.0 |
157.2 |
157.7 |
157.8 |
1168.3 |
NVFP4 |
300 |
64 |
5 |
261.9 |
266.5 |
267.1 |
267.5 |
1167.7 |
NVFP4 |
512 |
64 |
1 |
118.1 |
118.1 |
118.5 |
118.7 |
511.4 |
NVFP4 |
512 |
64 |
3 |
317.8 |
320.7 |
321.5 |
321.7 |
585.8 |
NVFP4 |
512 |
64 |
5 |
529.8 |
540.0 |
541.1 |
541.4 |
584.6 |
BF16 |
20 |
1 |
1 |
13.5 |
13.5 |
13.5 |
13.5 |
66.1 |
BF16 |
20 |
1 |
3 |
13.8 |
13.8 |
13.9 |
14.0 |
189.8 |
BF16 |
20 |
1 |
5 |
13.8 |
13.8 |
13.9 |
14.0 |
323.6 |
BF16 |
20 |
1 |
7 |
14.2 |
14.1 |
14.5 |
14.5 |
421.6 |
BF16 |
20 |
1 |
9 |
14.7 |
14.7 |
14.9 |
15.0 |
510.0 |
BF16 |
20 |
1 |
11 |
15.3 |
15.3 |
15.6 |
15.7 |
591.9 |
BF16 |
20 |
1 |
13 |
15.6 |
15.6 |
15.9 |
16.0 |
729.1 |
BF16 |
20 |
1 |
15 |
16.3 |
16.4 |
17.0 |
17.1 |
792.6 |
BF16 |
300 |
64 |
1 |
202.8 |
202.3 |
204.1 |
206.5 |
305.5 |
BF16 |
300 |
64 |
3 |
544.3 |
548.5 |
553.0 |
554.0 |
344.9 |
BF16 |
300 |
64 |
5 |
903.2 |
920.4 |
925.3 |
925.9 |
344.7 |
BF16 |
512 |
64 |
1 |
517.9 |
515.3 |
527.5 |
528.7 |
121.9 |
BF16 |
512 |
64 |
3 |
1449.8 |
1463.3 |
1477.5 |
1482.5 |
130.5 |
BF16 |
512 |
64 |
5 |
2397.6 |
2443.2 |
2462.1 |
2467.2 |
130.5 |
NVIDIA GB200#
The following table contains the performance data for Nemotron 3 Embed 1B on NVIDIA GB200.
Precision |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
NVFP4 |
20 |
1 |
1 |
1.7 |
1.7 |
1.7 |
1.7 |
452.5 |
NVFP4 |
20 |
1 |
3 |
2.4 |
2.4 |
2.4 |
2.5 |
1006.3 |
NVFP4 |
20 |
1 |
5 |
2.9 |
2.4 |
2.5 |
3.6 |
1433.7 |
NVFP4 |
20 |
1 |
7 |
3.8 |
2.4 |
12.0 |
12.1 |
1557.6 |
NVFP4 |
20 |
1 |
9 |
3.5 |
2.6 |
3.5 |
12.4 |
2103.4 |
NVFP4 |
20 |
1 |
11 |
3.8 |
2.7 |
12.6 |
12.8 |
2356.3 |
NVFP4 |
20 |
1 |
13 |
2.8 |
2.7 |
4.2 |
4.3 |
3540.1 |
NVFP4 |
20 |
1 |
15 |
4.5 |
2.9 |
13.1 |
13.2 |
2764.9 |
NVFP4 |
300 |
64 |
1 |
36.8 |
36.8 |
37.3 |
38.0 |
1500.6 |
NVFP4 |
300 |
64 |
3 |
69.2 |
69.3 |
70.5 |
70.6 |
2510.3 |
NVFP4 |
300 |
64 |
5 |
118.4 |
120.1 |
121.0 |
121.4 |
2512.4 |
NVFP4 |
512 |
64 |
1 |
59.4 |
59.4 |
60.1 |
60.2 |
980.0 |
NVFP4 |
512 |
64 |
3 |
127.9 |
128.7 |
129.6 |
129.9 |
1414.0 |
NVFP4 |
512 |
64 |
5 |
215.1 |
218.3 |
219.5 |
219.7 |
1413.7 |
BF16 |
20 |
1 |
1 |
12.0 |
12.1 |
12.1 |
12.1 |
79.6 |
BF16 |
20 |
1 |
3 |
12.3 |
12.2 |
12.3 |
12.7 |
228.8 |
BF16 |
20 |
1 |
5 |
12.5 |
12.4 |
12.8 |
13.1 |
380.5 |
BF16 |
20 |
1 |
7 |
12.6 |
12.6 |
12.7 |
13.1 |
503.0 |
BF16 |
20 |
1 |
9 |
12.8 |
12.7 |
13.3 |
13.4 |
619.1 |
BF16 |
20 |
1 |
11 |
13.7 |
13.1 |
16.8 |
16.9 |
701.0 |
BF16 |
20 |
1 |
13 |
13.7 |
13.3 |
15.7 |
15.7 |
862.3 |
BF16 |
20 |
1 |
15 |
14.4 |
13.7 |
17.5 |
17.6 |
942.3 |
BF16 |
300 |
64 |
1 |
136.5 |
137.2 |
141.0 |
142.3 |
450.1 |
BF16 |
300 |
64 |
3 |
330.3 |
333.3 |
341.7 |
344.1 |
565.3 |
BF16 |
300 |
64 |
5 |
549.0 |
558.9 |
572.4 |
574.7 |
565.1 |
BF16 |
512 |
64 |
1 |
345.8 |
350.4 |
356.2 |
358.3 |
182.1 |
BF16 |
512 |
64 |
3 |
912.8 |
923.8 |
942.9 |
946.2 |
206.8 |
BF16 |
512 |
64 |
5 |
1510.6 |
1540.2 |
1571.9 |
1574.0 |
206.8 |
NVIDIA H100 80GB HBM3#
The following table contains the performance data for Nemotron 3 Embed 1B on NVIDIA H100 80GB HBM3.
Precision |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
BF16 |
20 |
1 |
1 |
3.3 |
3.2 |
3.3 |
3.3 |
202.8 |
BF16 |
20 |
1 |
3 |
3.4 |
3.3 |
3.6 |
3.8 |
591.6 |
BF16 |
20 |
1 |
5 |
6.6 |
6.6 |
6.9 |
7.0 |
594.2 |
BF16 |
20 |
1 |
7 |
9.8 |
9.9 |
10.1 |
10.1 |
594.4 |
BF16 |
20 |
1 |
9 |
12.9 |
13.3 |
13.5 |
13.6 |
593.1 |
BF16 |
20 |
1 |
11 |
16.0 |
16.6 |
16.8 |
16.9 |
593.9 |
BF16 |
20 |
1 |
13 |
19.0 |
19.9 |
20.1 |
20.5 |
592.3 |
BF16 |
20 |
1 |
15 |
21.9 |
23.2 |
23.4 |
23.5 |
592.1 |
BF16 |
300 |
64 |
1 |
292.9 |
292.8 |
294.9 |
295.1 |
213.2 |
BF16 |
300 |
64 |
3 |
702.8 |
706.1 |
710.8 |
711.9 |
268.4 |
BF16 |
300 |
64 |
5 |
1179.9 |
1180.1 |
1185.8 |
1189.8 |
269.1 |
BF16 |
512 |
64 |
1 |
499.4 |
499.6 |
509.5 |
513.1 |
126.2 |
BF16 |
512 |
64 |
3 |
1277.2 |
1284.9 |
1302.2 |
1306.3 |
148.2 |
BF16 |
512 |
64 |
5 |
2146.0 |
2143.2 |
2171.2 |
2174.5 |
148.5 |
NVIDIA H100 NVL#
The following table contains the performance data for Nemotron 3 Embed 1B on NVIDIA H100 NVL.
Precision |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
BF16 |
20 |
1 |
1 |
3.2 |
3.2 |
3.2 |
3.3 |
208.1 |
BF16 |
20 |
1 |
3 |
3.3 |
3.3 |
3.6 |
3.7 |
590.6 |
BF16 |
20 |
1 |
5 |
6.4 |
6.4 |
6.6 |
6.7 |
612.4 |
BF16 |
20 |
1 |
7 |
9.5 |
9.6 |
9.8 |
9.8 |
612.2 |
BF16 |
20 |
1 |
9 |
12.5 |
12.8 |
13.1 |
13.2 |
612.5 |
BF16 |
20 |
1 |
11 |
15.5 |
16.1 |
16.3 |
16.4 |
611.6 |
BF16 |
20 |
1 |
13 |
18.4 |
19.3 |
19.5 |
19.6 |
612.3 |
BF16 |
20 |
1 |
15 |
21.3 |
22.5 |
22.8 |
22.9 |
611.0 |
BF16 |
300 |
64 |
1 |
303.3 |
303.3 |
304.9 |
305.3 |
206.4 |
BF16 |
300 |
64 |
3 |
747.4 |
751.7 |
755.8 |
756.9 |
252.8 |
BF16 |
300 |
64 |
5 |
1254.0 |
1255.5 |
1263.7 |
1264.9 |
253.4 |
BF16 |
512 |
64 |
1 |
533.4 |
532.7 |
545.2 |
547.5 |
118.5 |
BF16 |
512 |
64 |
3 |
1401.8 |
1409.8 |
1428.6 |
1434.9 |
135.3 |
BF16 |
512 |
64 |
5 |
2355.3 |
2351.4 |
2382.5 |
2385.2 |
135.4 |
NVIDIA A100 SXM4 80GB#
The following table contains the performance data for Nemotron 3 Embed 1B on NVIDIA A100 SXM4 80GB.
Precision |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
BF16 |
20 |
1 |
1 |
3.2 |
3.3 |
3.3 |
3.3 |
252.1 |
BF16 |
20 |
1 |
3 |
6.8 |
6.6 |
7.5 |
7.5 |
388.8 |
BF16 |
20 |
1 |
5 |
12.5 |
12.9 |
13.0 |
13.1 |
367.6 |
BF16 |
20 |
1 |
7 |
16.9 |
16.5 |
18.5 |
18.6 |
379.3 |
BF16 |
20 |
1 |
9 |
22.5 |
23.8 |
24.0 |
24.1 |
367.0 |
BF16 |
20 |
1 |
11 |
26.1 |
26.1 |
29.5 |
29.6 |
384.5 |
BF16 |
20 |
1 |
13 |
32.4 |
34.7 |
35.0 |
35.2 |
365.0 |
BF16 |
20 |
1 |
15 |
35.0 |
36.1 |
40.6 |
40.8 |
387.2 |
BF16 |
300 |
64 |
1 |
510.2 |
510.1 |
513.8 |
514.9 |
123.2 |
BF16 |
300 |
64 |
3 |
1412.2 |
1420.5 |
1428.5 |
1429.9 |
134.1 |
BF16 |
300 |
64 |
5 |
2374.8 |
2376.4 |
2387.5 |
2388.2 |
134.2 |
BF16 |
512 |
64 |
1 |
913.2 |
912.1 |
937.1 |
942.0 |
69.3 |
BF16 |
512 |
64 |
3 |
2550.6 |
2563.2 |
2608.4 |
2612.8 |
74.5 |
BF16 |
512 |
64 |
5 |
4285.0 |
4280.4 |
4344.7 |
4348.0 |
74.5 |
NVIDIA L40S#
The following table contains the performance data for Nemotron 3 Embed 1B on NVIDIA L40S.
Precision |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
BF16 |
20 |
1 |
1 |
4.8 |
4.8 |
4.9 |
4.9 |
156.4 |
BF16 |
20 |
1 |
3 |
8.7 |
8.7 |
8.9 |
9.1 |
293.6 |
BF16 |
20 |
1 |
5 |
15.1 |
15.3 |
15.6 |
15.7 |
293.8 |
BF16 |
20 |
1 |
7 |
21.6 |
22.1 |
22.4 |
22.5 |
293.6 |
BF16 |
20 |
1 |
9 |
27.8 |
28.8 |
29.1 |
29.2 |
293.7 |
BF16 |
20 |
1 |
11 |
34.1 |
35.7 |
36.0 |
36.0 |
293.9 |
BF16 |
20 |
1 |
13 |
40.2 |
42.4 |
42.6 |
42.7 |
293.6 |
BF16 |
20 |
1 |
15 |
46.0 |
49.0 |
49.3 |
49.4 |
293.6 |
BF16 |
300 |
64 |
1 |
493.7 |
493.8 |
495.9 |
496.2 |
127.9 |
BF16 |
300 |
64 |
3 |
1344.4 |
1353.0 |
1356.4 |
1358.2 |
141.1 |
BF16 |
300 |
64 |
5 |
2271.3 |
2272.4 |
2276.9 |
2278.5 |
140.4 |
BF16 |
512 |
64 |
1 |
796.9 |
795.1 |
820.0 |
822.4 |
79.6 |
BF16 |
512 |
64 |
3 |
2240.7 |
2255.1 |
2287.4 |
2292.2 |
84.8 |
BF16 |
512 |
64 |
5 |
3765.1 |
3766.5 |
3813.4 |
3817.6 |
84.8 |
NVIDIA A10G#
The following table contains the performance data for Nemotron 3 Embed 1B on NVIDIA A10G.
Precision |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
BF16 |
20 |
1 |
1 |
5.8 |
5.8 |
5.8 |
5.8 |
148.5 |
BF16 |
20 |
1 |
3 |
13.9 |
14.0 |
14.4 |
14.6 |
196.5 |
BF16 |
20 |
1 |
5 |
23.7 |
24.1 |
24.4 |
24.4 |
196.6 |
BF16 |
20 |
1 |
7 |
33.5 |
34.4 |
34.7 |
34.7 |
196.7 |
BF16 |
20 |
1 |
9 |
42.8 |
44.4 |
44.9 |
45.0 |
196.3 |
BF16 |
20 |
1 |
11 |
52.0 |
54.6 |
55.0 |
55.1 |
196.0 |
BF16 |
20 |
1 |
13 |
61.0 |
64.7 |
65.0 |
65.1 |
196.4 |
BF16 |
20 |
1 |
15 |
69.7 |
74.8 |
75.2 |
75.3 |
196.1 |
BF16 |
300 |
64 |
1 |
1148.5 |
1147.6 |
1152.8 |
1157.4 |
55.2 |
BF16 |
300 |
64 |
3 |
3306.7 |
3326.1 |
3342.6 |
3343.9 |
57.6 |
BF16 |
300 |
64 |
5 |
5546.9 |
5547.5 |
5553.8 |
5556.8 |
57.6 |
BF16 |
512 |
64 |
1 |
2113.1 |
2114.9 |
2179.9 |
2194.4 |
30.1 |
BF16 |
512 |
64 |
3 |
6117.3 |
6157.5 |
6257.9 |
6276.2 |
31.1 |
BF16 |
512 |
64 |
5 |
10270.1 |
10277.2 |
10429.7 |
10455.3 |
31.1 |
Llama Nemotron Embed VL 1B v2#
Memory Footprint#
The following table contains the memory footprint data for Llama Nemotron Embed VL 1B v2.
Precision |
Approximate GPU Memory Size (GiB) |
|
|---|---|---|
12.0 |
fp8 |
7.96 |
10.0 |
fp8 |
9.04 |
9.0 |
fp8 |
8.87 |
8.9 |
fp8 |
8.21 |
8.9 |
fp16 |
5.26 |
8.6 |
fp16 |
5.33 |
8.0 |
fp16 |
5.49 |
NVIDIA RTX PRO 6000 Blackwell Server Edition#
The following table contains the performance data for Llama Nemotron Embed VL 1B v2 on NVIDIA RTX PRO 6000 Blackwell Server Edition.
Precision |
Input Type |
Image Size |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|---|
Mixed |
text |
20 |
1 |
1 |
3.8 |
3.8 |
3.9 |
3.9 |
183.8 |
|
Mixed |
text |
20 |
1 |
3 |
5.8 |
5.7 |
6.0 |
6.1 |
407.0 |
|
Mixed |
text |
20 |
1 |
5 |
10.6 |
10.7 |
10.9 |
11.0 |
403.2 |
|
Mixed |
text |
20 |
1 |
7 |
15.3 |
15.6 |
15.9 |
16.1 |
401.5 |
|
Mixed |
text |
20 |
1 |
9 |
20.5 |
21.2 |
21.4 |
21.5 |
393.4 |
|
Mixed |
text |
20 |
1 |
11 |
25.1 |
26.2 |
26.4 |
26.5 |
393.2 |
|
Mixed |
text |
20 |
1 |
13 |
29.6 |
31.2 |
31.5 |
31.6 |
392.9 |
|
Mixed |
text |
20 |
1 |
15 |
34.0 |
36.2 |
36.5 |
36.6 |
392.7 |
|
Mixed |
text |
300 |
64 |
1 |
260.6 |
260.5 |
261.0 |
261.2 |
239.4 |
|
Mixed |
text |
300 |
64 |
3 |
752.0 |
759.2 |
760.3 |
760.5 |
250.5 |
|
Mixed |
text |
300 |
64 |
5 |
1245.2 |
1270.1 |
1271.1 |
1271.3 |
250.5 |
|
Mixed |
text |
512 |
64 |
1 |
401.9 |
401.3 |
411.2 |
412.8 |
156.6 |
|
Mixed |
text |
512 |
64 |
3 |
1170.1 |
1180.9 |
1194.8 |
1201.3 |
161.5 |
|
Mixed |
text |
512 |
64 |
5 |
1932.8 |
1972.3 |
1988.9 |
1990.8 |
161.6 |
|
Mixed |
image passage |
336x336 JPEG |
1 |
1 |
12.3 |
12.3 |
12.5 |
12.7 |
81.1 |
|
Mixed |
image passage |
336x336 JPEG |
1 |
2 |
19.2 |
19.2 |
19.4 |
19.5 |
104.4 |
|
Mixed |
image passage |
336x336 JPEG |
1 |
3 |
29.5 |
29.7 |
29.9 |
30.0 |
102.5 |
|
Mixed |
image passage |
336x336 JPEG |
1 |
5 |
49.8 |
50.6 |
50.8 |
50.9 |
103.3 |
|
Mixed |
image passage |
336x336 JPEG |
1 |
8 |
79.5 |
82.1 |
82.3 |
82.5 |
108.7 |
|
Mixed |
image passage |
336x336 JPEG |
4 |
1 |
11.1 |
11.1 |
11.1 |
11.2 |
90.1 |
|
Mixed |
image passage |
336x336 JPEG |
4 |
2 |
20.4 |
20.5 |
20.6 |
20.6 |
98.3 |
|
Mixed |
image passage |
336x336 JPEG |
4 |
3 |
30.6 |
30.9 |
30.9 |
30.9 |
99.2 |
|
Mixed |
image passage |
336x336 JPEG |
4 |
5 |
50.9 |
51.8 |
51.9 |
51.9 |
102.1 |
|
Mixed |
image passage |
336x336 JPEG |
4 |
8 |
80.5 |
83.3 |
83.4 |
83.4 |
108.4 |
|
Mixed |
image passage |
336x336 JPEG |
8 |
1 |
10.9 |
10.9 |
10.9 |
10.9 |
91.5 |
|
Mixed |
image passage |
336x336 JPEG |
8 |
2 |
20.6 |
20.6 |
20.7 |
20.8 |
97.7 |
|
Mixed |
image passage |
336x336 JPEG |
8 |
3 |
30.8 |
31.1 |
31.1 |
31.2 |
98.6 |
|
Mixed |
image passage |
336x336 JPEG |
8 |
5 |
51.1 |
52.1 |
52.2 |
52.2 |
101.7 |
|
Mixed |
image passage |
336x336 JPEG |
8 |
8 |
80.8 |
83.5 |
83.7 |
84.5 |
108.3 |
|
Mixed |
image passage |
512x512 JPEG |
1 |
1 |
12.2 |
12.2 |
12.4 |
12.4 |
81.7 |
|
Mixed |
image passage |
512x512 JPEG |
1 |
2 |
19.1 |
19.2 |
19.3 |
19.4 |
104.7 |
|
Mixed |
image passage |
512x512 JPEG |
1 |
3 |
29.5 |
29.7 |
30.0 |
30.3 |
102.5 |
|
Mixed |
image passage |
512x512 JPEG |
1 |
5 |
49.7 |
50.6 |
50.8 |
50.8 |
103.9 |
|
Mixed |
image passage |
512x512 JPEG |
1 |
8 |
79.4 |
82.1 |
82.2 |
82.3 |
108.6 |
|
Mixed |
image passage |
512x512 JPEG |
4 |
1 |
11.2 |
11.2 |
11.3 |
11.3 |
89.1 |
|
Mixed |
image passage |
512x512 JPEG |
4 |
2 |
20.3 |
20.4 |
20.5 |
20.5 |
98.7 |
|
Mixed |
image passage |
512x512 JPEG |
4 |
3 |
30.6 |
30.9 |
30.9 |
30.9 |
99.3 |
|
Mixed |
image passage |
512x512 JPEG |
4 |
5 |
50.9 |
51.8 |
51.9 |
51.9 |
102.1 |
|
Mixed |
image passage |
512x512 JPEG |
4 |
8 |
80.4 |
83.3 |
83.3 |
83.3 |
108.3 |
|
Mixed |
image passage |
512x512 JPEG |
8 |
1 |
11.1 |
11.1 |
11.1 |
11.1 |
90.4 |
|
Mixed |
image passage |
512x512 JPEG |
8 |
2 |
20.5 |
20.6 |
20.6 |
20.6 |
97.8 |
|
Mixed |
image passage |
512x512 JPEG |
8 |
3 |
30.8 |
31.1 |
31.1 |
31.1 |
98.6 |
|
Mixed |
image passage |
512x512 JPEG |
8 |
5 |
51.0 |
52.0 |
52.1 |
52.1 |
101.8 |
|
Mixed |
image passage |
512x512 JPEG |
8 |
8 |
80.7 |
83.6 |
83.7 |
83.7 |
108.2 |
|
Mixed |
image passage |
1024x1024 JPEG |
1 |
1 |
12.7 |
12.7 |
13.0 |
13.0 |
78.5 |
|
Mixed |
image passage |
1024x1024 JPEG |
1 |
2 |
19.0 |
19.0 |
19.2 |
19.3 |
105.6 |
|
Mixed |
image passage |
1024x1024 JPEG |
1 |
3 |
29.3 |
29.6 |
29.8 |
29.8 |
103.1 |
|
Mixed |
image passage |
1024x1024 JPEG |
1 |
5 |
49.7 |
50.5 |
50.7 |
50.8 |
103.7 |
|
Mixed |
image passage |
1024x1024 JPEG |
1 |
8 |
79.3 |
81.9 |
82.2 |
82.2 |
108.3 |
|
Mixed |
image passage |
1024x1024 JPEG |
4 |
1 |
11.8 |
11.8 |
11.9 |
12.0 |
84.8 |
|
Mixed |
image passage |
1024x1024 JPEG |
4 |
2 |
20.3 |
20.3 |
20.3 |
20.4 |
99.0 |
|
Mixed |
image passage |
1024x1024 JPEG |
4 |
3 |
30.5 |
30.8 |
30.8 |
30.8 |
99.4 |
|
Mixed |
image passage |
1024x1024 JPEG |
4 |
5 |
50.8 |
51.7 |
52.0 |
52.4 |
101.8 |
|
Mixed |
image passage |
1024x1024 JPEG |
4 |
8 |
80.3 |
83.1 |
83.2 |
83.2 |
108.0 |
|
Mixed |
image passage |
1024x1024 JPEG |
8 |
1 |
11.7 |
11.7 |
11.8 |
11.8 |
85.5 |
|
Mixed |
image passage |
1024x1024 JPEG |
8 |
2 |
20.4 |
20.5 |
20.5 |
20.5 |
98.4 |
|
Mixed |
image passage |
1024x1024 JPEG |
8 |
3 |
30.7 |
30.9 |
31.0 |
31.1 |
98.8 |
|
Mixed |
image passage |
1024x1024 JPEG |
8 |
5 |
51.0 |
51.9 |
52.0 |
52.0 |
101.6 |
|
Mixed |
image passage |
1024x1024 JPEG |
8 |
8 |
80.5 |
83.3 |
83.4 |
83.4 |
107.9 |
|
Mixed |
image passage |
2048x2048 JPEG |
1 |
1 |
14.4 |
14.4 |
15.0 |
15.0 |
69.6 |
|
Mixed |
image passage |
2048x2048 JPEG |
1 |
2 |
18.7 |
18.6 |
18.9 |
18.9 |
107.4 |
|
Mixed |
image passage |
2048x2048 JPEG |
1 |
3 |
29.0 |
29.2 |
29.5 |
29.5 |
104.0 |
|
Mixed |
image passage |
2048x2048 JPEG |
1 |
5 |
49.3 |
50.1 |
50.4 |
50.5 |
103.9 |
|
Mixed |
image passage |
2048x2048 JPEG |
1 |
8 |
79.3 |
81.6 |
82.1 |
82.5 |
107.4 |
|
Mixed |
image passage |
2048x2048 JPEG |
4 |
1 |
13.5 |
13.5 |
13.9 |
14.0 |
74.3 |
|
Mixed |
image passage |
2048x2048 JPEG |
4 |
2 |
20.0 |
20.0 |
20.1 |
20.1 |
100.3 |
|
Mixed |
image passage |
2048x2048 JPEG |
4 |
3 |
30.3 |
30.5 |
30.5 |
30.6 |
99.8 |
|
Mixed |
image passage |
2048x2048 JPEG |
4 |
5 |
50.5 |
51.3 |
51.5 |
51.5 |
101.8 |
|
Mixed |
image passage |
2048x2048 JPEG |
4 |
8 |
80.2 |
82.8 |
82.9 |
82.9 |
107.2 |
|
Mixed |
image passage |
2048x2048 JPEG |
8 |
1 |
13.1 |
13.1 |
13.3 |
13.4 |
76.4 |
|
Mixed |
image passage |
2048x2048 JPEG |
8 |
2 |
20.0 |
20.1 |
20.1 |
20.2 |
100.0 |
|
Mixed |
image passage |
2048x2048 JPEG |
8 |
3 |
30.3 |
30.5 |
30.6 |
30.7 |
99.8 |
|
Mixed |
image passage |
2048x2048 JPEG |
8 |
5 |
50.7 |
51.6 |
51.7 |
52.0 |
101.6 |
|
Mixed |
image passage |
2048x2048 JPEG |
8 |
8 |
80.3 |
82.9 |
83.1 |
83.1 |
106.9 |
NVIDIA B200#
The following table contains the performance data for Llama Nemotron Embed VL 1B v2 on NVIDIA B200.
Precision |
Input Type |
Image Size |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|---|
FP8 |
text |
20 |
1 |
1 |
2.9 |
2.9 |
3.0 |
3.0 |
220.2 |
|
FP8 |
text |
20 |
1 |
3 |
3.9 |
4.3 |
4.9 |
4.9 |
530.0 |
|
FP8 |
text |
20 |
1 |
5 |
4.6 |
4.7 |
6.6 |
6.9 |
793.7 |
|
FP8 |
text |
20 |
1 |
7 |
4.9 |
4.7 |
9.1 |
9.4 |
1042.5 |
|
FP8 |
text |
20 |
1 |
9 |
5.0 |
4.6 |
6.7 |
11.0 |
1315.1 |
|
FP8 |
text |
20 |
1 |
11 |
5.4 |
5.2 |
7.6 |
9.5 |
1476.3 |
|
FP8 |
text |
20 |
1 |
13 |
5.2 |
4.4 |
10.8 |
11.1 |
1871.1 |
|
FP8 |
text |
20 |
1 |
15 |
6.1 |
5.5 |
11.6 |
11.8 |
1786.1 |
|
FP8 |
text |
300 |
64 |
1 |
58.9 |
58.6 |
60.5 |
60.9 |
971.2 |
|
FP8 |
text |
300 |
64 |
3 |
154.6 |
155.8 |
159.4 |
161.0 |
1177.6 |
|
FP8 |
text |
300 |
64 |
5 |
261.1 |
266.3 |
270.2 |
270.9 |
1170.5 |
|
FP8 |
text |
512 |
64 |
1 |
96.4 |
96.4 |
97.3 |
97.5 |
618.8 |
|
FP8 |
text |
512 |
64 |
3 |
267.0 |
269.6 |
271.0 |
271.5 |
693.8 |
|
FP8 |
text |
512 |
64 |
5 |
445.3 |
453.7 |
457.5 |
457.8 |
693.2 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
1 |
15.3 |
15.3 |
15.4 |
15.4 |
65.2 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
2 |
23.7 |
22.0 |
27.1 |
37.3 |
87.0 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
3 |
33.8 |
34.0 |
34.2 |
34.3 |
89.5 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
5 |
56.8 |
57.7 |
58.0 |
58.1 |
90.5 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
8 |
90.4 |
93.4 |
93.7 |
93.8 |
94.6 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
1 |
12.9 |
12.9 |
12.9 |
12.9 |
77.7 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
2 |
23.1 |
23.2 |
23.3 |
23.3 |
86.8 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
3 |
36.2 |
35.1 |
35.2 |
35.4 |
85.7 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
5 |
57.9 |
59.0 |
59.0 |
59.0 |
89.7 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
8 |
91.4 |
94.6 |
94.7 |
94.7 |
95.3 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
1 |
12.5 |
12.5 |
12.5 |
12.5 |
80.2 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
2 |
23.4 |
23.5 |
23.5 |
23.5 |
86.0 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
3 |
35.0 |
35.3 |
35.4 |
35.4 |
86.8 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
5 |
58.6 |
59.1 |
59.1 |
63.2 |
88.9 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
8 |
91.5 |
94.8 |
94.8 |
94.8 |
95.5 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
1 |
15.5 |
15.5 |
15.6 |
15.6 |
64.6 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
2 |
22.1 |
22.1 |
22.3 |
22.5 |
90.6 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
3 |
34.0 |
34.2 |
34.5 |
34.6 |
88.8 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
5 |
59.9 |
58.0 |
59.1 |
76.1 |
87.5 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
8 |
90.7 |
93.6 |
94.0 |
94.1 |
94.3 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
1 |
13.0 |
13.0 |
13.1 |
13.1 |
76.8 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
2 |
23.2 |
23.3 |
23.3 |
23.3 |
86.6 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
3 |
34.9 |
35.2 |
35.2 |
35.2 |
87.0 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
5 |
60.1 |
59.0 |
59.1 |
79.0 |
87.6 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
8 |
92.4 |
94.8 |
96.4 |
105.1 |
94.5 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
1 |
12.7 |
12.6 |
12.7 |
12.7 |
78.8 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
2 |
23.8 |
23.5 |
23.5 |
23.5 |
84.9 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
3 |
35.4 |
35.4 |
35.5 |
39.0 |
85.9 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
5 |
58.1 |
59.2 |
59.2 |
59.3 |
89.5 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
8 |
91.7 |
95.0 |
95.1 |
95.2 |
95.2 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
1 |
52.8 |
52.7 |
53.0 |
53.0 |
19.0 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
2 |
95.9 |
95.6 |
95.9 |
96.2 |
21.0 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
3 |
144.7 |
144.4 |
144.8 |
150.3 |
21.0 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
5 |
237.9 |
242.2 |
244.5 |
244.9 |
21.8 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
8 |
374.9 |
388.0 |
388.5 |
388.7 |
23.2 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
1 |
50.5 |
50.3 |
50.4 |
50.5 |
19.8 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
2 |
96.8 |
96.8 |
98.0 |
98.1 |
20.8 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
3 |
145.9 |
145.6 |
146.9 |
156.8 |
20.9 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
5 |
240.7 |
243.0 |
250.3 |
257.5 |
21.6 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
8 |
380.2 |
389.4 |
399.9 |
428.6 |
23.1 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
1 |
50.2 |
49.9 |
50.0 |
50.4 |
19.9 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
2 |
97.2 |
97.0 |
98.4 |
98.8 |
20.7 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
3 |
145.1 |
145.7 |
146.1 |
151.0 |
21.0 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
5 |
240.2 |
243.3 |
248.7 |
251.2 |
21.7 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
8 |
378.5 |
389.7 |
395.7 |
408.4 |
23.2 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
1 |
54.3 |
54.3 |
55.0 |
55.0 |
18.4 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
2 |
98.7 |
96.0 |
96.3 |
96.4 |
20.7 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
3 |
143.8 |
145.1 |
145.4 |
145.4 |
21.1 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
5 |
238.8 |
243.3 |
243.7 |
243.8 |
21.7 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
8 |
379.3 |
390.9 |
391.3 |
391.7 |
23.0 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
1 |
52.0 |
51.8 |
52.2 |
52.3 |
19.2 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
2 |
97.0 |
97.2 |
97.4 |
97.9 |
20.7 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
3 |
146.6 |
146.5 |
148.1 |
148.4 |
20.8 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
5 |
243.4 |
244.7 |
249.7 |
260.2 |
21.4 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
8 |
379.3 |
392.1 |
394.5 |
399.7 |
23.0 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
1 |
51.7 |
51.5 |
51.8 |
51.9 |
19.4 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
2 |
97.0 |
97.3 |
97.5 |
99.1 |
20.7 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
3 |
145.8 |
146.5 |
147.6 |
149.1 |
20.9 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
5 |
240.9 |
244.7 |
247.6 |
248.2 |
21.6 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
8 |
380.6 |
392.3 |
396.6 |
409.9 |
23.0 |
NVIDIA H100 80GB HBM3#
The following table contains the performance data for Llama Nemotron Embed VL 1B v2 on NVIDIA H100 80GB HBM3.
Precision |
Input Type |
Image Size |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|---|
FP8 |
text |
20 |
1 |
1 |
3.5 |
3.5 |
3.6 |
3.7 |
193.8 |
|
FP8 |
text |
20 |
1 |
3 |
5.1 |
5.3 |
6.1 |
6.4 |
436.0 |
|
FP8 |
text |
20 |
1 |
5 |
5.2 |
5.3 |
6.2 |
6.3 |
710.2 |
|
FP8 |
text |
20 |
1 |
7 |
5.6 |
5.4 |
6.1 |
11.2 |
933.0 |
|
FP8 |
text |
20 |
1 |
9 |
7.1 |
5.6 |
14.7 |
16.0 |
999.3 |
|
FP8 |
text |
20 |
1 |
11 |
6.8 |
6.0 |
10.5 |
12.6 |
1218.5 |
|
FP8 |
text |
20 |
1 |
13 |
9.8 |
8.0 |
16.8 |
19.4 |
1099.3 |
|
FP8 |
text |
20 |
1 |
15 |
6.3 |
6.1 |
9.0 |
9.4 |
1809.5 |
|
FP8 |
text |
300 |
64 |
1 |
72.0 |
71.8 |
73.1 |
73.5 |
805.6 |
|
FP8 |
text |
300 |
64 |
3 |
193.1 |
194.7 |
196.5 |
196.8 |
947.6 |
|
FP8 |
text |
300 |
64 |
5 |
323.1 |
329.2 |
331.0 |
331.5 |
948.1 |
|
FP8 |
text |
512 |
64 |
1 |
122.6 |
122.5 |
123.8 |
124.1 |
491.6 |
|
FP8 |
text |
512 |
64 |
3 |
344.8 |
347.7 |
350.9 |
351.6 |
539.1 |
|
FP8 |
text |
512 |
64 |
5 |
571.6 |
582.8 |
585.8 |
586.3 |
541.1 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
1 |
17.9 |
17.6 |
18.7 |
19.3 |
56.0 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
2 |
27.4 |
27.0 |
28.9 |
29.9 |
73.3 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
3 |
41.6 |
41.2 |
44.1 |
46.2 |
72.9 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
5 |
69.7 |
70.1 |
73.5 |
74.2 |
73.9 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
8 |
111.0 |
113.5 |
118.9 |
123.5 |
77.7 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
1 |
15.6 |
15.3 |
16.5 |
17.0 |
64.1 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
2 |
28.6 |
28.1 |
30.6 |
30.7 |
70.4 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
3 |
42.8 |
42.6 |
44.9 |
45.9 |
71.0 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
5 |
70.2 |
71.2 |
73.2 |
73.6 |
74.0 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
8 |
111.9 |
115.1 |
119.0 |
122.3 |
78.2 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
1 |
15.1 |
14.9 |
15.8 |
16.5 |
66.3 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
2 |
28.7 |
28.6 |
30.2 |
31.0 |
70.1 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
3 |
42.9 |
43.0 |
44.3 |
44.6 |
70.9 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
5 |
70.7 |
72.0 |
73.5 |
73.5 |
73.7 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
8 |
113.3 |
116.2 |
120.0 |
127.6 |
77.4 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
1 |
18.2 |
17.8 |
19.2 |
20.5 |
55.1 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
2 |
27.6 |
27.1 |
29.6 |
30.9 |
72.8 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
3 |
42.3 |
41.5 |
46.1 |
49.3 |
71.7 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
5 |
69.6 |
70.1 |
72.6 |
76.8 |
73.6 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
8 |
113.5 |
115.2 |
124.8 |
125.2 |
76.3 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
1 |
15.9 |
15.5 |
16.5 |
16.8 |
63.3 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
2 |
28.7 |
28.4 |
29.9 |
30.4 |
70.2 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
3 |
42.5 |
42.7 |
44.2 |
44.4 |
71.4 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
5 |
70.9 |
72.2 |
74.1 |
74.3 |
72.7 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
8 |
111.8 |
115.3 |
117.6 |
118.2 |
77.4 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
1 |
15.3 |
15.1 |
15.9 |
16.1 |
65.6 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
2 |
28.8 |
28.7 |
29.8 |
30.0 |
69.7 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
3 |
43.1 |
43.5 |
44.7 |
45.5 |
70.5 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
5 |
71.5 |
72.3 |
74.2 |
75.1 |
72.7 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
8 |
112.6 |
116.5 |
118.4 |
118.7 |
77.4 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
1 |
64.7 |
64.0 |
66.8 |
67.7 |
15.5 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
2 |
118.6 |
117.9 |
122.3 |
124.6 |
16.9 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
3 |
178.5 |
179.7 |
184.9 |
185.7 |
17.0 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
5 |
296.6 |
300.1 |
308.9 |
313.2 |
17.4 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
8 |
470.5 |
482.9 |
495.8 |
516.7 |
18.4 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
1 |
62.2 |
61.6 |
62.8 |
63.0 |
16.1 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
2 |
120.6 |
120.3 |
121.7 |
126.5 |
16.7 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
3 |
179.4 |
181.0 |
182.6 |
183.5 |
17.0 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
5 |
296.9 |
301.8 |
304.3 |
306.1 |
17.6 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
8 |
469.0 |
484.2 |
497.5 |
498.9 |
18.7 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
1 |
62.0 |
62.0 |
62.5 |
62.8 |
16.1 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
2 |
120.0 |
120.5 |
121.7 |
122.3 |
16.7 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
3 |
179.8 |
181.2 |
182.1 |
187.6 |
16.9 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
5 |
296.4 |
302.2 |
303.6 |
304.9 |
17.6 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
8 |
468.1 |
484.3 |
486.2 |
493.5 |
18.7 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
1 |
66.3 |
66.1 |
68.3 |
68.8 |
15.1 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
2 |
118.4 |
118.2 |
121.1 |
124.6 |
17.0 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
3 |
178.7 |
179.4 |
183.0 |
186.0 |
17.0 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
5 |
296.4 |
301.7 |
306.3 |
306.9 |
17.5 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
8 |
469.7 |
486.0 |
491.8 |
492.2 |
18.6 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
1 |
63.7 |
63.5 |
64.7 |
65.2 |
15.7 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
2 |
120.5 |
120.7 |
122.4 |
124.4 |
16.7 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
3 |
180.2 |
181.8 |
183.8 |
184.9 |
16.9 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
5 |
298.1 |
303.7 |
306.0 |
306.4 |
17.4 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
8 |
471.1 |
486.3 |
488.7 |
489.3 |
18.2 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
1 |
63.4 |
63.3 |
63.9 |
64.2 |
15.8 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
2 |
119.6 |
120.0 |
120.7 |
121.5 |
16.8 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
3 |
180.1 |
181.1 |
183.4 |
184.1 |
16.9 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
5 |
297.7 |
302.6 |
309.0 |
309.9 |
17.5 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
8 |
469.8 |
485.5 |
487.6 |
498.2 |
18.6 |
NVIDIA A100 SXM4 80GB#
The following table contains the performance data for Llama Nemotron Embed VL 1B v2 on NVIDIA A100 SXM4 80GB.
Precision |
Input Type |
Image Size |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|---|
FP8 |
text |
20 |
1 |
1 |
3.5 |
3.4 |
3.8 |
3.8 |
229.4 |
|
FP8 |
text |
20 |
1 |
3 |
5.7 |
5.8 |
6.2 |
6.2 |
439.6 |
|
FP8 |
text |
20 |
1 |
5 |
6.7 |
6.7 |
7.1 |
7.4 |
624.6 |
|
FP8 |
text |
20 |
1 |
7 |
6.7 |
6.6 |
8.2 |
8.4 |
872.2 |
|
FP8 |
text |
20 |
1 |
9 |
9.8 |
10.0 |
10.5 |
10.6 |
773.6 |
|
FP8 |
text |
20 |
1 |
11 |
10.1 |
10.1 |
11.1 |
12.2 |
923.5 |
|
FP8 |
text |
20 |
1 |
13 |
9.0 |
9.0 |
10.9 |
11.3 |
1170.8 |
|
FP8 |
text |
20 |
1 |
15 |
11.5 |
11.7 |
12.6 |
13.8 |
1073.4 |
|
FP8 |
text |
300 |
64 |
1 |
382.2 |
382.1 |
384.0 |
384.7 |
163.4 |
|
FP8 |
text |
300 |
64 |
3 |
1100.1 |
1110.5 |
1114.5 |
1115.5 |
171.2 |
|
FP8 |
text |
300 |
64 |
5 |
1822.5 |
1859.4 |
1863.0 |
1863.9 |
171.2 |
|
FP8 |
text |
512 |
64 |
1 |
840.2 |
839.6 |
848.1 |
849.3 |
75.2 |
|
FP8 |
text |
512 |
64 |
3 |
2459.8 |
2484.6 |
2496.9 |
2498.0 |
77.0 |
|
FP8 |
text |
512 |
64 |
5 |
4064.9 |
4146.5 |
4163.6 |
4167.8 |
77.0 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
1 |
34.5 |
34.3 |
34.7 |
35.5 |
29.0 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
2 |
64.9 |
64.8 |
66.7 |
68.2 |
30.9 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
3 |
97.3 |
97.6 |
100.3 |
102.8 |
31.2 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
5 |
161.7 |
163.2 |
169.7 |
171.1 |
32.2 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
8 |
254.7 |
261.6 |
268.2 |
269.5 |
33.9 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
1 |
33.8 |
33.5 |
35.2 |
35.7 |
29.6 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
2 |
65.6 |
65.4 |
67.8 |
68.7 |
30.7 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
3 |
97.8 |
98.1 |
100.9 |
101.5 |
31.1 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
5 |
162.1 |
164.3 |
168.2 |
169.5 |
32.1 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
8 |
255.1 |
264.1 |
266.1 |
267.5 |
34.4 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
1 |
33.6 |
33.3 |
34.5 |
35.1 |
29.8 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
2 |
65.6 |
65.4 |
67.2 |
67.9 |
30.6 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
3 |
98.2 |
98.7 |
101.0 |
101.4 |
31.0 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
5 |
162.4 |
165.6 |
167.4 |
167.7 |
32.1 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
8 |
255.5 |
264.8 |
265.5 |
265.8 |
34.2 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
1 |
34.7 |
34.5 |
35.6 |
36.2 |
28.8 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
2 |
65.1 |
64.7 |
67.3 |
68.9 |
30.9 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
3 |
97.9 |
97.5 |
102.7 |
109.0 |
31.1 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
5 |
161.9 |
163.1 |
169.5 |
171.3 |
31.9 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
8 |
255.1 |
261.4 |
274.1 |
276.0 |
34.2 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
1 |
33.9 |
33.7 |
34.8 |
35.2 |
29.5 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
2 |
65.3 |
65.2 |
66.9 |
67.4 |
30.8 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
3 |
97.8 |
98.2 |
100.9 |
101.6 |
31.1 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
5 |
162.0 |
164.5 |
167.7 |
168.0 |
32.2 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
8 |
256.9 |
265.9 |
269.7 |
270.0 |
34.1 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
1 |
33.7 |
33.5 |
34.6 |
35.0 |
29.7 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
2 |
65.5 |
65.4 |
67.2 |
67.3 |
30.7 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
3 |
98.0 |
98.5 |
100.3 |
100.8 |
31.0 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
5 |
162.4 |
165.6 |
167.4 |
167.6 |
32.1 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
8 |
255.4 |
264.1 |
265.5 |
265.6 |
34.1 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
1 |
181.2 |
180.5 |
182.2 |
186.8 |
5.5 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
2 |
359.8 |
360.3 |
368.4 |
369.0 |
5.6 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
3 |
538.6 |
541.1 |
552.1 |
552.9 |
5.7 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
5 |
888.7 |
904.5 |
914.1 |
914.6 |
5.9 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
8 |
1400.0 |
1451.2 |
1458.1 |
1458.5 |
6.3 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
1 |
181.5 |
181.1 |
183.0 |
184.1 |
5.5 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
2 |
361.0 |
363.3 |
364.1 |
364.4 |
5.6 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
3 |
539.2 |
544.4 |
546.7 |
547.0 |
5.6 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
5 |
890.1 |
908.6 |
910.1 |
910.4 |
5.9 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
8 |
1402.7 |
1453.7 |
1454.6 |
1454.9 |
6.3 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
1 |
182.0 |
181.6 |
183.1 |
183.6 |
5.5 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
2 |
361.2 |
362.8 |
364.1 |
364.7 |
5.6 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
3 |
539.3 |
544.6 |
545.6 |
545.9 |
5.6 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
5 |
890.1 |
908.2 |
909.1 |
909.4 |
5.9 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
8 |
1403.2 |
1454.2 |
1454.7 |
1454.9 |
6.3 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
1 |
181.5 |
180.5 |
183.6 |
186.9 |
5.5 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
2 |
359.2 |
359.5 |
366.5 |
369.9 |
5.6 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
3 |
536.7 |
540.4 |
548.6 |
550.4 |
5.7 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
5 |
1006.9 |
910.8 |
933.4 |
1871.2 |
5.6 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
8 |
1400.5 |
1450.3 |
1456.7 |
1457.6 |
6.3 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
1 |
183.7 |
183.3 |
185.3 |
186.1 |
5.4 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
2 |
360.4 |
362.2 |
363.3 |
363.9 |
5.6 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
3 |
538.4 |
543.7 |
545.1 |
545.8 |
5.7 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
5 |
889.5 |
907.7 |
908.7 |
908.9 |
5.9 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
8 |
1401.9 |
1452.9 |
1454.0 |
1454.5 |
6.3 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
1 |
184.0 |
183.7 |
185.4 |
185.9 |
5.4 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
2 |
360.4 |
362.1 |
363.1 |
363.2 |
5.6 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
3 |
538.6 |
543.8 |
545.0 |
545.3 |
5.7 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
5 |
889.5 |
907.5 |
908.5 |
909.2 |
5.9 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
8 |
1402.2 |
1453.3 |
1454.1 |
1454.3 |
6.3 |
NVIDIA L40S#
The following table contains the performance data for Llama Nemotron Embed VL 1B v2 on NVIDIA L40S.
Precision |
Input Type |
Image Size |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|---|
FP8 |
text |
20 |
1 |
1 |
4.8 |
4.8 |
4.9 |
4.9 |
157.6 |
|
FP8 |
text |
20 |
1 |
3 |
6.6 |
6.7 |
6.9 |
7.0 |
361.0 |
|
FP8 |
text |
20 |
1 |
5 |
8.6 |
8.7 |
9.0 |
9.0 |
480.0 |
|
FP8 |
text |
20 |
1 |
7 |
6.8 |
6.9 |
7.0 |
7.0 |
796.8 |
|
FP8 |
text |
20 |
1 |
9 |
8.9 |
9.0 |
9.4 |
9.6 |
808.3 |
|
FP8 |
text |
20 |
1 |
11 |
7.9 |
8.0 |
8.3 |
9.2 |
1124.1 |
|
FP8 |
text |
20 |
1 |
13 |
8.2 |
8.5 |
9.5 |
9.8 |
1256.2 |
|
FP8 |
text |
20 |
1 |
15 |
8.8 |
8.8 |
9.8 |
9.9 |
1354.3 |
|
FP8 |
text |
300 |
64 |
1 |
233.0 |
232.9 |
233.7 |
233.9 |
267.0 |
|
FP8 |
text |
300 |
64 |
3 |
675.3 |
681.5 |
685.2 |
687.5 |
278.7 |
|
FP8 |
text |
300 |
64 |
5 |
1117.7 |
1139.9 |
1145.5 |
1147.4 |
278.9 |
|
FP8 |
text |
512 |
64 |
1 |
394.7 |
394.3 |
399.1 |
399.5 |
159.3 |
|
FP8 |
text |
512 |
64 |
3 |
1164.2 |
1175.8 |
1181.4 |
1183.7 |
162.3 |
|
FP8 |
text |
512 |
64 |
5 |
1927.5 |
1967.5 |
1974.9 |
1976.5 |
162.1 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
1 |
26.6 |
26.6 |
26.7 |
26.7 |
37.6 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
2 |
45.2 |
45.3 |
45.5 |
45.7 |
44.4 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
3 |
68.2 |
68.8 |
68.9 |
68.9 |
44.4 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
5 |
113.7 |
115.8 |
115.9 |
116.0 |
45.5 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
8 |
180.1 |
186.3 |
186.4 |
186.5 |
48.1 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
1 |
24.4 |
24.4 |
24.4 |
24.5 |
41.0 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
2 |
46.3 |
46.5 |
46.6 |
46.7 |
43.3 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
3 |
69.4 |
70.1 |
70.1 |
70.1 |
43.8 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
5 |
114.9 |
117.1 |
117.2 |
117.2 |
45.3 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
8 |
181.4 |
187.8 |
188.2 |
189.3 |
48.2 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
1 |
24.1 |
24.1 |
24.1 |
24.1 |
41.5 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
2 |
46.6 |
46.8 |
46.8 |
46.9 |
43.1 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
3 |
69.7 |
70.4 |
70.4 |
70.5 |
43.6 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
5 |
115.2 |
117.5 |
117.5 |
117.5 |
45.2 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
8 |
181.6 |
188.1 |
188.2 |
188.2 |
48.2 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
1 |
26.7 |
26.7 |
26.8 |
26.9 |
37.4 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
2 |
45.2 |
45.3 |
45.5 |
45.5 |
44.4 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
3 |
68.2 |
68.8 |
69.0 |
69.0 |
44.4 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
5 |
113.8 |
115.9 |
116.0 |
116.1 |
45.5 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
8 |
180.2 |
186.4 |
186.5 |
186.6 |
48.1 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
1 |
24.5 |
24.5 |
24.6 |
24.6 |
40.8 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
2 |
46.3 |
46.5 |
46.6 |
46.6 |
43.3 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
3 |
69.4 |
70.1 |
70.1 |
70.2 |
43.8 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
5 |
115.1 |
117.1 |
117.4 |
118.7 |
45.2 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
8 |
181.3 |
187.8 |
187.9 |
187.9 |
48.2 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
1 |
24.2 |
24.2 |
24.3 |
24.3 |
41.3 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
2 |
46.6 |
46.8 |
46.8 |
46.8 |
43.1 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
3 |
69.7 |
70.3 |
70.4 |
70.4 |
43.6 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
5 |
115.3 |
117.4 |
118.0 |
118.3 |
45.2 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
8 |
182.0 |
188.1 |
188.7 |
192.2 |
48.1 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
1 |
123.1 |
123.1 |
123.7 |
123.8 |
8.1 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
2 |
236.1 |
237.2 |
237.9 |
238.1 |
8.5 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
3 |
353.7 |
357.2 |
358.0 |
358.2 |
8.6 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
5 |
584.8 |
596.5 |
597.9 |
598.1 |
8.9 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
8 |
922.0 |
954.9 |
956.9 |
957.1 |
9.5 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
1 |
121.0 |
121.0 |
121.2 |
121.3 |
8.3 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
2 |
237.8 |
239.1 |
239.3 |
239.4 |
8.4 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
3 |
355.6 |
359.1 |
359.3 |
360.3 |
8.6 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
5 |
586.9 |
598.8 |
599.1 |
599.5 |
8.9 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
8 |
924.9 |
958.5 |
958.9 |
960.6 |
9.5 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
1 |
120.8 |
120.8 |
121.0 |
121.0 |
8.3 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
2 |
238.2 |
239.4 |
239.5 |
239.5 |
8.4 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
3 |
355.6 |
359.3 |
359.4 |
359.5 |
8.6 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
5 |
587.6 |
599.3 |
601.1 |
601.9 |
8.9 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
8 |
925.4 |
959.1 |
959.3 |
959.7 |
9.5 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
1 |
125.3 |
125.1 |
125.8 |
125.9 |
8.0 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
2 |
236.4 |
237.6 |
238.0 |
238.0 |
8.5 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
3 |
354.0 |
357.5 |
358.1 |
358.2 |
8.6 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
5 |
585.4 |
597.3 |
597.9 |
597.9 |
8.9 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
8 |
923.9 |
957.0 |
958.0 |
958.2 |
9.5 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
1 |
122.4 |
122.5 |
122.7 |
122.9 |
8.2 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
2 |
237.7 |
238.9 |
239.1 |
239.1 |
8.5 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
3 |
355.3 |
358.9 |
359.1 |
359.1 |
8.6 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
5 |
587.2 |
598.8 |
599.1 |
601.8 |
8.9 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
8 |
925.5 |
958.6 |
959.7 |
964.1 |
9.5 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
1 |
121.9 |
121.8 |
122.4 |
122.5 |
8.2 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
2 |
238.0 |
239.0 |
239.2 |
239.3 |
8.4 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
3 |
355.5 |
359.0 |
359.2 |
359.3 |
8.6 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
5 |
587.0 |
598.8 |
599.0 |
599.0 |
8.9 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
8 |
925.3 |
958.7 |
958.8 |
958.9 |
9.5 |
NVIDIA L4#
The following table contains the performance data for Llama Nemotron Embed VL 1B v2 on NVIDIA L4.
Precision |
Input Type |
Image Size |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|---|
FP8 |
text |
20 |
1 |
1 |
10.2 |
10.1 |
10.2 |
10.3 |
88.8 |
|
FP8 |
text |
20 |
1 |
3 |
17.6 |
18.1 |
18.5 |
18.6 |
157.7 |
|
FP8 |
text |
20 |
1 |
5 |
19.5 |
19.2 |
19.9 |
21.6 |
236.2 |
|
FP8 |
text |
20 |
1 |
7 |
20.1 |
20.2 |
21.2 |
21.5 |
319.3 |
|
FP8 |
text |
20 |
1 |
9 |
21.1 |
21.3 |
22.5 |
22.7 |
380.9 |
|
FP8 |
text |
20 |
1 |
11 |
22.4 |
22.7 |
23.9 |
24.4 |
436.0 |
|
FP8 |
text |
20 |
1 |
13 |
31.4 |
32.2 |
41.5 |
42.1 |
378.3 |
|
FP8 |
text |
20 |
1 |
15 |
37.3 |
36.8 |
46.2 |
46.9 |
366.9 |
|
FP8 |
text |
300 |
64 |
1 |
1096.9 |
1098.2 |
1106.0 |
1108.7 |
57.9 |
|
FP8 |
text |
300 |
64 |
3 |
3239.8 |
3271.2 |
3291.7 |
3293.9 |
58.5 |
|
FP8 |
text |
300 |
64 |
5 |
5352.3 |
5464.2 |
5478.1 |
5481.6 |
58.5 |
|
FP8 |
text |
512 |
64 |
1 |
2159.2 |
2157.8 |
2182.8 |
2191.2 |
29.5 |
|
FP8 |
text |
512 |
64 |
3 |
6405.2 |
6471.2 |
6501.6 |
6507.9 |
29.6 |
|
FP8 |
text |
512 |
64 |
5 |
10591.4 |
10810.3 |
10853.6 |
10862.4 |
29.6 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
1 |
85.3 |
85.0 |
87.0 |
87.1 |
11.7 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
2 |
167.1 |
168.0 |
169.5 |
169.9 |
12.0 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
3 |
250.5 |
252.5 |
254.2 |
254.6 |
12.2 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
5 |
412.9 |
421.1 |
423.1 |
423.9 |
12.6 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
8 |
655.7 |
679.3 |
681.0 |
681.7 |
13.4 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
1 |
85.2 |
85.2 |
85.5 |
85.7 |
11.7 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
2 |
168.6 |
169.4 |
169.7 |
169.9 |
11.9 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
3 |
251.8 |
254.3 |
254.6 |
254.7 |
12.1 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
5 |
416.1 |
424.4 |
425.0 |
425.8 |
12.5 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
8 |
655.8 |
679.5 |
680.1 |
680.3 |
13.4 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
1 |
85.3 |
85.3 |
85.5 |
85.7 |
11.7 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
2 |
169.6 |
170.5 |
170.8 |
171.0 |
11.9 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
3 |
252.4 |
254.9 |
255.4 |
255.5 |
12.1 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
5 |
416.7 |
425.2 |
425.7 |
425.9 |
12.5 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
8 |
658.8 |
682.8 |
683.6 |
683.8 |
13.3 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
1 |
86.6 |
86.5 |
88.0 |
88.6 |
11.6 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
2 |
167.4 |
168.3 |
170.0 |
170.3 |
12.0 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
3 |
250.2 |
252.6 |
254.3 |
254.5 |
12.2 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
5 |
414.1 |
422.4 |
424.1 |
424.4 |
12.6 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
8 |
653.6 |
677.0 |
678.4 |
678.8 |
13.4 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
1 |
85.4 |
85.4 |
85.8 |
85.9 |
11.7 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
2 |
168.4 |
169.3 |
169.5 |
169.6 |
11.9 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
3 |
251.7 |
254.3 |
254.6 |
254.8 |
12.1 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
5 |
415.7 |
424.1 |
424.7 |
425.0 |
12.5 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
8 |
655.3 |
679.0 |
679.5 |
679.6 |
13.4 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
1 |
86.1 |
86.1 |
86.5 |
86.6 |
11.6 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
2 |
168.8 |
169.6 |
169.9 |
170.0 |
11.9 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
3 |
252.2 |
254.7 |
255.1 |
255.2 |
12.1 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
5 |
417.7 |
426.4 |
426.8 |
427.1 |
12.5 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
8 |
656.9 |
680.7 |
682.4 |
682.6 |
13.4 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
1 |
457.4 |
457.6 |
458.9 |
459.3 |
2.2 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
2 |
902.1 |
906.8 |
909.6 |
909.8 |
2.2 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
3 |
1348.7 |
1362.4 |
1365.4 |
1365.8 |
2.3 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
5 |
2223.0 |
2268.0 |
2272.1 |
2272.9 |
2.3 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
8 |
3508.4 |
3634.6 |
3641.6 |
3643.4 |
2.5 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
1 |
456.6 |
456.5 |
457.6 |
457.7 |
2.2 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
2 |
903.8 |
908.1 |
909.6 |
910.0 |
2.2 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
3 |
1348.2 |
1361.7 |
1363.1 |
1363.5 |
2.3 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
5 |
2225.1 |
2270.6 |
2272.7 |
2273.1 |
2.3 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
8 |
3506.3 |
3633.0 |
3636.6 |
3637.4 |
2.5 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
1 |
456.0 |
456.0 |
456.6 |
456.7 |
2.2 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
2 |
903.3 |
907.9 |
908.7 |
908.9 |
2.2 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
3 |
1349.3 |
1362.8 |
1364.2 |
1364.4 |
2.3 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
5 |
2226.2 |
2271.5 |
2273.4 |
2273.7 |
2.3 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
8 |
3507.6 |
3634.5 |
3637.3 |
3637.8 |
2.5 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
1 |
459.5 |
459.2 |
461.4 |
461.8 |
2.2 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
2 |
899.8 |
904.0 |
907.0 |
907.5 |
2.2 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
3 |
1345.5 |
1358.8 |
1361.8 |
1362.2 |
2.3 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
5 |
2219.2 |
2264.1 |
2268.1 |
2269.1 |
2.3 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
8 |
3500.3 |
3626.5 |
3632.9 |
3633.8 |
2.5 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
1 |
457.0 |
457.0 |
457.9 |
458.2 |
2.2 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
2 |
902.4 |
906.8 |
908.2 |
908.8 |
2.2 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
3 |
1346.7 |
1360.3 |
1362.2 |
1362.3 |
2.3 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
5 |
2224.7 |
2269.9 |
2272.8 |
2273.3 |
2.3 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
8 |
3505.9 |
3632.6 |
3636.4 |
3636.7 |
2.5 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
1 |
456.6 |
456.6 |
457.5 |
457.7 |
2.2 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
2 |
903.0 |
907.6 |
908.7 |
908.8 |
2.2 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
3 |
1347.9 |
1361.6 |
1362.8 |
1363.0 |
2.3 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
5 |
2225.5 |
2270.8 |
2272.4 |
2272.7 |
2.3 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
8 |
3503.6 |
3630.4 |
3633.7 |
3634.6 |
2.5 |
NVIDIA A10G#
The following table contains the performance data for Llama Nemotron Embed VL 1B v2 on NVIDIA A10G.
Precision |
Input Type |
Image Size |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|---|
FP8 |
text |
20 |
1 |
1 |
6.9 |
7.0 |
7.2 |
7.3 |
124.9 |
|
FP8 |
text |
20 |
1 |
3 |
10.9 |
10.9 |
11.2 |
11.3 |
249.3 |
|
FP8 |
text |
20 |
1 |
5 |
11.8 |
11.7 |
12.3 |
12.6 |
380.2 |
|
FP8 |
text |
20 |
1 |
7 |
17.6 |
17.6 |
19.5 |
19.7 |
357.7 |
|
FP8 |
text |
20 |
1 |
9 |
14.9 |
15.1 |
15.6 |
16.9 |
532.9 |
|
FP8 |
text |
20 |
1 |
11 |
16.5 |
16.4 |
17.8 |
19.1 |
586.1 |
|
FP8 |
text |
20 |
1 |
13 |
21.7 |
23.0 |
24.0 |
24.2 |
534.9 |
|
FP8 |
text |
20 |
1 |
15 |
21.4 |
22.1 |
23.5 |
23.6 |
616.3 |
|
FP8 |
text |
300 |
64 |
1 |
951.5 |
951.6 |
954.7 |
955.1 |
66.6 |
|
FP8 |
text |
300 |
64 |
3 |
2787.7 |
2814.7 |
2822.9 |
2824.8 |
67.9 |
|
FP8 |
text |
300 |
64 |
5 |
4606.7 |
4700.2 |
4706.6 |
4708.7 |
67.9 |
|
FP8 |
text |
512 |
64 |
1 |
1806.0 |
1804.9 |
1819.4 |
1821.3 |
35.2 |
|
FP8 |
text |
512 |
64 |
3 |
5322.8 |
5377.1 |
5394.6 |
5397.8 |
35.6 |
|
FP8 |
text |
512 |
64 |
5 |
8788.5 |
8965.8 |
8996.8 |
9001.3 |
35.6 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
1 |
72.7 |
72.7 |
72.9 |
72.9 |
13.8 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
2 |
138.0 |
138.6 |
138.8 |
139.0 |
14.6 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
3 |
206.6 |
208.6 |
208.9 |
209.0 |
14.7 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
5 |
342.0 |
348.4 |
349.1 |
352.5 |
15.2 |
|
FP8 |
image passage |
336x336 JPEG |
1 |
8 |
539.0 |
558.3 |
558.6 |
558.7 |
16.2 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
1 |
70.9 |
70.9 |
71.0 |
71.0 |
14.1 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
2 |
138.9 |
139.4 |
139.6 |
139.8 |
14.5 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
3 |
207.3 |
209.3 |
209.5 |
209.6 |
14.7 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
5 |
342.6 |
349.2 |
349.5 |
351.1 |
15.2 |
|
FP8 |
image passage |
336x336 JPEG |
4 |
8 |
539.9 |
559.1 |
559.5 |
562.8 |
16.2 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
1 |
70.6 |
70.5 |
70.6 |
70.7 |
14.2 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
2 |
138.9 |
139.5 |
139.6 |
139.7 |
14.5 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
3 |
207.6 |
209.5 |
209.7 |
210.9 |
14.7 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
5 |
342.6 |
349.4 |
349.5 |
350.5 |
15.2 |
|
FP8 |
image passage |
336x336 JPEG |
8 |
8 |
540.0 |
559.3 |
559.4 |
561.9 |
16.2 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
1 |
73.2 |
72.9 |
73.6 |
74.0 |
13.7 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
2 |
137.7 |
138.4 |
138.6 |
138.8 |
14.6 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
3 |
206.9 |
208.3 |
208.9 |
209.3 |
14.7 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
5 |
341.6 |
348.3 |
349.0 |
349.7 |
15.2 |
|
FP8 |
image passage |
512x512 JPEG |
1 |
8 |
538.8 |
558.0 |
558.4 |
558.5 |
16.2 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
1 |
71.1 |
71.1 |
71.3 |
71.4 |
14.1 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
2 |
138.8 |
139.3 |
139.5 |
139.5 |
14.5 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
3 |
207.3 |
209.3 |
209.5 |
209.6 |
14.7 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
5 |
342.3 |
349.2 |
349.4 |
349.5 |
15.2 |
|
FP8 |
image passage |
512x512 JPEG |
4 |
8 |
539.6 |
559.0 |
559.3 |
559.3 |
16.2 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
1 |
70.7 |
70.7 |
70.8 |
70.8 |
14.1 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
2 |
138.8 |
139.5 |
139.6 |
139.6 |
14.5 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
3 |
207.4 |
209.5 |
209.6 |
209.8 |
14.7 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
5 |
342.4 |
349.3 |
349.5 |
350.3 |
15.2 |
|
FP8 |
image passage |
512x512 JPEG |
8 |
8 |
539.9 |
559.2 |
559.4 |
561.0 |
16.2 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
1 |
371.7 |
371.6 |
372.1 |
372.5 |
2.7 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
2 |
729.0 |
732.5 |
733.5 |
733.9 |
2.8 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
3 |
1089.1 |
1099.5 |
1101.5 |
1103.1 |
2.8 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
5 |
1799.1 |
1835.2 |
1837.1 |
1838.6 |
2.9 |
|
FP8 |
image passage |
1024x1024 JPEG |
1 |
8 |
2834.2 |
2935.6 |
2940.2 |
2942.3 |
3.1 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
1 |
369.2 |
369.1 |
369.5 |
369.8 |
2.7 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
2 |
729.7 |
733.1 |
733.9 |
734.4 |
2.8 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
3 |
1089.9 |
1100.5 |
1102.4 |
1103.0 |
2.8 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
5 |
1799.1 |
1835.3 |
1836.5 |
1840.2 |
2.9 |
|
FP8 |
image passage |
1024x1024 JPEG |
4 |
8 |
2834.4 |
2937.1 |
2937.9 |
2938.1 |
3.1 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
1 |
368.8 |
368.6 |
369.2 |
370.0 |
2.7 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
2 |
730.0 |
733.4 |
734.4 |
734.8 |
2.8 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
3 |
1090.3 |
1101.0 |
1102.7 |
1103.2 |
2.8 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
5 |
1798.8 |
1835.3 |
1836.2 |
1836.4 |
2.9 |
|
FP8 |
image passage |
1024x1024 JPEG |
8 |
8 |
2834.7 |
2937.2 |
2938.3 |
2938.7 |
3.1 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
1 |
374.0 |
373.7 |
375.3 |
376.5 |
2.7 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
2 |
729.3 |
732.7 |
733.7 |
734.4 |
2.8 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
3 |
1089.6 |
1100.0 |
1102.5 |
1104.6 |
2.8 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
5 |
1801.1 |
1837.5 |
1839.2 |
1839.5 |
2.9 |
|
FP8 |
image passage |
2048x2048 JPEG |
1 |
8 |
2837.4 |
2939.5 |
2941.3 |
2945.5 |
3.1 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
1 |
371.0 |
370.9 |
371.6 |
372.2 |
2.7 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
2 |
733.2 |
736.8 |
740.3 |
740.8 |
2.7 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
3 |
1090.2 |
1101.1 |
1102.0 |
1102.2 |
2.8 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
5 |
1800.8 |
1837.0 |
1839.1 |
1840.4 |
2.9 |
|
FP8 |
image passage |
2048x2048 JPEG |
4 |
8 |
2838.2 |
2940.6 |
2942.0 |
2942.3 |
3.1 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
1 |
370.3 |
370.2 |
370.9 |
371.1 |
2.7 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
2 |
730.6 |
734.2 |
734.6 |
735.4 |
2.8 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
3 |
1090.9 |
1101.6 |
1103.2 |
1104.6 |
2.8 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
5 |
1800.8 |
1837.2 |
1838.4 |
1838.9 |
2.9 |
|
FP8 |
image passage |
2048x2048 JPEG |
8 |
8 |
2838.2 |
2940.7 |
2942.0 |
2942.3 |
3.1 |
Llama Nemotron Embed 1B v2#
Memory Footprint#
The following table contains the memory footprint data for Llama Nemotron Embed 1B v2.
Precision |
Approximate GPU Memory Size (GiB) |
|
|---|---|---|
12.0 |
fp8 |
2.98 |
12.0 |
fp16 |
6.53 |
10.0 |
fp8 |
2.98 |
10.0 |
fp16 |
6.09 |
9.0 |
fp8 |
2.98 |
9.0 |
fp16 |
4.91 |
8.9 |
fp8 |
2.98 |
8.9 |
fp16 |
4.91 |
8.6 |
fp16 |
5.22 |
8.0 |
fp16 |
5.09 |
NVIDIA H100 80GB HBM3#
The following table contains the performance data for Llama Nemotron Embed 1B v2 on NVIDIA H100 80GB HBM3.
Precision |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|
FP8 |
passage |
300 |
64 |
1 |
159 |
159 |
162 |
163 |
401.8 |
FP8 |
passage |
300 |
64 |
3 |
267 |
269 |
275 |
277 |
709.4 |
FP8 |
passage |
300 |
64 |
5 |
392 |
390 |
401 |
411 |
814.5 |
FP8 |
passage |
512 |
64 |
1 |
222 |
218 |
233 |
235 |
286.3 |
FP8 |
passage |
512 |
64 |
3 |
431 |
430 |
450 |
455 |
440.1 |
FP8 |
passage |
512 |
64 |
5 |
615 |
617 |
694 |
701 |
504.3 |
FP8 |
query |
20 |
1 |
1 |
7 |
7 |
8 |
8 |
140.7 |
FP8 |
query |
20 |
1 |
3 |
10 |
10 |
10 |
10 |
291.7 |
FP8 |
query |
20 |
1 |
5 |
16 |
17 |
17 |
17 |
307.0 |
FP8 |
query |
20 |
1 |
7 |
22 |
23 |
24 |
24 |
310.9 |
FP8 |
query |
20 |
1 |
9 |
25 |
26 |
31 |
31 |
329.7 |
FP8 |
query |
20 |
1 |
11 |
30 |
32 |
38 |
38 |
332.8 |
FP8 |
query |
20 |
1 |
13 |
36 |
37 |
43 |
44 |
340.0 |
FP8 |
query |
20 |
1 |
15 |
42 |
43 |
48 |
50 |
327.5 |
FP16 |
passage |
300 |
64 |
1 |
186 |
186 |
190 |
192 |
342.7 |
FP16 |
passage |
300 |
64 |
3 |
345 |
346 |
354 |
356 |
550.7 |
FP16 |
passage |
300 |
64 |
5 |
525 |
524 |
535 |
544 |
608.3 |
FP16 |
passage |
512 |
64 |
1 |
269 |
265 |
280 |
284 |
237.2 |
FP16 |
passage |
512 |
64 |
3 |
563 |
564 |
581 |
584 |
337.3 |
FP16 |
passage |
512 |
64 |
5 |
846 |
861 |
946 |
956 |
366.0 |
FP16 |
query |
20 |
1 |
1 |
8 |
8 |
8 |
9 |
125.6 |
FP16 |
query |
20 |
1 |
3 |
12 |
12 |
13 |
13 |
246.1 |
FP16 |
query |
20 |
1 |
5 |
20 |
21 |
22 |
22 |
247.9 |
FP16 |
query |
20 |
1 |
7 |
25 |
26 |
30 |
30 |
267.5 |
FP16 |
query |
20 |
1 |
9 |
34 |
34 |
38 |
39 |
251.5 |
FP16 |
query |
20 |
1 |
11 |
43 |
46 |
47 |
48 |
237.7 |
FP16 |
query |
20 |
1 |
13 |
46 |
49 |
53 |
54 |
260.9 |
FP16 |
query |
20 |
1 |
15 |
55 |
58 |
65 |
65 |
248.4 |
NVIDIA A100 SXM4 80GB#
The following table contains the performance data for Llama Nemotron Embed 1B v2 on NVIDIA A100 SXM4 80GB.
Precision |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|
FP16 |
passage |
300 |
64 |
1 |
325 |
324 |
331 |
333 |
196.9 |
FP16 |
passage |
300 |
64 |
3 |
699 |
699 |
717 |
723 |
272.1 |
FP16 |
passage |
300 |
64 |
5 |
1095 |
1093 |
1109 |
1113 |
291.8 |
FP16 |
passage |
512 |
64 |
1 |
496 |
497 |
502 |
518 |
128.8 |
FP16 |
passage |
512 |
64 |
3 |
1165 |
1169 |
1199 |
1209 |
163.2 |
FP16 |
passage |
512 |
64 |
5 |
1872 |
1870 |
1904 |
1916 |
170.5 |
FP16 |
query |
20 |
1 |
1 |
8 |
8 |
9 |
10 |
122.3 |
FP16 |
query |
20 |
1 |
3 |
11 |
12 |
14 |
14 |
252.9 |
FP16 |
query |
20 |
1 |
5 |
20 |
19 |
23 |
24 |
250.9 |
FP16 |
query |
20 |
1 |
7 |
27 |
27 |
31 |
32 |
251.6 |
FP16 |
query |
20 |
1 |
9 |
30 |
30 |
37 |
42 |
280.1 |
FP16 |
query |
20 |
1 |
11 |
41 |
42 |
47 |
51 |
250.9 |
FP16 |
query |
20 |
1 |
13 |
46 |
49 |
53 |
57 |
262.0 |
FP16 |
query |
20 |
1 |
15 |
50 |
51 |
60 |
63 |
269.7 |
NVIDIA A100 PCIe 80GB#
The following table contains the performance data for Llama Nemotron Embed 1B v2 on NVIDIA A100 PCIe 80GB.
Precision |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|
FP16 |
passage |
300 |
64 |
1 |
362 |
362 |
367 |
369 |
176.8 |
FP16 |
passage |
300 |
64 |
3 |
786 |
789 |
803 |
812 |
241.9 |
FP16 |
passage |
300 |
64 |
5 |
1240 |
1239 |
1258 |
1262 |
257.6 |
FP16 |
passage |
512 |
64 |
1 |
550 |
548 |
565 |
570 |
116.2 |
FP16 |
passage |
512 |
64 |
3 |
1327 |
1335 |
1354 |
1359 |
143.4 |
FP16 |
passage |
512 |
64 |
5 |
2145 |
2145 |
2180 |
2187 |
149.0 |
FP16 |
query |
20 |
1 |
1 |
10 |
9 |
10 |
10 |
104.7 |
FP16 |
query |
20 |
1 |
3 |
12 |
12 |
12 |
12 |
246.7 |
FP16 |
query |
20 |
1 |
5 |
18 |
19 |
20 |
20 |
276.7 |
FP16 |
query |
20 |
1 |
7 |
25 |
24 |
28 |
28 |
268.9 |
FP16 |
query |
20 |
1 |
9 |
30 |
31 |
36 |
36 |
291.4 |
FP16 |
query |
20 |
1 |
11 |
38 |
39 |
44 |
44 |
270.3 |
FP16 |
query |
20 |
1 |
13 |
43 |
44 |
52 |
52 |
272.4 |
FP16 |
query |
20 |
1 |
15 |
49 |
52 |
56 |
57 |
276.4 |
NVIDIA L40S#
The following table contains the performance data for Llama Nemotron Embed 1B v2 on NVIDIA L40S.
Precision |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|
FP8 |
passage |
300 |
64 |
1 |
277 |
278 |
280 |
281 |
230.3 |
FP8 |
passage |
300 |
64 |
3 |
615 |
617 |
629 |
632 |
309.4 |
FP8 |
passage |
300 |
64 |
5 |
976 |
975 |
983 |
987 |
327.4 |
FP8 |
passage |
512 |
64 |
1 |
443 |
441 |
448 |
454 |
144.3 |
FP8 |
passage |
512 |
64 |
3 |
1071 |
1077 |
1089 |
1090 |
177.6 |
FP8 |
passage |
512 |
64 |
5 |
1736 |
1735 |
1752 |
1758 |
184.1 |
FP8 |
query |
20 |
1 |
1 |
10 |
10 |
10 |
11 |
98.0 |
FP8 |
query |
20 |
1 |
3 |
15 |
17 |
17 |
17 |
192.5 |
FP8 |
query |
20 |
1 |
5 |
24 |
23 |
28 |
28 |
200.8 |
FP8 |
query |
20 |
1 |
7 |
34 |
34 |
39 |
40 |
197.0 |
FP8 |
query |
20 |
1 |
9 |
44 |
45 |
51 |
51 |
197.3 |
FP8 |
query |
20 |
1 |
11 |
53 |
56 |
62 |
62 |
196.8 |
FP8 |
query |
20 |
1 |
13 |
63 |
67 |
73 |
73 |
190.6 |
FP8 |
query |
20 |
1 |
15 |
73 |
79 |
84 |
84 |
188.3 |
FP16 |
passage |
300 |
64 |
1 |
350 |
350 |
353 |
354 |
182.8 |
FP16 |
passage |
300 |
64 |
3 |
821 |
823 |
831 |
842 |
231.7 |
FP16 |
passage |
300 |
64 |
5 |
1320 |
1317 |
1330 |
1346 |
242.1 |
FP16 |
passage |
512 |
64 |
1 |
567 |
566 |
573 |
575 |
112.7 |
FP16 |
passage |
512 |
64 |
3 |
1440 |
1448 |
1464 |
1475 |
132.2 |
FP16 |
passage |
512 |
64 |
5 |
2370 |
2371 |
2391 |
2396 |
134.8 |
FP16 |
query |
20 |
1 |
1 |
12 |
12 |
12 |
12 |
85.7 |
FP16 |
query |
20 |
1 |
3 |
19 |
21 |
22 |
22 |
150.3 |
FP16 |
query |
20 |
1 |
5 |
32 |
29 |
36 |
36 |
152.8 |
FP16 |
query |
20 |
1 |
7 |
46 |
50 |
50 |
50 |
148.0 |
FP16 |
query |
20 |
1 |
9 |
55 |
57 |
65 |
65 |
156.3 |
FP16 |
query |
20 |
1 |
11 |
68 |
72 |
79 |
79 |
154.5 |
FP16 |
query |
20 |
1 |
13 |
79 |
86 |
93 |
93 |
154.4 |
FP16 |
query |
20 |
1 |
15 |
84 |
86 |
95 |
101 |
165.0 |
NVIDIA L4#
The following table contains the performance data for Llama Nemotron Embed 1B v2 on NVIDIA L4.
Precision |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|
FP8 |
passage |
300 |
64 |
1 |
724 |
725 |
730 |
733 |
88.3 |
FP8 |
passage |
300 |
64 |
3 |
1865 |
1876 |
1891 |
1892 |
102.1 |
FP8 |
passage |
300 |
64 |
5 |
3117 |
3127 |
3147 |
3150 |
102.1 |
FP8 |
passage |
512 |
64 |
1 |
1300 |
1300 |
1314 |
1318 |
49.2 |
FP8 |
passage |
512 |
64 |
3 |
3551 |
3577 |
3606 |
3618 |
53.6 |
FP8 |
passage |
512 |
64 |
5 |
5940 |
5968 |
6000 |
6007 |
53.6 |
FP8 |
query |
20 |
1 |
1 |
13 |
13 |
14 |
15 |
74.2 |
FP8 |
query |
20 |
1 |
3 |
21 |
23 |
23 |
23 |
138.0 |
FP8 |
query |
20 |
1 |
5 |
32 |
31 |
39 |
39 |
152.2 |
FP8 |
query |
20 |
1 |
7 |
46 |
46 |
53 |
54 |
145.8 |
FP8 |
query |
20 |
1 |
9 |
56 |
61 |
69 |
69 |
150.3 |
FP8 |
query |
20 |
1 |
11 |
67 |
69 |
76 |
77 |
154.0 |
FP8 |
query |
20 |
1 |
13 |
86 |
91 |
98 |
99 |
141.2 |
FP8 |
query |
20 |
1 |
15 |
98 |
105 |
113 |
114 |
141.0 |
FP16 |
passage |
300 |
64 |
1 |
1011 |
1011 |
1019 |
1019 |
63.3 |
FP16 |
passage |
300 |
64 |
3 |
2702 |
2716 |
2734 |
2754 |
70.6 |
FP16 |
passage |
300 |
64 |
5 |
4527 |
4527 |
4550 |
4571 |
70.6 |
FP16 |
passage |
512 |
64 |
1 |
1705 |
1710 |
1733 |
1740 |
37.5 |
FP16 |
passage |
512 |
64 |
3 |
4770 |
4799 |
4860 |
4872 |
39.9 |
FP16 |
passage |
512 |
64 |
5 |
8001 |
8025 |
8079 |
8085 |
39.9 |
FP16 |
query |
20 |
1 |
1 |
18 |
18 |
18 |
19 |
55.8 |
FP16 |
query |
20 |
1 |
3 |
33 |
37 |
38 |
38 |
87.7 |
FP16 |
query |
20 |
1 |
5 |
55 |
62 |
63 |
63 |
87.4 |
FP16 |
query |
20 |
1 |
7 |
71 |
75 |
88 |
88 |
94.6 |
FP16 |
query |
20 |
1 |
9 |
96 |
101 |
114 |
114 |
90.3 |
FP16 |
query |
20 |
1 |
11 |
115 |
126 |
139 |
139 |
89.8 |
FP16 |
query |
20 |
1 |
13 |
131 |
138 |
164 |
164 |
91.4 |
FP16 |
query |
20 |
1 |
15 |
154 |
157 |
189 |
189 |
88.4 |
NVIDIA A10G#
The following table contains the performance data for Llama Nemotron Embed 1B v2 on NVIDIA A10G.
Precision |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|
FP16 |
passage |
300 |
64 |
1 |
888 |
888 |
899 |
899 |
72.1 |
FP16 |
passage |
300 |
64 |
3 |
2272 |
2280 |
2329 |
2341 |
83.9 |
FP16 |
passage |
300 |
64 |
5 |
3795 |
3801 |
3828 |
3840 |
84.2 |
FP16 |
passage |
512 |
64 |
1 |
1451 |
1451 |
1471 |
1473 |
44.1 |
FP16 |
passage |
512 |
64 |
3 |
3926 |
3947 |
3988 |
4009 |
48.6 |
FP16 |
passage |
512 |
64 |
5 |
6577 |
6571 |
6632 |
6657 |
48.6 |
FP16 |
query |
20 |
1 |
1 |
21 |
21 |
21 |
21 |
48.6 |
FP16 |
query |
20 |
1 |
3 |
39 |
44 |
45 |
45 |
75.8 |
FP16 |
query |
20 |
1 |
5 |
61 |
60 |
75 |
75 |
80.2 |
FP16 |
query |
20 |
1 |
7 |
85 |
89 |
104 |
105 |
79.2 |
FP16 |
query |
20 |
1 |
9 |
109 |
119 |
134 |
134 |
79.0 |
FP16 |
query |
20 |
1 |
11 |
144 |
149 |
164 |
164 |
73.2 |
FP16 |
query |
20 |
1 |
13 |
159 |
164 |
194 |
194 |
76.5 |
FP16 |
query |
20 |
1 |
15 |
175 |
179 |
209 |
209 |
79.4 |
Llama Nemotron Embed 300m v2#
Memory Footprint#
The following table contains the memory footprint data for Llama Nemotron Embed 300m v2.
Precision |
Approximate GPU Memory Size (GiB) |
|
|---|---|---|
12.0 |
fp8 |
2.04 |
12.0 |
fp16 |
3.53 |
10.0 |
fp8 |
2.04 |
10.0 |
fp16 |
3.19 |
9.0 |
fp8 |
2.04 |
9.0 |
fp16 |
3.17 |
8.9 |
fp8 |
2.04 |
8.9 |
fp16 |
2.67 |
8.6 |
fp16 |
2.86 |
8.0 |
fp16 |
2.86 |
NVIDIA H100 80GB HBM3#
The following table contains the performance data for Llama Nemotron Embed 300m v2 on NVIDIA H100 80GB HBM3.
Precision |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
FP8 |
20 |
1 |
1 |
7.78 |
8 |
8 |
8.33 |
126 |
FP8 |
20 |
1 |
3 |
8.29 |
8 |
9 |
8.77 |
348 |
FP8 |
20 |
1 |
5 |
8.75 |
8 |
9 |
10.35 |
547 |
FP8 |
20 |
1 |
7 |
10.85 |
11 |
12 |
11.86 |
585 |
FP8 |
20 |
1 |
9 |
13.18 |
13 |
15 |
15.14 |
643 |
FP8 |
20 |
1 |
11 |
15.53 |
16 |
18 |
18.35 |
652 |
FP8 |
20 |
1 |
13 |
17.54 |
17 |
21 |
21.46 |
673 |
FP8 |
20 |
1 |
15 |
20.59 |
21 |
24 |
24.02 |
662 |
FP8 |
300 |
64 |
1 |
69.49 |
69 |
71 |
72.27 |
896 |
FP8 |
300 |
64 |
3 |
103.50 |
104 |
107 |
108.34 |
1779 |
FP8 |
300 |
64 |
5 |
172.27 |
173 |
178 |
179.91 |
1799 |
FP8 |
512 |
64 |
1 |
103.64 |
101 |
116 |
120.12 |
606 |
FP8 |
512 |
64 |
3 |
180.56 |
182 |
192 |
193.55 |
1040 |
FP8 |
512 |
64 |
5 |
298.65 |
300 |
318 |
325.06 |
1041 |
FP16 |
20 |
1 |
1 |
8.02 |
8 |
9 |
8.93 |
122 |
FP16 |
20 |
1 |
3 |
8.58 |
8 |
9 |
9.17 |
340 |
FP16 |
20 |
1 |
5 |
8.79 |
9 |
9 |
9.22 |
548 |
FP16 |
20 |
1 |
7 |
10.91 |
11 |
12 |
12.59 |
575 |
FP16 |
20 |
1 |
9 |
13.33 |
14 |
15 |
15.16 |
638 |
FP16 |
20 |
1 |
11 |
15.36 |
16 |
17 |
17.92 |
668 |
FP16 |
20 |
1 |
13 |
17.60 |
17 |
21 |
21.27 |
688 |
FP16 |
20 |
1 |
15 |
21.16 |
22 |
24 |
24.85 |
645 |
FP16 |
300 |
64 |
1 |
70.88 |
71 |
73 |
74.33 |
880 |
FP16 |
300 |
64 |
3 |
112.79 |
113 |
117 |
118.36 |
1652 |
FP16 |
300 |
64 |
5 |
187.74 |
189 |
194 |
194.95 |
1653 |
FP16 |
512 |
64 |
1 |
109.49 |
108 |
128 |
128.95 |
574 |
FP16 |
512 |
64 |
3 |
202.49 |
203 |
216 |
219.39 |
919 |
FP16 |
512 |
64 |
5 |
336.37 |
338 |
352 |
356.86 |
926 |
NVIDIA H100 NVL#
The following table contains the performance data for Llama Nemotron Embed 300m v2 on NVIDIA H100 NVL.
Precision |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
FP8 |
20 |
1 |
1 |
8.18 |
8 |
9 |
8.63 |
120 |
FP8 |
20 |
1 |
3 |
8.47 |
8 |
9 |
9.15 |
345 |
FP8 |
20 |
1 |
5 |
9.41 |
9 |
10 |
10.85 |
508 |
FP8 |
20 |
1 |
7 |
11.40 |
11 |
13 |
13.37 |
576 |
FP8 |
20 |
1 |
9 |
14.38 |
15 |
17 |
16.93 |
584 |
FP8 |
20 |
1 |
11 |
16.63 |
16 |
20 |
20.25 |
602 |
FP8 |
20 |
1 |
13 |
20.25 |
21 |
24 |
24.18 |
590 |
FP8 |
20 |
1 |
15 |
22.61 |
23 |
26 |
26.69 |
600 |
FP8 |
300 |
64 |
1 |
68.78 |
68 |
72 |
73.08 |
907 |
FP8 |
300 |
64 |
3 |
119.34 |
120 |
125 |
126.38 |
1552 |
FP8 |
300 |
64 |
5 |
199.83 |
202 |
208 |
210.33 |
1555 |
FP8 |
512 |
64 |
1 |
112.73 |
112 |
125 |
131.93 |
559 |
FP8 |
512 |
64 |
3 |
229.52 |
232 |
242 |
246.59 |
814 |
FP8 |
512 |
64 |
5 |
383.51 |
391 |
401 |
406.96 |
813 |
FP16 |
20 |
1 |
1 |
8.16 |
8 |
8 |
8.70 |
120 |
FP16 |
20 |
1 |
3 |
8.31 |
8 |
9 |
8.83 |
339 |
FP16 |
20 |
1 |
5 |
8.73 |
9 |
9 |
9.28 |
535 |
FP16 |
20 |
1 |
7 |
10.72 |
11 |
12 |
11.83 |
597 |
FP16 |
20 |
1 |
9 |
13.20 |
13 |
15 |
15.20 |
642 |
FP16 |
20 |
1 |
11 |
15.52 |
16 |
18 |
18.31 |
654 |
FP16 |
20 |
1 |
13 |
18.43 |
19 |
21 |
21.44 |
650 |
FP16 |
20 |
1 |
15 |
20.82 |
21 |
24 |
24.46 |
668 |
FP16 |
300 |
64 |
1 |
78.69 |
79 |
82 |
82.79 |
795 |
FP16 |
300 |
64 |
3 |
149.74 |
151 |
156 |
157.84 |
1250 |
FP16 |
300 |
64 |
5 |
249.85 |
254 |
260 |
261.95 |
1246 |
FP16 |
512 |
64 |
1 |
129.64 |
127 |
142 |
150.17 |
487 |
FP16 |
512 |
64 |
3 |
283.14 |
286 |
299 |
302.77 |
660 |
FP16 |
512 |
64 |
5 |
474.03 |
482 |
499 |
502.43 |
658 |
NVIDIA A100 SXM4 80GB#
The following table contains the performance data for Llama Nemotron Embed 300m v2 on NVIDIA A100 SXM4 80GB.
Precision |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
FP16 |
20 |
1 |
1 |
5.53 |
5 |
7 |
6.98 |
180 |
FP16 |
20 |
1 |
3 |
7.01 |
7 |
7 |
7.13 |
421 |
FP16 |
20 |
1 |
5 |
10.47 |
11 |
12 |
11.85 |
455 |
FP16 |
20 |
1 |
7 |
12.63 |
12 |
16 |
16.39 |
529 |
FP16 |
20 |
1 |
9 |
18.47 |
20 |
21 |
21.29 |
466 |
FP16 |
20 |
1 |
11 |
19.63 |
19 |
23 |
25.16 |
524 |
FP16 |
20 |
1 |
13 |
24.63 |
25 |
28 |
30.14 |
481 |
FP16 |
20 |
1 |
15 |
26.42 |
28 |
30 |
30.31 |
509 |
FP16 |
300 |
64 |
1 |
115.68 |
115 |
120 |
120.39 |
544 |
FP16 |
300 |
64 |
3 |
229.73 |
231 |
236 |
239.21 |
817 |
FP16 |
300 |
64 |
5 |
381.63 |
385 |
393 |
397.63 |
817 |
FP16 |
512 |
64 |
1 |
199.57 |
195 |
237 |
242.43 |
317 |
FP16 |
512 |
64 |
3 |
431.60 |
433 |
447 |
448.86 |
438 |
FP16 |
512 |
64 |
5 |
715.00 |
724 |
740 |
746.65 |
437 |
NVIDIA L40S#
The following table contains the performance data for Llama Nemotron Embed 300m v2 on NVIDIA L40S.
Precision |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
FP8 |
20 |
1 |
1 |
8.31 |
8 |
8 |
8.66 |
118 |
FP8 |
20 |
1 |
3 |
8.96 |
9 |
9 |
9.50 |
320 |
FP8 |
20 |
1 |
5 |
9.95 |
10 |
11 |
11.45 |
476 |
FP8 |
20 |
1 |
7 |
11.51 |
12 |
14 |
13.74 |
573 |
FP8 |
20 |
1 |
9 |
14.40 |
14 |
16 |
16.49 |
583 |
FP8 |
20 |
1 |
11 |
17.76 |
18 |
21 |
21.69 |
518 |
FP8 |
20 |
1 |
13 |
20.21 |
20 |
24 |
23.74 |
582 |
FP8 |
20 |
1 |
15 |
20.48 |
21 |
24 |
24.73 |
664 |
FP8 |
300 |
64 |
1 |
97.32 |
97 |
99 |
100.20 |
646 |
FP8 |
300 |
64 |
3 |
199.96 |
202 |
204 |
204.93 |
940 |
FP8 |
300 |
64 |
5 |
332.69 |
336 |
340 |
340.51 |
938 |
FP8 |
512 |
64 |
1 |
199.93 |
189 |
261 |
268.05 |
317 |
FP8 |
512 |
64 |
3 |
415.89 |
419 |
427 |
428.23 |
455 |
FP8 |
512 |
64 |
5 |
690.07 |
701 |
711 |
712.80 |
453 |
FP16 |
20 |
1 |
1 |
8.58 |
9 |
9 |
9.05 |
114 |
FP16 |
20 |
1 |
3 |
9.29 |
9 |
10 |
10.16 |
312 |
FP16 |
20 |
1 |
5 |
10.49 |
10 |
11 |
11.16 |
459 |
FP16 |
20 |
1 |
7 |
14.37 |
14 |
15 |
21.76 |
453 |
FP16 |
20 |
1 |
9 |
17.65 |
18 |
20 |
20.21 |
476 |
FP16 |
20 |
1 |
11 |
20.24 |
21 |
25 |
24.91 |
499 |
FP16 |
20 |
1 |
13 |
24.52 |
24 |
27 |
28.41 |
480 |
FP16 |
20 |
1 |
15 |
28.73 |
30 |
33 |
33.26 |
471 |
FP16 |
300 |
64 |
1 |
124.10 |
124 |
126 |
127.18 |
508 |
FP16 |
300 |
64 |
3 |
273.10 |
275 |
278 |
279.24 |
692 |
FP16 |
300 |
64 |
5 |
451.25 |
458 |
463 |
463.71 |
692 |
FP16 |
512 |
64 |
1 |
239.63 |
230 |
297 |
300.99 |
265 |
FP16 |
512 |
64 |
3 |
529.88 |
534 |
541 |
542.72 |
357 |
FP16 |
512 |
64 |
5 |
876.35 |
892 |
900 |
901.77 |
357 |
NVIDIA L4#
The following table contains the performance data for Llama Nemotron Embed 300m v2 on NVIDIA L4.
Precision |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
FP16 |
20 |
1 |
1 |
8.77 |
8 |
10 |
10.01 |
113 |
FP16 |
20 |
1 |
3 |
11.12 |
11 |
12 |
11.77 |
259 |
FP16 |
20 |
1 |
5 |
16.35 |
16 |
19 |
18.90 |
295 |
FP16 |
20 |
1 |
7 |
23.27 |
22 |
26 |
26.77 |
290 |
FP16 |
20 |
1 |
9 |
29.48 |
30 |
33 |
33.85 |
289 |
FP16 |
20 |
1 |
11 |
34.20 |
34 |
40 |
40.63 |
302 |
FP16 |
20 |
1 |
13 |
41.65 |
44 |
47 |
48.09 |
288 |
FP16 |
20 |
1 |
15 |
45.74 |
48 |
55 |
55.23 |
300 |
FP16 |
300 |
64 |
1 |
339.59 |
339 |
343 |
345.46 |
187 |
FP16 |
300 |
64 |
3 |
918.26 |
927 |
933 |
934.58 |
207 |
FP16 |
300 |
64 |
5 |
1517.32 |
1547 |
1554 |
1555.52 |
206 |
FP16 |
512 |
64 |
1 |
642.27 |
643 |
657 |
661.90 |
99 |
FP16 |
512 |
64 |
3 |
1795.35 |
1818 |
1835 |
1841.23 |
105 |
FP16 |
512 |
64 |
5 |
2976.58 |
3034 |
3066 |
3075.53 |
105 |
NVIDIA A10G#
The following table contains the performance data for Llama Nemotron Embed 300m v2 on NVIDIA A10G.
Precision |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
FP16 |
20 |
1 |
1 |
9.60 |
9 |
10 |
10.98 |
103 |
FP16 |
20 |
1 |
3 |
12.57 |
13 |
14 |
14.24 |
232 |
FP16 |
20 |
1 |
5 |
20.57 |
22 |
23 |
23.06 |
237 |
FP16 |
20 |
1 |
7 |
28.32 |
31 |
32 |
31.69 |
234 |
FP16 |
20 |
1 |
9 |
35.08 |
36 |
41 |
41.84 |
244 |
FP16 |
20 |
1 |
11 |
43.33 |
45 |
50 |
50.38 |
236 |
FP16 |
20 |
1 |
13 |
52.65 |
54 |
59 |
58.80 |
234 |
FP16 |
20 |
1 |
15 |
57.97 |
60 |
68 |
68.68 |
234 |
FP16 |
300 |
64 |
1 |
304.86 |
305 |
309 |
310.32 |
209 |
FP16 |
300 |
64 |
3 |
780.64 |
791 |
799 |
800.03 |
241 |
FP16 |
300 |
64 |
5 |
1296.85 |
1320 |
1330 |
1331.81 |
241 |
FP16 |
512 |
64 |
1 |
520.74 |
519 |
533 |
538.60 |
122 |
FP16 |
512 |
64 |
3 |
1385.52 |
1404 |
1424 |
1428.40 |
137 |
FP16 |
512 |
64 |
5 |
2294.26 |
2341 |
2362 |
2368.31 |
137 |
NVIDIA QA E5 Embedding#
Memory Footprint#
The following table contains the memory footprint data for NVIDIA QA E5 Embedding.
Precision |
Approximate GPU Memory Size (GiB) |
|
|---|---|---|
12.0 |
fp16 |
0.87 |
10.0 |
fp16 |
0.87 |
9.0 |
fp16 |
0.87 |
8.9 |
fp16 |
0.87 |
8.6 |
fp16 |
0.88 |
8.0 |
fp16 |
0.87 |
NVIDIA H100 80GB HBM3#
The following table contains the performance data for NVIDIA QA E5 Embedding on NVIDIA H100 80GB HBM3.
Precision |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|
FP16 |
passage |
300 |
64 |
1 |
99.82703581 |
100.904438 |
107.924179 |
108.6330097 |
639.0343548 |
FP16 |
passage |
300 |
64 |
3 |
143.8097451 |
143.2762855 |
156.6011532 |
159.0137593 |
1330.002429 |
FP16 |
passage |
300 |
64 |
5 |
239.6826504 |
239.732372 |
259.140126 |
264.9664443 |
1330.975643 |
FP16 |
passage |
512 |
64 |
1 |
114.5962003 |
114.388121 |
115.9079986 |
117.0048043 |
556.5479079 |
FP16 |
passage |
512 |
64 |
3 |
170.1910781 |
169.862901 |
171.2404072 |
171.8325308 |
1124.195819 |
FP16 |
passage |
512 |
64 |
5 |
284.5675255 |
284.516985 |
285.61744 |
286.059655 |
1121.352035 |
FP16 |
query |
20 |
1 |
1 |
5.080810388 |
5.077455 |
5.35184 |
5.449206 |
196.2536704 |
FP16 |
query |
20 |
1 |
3 |
6.005669286 |
5.5439535 |
7.449483 |
7.6433133 |
498.5059144 |
FP16 |
query |
20 |
1 |
5 |
11.90364799 |
12.3012425 |
12.7891541 |
12.93557995 |
418.3266895 |
FP16 |
query |
20 |
1 |
7 |
16.50119069 |
17.2286925 |
17.9718198 |
18.13005535 |
422.0442266 |
FP16 |
query |
20 |
1 |
9 |
21.43597982 |
22.277524 |
23.323499 |
23.553763 |
418.3116944 |
FP16 |
query |
20 |
1 |
11 |
26.03332591 |
25.9899645 |
28.3617327 |
28.5704181 |
421.3414301 |
FP16 |
query |
20 |
1 |
13 |
30.71904992 |
30.885644 |
33.145684 |
33.59465 |
422.1799737 |
FP16 |
query |
20 |
1 |
15 |
37.27936682 |
37.933293 |
39.0920083 |
39.3421219 |
401.4463907 |
NVIDIA A100 SXM4 80GB#
The following table contains the performance data for NVIDIA QA E5 Embedding on NVIDIA A100 SXM4 80GB.
Precision |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|
FP16 |
passage |
300 |
64 |
1 |
176.5200289 |
177.123774 |
188.566657 |
190.397038 |
362.1932991 |
FP16 |
passage |
300 |
64 |
3 |
336.1264122 |
336.965212 |
359.1192051 |
365.7920741 |
570.1898092 |
FP16 |
passage |
300 |
64 |
5 |
560.1892229 |
562.919036 |
592.343194 |
634.589014 |
569.8085691 |
FP16 |
passage |
512 |
64 |
1 |
205.2655254 |
204.74978 |
208.1572084 |
210.8121596 |
311.4421668 |
FP16 |
passage |
512 |
64 |
3 |
410.9122943 |
411.082123 |
412.4502242 |
412.745361 |
466.4179481 |
FP16 |
passage |
512 |
64 |
5 |
681.5252158 |
681.9785185 |
683.5981631 |
684.081592 |
468.6722081 |
FP16 |
query |
20 |
1 |
1 |
5.349868331 |
5.286015 |
5.6207312 |
5.717331 |
186.3291474 |
FP16 |
query |
20 |
1 |
3 |
7.411916441 |
7.4081655 |
7.5194672 |
7.6627225 |
403.8435598 |
FP16 |
query |
20 |
1 |
5 |
11.90756041 |
12.35885 |
12.5979345 |
12.761145 |
419.1684076 |
FP16 |
query |
20 |
1 |
7 |
16.58563318 |
17.30968 |
17.5332659 |
17.618394 |
421.4802771 |
FP16 |
query |
20 |
1 |
9 |
21.2083815 |
22.1409605 |
22.4645555 |
22.57402875 |
423.9030531 |
FP16 |
query |
20 |
1 |
11 |
26.13345736 |
27.168601 |
27.7088854 |
27.8043151 |
420.5148337 |
FP16 |
query |
20 |
1 |
13 |
30.75539894 |
31.155936 |
32.590465 |
32.686162 |
422.3358847 |
FP16 |
query |
20 |
1 |
15 |
36.41082533 |
37.3229475 |
37.8944988 |
37.9994324 |
411.6554532 |
NVIDIA A100 PCIe 80GB#
The following table contains the performance data for NVIDIA QA E5 Embedding on NVIDIA A100 PCIe 80GB.
Precision |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|
FP16 |
passage |
300 |
64 |
1 |
188.3624994 |
191.678584 |
197.67535 |
198.7671405 |
338.7085017 |
FP16 |
passage |
300 |
64 |
3 |
371.7164853 |
372.696551 |
393.8187446 |
470.9664648 |
515.255286 |
FP16 |
passage |
300 |
64 |
5 |
619.4848889 |
621.652637 |
648.4527444 |
728.8011656 |
515.0944739 |
FP16 |
passage |
512 |
64 |
1 |
222.7213055 |
222.302109 |
226.0124854 |
227.4071962 |
286.5090374 |
FP16 |
passage |
512 |
64 |
3 |
446.9578206 |
446.956036 |
448.6925762 |
449.3574626 |
428.4463769 |
FP16 |
passage |
512 |
64 |
5 |
742.316976 |
742.838223 |
744.9883932 |
745.4736348 |
430.0964269 |
FP16 |
query |
20 |
1 |
1 |
6.61186138 |
6.582944 |
6.977546 |
7.1138479 |
149.3209374 |
FP16 |
query |
20 |
1 |
3 |
7.36042852 |
7.345678 |
7.5866995 |
7.65647025 |
404.7664237 |
FP16 |
query |
20 |
1 |
5 |
11.7962158 |
12.222473 |
12.503024 |
12.59291 |
421.5411133 |
FP16 |
query |
20 |
1 |
7 |
16.35367295 |
17.0629145 |
17.4296151 |
17.541723 |
426.3990987 |
FP16 |
query |
20 |
1 |
9 |
20.86020568 |
21.905815 |
22.2696387 |
22.37563665 |
429.9399649 |
FP16 |
query |
20 |
1 |
11 |
25.69697501 |
26.830452 |
27.410482 |
27.724138 |
427.1729204 |
FP16 |
query |
20 |
1 |
13 |
30.3572733 |
31.467969 |
32.107687 |
32.228261 |
427.3957507 |
FP16 |
query |
20 |
1 |
15 |
35.59845166 |
36.40144 |
37.911859 |
38.023954 |
420.5844606 |
NVIDIA L40S#
The following table contains the performance data for NVIDIA QA E5 Embedding on NVIDIA L40S.
Precision |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|
FP16 |
passage |
300 |
64 |
1 |
2554.319287 |
2563.941207 |
2678.142464 |
2698.264006 |
25.04941137 |
FP16 |
passage |
300 |
64 |
3 |
7349.208273 |
7502.055819 |
7889.257348 |
7968.054606 |
25.46501769 |
FP16 |
passage |
300 |
64 |
5 |
11913.18547 |
12461.92393 |
12893.43705 |
12969.40433 |
25.62482416 |
FP16 |
passage |
512 |
64 |
1 |
3701.910032 |
3701.626575 |
3703.128269 |
3703.37542 |
17.28565985 |
FP16 |
passage |
512 |
64 |
3 |
10730.17861 |
10985.16205 |
10987.04189 |
11029.24857 |
17.46973511 |
FP16 |
passage |
512 |
64 |
5 |
17355.37756 |
14691.33609 |
22035.36884 |
22035.66547 |
17.42093242 |
FP16 |
query |
20 |
1 |
1 |
32.39684445 |
32.390068 |
32.697962 |
32.812839 |
30.71068928 |
FP16 |
query |
20 |
1 |
3 |
82.5131236 |
85.641417 |
85.943178 |
86.0222786 |
36.28989135 |
FP16 |
query |
20 |
1 |
5 |
135.5158144 |
142.938266 |
143.2624068 |
143.3466052 |
36.83096757 |
FP16 |
query |
20 |
1 |
7 |
191.652608 |
200.174195 |
200.5275478 |
200.61249 |
36.46886827 |
FP16 |
query |
20 |
1 |
9 |
246.8746939 |
257.437655 |
257.8359189 |
257.9366438 |
36.40752495 |
FP16 |
query |
20 |
1 |
11 |
301.7433573 |
314.645817 |
315.1281346 |
315.2351826 |
36.38786195 |
FP16 |
query |
20 |
1 |
13 |
356.6215151 |
371.558847 |
372.237291 |
372.367342 |
36.38148395 |
FP16 |
query |
20 |
1 |
15 |
409.5395827 |
401.376161 |
429.7671046 |
429.915561 |
36.54702515 |
NVIDIA L4#
The following table contains the performance data for NVIDIA QA E5 Embedding on NVIDIA L4.
Precision |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|
FP16 |
passage |
300 |
64 |
1 |
483.6478251 |
483.56992 |
505.150513 |
509.4594582 |
132.2515378 |
FP16 |
passage |
300 |
64 |
3 |
1327.952502 |
1334.361511 |
1367.915559 |
1379.568671 |
143.9367794 |
FP16 |
passage |
300 |
64 |
5 |
2181.789118 |
2203.719238 |
2241.932492 |
2250.362263 |
145.4517364 |
FP16 |
passage |
512 |
64 |
1 |
633.7960827 |
633.8213355 |
638.6268792 |
639.4460989 |
100.9307126 |
FP16 |
passage |
512 |
64 |
3 |
1744.892764 |
1755.272823 |
1761.547025 |
1763.103427 |
109.3979277 |
FP16 |
passage |
512 |
64 |
5 |
2892.159813 |
2923.949474 |
2934.835412 |
2936.818463 |
109.4116347 |
FP16 |
query |
20 |
1 |
1 |
8.00527288 |
7.97713 |
8.255327 |
8.3389483 |
124.1064504 |
FP16 |
query |
20 |
1 |
3 |
11.2179374 |
12.2274185 |
12.643952 |
12.7737285 |
266.1168591 |
FP16 |
query |
20 |
1 |
5 |
19.89864006 |
20.623372 |
21.068909 |
21.218374 |
250.3263155 |
FP16 |
query |
20 |
1 |
7 |
27.61174325 |
28.90997 |
29.4092535 |
29.5545445 |
252.95772 |
FP16 |
query |
20 |
1 |
9 |
35.05091948 |
36.7263 |
37.341008 |
37.513897 |
256.1157064 |
FP16 |
query |
20 |
1 |
11 |
42.73745639 |
44.63778 |
45.480588 |
45.658214 |
256.9000856 |
FP16 |
query |
20 |
1 |
13 |
50.74195156 |
50.275669 |
54.0248513 |
54.245848 |
255.897636 |
FP16 |
query |
20 |
1 |
15 |
57.40762279 |
57.9185695 |
62.191041 |
62.46531895 |
260.999059 |
NVIDIA A10G#
The following table contains the performance data for NVIDIA QA E5 Embedding on NVIDIA A10G.
Precision |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|---|
FP16 |
passage |
300 |
64 |
1 |
377.9075444 |
376.386471 |
396.5482632 |
404.4754582 |
169.1417219 |
FP16 |
passage |
300 |
64 |
3 |
929.9058353 |
932.259007 |
972.005973 |
979.1739983 |
205.8495868 |
FP16 |
passage |
300 |
64 |
5 |
1545.528521 |
1555.163799 |
1597.331709 |
1610.341433 |
205.8607711 |
FP16 |
passage |
512 |
64 |
1 |
469.4904227 |
468.610089 |
473.875774 |
475.110078 |
136.232079 |
FP16 |
passage |
512 |
64 |
3 |
1178.924205 |
1182.383929 |
1183.031068 |
1183.212201 |
162.2102538 |
FP16 |
passage |
512 |
64 |
5 |
1958.134001 |
1970.934197 |
1971.589458 |
1971.823967 |
162.2009027 |
FP16 |
query |
20 |
1 |
1 |
11.05668374 |
11.059908 |
11.4824455 |
11.6029265 |
89.77224972 |
FP16 |
query |
20 |
1 |
3 |
19.26746096 |
20.2599865 |
20.804533 |
20.961988 |
154.9430338 |
FP16 |
query |
20 |
1 |
5 |
32.06569956 |
33.962209 |
34.599702 |
34.76241 |
155.5098235 |
FP16 |
query |
20 |
1 |
7 |
44.76415285 |
47.4440885 |
48.0757631 |
48.2137225 |
156.015683 |
FP16 |
query |
20 |
1 |
9 |
57.69580393 |
60.895778 |
61.8394424 |
62.03314885 |
155.7518518 |
FP16 |
query |
20 |
1 |
11 |
70.60605841 |
74.0469635 |
75.482615 |
75.69873895 |
155.546507 |
FP16 |
query |
20 |
1 |
13 |
83.83823323 |
82.7783115 |
89.2197166 |
89.615594 |
154.8850883 |
FP16 |
query |
20 |
1 |
15 |
97.47471991 |
96.5648555 |
103.1160317 |
103.4140632 |
153.7165527 |