Performance and Memory for NVIDIA NeMo Retriever Reranking NIM#
To benchmark the performance of NVIDIA NeMo Retriever Reranking NIM, you can use the AIPerf tool.
To run a performance benchmark with representative input data, create a JSONL file where each line represents a ranking request with a query and one or more passages.
Example#
Create a file named rankings.jsonl that contains the following content.
{"texts": [{"name":"query","contents":["What was the first car ever driven?"]},{"name":"passages","contents":["The first car was driven in the late 19th century.","A steam-powered vehicle was demonstrated before gasoline automobiles became common."]}]}
{"texts": [{"name":"query","contents":["Who served as the 5th President of the United States of America?"]},{"name":"passages","contents":["James Monroe served as the fifth president of the United States.","George Washington was the first president of the United States."]}]}
{"texts": [{"name":"query","contents":["Is the Sydney Opera House located in Australia?"]},{"name":"passages","contents":["The Sydney Opera House is located in Sydney, Australia.","The Eiffel Tower is located in Paris, France."]}]}
{"texts": [{"name":"query","contents":["In what state did they film Shrek 2?"]},{"name":"passages","contents":["Principal voice recording for Shrek 2 took place in California.","DreamWorks Animation produced Shrek 2."]}]}
Install the verified AIPerf version in a Python environment that can reach the NIM endpoint.
pip install aiperf==0.9.0
Run the following command to benchmark the ranking endpoint by using AIPerf.
aiperf profile \
--model nvidia/llama-nemotron-rerank-vl-1b-v2 \
--endpoint-type nim_rankings \
--url localhost:8000 \
--input-file rankings.jsonl \
--custom-dataset-type single_turn \
--extra-inputs truncate:END \
--request-count 20 \
--concurrency 5
You can see the full set of command line options in the AIPerf command line options documentation.
Benchmarks#
All latency measurements are reported in milliseconds.
The previous AIPerf example benchmarks ranking requests using custom queries and passages. The benchmark tables use a separate synthetic AIPerf benchmark matrix for consistent SKU comparisons. The benchmark tables report measurements for the supported GPU SKUs and precisions that have benchmark data. For text rows, batch size is text passages per request and throughput is text passages per second. For image rows, batch size is image passages per request, image size is the generated JPEG resolution, and throughput is image passages per second.
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
passage |
514 |
40 |
1 |
160.8 |
160.7 |
162.4 |
163.3 |
246.1 |
image passage (1024x1024 JPEG) |
1 |
1 |
34.2 |
34.2 |
34.5 |
34.6 |
29.2 |
|
passage |
514 |
20 |
1 |
85.4 |
85.3 |
85.8 |
86.2 |
230.0 |
image passage (560x560 JPEG) |
1 |
1 |
12.1 |
12.1 |
12.1 |
12.1 |
82.9 |
|
image passage (336x336 JPEG) |
1 |
1 |
12.1 |
12.1 |
12.2 |
12.2 |
82.8 |
|
passage |
515 |
10 |
1 |
40.5 |
40.5 |
40.8 |
40.8 |
236.8 |
image passage (1024x1024 JPEG) |
1 |
3 |
88.6 |
89.2 |
90.4 |
90.5 |
11.3 |
|
passage |
515 |
10 |
3 |
109.4 |
110.0 |
110.4 |
110.7 |
268.2 |
image passage (560x560 JPEG) |
1 |
3 |
25.8 |
25.8 |
26.0 |
26.3 |
39.1 |
|
passage |
514 |
40 |
3 |
445.9 |
450.1 |
453.0 |
454.0 |
265.4 |
image passage (336x336 JPEG) |
1 |
3 |
26.0 |
26.1 |
26.3 |
26.3 |
38.7 |
|
passage |
514 |
20 |
3 |
224.2 |
210.5 |
300.9 |
301.6 |
264.4 |
passage |
514 |
20 |
5 |
370.7 |
379.5 |
420.4 |
524.7 |
264.0 |
passage |
515 |
10 |
5 |
185.9 |
187.8 |
188.6 |
189.2 |
263.3 |
image passage (1024x1024 JPEG) |
1 |
5 |
153.7 |
154.8 |
155.8 |
156.1 |
6.7 |
|
image passage (336x336 JPEG) |
1 |
5 |
41.4 |
41.7 |
41.9 |
42.0 |
24.3 |
|
passage |
514 |
40 |
5 |
737.6 |
751.9 |
755.9 |
756.5 |
265.1 |
image passage (560x560 JPEG) |
1 |
5 |
41.1 |
41.2 |
41.5 |
41.8 |
24.8 |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
image passage (560x560 JPEG) |
1 |
1 |
8.1 |
8.1 |
8.3 |
8.4 |
122.9 |
|
passage |
514 |
40 |
1 |
83.2 |
83.1 |
84.2 |
84.3 |
470.5 |
passage |
514 |
20 |
1 |
42.6 |
42.7 |
43.1 |
43.2 |
450.6 |
image passage (336x336 JPEG) |
1 |
1 |
8.0 |
8.0 |
8.2 |
8.2 |
124.5 |
|
image passage (1024x1024 JPEG) |
1 |
1 |
19.4 |
19.2 |
20.3 |
20.3 |
51.6 |
|
passage |
515 |
10 |
1 |
30.6 |
30.8 |
31.1 |
31.1 |
310.2 |
passage |
514 |
40 |
3 |
203.8 |
206.0 |
207.4 |
208.0 |
578.1 |
passage |
514 |
20 |
3 |
90.1 |
83.0 |
120.9 |
121.7 |
648.8 |
image passage (1024x1024 JPEG) |
1 |
3 |
43.5 |
43.6 |
44.0 |
44.1 |
23.2 |
|
image passage (336x336 JPEG) |
1 |
3 |
14.0 |
14.0 |
14.1 |
14.2 |
72.0 |
|
image passage (560x560 JPEG) |
1 |
3 |
13.7 |
13.7 |
14.0 |
14.1 |
73.3 |
|
passage |
515 |
10 |
3 |
56.3 |
56.6 |
56.9 |
57.0 |
512.6 |
passage |
515 |
10 |
5 |
75.7 |
75.8 |
76.8 |
77.2 |
633.8 |
passage |
514 |
40 |
5 |
336.8 |
343.6 |
346.1 |
346.4 |
579.0 |
image passage (560x560 JPEG) |
1 |
5 |
21.5 |
21.5 |
21.8 |
21.9 |
47.0 |
|
image passage (336x336 JPEG) |
1 |
5 |
21.9 |
21.8 |
22.1 |
22.6 |
46.3 |
|
passage |
514 |
20 |
5 |
149.4 |
154.4 |
168.7 |
209.1 |
650.4 |
image passage (1024x1024 JPEG) |
1 |
5 |
74.8 |
75.1 |
76.1 |
76.8 |
13.7 |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
passage |
515 |
10 |
1 |
37.0 |
37.0 |
37.5 |
37.6 |
259.9 |
passage |
514 |
40 |
1 |
122.6 |
122.5 |
124.1 |
124.5 |
321.8 |
passage |
514 |
20 |
1 |
64.9 |
64.9 |
65.5 |
65.6 |
300.1 |
image passage (336x336 JPEG) |
1 |
1 |
9.5 |
9.5 |
9.6 |
9.6 |
105.6 |
|
image passage (1024x1024 JPEG) |
1 |
1 |
27.1 |
27.2 |
27.6 |
27.6 |
36.9 |
|
image passage (560x560 JPEG) |
1 |
1 |
9.5 |
9.5 |
9.6 |
9.7 |
104.9 |
|
passage |
514 |
40 |
3 |
318.2 |
321.3 |
323.7 |
324.9 |
371.3 |
passage |
515 |
10 |
3 |
84.7 |
85.0 |
86.1 |
86.6 |
344.6 |
image passage (336x336 JPEG) |
1 |
3 |
18.1 |
18.0 |
18.2 |
18.5 |
55.6 |
|
passage |
514 |
20 |
3 |
155.3 |
147.4 |
209.1 |
210.2 |
380.4 |
image passage (1024x1024 JPEG) |
1 |
3 |
69.5 |
69.3 |
69.8 |
70.5 |
14.5 |
|
image passage (560x560 JPEG) |
1 |
3 |
17.7 |
17.7 |
18.0 |
18.3 |
56.7 |
|
image passage (336x336 JPEG) |
1 |
5 |
28.0 |
27.8 |
30.0 |
31.4 |
36.2 |
|
passage |
514 |
20 |
5 |
255.3 |
263.8 |
287.9 |
357.0 |
382.4 |
image passage (1024x1024 JPEG) |
1 |
5 |
122.0 |
122.5 |
123.3 |
130.5 |
8.3 |
|
passage |
515 |
10 |
5 |
128.8 |
130.0 |
131.4 |
131.8 |
378.6 |
image passage (560x560 JPEG) |
1 |
5 |
27.3 |
27.4 |
27.7 |
27.8 |
37.1 |
|
passage |
514 |
40 |
5 |
530.4 |
540.6 |
544.2 |
545.1 |
368.2 |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
passage |
514 |
40 |
1 |
277.9 |
277.8 |
280.3 |
281.0 |
143.0 |
passage |
514 |
20 |
1 |
143.9 |
143.7 |
145.3 |
146.2 |
137.3 |
image passage (1024x1024 JPEG) |
1 |
1 |
58.6 |
58.5 |
59.3 |
59.3 |
17.1 |
|
image passage (560x560 JPEG) |
1 |
1 |
14.9 |
14.9 |
15.0 |
15.0 |
67.3 |
|
passage |
515 |
10 |
1 |
64.6 |
64.6 |
65.6 |
65.9 |
151.3 |
image passage (336x336 JPEG) |
1 |
1 |
14.7 |
14.7 |
14.8 |
14.8 |
68.2 |
|
passage |
514 |
40 |
3 |
805.0 |
812.4 |
818.4 |
820.1 |
147.2 |
image passage (336x336 JPEG) |
1 |
3 |
35.8 |
35.9 |
36.4 |
36.5 |
28.1 |
|
passage |
514 |
20 |
3 |
412.6 |
386.3 |
553.5 |
554.9 |
144.1 |
image passage (1024x1024 JPEG) |
1 |
3 |
173.5 |
174.8 |
175.4 |
175.6 |
5.8 |
|
image passage (560x560 JPEG) |
1 |
3 |
35.3 |
35.6 |
36.1 |
36.3 |
28.4 |
|
passage |
515 |
10 |
3 |
197.3 |
198.6 |
201.6 |
201.9 |
149.9 |
passage |
514 |
20 |
5 |
689.2 |
707.0 |
783.5 |
798.3 |
142.3 |
image passage (1024x1024 JPEG) |
1 |
5 |
279.4 |
279.1 |
280.3 |
281.8 |
3.6 |
|
passage |
515 |
10 |
5 |
342.1 |
346.9 |
350.1 |
351.5 |
143.7 |
image passage (560x560 JPEG) |
1 |
5 |
58.2 |
58.6 |
58.9 |
59.1 |
17.5 |
|
image passage (336x336 JPEG) |
1 |
5 |
57.5 |
57.9 |
58.4 |
60.0 |
17.6 |
|
passage |
514 |
40 |
5 |
1331.8 |
1357.9 |
1366.4 |
1368.6 |
147.0 |
Input Type |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|---|
passage |
514 |
40 |
1 |
275.8 |
275.8 |
279.2 |
280.3 |
144.4 |
passage |
515 |
10 |
1 |
69.7 |
70.2 |
70.6 |
70.8 |
141.6 |
image passage (560x560 JPEG) |
1 |
1 |
15.0 |
15.0 |
15.2 |
15.2 |
66.7 |
|
image passage (336x336 JPEG) |
1 |
1 |
14.7 |
14.6 |
15.0 |
15.0 |
67.9 |
|
image passage (1024x1024 JPEG) |
1 |
1 |
57.3 |
57.3 |
58.2 |
58.3 |
17.4 |
|
passage |
514 |
20 |
1 |
144.1 |
143.8 |
145.8 |
146.8 |
137.9 |
passage |
514 |
20 |
3 |
377.8 |
355.3 |
508.7 |
510.0 |
157.5 |
passage |
514 |
40 |
3 |
762.2 |
769.7 |
775.7 |
777.6 |
155.6 |
image passage (336x336 JPEG) |
1 |
3 |
35.9 |
36.2 |
36.5 |
36.6 |
28.1 |
|
passage |
515 |
10 |
3 |
191.6 |
192.9 |
193.9 |
194.1 |
154.7 |
image passage (560x560 JPEG) |
1 |
3 |
35.6 |
35.7 |
36.4 |
36.4 |
28.3 |
|
image passage (1024x1024 JPEG) |
1 |
3 |
159.4 |
160.1 |
161.1 |
161.7 |
6.3 |
|
passage |
514 |
40 |
5 |
1259.2 |
1284.2 |
1292.5 |
1296.4 |
155.5 |
passage |
514 |
20 |
5 |
624.8 |
640.1 |
879.2 |
884.0 |
157.0 |
image passage (1024x1024 JPEG) |
1 |
5 |
273.8 |
275.8 |
277.3 |
277.5 |
3.8 |
|
passage |
515 |
10 |
5 |
315.7 |
319.7 |
321.4 |
321.7 |
155.5 |
image passage (336x336 JPEG) |
1 |
5 |
58.5 |
59.0 |
59.6 |
59.7 |
17.3 |
|
image passage (560x560 JPEG) |
1 |
5 |
57.5 |
57.8 |
58.8 |
59.3 |
17.7 |
Input tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
33.0 |
31.0 |
37.0 |
37.0 |
307.3 |
512 |
10 |
3 |
62.0 |
63.0 |
72.0 |
74.0 |
474.9 |
512 |
10 |
5 |
103.0 |
104.0 |
113.0 |
116.0 |
471.7 |
512 |
20 |
1 |
57.0 |
59.0 |
64.0 |
65.0 |
351.1 |
512 |
20 |
3 |
124.0 |
123.0 |
139.0 |
140.0 |
475.7 |
512 |
20 |
5 |
206.0 |
207.0 |
227.0 |
230.0 |
477.7 |
512 |
40 |
1 |
99.0 |
99.0 |
109.0 |
110.0 |
402.8 |
512 |
40 |
3 |
244.0 |
248.0 |
261.0 |
267.0 |
483.4 |
512 |
40 |
5 |
405.0 |
414.0 |
429.0 |
434.0 |
483.7 |
Input tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
40.0 |
39.0 |
43.0 |
44.0 |
250.4 |
512 |
10 |
3 |
88.0 |
89.0 |
96.0 |
104.0 |
337.7 |
512 |
10 |
5 |
145.0 |
148.0 |
153.0 |
154.0 |
337.1 |
512 |
20 |
1 |
71.0 |
74.0 |
78.0 |
79.0 |
280.1 |
512 |
20 |
3 |
171.0 |
172.0 |
184.0 |
187.0 |
345.0 |
512 |
20 |
5 |
284.0 |
288.0 |
305.0 |
310.0 |
345.3 |
512 |
40 |
1 |
129.0 |
129.0 |
139.0 |
140.0 |
309.5 |
512 |
40 |
3 |
341.0 |
346.0 |
358.0 |
361.0 |
347.5 |
512 |
40 |
5 |
565.0 |
575.0 |
592.0 |
601.0 |
347.2 |
Input tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
58.0 |
58.0 |
61.0 |
61.0 |
170.8 |
512 |
10 |
3 |
149.0 |
150.0 |
159.0 |
204.0 |
197.8 |
512 |
10 |
5 |
246.0 |
250.0 |
257.0 |
260.0 |
198.8 |
512 |
20 |
1 |
119.0 |
117.0 |
123.0 |
124.0 |
168.5 |
512 |
20 |
3 |
315.0 |
319.0 |
325.0 |
326.0 |
187.9 |
512 |
20 |
5 |
523.0 |
532.0 |
540.0 |
540.0 |
187.5 |
512 |
40 |
1 |
234.0 |
234.0 |
242.0 |
243.0 |
171.1 |
512 |
40 |
3 |
652.0 |
661.0 |
670.0 |
674.0 |
181.6 |
512 |
40 |
5 |
1080.0 |
1101.0 |
1118.0 |
1120.0 |
181.4 |
Input tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
78.0 |
78.0 |
80.0 |
80.0 |
128.2 |
512 |
10 |
3 |
205.0 |
207.0 |
210.0 |
212.0 |
144.8 |
512 |
10 |
5 |
340.0 |
346.0 |
354.0 |
357.0 |
144.0 |
512 |
20 |
1 |
158.0 |
157.0 |
162.0 |
163.0 |
126.8 |
512 |
20 |
3 |
430.0 |
436.0 |
443.0 |
446.0 |
137.2 |
512 |
20 |
5 |
716.0 |
728.0 |
739.0 |
744.0 |
136.9 |
512 |
40 |
1 |
312.0 |
312.0 |
320.0 |
321.0 |
128.0 |
512 |
40 |
3 |
886.0 |
896.0 |
907.0 |
910.0 |
134.0 |
512 |
40 |
5 |
1463.0 |
1492.0 |
1512.0 |
1515.0 |
134.0 |
512 |
40 |
5 |
1463.0 |
1492.0 |
1512.0 |
1515.0 |
134.0 |
Input tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
74.0 |
74.0 |
79.0 |
80.0 |
134.8 |
512 |
10 |
3 |
185.0 |
188.0 |
202.0 |
244.0 |
157.3 |
512 |
10 |
5 |
311.0 |
315.0 |
325.0 |
329.0 |
157.6 |
512 |
20 |
1 |
139.0 |
137.0 |
149.0 |
150.0 |
143.5 |
512 |
20 |
3 |
371.0 |
373.0 |
394.0 |
398.0 |
159.8 |
512 |
20 |
5 |
615.0 |
622.0 |
644.0 |
648.0 |
159.5 |
512 |
40 |
1 |
267.0 |
266.0 |
286.0 |
290.0 |
149.8 |
512 |
40 |
3 |
744.0 |
752.0 |
788.0 |
799.0 |
159.1 |
512 |
40 |
5 |
1231.0 |
1252.0 |
1296.0 |
1316.0 |
159.0 |
Input tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
79.0 |
79.0 |
84.0 |
85.0 |
126.4 |
512 |
10 |
3 |
203.0 |
206.0 |
217.0 |
271.0 |
144.5 |
512 |
10 |
5 |
339.0 |
346.0 |
354.0 |
358.0 |
144.4 |
512 |
20 |
1 |
151.0 |
151.0 |
160.0 |
161.0 |
132.0 |
512 |
20 |
3 |
405.0 |
411.0 |
428.0 |
435.0 |
145.4 |
512 |
20 |
5 |
672.0 |
685.0 |
710.0 |
717.0 |
145.5 |
512 |
40 |
1 |
289.0 |
287.0 |
307.0 |
310.0 |
138.2 |
512 |
40 |
3 |
811.0 |
817.0 |
842.0 |
851.0 |
146.3 |
512 |
40 |
5 |
1340.0 |
1357.0 |
1405.0 |
1412.0 |
146.2 |
Input tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
230.0 |
229.0 |
240.0 |
244.0 |
43.5 |
512 |
10 |
3 |
639.0 |
644.0 |
667.0 |
675.0 |
46.5 |
512 |
10 |
5 |
1055.0 |
1076.0 |
1101.0 |
1107.0 |
46.4 |
512 |
20 |
1 |
447.0 |
442.0 |
464.0 |
468.0 |
44.7 |
512 |
20 |
3 |
1257.0 |
1276.0 |
1310.0 |
1320.0 |
46.9 |
512 |
20 |
5 |
2088.0 |
2126.0 |
2172.0 |
2185.0 |
46.9 |
512 |
40 |
1 |
877.0 |
879.0 |
902.0 |
906.0 |
45.6 |
512 |
40 |
3 |
2534.0 |
2566.0 |
2605.0 |
2617.0 |
46.7 |
512 |
40 |
5 |
4194.0 |
4273.0 |
4321.0 |
4339.0 |
46.7 |
Input tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
190.0 |
187.0 |
197.0 |
199.0 |
52.7 |
512 |
10 |
3 |
534.0 |
543.0 |
554.0 |
557.0 |
55.2 |
512 |
10 |
5 |
888.0 |
907.0 |
924.0 |
925.0 |
55.2 |
512 |
20 |
1 |
383.0 |
384.0 |
395.0 |
397.0 |
52.1 |
512 |
20 |
3 |
1117.0 |
1134.0 |
1150.0 |
1153.0 |
53.0 |
512 |
20 |
5 |
1850.0 |
1885.0 |
1908.0 |
1916.0 |
53.0 |
512 |
40 |
1 |
768.0 |
768.0 |
785.0 |
787.0 |
52.1 |
512 |
40 |
3 |
2254.0 |
2279.0 |
2302.0 |
2308.0 |
52.7 |
512 |
40 |
5 |
3712.0 |
3795.0 |
3834.0 |
3839.0 |
52.7 |
Input tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
188.0 |
188.0 |
196.0 |
197.0 |
53.2 |
512 |
10 |
3 |
533.0 |
541.0 |
554.0 |
559.0 |
55.4 |
512 |
10 |
5 |
884.0 |
902.0 |
917.0 |
922.0 |
55.4 |
512 |
20 |
1 |
383.0 |
382.0 |
396.0 |
399.0 |
52.2 |
512 |
20 |
3 |
1119.0 |
1134.0 |
1147.0 |
1152.0 |
53.1 |
512 |
20 |
5 |
1848.0 |
1884.0 |
1910.0 |
1916.0 |
53.0 |
512 |
40 |
1 |
767.0 |
767.0 |
784.0 |
788.0 |
52.2 |
512 |
40 |
3 |
2245.0 |
2276.0 |
2301.0 |
2308.0 |
52.8 |
512 |
40 |
5 |
3714.0 |
3790.0 |
3825.0 |
3831.0 |
52.8 |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
27.1 |
27.0 |
30.0 |
31.3 |
367.0 |
512 |
10 |
3 |
33.2 |
33.0 |
36.0 |
37.2 |
878.0 |
512 |
10 |
5 |
54.5 |
55.0 |
60.0 |
62.9 |
896.0 |
512 |
20 |
1 |
38.7 |
38.0 |
42.0 |
43.1 |
513.0 |
512 |
20 |
3 |
60.5 |
61.0 |
67.0 |
67.7 |
967.0 |
512 |
20 |
5 |
100.9 |
101.0 |
110.0 |
111.2 |
971.0 |
512 |
40 |
1 |
66.6 |
67.0 |
72.0 |
72.6 |
598.0 |
512 |
40 |
3 |
123.8 |
124.0 |
133.0 |
135.7 |
958.0 |
512 |
40 |
5 |
205.0 |
208.0 |
220.0 |
224.3 |
956.0 |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
29.7 |
29.0 |
33.0 |
34.3 |
334.0 |
512 |
10 |
3 |
41.1 |
41.0 |
44.0 |
45.6 |
717.0 |
512 |
10 |
5 |
68.1 |
68.0 |
73.0 |
77.7 |
718.0 |
512 |
20 |
1 |
44.2 |
43.0 |
48.0 |
48.7 |
450.0 |
512 |
20 |
3 |
78.1 |
78.0 |
84.0 |
86.4 |
759.0 |
512 |
20 |
5 |
129.0 |
130.0 |
137.0 |
137.7 |
759.0 |
512 |
40 |
1 |
77.1 |
77.0 |
82.0 |
82.7 |
517.0 |
512 |
40 |
3 |
157.6 |
158.0 |
166.0 |
169.9 |
753.0 |
512 |
40 |
5 |
260.7 |
264.0 |
272.0 |
277.6 |
751.0 |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
28.2 |
28.0 |
32.0 |
32.6 |
352.0 |
512 |
10 |
3 |
43.3 |
43.0 |
47.0 |
55.3 |
675.0 |
512 |
10 |
5 |
71.3 |
72.0 |
80.0 |
88.3 |
685.0 |
512 |
20 |
1 |
43.0 |
42.0 |
47.0 |
47.5 |
462.0 |
512 |
20 |
3 |
83.2 |
84.0 |
91.0 |
92.3 |
710.0 |
512 |
20 |
5 |
138.4 |
141.0 |
148.0 |
148.5 |
707.0 |
512 |
40 |
1 |
76.1 |
76.0 |
81.0 |
81.9 |
524.0 |
512 |
40 |
3 |
166.9 |
168.0 |
179.0 |
179.5 |
707.0 |
512 |
40 |
5 |
278.2 |
283.0 |
291.0 |
292.2 |
703.0 |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
32.7 |
33.0 |
35.0 |
37.0 |
304.0 |
512 |
10 |
3 |
58.0 |
58.0 |
62.0 |
68.5 |
511.0 |
512 |
10 |
5 |
95.6 |
97.0 |
104.0 |
113.9 |
510.0 |
512 |
20 |
1 |
53.7 |
54.0 |
58.0 |
58.7 |
371.0 |
512 |
20 |
3 |
114.1 |
116.0 |
124.0 |
125.9 |
515.0 |
512 |
20 |
5 |
191.0 |
195.0 |
204.0 |
204.9 |
513.0 |
512 |
40 |
1 |
96.1 |
96.0 |
103.0 |
104.6 |
415.0 |
512 |
40 |
3 |
232.1 |
236.0 |
246.0 |
247.9 |
506.0 |
512 |
40 |
5 |
387.5 |
396.0 |
409.0 |
413.3 |
505.0 |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
39.7 |
38.0 |
52.0 |
53.9 |
251.0 |
512 |
10 |
3 |
67.2 |
67.0 |
70.0 |
74.3 |
441.0 |
512 |
10 |
5 |
111.0 |
112.0 |
117.0 |
120.1 |
440.0 |
512 |
20 |
1 |
70.8 |
71.0 |
75.0 |
78.2 |
282.0 |
512 |
20 |
3 |
146.8 |
148.0 |
153.0 |
153.9 |
404.0 |
512 |
20 |
5 |
242.8 |
246.0 |
251.0 |
253.1 |
403.0 |
512 |
40 |
1 |
134.2 |
133.0 |
144.0 |
145.2 |
297.0 |
512 |
40 |
3 |
305.4 |
307.0 |
314.0 |
316.2 |
389.0 |
512 |
40 |
5 |
504.5 |
513.0 |
522.0 |
527.0 |
388.0 |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
53.7 |
51.0 |
66.0 |
68.1 |
186.0 |
512 |
10 |
3 |
105.4 |
106.0 |
109.0 |
110.8 |
280.0 |
512 |
10 |
5 |
175.5 |
178.0 |
184.0 |
186.3 |
279.0 |
512 |
20 |
1 |
94.5 |
95.0 |
99.0 |
100.1 |
211.0 |
512 |
20 |
3 |
218.6 |
222.0 |
228.0 |
229.5 |
270.0 |
512 |
20 |
5 |
364.1 |
370.0 |
377.0 |
379.9 |
269.0 |
512 |
40 |
1 |
186.6 |
185.0 |
195.0 |
198.8 |
214.0 |
512 |
40 |
3 |
459.2 |
462.0 |
470.0 |
472.0 |
258.0 |
512 |
40 |
5 |
759.3 |
773.0 |
785.0 |
787.0 |
258.0 |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
47.8 |
47.0 |
53.0 |
58.0 |
209.0 |
512 |
10 |
3 |
96.3 |
96.0 |
102.0 |
109.3 |
305.0 |
512 |
10 |
5 |
160.0 |
161.0 |
167.0 |
181.6 |
306.0 |
512 |
20 |
1 |
83.0 |
84.0 |
88.0 |
89.3 |
240.0 |
512 |
20 |
3 |
192.4 |
194.0 |
201.0 |
204.7 |
309.0 |
512 |
20 |
5 |
317.6 |
323.0 |
331.0 |
334.2 |
308.0 |
512 |
40 |
1 |
156.4 |
156.0 |
164.0 |
165.4 |
255.0 |
512 |
40 |
3 |
380.9 |
384.0 |
401.0 |
405.1 |
309.0 |
512 |
40 |
5 |
632.9 |
641.0 |
658.0 |
671.0 |
310.0 |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
124.7 |
125.0 |
131.0 |
132.9 |
80.0 |
512 |
10 |
3 |
326.5 |
330.0 |
338.0 |
339.4 |
91.0 |
512 |
10 |
5 |
542.0 |
552.0 |
562.0 |
565.1 |
90.0 |
512 |
20 |
1 |
238.0 |
238.0 |
245.0 |
246.1 |
84.0 |
512 |
20 |
3 |
656.1 |
662.0 |
672.0 |
675.0 |
91.0 |
512 |
20 |
5 |
1084.7 |
1105.0 |
1119.0 |
1120.6 |
90.0 |
512 |
40 |
1 |
469.6 |
470.0 |
476.0 |
478.6 |
85.0 |
512 |
40 |
3 |
1308.2 |
1330.0 |
1345.0 |
1346.8 |
90.0 |
512 |
40 |
5 |
2173.5 |
2219.0 |
2235.0 |
2240.1 |
90.0 |
Input Tokens |
Batch Size |
Concurrency |
Avg Latency |
P50 Latency |
P90 Latency |
P95 Latency |
Throughput (inputs/s) |
|---|---|---|---|---|---|---|---|
512 |
10 |
1 |
136.5 |
137.0 |
140.0 |
143.2 |
73.0 |
512 |
10 |
3 |
379.0 |
384.0 |
392.0 |
393.7 |
78.0 |
512 |
10 |
5 |
627.5 |
640.0 |
649.0 |
651.9 |
78.0 |
512 |
20 |
1 |
275.9 |
276.0 |
282.0 |
285.2 |
72.0 |
512 |
20 |
3 |
786.4 |
795.0 |
806.0 |
809.3 |
76.0 |
512 |
20 |
5 |
1298.2 |
1324.0 |
1339.0 |
1342.5 |
75.0 |
512 |
40 |
1 |
549.2 |
549.0 |
558.0 |
559.7 |
73.0 |
512 |
40 |
3 |
1567.2 |
1594.0 |
1609.0 |
1610.5 |
75.0 |
512 |
40 |
5 |
2603.4 |
2654.0 |
2677.0 |
2684.0 |
75.0 |
Memory Footprint#
The following table provides the set of valid configurations and the associated approximate memory footprints for the model.
For llama-nemotron-rerank-vl-1b-v2, use the optimized SKU table in Support Matrix for NVIDIA NeMo Retriever Reranking NIM to determine which precision is supported for a release GPU SKU.
Llama Nemotron Rerank VL 1B v2#
Precision |
Approximate GPU Memory Size (GiB) |
|
|---|---|---|
12.0 |
fp8 |
6.09 |
10.0 |
fp8 |
6.21 |
9.0 |
fp8 |
6.12 |
8.0 |
fp16 |
5.67 |
Llama Nemotron Rerank 1B#
Precision |
Approximate GPU Memory Size (GiB) |
|
|---|---|---|
12.0 |
fp8 |
3.68 |
12.0 |
fp16 |
7.59 |
10.0 |
fp8 |
3.91 |
10.0 |
fp16 |
6.69 |
9.0 |
fp8 |
3.65 |
9.0 |
fp16 |
6.51 |
8.9 |
fp8 |
3.56 |
8.9 |
fp16 |
5.84 |
8.6 |
fp16 |
6.06 |
8.0 |
fp16 |
6.53 |
Llama Nemotron Rerank 500m#
GPU |
Precision |
Max Batch Size |
Max Sequence Length |
Approximate GPU Memory Size (GiB) |
|---|---|---|---|---|
a100-sxm4-40gb |
fp16 |
1 |
8192 |
2.53 |
a100-sxm4-40gb |
fp16 |
8 |
8192 |
9.63 |
a100-sxm4-40gb |
fp16 |
16 |
8192 |
18.25 |
a100-sxm4-40gb |
fp16 |
30 |
1024 |
5.19 |
a100-sxm4-40gb |
fp16 |
30 |
2048 |
8.65 |
a100-sxm4-40gb |
fp16 |
30 |
4096 |
16.27 |
a100-sxm4-40gb |
fp16 |
30 |
8192 |
32.91 |
a100-sxm4-80gb |
fp16 |
1 |
8192 |
2.53 |
a100-sxm4-80gb |
fp16 |
8 |
8192 |
9.63 |
a100-sxm4-80gb |
fp16 |
16 |
8192 |
18.25 |
a100-sxm4-80gb |
fp16 |
30 |
1024 |
5.19 |
a100-sxm4-80gb |
fp16 |
30 |
2048 |
8.65 |
a100-sxm4-80gb |
fp16 |
30 |
4096 |
16.27 |
a100-sxm4-80gb |
fp16 |
30 |
8192 |
32.91 |
a10g |
fp16 |
1 |
8192 |
2.78 |
a10g |
fp16 |
8 |
8192 |
9.88 |
a10g |
fp16 |
16 |
1024 |
3.66 |
a10g |
fp16 |
16 |
2048 |
5.5 |
a10g |
fp16 |
16 |
4096 |
9.38 |
a10g |
fp16 |
16 |
8192 |
18.0 |
h100-hbm3-80gb |
fp8 |
1 |
8192 |
1.4 |
h100-hbm3-80gb |
fp8 |
8 |
8192 |
4.5 |
h100-hbm3-80gb |
fp8 |
16 |
8192 |
8.04 |
h100-hbm3-80gb |
fp8 |
30 |
1024 |
2.16 |
h100-hbm3-80gb |
fp8 |
30 |
2048 |
3.58 |
h100-hbm3-80gb |
fp8 |
30 |
4096 |
6.66 |
h100-hbm3-80gb |
fp8 |
30 |
8192 |
14.21 |
h100-hbm3-80gb |
fp16 |
1 |
8192 |
3.04 |
h100-hbm3-80gb |
fp16 |
8 |
8192 |
10.38 |
h100-hbm3-80gb |
fp16 |
16 |
8192 |
18.75 |
h100-hbm3-80gb |
fp16 |
30 |
1024 |
5.69 |
h100-hbm3-80gb |
fp16 |
30 |
2048 |
9.15 |
h100-hbm3-80gb |
fp16 |
30 |
4096 |
16.77 |
h100-hbm3-80gb |
fp16 |
30 |
8192 |
33.41 |
h100-nvl |
fp8 |
1 |
8192 |
1.4 |
h100-nvl |
fp8 |
8 |
8192 |
4.5 |
h100-nvl |
fp8 |
16 |
8192 |
8.04 |
h100-nvl |
fp8 |
30 |
1024 |
2.16 |
h100-nvl |
fp8 |
30 |
2048 |
3.58 |
h100-nvl |
fp8 |
30 |
4096 |
6.66 |
h100-nvl |
fp8 |
30 |
8192 |
14.21 |
h100-nvl |
fp16 |
1 |
8192 |
3.04 |
h100-nvl |
fp16 |
8 |
8192 |
10.38 |
h100-nvl |
fp16 |
16 |
8192 |
18.75 |
h100-nvl |
fp16 |
30 |
1024 |
5.69 |
h100-nvl |
fp16 |
30 |
2048 |
9.15 |
h100-nvl |
fp16 |
30 |
4096 |
16.77 |
h100-nvl |
fp16 |
30 |
8192 |
33.41 |
l4 |
fp16 |
1 |
8192 |
2.27 |
l4 |
fp16 |
8 |
8192 |
9.81 |
l4 |
fp16 |
16 |
1024 |
3.5 |
l4 |
fp16 |
16 |
2048 |
5.38 |
l4 |
fp16 |
16 |
4096 |
9.31 |
l4 |
fp16 |
16 |
8192 |
18.06 |
l40s |
fp8 |
1 |
8192 |
1.52 |
l40s |
fp8 |
8 |
8192 |
4.5 |
l40s |
fp8 |
16 |
8192 |
8.03 |
l40s |
fp8 |
30 |
1024 |
2.16 |
l40s |
fp8 |
30 |
2048 |
3.58 |
l40s |
fp8 |
30 |
4096 |
6.65 |
l40s |
fp8 |
30 |
8192 |
14.21 |
l40s |
fp16 |
1 |
8192 |
2.6 |
l40s |
fp16 |
8 |
8192 |
9.69 |
l40s |
fp16 |
16 |
8192 |
17.81 |
l40s |
fp16 |
30 |
1024 |
5.14 |
l40s |
fp16 |
30 |
2048 |
8.59 |
l40s |
fp16 |
30 |
4096 |
15.86 |
l40s |
fp16 |
30 |
8192 |
32.03 |