Performance and Memory for NVIDIA NeMo Retriever Reranking NIM

To benchmark the performance of NVIDIA NeMo Retriever Reranking NIM, you can use the AIPerf tool.

To run a performance benchmark with representative input data, create a JSONL file where each line represents a ranking request with a query and one or more passages.

Example

Create a file named rankings.jsonl that contains the following content.

{"texts": [{"name":"query","contents":["What was the first car ever driven?"]},{"name":"passages","contents":["The first car was driven in the late 19th century.","A steam-powered vehicle was demonstrated before gasoline automobiles became common."]}]}
{"texts": [{"name":"query","contents":["Who served as the 5th President of the United States of America?"]},{"name":"passages","contents":["James Monroe served as the fifth president of the United States.","George Washington was the first president of the United States."]}]}
{"texts": [{"name":"query","contents":["Is the Sydney Opera House located in Australia?"]},{"name":"passages","contents":["The Sydney Opera House is located in Sydney, Australia.","The Eiffel Tower is located in Paris, France."]}]}
{"texts": [{"name":"query","contents":["In what state did they film Shrek 2?"]},{"name":"passages","contents":["Principal voice recording for Shrek 2 took place in California.","DreamWorks Animation produced Shrek 2."]}]}

Install the verified AIPerf version in a Python environment that can reach the NIM endpoint.

pip install aiperf==0.9.0

Run the following command to benchmark the ranking endpoint by using AIPerf.

aiperf profile \
    --model nvidia/Nemotron-3.5-Rerank-8B-BF16 \
    --endpoint-type nim_rankings \
    --url localhost:8000 \
    --input-file rankings.jsonl \
    --custom-dataset-type single_turn \
    --extra-inputs truncate:END \
    --request-count 20 \
    --concurrency 5

For the full set of command line options, refer to the AIPerf command line options documentation.

About the Measurements

The following sections report the memory footprint and the benchmark results for each model. All latency measurements are reported in milliseconds.

The previous AIPerf example benchmarks ranking requests using custom queries and passages. The benchmark tables use a separate synthetic AIPerf benchmark matrix for consistent GPU comparisons. For text rows, batch size is text passages per request and throughput is text passages per second. For image rows, batch size is image passages per request, image size is the generated JPEG resolution, and throughput is image passages per second.

Nemotron 3.5 Rerank 8B

The Nemotron 3.5 Rerank 8B data was measured on release candidate builds of the 2.4 container with FP16 precision and NIM_PERFORMANCE_MODE=1.

Memory Footprint

The following table contains the GPU memory that the NIM uses at server readiness, after the model loads and before it serves requests. The values are the lowest and highest observed across benchmark runs with different server settings. They aren’t peak memory during inference or a minimum GPU memory requirement.

GPU	Precision	GPU Memory at Server Readiness (GiB)
NVIDIA RTX PRO 6000 Blackwell Server Edition	FP16	16.9–19.1
NVIDIA GB200	FP16	17.1–19.2
NVIDIA B300	FP16	17.1–19.2
NVIDIA H100 80GB HBM3	FP16	16.9–19.1
NVIDIA H100 NVL	FP16	16.9–19.1
NVIDIA A100 SXM4 80GB	FP16	16.8–18.9
NVIDIA L40S	FP16	16.8–18.9
NVIDIA A10G	FP16	16.6–18.7

NVIDIA GB10 (DGX Spark) uses unified memory, which the CPU and GPU share. On GB10, the NIM used 55.1–62.1 GiB of unified memory at server readiness. This value isn’t comparable to the GPU memory values in the preceding table.

Benchmark Workloads

The performance tables report two workloads for each GPU. Both workloads use a concurrency of 5, 100 measured requests after 20 warmup requests, and a sequence limit of 10240 tokens. The following list describes the workloads:

• Text: Each request has a query of about 20 tokens and 40 text passages of about 512 tokens each, with truncate set to END. Batch Size is the number of passages per request.

• Image: Each request has a query of about 50 tokens and one 1120x1120 JPEG image passage.

Latency values are in milliseconds for each request. Throughput is passages per second.

NVIDIA RTX PRO 6000 Blackwell Server Edition

The following table contains the performance data for Nemotron 3.5 Rerank 8B on NVIDIA RTX PRO 6000 Blackwell Server Edition.

Precision	Input Type	Image Size	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
FP16	text		512	40	5	4923.8	5036.9	5055.4	5066.5	39.8
FP16	image passage	1120x1120		1	5	709.2	713.6	715.7	716.2	6.9

NVIDIA GB200

The following table contains the performance data for Nemotron 3.5 Rerank 8B on NVIDIA GB200.

Precision	Input Type	Image Size	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
FP16	text		512	40	5	1592.1	1621.9	1646.0	1651.5	123.0
FP16	image passage	1120x1120		1	5	434.8	438.1	440.6	440.7	11.1

NVIDIA B300

The following table contains the performance data for Nemotron 3.5 Rerank 8B on NVIDIA B300.

Precision	Input Type	Image Size	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
FP16	text		512	40	5	1662.3	1697.4	1700.1	1703.2	117.8
FP16	image passage	1120x1120		1	5	450.5	450.2	456.8	461.1	10.5

NVIDIA GB10 (DGX Spark)

The following table contains the performance data for Nemotron 3.5 Rerank 8B on NVIDIA GB10 (DGX Spark).

Precision	Input Type	Image Size	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
FP16	text		512	40	5	21925.1	22341.9	22482.3	22543.9	8.9
FP16	image passage	1120x1120		1	5	3062.6	3086.2	3101.4	3105.4	1.6

NVIDIA H100 80GB HBM3

The following table contains the performance data for Nemotron 3.5 Rerank 8B on NVIDIA H100 80GB HBM3.

Precision	Input Type	Image Size	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
FP16	text		512	40	5	2933.4	2993.8	3017.9	3028.2	66.8
FP16	image passage	1120x1120		1	5	640.3	645.1	648.4	649.7	7.5

NVIDIA H100 NVL

The following table contains the performance data for Nemotron 3.5 Rerank 8B on NVIDIA H100 NVL.

Precision	Input Type	Image Size	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
FP16	text		512	40	5	4101.9	4196.6	4281.4	4327.2	47.7
FP16	image passage	1120x1120		1	5	804.2	810.2	813.9	814.5	6.0

NVIDIA A100 SXM4 80GB

The following table contains the performance data for Nemotron 3.5 Rerank 8B on NVIDIA A100 SXM4 80GB.

Precision	Input Type	Image Size	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
FP16	text		512	40	5	7887.4	8051.2	8089.6	8094.2	24.8
FP16	image passage	1120x1120		1	5	1302.1	1313.9	1320.0	1320.5	3.7

NVIDIA L40S

The following table contains the performance data for Nemotron 3.5 Rerank 8B on NVIDIA L40S.

Precision	Input Type	Image Size	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
FP16	text		512	40	5	10117.7	10395.1	10458.3	10466.4	19.4
FP16	image passage	1120x1120		1	5	1164.2	1175.6	1187.3	1192.5	4.2

NVIDIA A10G

The following table contains the performance data for Nemotron 3.5 Rerank 8B on NVIDIA A10G.

Precision	Input Type	Image Size	Input Tokens	Batch Size	Concurrency	Avg Latency	P50 Latency	P90 Latency	P95 Latency	Throughput (inputs/s)
FP16	text		512	40	5	25876.3	26381.0	26538.5	26640.6	7.6
FP16	image passage	1120x1120		1	5	3409.2	3432.1	3449.8	3451.7	1.4