Support Matrix for NVIDIA NeMo Retriever Reranking NIM

This documentation describes the software and hardware that NVIDIA NeMo Retriever Reranking NIM supports.

CPU

NeMo Retriever Reranking NIM requires the following:

• x86 processor with at least 8 cores. For a list of supported systems, refer to NVIDIA Certified Systems Catalog.

Models

NeMo Retriever Reranking NIM supports the following models.

Publisher	Model ID	Max Tokens (Optimized Models)	Model Card
NVIDIA	nvidia/llama-nemotron-rerank-vl-1b-v2	10240	-
NVIDIA	nvidia/llama-nemotron-rerank-1b-v2	8192	Link
NVIDIA	nvidia/llama-nemotron-rerank-500m-v2	8192	Link

Note that when truncate is set to END, any Query / Passage pair that is longer than the maximum token length will be truncated from the right, starting with the passage.

Optimized vs Non Optimized Models

The following models are optimized using TRT and are available as pre-built, optimized engines on NGC. These optimized models are GPU specific and require a minimum GPU memory value as specified in the Optimized configuration sections of each model.

NVIDIA also provides generic model profiles that operate with any NVIDIA GPU (or set of GPUs) with sufficient memory capacity. These generic profiles are known as non-optimized configuration. On systems where there are no compatible optimized profiles, generic profiles are chosen automatically. Optimized profiles are preferred over generic profiles when available, but you can choose to deploy a generic profile on any system by following the steps in the Overriding Profile Selection section.

Compute Capability and Automatic Profile Selection

NVIDIA NeMo Retriever Reranking NIM supports TensorRT engines that are compiled with the option kSAME_COMPUTE_CAPABILITY. This option builds engines that are compatible with GPUs having the same compute capability as the one on which the engine was built. For more information, refer to Same Compute Capability Compatibility Level.

To see the mapping of CUDA GPU compute capability versions to supported GPU SKUs, refer to CUDA GPU Compute Capability. If you run a NIM on a GPU that has the same compute capability as one of the engines, then that engine should appear as compatible when you run list-model-profiles.

Automatic profile selection uses the following order to choose a profile:

1. A GPU-specific engine (for example, gpu:NVIDIA B200)

2. A compute capability engine (for example, compute_capability:10.0)

3. ONNX or Pytorch (for example, model_type:onnx)

Note: Certain NIMs may include both GPU-specific engines and compute capability engines, while others may include only a single engine type.

Supported Hardware

Note

Currently, GPU clusters with GPUs in Multi-instance GPU mode (MIG) are not supported.

llama-nemotron-rerank-vl-1b-v2

Optimized configuration

Compute Capability	Precision	Max Tokens
12.0	FP16 & FP8	10240
10.0	FP16 & FP8	10240
9.0	FP16 & FP8	10240
8.9	FP16 & FP8	10240
8.6	FP16	8192
8.0	FP16	10240

Non-optimized configuration

The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.

GPUs	GPU Memory	Precision	Disk Space	Max Tokens
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory	7.30	FP16	3.10	8191

llama-nemotron-rerank-1b-v2

Optimized configuration

Compute Capability	Precision
12.0	FP16 & FP8
10.0	FP16 & FP8
9.0	FP16 & FP8
8.9	FP16 & FP8
8.6	FP16
8.0	FP16

Non-optimized configuration

The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.

GPUs	GPU Memory	Precision	Disk Space	Max Tokens
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory	3.6	FP16	9.5	4096

Warning

The maximum token length of the non-optimized configuration is smaller (4096) than the other profiles (8192).

llama-nemotron-rerank-500m-v2

Optimized configuration

Compute Capability	Precision
12.0	FP16 & FP8
10.0	FP16 & FP8
9.0	FP16 & FP8
8.9	FP16 & FP8
8.6	FP16
8.0	FP16

Non-optimized configuration

The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.

GPUs	GPU Memory	Precision	Disk Space	Max Tokens
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory	3.6	FP16	9.5	4096

Warning

The maximum token length of the non-optimized configuration is smaller (4096) than the other profiles (8192).

Memory Footprint

The following table provides the set of valid configurations and the associated approximate memory footprints for the model.

nvidia/llama-nemotron-rerank-vl-1b-v2

12.0

fp8

Approximate GPU Memory Size (GiB)
3.95

fp16

Approximate GPU Memory Size (GiB)
5.0

10.0

fp8

Approximate GPU Memory Size (GiB)
4.24

fp16

Approximate GPU Memory Size (GiB)
5.32

9.0

fp8

Approximate GPU Memory Size (GiB)
3.88

fp16

Approximate GPU Memory Size (GiB)
4.94

8.9

fp8

Approximate GPU Memory Size (GiB)
3.72

fp16

Approximate GPU Memory Size (GiB)
4.77

8.6

fp16

Approximate GPU Memory Size (GiB)
4.27

8.0

fp16

Approximate GPU Memory Size (GiB)
4.74

nvidia/llama-nemotron-rerank-1b-v2

12.0

fp8

Approximate GPU Memory Size (GiB)
3.68

fp16

Approximate GPU Memory Size (GiB)
7.59

10.0

fp8

Approximate GPU Memory Size (GiB)
3.91

fp16

Approximate GPU Memory Size (GiB)
6.69

9.0

fp8

Approximate GPU Memory Size (GiB)
3.65

fp16

Approximate GPU Memory Size (GiB)
6.51

8.9

fp8

Approximate GPU Memory Size (GiB)
3.56

fp16

Approximate GPU Memory Size (GiB)
5.84

8.6

fp16

Approximate GPU Memory Size (GiB)
6.06

8.0

fp16

Approximate GPU Memory Size (GiB)
6.53

nvidia/llama-nemotron-rerank-500m-v2

a100-sxm4-40gb

fp16

Max Batch Size	Max Sequence Length	Approximate GPU Memory Size (GiB)
1	8192	2.53
8	8192	9.63
16	8192	18.25
30	1024	5.19
30	2048	8.65
30	4096	16.27
30	8192	32.91

a100-sxm4-80gb

fp16

Max Batch Size	Max Sequence Length	Approximate GPU Memory Size (GiB)
1	8192	2.53
8	8192	9.63
16	8192	18.25
30	1024	5.19
30	2048	8.65
30	4096	16.27
30	8192	32.91

a10g

fp16

Max Batch Size	Max Sequence Length	Approximate GPU Memory Size (GiB)
1	8192	2.78
8	8192	9.88
16	1024	3.66
16	2048	5.5
16	4096	9.38
16	8192	18.0

h100-hbm3-80gb

fp16

Max Batch Size	Max Sequence Length	Approximate GPU Memory Size (GiB)
1	8192	3.04
8	8192	10.38
16	8192	18.75
30	1024	5.69
30	2048	9.15
30	4096	16.77
30	8192	33.41

fp8

Max Batch Size	Max Sequence Length	Approximate GPU Memory Size (GiB)
1	8192	1.4
8	8192	4.5
16	8192	8.04
30	1024	2.16
30	2048	3.58
30	4096	6.66
30	8192	14.21

h100-nvl

fp16

Max Batch Size	Max Sequence Length	Approximate GPU Memory Size (GiB)
1	8192	3.04
8	8192	10.38
16	8192	18.75
30	1024	5.69
30	2048	9.15
30	4096	16.77
30	8192	33.41

fp8

Max Batch Size	Max Sequence Length	Approximate GPU Memory Size (GiB)
1	8192	1.4
8	8192	4.5
16	8192	8.04
30	1024	2.16
30	2048	3.58
30	4096	6.66
30	8192	14.21

l4

fp16

Max Batch Size	Max Sequence Length	Approximate GPU Memory Size (GiB)
1	8192	2.27
8	8192	9.81
16	1024	3.5
16	2048	5.38
16	4096	9.31
16	8192	18.06

l40s

fp16

Max Batch Size	Max Sequence Length	Approximate GPU Memory Size (GiB)
1	8192	2.6
8	8192	9.69
16	8192	17.81
30	1024	5.14
30	2048	8.59
30	4096	15.86
30	8192	32.03

fp8

Max Batch Size	Max Sequence Length	Approximate GPU Memory Size (GiB)
1	8192	1.52
8	8192	4.5
16	8192	8.03
30	1024	2.16
30	2048	3.58
30	4096	6.65
30	8192	14.21

Software

NVIDIA Driver

Release 1.6.0+ uses NVIDIA Optimized Frameworks 25.01. For NVIDIA driver support, refer to the Frameworks Support Matrix.

Ensure that the latest compatible NVIDIA driver is installed on your system before launching NIM containers. If you experience issues starting the containers, verify that your driver is up-to-date.

NVIDIA Container Toolkit

Your Docker environment must support NVIDIA GPUs. For more information, refer to NVIDIA Container Toolkit.

Related Topics

• Release Notes for NVIDIA NeMo Retriever Reranking NIM

• Deploy NVIDIA NeMo Retriever Reranking NIM on Kubernetes