Is this page helpful?

Support Matrix for NVIDIA NeMo Retriever Reranking NIM#

This documentation describes the models, software, and hardware that NVIDIA NeMo Retriever Reranking NIM supports.

Models#

Reranking NIM supports the following models.

Model ID

Model Card

Max Tokens
(Optimized Models)

nvidia/llama-nemotron-rerank-vl-1b-v2

Model card

8192

nvidia/llama-nemotron-rerank-1b-v2

Model card

8192

nvidia/llama-nemotron-rerank-500m-v2

Model card

8192

Note that when truncate is set to END, any Query / Passage pair that is longer than the maximum token length is truncated from the right, starting with the passage.

Software#

  • NVIDIA Driver: Release 1.6.0+ uses NVIDIA Optimized Frameworks 25.01. For NVIDIA driver support, refer to the Frameworks Support Matrix. Ensure that the latest compatible NVIDIA driver is installed on your system before launching NIM containers. If you experience issues starting the containers, verify that your driver is up-to-date.

  • NVIDIA Container Toolkit: Your Docker environment must support NVIDIA GPUs. For more information, refer to NVIDIA Container Toolkit.

CPUs#

Reranking NIM supports the following host CPU configurations:

GPUs#

Reranking NIM supports the following GPUs:

Optimized and Non-optimized Configurations#

Starting in version 2.0, optimized configurations for nvidia/llama-nemotron-rerank-vl-1b-v2 use runtime CUDA kernels and just-in-time compilation. The NIM uses FP16 kernels by default. FP8 kernels are available only for the GPUs listed with FP8 support and must be requested explicitly with NIM_ENGINE_PRECISION=fp8.

The optimized configuration table in the section for each model lists the GPUs and precisions that are tuned and validated for the release. Optimized attention kernels are also an explicit runtime configuration; enable them only on supported hardware.

Non-optimized configurations use a fallback kernel feature set intended for broad compatibility, such as FP16 architecture-agnostic kernels. Fallback configurations can run on GPUs with sufficient memory, but they might not support every optimized feature or deliver the same performance as the optimized configurations.

Compute Capability and Kernel Configuration#

Starting in version 2.0, the NIM does not perform automatic kernel selection for nvidia/llama-nemotron-rerank-vl-1b-v2. The runtime uses the default FP16 kernel path unless you explicitly request another supported configuration.

Use the optimized configuration table for llama-nemotron-rerank-vl-1b-v2 to determine whether the target GPU supports FP8. To opt into FP8 kernels, set NIM_ENGINE_PRECISION=fp8. To use optimized attention kernels, enable the corresponding runtime configuration for the deployment. To override the default FP16 precision, refer to Precision Override.

To see the mapping of CUDA GPU compute capability versions to GPUs, refer to CUDA GPU Compute Capability.

For details about how the runtime selects an inference pipeline at startup, refer to Automatic Pipeline Selection.

Supported Hardware for each Model#

Note

Currently, GPU clusters with GPUs in Multi-instance GPU mode (MIG) are not supported.

llama-nemotron-rerank-vl-1b-v2#

Optimized configuration#

GPU

Precision

Max Tokens

NVIDIA GB10 (DGX Spark)

FP16

8192

NVIDIA RTX PRO 6000 Blackwell Server Edition

FP16 and FP8

8192

NVIDIA B200

FP16 and FP8

8192

NVIDIA H100 NVL

FP16 and FP8

8192

NVIDIA H100 80GB HBM3

FP16 and FP8

8192

NVIDIA A100 SXM4 80GB

FP16

8192

Note

NVIDIA DGX Spark uses a unified memory architecture in which the CPU and GPU share system memory, rather than dedicated GPU framebuffer memory. For NVIDIA-GB10, conventional VRAM total, used, and free measurements are therefore not available or directly comparable with measurements from discrete GPUs. This does not affect NIM support. For details, refer to Guidance for reporting memory resources with unified memory architecture.

FP8 availability for llama-nemotron-rerank-vl-1b-v2 is GPU-specific. Use the precision listed for the target GPU. For NVIDIA-GB10, NIM_ENGINE_PRECISION=fp8 is accepted but falls back to FP16 numerics; native FP8 acceleration is unavailable.

By default, the runtime uses NIM_ENGINE_COUNT=1. For the maximum compatibility profile, keep or set NIM_ENGINE_COUNT=1 explicitly. For maximum performance on NVIDIA DGX Spark (NVIDIA-GB10) or discrete GPUs with at least 80 GB of VRAM, set NIM_ENGINE_COUNT=2.

Non-optimized configuration#

The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.

GPUs

GPU Memory

Precision

Disk Space

Max Tokens

Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory

7.30

FP16

3.10

8192

llama-nemotron-rerank-1b-v2#

Optimized configuration#

Compute Capability

Precision

12.0

FP16 and FP8

10.0

FP16 and FP8

9.0

FP16 and FP8

8.9

FP16 and FP8

8.6

FP16

8.0

FP16

Non-optimized configuration#

The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.

GPUs

GPU Memory

Precision

Disk Space

Max Tokens

Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory

3.6

FP16

9.5

4096

Warning

The maximum token length of the non-optimized configuration is smaller (4096) than the other profiles (8192).

llama-nemotron-rerank-500m-v2#

Optimized configuration#

Compute Capability

Precision

12.0

FP16 and FP8

10.0

FP16 and FP8

9.0

FP16 and FP8

8.9

FP16 and FP8

8.6

FP16

8.0

FP16

Non-optimized configuration#

The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.

GPUs

GPU Memory

Precision

Disk Space

Max Tokens

Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory

3.6

FP16

9.5

4096

Warning

The maximum token length of the non-optimized configuration is smaller (4096) than the other profiles (8192).