Support Matrix for NVIDIA NeMo Retriever Reranking NIM#

This documentation describes the models, software, and hardware that NVIDIA NeMo Retriever Reranking NIM supports.

Models#

Reranking NIM supports the following models. When truncate is set to END, any Query / Passage pair that is longer than the maximum token length is truncated from the right, starting with the passage.

Model ID	Model Card	Max Tokens (Optimized Models)
nvidia/Nemotron-3.5-Rerank-8B-BF16	HF, NVIDIA	32768

The Max Tokens column lists the maximum that each model supports. For Nemotron 3.5 Rerank 8B, the NIM limits each query and passage pair to 10240 tokens by default. To allow up to 32768 tokens, set NIM_PIPELINE_MAX_SEQ_LEN. For more information, refer to Environment Variables for NVIDIA NeMo Retriever Reranking NIM.

Software#

NVIDIA Driver: Release 2.4 is built with CUDA 13.0.2. Use an NVIDIA driver that supports CUDA 13.0. For driver requirements, refer to CUDA Compatibility. Ensure that the latest compatible NVIDIA driver is installed on your system before launching NIM containers. If you experience issues starting the containers, verify that your driver is up-to-date.
NVIDIA Container Toolkit: Your Docker environment must support NVIDIA GPUs. For more information, refer to NVIDIA Container Toolkit.

CPUs#

Reranking NIM supports the following host CPU configurations:

x86 processor with at least 8 cores. For a list of supported systems, refer to NVIDIA Certified Systems Catalog.
NVIDIA DGX Spark with the integrated Arm CPU (aarch64/Arm64). DGX Spark support starts with version 2.3.

GPUs#

Reranking NIM supports the following GPUs. For details, refer to the optimized configuration table for each model later in this documentation.

Note

Currently, GPU clusters with GPUs in Multi-instance GPU mode (MIG) are not supported.

Optimized and Non-optimized Configurations#

The optimized configuration table in the section for each model lists the GPUs and precisions that are tuned and validated for the release. Optimized attention kernels are also an explicit runtime configuration; enable them only on supported hardware.

Non-optimized configurations use a fallback kernel feature set intended for broad compatibility, such as FP16 architecture-agnostic kernels. Fallback configurations can run on GPUs with sufficient memory, but they might not support every optimized feature or deliver the same performance as the optimized configurations.

Compute Capability and Kernel Configuration#

For the mapping of CUDA GPU compute capability versions to GPUs, refer to CUDA GPU Compute Capability.

For details about how the runtime selects an inference pipeline at startup, refer to Automatic Pipeline Selection.

Nemotron 3.5 Rerank 8B#

The model checkpoint is BF16, as the BF16 in the model ID shows. The NIM runs the model in FP16 by default (NIM_ENGINE_PRECISION=fp16). For more information, refer to Precision Override.

The following table contains the optimized configurations for Nemotron 3.5 Rerank 8B.

GPU	Precision	Max Tokens
NVIDIA RTX PRO 6000 Blackwell Server Edition	FP16	32768
NVIDIA GB200	FP16	32768
NVIDIA B300	FP16	32768
NVIDIA GB10 (DGX Spark)	FP16	32768
NVIDIA H100 80GB HBM3	FP16	32768
NVIDIA H100 NVL	FP16	32768
NVIDIA A100 SXM4 80GB	FP16	32768
NVIDIA L40S	FP16	32768
NVIDIA A10G	FP16	32768