Is this page helpful?

Optimization for NVIDIA NeMo Retriever Reranking NIM#

Use this documentation to learn about runtime optimization for NVIDIA NeMo Retriever Reranking NIM.

Automatic Pipeline Selection#

Starting in version 2.0, at startup NeMo Retriever Reranking NIM detects the compute capability of the GPU and automatically selects an optimized inference pipeline for that GPU. The selected pipeline determines the CUDA kernels, precompiled cuDNN plans, attention implementation, and default precision. No user action is required to select profiles, and the list-model-profiles command is not available.

The selected pipeline and its precision are logged at startup. For example, the log includes a line similar to the following.

INFO CUDA device ready gpu="NVIDIA H100 80GB HBM3" cc=90
INFO loading rerank engines engine_count=2 ...

Precision Override#

Precision support depends on the model and its container runtime, not only on the GPU.

Nemotron 3.5 Rerank 8B#

For Nemotron 3.5 Rerank 8B, the NIM uses FP16 by default. To have the NIM choose the precision from the model checkpoint, set NIM_ENGINE_PRECISION=auto.

The following table lists the NIM_ENGINE_PRECISION values for Nemotron 3.5 Rerank 8B.

Value

Result

fp16

Uses FP16. This is the default.

auto

Uses FP16, or NVFP4 when the checkpoint has NVFP4 weights and the GPU supports NVFP4.

nvfp4

Uses NVFP4. Requires a checkpoint with NVFP4 weights and an NVIDIA Blackwell GPU.

bf16, fp8, fp8-e4m3

Not supported. The NIM fails to start.

If you set a precision that the model, checkpoint, or GPU doesn’t support, the NIM fails to start and logs an error. It doesn’t fall back to another precision.

docker run ... \
  -e NIM_ENGINE_PRECISION=auto \
  $IMG_NAME

Override Profile Selection#

Starting in version 2.0, profile selection is automatic and you can’t override it by using NIM_MODEL_PROFILE. To request a specific precision, refer to Precision Override.