Is this page helpful?

Release Notes for NVIDIA NeMo Retriever Reranking NIM#

This documentation contains the release notes for NVIDIA NeMo Retriever Reranking NIM.

Note

Some releases are labelled “Production Branch” or “(PB)”. Production Branches provide reliable, stable versions of the NIM. Non-production branch releases (sometimes called Feature Branch (FB) releases) contain the latest features, improvements, and optimizations.

Release 2.0.0#

Summary#

  • Major runtime upgrade for the nvidia/llama-nemotron-rerank-vl-1b-v2 NIM that includes a new purpose-built reranking inference stack. Compared to earlier versions, the new runtime delivers higher throughput and lower latency across all supported GPU SKUs, smaller VRAM footprint, faster startup time, and smaller container size.

  • The model, the multimodal capability, and the OpenAI-compatible /v1/ranking API surface are unchanged.

  • The runtime selects optimized CUDA kernels automatically at startup based on the GPU’s compute capability. No manual profile selection steps are required.

  • The supported optimized SKUs are the following. For details, refer to Support Matrix for NVIDIA NeMo Retriever Reranking NIM.

    • FP16 on: B200, RTX PRO 6000, H100, H200, L40s, A100, A10G, L4

    • FP8 on: H100, H200, RTX PRO 6000

  • The default precision is now FP16 when supported. Set NIM_PRECISION=fp8 to opt into FP8.

  • Added support for loading model artifacts from Hugging Face or NGC.

    • To use Hugging Face (default), set HF_TOKEN.

    • To use NGC, set NIM_MODEL_DOWNLOAD_PROVIDER=ngc and NGC_API_KEY.

  • The NIM_ENGINE_COUNT env var defaults to 1. For details, refer to Engine Count.

  • Added the SHOW_CONFIG environment variable. Setting SHOW_CONFIG=1 at runtime lists the environment variables configured for the NIM.

  • You can opt-in to gRPC by setting NIM_GRPC_BIND_ADDR.

  • New environment variables. For details, refer to Environment Variables for NVIDIA NeMo Retriever Reranking NIM.

  • NIM_MODEL_NAME is supported for served API model aliasing. NIM_SERVED_MODEL_NAME remains supported for served API aliasing and currently takes precedence when both variables are configured.

  • The following environment variables are renamed in this version:

    • NIM_HTTP_API_PORT is now NIM_BIND_ADDR.

    • NIM_LOG_LEVEL is now RUST_LOG.

    • NIM_LOGGING_JSONL is now LOG_FORMAT=json.

    • NIM_TRITON_GRPC_PORT is now NIM_GRPC_BIND_ADDR.

  • The following environment variables are deprecated aliases in this release. The aliases still work, but will be removed in a future release.

    • NIM_NUM_MODEL_INSTANCES and NIM_TRITON_MODEL_INSTANCE_COUNT are now NIM_ENGINE_COUNT.

  • The following environment variables are removed in this release with no replacement. NIM_CACHE_PATH, NIM_HTTP_MAX_WORKERS, NIM_HTTP_TRITON_PORT, NIM_IGNORE_MODEL_DOWNLOAD_FAIL, NIM_MANIFEST_ALLOW_UNSAFE, NIM_MANIFEST_PATH, NIM_MODEL_PROFILE, NIM_NUM_TOKENIZERS, NIM_REPOSITORY_OVERRIDE, NIM_TELEMETRY_MODE, NIM_TELEMETRY_ENABLE_ON_RTX, NIM_TELEMETRY_INTERVAL_MINUTES, NIM_TRITON_DYNAMIC_BATCHING_MAX_QUEUE_DELAY_MICROSECONDS, NIM_TRITON_LOG_VERBOSE, NIM_TRITON_PERFORMANCE_MODE.

Known Issues#

Release 1.11.0#

Highlights#

Release 1.10.0#

Highlights#

  • Rename existing models to the new Nemotron brand. The impacted models are the following:

    • The llama-3.2-nemoretriever-500m-rerank-v2 model is now named llama-nemotron-rerank-500m-v2.

    • The llama-3.2-nv-rerankqa-1b-v2 model is now named llama-nemotron-rerank-1b-v2.

  • Add fixes for high and critical vulnerabilities.

Fixed Known Issues#

The following are the known issues that are fixed in this version:

  • Fixed an issue with the persistence.enabled helm chart value. Persistent storage options (persistence.storageClass, persistence.existingClaim, hostPath.enabled) are now fully functional.

Release 1.9 - Production Branch Only#

This release is a production branch.

Highlights#

  • 1.9.0: Production branch release of llama-3.2-nv-rerankqa-1b-v2 with STIG/FIPS base image.

  • 1.9.0: Upgraded to use Triton Inference server version 25.08.03 to address CVEs.

  • CUDA version changed from 12.9 to 13. For details, refer to What’s New and Important in CUDA Toolkit 13.0.

  • 1.9.1 - 1.9.x: CVE fixes for high & critical vulnerabilities.

  • 1.9.1: Updated the API to return HTTP 422 (Unprocessable Content) instead of HTTP 400 (Bad Request) when the input text exceeds the maximum token length.

Known Issues#

There are no known issues in this release.

Release 1.8.0#

Summary#

  • Upgraded to use Triton Inference Server 25.08 to address CVEs.

  • Added TRT optimized engines for CUDA GPU Compute Capability. Support includes 12.0, 10.0, 9.0, 8.9, 8.6, and 8.0.

Known Issues#

  • The persistence.enabled value and all related dependent configuration flags are currently non-functional in the NIM helm chart.

Release 1.7.0#

Summary#

Known Issues#

  • The persistence.enabled value and all related dependent configuration flags are currently non-functional in the NIM helm chart.

Release 1.6.0#

Summary#

  • Added support for Llama-3.2-nemoretriever-500m-rerank-v2 reranking model.

Release 1.5.0#

Summary#

  • Added support for B200 GPU.

Known Issues#

  • The list-model-profiles command incorrectly lists compatible model profiles as incompatible. Select the profile that matches your hardware configuration. This bug does not impact automatic profile selection.

  • Slight performance degradation observed since 1.3.1 release.

Release 1.4.0#

Summary#

  • Added support for configurable memory footprint by allowing users to set batch size and sequence length.

  • Added the NIM_TRITON_MAX_BATCH_SIZE environment variable.

  • Reduced container image sizes.

  • Removed model profiles for A100 PCIe 40GB & H100 PCIe 80GB configurations.

  • Fixed bug where list-model-profiles command fails to run on hosts that don’t have an NVIDIA GPUs, even when NIM_CPU_ONLY is set.

Known Issues#

  • The list-model-profiles command incorrectly lists compatible model profiles as incompatible. Select the profile that matches your hardware configuration. This bug does not impact automatic profile selection.

  • Slight performance degradation observed since 1.3.1 release.

Release 1.3.1#

Release 1.3.0#

  • Added support for Llama-3.2-NV-RerankQA-1B-v2 reranking model.

  • Added NIM_NUM_MODEL_INSTANCES and NIM_NUM_TOKENIZERS environment variables.

  • Added support for dynamic batching in the underlying Triton Inference Server process.

Known Issues#

  • The current version of langchain-nvidia-ai-endpoints used in the LangChain playbook is not compatible with the Llama-3.2-NV-RerankQA-1B-v2 NIM.

Release 1.0.2#

  • Improved accuracy for model running on A100 and A10G GPUs.

Release 1.0.1#

  • Added support for NGC Personal/Service API keys in addition to the NGC API Key (Original).

  • NGC_API_KEY is no longer required when running a container with a pre-populated cache (NIM_CACHE_PATH).

  • list-model-profiles command updated to check the correct location for model artifacts.

Release 1.0.0#

Summary#

This is the first general release of the NVIDIA NeMo Retriever Reranking NIM.

Reranking Models#

  • NV-RerankQA-Mistral4B-v3