Is this page helpful?

Release Notes for NVIDIA NeMo Retriever Embedding NIM#

This documentation contains the release notes for NVIDIA NeMo Retriever Embedding NIM.

Note

Some releases are labelled “Production Branch” or “(PB)”. Production Branches provide reliable, stable versions of the NIM. Non-production branch releases (sometimes called Feature Branch (FB) releases) contain the latest features, improvements, and optimizations.

Release 2.3.0#

This release adds support for NVIDIA DGX Spark to the nvidia/llama-nemotron-embed-vl-1b-v2 NIM.

Highlights#

  • Added FP16 and FP8 support for NVIDIA GB10 on DGX Spark.

  • Added aarch64/Arm64 container support for DGX Spark deployments.

  • The standard single-GPU Docker deployment workflow now supports DGX Spark. For details, refer to Get Started.

  • Removed NVIDIA RTX PRO 6000 Blackwell Workstation Edition from the claimed support set. NVIDIA RTX PRO 6000 Blackwell Server Edition remains supported.

For details, refer to Support Matrix.

Release 2.2.2#

This patch release includes the following updates:

  • Adds the com.nvidia.nim.model_download_protocol=native-v1 container label for NIM Operator deployment compatibility (requires NIM Operator >= 3.1.2).

  • The non-default RUST_LOG configuration is removed so that logging is controlled by the product default or explicit user configuration.

Release 2.2.1#

Release 2.2.1 was intentionally skipped.

Release 2.2.0#

This release adds the nvidia/nemotron-3-embed-1b NIM, a text embedding NIM for query and passage embeddings.

Highlights#

This release contains the following key changes:

  • Added support for nvidia/nemotron-3-embed-1b.

  • Added the NIM_PERFORMANCE_MODE environment variable for selecting latency-oriented or throughput-oriented defaults for unset batching and engine-count environment variables. For details, refer to Environment Variables for NVIDIA NeMo Retriever Embedding NIM.

Support Matrix Updates#

  • Added NVFP4 optimized pipelines for NVIDIA GB200 and NVIDIA RTX PRO 6000 Blackwell Server Edition.

  • Added support for BF16 pipelines on supported NVIDIA GPUs, including NVIDIA GB10.

For details, refer to Support Matrix for NVIDIA NeMo Retriever Embedding NIM.

Compatibility Updates#

  • nvidia/nemotron-3-embed-1b supports text inputs with the /v1/embeddings endpoint.

  • The validated maximum sequence length is 4096 tokens.

  • The model returns native 2048-dimensional embeddings when embedding_type is float. For supported compressed embedding types, refer to Support Matrix for NVIDIA NeMo Retriever Embedding NIM.

  • Dynamic embedding dimensions and image inputs are not supported for this model.

Release 2.1.0#

Release 2.1.0 was intentionally skipped.

Release 2.0.0#

This release represents a major runtime upgrade for the nvidia/llama-nemotron-embed-vl-1b-v2 NIM that includes a new purpose-built embedding inference stack. Compared to earlier versions, the new runtime delivers higher throughput and lower latency across all supported GPU SKUs, smaller VRAM footprint, faster startup time, and smaller container size.

Note

The model, supported modalities (text | image | text_image), and API are unchanged.

Highlights#

This release contains the following key changes:

  • The runtime selects optimized CUDA kernels automatically at startup based on the GPU’s compute capability. No manual profile selection steps are required.

  • Added support for loading model artifacts from Hugging Face or NGC.

    • To use Hugging Face (default), set HF_TOKEN.

    • To use NGC, set NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc and NGC_API_KEY.

  • Added the NIM_PERFORMANCE_MODE environment variable. Set NIM_PERFORMANCE_MODE=1 to use throughput-oriented defaults for unset batching and engine-count variables. Explicit environment variables override these defaults.

  • The NIM_ENGINE_COUNT env var defaults to 1, or 2 when NIM_PERFORMANCE_MODE=1 and NIM_ENGINE_COUNT is unset.

  • Added the SHOW_CONFIG environment variable. Setting SHOW_CONFIG=1 at runtime lists the environment variables configured for the NIM.

  • You can opt-in to gRPC by setting NIM_SERVER_GRPC_BIND_ADDR.

  • New environment variables. For details, refer to Environment Variables for NVIDIA NeMo Retriever Embedding NIM.

  • NIM_ENGINE_MODEL_NAME configures the model name returned in embedding responses. NIM_SERVER_SERVED_MODEL_NAME configures legacy served model aliases for /v1/models and request model matching.

  • Environment variables use the NIM_SERVER_*, NIM_PIPELINE_*, and NIM_ENGINE_* namespaces in this version. Earlier flat and Triton-era names are not accepted as aliases. Common migrations include the following:

    • NIM_HTTP_API_PORT is now NIM_SERVER_BIND_ADDR.

    • NIM_TRITON_GRPC_PORT is now NIM_SERVER_GRPC_BIND_ADDR.

    • NIM_MODEL_NAME is now NIM_ENGINE_MODEL_NAME.

    • NIM_SERVED_MODEL_NAME is now NIM_SERVER_SERVED_MODEL_NAME.

    • NIM_MODEL_DOWNLOAD_PROVIDER is now NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER.

    • NIM_MODEL_PATH is now NIM_ENGINE_MODEL_PATH.

    • NIM_PRECISION is now NIM_ENGINE_PRECISION.

    • NIM_PRECOMPILE_ONLY is now NIM_ENGINE_PRECOMPILE_ONLY.

    • NIM_MAX_BATCH_SIZE is now NIM_PIPELINE_MAX_BATCH_SIZE. The default is 1 in latency mode and 64 in throughput mode. Set NIM_PERFORMANCE_MODE=1 or configure NIM_PIPELINE_MAX_BATCH_SIZE=64 explicitly for the throughput-oriented batch size.

    • NIM_MAX_SEQ_LEN is now NIM_PIPELINE_MAX_SEQ_LEN.

    • NIM_LOG_LEVEL is now RUST_LOG.

    • NIM_LOGGING_JSONL is now LOG_FORMAT=json.

    • NIM_NUM_MODEL_INSTANCES and NIM_TRITON_MODEL_INSTANCE_COUNT are now NIM_ENGINE_COUNT.

    • NIM_TRITON_DYNAMIC_BATCHING_MAX_QUEUE_DELAY_MICROSECONDS is now NIM_SERVER_MAX_WAIT_MS. Convert the old value from microseconds to milliseconds by dividing it by 1,000. For example, 10000 microseconds becomes 10 milliseconds. The new variable accepts only whole milliseconds, so values that are not multiples of 1,000 cannot be represented exactly.

  • The following environment variables are removed in this release with no replacement. NIM_CACHE_PATH, NIM_CUSTOM_MODEL, NIM_HTTP_MAX_WORKERS, NIM_HTTP_TRITON_PORT, NIM_IGNORE_MODEL_DOWNLOAD_FAIL, NIM_MANIFEST_ALLOW_UNSAFE, NIM_MANIFEST_PATH, NIM_MODEL_PROFILE, NIM_NUM_TOKENIZERS, NIM_REPOSITORY_OVERRIDE, NIM_TELEMETRY_MODE, NIM_TELEMETRY_ENABLE_ON_RTX, NIM_TELEMETRY_INTERVAL_MINUTES, NIM_TRITON_LOG_VERBOSE, NIM_TRITON_PERFORMANCE_MODE.

Support Matrix and Compatibility Updates#

  • The supported optimized SKUs are the following. For details, refer to Support Matrix for NVIDIA NeMo Retriever Embedding NIM.

    • FP16 on: B200, GB200, RTX PRO 6000, H100, H200, L40S, A100, A10G, L4

    • FP8 on B200, GB200, RTX PRO 6000, H100, H200, L40S

  • The default precision is now determined automatically based on the GPU architecture. Set NIM_ENGINE_PRECISION=fp16 to opt into FP16.

All Known Issues#

The known issues for NeMo Retriever Embedding NIM are the following:

  • The container health check used by Docker and Docker Compose can report the container as unhealthy even when the NIM is ready. This can block Compose dependencies that require service_healthy.

    Workaround: Override the health check in Docker Compose:

    healthcheck:
  test: ["CMD", "/usr/bin/curl", "-fsS", "http://127.0.0.1:8000/v1/health/ready"]

  • For nvidia/nemotron-3-embed-1b, setting NIM_SERVER_MAX_QUEUE_SIZE=0 can cause the NIM to log an error during padded-batcher startup instead of rejecting the invalid configuration. Use the default value, 1024, or set NIM_SERVER_MAX_QUEUE_SIZE to a positive integer.

Release Notes for Previous Versions#