Support Matrix for NVIDIA NeMo Retriever Embedding NIM#
This documentation describes the models, software, and hardware that NVIDIA NeMo Retriever Embedding NIM supports.
Models#
Embedding NIM supports the following models.
Model ID |
Model Card |
Supported Embedding Types |
Max Tokens |
Parameters |
Total Parameters |
Embedding |
Dynamic Embeddings |
|---|---|---|---|---|---|---|---|
nvidia/nemotron-3-embed-1b |
|
4096 |
872 |
1141 |
20482 |
no |
|
nvidia/llama-nemotron-embed-vl-1b-v2 |
|
2048 |
1414 |
1678 |
2048 |
yes |
|
nvidia/llama-nemotron-embed-1b-v2 |
|
8192 |
973 |
1236 |
2048 |
yes |
|
nvidia/llama-nemotron-embed-300m-v2 |
|
8192 |
307 |
569 |
2048 |
yes |
|
nvidia/nv-embedqa-e5-v5 |
|
512 |
303 |
335 |
1024 |
no |
|
baai/bge-m3 |
|
8192 |
303 |
568 |
1024 |
no |
|
baai/bge-large-zh-v1.5 |
|
512 |
303 |
325 |
1024 |
no |
Note
1 The “Parameters (excl. embeddings)” column shows the count of parameters that directly impact inference performance and computational cost. Embedding layer parameters are excluded because they primarily affect model size rather than inference speed. For example, models with different vocabulary sizes may have different total parameter counts but the same inference-relevant parameter count.
Note
2 The dimensions request parameter is model-specific. Models that support dynamic embeddings can return reduced embedding sizes. The model nvidia/nemotron-3-embed-1b does not support reduced dimensions. Its native embedding dimension is 2048, and the only supported dimensions value is 2048. You can also omit the parameter.
Software#
NVIDIA Driver: Release 1.7.0+ uses NVIDIA Optimized Frameworks 25.01. For NVIDIA driver support, refer to the Frameworks Support Matrix. Ensure that the latest compatible NVIDIA driver is installed on your system before launching NIM containers. If you experience issues starting the containers, verify that your driver is up-to-date.
NVIDIA Container Toolkit: Your Docker environment must support NVIDIA GPUs. For more information, refer to NVIDIA Container Toolkit.
CPUs#
Embedding NIM supports the following host CPU configurations:
x86 processor with at least 8 cores. For a list of supported systems, refer to NVIDIA Certified Systems Catalog.
NVIDIA DGX Spark with the integrated Arm CPU (
aarch64/Arm64). DGX Spark support starts with version 2.3.
GPUs#
Embedding NIM supports the following GPUs. For details, refer to the optimized configuration table for each model later in this documentation.
NVIDIA H20
NVIDIA L20
Note
Currently, GPU clusters with GPUs in Multi-instance GPU mode (MIG) are not supported.
Optimized and Non-optimized Configurations#
Starting in version 2.0, optimized configurations use runtime CUDA kernels and just-in-time compilation. At startup, the NIM selects a kernel feature set for the detected GPU architecture. Depending on the selected feature set, the NIM might compile kernels, load precompiled kernels optimized for that architecture, or use both.
The optimized configuration table in the section for each model lists the compute capability families or GPUs that have optimized kernel support for the listed precision. These configurations are tuned and validated for the release.
Non-optimized configurations use a fallback kernel feature set intended for broad compatibility, such as FP16 architecture-agnostic kernels. Fallback configurations can run on GPUs with sufficient memory, but they might not support every optimized feature or deliver the same performance as the optimized configurations.
Compute Capability and Automatic Kernel Selection#
Starting in version 2.0, the NIM detects the GPU compute capability at startup and selects the supported kernel feature set for that compute capability family.
The selected feature set determines which CUDA kernels, attention implementation, precompiled kernel artifacts, and default precision are used. If an optimized feature set is not available for the detected GPU, the NIM uses the compatible fallback feature set.
To request a precision explicitly, set NIM_ENGINE_PRECISION to a value supported by the model and GPU. FP8 and NVFP4 are available only on the compute capability families or GPUs listed in the optimized configuration table in the section for each model. To override the automatically selected precision, refer to Precision Override.
For details about how the runtime selects an inference pipeline at startup, refer to Automatic Pipeline Selection.
Nemotron 3 Embed 1B#
The following table contains the optimized configurations for Nemotron 3 Embed 1B.
GPU |
Precision |
|---|---|
NVIDIA RTX PRO 6000 Blackwell Server Edition |
NVFP4 and FP16 |
NVIDIA GB200 |
NVFP4 and FP16 |
NVIDIA A100 SXM4 80GB |
FP16 |
NVIDIA H100 80GB HBM3 |
FP16 |
NVIDIA H100 NVL |
FP16 |
NVIDIA L40S |
FP16 |
NVIDIA A10G |
FP16 |
NVIDIA GB10 (DGX Spark) |
FP16 |
Note
NVIDIA DGX Spark uses a unified memory architecture in which the CPU and GPU share system memory, rather than dedicated GPU framebuffer memory. For NVIDIA-GB10, conventional VRAM total, used, and free measurements are therefore not available or directly comparable with measurements from discrete GPUs. This does not affect NIM support. For details, refer to Guidance for reporting memory resources with unified memory architecture.
Fallback behavior on GPUs that are not listed has not been verified for this model.
Note
nvidia/nemotron-3-embed-1b is a text-only embedding model. It supports query and passage inputs, returns native 2048-dimensional embeddings when embedding_type is float, and is validated with a maximum sequence length of 4096 tokens. To use the native embedding dimension, omit the dimensions request parameter or set it to 2048.
NVFP4 Batch Token Budget#
NIM_PIPELINE_MAX_BATCH_TOKENS is an optional post-tokenization token budget for each Nemotron 3 embedding GPU dispatch. Set it to 0, or leave it unset, to disable token-budget chunking. This setting does not change NIM_PIPELINE_MAX_SEQ_LEN or API response semantics. The runtime splits CPU-tokenized batches only on request boundaries when possible. If a single request exceeds the configured token budget, it can still run alone.
Use the GPU-specific value from the validated release manifest. Do not copy values between GPUs without validation.
GPU / profile |
Pipeline |
Recommended value |
Notes |
|---|---|---|---|
NVIDIA RTX PRO 6000 Blackwell Server Edition, SM120 |
|
|
Dedicated RTX PRO release manifest value. Runtime profile uses |
NVIDIA GB200, SM100 |
|
|
Dedicated GB200 release manifest value. Do not reuse the NVIDIA B200 |
Llama Nemotron Embed VL 1B v2#
The following table contains the optimized configurations for Llama Nemotron Embed VL 1B v2.
GPU |
Precision |
|---|---|
NVIDIA GB10 (DGX Spark) |
FP8 and FP16 |
NVIDIA RTX PRO 6000 Blackwell Server Edition |
FP8 and FP16 |
NVIDIA B200 |
FP8 and FP16 |
NVIDIA GB200 |
FP8 and FP16 |
NVIDIA H200 |
FP8 and FP16 |
NVIDIA A100 SXM4 80GB |
FP16 |
NVIDIA H100 NVL |
FP8 and FP16 |
NVIDIA H100 80GB HBM3 |
FP8 and FP16 |
NVIDIA L4 |
FP16 |
NVIDIA L40S |
FP8 and FP16 |
NVIDIA A10G |
FP16 |
Note
NVIDIA DGX Spark uses a unified memory architecture in which the CPU and GPU share system memory, rather than dedicated GPU framebuffer memory. For NVIDIA-GB10, conventional VRAM total, used, and free measurements are therefore not available or directly comparable with measurements from discrete GPUs. This does not affect NIM support. For details, refer to Guidance for reporting memory resources with unified memory architecture.
Fallback behavior on GPUs that are not listed has not been verified for this model.
Note
The default VLM profile uses a maximum sequence length of 2048 tokens. Image inputs are supported only as document or passage inputs.
Llama Nemotron Embed 1B v2#
The following table contains the optimized configurations for Llama Nemotron Embed 1B v2.
Precision |
|
|---|---|
12.0 |
FP16 and FP8 |
10.0 |
FP16 and FP8 |
9.0 |
FP16 and FP8 |
8.9 |
FP16 and FP8 |
8.6 |
FP16 |
8.0 |
FP16 |
The following table contains the non-optimized configurations for Llama Nemotron Embed 1B v2. The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model. If you run this model on RTX 40xx or later, you need a minimum of 8GB of VRAM.
GPUs |
GPU Memory |
Precision |
Disk Space |
Max Tokens |
|---|---|---|---|---|
Any single NVIDIA GPU that has sufficient memory, or multiple homogenous NVIDIA GPUs that have sufficient memory in total. |
3.6 |
FP16 |
9 |
4096 |
Warning
The maximum token length of the non-optimized configuration is smaller (4096) than the other profiles (8192).
Llama Nemotron Embed 300m v2#
The following table contains the optimized configurations for Llama Nemotron Embed 300m v2.
Precision |
|
|---|---|
12.0 |
FP16 and FP8 |
10.0 |
FP16 and FP8 |
9.0 |
FP16 and FP8 |
8.9 |
FP16 and FP8 |
8.6 |
FP16 |
8.0 |
FP16 |
The following table contains the non-optimized configurations for Llama Nemotron Embed 300m v2. The GPU Memory and Disk Space values are in GB unless otherwise specified; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
Max Tokens |
|---|---|---|---|---|
Any single NVIDIA GPU that has sufficient memory, or multiple homogenous NVIDIA GPUs that have sufficient memory in total. |
Min: 2.4 GiB, Max: 25.2 GiB |
FP16 |
7.49 GiB |
4096 |
Warning
The maximum token length of the non-optimized configuration is smaller (4096) than the other profiles (8192).
NVIDIA QA E5 Embedding#
The following table contains the optimized configurations for NVIDIA QA E5 Embedding.
Precision |
|
|---|---|
12.0 |
FP16 |
10.0 |
FP16 |
9.0 |
FP16 |
8.9 |
FP16 |
8.6 |
FP16 |
8.0 |
FP16 |
The following table contains the non-optimized configurations for NVIDIA QA E5 Embedding. The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
|---|---|---|---|
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory |
2 |
FP16 |
8.5 |
BAAI bge-large-zh-v1.5#
The following table contains the optimized configurations for BAAI bge-large-zh-v1.5.
GPU |
GPU Memory (GB) |
Precision |
|---|---|---|
H20 |
96 |
FP16 |
L20 |
48 |
FP16 |
The following table contains the non-optimized configurations for BAAI bge-large-zh-v1.5. The GPU Memory and Disk Space values in the following table are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
|---|---|---|---|
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory |
10 |
FP16 |
8.1 |
BAAI bge-m3#
The following table contains the optimized configurations for BAAI bge-m3.
GPU |
GPU Memory (GB) |
Precision |
|---|---|---|
A100 SXM4 |
80 |
FP16 |
H100 HBM3 |
80 |
FP16 |
L40S |
48 |
FP16 |
A10G |
24 |
FP16 |
L20 |
48 |
FP16 |
H20 |
96 |
FP16 |
The following table contains the non-optimized configurations for BAAI bge-m3. The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
|---|---|---|---|
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory |
33 |
FP16 |
8.8 |