Support Matrix for NVIDIA NeMo Retriever Embedding NIM#
This documentation describes the models, software, and hardware that NVIDIA NeMo Retriever Embedding NIM supports.
Models#
Embedding NIM supports the following models.
Model ID |
Model Card |
Supported Embedding Types |
Max Tokens |
Parameters |
Total Parameters |
Embedding |
Dynamic Embeddings |
|---|---|---|---|---|---|---|---|
nvidia/nemotron-3-embed-1b |
|
4096 |
872 |
1141 |
20482 |
no |
|
nvidia/llama-nemotron-embed-vl-1b-v2 |
|
2048 |
1414 |
1678 |
2048 |
yes |
|
nvidia/llama-nemotron-embed-1b-v2 |
|
8192 |
973 |
1236 |
2048 |
yes |
|
nvidia/llama-nemotron-embed-300m-v2 |
|
8192 |
307 |
569 |
2048 |
yes |
|
nvidia/nv-embedqa-e5-v5 |
|
512 |
303 |
335 |
1024 |
no |
|
baai/bge-m3 |
|
8192 |
303 |
568 |
1024 |
no |
|
baai/bge-large-zh-v1.5 |
|
512 |
303 |
325 |
1024 |
no |
Note
1 The “Parameters (excl. embeddings)” column shows the count of parameters that directly impact inference performance and computational cost. Embedding layer parameters are excluded because they primarily affect model size rather than inference speed. For example, models with different vocabulary sizes may have different total parameter counts but the same inference-relevant parameter count.
Note
2 The dimensions request parameter is model-specific. Models that support dynamic embeddings can return reduced embedding sizes. The model nvidia/nemotron-3-embed-1b does not support reduced dimensions. Its native embedding dimension is 2048, and the only supported dimensions value is 2048. You can also omit the parameter.
Software#
NVIDIA Driver: Release 1.7.0+ uses NVIDIA Optimized Frameworks 25.01. For NVIDIA driver support, refer to the Frameworks Support Matrix. Ensure that the latest compatible NVIDIA driver is installed on your system before launching NIM containers. If you experience issues starting the containers, verify that your driver is up-to-date.
NVIDIA Container Toolkit: Your Docker environment must support NVIDIA GPUs. For more information, refer to NVIDIA Container Toolkit.
CPUs#
Embedding NIM supports the following host CPU configurations:
x86 processor with at least 8 cores. For a list of supported systems, refer to NVIDIA Certified Systems Catalog.
NVIDIA DGX Spark with the integrated Arm CPU (
aarch64/Arm64). DGX Spark support starts with version 2.3.
GPUs#
Embedding NIM supports the following GPUs:
Optimized and Non-optimized Configurations#
Starting in version 2.0, optimized configurations use runtime CUDA kernels and just-in-time compilation. At startup, the NIM selects a kernel feature set for the detected GPU architecture. Depending on the selected feature set, the NIM might compile kernels, load precompiled kernels optimized for that architecture, or use both.
The optimized configuration table in the section for each model lists the compute capability families or GPUs that have optimized kernel support for the listed precision. These configurations are tuned and validated for the release.
Non-optimized configurations use a fallback kernel feature set intended for broad compatibility, such as FP16 architecture-agnostic kernels. Fallback configurations can run on GPUs with sufficient memory, but they might not support every optimized feature or deliver the same performance as the optimized configurations.
Compute Capability and Automatic Kernel Selection#
Starting in version 2.0, the NIM detects the GPU compute capability at startup and selects the supported kernel feature set for that compute capability family.
The selected feature set determines which CUDA kernels, attention implementation, precompiled kernel artifacts, and default precision are used. If an optimized feature set is not available for the detected GPU, the NIM uses the compatible fallback feature set.
To request a precision explicitly, set NIM_ENGINE_PRECISION to a value supported by the model and GPU. FP8 and NVFP4 are available only on the compute capability families or GPUs listed in the optimized configuration table in the section for each model. To override the automatically selected precision, refer to Precision Override.
For details about how the runtime selects an inference pipeline at startup, refer to Automatic Pipeline Selection.
Supported Hardware for each Model#
Note
Currently, GPU clusters with GPUs in Multi-instance GPU mode (MIG) are not supported.
Nemotron 3 Embed 1B (nemotron-3-embed-1b)#
Supported GPUs#
GPU |
Precision |
|---|---|
NVIDIA RTX PRO 6000 Blackwell Server Edition |
NVFP4 |
NVIDIA GB200 |
NVFP4 |
NVIDIA A100 SXM4 80GB |
BF16 |
NVIDIA H100 80GB HBM3 |
BF16 |
NVIDIA L40S |
BF16 |
NVIDIA A10G |
BF16 |
NVIDIA GB10 (DGX Spark) |
BF16 |
Note
NVIDIA DGX Spark uses a unified memory architecture in which the CPU and GPU share system memory, rather than dedicated GPU framebuffer memory. For NVIDIA-GB10, conventional VRAM total, used, and free measurements are therefore not available or directly comparable with measurements from discrete GPUs. This does not affect NIM support. For details, refer to Guidance for reporting memory resources with unified memory architecture.
Non-optimized configuration#
Fallback behavior on GPUs that are not listed has not been verified for this model.
Note
nvidia/nemotron-3-embed-1b is a text-only embedding model. It supports query and passage inputs, returns native 2048-dimensional embeddings when embedding_type is float, and is validated with a maximum sequence length of 4096 tokens. To use the native embedding dimension, omit the dimensions request parameter or set it to 2048.
NVFP4 Batch Token Budget#
NIM_PIPELINE_MAX_BATCH_TOKENS is an optional post-tokenization token budget for each Nemotron 3 embedding GPU dispatch. Set it to 0, or leave it unset, to disable token-budget chunking. This setting does not change NIM_PIPELINE_MAX_SEQ_LEN or API response semantics. The runtime splits CPU-tokenized batches only on request boundaries when possible. If a single request exceeds the configured token budget, it can still run alone.
Use the GPU-specific value from the validated release manifest. Do not copy values between GPUs without validation.
GPU / profile |
Pipeline |
Recommended value |
Notes |
|---|---|---|---|
NVIDIA RTX PRO 6000 Blackwell Server Edition, SM120 |
|
|
Dedicated RTX PRO release manifest value. Runtime profile uses |
NVIDIA GB200, SM100 |
|
|
Dedicated GB200 release manifest value. Do not reuse the NVIDIA B200 |
Llama Nemotron Embed 300m v2 (llama-nemotron-embed-300m-v2)#
Optimized configuration#
Precision |
|
|---|---|
12.0 |
FP16 and FP8 |
10.0 |
FP16 and FP8 |
9.0 |
FP16 and FP8 |
8.9 |
FP16 and FP8 |
8.6 |
FP16 |
8.0 |
FP16 |
Non-optimized configuration#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
Max Tokens |
|---|---|---|---|---|
Any single NVIDIA GPU that has sufficient memory, or multiple homogenous NVIDIA GPUs that have sufficient memory in total. |
Min: 2.4 GiB, Max: 25.2 GiB |
FP16 |
7.49 GiB |
4096 |
Warning
The maximum token length of the non-optimized configuration is smaller (4096) than the other profiles (8192).
Llama Nemotron Embed Vision Language 1B (llama-nemotron-embed-vl-1b-v2)#
Supported GPUs#
GPU |
Precision |
|---|---|
NVIDIA GB10 (DGX Spark) |
FP8 and FP16 |
NVIDIA RTX PRO 6000 Blackwell Server Edition |
FP8 and FP16 |
NVIDIA B200 |
FP8 and FP16 |
NVIDIA GB200 |
FP8 and FP16 |
NVIDIA H200 |
FP8 and FP16 |
NVIDIA A100 SXM4 80GB |
FP16 |
NVIDIA H100 NVL |
FP8 and FP16 |
NVIDIA H100 80GB HBM3 |
FP8 and FP16 |
NVIDIA L4 |
FP16 |
NVIDIA L40S |
FP8 and FP16 |
NVIDIA A10G |
FP16 |
Note
NVIDIA DGX Spark uses a unified memory architecture in which the CPU and GPU share system memory, rather than dedicated GPU framebuffer memory. For NVIDIA-GB10, conventional VRAM total, used, and free measurements are therefore not available or directly comparable with measurements from discrete GPUs. This does not affect NIM support. For details, refer to Guidance for reporting memory resources with unified memory architecture.
Non-optimized configuration#
Fallback behavior on GPUs that are not listed has not been verified for this model.
Note
The default VLM profile uses a maximum sequence length of 2048 tokens. Image inputs are supported only as document or passage inputs.
bge-large-zh-v1.5#
Optimized configuration#
GPU |
GPU Memory (GB) |
Precision |
|---|---|---|
H20 |
96 |
FP16 |
L20 |
48 |
FP16 |
Non-optimized configuration#
The GPU Memory and Disk Space values in the following table are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
|---|---|---|---|
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory |
10 |
FP16 |
8.1 |
bge-m3#
Optimized configuration#
GPU |
GPU Memory (GB) |
Precision |
|---|---|---|
A100 SXM4 |
80 |
FP16 |
H100 HBM3 |
80 |
FP16 |
L40S |
48 |
FP16 |
A10G |
24 |
FP16 |
L20 |
48 |
FP16 |
H20 |
96 |
FP16 |
Non-optimized configuration#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
|---|---|---|---|
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory |
33 |
FP16 |
8.8 |
Llama Nemotron Embed 1B v2#
Optimized configuration#
Precision |
|
|---|---|
12.0 |
FP16 and FP8 |
10.0 |
FP16 and FP8 |
9.0 |
FP16 and FP8 |
8.9 |
FP16 and FP8 |
8.6 |
FP16 |
8.0 |
FP16 |
Non-optimized configuration#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
Max Tokens |
|---|---|---|---|---|
Any single NVIDIA GPU that has sufficient memory, or multiple homogenous NVIDIA GPUs that have sufficient memory in total. |
3.6 |
FP16 |
9 |
4096 |
If you run this model on RTX 40xx or later, you need a minimum of 8GB of VRAM.
Warning
The maximum token length of the non-optimized configuration is smaller (4096) than the other profiles (8192).
NV-EmbedQA-E5-v5#
Optimized configuration#
Precision |
|
|---|---|
12.0 |
FP16 |
10.0 |
FP16 |
9.0 |
FP16 |
8.9 |
FP16 |
8.6 |
FP16 |
8.0 |
FP16 |
Non-optimized configuration#
The GPU Memory and Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
|---|---|---|---|
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory |
2 |
FP16 |
8.5 |