Support Matrix for NVIDIA NeMo Retriever Embedding NIM#
This documentation describes the models, software, and hardware that NVIDIA NeMo Retriever Embedding NIM supports.
Models#
Embedding NIM supports the following models.
Model ID |
Model Card |
Supported Embedding Types |
Max Tokens |
Parameters |
Total Parameters |
Embedding |
Dynamic Embeddings |
|---|---|---|---|---|---|---|---|
nvidia/Nemotron-3.5-Embed-1B-BF16 |
|
32768 |
1286 |
1555 |
20482 |
yes |
|
nvidia/Nemotron-3.5-Embed-1B-NVFP4 |
|
32768 |
1286 |
1555 |
20482 |
yes |
|
nvidia/Nemotron-3.5-Embed-8B-BF16 |
|
32768 |
8000 |
8000 |
20482 |
yes |
|
nvidia/nemotron-3-embed-1b |
|
4096 |
872 |
1141 |
20482 |
no |
|
nvidia/llama-nemotron-embed-vl-1b-v2 |
|
2048 |
1414 |
1678 |
2048 |
yes |
|
nvidia/llama-nemotron-embed-1b-v2 |
|
8192 |
973 |
1236 |
2048 |
yes |
|
nvidia/llama-nemotron-embed-300m-v2 |
|
8192 |
307 |
569 |
2048 |
yes |
|
nvidia/nv-embedqa-e5-v5 |
|
512 |
303 |
335 |
1024 |
no |
|
baai/bge-m3 |
|
8192 |
303 |
568 |
1024 |
no |
|
baai/bge-large-zh-v1.5 |
|
512 |
303 |
325 |
1024 |
no |
Note
For Nemotron 3.5 Embed 1B, release 2.4 supports three embedding_type values: float, int8, and uint8. The default is float. The values binary and ubinary are not supported in release 2.4.
This output support applies to both BF16 and NVFP4 model profiles. Model-weight and inference precision are separate from embedding output quantization.
The independent encoding_format parameter supports float and base64. Base64 encodes the selected output type; it does not enable an unsupported embedding type.
Note
1 The “Parameters (excl. embeddings)” column shows the count of parameters that directly impact inference performance and computational cost. Embedding layer parameters are excluded because they primarily affect model size rather than inference speed. For example, models with different vocabulary sizes may have different total parameter counts but the same inference-relevant parameter count.
Note
2 The dimensions request parameter is model-specific. Models that support dynamic embeddings can return reduced embedding sizes. Nemotron 3.5 Embed 1B accepts any integer dimensions value from 128 through 2048. The model nvidia/nemotron-3-embed-1b does not support reduced dimensions. Its native embedding dimension is 2048, and the only supported dimensions value is 2048. You can also omit the parameter.
Software#
NVIDIA Driver: Release 2.4 is built with CUDA 13.0.2. Use an NVIDIA driver that supports CUDA 13.0. For driver requirements, refer to CUDA Compatibility. Ensure that the latest compatible NVIDIA driver is installed on your system before launching NIM containers. If you experience issues starting the containers, verify that your driver is up-to-date.
NVIDIA Container Toolkit: Your Docker environment must support NVIDIA GPUs. For more information, refer to NVIDIA Container Toolkit.
CPUs#
Embedding NIM supports the following host CPU configurations:
x86 processor with at least 8 cores. For a list of supported systems, refer to NVIDIA Certified Systems Catalog.
NVIDIA DGX Spark with the integrated Arm CPU (
aarch64/Arm64). DGX Spark support starts with version 2.3.
GPUs#
Embedding NIM supports the following GPUs. For details, refer to the optimized configuration table for each model later in this documentation.
NVIDIA H20
NVIDIA L20
Multi-Instance GPU (MIG)#
Nemotron 3.5 Embed 1B supports Multi-Instance GPU (MIG) in release 2.4 with the default NIM configuration on the following profiles.
GPU |
Nominal memory allocation per GPU instance |
GPU instance profile |
|---|---|---|
NVIDIA GB200 |
24 GB |
|
NVIDIA RTX PRO 6000 Blackwell Server Edition |
24 GB |
|
Use one compute instance that occupies the entire GPU instance, without subdividing it into multiple compute instances.
Profile names can differ between GPUs and driver versions. Before you create the GPU instance, run nvidia-smi mig -lgip on the target GPU to check the profiles that it reports.
The nominal 24 GB allocation does not specify driver-reported usable memory and is not a minimum memory requirement.
Note
MIG is validated for the default-selected precision and runtime settings. Other models, GPUs, MIG profiles, compute-instance subdivisions, and non-default precision or runtime settings are outside this validated scope.
Optimized and Non-optimized Configurations#
Starting in version 2.0, optimized configurations use runtime CUDA kernels and just-in-time compilation. At startup, the NIM selects a kernel feature set for the detected GPU architecture. Depending on the selected feature set, the NIM might compile kernels, load precompiled kernels optimized for that architecture, or use both.
The optimized configuration table in the section for each model lists the compute capability families or GPUs that have optimized kernel support for the listed precision. These configurations are tuned and validated for the release.
Non-optimized configurations use a fallback kernel feature set intended for broad compatibility, such as FP16 architecture-agnostic kernels. Fallback configurations can run on GPUs with sufficient memory, but they might not support every optimized feature or deliver the same performance as the optimized configurations.
Compute Capability and Automatic Kernel Selection#
Starting in version 2.0, the NIM detects the GPU compute capability at startup and selects the supported kernel feature set for that compute capability family.
The selected feature set determines which CUDA kernels, attention implementation, precompiled kernel artifacts, and default precision are used. If an optimized feature set is not available for the detected GPU, the NIM uses the compatible fallback feature set.
To request a precision explicitly, set NIM_ENGINE_PRECISION to a value supported by the model and GPU. FP8 and NVFP4 are available only on the compute capability families or GPUs listed in the optimized configuration table in the section for each model. To override the automatically selected precision, refer to Precision Override.
For details about how the runtime selects an inference pipeline at startup, refer to Automatic Pipeline Selection.
Nemotron 3.5 Embed 1B#
Sequence Length Limits#
For the effective token capacity of text GPU dispatches, see Text Dispatch Token Budget. This budget does not reduce the per-row sequence length limit.
The finalized BF16 and NVFP4 checkpoints each declare a maximum input length of 32,768 tokens. This model limit is independent of the deployment default and the runtime fallback when NIM_PIPELINE_MAX_SEQ_LEN is unset.
The nemotron-embed deployment sets NIM_PIPELINE_MAX_SEQ_LEN=32768. Without that setting, the embedding server falls back to 512. Operators can select a lower sequence limit; an override does not extend the checkpoint’s supported maximum. See Pipeline Environment Variables for an override example.
The token limit does not establish support for other GPU or precision combinations; use the configurations listed below.
Optimized Configurations#
The following table contains the optimized configurations for Nemotron 3.5 Embed 1B.
BF16 in nvidia/Nemotron-3.5-Embed-1B-BF16 identifies the checkpoint, not the inference precision setting.
The container runs this dense checkpoint with FP16.
To select it explicitly, set NIM_ENGINE_MODEL_NAME=nvidia/Nemotron-3.5-Embed-1B-BF16 and NIM_ENGINE_PRECISION=fp16.
Do not set NIM_ENGINE_PRECISION=bf16: explicit BF16 execution is not implemented.
For the NVFP4 model on a supported GPU, set NIM_ENGINE_MODEL_NAME=nvidia/Nemotron-3.5-Embed-1B-NVFP4 and NIM_ENGINE_PRECISION=nvfp4.
GPU |
Precision |
|---|---|
NVIDIA GB200 |
NVFP4 and FP16 |
NVIDIA B300 |
NVFP4 and FP16 |
NVIDIA RTX PRO 6000 Blackwell Server Edition |
NVFP4 and FP16 |
NVIDIA H100 80GB HBM3 |
FP16 |
NVIDIA H100 NVL |
FP16 |
NVIDIA A100 SXM4 80GB |
FP16 |
NVIDIA L40S |
FP16 |
NVIDIA A10G |
FP16 |
NVIDIA GB10 (DGX Spark) |
FP16 |
Nemotron 3 Embed 1B#
The following table contains the optimized configurations for Nemotron 3 Embed 1B.
GPU |
Precision |
|---|---|
NVIDIA RTX PRO 6000 Blackwell Server Edition |
NVFP4 and FP16 |
NVIDIA GB200 |
NVFP4 and FP16 |
NVIDIA A100 SXM4 80GB |
FP16 |
NVIDIA H100 80GB HBM3 |
FP16 |
NVIDIA H100 NVL |
FP16 |
NVIDIA L40S |
FP16 |
NVIDIA A10G |
FP16 |
NVIDIA GB10 (DGX Spark) |
FP16 |
Note
NVIDIA DGX Spark uses a unified memory architecture in which the CPU and GPU share system memory, rather than dedicated GPU framebuffer memory. For NVIDIA-GB10, conventional VRAM total, used, and free measurements are therefore not available or directly comparable with measurements from discrete GPUs. This does not affect NIM support. For details, refer to Guidance for reporting memory resources with unified memory architecture.
Fallback behavior on GPUs that are not listed has not been verified for this model.
Note
nvidia/nemotron-3-embed-1b is a text-only embedding model. It supports query and passage inputs, returns native 2048-dimensional embeddings when embedding_type is float, and is validated with a maximum sequence length of 4096 tokens. To use the native embedding dimension, omit the dimensions request parameter or set it to 2048.
NVFP4 Batch Token Budget#
NIM_PIPELINE_MAX_BATCH_TOKENS is an optional post-tokenization token budget for each Nemotron 3 embedding GPU dispatch. Set it to 0, or leave it unset, to disable token-budget chunking. This setting does not change NIM_PIPELINE_MAX_SEQ_LEN or API response semantics. The runtime splits CPU-tokenized batches only on request boundaries when possible. If a single request exceeds the configured token budget, it can still run alone.
Use the GPU-specific value from the validated release manifest. Do not copy values between GPUs without validation.
GPU / profile |
Pipeline |
Recommended value |
Notes |
|---|---|---|---|
NVIDIA RTX PRO 6000 Blackwell Server Edition, SM120 |
|
|
Dedicated RTX PRO release manifest value. Runtime profile uses |
NVIDIA GB200, SM100 |
|
|
Dedicated GB200 release manifest value. Do not reuse the NVIDIA B200 |
Llama Nemotron Embed VL 1B v2#
The following table contains the optimized configurations for Llama Nemotron Embed VL 1B v2.
GPU |
Precision |
|---|---|
NVIDIA GB10 (DGX Spark) |
FP8 and FP16 |
NVIDIA RTX PRO 6000 Blackwell Server Edition |
FP8 and FP16 |
NVIDIA B200 |
FP8 and FP16 |
NVIDIA GB200 |
FP8 and FP16 |
NVIDIA H200 |
FP8 and FP16 |
NVIDIA A100 SXM4 80GB |
FP16 |
NVIDIA H100 NVL |
FP8 and FP16 |
NVIDIA H100 80GB HBM3 |
FP8 and FP16 |
NVIDIA L4 |
FP16 |
NVIDIA L40S |
FP8 and FP16 |
NVIDIA A10G |
FP16 |
Note
NVIDIA DGX Spark uses a unified memory architecture in which the CPU and GPU share system memory, rather than dedicated GPU framebuffer memory. For NVIDIA-GB10, conventional VRAM total, used, and free measurements are therefore not available or directly comparable with measurements from discrete GPUs. This does not affect NIM support. For details, refer to Guidance for reporting memory resources with unified memory architecture.
Fallback behavior on GPUs that are not listed has not been verified for this model.
Note
The default VLM profile uses a maximum sequence length of 2048 tokens. Image inputs are supported only as document or passage inputs.
Llama Nemotron Embed 1B v2#
The following table contains the optimized configurations for Llama Nemotron Embed 1B v2.
Precision |
|
|---|---|
12.0 |
FP16 and FP8 |
10.0 |
FP16 and FP8 |
9.0 |
FP16 and FP8 |
8.9 |
FP16 and FP8 |
8.6 |
FP16 |
8.0 |
FP16 |
The following table contains the non-optimized configurations for Llama Nemotron Embed 1B v2. The GPU Memory values are in GiB and the Disk Space values are in GB; Disk Space is for both the container and the model. If you run this model on RTX 40xx or later, you need a minimum of 8GB of VRAM.
GPUs |
GPU Memory |
Precision |
Disk Space |
Max Tokens |
|---|---|---|---|---|
Any single NVIDIA GPU that has sufficient memory, or multiple homogenous NVIDIA GPUs that have sufficient memory in total. |
3.6 |
FP16 |
9 |
4096 |
Warning
The maximum token length of the non-optimized configuration is smaller (4096) than the other profiles (8192).
Llama Nemotron Embed 300m v2#
The following table contains the optimized configurations for Llama Nemotron Embed 300m v2.
Precision |
|
|---|---|
12.0 |
FP16 and FP8 |
10.0 |
FP16 and FP8 |
9.0 |
FP16 and FP8 |
8.9 |
FP16 and FP8 |
8.6 |
FP16 |
8.0 |
FP16 |
The following table contains the non-optimized configurations for Llama Nemotron Embed 300m v2. The GPU Memory values are in GiB and the Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
Max Tokens |
|---|---|---|---|---|
Any single NVIDIA GPU that has sufficient memory, or multiple homogenous NVIDIA GPUs that have sufficient memory in total. |
Min: 2.4, Max: 25.2 |
FP16 |
8.04 |
4096 |
Warning
The maximum token length of the non-optimized configuration is smaller (4096) than the other profiles (8192).
NVIDIA QA E5 Embedding#
The following table contains the optimized configurations for NVIDIA QA E5 Embedding.
Precision |
|
|---|---|
12.0 |
FP16 |
10.0 |
FP16 |
9.0 |
FP16 |
8.9 |
FP16 |
8.6 |
FP16 |
8.0 |
FP16 |
The following table contains the non-optimized configurations for NVIDIA QA E5 Embedding. The GPU Memory values are in GiB and the Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
|---|---|---|---|
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory |
2 |
FP16 |
8.5 |
BAAI bge-large-zh-v1.5#
The following table contains the optimized configurations for BAAI bge-large-zh-v1.5.
GPU |
GPU Memory (GiB) |
Precision |
|---|---|---|
H20 |
96 |
FP16 |
L20 |
48 |
FP16 |
The following table contains the non-optimized configurations for BAAI bge-large-zh-v1.5. The GPU Memory values in the following table are in GiB and the Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
|---|---|---|---|
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory |
10 |
FP16 |
8.1 |
BAAI bge-m3#
The following table contains the optimized configurations for BAAI bge-m3.
GPU |
GPU Memory (GiB) |
Precision |
|---|---|---|
A100 SXM4 |
80 |
FP16 |
H100 HBM3 |
80 |
FP16 |
L40S |
48 |
FP16 |
A10G |
24 |
FP16 |
L20 |
48 |
FP16 |
H20 |
96 |
FP16 |
The following table contains the non-optimized configurations for BAAI bge-m3. The GPU Memory values are in GiB and the Disk Space values are in GB; Disk Space is for both the container and the model.
GPUs |
GPU Memory |
Precision |
Disk Space |
|---|---|---|---|
Any NVIDIA GPU with sufficient GPU memory or on multiple, homogenous NVIDIA GPUs with sufficient aggregate memory |
33 |
FP16 |
8.8 |