> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# Embedding Models

## Introduction

Text embedding models transform text into dense vector representations that power semantic search, dense retrieval, retrieval-augmented generation (RAG), and classification tasks. NeMo AutoModel includes custom bidirectional backbones and configures supported Hugging Face `AutoModel` backbones with non-causal attention for retrieval training.

For cross-encoder pairwise scoring, see [Reranking Models](/model-coverage/reranking-models/overview).

Embedding models use bi-encoders to produce dense representations for queries and documents independently. They are the standard path for embedding generation and first-stage dense retrieval.

### Optimized Backbones (Bidirectional Attention)

| Owner      | Model                                                                                | Architecture                              | Auto Class                                                                                                                                                         | Tasks                      |
| ---------- | ------------------------------------------------------------------------------------ | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------- |
| NVIDIA     | [Llama (Bidirectional)](/model-coverage/embedding-models/nvidia/llama-bidirectional) | `LlamaBidirectionalModel`                 | [`NeMoAutoModelBiEncoder`](https://github.com/NVIDIA-NeMo/Automodel/blob/8dc00dcb4a35c2413c52c6e7eb7ac8f1c24836aa/nemo_automodel/_transformers/auto_model.py#L991) | Embedding, Dense Retrieval |
| NVIDIA     | [Llama Nemotron VL](/model-coverage/embedding-models/nvidia/llama-bidirectional)     | `LlamaNemotronVLModel`                    | [`NeMoAutoModelBiEncoder`](https://github.com/NVIDIA-NeMo/Automodel/blob/8dc00dcb4a35c2413c52c6e7eb7ac8f1c24836aa/nemo_automodel/_transformers/auto_model.py#L991) | Embedding, Dense Retrieval |
| Mistral AI | [Ministral3](/model-coverage/embedding-models/mistralai/ministral3-bidirectional)    | `Ministral3Model` with `is_causal: false` | [`NeMoAutoModelBiEncoder`](https://github.com/NVIDIA-NeMo/Automodel/blob/8dc00dcb4a35c2413c52c6e7eb7ac8f1c24836aa/nemo_automodel/_transformers/auto_model.py#L991) | Embedding, Dense Retrieval |

The stock `ministral3` path is preferred. The `ministral3_bidirec` model type remains available for compatibility with legacy checkpoints and resolves to the custom `Ministral3BidirectionalModel`.

### Hugging Face Auto Backbones

Any Hugging Face model that can be loaded with `AutoModel` can be used as an embedding backbone. The fallback path sets `config.is_causal: false`; verify that an unregistered architecture honors this flag before using it for retrieval training.

## Example Recipes

| Recipe                                                                                                                                                                   | Description                                              |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------- |
| [llama3\_2\_1b.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/retrieval/bi_encoder/llama3_2_1b.yaml)                                                  | Bi-encoder — Llama 3.2 1B embedding model                |
| [llama\_embed\_nemotron\_8b.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/retrieval/bi_encoder/llama_embed_nemotron_8b/llama_embed_nemotron_8b.yaml) | Bi-encoder — Llama-Embed-Nemotron-8B reproduction recipe |
| [ministral3\_3b\_instruct.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/retrieval/bi_encoder/ministral3_3b_instruct.yaml)                            | Bi-encoder — Ministral3-3B recipe                        |

## Supported Models

This table combines recipe-backed checkpoints with documented model families. Dates show when the current checkpoint first appeared in a recipe, or when a documentation-only family page was added. See the [combined model support log](/model-coverage/release-log) for recipe-backed checkpoints of every model type.

| Date       | Model                                                                                                           | Architectures                                                                                  | Recipe                                                                                                                                          |
| :--------- | :-------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-08-07 | [Ministral-3-3B-Instruct-2512-BF16](https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512-BF16)         |                                                                                                | [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/retrieval/bi_encoder/ministral3_3b_instruct.yaml)                          |
| 2026-06-27 | [Llama-nemotron-embed-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-embed-vl-1b-v2)                    |                                                                                                | [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/retrieval/bi_encoder/nemotron_vl_1b/nemotron_vl_1b_example.yaml)           |
| 2026-05-21 | [Ministral3 for Embedding](/model-coverage/embedding-models/mistralai/ministral3-bidirectional) (documentation) | `Ministral3Model` with `is_causal: false` (preferred); `Ministral3BidirectionalModel` (legacy) |                                                                                                                                                 |
| 2025-12-16 | [Llama-3.1-8B](/model-coverage/embedding-models/nvidia/llama-bidirectional)                                     | `LlamaBidirectionalModel`                                                                      | [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/retrieval/bi_encoder/llama_embed_nemotron_8b/llama_embed_nemotron_8b.yaml) |
| 2025-11-18 | [Llama-3.2-1B](/model-coverage/embedding-models/nvidia/llama-bidirectional)                                     | `LlamaBidirectionalModel`                                                                      | [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/retrieval/bi_encoder/llama3_2_1b.yaml)                                     |

## Supported Workflows

* **Fine-tuning (Bi-Encoder):** Contrastive learning on query-document pairs to produce embedding models
* **LoRA/PEFT:** Parameter-efficient fine-tuning for embedding backbones
* **ONNX Export:** Export trained embedding models for deployment (case by case, model dependent)

## Dataset

Retrieval fine-tuning requires query-document pairs: each example is a query paired with one positive document and one or more negative documents. Both inline JSONL and corpus ID-based JSON formats are supported. See the [Retrieval Dataset](/datasets/retrieval-dataset) guide.