> This page is for version 0.7.0 · 26.10.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# Embedding Models

> Overview of NeMo AutoModel embedding models, including bidirectional Llama and Ministral3 backbones, Hugging Face AutoModel fallback, and bi-encoder recipes.

## Introduction

Text embedding models transform text into dense vector representations that power semantic search, dense retrieval, retrieval-augmented generation (RAG), and classification tasks. NeMo AutoModel includes custom bidirectional backbones and configures supported Hugging Face `AutoModel` backbones with non-causal attention for retrieval training.

For cross-encoder pairwise scoring, see [Reranking Models](/nemo/automodel/v0.7/model-coverage/reranking-models/overview).

Embedding models use bi-encoders to produce dense representations for queries and documents independently. They are the standard path for embedding generation and first-stage dense retrieval.

### Optimized Backbones

| Owner      | Model                                                                                    | Architecture                                  | Auto Class                                                                                                                                                         | Tasks                      |
| ---------- | ---------------------------------------------------------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------- |
| NVIDIA     | [Llama (Bidirectional)](/nemo/automodel/v0.7/model-coverage/embedding-models/nvidia/llama-embed-nemotron-8b) | `LlamaBidirectionalModel`                     | [`NeMoAutoModelBiEncoder`](https://github.com/NVIDIA-NeMo/Automodel/blob/8dc00dcb4a35c2413c52c6e7eb7ac8f1c24836aa/nemo_automodel/_transformers/auto_model.py#L991) | Embedding, Dense Retrieval |
| NVIDIA     | [Llama Nemotron VL](/nemo/automodel/v0.7/model-coverage/embedding-models/nvidia/llama-embed-nemotron-8b)     | `LlamaNemotronVLModel`                        | [`NeMoAutoModelBiEncoder`](https://github.com/NVIDIA-NeMo/Automodel/blob/8dc00dcb4a35c2413c52c6e7eb7ac8f1c24836aa/nemo_automodel/_transformers/auto_model.py#L991) | Embedding, Dense Retrieval |
| Mistral AI | [Ministral3](/nemo/automodel/v0.7/model-coverage/embedding-models/mistralai/Ministral-3-3B-Base-2512)        | `Ministral3Model` with configurable attention | [`NeMoAutoModelBiEncoder`](https://github.com/NVIDIA-NeMo/Automodel/blob/8dc00dcb4a35c2413c52c6e7eb7ac8f1c24836aa/nemo_automodel/_transformers/auto_model.py#L991) | Embedding, Dense Retrieval |

The stock `ministral3` path is preferred. The `ministral3_bidirec` model type remains available for compatibility with legacy checkpoints and resolves to the custom `Ministral3BidirectionalModel`.

### Hugging Face Auto Backbones

Unregistered text architectures load through Hugging Face `AutoModel`. The fallback path applies `model.is_causal`, restores a saved text-config value when the option is omitted, and otherwise defaults to bidirectional attention. Verify that an unregistered architecture honors the selected policy before using it for retrieval training.

Composite backbones must expose matching text configuration and decoder objects through `get_text_config(decoder=True)` and `get_decoder()`. Loading fails if these methods do not identify the same text tower. The attention policy changes only that tower, leaving vision attention unchanged.

## Example Recipes

| Recipe                                                                                                                                                                  | Description                                                                                                                                                                                               |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Nemotron end-to-end text bi-encoder recipe](https://github.com/NVIDIA-NeMo/Nemotron/tree/main/docs/nemotron/embed)                                                     | Adapt an embedding model to your domain-specific data with a complete multi-stage pipeline, from synthetic data generation and data preparation to model fine-tuning, evaluation, export, and deployment. |
| [llama3_2_1b.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/r0.7.0/examples/retrieval/bi_encoder/llama3_2_1b.yaml)                                                 | Llama-3.2-1B based text embedding model training recipe.                                                                                                                                                  |
| [llama_embed_nemotron_8b.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/r0.7.0/examples/retrieval/bi_encoder/llama_embed_nemotron_8b/llama_embed_nemotron_8b.yaml) | [Llama-Embed-Nemotron-8B](https://huggingface.co/nvidia/llama-embed-nemotron-8b) text embedding model reproduction recipe.                                                                                |
| [ministral3_3b_instruct.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/r0.7.0/examples/retrieval/bi_encoder/ministral3_3b_instruct.yaml)                           | Ministral3-3B based text embedding model training recipe.                                                                                                                                                 |

## Supported Models

This table combines recipe-backed checkpoints with documented model families. Dates show when the current checkpoint first appeared in a recipe, or when a documentation-only family page was added. See the [combined model support log](/nemo/automodel/v0.7/model-coverage/release-log) for recipe-backed checkpoints of every model type.

## Supported Workflows

* **Fine-tuning (Bi-Encoder):** Contrastive learning on query-document pairs to produce embedding models
* **LoRA/PEFT:** Parameter-efficient fine-tuning for embedding backbones
* **ONNX Export:** Export trained embedding models for deployment (case-by-case, model-dependent)

## Dataset

Retrieval fine-tuning requires query-document pairs: each example is a query paired with one positive document and one or more negative documents. Both inline JSONL and corpus ID-based JSON formats are supported. See the [Retrieval Dataset](/nemo/automodel/v0.7/datasets/retrieval-dataset) guide.

## Train Embedding Models

For a complete walkthrough of training configuration, model-specific settings, and launch commands, see the [Embedding and Reranking Fine-Tuning Guide](/nemo/automodel/v0.7/recipes-e2e-examples/retrieval-finetune).