> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# Embedding Models

## Introduction

Text embedding models transform text into dense vector representations that power semantic search, dense retrieval, retrieval-augmented generation (RAG), and classification tasks. NeMo AutoModel includes a training recipe for converting Llama decoder-only models into encoder architectures with bidirectional attention, and falls back to Hugging Face AutoModel for other encoder backbones.

For cross-encoder pairwise scoring, see [Reranking Models](/model-coverage/reranking-models/overview).

Embedding models use bi-encoders to produce dense representations for queries and documents independently. They are the standard path for embedding generation and first-stage dense retrieval.

### Optimized Backbones (Bidirectional Attention)

| Owner      | Model                                                                                   | Architecture                   | Auto Class                                                                                                                                                         | Tasks                      |
| ---------- | --------------------------------------------------------------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------- |
| NVIDIA     | [Llama (Bidirectional)](/model-coverage/embedding-models/llama-bidirectional)           | `LlamaBidirectionalModel`      | [`NeMoAutoModelBiEncoder`](https://github.com/NVIDIA-NeMo/Automodel/blob/8dc00dcb4a35c2413c52c6e7eb7ac8f1c24836aa/nemo_automodel/_transformers/auto_model.py#L991) | Embedding, Dense Retrieval |
| Mistral AI | [Ministral3 (Bidirectional)](/model-coverage/embedding-models/ministral3-bidirectional) | `Ministral3BidirectionalModel` | [`NeMoAutoModelBiEncoder`](https://github.com/NVIDIA-NeMo/Automodel/blob/8dc00dcb4a35c2413c52c6e7eb7ac8f1c24836aa/nemo_automodel/_transformers/auto_model.py#L991) | Embedding, Dense Retrieval |

### Hugging Face Auto Backbones

Any Hugging Face model that can be loaded with `AutoModel` can be used as an embedding backbone. This fallback path uses the model's native attention; no bidirectional conversion is applied.

## Example Recipes

| Recipe                                                                                                                                                                   | Description                                              |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------- |
| [llama3\_2\_1b.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/retrieval/bi_encoder/llama3_2_1b.yaml)                                                  | Bi-encoder — Llama 3.2 1B embedding model                |
| [llama\_embed\_nemotron\_8b.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/retrieval/bi_encoder/llama_embed_nemotron_8b/llama_embed_nemotron_8b.yaml) | Bi-encoder — Llama-Embed-Nemotron-8B reproduction recipe |
| [ministral3\_3b\_instruct.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/retrieval/bi_encoder/ministral3_3b_instruct.yaml)                            | Bi-encoder — Ministral3-3B recipe                        |

## Supported Workflows

* **Fine-tuning (Bi-Encoder):** Contrastive learning on query-document pairs to produce embedding models
* **LoRA/PEFT:** Parameter-efficient fine-tuning for embedding backbones
* **ONNX Export:** Export trained embedding models for deployment (case by case, model dependent)

## Dataset

Retrieval fine-tuning requires query-document pairs: each example is a query paired with one positive document and one or more negative documents. Both inline JSONL and corpus ID-based JSON formats are supported. See the [Retrieval Dataset](/datasets/retrieval-dataset) guide.