Embedding Models

Introduction

Text embedding models transform text into dense vector representations that power semantic search, dense retrieval, retrieval-augmented generation (RAG), and classification tasks. NeMo AutoModel includes custom bidirectional backbones and configures supported Hugging Face AutoModel backbones with non-causal attention for retrieval training.

For cross-encoder pairwise scoring, see Reranking Models.

Embedding models use bi-encoders to produce dense representations for queries and documents independently. They are the standard path for embedding generation and first-stage dense retrieval.

Optimized Backbones (Bidirectional Attention)

Owner	Model	Architecture	Auto Class	Tasks
NVIDIA	Llama (Bidirectional)	`LlamaBidirectionalModel`	`NeMoAutoModelBiEncoder`	Embedding, Dense Retrieval
NVIDIA	Llama Nemotron VL	`LlamaNemotronVLModel`	`NeMoAutoModelBiEncoder`	Embedding, Dense Retrieval
Mistral AI	Ministral3	`Ministral3Model` with `is_causal: false`	`NeMoAutoModelBiEncoder`	Embedding, Dense Retrieval

The stock ministral3 path is preferred. The ministral3_bidirec model type remains available for compatibility with legacy checkpoints and resolves to the custom Ministral3BidirectionalModel.

Hugging Face Auto Backbones

Any Hugging Face model that can be loaded with AutoModel can be used as an embedding backbone. The fallback path sets config.is_causal: false; verify that an unregistered architecture honors this flag before using it for retrieval training.

Example Recipes

Recipe	Description
llama3_2_1b.yaml	Bi-encoder — Llama 3.2 1B embedding model
llama_embed_nemotron_8b.yaml	Bi-encoder — Llama-Embed-Nemotron-8B reproduction recipe
ministral3_3b_instruct.yaml	Bi-encoder — Ministral3-3B recipe

Supported Models

This table combines recipe-backed checkpoints with documented model families. Dates show when the current checkpoint first appeared in a recipe, or when a documentation-only family page was added. See the combined model support log for recipe-backed checkpoints of every model type.

Date	Model	Architectures	Recipe
2026-08-07	Ministral-3-3B-Instruct-2512-BF16		recipe
2026-06-27	Llama-nemotron-embed-vl-1b-v2		recipe
2026-05-21	Ministral3 for Embedding (documentation)	`Ministral3Model` with `is_causal: false` (preferred); `Ministral3BidirectionalModel` (legacy)
2025-12-16	Llama-3.1-8B	`LlamaBidirectionalModel`	recipe
2025-11-18	Llama-3.2-1B	`LlamaBidirectionalModel`	recipe

Supported Workflows

Fine-tuning (Bi-Encoder): Contrastive learning on query-document pairs to produce embedding models
LoRA/PEFT: Parameter-efficient fine-tuning for embedding backbones
ONNX Export: Export trained embedding models for deployment (case by case, model dependent)

Dataset

Retrieval fine-tuning requires query-document pairs: each example is a query paired with one positive document and one or more negative documents. Both inline JSONL and corpus ID-based JSON formats are supported. See the Retrieval Dataset guide.