Ministral3 (Bidirectional) for Embedding
Ministral3 (Bidirectional) for Embedding
NeMo AutoModel provides a bidirectional variant of Mistral AI’s Ministral3 for embedding and dense retrieval tasks. Unlike the standard causal (left-to-right) Ministral3 used for text generation, this variant uses bidirectional attention, so each token can attend to both past and future tokens in the sequence, producing richer representations for semantic similarity and dense retrieval.
The bidirectional encoder can be loaded directly from text-only checkpoints (e.g. mistralai/Ministral-3B-Instruct) and also automatically extracts the language model from Ministral3 VLM checkpoints (e.g. mistralai/Ministral-3-3B-Base-2512 or mistralai/Ministral-3-3B-Instruct-2512).
Available Models
Any Ministral3 checkpoint can be loaded as a bidirectional backbone. The following configurations are tested:
- Ministral-3-3B-Base-2512 — VLM checkpoint, language model is extracted automatically
- Ministral-3-3B-Instruct-2512 — VLM checkpoint, language model is extracted automatically
Embedding Models
The bidirectional bi-encoder path is used for embedding generation and dense retrieval.
Pooling Strategies
The bi-encoder supports multiple pooling strategies to aggregate token representations into a single embedding vector:
Example HF Models
Try with NeMo AutoModel
1. Install NeMo AutoModel. Refer to the (Installation Guide) for information:
2. Clone the repo to get the example recipes:
3. Run the recipe from inside the repo (point any Llama bi-encoder recipe at a Ministral3 checkpoint, or write a recipe targeting mistralai/Ministral-3-3B-Base-2512):
See the Installation Guide.