Model CoverageEmbedding Models

Ministral3 for Embedding

View as Markdown

NeMo AutoModel loads Mistral AI’s Ministral3 with the stock Hugging Face Ministral3Model and sets is_causal: false for embedding and dense retrieval tasks. Each token can therefore attend to both past and future tokens without requiring a custom model implementation.

The encoder can be loaded directly from text-only checkpoints (e.g. mistralai/Ministral-3B-Instruct) and also automatically extracts the language model from Ministral3 VLM checkpoints (e.g. mistralai/Ministral-3-3B-Base-2512 or mistralai/Ministral-3-3B-Instruct-2512). The custom Ministral3BidirectionalModel remains available through the legacy ministral3_bidirec model type for compatibility with existing checkpoints.

TasksEmbedding, Dense Retrieval
ArchitectureMinistral3Model with is_causal: false (preferred); Ministral3BidirectionalModel (legacy)
Parameters3B
HF Orgmistralai

Available Models

Any Ministral3 checkpoint can be loaded as a non-causal backbone. The following configurations are tested:

  • Ministral-3-3B-Base-2512 — VLM checkpoint, language model is extracted automatically
  • Ministral-3-3B-Instruct-2512 — VLM checkpoint, language model is extracted automatically

Embedding Models

The stock non-causal bi-encoder path is used for embedding generation and dense retrieval.

ArchitectureTaskAuto ClassDescription
Ministral3Model with is_causal: falseEmbeddingNeMoAutoModelBiEncoderPreferred stock Hugging Face path for bidirectional dense embeddings
Ministral3BidirectionalModelEmbeddingNeMoAutoModelBiEncoderLegacy compatibility path for custom retrieval checkpoints

For cross-encoder scoring, stock ministral3 checkpoints use Hugging Face Ministral3ForSequenceClassification. The legacy ministral3_bidirec type does not provide a cross-encoder scoring class.

Pooling Strategies

The bi-encoder supports multiple pooling strategies to aggregate token representations into a single embedding vector:

StrategyDescription
avgAverage of all token hidden states (default)
clsFirst token hidden state
lastLast non-padding token hidden state
weighted_avgWeighted average of token hidden states

Example HF Models

ModelHF ID
Ministral-3 3B Basemistralai/Ministral-3-3B-Base-2512
Ministral-3 3B Instructmistralai/Ministral-3-3B-Instruct-2512

Try with NeMo AutoModel

1. Clone and install from source (full instructions):

$git clone https://github.com/NVIDIA-NeMo/Automodel.git
$cd Automodel
$uv sync --locked --all-groups --all-extras

2. Run the Ministral3 recipe from inside the repo:

$uv run automodel examples/retrieval/bi_encoder/ministral3_3b_instruct.yaml --nproc-per-node 8

See the Installation Guide.

Hugging Face Model Cards