> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# Ministral3 for Embedding

NeMo AutoModel loads [Mistral AI's Ministral3](https://mistral.ai/news/ministraux/) with the stock Hugging Face `Ministral3Model` and sets `is_causal: false` for embedding and dense retrieval tasks. Each token can therefore attend to both past and future tokens without requiring a custom model implementation.

The encoder can be loaded directly from text-only checkpoints (e.g. `mistralai/Ministral-3B-Instruct`) and also automatically extracts the language model from Ministral3 VLM checkpoints (e.g. `mistralai/Ministral-3-3B-Base-2512` or `mistralai/Ministral-3-3B-Instruct-2512`). The custom `Ministral3BidirectionalModel` remains available through the legacy `ministral3_bidirec` model type for compatibility with existing checkpoints.

|                  |                                                                                                |
| ---------------- | ---------------------------------------------------------------------------------------------- |
| **Tasks**        | Embedding, Dense Retrieval                                                                     |
| **Architecture** | `Ministral3Model` with `is_causal: false` (preferred); `Ministral3BidirectionalModel` (legacy) |
| **Parameters**   | 3B                                                                                             |
| **HF Org**       | [mistralai](https://huggingface.co/mistralai)                                                  |

## Available Models

Any Ministral3 checkpoint can be loaded as a non-causal backbone. The following configurations are tested:

* **Ministral-3-3B-Base-2512** — VLM checkpoint, language model is extracted automatically
* **Ministral-3-3B-Instruct-2512** — VLM checkpoint, language model is extracted automatically

## Embedding Models

The stock non-causal bi-encoder path is used for embedding generation and dense retrieval.

| Architecture                              | Task      | Auto Class                                                                                                                                                         | Description                                                          |
| ----------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------- |
| `Ministral3Model` with `is_causal: false` | Embedding | [`NeMoAutoModelBiEncoder`](https://github.com/NVIDIA-NeMo/Automodel/blob/8dc00dcb4a35c2413c52c6e7eb7ac8f1c24836aa/nemo_automodel/_transformers/auto_model.py#L991) | Preferred stock Hugging Face path for bidirectional dense embeddings |
| `Ministral3BidirectionalModel`            | Embedding | [`NeMoAutoModelBiEncoder`](https://github.com/NVIDIA-NeMo/Automodel/blob/8dc00dcb4a35c2413c52c6e7eb7ac8f1c24836aa/nemo_automodel/_transformers/auto_model.py#L991) | Legacy compatibility path for custom retrieval checkpoints           |

For cross-encoder scoring, stock `ministral3` checkpoints use Hugging Face `Ministral3ForSequenceClassification`. The legacy `ministral3_bidirec` type does not provide a cross-encoder scoring class.

## Pooling Strategies

The bi-encoder supports multiple pooling strategies to aggregate token representations into a single embedding vector:

| Strategy       | Description                                  |
| -------------- | -------------------------------------------- |
| `avg`          | Average of all token hidden states (default) |
| `cls`          | First token hidden state                     |
| `last`         | Last non-padding token hidden state          |
| `weighted_avg` | Weighted average of token hidden states      |

## Example HF Models

| Model                   | HF ID                                                                                                     |
| ----------------------- | --------------------------------------------------------------------------------------------------------- |
| Ministral-3 3B Base     | [`mistralai/Ministral-3-3B-Base-2512`](https://huggingface.co/mistralai/Ministral-3-3B-Base-2512)         |
| Ministral-3 3B Instruct | [`mistralai/Ministral-3-3B-Instruct-2512`](https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512) |

## Try with NeMo AutoModel

**1. Clone and install from source** ([full instructions](/get-started/installation)):

```bash
git clone https://github.com/NVIDIA-NeMo/Automodel.git
cd Automodel
uv sync --locked --all-groups --all-extras
```

**2. Run the Ministral3 recipe** from inside the repo:

```bash
uv run automodel examples/retrieval/bi_encoder/ministral3_3b_instruct.yaml --nproc-per-node 8
```

See the [Installation Guide](/get-started/installation).

## Hugging Face Model Cards

* [mistralai/Ministral-3-3B-Base-2512](https://huggingface.co/mistralai/Ministral-3-3B-Base-2512)
* [mistralai/Ministral-3-3B-Instruct-2512](https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512)