Mistral-Large-4-1T-A52B

Mistral Large 4 is Mistral AI’s 1T-parameter (52B active) multimodal Mixture-of-Experts (MoE) model. NeMo AutoModel provides native text (MistralLarge4ForCausalLM) and image-text (MistralLarge4ForConditionalGeneration) implementations.

Fine-Tune Mistral Large 4

Follow the installation instructions. The full-model recipes run on 32 nodes x 4 GB200 (128 GPUs); use the Slurm launcher guide for multi-node runs.

uv run torchrun --nnodes=32 --nproc-per-node=4 \
  --node-rank="$NODE_RANK" --master-addr="$MASTER_ADDR" --master-port="$MASTER_PORT" \
  examples/vlm_finetune/finetune.py \
  --config examples/vlm_finetune/mistral/mistral_large4_medpix_ep64_pp2.yaml

Choose a Workflow

Goal	Start Here
Full image-text SFT on MedPix	mistral_large4_medpix_ep64_pp2.yaml
128K packed text SFT on Tulu3	mistral_large4_tulu3_packed128k_ep64_pp2_cp32_tilelang.yaml
Throughput benchmark	mistral_large4_gb200.yaml

Configuration

Setting	MedPix SFT	Tulu3 128K
GPUs	128	128
Parallelism	EP64, PP2, CP1, TP1; FSDP2	EP64, PP2, CP32, TP1, DP2; FSDP2
Pipeline schedule	Interleaved 1F1B, four stages of 18 layers	1F1B, two stages of 36 layers
Sequence	Nonpacked, 4,096 tokens	Packed THD, 131,072 tokens
Batch	Global 2,048, local 32	Global 8, local 4
Attention	SDPA	TileLang sparse MLA
Experts	`torch_mm`, HybridEP dispatcher	`torch_mm`, HybridEP dispatcher
Optimizer	AdamW, LR 5e-6, betas 0.9/0.95, weight decay 0.1, clip 1.0	FusedAdam, LR 5e-6, BF16 moments, FP32 master weights

Both SFT recipes use full activation checkpointing and keep the DSA indexer frozen. Both recipes freeze the vision tower. W&B is opt-in: set wandb.enable: true and configure your entity and project when launching. With TP1, the data-parallel degree for batch accounting is world_size / (PP * CP).

Checkpoint

The recipes load mistralai/Mistral-Large-4-1T-A52B directly. checkpoint.dequantize_base_checkpoint: true reconstructs the FP8 payloads in FP32 and stores the weights in BF16. Router, LeRoPE, and indexer-normalization state stays in FP32. FP8 indexer and KV-cache execution are disabled for training.

Tulu3 Data

The 128K recipe tokenizes and packs allenai/tulu-3-sft-mixture online. Labels are assistant-only, and each document is padded to a multiple of 2 * cp_size tokens.

Training Results

Run	Configuration	Result
Tulu3 128K SFT	128 GB200, EP64/PP2/CP32, packed 131,072 tokens, GBS 8	50 steps; loss 0.9597 to 0.6602; 9,087 tokens/s; 88.8 GiB on the reporting rank

Training logs: 4K MedPix SFT and 128K Tulu3 SFT. These training measurements do not close the numerical gates listed below.

Numerical Validation

Check	Result
Four-layer logits vs. an independent FP32 reference (3 seeds)	Mean absolute error 6.1e-9 to 7.0e-9, p95 1.5e-8 to 2.1e-8, max 4.5e-8; top-1 agreement 100%
CP2 image-text vs. CP1 (FP32)	Max absolute logit error 7.5e-8, parameter-gradient error 1.2e-7
CP2/PP2/EP2 with HybridEP (BF16)	Identical logits; aggregate gradient relative L2 0.45%
TileLang sparse MLA (CP1, CP2) vs. FP32 oracle	Output relative L2 0.24%; query/KV/value-weight gradients 0.31% / 0.28% / 0.24%

Model Reference

Model Architecture

Property	Value
Task	Image-Text-to-Text
Architectures	`MistralLarge4ForCausalLM`, `MistralLarge4ForConditionalGeneration`; `model_type: mistral_large4`
Parameters	1T total, 52B active
Language model	72 layers, hidden size 6,144, vocabulary 204,800
Attention	MLA with sparse DSA selection (top-k 2,048) and IndexShare; 64 heads, 512-dimensional KV latent
Experts	128 routed (top-4) and one shared; the first layer uses a dense FFN
Positions	Trainable LeRoPE/YaRN, periodic no-positional-embedding (NoPE) layers, SSMax, cross-layer KV sharing from layer 35
Vision	30 layers, hidden size 2,048, 16 heads, patch size 14, 2x2 patch merger
Output	Logit soft cap 30
Checkpoint precision	FP8 linear and expert weights with FP32 scales; BF16 vision and projector

Available Models

Model	Hugging Face ID
Mistral Large 4	`mistralai/Mistral-Large-4-1T-A52B`