> This page is for version Nightly (default). > For other versions, use one of these documentation indexes: > - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt > - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt > - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt > - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt > - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server. # Mistral-Large-4-1T-A52B > Fine-tune Mistral-Large-4-1T-A52B with NeMo AutoModel on MedPix and packed Tulu3 data using HybridEP, pipeline parallelism, and context parallelism. [Mistral Large 4](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B) is Mistral AI's 1T-parameter (52B active) multimodal Mixture-of-Experts (MoE) model. NeMo AutoModel provides native text (`MistralLarge4ForCausalLM`) and image-text (`MistralLarge4ForConditionalGeneration`) implementations. ## Fine-Tune Mistral Large 4 Follow the [installation instructions](/nemo/automodel/get-started/installation). The full-model recipes run on 32 nodes x 4 GB200 (128 GPUs); use the [Slurm launcher guide](/nemo/automodel/job-launchers/slurm-cluster) for multi-node runs. ```bash uv run torchrun --nnodes=32 --nproc-per-node=4 \ --node-rank="$NODE_RANK" --master-addr="$MASTER_ADDR" --master-port="$MASTER_PORT" \ examples/vlm_finetune/finetune.py \ --config examples/vlm_finetune/mistral/mistral_large4_medpix_ep64_pp2.yaml ``` ## Choose a Workflow | Goal | Start Here | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Full image-text SFT on MedPix | [mistral_large4_medpix_ep64_pp2.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/mistral/mistral_large4_medpix_ep64_pp2.yaml) | | 128K packed text SFT on Tulu3 | [mistral_large4_tulu3_packed128k_ep64_pp2_cp32_tilelang.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/mistral/mistral_large4_tulu3_packed128k_ep64_pp2_cp32_tilelang.yaml) | | Throughput benchmark | [mistral_large4_gb200.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_benchmark/mistral/mistral_large4_gb200.yaml) | ### Configuration | Setting | MedPix SFT | Tulu3 128K | | ----------------- | ---------------------------------------------------------- | ----------------------------------------------------- | | GPUs | 128 | 128 | | Parallelism | EP64, PP2, CP1, TP1; FSDP2 | EP64, PP2, CP32, TP1, DP2; FSDP2 | | Pipeline schedule | Interleaved 1F1B, four stages of 18 layers | 1F1B, two stages of 36 layers | | Sequence | Nonpacked, 4,096 tokens | Packed THD, 131,072 tokens | | Batch | Global 2,048, local 32 | Global 8, local 4 | | Attention | SDPA | TileLang sparse MLA | | Experts | `torch_mm`, HybridEP dispatcher | `torch_mm`, HybridEP dispatcher | | Optimizer | AdamW, LR 5e-6, betas 0.9/0.95, weight decay 0.1, clip 1.0 | FusedAdam, LR 5e-6, BF16 moments, FP32 master weights | Both SFT recipes use full activation checkpointing and keep the DSA indexer frozen. Both recipes freeze the vision tower. W\&B is opt-in: set `wandb.enable: true` and configure your entity and project when launching. With TP1, the data-parallel degree for batch accounting is `world_size / (PP * CP)`. ## Checkpoint The recipes load `mistralai/Mistral-Large-4-1T-A52B` directly. `checkpoint.dequantize_base_checkpoint: true` reconstructs the FP8 payloads in FP32 and stores the weights in BF16. Router, LeRoPE, and indexer-normalization state stays in FP32. FP8 indexer and KV-cache execution are disabled for training. ## Tulu3 Data The 128K recipe tokenizes and packs `allenai/tulu-3-sft-mixture` online. Labels are assistant-only, and each document is padded to a multiple of `2 * cp_size` tokens. ## Training Results | Run | Configuration | Result | | -------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------- | | Tulu3 128K SFT | 128 GB200, EP64/PP2/CP32, packed 131,072 tokens, GBS 8 | 50 steps; loss 0.9597 to 0.6602; 9,087 tokens/s; 88.8 GiB on the reporting rank | Training logs: [4K MedPix SFT](https://forge.coreweave.com/wandb/nvidia/automodel/runs/d3295d876730) and [128K Tulu3 SFT](https://forge.coreweave.com/wandb/nvidia/automodel/runs/286e3fe29adb). These training measurements do not close the numerical gates listed below. ## Numerical Validation | Check | Result | | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | Four-layer logits vs. an independent FP32 reference (3 seeds) | Mean absolute error 6.1e-9 to 7.0e-9, p95 1.5e-8 to 2.1e-8, max 4.5e-8; top-1 agreement 100% | | CP2 image-text vs. CP1 (FP32) | Max absolute logit error 7.5e-8, parameter-gradient error 1.2e-7 | | CP2/PP2/EP2 with HybridEP (BF16) | Identical logits; aggregate gradient relative L2 0.45% | | TileLang sparse MLA (CP1, CP2) vs. FP32 oracle | Output relative L2 0.24%; query/KV/value-weight gradients 0.31% / 0.28% / 0.24% | ## Model Reference ### Model Architecture | Property | Value | | -------------------- | ------------------------------------------------------------------------------------------------------------------ | | Task | Image-Text-to-Text | | Architectures | `MistralLarge4ForCausalLM`, `MistralLarge4ForConditionalGeneration`; `model_type: mistral_large4` | | Parameters | 1T total, 52B active | | Language model | 72 layers, hidden size 6,144, vocabulary 204,800 | | Attention | MLA with sparse DSA selection (top-k 2,048) and IndexShare; 64 heads, 512-dimensional KV latent | | Experts | 128 routed (top-4) and one shared; the first layer uses a dense FFN | | Positions | Trainable LeRoPE/YaRN, periodic no-positional-embedding (NoPE) layers, SSMax, cross-layer KV sharing from layer 35 | | Vision | 30 layers, hidden size 2,048, 16 heads, patch size 14, 2x2 patch merger | | Output | Logit soft cap 30 | | Checkpoint precision | FP8 linear and expert weights with FP32 scales; BF16 vision and projector | ### Available Models | Model | Hugging Face ID | | --------------- | ----------------------------------------------------------------------------------------------- | | Mistral Large 4 | [`mistralai/Mistral-Large-4-1T-A52B`](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B) | ## Related Resources * [Multi-modal dataset guide](/nemo/automodel/datasets/multi-modal-dataset) * [Run on a cluster](/nemo/automodel/job-launchers/slurm-cluster)