> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/nightly/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server. # NeMo AutoModel Documentation > NeMo AutoModel is a PyTorch DTensor-native SPMD open source training library for scalable LLM and VLM training and fine-tuning with day-0 Hugging Face model support. PyTorch-native training that scales from 1 GPU to thousands with a single config change. Load any Hugging Face model, point at your data, and start training; no checkpoint conversion and no boilerplate. ## Documentation Map * [Get Started](/get-started/llms.txt): installation, configuration, Hugging Face compatibility, and repository structure. * [Recipes and End-to-End Examples](/recipes-e2e-examples/llms.txt): task-oriented training and fine-tuning guides. * [Model Coverage](/model-coverage/llms.txt): supported architectures, model pages, and the model support log. * [Development](/development/llms.txt): parallelism, precision, checkpointing, and performance features. * [API Reference](/reference/llms.txt): generated Python API documentation. 🤗 HF Compatible 🚀 Performance 📐 Scalability 🎯 SFT and PEFT 🎨 Diffusion 👁️ VLM 🌐 Omni 🌊 dLLM 🔊 Audio ⚡ Speculative #### [About](/get-started/about) Overview of NeMo AutoModel and its capabilities. #### [Key Features](/get-started/key-features) Supported workflows, parallelism, recipes, and benchmarks. #### [🤗 HF Integration](/get-started/hf-compatibility) A `transformers`-compatible library with accelerated model implementations. #### [Model Coverage](/model-coverage/overview) Built on `transformers` for day-0 model support and OOTB compatibility. ## Get Started ```bash git clone https://github.com/NVIDIA-NeMo/Automodel.git cd Automodel uv sync --locked --all-groups --extra all uv run automodel examples/llm_finetune/llama3_2/llama3_2_1b_squad.yaml --nproc-per-node 2 ``` See the [installation guide](/get-started/installation) for Docker, source builds, and multi-node setup. See the [configuration guide](/get-started/configuration) for YAML recipes and CLI overrides. Launch on a [local workstation](/job-launchers/local-workstation) or [Slurm cluster](/job-launchers/slurm-cluster). ## 10 Most Recently Supported Models This short excerpt highlights the 10 newest additions. Pick a model below to start fine-tuning, or see the [full model support log](/model-coverage/release-log) for the complete history. | Date | Type | Model | Recipe | | :--------- | :-------- | :-------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------- | | 2026-10-16 | LLM | [Beam-1.0-501B-A23B](/model-coverage/large-language-models/reflection/Beam-1.0-501B-A23B) | [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/beam/beam_1_0_501b_a23b_tulu3_ep64_8nodes.yaml) | | 2026-10-16 | LLM | [Muse-Spark-open-816B-A42B](/model-coverage/vision-language-models/meta-models/Muse-Spark-open-816B-A42B) | [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/muse_spark/muse_spark_open_tulu3_packed128k_ep32_pp2_cp32_magi.yaml) | | 2026-10-16 | LLM | [Step-5-Preview-BF16](/model-coverage/large-language-models/stepfun-ai/Step-5-Preview) | [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_benchmark/step/step_5_preview_full92_mock_h100.yaml) | | 2026-10-16 | VLM | [Muse-Spark-open-816B-A42B](/model-coverage/vision-language-models/meta-models/Muse-Spark-open-816B-A42B) | [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/muse_spark/muse_spark_open_medpix_nonpacked4k_ep64_pp4_100steps.yaml) | | 2026-10-15 | Diffusion | [FLUX-3-Image-Klein-Base](/model-coverage/diffusion/black-forest-labs/flux-3-image-klein-base) | [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/diffusion/finetune/flux3_t2i_flow_lora.yaml) | | 2026-10-14 | VLM | [Step-5-Preview](/model-coverage/vision-language-models/stepfun-ai/Step-5-Preview) | | | 2026-10-05 | Reranking | [Qwen3-Reranker-4B](/model-coverage/reranking-models/qwen/Qwen3-Reranker-4B) | [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/retrieval/cross_encoder/qwen3_4b_reranker_agentir.yaml) | | 2026-10-04 | Diffusion | [HunyuanImage-3.0](/model-coverage/diffusion/tencent/HunyuanImage-3.0) | [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/diffusion/finetune/hunyuan_image3_t2i_flow.yaml) | | 2026-10-03 | Reranking | [Ministral-3-3B-Instruct-2512-BF16](/model-coverage/embedding-models/mistralai/Ministral-3-3B-Base-2512) | [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/retrieval/cross_encoder/mistral3_vl_reranker.yaml) | | 2026-10-02 | LLM | [MiMo-V2.6-Pro-RL](/model-coverage/large-language-models/xiaomimimo/MiMo-V2.6-Pro-RL) | [recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/mimo_v2_flash/mimo_v2_6_pro_rl_tulu3_packed64k_ep64pp2cp8_100steps.yaml) | ## Recipes and Guides Find the right guide for your task: fine-tuning, pretraining, distillation, diffusion, and more. | I want to... | Choose this when... | Input Data | Model | Guide | | ------------------------------------------ | ------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **SFT (full fine-tune)** | You need maximum accuracy and have the GPU budget to update all weights | Instruction / chat dataset | LLM | [Start fine-tuning](/recipes-e2e-examples/sft-peft) | | **PEFT (LoRA)** | You want to fine-tune on limited GPU memory; updates \<1% of parameters | Instruction / chat dataset | LLM | [Start LoRA](/recipes-e2e-examples/sft-peft) | | **Tool / function calling** | Your model needs to call APIs or tools with structured arguments | Function-calling dataset (queries + tool schemas) | LLM | [Add tool calling](/recipes-e2e-examples/function-calling) | | **Fine-tune VLM** | Your task involves both images and text (e.g., visual QA, captioning) | Image + text dataset | VLM | [Fine-tune VLM](/recipes-e2e-examples/gemma-3-3n) | | **Fine-tune NVIDIA Nemotron 3.5 Super VL** | Adapt the 120B-A12B model with full SFT or LoRA, or train on 256K text packs | CORD-v2 images + text; Tulu-3 text | VLM (MoE) | [CORD-v2 SFT and PEFT](/recipes-e2e-examples/nemotron-3-5-super-vl), [256K GB200 recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/nemotron_3_5_super_vl/nemotron_3_5_super_vl_tulu3_packed_256k_cp8_ep16_gb200.yaml) | | **Fine-tune Gemma 4** | You want to fine-tune Gemma 4 for structured extraction from images (e.g., receipts) | Image + text dataset | VLM | [Fine-tune Gemma 4](/recipes-e2e-examples/gemma-4) | | **Fine-tune dLLM** | You want to fine-tune a diffusion language model (e.g., LLaDA) using masked denoising | Instruction / chat dataset | dLLM | [Fine-tune dLLM](/recipes-e2e-examples/dllm-fine-tuning) | | **Fine-tune Diffusion** | You want to fine-tune a diffusion model for image or video generation | Video / Image dataset | Diffusion | [Fine-tune Diffusion](/recipes-e2e-examples/diffusion-fine-tuning) | | **Fine-tune VLM-MoE** | You need large-scale vision-language training with sparse MoE efficiency | Image + text dataset | VLM (MoE) | [Fine-tune VLM-MoE](/recipes-e2e-examples/qwen3-5-vl) | | **Fine-tune agentic VLM-MoE** | You need image/video context for agentic developer workflows | Image / video + text dataset | VLM (MoE) | [Fine-tune Step-3.7-Flash](/recipes-e2e-examples/step-3-7) | | **Fine-tune Audio ASR** | Adapt Qwen3-Omni for speech recognition on HF audio datasets | Audio + transcript dataset | Qwen3-Omni | [Fine-tune Qwen3-Omni ASR](/recipes-e2e-examples/qwen3-omni-asr) | | **Embedding fine-tune** | You want to improve text similarity for search, retrieval, or RAG | Text pairs / retrieval corpus | LLM | [Train a bi-encoder or cross-encoder](/recipes-e2e-examples/retrieval-finetune), [Nemotron end-to-end text bi-encoder recipe](https://github.com/NVIDIA-NeMo/Nemotron/tree/main/docs/nemotron/embed) | | **Fine-tune a large MoE** | You are adapting a large sparse MoE model (DeepSeek-V3, GLM-5, etc.) to your domain | Text dataset (e.g., HellaSwag) | LLM (MoE) | [Fine-tune MoE](/recipes-e2e-examples/large-moe-fine-tuning) | | **Fine-tune DeepSeek-V4 Flash** | You want to fine-tune the DeepSeek-V4 Flash hybrid-attention MoE (SWA / CSA / HCA + hash-routing) | Text dataset (e.g., HellaSwag) | LLM (MoE) | [Fine-tune DeepSeek-V4 Flash](/recipes-e2e-examples/deepseek-v4-flash) | | **Fine-tune Hy3-preview** | You want to fine-tune Tencent's 295B MoE with sigmoid routing and per-head QK RMSNorm | Text dataset (e.g., HellaSwag) | LLM (MoE) | [Fine-tune Hy3-preview](/recipes-e2e-examples/hy3-preview) | | **Fine-tune Nemotron-3 Ultra** | You want to fine-tune NVIDIA's 550B-A55B hybrid Mamba-2 / LatentMoE model with MTP | Text dataset (e.g., HellaSwag) | LLM (MoE) | [Fine-tune Nemotron-3 Ultra](/recipes-e2e-examples/nemotron-3-ultra) | | **Sequence classification** | You need to classify text into categories (sentiment, topic, NLI) | Text + labels (e.g., GLUE MRPC) | LLM | [Train classifier](/recipes-e2e-examples/sequence-classification) | | **QAT fine-tune** | You want a quantized model that keeps accuracy for efficient deployment | Text dataset | LLM | [Enable QAT](/recipes-e2e-examples/qat) | | **Knowledge distillation** | You want a smaller, faster model that retains most of the teacher's quality | Instruction dataset + teacher model | LLM | [Distill a model](/recipes-e2e-examples/knowledge-distillation) | | **Pretrain an LLM** | You are building a base model from scratch on your own corpus | Large unlabeled text corpus (e.g., FineWeb-Edu) | LLM | [Start pretraining](/recipes-e2e-examples/pretraining) | | **Pretrain (NanoGPT)** | You want quick pretraining experiments on a single node | FineWeb / text corpus | LLM | [Try NanoGPT](/recipes-e2e-examples/nanogpt-pretraining) | ## Performance Training throughput on NVIDIA GPUs with optimized kernels for Hugging Face models. | Model | GPUs | TFLOPs/sec/GPU | Tokens/sec/GPU | Optimizations | | ---------------- | ---- | -------------- | -------------- | ---------------------- | | DeepSeek V3 671B | 256 | 250 | 1,002 | TE + DeepEP | | GPT-OSS 20B | 8 | 279 | 13,058 | TE + DeepEP + FlexAttn | | Qwen3 MoE 30B | 8 | 277 | 12,040 | TE + DeepEP | See the [full benchmark results](/performance/performance-summary) for configuration details and more models. ## Advanced Topics Parallelism, precision, checkpointing strategies, and experiment tracking. #### [Pipeline Parallelism](/development/pipeline-parallelism) Torch-native pipelining composable with FSDP2 and DTensor. 3d-parallelism #### [FP8 Training](/development/fp8-training) Mixed-precision FP8 training with torchao. FP8 mixed-precision #### [Mixed-Precision Training](/development/mixed-precision-training) FP32 master weights, BF16 compute, and the precision traps to avoid. BF16 mixed-precision #### [Checkpointing](/development/checkpointing) Distributed checkpoints with SafeTensors output. DCP safetensors #### [Gradient Checkpointing](/development/gradient-checkpointing) Trade compute for memory with activation checkpointing. memory-efficiency #### [Quantization-Aware Training](/recipes-e2e-examples/qat) Train with quantization for deployment-ready models. QAT #### [Experiment Tracking](/development/mlflow-logging) Track experiments and metrics with MLflow and W\&B. MLflow Wandb ## For Developers #### [Repo Internals](/get-started/repo-structure) Components, recipes, and CLI architecture. #### [API Reference](/reference/api-reference) Auto-generated Python API documentation. #### [Use as a Library](/get-started/about) Drop-in accelerated backend for TRL, lm-eval-harness, OpenRLHF, or any code that loads Hugging Face models. --- :: :: :: :: :: :: :: :: > NeMo AutoModel is a PyTorch DTensor-native SPMD open source training library for scalable LLM and VLM training and fine-tuning with day-0 Hugging Face model support.