> This page is for version 0.7.0 ยท 26.10. > For other versions, use one of these documentation indexes: > - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt > - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt > - 0.7.0 ยท 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt > - 0.5.0 ยท 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt > - 0.4.0 ยท 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server. # NeMo AutoModel Documentation > NeMo AutoModel is a PyTorch DTensor-native SPMD open source training library for scalable LLM and VLM training and fine-tuning with day-0 Hugging Face model support. PyTorch-native training that scales from 1 GPU to thousands with a single config change. Load any Hugging Face model, point at your data, and start training; no checkpoint conversion and no boilerplate. ## Documentation Map * [Get Started](/nemo/automodel/get-started/llms.txt): installation, configuration, Hugging Face compatibility, and repository structure. * [Training Recipes](/nemo/automodel/v0.7/recipes-e2e-examples/overview): reusable training workflows and example configurations. * [Tutorials](/nemo/automodel/v0.7/tutorials/overview): walkthroughs for specific models and tasks. * [Model Coverage](/nemo/automodel/model-coverage/llms.txt): supported architectures, model pages, and the model support log. * [Development](/nemo/automodel/development/llms.txt): parallelism, precision, checkpointing, and performance features. * [API Reference](/nemo/automodel/reference/llms.txt): generated Python API documentation. ๐Ÿค— HF Compatible ๐Ÿš€ Performance ๐Ÿ“ Scalability ๐ŸŽฏ SFT and PEFT ๐ŸŽจ Diffusion ๐Ÿ‘๏ธ VLM ๐ŸŒ Omni ๐ŸŒŠ dLLM ๐Ÿ”Š Audio โšก Speculative #### [About](/nemo/automodel/v0.7/get-started/about) Overview of NeMo AutoModel and its capabilities. #### [Key Features](/nemo/automodel/v0.7/get-started/key-features) Supported workflows, parallelism, recipes, and benchmarks. #### [๐Ÿค— HF Integration](/nemo/automodel/v0.7/get-started/hf-compatibility) A `transformers`-compatible library with accelerated model implementations. #### [Model Coverage](/nemo/automodel/v0.7/model-coverage/overview) Built on `transformers` for day-0 model support and OOTB compatibility. ## Get Started ```bash mkdir -p checkpoints docker run --gpus all -it --rm --shm-size=8g \ -v "$PWD/checkpoints:/opt/Automodel/checkpoints" \ nvcr.io/nvidia/nemo-automodel:26.10.00 \ automodel /opt/Automodel/examples/llm_finetune/qwen/qwen3_0p6b_hellaswag.yaml \ --nproc-per-node 2 \ --step_scheduler.max_steps 10 ``` This 10-step example fine-tunes [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) on HellaSwag using two GPUs. The model and dataset are public; no Hugging Face token is required. Checkpoints are enabled and saved to the mounted `checkpoints` directory. See the [installation guide](/nemo/automodel/v0.7/get-started/installation) for Docker prerequisites, source builds, and multi-node setup. See the [configuration guide](/nemo/automodel/v0.7/get-started/configuration) for YAML recipes and CLI overrides. Launch on a [local workstation](/nemo/automodel/v0.7/job-launchers/local-workstation) or [Slurm cluster](/nemo/automodel/v0.7/job-launchers/slurm-cluster). ## 10 Most Recently Supported Models This short excerpt highlights the 10 newest additions. Pick a model below to start fine-tuning, or see the [full model support log](/nemo/automodel/v0.7/model-coverage/release-log) for the complete history. ## Recipes and Guides Find the right guide for your task: fine-tuning, pretraining, distillation, diffusion, and more. | I want to... | Choose this when... | Input Data | Model | Guide | | ------------------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **SFT (full fine-tune)** | You need maximum accuracy and have the GPU budget to update all weights | Instruction / chat dataset | LLM | [Start fine-tuning](/nemo/automodel/v0.7/recipes-e2e-examples/sft-peft) | | **PEFT (LoRA)** | You want to fine-tune on limited GPU memory; updates \<1% of parameters | Instruction / chat dataset | LLM | [Start LoRA](/nemo/automodel/v0.7/recipes-e2e-examples/sft-peft) | | **Tool / function calling** | Your model needs to call APIs or tools with structured arguments | Function-calling dataset (queries + tool schemas) | LLM | [Add tool calling](/nemo/automodel/v0.7/recipes-e2e-examples/function-calling) | | **Fine-tune VLM** | Your task involves both images and text (e.g., visual QA, captioning) | Image + text dataset | VLM | [Fine-tune VLM](/nemo/automodel/v0.7/recipes-e2e-examples/gemma-3-3n) | | **Fine-tune Gemma 4** | You want to fine-tune Gemma 4 for structured extraction from images (e.g., receipts) | Image + text dataset | VLM | [Fine-tune Gemma 4](/nemo/automodel/v0.7/recipes-e2e-examples/gemma-4) | | **Fine-tune dLLM** | You want to fine-tune a diffusion language model (e.g., LLaDA) using masked denoising | Instruction / chat dataset | dLLM | [Fine-tune dLLM](/nemo/automodel/v0.7/recipes-e2e-examples/dllm-fine-tuning) | | **Fine-tune Diffusion** | You want to fine-tune a diffusion model for image or video generation | Video / Image dataset | Diffusion | [Fine-tune Diffusion](/nemo/automodel/v0.7/recipes-e2e-examples/diffusion-fine-tuning) | | **Fine-tune VLM-MoE** | You need large-scale vision-language training with sparse MoE efficiency | Image + text dataset | VLM (MoE) | [Fine-tune VLM-MoE](/nemo/automodel/v0.7/recipes-e2e-examples/qwen3-5-vl) | | **Fine-tune agentic VLM-MoE** | You need image/video context for agentic developer workflows | Image / video + text dataset | VLM (MoE) | [Fine-tune Step-3.7-Flash](/nemo/automodel/v0.7/recipes-e2e-examples/step-3-7) | | **Fine-tune Audio ASR** | Adapt Qwen3-Omni for speech recognition on HF audio datasets | Audio + transcript dataset | Qwen3-Omni | [Fine-tune Qwen3-Omni ASR](/nemo/automodel/v0.7/recipes-e2e-examples/qwen3-omni-asr) | | **Embedding fine-tune** | You want to improve text similarity for search, retrieval, or RAG | Text pairs / retrieval corpus | LLM | [Train a bi-encoder or cross-encoder](/nemo/automodel/v0.7/recipes-e2e-examples/retrieval-finetune), [Nemotron end-to-end text bi-encoder recipe](https://github.com/NVIDIA-NeMo/Nemotron/tree/main/docs/nemotron/embed) | | **Fine-tune a large MoE** | You are adapting a large sparse MoE model (DeepSeek-V3, GLM-5, etc.) to your domain | Text dataset (e.g., HellaSwag) | LLM (MoE) | [Fine-tune MoE](/nemo/automodel/v0.7/recipes-e2e-examples/large-moe-fine-tuning) | | **Fine-tune DeepSeek-V4 Flash** | You want to fine-tune the DeepSeek-V4 Flash hybrid-attention MoE (SWA / CSA / HCA + hash-routing) | Text dataset (e.g., HellaSwag) | LLM (MoE) | [Fine-tune DeepSeek-V4 Flash](/nemo/automodel/v0.7/recipes-e2e-examples/deepseek-v4-flash) | | **Fine-tune Hy3-preview** | You want to fine-tune Tencent's 295B MoE with sigmoid routing and per-head QK RMSNorm | Text dataset (e.g., HellaSwag) | LLM (MoE) | [Fine-tune Hy3-preview](/nemo/automodel/v0.7/recipes-e2e-examples/hy3-preview) | | **Fine-tune Nemotron-3 Ultra** | You want to fine-tune NVIDIA's 550B-A55B hybrid Mamba-2 / LatentMoE model with MTP | Text dataset (e.g., HellaSwag) | LLM (MoE) | [Fine-tune Nemotron-3 Ultra](/nemo/automodel/v0.7/recipes-e2e-examples/nemotron-3-ultra) | | **Sequence classification** | You need to classify text into categories (sentiment, topic, NLI) | Text + labels (e.g., GLUE MRPC) | LLM | [Train classifier](/nemo/automodel/v0.7/recipes-e2e-examples/sequence-classification) | | **QAT fine-tune** | You want a quantized model that keeps accuracy for efficient deployment | Text dataset | LLM | [Enable QAT](/nemo/automodel/v0.7/recipes-e2e-examples/qat) | | **Knowledge distillation** | You want a smaller, faster model that retains most of the teacher's quality | Instruction dataset + teacher model | LLM | [Distill a model](/nemo/automodel/v0.7/recipes-e2e-examples/knowledge-distillation) | | **Pretrain an LLM** | You are building a base model from scratch on your own corpus | Large unlabeled text corpus (e.g., FineWeb-Edu) | LLM | [Start pretraining](/nemo/automodel/v0.7/recipes-e2e-examples/pretraining) | | **Pretrain (NanoGPT)** | You want quick pretraining experiments on a single node | FineWeb / text corpus | LLM | [Try NanoGPT](/nemo/automodel/v0.7/recipes-e2e-examples/nanogpt-pretraining) | ## Performance Recent BF16 full fine-tuning results on NVIDIA H100 80GB GPUs, using the workloads and sequence lengths below. | Model | GPUs | Workload | Sequence Length | Tokens/sec/GPU | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---: | ------------- | --------------: | -------------: | | [Qwen3.8-27B](https://github.com/NVIDIA-NeMo/Automodel/blob/b203f3f9f9f3140c660942a088ff36bbc3a5e856/examples/vlm_finetune/qwen3_8/qwen3_8_27b.yaml) | 8 | MedPix VLM | Variable | 233.07 | | [GLM-5.3](https://github.com/NVIDIA-NeMo/Automodel/blob/b203f3f9f9f3140c660942a088ff36bbc3a5e856/examples/llm_finetune/glm/glm_5.3_tulu3_4k_cudnn_100step.yaml) | 256 | Tulu-3 packed | 4096 | 162.56 | | [MiMo-V2.6-Flash-RL](https://github.com/NVIDIA-NeMo/Automodel/blob/b203f3f9f9f3140c660942a088ff36bbc3a5e856/examples/llm_finetune/mimo_v2_flash/mimo_v2_6_flash_rl_tulu3_packed4k_ep64_cp2_100steps.yaml) | 64 | Tulu-3 packed | 4096 | 640.80 | | [Nemotron-3.5-Lightning-30B-A3B-BF16](https://github.com/NVIDIA-NeMo/Automodel/blob/b203f3f9f9f3140c660942a088ff36bbc3a5e856/examples/llm_finetune/nemotron/nemotron_nano_v3_5_lightning_tulu3_packed_cp8_64k.yaml) | 8 | Tulu-3 packed | 8192 | 6,895.85 | Qwen3.8-27B freezes the vision encoder. See the [full benchmark results](/nemo/automodel/v0.7/performance/performance-summary) for measurement settings, pinned recipes, and more models. ## Advanced Topics Parallelism, precision, checkpointing strategies, and experiment tracking. #### [Pipeline Parallelism](/nemo/automodel/v0.7/development/pipeline-parallelism) Torch-native pipelining composable with FSDP2 and DTensor. 3d-parallelism #### [FP8 Training](/nemo/automodel/v0.7/development/fp8-training) Mixed-precision FP8 training with torchao. FP8 mixed-precision #### [Mixed-Precision Training](/nemo/automodel/v0.7/development/mixed-precision-training) FP32 master weights, BF16 compute, and the precision traps to avoid. BF16 mixed-precision #### [Checkpointing](/nemo/automodel/v0.7/development/checkpointing) Distributed checkpoints with SafeTensors output. DCP safetensors #### [Gradient Checkpointing](/nemo/automodel/v0.7/development/gradient-checkpointing) Trade compute for memory with activation checkpointing. memory-efficiency #### [Quantization-Aware Training](/nemo/automodel/v0.7/recipes-e2e-examples/qat) Train with quantization for deployment-ready models. QAT #### [Experiment Tracking](/nemo/automodel/v0.7/development/mlflow-logging) Track experiments and metrics with MLflow and W\&B. MLflow Wandb ## For Developers #### [Repo Internals](/nemo/automodel/v0.7/get-started/repo-structure) Components, recipes, and CLI architecture. #### [API Reference](/nemo/automodel/v0.7/reference/api-reference) Auto-generated Python API documentation. #### [Use as a Library](/nemo/automodel/v0.7/get-started/about) Drop-in accelerated backend for TRL, lm-eval-harness, OpenRLHF, or any code that loads Hugging Face models. --- :: :: :: :: :: :: :: ::