What's New

Release Notes

View as Markdown

0.5.0 · 26.06 · PyPI · GH · NGC Docker

Highlights

  • Agent SFT. Train Qwen2.5-3B on multi-turn, function-calling ChatML or ShareGPT data with full-SFT and LoRA recipes. The data adapter preserves tool calls, tool responses, and optional reasoning_content.
  • Speculative decoding. Train EAGLE-1/2/3/3.1, P-EAGLE, and DFlash drafts for Llama, Phi, Qwen3, Qwen3-MoE, GPT-OSS, and Nemotron Nano targets.
  • Discrete-diffusion LLMs (dLLMs). Train DFlash drafts and fine-tune LLaDA2 or Nemotron-Labs-Diffusion with the new discrete-diffusion workflows.
  • Diffusion. Fine-tune and generate with FLUX.2-dev, Qwen-Image, and Wan 2.2 T2V-A14B; updated FLUX, Hunyuan, and Wan 2.1 recipes add LoRA and throughput improvements.
  • Large-scale MoE. Run DeepSeek V4 with TileLang kernels, train MoE models with MXFP8 on GB200, and use context parallelism for Qwen3.5/3.6 MoE VLMs.

New and Expanded Model Support

LLM and MoE

  • DeepSeek V4 Flash: A native model and checkpoint adapter; HellaSwag full-SFT, LoRA, and packed-sequence recipes; and large-scale TileLang PP/EP recipes. The implementation covers hash routing, hyper-connections, sparse/compressed attention, FP4/FP8 checkpoint loading, and multi-token prediction.
  • ERNIE 4.5: HellaSwag recipes for the 0.3B dense and 21B-A3B MoE checkpoints.
  • MiMo-V2-Flash: A HellaSwag fine-tuning recipe and native model implementation with checkpoint conversion.
  • Ling 2.0: Mini, Flash, and 1T model paths with SFT, LoRA, HellaSwag, SQuAD, and pipeline-parallel recipes.
  • Hy3-preview: DeepEP full-SFT and LoRA recipes for the 295B MoE model, with model-specific expert-bias handling.
  • HY-MT2-30B-A3B: Native model support and a supervised fine-tuning recipe.
  • MiniMax-M2.7: Pipeline-parallel full-SFT and LoRA recipes.
  • Falcon H1: Full-SFT recipes for 0.5B, 1.5B, and 7B checkpoints, plus 7B and 34B PEFT recipes.
  • MiniCPM5-1B: Full-SFT and PEFT recipes.
  • Nemotron-3-Ultra-550B-A55B: A 16-node/128-GPU throughput benchmark and a 32-node/256-GPU full-SFT recipe using a real router, THD sequence packing, and a repeated MTP head.
  • Qwen3.5/3.6 dense and MoE: MTP support and a native dense Qwen3.5 backbone that keeps its SSM-gate parameters in fp32.

Vision-Language Models

  • Qwen3.6 VLM: 27B dense and 35B-A3B SFT/LoRA recipes, including a 4K MedPix EP8/CP2 recipe for the MoE model.
  • Mistral Medium 3.5: A 128B MedPix full fine-tuning recipe and matching LoRA recipe.
  • LLaVA-OneVision 1.5: A 4B full fine-tuning recipe and an 8B LoRA recipe.
  • VLM QLoRA: The existing quantization: configuration block now works with VLM fine-tuning, including BitsAndBytes 4-bit NF4 base weights with LoRA adapters.
  • VLM knowledge distillation: A Qwen3.5-VL 9B-to-4B KD recipe with a chunked KD loss, frozen vision/audio towers, and separate CE/KD metrics.
  • Step-3.7-Flash: Native support and MedPix SFT/LoRA recipes using EP and PP. The full-SFT recipe targets 16 nodes/128 GPUs; the LoRA recipe targets 8 nodes/64 GPUs.

Retrieval and Embeddings

  • Late-interaction retrieval: Multi-vector/MaxSim training for ColBERT-style encoders, including distributed in-batch negatives.
  • Bi-encoder positives: Cycling over every positive passage instead of choosing only one positive document per query.
  • Embedding and reranker recipes: A Ministral3 3B bi-encoder recipe and model-coverage pages for bidirectional Ministral and Llama models.

Agent SFT and Speculative Decoding

Agent SFT

  • The agent-chat data adapter accepts ChatML messages and ShareGPT conversations, merges consecutive tool calls, pairs tool responses to call IDs, and supervises assistant content and tool-call arguments.
  • train_on_last_turn_only masks every earlier assistant response and trains only on the final assistant turn.
  • reasoning_content is preserved during data conversion; mask_reasoning_content can include the trace in the prompt while excluding it from loss.
  • Generation-based tool-call evaluation reports call presence, tool-name accuracy, JSON validity, argument precision/recall, and exact argument match.

Speculative Decoding

  • EAGLE-1/2 recipes support Llama, Phi, Qwen3, and Qwen3-MoE targets. EAGLE-3 adds GPT-OSS, and EAGLE-3.1 recipes cover Llama targets.
  • A remote target server/client path can train a draft model while target supervision runs on a separate GPU.
  • P-EAGLE sequence_partitions splits a long-context draft step into loss-equivalent segments to reduce activation memory.
  • Packed variable-length training uses block-causal attention and controls to keep, save, or mask target-model reasoning traces.
  • Performance tools include SGLang acceptance/speedup benchmarking, offline target-output caching, and a fused Triton soft-target cross-entropy kernel that avoids allocating a full fp32 log-probability tensor.

Discrete-Diffusion LLMs

  • DFlash: A draft-training strategy that distills frozen target hidden states with a position-decay loss, with Nemotron Nano 30B and Qwen3-4B example recipes.
  • LLaDA2: A discrete-diffusion SFT recipe alongside the existing LLaDA workflow.
  • Nemotron-Labs-Diffusion: A hybrid diffusion-LLM SFT recipe.

Diffusion

  • FLUX.2-dev: A model adapter, dataset processor, full fine-tuning recipe, LoRA recipe, and generation configuration for its 4D positional-ID and Mistral3 text-embedding architecture.
  • Qwen-Image: Model and processor support plus full fine-tuning, pretraining, generation, and LoRA recipes. The LoRA recipe targets attention and image/text MLP layers; the data collator pads variable-length cached prompt embeddings.
  • Wan 2.2 T2V-A14B: Preprocessing, high-noise and low-noise stage fine-tuning, and generation that loads independently trained checkpoints into the matching denoisers.
  • FLUX, Hunyuan, and Wan 2.1: LoRA recipes for each workflow, a multi-node Wan 2.1 recipe, and tuned full-SFT/LoRA settings including compile and FSDP reduction-dtype controls.

Omni and Multimodal

  • Bagel: Model, dataset, pretraining and fine-tuning recipes, EMA support, checkpoint adapter, and distributed initialization. The one-node examples use token budgets validated for parity runs.
  • Qwen2.5-Omni ASR: AMI and multi-language SFT recipes for the 3B and 7B checkpoints.
  • Qwen3-Omni ASR: AMI and multi-language audio SFT recipes.
  • Nemotron-Omni vision: A dynamic-resolution image path aligned with vLLM image preprocessing, preventing training/rollout vision-embedding divergence.
  • Nemotron-Omni parallelism: Activation checkpointing for the nested language model and an EP8/CP2 VLM recipe that prepares and shards multimodal embeddings across the context mesh.

Gemma 4

  • TP/PP VLM support: TP4/PP2 and TP4/PP4 recipes for Gemma 4 31B, including pipeline-stage handling for image position IDs.
  • LoRA coverage: PEFT recipes for the 2B, 4B, 31B, and 26B-A4B MoE Gemma 4 VLM models.
  • Joint drafter: A Gemma 4 joint-drafter model, MedPix recipe, and MTP inference benchmark.

Training, Parallelism, and Performance

  • Selective FSDP2 activation checkpointing: distributed.activation_checkpointing: selective saves attention and communication operations while alternating save/recompute for matrix multiplies and grouped MoE GEMMs. It supports dense models, expert-parallel MoE, single-GPU runs, and torch.compile.
  • Memory-efficient LoRA: use_memory_efficient_lora reduces adapter training memory requirements.
  • Dion-family optimizers: Typed recipe configurations for Dion, Dion2, Muon, and NorMuon, plus a Qwen2.5-7B Muon fine-tuning recipe and corrected FSDP mesh alignment.
  • Qwen3.5/3.6 VLM context parallelism: Image/video embeddings and mRoPE positions are prepared before sequence sharding; the dense sequence index is passed to linear attention, with a CP-aware validation denominator.
  • Nemotron 3 parallelism: Context and pipeline parallelism plus MTP-related train-loop support.
  • GB200 MXFP8 MoE: Transformer Engine MXFP8 grouped-expert training for Qwen3-MoE-30B, GPT-OSS-120B, and Qwen3-MoE-235B; the bf16 expert backend now uses Transformer Engine grouped linear layers.
  • Cross-entropy kernels: Fused linear cross-entropy for custom models and a CUDA Triton implementation of EAGLE soft-target cross-entropy.
  • DeepSeek V4 kernels: TileLang implementations for sparse MLA, the lightning indexer, and MHC Sinkhorn. The CUDA environment installs TileLang and TileKernels for the DeepSeek V4 recipes.
  • Large-checkpoint loading: Memory-mapped local HF DCP reads let each rank map only its requested tensor slice instead of copying full tensors into host RAM.

Data, Recipes, and Operations

  • Added S3 and multi-storage-client object sources to the Megatron pretraining dataset path.
  • data_dir_list accepts [num_samples, path] entries for deterministic per-source sampling before concatenation while retaining plain-path entries.
  • Added flat-list blend JSON files and lazy dataset preprocessing.
  • Retrieval runtime tuning exposes DDP bucket size, static graph, buffer broadcast, unused-parameter, and gradient-bucket-view controls; FSDP2 reshard_after_forward; and averaged retrieval-loss logging.
  • Added runnable examples for agent SFT, EAGLE/DFlash, dLLM, diffusion, ASR, retrieval, DeepSeek V4, Gemma 4, Qwen3.6, Nemotron Ultra, and Bagel.
  • Added NeMo-Run as a managed job launcher alongside local, Slurm, SkyPilot, and Kubernetes workflows.

Packaging and Media Dependencies

  • The package is classified as Production/Stable, pins Transformers 5.8.1, requires Transformer Engine 2.14.1 or newer, and resolves Linux PyTorch packages from the CUDA 13.0 index.

  • Video and media packages are no longer installed by default through the base, VLM, diffusion, or all extras. Install the media union when a workflow needs OpenCV, decord, Qwen media utilities, or FFmpeg:

    $uv venv
    $source .venv/bin/activate
    $uv pip install "nemo-automodel[media]"

    Use nemo-automodel[vlm-media] or nemo-automodel[diffusion-media] when only one media stack is required.

Breaking Changes

FSDP2 now defaults gradient reduction to float32 while forward and backward compute remain in bfloat16. See Breaking Changes for the compatibility override and migration guidance.

0.4.0 · 26.04 (2026-04-28) · PyPI · GH · NGC Docker

Highlights

  • Discrete-diffusion LLMs (dLLM). SFT and generation support for dLLM models, including Llada.
  • Embedding and retrieval training. Reranker training, biencoder datasets loaded directly from the Hugging Face Hub, in-batch negative sampling, and ONNX export for biencoder models.
  • SkyPilot launcher. Native multi-node launch on cloud (SkyPilot, including Kubernetes), in addition to local interactive runs. SkyPilot and NeMo Run launchers are selected with YAML sections in the config; SLURM jobs use the sbatch slurm.sub workflow.
  • CLI install profile. The nemo-automodel[cli] extra declares pyyaml beyond the package’s base dependencies for job-submission configs.
  • Refreshed CLI. automodel <config.yaml> (alias am) replaces the older automodel <command> <domain> -c <config> form.

New Models

  • LLM: GLM-5, MiniMax-M2.5, Nemotron Super v3, Nemotron Nano 4B/8B.
  • MoE / VLM: Qwen3.5-MoE (397B-A17B, 35B-A3B).
  • VLM: Gemma 4, Mistral Small 4, Qwen3.5 small dense models.
  • Diffusion: FLUX.1-dev, Wan 2.1 T2V, HunyuanVideo 1.5; Wan multi-resolution and LoRA recipes for diffusion.

Distributed Training

  • Context parallelism for Qwen3.5-MoE and Nemotron v3.
  • Pipeline parallelism for knowledge distillation.
  • HybridEP and UCCL-EP as alternative expert-parallel dispatchers.
  • FSDP2 weight prefetching and async TP optimization.
  • TP > 1 in knowledge distillation.

Performance and Kernels

  • TE Linear layers enabled for PEFT/LoRA.
  • torch._grouped_mm expert backend.
  • fp32 RMSNorm backend and cast_model_to_dtype controls.
  • TP-aware KD loss with distributed softmax and T² scaling.
  • FlashOptim optimizer integration.
  • Sequence-packing updates: Qwen3.5-MoE VLM neat-packing recipe with EP+PP; Generic THD collation for chat datasets; CP/BSHD padding fixes.

PEFT

  • MoE LoRA: rank scaling, torch_mm integration, expert-LoRA init using config.expert_dim.
  • merge_lora tool for materializing adapters into the base model.
  • QLoRA PEFT checkpoints saved with the HF adapter prefix.

Recipes and Workflow

  • New recipes for Gemma 4 (LoRA), Nemotron Nano 4B SQuAD, Mistral Small 4, Tulu-3 E2E convergence, GPT-OSS 20B / Moonlight 16B convergence, and reranker / biencoder training.
  • MFU logging for LLM and dLLM train recipes.
  • Native Comet ML experiment tracking.
  • NEFTune noisy embeddings for instruction fine-tuning.
  • Scheduler-driven manual garbage collection.
  • Common inference utility and .generate() with KV cache for Nemotron v3.

Checkpointing

  • v4_compatible checkpoint format.
  • Diffusion full fine-tuning and pretraining examples use safetensors checkpoint format; diffusion LoRA examples use torch_save.
  • QLoRA / LoRA loading robustness; tied-weight handling moved out of _init_model.

Fixes

  • FSDP2 meta-device crash for Qwen3.5 GatedDeltaNet fp32 params.
  • Activation checkpointing silently skipped on registered VLMs (ModuleList flattening).
  • Gradient checkpointing for MoE models on single GPU (ep_size=1).
  • Gradient clipping with torch_mm + EP (GPT-OSS 120B recipe).
  • Rotary embeddings for v4 models; inputs_embeds passthrough for Nano v3.

Breaking Changes

A migration guide for the new CLI, the recipe YAML section, the SLURM sbatch-script workflow, and the nemo-automodel[cli] install profile is in Breaking Changes.

0.3.0 · 26.02 (2026-02-26) · PyPI · GH · NGC Docker

Highlights

  • Transformers v4 / v5 alignment. New transformers v4 API support and a v5 refactor for device-mesh-only model init.
  • Streaming safetensors writer for faster checkpoint export.
  • Faster fp8 dequant kernels with DTensor dequantization fixes for DSv3.

New Models

  • LLM: DeepSeek V3.2, Step-3.5-Flash, MiniMax-M2.1, Nemotron-3-Nano-30B-A3B, Nemotron Flash 1B, GLM-4.7, Devstral-Small-2-24B.
  • MoE / VLM / Omni: Qwen3-VL (4B/8B), Qwen3-VL-MoE (30B/235B), Kimi-VL, Kimi-K2.5 VL, Nemotron-Parse VLM, InternVL3.5-4B, Ministral3 (3B/8B/14B), Phi-4-multimodal.

Distributed Training

  • v5 refactor: device-mesh-only model init.
  • TP plan for Ministral; Ministral3 ported to transformers v4.
  • Pipeline-parallelism validation support.
  • Parallel diffusers generate().

Performance and Kernels

  • TE fp8 for models that support it.
  • GroupedExpertsTE backend (prerequisite for MoE fp8).
  • TE RoPE fusion for custom MoE models; norm fusion and RoPE cache for dense models.
  • Improved import time.

PEFT

  • DoRA implementation.
  • LoRA support for custom MoEs.
  • LoRA support in Biencoder.

Datasets and Workflow

  • Databricks Delta Lake dataset support; consolidation for Databricks.
  • Parquet file support; inline text dataset format.
  • ColumnMapped: configurable special tokens, chat-template flags, and answer-only masking.
  • Hard negative mining and biencoder + inline-dataset tests.
  • nsys benchmark support and model-layer name scoping in the CLI.
  • Updated checkpoint auto-loading with explicit restore_from.
  • Dion optimizer.
  • Functiongemma + xlam tool-calling recipes.

Fixes

  • inputs_embeds passthrough for Nano v3.
  • from_pretrained / from_config simplification with model-id pass-through.
  • Tied-embedding detection improvements.

0.2.0 · 25.11 (2025-12-04) · PyPI · GH · NGC Docker

Highlights

  • Async checkpointing. Checkpoint refactor with async DCP and HF safetensors backport / consolidation.
  • Custom MoE optimizations. FSDP optimizations, packed-sequence + context parallel through TE, configurable router precision, fp32 lm_head and fp32 apply_rope.
  • Performance documentation. New performance-summary doc and benchmarking recipe with configs.
  • Multinode + cluster guidance. Multinode configs and updated launcher docs.

New Models

  • MoE: Qwen3 MoE custom implementation, Qwen3 Next, GPT-OSS (custom implementation, dequantization, DGX Spark recipe), GLM 4 / 4.5 / 4.6 MoE, GLM 4.5 Air, Moonlight 2L test, Phi 4 (TP plan).
  • Omni / VLM: Qwen3-Omni OOTB recipe and custom implementation.
  • DeepSeek v3 with fp8 base checkpoint loading.
  • Sequence classification: Qwen3ForSequenceClassification registered; generic SFT sequence-classification recipe.

Distributed Training

  • VLM expert-parallel recipe support.
  • PP for VLM; PEFT with PP.
  • Sharding optimization for SP / LoRA.
  • clip_grad_norm across all parallelism modes.
  • fully_shard_by_dtype option.
  • Out-of-tree (OOT) parallelism decorator.

Performance and Kernels

  • Mask creation moved into the data pipeline for better performance.
  • TE attention for GPT-OSS.
  • Faster fp8 dequant; auto-detect base-weights dequant.

PEFT

  • LoRA-aware ColwiseParallel / RowwiseParallel.
  • LoRA + TE.
  • MFU estimation for LoRA.
  • Additional PEFT LoRA recipes.

Datasets and Recipes

  • Multiturn chat dataset; VLM multiturn chat support.
  • Tool-calling dataset and recipe.
  • Streaming dataset.
  • Multiple validation datasets with per-dataset logging.
  • ColumnMapped: surface truncating + padding options.
  • Configurable max-clip-grad; configurable remote-logging frequency using step_scheduler.
  • Validation-loss checkpoint, run-val-at-ckpt, best-ckpt symlink.
  • InternVL recipe; Qwen3-VL 30B recipe; Llama-Embed-Nemotron-8B training.

Logging and Observability

  • MLflow integration.
  • Metric logger with JSONL output.
  • YAML logging-to-stdout improvements.

Workflow

  • Knowledge-distillation custom validation step; ScopedModuleOffloading to reduce memory.
  • Model Registry component.
  • SIGTERM handling.
  • NEMO_ENABLE_USER_MODULES for user-extension modules.
  • Rank-0 download for custom models.
  • Dereference env vars in YAML.

0.1.2 (2025-10-23) · PyPI · GH

Patch release.

  • Fix: max_steps now set inside the constructor (#650).
  • Fix: step scheduler switched to zero-based indexing (#627).
  • Fix: sample-limit handling for ColumnMapped datasets (#521).

0.1.0 (2025-10-08) · PyPI · GH

Initial public release of NeMo AutoModel.

Highlights

  • PyTorch-native training framework for LLMs and VLMs with Hugging Face Transformers integration via NeMoAuto* wrapper classes.
  • YAML-driven recipes for SFT and PEFT.
  • FSDP2 / HSDP / DDP distributed training with DTensor sharding.
  • Megatron-FSDP available as the default heavy-duty sharding option (replaces the earlier nvFSDP path).
  • Knowledge distillation recipe.
  • MoE component with DeepSeek v3 model implementation.
  • ColumnMappedTextInstructionDataset for instruction tuning.
  • Gradient checkpointing.
  • SLURM launcher.

For the list of newly supported models per release, see the Model Coverage Release Log.