Skip to navigation
What's NewRelease Notes

Release Notes

View as Markdown

0.7.0 · 26.10 (2026-10-21) · PyPI · GH · NGC Docker

Highlights

  • Expanded model coverage. New training paths for DeepSeek-V4.1-Flash, GLM-5.3-Flash, Qwen3.8-Flash-Next, MiMo-V2.5-Pro and MiMo-V2.6, and additional vision-language models.
  • More MoE training options. Mixture-of-Kittens (MoK) BF16 and MXFP8 experts, compact HybridEP routing, and MXFP4-resident DeepSeek-V4-Flash experts for LoRA.
  • Speculative and discrete-diffusion training. DFlash 2, additional Kimi K3 and DeepSeek-V4.1 draft-training paths, streaming EAGLE-3 feature components, I-DLM training, and SCDD training and sampling.
  • Text-to-image fine-tuning. HunyuanImage-3.0 with expert parallelism and Qwen-Image-2.1 with full-parameter and LoRA recipes.
  • Training and integration improvements. Native FlashAttention-4, explicit packing contracts, weighted domain mixtures, lower-memory checkpoint loading, and public registration of custom model architectures.

New and Expanded Model Support

LLM and MoE

  • DeepSeek-V4.1-Flash: Native model and checkpoint conversion, attention and expert LoRA, packed SFT and context parallelism, H100-80GB recipes, and MedPix vision SFT. The release includes the CP8 gradient-synchronization fix. (model guide, #3855, #3872, #3879, #3962, #4221)
  • GLM-5.3-Flash: Native hybrid KDA/DSA model, vision processing, distributed checkpoint loading, and packed context-parallel training. The MedPix recipe uses EP72/CP2 on 72 H100 GPUs; sparse attention combines FlashMLA forward with cuDNN backward. TP, PP, video training, and full-model single-GPU loading are outside this model’s supported path. GLM-5.2 and GLM-5.3 also gain cuDNN DSA training backends. (model guide, #3699, #3127, #3747)
  • Qwen3.8: Adds 27B fine-tuning recipes and native Flash-Next language-only training with a sharded Engram embedding table, sparse QSA attention, packed sequences, and contiguous context parallelism. Flash-Next’s QSA indexers remain frozen; its checkpoint MTP head is not loaded. TP and PP are disabled in the documented Flash-Next configurations. (model guide, #3560, #3690)
  • MiMo: Adds MiMo-V2.5-Pro, MiMo-V2.6, and MiMo-V2.6-Pro support. The MiMo-V2.6-Pro-RL recipes cover packed 64K Tulu3 text, MedPix full fine-tuning, and MedPix LoRA on GB200. Keep expert-parallel groups within an NVLink domain. (recipe guide, #2514, #3992, #4054)
  • Additional architectures: Dense Nemotron-H Nano 4B, a Laguna XS 2.1 training recipe, and encoder-decoder fine-tuning through NeMoAutoModelForSeq2SeqLM, including a T5-small SQuAD example. The seq2seq path does not wire up CP, TP, PP, or sequence packing. (#2670, #3639, #2860)

Vision-Language Models

  • DeepSeek-V4-Flash Vision and Muse Glimmer: Adds the DeepSeek-V4 vision path and native Muse Glimmer training, checkpoint conversion, and distributed model support. (#3772, #3476)
  • Cohere North Micro Vision: Adds Shopify product-catalogue and RDR LoRA recipes, including checkpoint resume. This preview model uses remote-code dependencies and is excluded from automatic release-scope CI discovery. (#3518, #3400)
  • Existing VLM improvements: Gemma 4 gains D512 FFPA attention on SM100 and sliding-window communication improvements. MiniMax-M3 gains SM100 sparse attention with a custom backward pass. Qwen3.5 gains packed vision training with Transformer Engine and MTP with VLM context parallelism. (#3742, #2984, #3809, #4303, #3990)

Speculative Decoding and Discrete-Diffusion LLMs

  • DFlash 2: Draft model, trainer, Qwen3 and Qwen3.8-27B recipes, dynamic convolution, and candidate-path selection. Sampling supports top-p and top-k. (#3605, #3627)
  • Additional draft targets and parallelism: Kimi K3 DSpark, EAGLE-3, and DFlash training; DeepSeek-V4.1 DSpark training; and Ulysses context parallelism for the EAGLE-3 draft. (#3263, #3286, #3287, #3875, #3356)
  • Streaming supervision components: EAGLE-3 feature production and loading, an asynchronous producer, and a safetensors-backed shared-directory feature store with backpressure. (#3085, #3094)
  • I-DLM and SCDD: I-DLM all-masked training uses dllm.mode: idlm; SCDD adds mixed mask/uniform corruption, training, and sampling with dllm.mode: scdd. Neither new mode supports context parallelism. SCDD does not currently maintain the reference method’s EMA weights. (#2437, #3354)

Diffusion

  • HunyuanImage-3.0: Native MoE text-to-image implementation with FSDP2, expert parallelism, LoRA, full fine-tuning, preprocessing, and generation. Recipes cover single-node LoRA and four-node full fine-tuning. TP, PP, CP, image-to-image conditioning, and text-generation training are unsupported. (model and recipe guide, #4113, #4129)
  • Qwen-Image-2.1: Full-parameter and LoRA text-to-image fine-tuning, a dedicated latent/prompt preprocessing path, and generation recipes. Requires a Diffusers build providing QwenImage21Pipeline. (model and recipe guide, #4027)
  • Validation: Periodic diffusion validation loss, retention of validation tail batches, and sample-weighted validation loss. (#3579, #4291)

Retrieval and Embeddings

  • Context-aware reranking: Train rerankers with query-context pairs and configurable prompt augmentation. (#3786)
  • Mistral3 vision-language retrieval: Embedding training, reranking, and multimodal hard-negative mining for text, images, and image-plus-text documents. Exports preserve retrieval prompts, pooling, normalization, attention policy, and scoring temperature; release backports fix portable reranker exports and Pixtral image-attention boundaries. (#3915, #4197, #4267)

Training, Parallelism, and Performance

  • MoK experts: Optional dispatcher: mok integration for fused dispatch and expert computation, plus MXFP8 experts. Supported shapes, precision, topology, and adapter combinations are checked explicitly. (#3422, #3606)
  • Lower-memory DeepSeek-V4-Flash LoRA: MXFP4-resident frozen experts avoid keeping a full BF16 expert copy. This path requires expert parallelism and torch_mm experts with the torch dispatcher; DeepEP experts are unsupported. (#2548)
  • MoE and Kimi K3 efficiency: Compact HybridEP routing, reuse of DeepEP dispatch layouts during activation-checkpoint recomputation, fused gradient norm reduction, pipeline receive-buffer reuse, and opt-in Kimi K3 kernels. (#3404, #3684, #3685, #3780, #3954)
  • Kimi Linear: Adds packed sequences and context parallelism. (#3243)
  • Weighted domain mixtures: Configure sampling frequency separately from each domain’s objective weight for Megatron continual pretraining, with per-domain token metrics and weighted validation. This initial path rejects packing, PP, CP, JSON blends, and other unvalidated combinations. (#3303)
  • Correctness fixes: Expert-parallel gradient clipping, TP replica synchronization, TP knowledge-distillation gradient reduction, per-token parallel cross-entropy gradients, and preservation of model input/output dtypes through FSDP. (#3817, #3822, #3777, #3853, #4150)

Data, Checkpointing, and Extensibility

  • Data paths: SynthTraces agent SFT, Shopify product-catalogue VLM data, typed Datum records and collate-time THD packing, and mask_generation_prompt to exclude generation-prompt tokens from loss. (#3371, #3400, #3514, #3739)
  • Checkpoint loading: DCP loading of standard Hugging Face safetensors, broader low-memory MoE loading, and bounded quantized/GPT-OSS MXFP4 loading. (#3616, #3610, #3619, #3623)
  • PEFT export and resume: Fixes for grouped-expert merged exports, DeepSeek-V4 LoRA reload/resume, Nemotron adapter namespaces and generation configuration, and processor artifacts in merged VLM adapters. (#3834, #4072, #3866, #3891, #3270)
  • Custom architectures: Register an architecture through the public nemo_automodel.register_architecture() API or the nemo_automodel.architectures entry-point group. (#3645)
  • Freezing: freeze_config.freeze_modules and unfreeze_modules accept module selectors and glob patterns. (#3681)

Compatibility and Upgrade Notes

  • Dependency baseline: The release branch pins transformers==5.18.0 and requires torchao>=0.16.0. The optional PyTorch base stage uses NVIDIA PyTorch 26.08-py3. Use the release branch’s dependency files when building an environment; individual model guides may still mention older intermediate Transformers pins. (package dependencies, container definition)
  • Optional experiment tracking: MLflow and Weights & Biases are optional extras. Install nemo-automodel[mlflow] for the tracking client, nemo-automodel[mlflow-full] for the full MLflow stack, or nemo-automodel[wandb] for W&B. (#3785, extra definitions)
  • Custom retrieval loaders: Pass backbone options such as attn_implementation="sdpa" by keyword; positional arguments after the checkpoint path are forwarded to the backbone constructor. (#3915)

Native FlashAttention-4

  • Select the native CuTe backend with model.backend.attn: fa4 in an INSTALL_FA4=true image. On models declaring native FA4 support, model.attn_implementation: flash_attention_4 also selects it when no backend config is supplied. An explicit backend config remains authoritative. HF-dispatched models continue to use the HF flash_attention_4 implementation.
  • Native FA4 accepts BSHD inputs and NEAT packing. Use cp_size: 1 and select NEAT explicitly for packed runs; legacy THD packing is unsupported. Dense attention masks are unsupported. Packed Qwen3Next recurrent attention and DeepSeek V3.2 sparse attention are also unsupported by this backend.
  • Finite native FA4 windows count visible keys including the current token, matching the SDPA window convention. The pinned SM100/SM110 FA4 kernel does not support local attention with 256-wide query/key and value heads; select model.backend.attn: sdpa for that combination.

NEAT packing API migration

  • Packing now follows the built model’s attention dispatch. The legacy YAML selector packed_sequence.attn_implementation is ignored with a warning.
  • Replace get_attn_implementation(cfg_model, model) with get_model_attn_implementation(model) from nemo_automodel.components.models.common.packing. For multiple local pipeline stages, use configure_packing_for_models(model_parts).
  • Import get_unpad_data and get_seqlens_in_batch from nemo_automodel.components.datasets.packing. HF flash packing uses explicit public varlen kwargs without modifying Transformers globals. Pass the contract returned by configure_packing as packing= to the collater or packing_contract= to the dataloader config’s build().
  • VlmDataloaderConfig.resolve_packing_attn_implementation was removed. Custom integrations should configure the built model and supply its packing contract to VlmDataloaderConfig.build() instead of selecting a backend in the dataset.
  • configure_packing now requires an attention implementation and returns a packing contract. Qwen3.5, Qwen3.5-MoE, Kimi Linear, and Kimi K3 require the resulting dataset metadata alongside indexed masks. Custom loops must pass the contract to their collater and forward its metadata; the legacy attn_implementation= collater argument cannot declare model-specific metadata requirements.
  • Qwen3.5 packing with load-balanced context parallelism now raises instead of silently losing document isolation. Use cp_size: 1, or the supported SDPA block-diagonal CP path for Qwen3.5-MoE.

External Contributors

Thank you to the following community contributors for the code, documentation, tests, and fixes included in 26.10, and to everyone who reported issues, reviewed changes, and helped validate the release.

  • @akx — expert-parallel collective deadlock fixes, Kimi-K2.5 checkpoint and quantization fixes, lint updates, and distributed-filesystem checkpoint handling. (#3631, #3632, #3633, #3686)
  • @Anai-Guo — removing a shadowed Kimi-K2.5-VL forward-test class. (#3873)
  • @andrewwhitecdw — removing an unused CI summary environment variable. (#3536)
  • @beccohov — typed packing metadata in place of the global Hugging Face monkeypatch, and periodic diffusion validation loss. (#3577, #3579)
  • @Butterfingrz — Gemma 4 D512 FFPA attention on SM100 and MiniMax-M3 SM100 sparse attention with a custom backward pass. (#3742, #3809)
  • @coderaBruce — preserving fused losses through MiniMax-M3-VL’s logits_to_keep contract. (#3511)
  • @coding-famer — saving processor artifacts when merging VLM LoRA adapters. (#3270)
  • @dundysm — accepting the singular embedding-key alias in Nemotron V3 Hugging Face checkpoints. (#3999)
  • @edjson — preserving FP32 routing weights in DeepSeek and Ling models. (#3746, #3861)
  • Daniel Shen (@excepshenal, @dshen-crusoe) — MXFP4-resident MoE experts for DeepSeek-V4-Flash LoRA. (#2548)
  • @Faisal-Alsrheed — correcting the training dataloader annotation and modernizing optional types. (#3534)
  • @haowen-han — preserving FP32 rotary frequencies in the Qwen3.5 vision tower. (#3856)
  • @HuayiJin — keeping Qwen3.8 Flash Next flex-QSA membership lookup in int64. (#3802)
  • @kabirvashisht4-glitch — documentation corrections and fixes to dataset logging and collation, checkpoint sizing, sequence-classification loss, InfoNCE validation, scheduler resume, launcher overrides, and the torchao dependency floor. (#3762, #3830, #3883, #4131)
  • @kashif — I-DLM and SCDD training, streaming EAGLE-3 features, DFlash 2 training and sampling, DeepSeek-V4.1 DSpark support, and DiffusionGemma documentation. (#2437, #3094, #3354, #3605, #3875)
  • @khazic — speculative-decoding training, correctness fixes and documentation; Kimi Linear packing and context parallelism; SynthTraces data support; DeepSeek-V4.1-Flash training; and HunyuanImage-3.0 fine-tuning with expert parallelism. (#3243, #3286, #3356, #3855, #4129)
  • @linmuchuiyang — selective activation-checkpoint replay fixes, DeepSeek-V4.1 H100 recipes and data preparation, and the CP8 gradient-synchronization fix backported into this release. (#3961, #3962, #4203, #4221)
  • @mikemikimike — preventing tensor-parallel knowledge-distillation gradient over-reduction. (#3777)
  • @MohammedAlkindi — documentation and coverage-link repairs, UTF-8 configuration loading, multi-node SkyPilot rendezvous, SQuAD handling, PEFT compatibility and wildcard fixes, validation-loss reduction, and optimizer-group scheduling. (#3769, #4037, #4043, #4045, #4050)
  • @MrCapricornLiu — Mistral tokenizers in the distillation collator, InfoNCE masking order, and per-token gradients in parallel cross-entropy. (#3849, #3850, #3853)
  • @nathan-az — Gemma 4 sliding-window attention communication and FlashAttention improvements. (#2984)
  • @piyushumate — making MLflow and Weights & Biases optional dependencies. (#3785)
  • @ralovets — base-weight loading with DDP and Megatron FSDP, RTX PRO 6000 MFU recognition, and multi-source dataset blending fixes. (#3597, #3598, #3919)
  • @rouchenzi — keeping Qwen3.8 Flash Next PLE hashing and convolution within packed document boundaries. (#4132)
  • @sahel-sh — configurable metric buffering, retaining retrieval metrics on preemption or error, and context-aware reranker training. (#3541, #3583, #3786)
  • @Simar-malhotra09 and credited co-author Matilda-Coder — MiMo-V2.5-Pro model support. (#2514)
  • @stanley1208 — FP8 loading checks, dense Nemotron-H and encoder-decoder support, PEFT checkpoint naming and export fixes, expert-parallel gradient clipping, generation-config export, and shared PEFT reload tests. (#2350, #2670, #2860, #3817, #3945)
  • @taran-dev4u — VLM processor serialization tests and Nemotron Omni PEFT save/reload key conversion. (#3379, #3986)
  • @waynel96 — GLM-5.3-Flash installation and parity documentation, KDA execution fixes, and the GLM5 Next training chat template. (#3939, #4077, #4110, #4165)
  • @yh0903 — avoiding retained routed inputs in chunked MoE checkpoints. (#4103)
  • @yuvalnissan — honoring the checkpoint’s GLM indexer rotary layout. (#4142)
  • @ZenAlexa — advancing the DeepSeek-V4 pipeline media cursor for text-only microbatches. (#3821)
  • @zeryx — the Shopify product-catalogue VLM dataset and Cohere Micro Vision model support. (#3400, #3518)

0.6.0 · 26.08 (2026-08-26) · PyPI · GH · NGC Docker

Highlights

  • Frontier-scale MoE and multimodal models. Train Moonshot AI’s Kimi K3 (2.8T/104B active), Thinking Machines Lab’s Inkling (975B/41B active), MiniMax AI’s MiniMax-M3 VL (428B/22B active), Poolside’s Laguna, and Zhipu AI’s GLM-5.2.

  • Long-context training. Shard packed multi-document sequences with block-diagonal variable-length context parallelism, and shard vision towers by frame to remove the redundant vision compute of a CP fold; recipes reach 128K tokens at EP8/CP32.

  • New attention and kernel backends. Select FlashAttention 3 and 4, MagiAttention, FFPA for headdim=512 layers, QuACK linear/RMSNorm/RoPE kernels, and scoped partial CUDA graphs for MoE.

  • Resilient checkpointing. Checkpoint on preemption signals, write to msc:// cloud directories, consolidate async saves in the background, bound checkpoint retention, and resume past an interrupted save.

  • Diffusion. Fine-tune and generate with LTX-2.3 joint video+audio and Qwen-Image-Edit-2511, and run context parallelism through the diffusers ContextParallelConfig API.

New Hardware and Precision Support

  • FP8 draft training. Extends the SFT recipes’ top-level fp8: block to every speculative recipe, swapping the draft’s nn.Linear layers to torchao Float8Linear on SM89 and newer, with emulate: true for older GPUs. (#2963)

  • Quantized checkpoint ingestion. Dequantizes MXFP4 dequantization for Kimi K3 and MXFP8 dequantization for MiniMax-M3 for BF16 training, keeps Kimi-K2.5-VL LoRA keys out of INT4 quantization on save, and fixes a TileLang boolx8 backward codegen failure on the DeepSeek-V4 path. (#3259, #3295, #3467)

New and Expanded Model Support

LLM and MoE

VLM

  • MiniMax AI: MiniMax-M3 VL (428B / 22B-active MoE) — MedPix full SFT at EP32/PP4 on 16 nodes, MedPix LoRA at PP4/EP8, a CP2 comparison recipe, and 16K packed Tulu-3 text SFT at CP8. (docs, #2538, #2551)

  • Thinking Machines Lab: Inkling (975B / 41B-active MoE; text, image, video, audio) — MedPix SFT at PP8/EP32 on 32 nodes. (docs, #3095, #3281, #3358)

  • Alibaba: Qwen3.5 and Qwen3.6 VLM (122B-A10B MoE, 27B dense) — 128K packed sequences at EP8/CP32 on 16 nodes with a trainable vision tower, dense context parallelism, and matched CP1/CP2 27B MedPix recipes. (docs, #2505, #3186, #3209)

  • Google: Gemma 4 (31B and E2B/E4B dense, 26B-A4B MoE) — context parallelism with 16K Tulu-3 at CP8, 64K Tulu-3 at CP16, and 4K MedPix at EP8/CP2; plus fp32 SDPA to avoid NaNs on Hopper and an exact fp32 reference router for MoE parity. (docs, #1914, #2592, #2621, #3141)

  • Moonshot AI: Kimi-K2.5 VL (MoE) — LoRA tensors kept out of INT4 expert quantization on save, so adapters reload on resume. (#3295, #3431)

  • NVIDIA: Nemotron-Parse — corrected preprocessing stopping the RADIO encoder from normalizing already-normalized images a second time. (#3331)

  • MTP on multimodal inputs. Pre-rolls pre-fused inputs_embeds per depth so audio and vision positions keep their continuous embedding instead of being re-embedded from a padding token id. (#2510)

  • Pre-extracted video frames. Loads a video field holding a list of image paths directly as a frame sequence with no decoder, padded to the processor’s temporal_patch_size. (#3211)

Diffusion

  • Lightricks: LTX-2.3 (video + audio) — full-finetuning, LoRA, and generation recipes that train video and audio jointly and mux the generated waveform into the output mp4; --processor ltx2 preprocessing requires an 8n+1 frame count. (docs, #3165, #3372)

  • Alibaba: Qwen-Image-Edit-2511 — cached, full-parameter instruction-based editing with an offline latent and prompt cache encoder, an eight-GPU BF16 recipe, and an image-edit preprocessing subcommand. (docs, #3217)

  • Diffusion context parallelism. Drives the diffusers ContextParallelConfig API from fsdp.cp_size, reusing the cp axis of the existing FSDP2 mesh; only pure Ulysses sharding is accepted. (#3157)

dLLM

  • Google: DiffusionGemma 26B-A4B (block-diffusion MoE) — a native implementation with GSM8K SFT and LoRA recipes at ep_size: 8. (docs, #2506)

  • Generation and LoRA. Exposes --sampler presets llada, llada2, nemotron, and gemma, an --adapter merge path, and llada_lora, llada2_lora, and nemotron_labs_diffusion_lora recipes. (#3092, #3161, #3163)

  • Reproducible corruption. Seeds corruption noise per sample from that example’s global index in the shuffled stream, so noise is independent of parallel topology and resume reproduces exactly. (#3162)

Speculative Decoding

  • DSpark. Introduces a semi-autoregressive draft in which a parallel backbone proposes a whole block, a serial Markov head adds intra-block dependency, and a confidence head predicts acceptance, with configs for Qwen3, Gemma 4, GLM-5.2, MiniMax-M3, and DeepSeek-V4-Flash. (docs, #2810, #2866, #2885, #2909)

  • Domino, JetSpec, and ViSpec. Adds a GRU correction head on the DFlash backbone (Domino), causal in-block attention distilled with a temperature-scaled forward KL (JetSpec), and two-stage VLM draft training (ViSpec). (#2819, #2867, #3173)

  • New EAGLE-3 targets. Registers a DeepSeek-V3 MLA draft class reusing the target’s low-rank q/kv projections, and registers Gemma 4 as a target with E2B, E4B, 31B, and 26B-A4B configs. (docs, #2849, #3071, #3077, #3079)

  • Sequence packing, TP, and CP. Enables block-causal packed training for every draft family, target tensor parallelism for EAGLE-1/2/3 and DFlash, and context parallelism for both target and draft through a differentiable ring attention. (#2827, #2918, #3002, #3005)

  • fp8, compile, LoRA, and new objectives. Exposes a compile: block, EAGLE-3 adapter-only training through peft:, DFlash loss_type: variable_prefix, and EAGLE-3 lk_loss_type: alpha/lambda. (#2963, #3032, #3081)

  • Feature-noise augmentation. Perturbs the target features fed to the EAGLE-1/2 draft with uniform noise on the target features fed to the EAGLE-1/2 draft through recipe_args.feature_noise, defaulting to the paper value 0.1. (#2470)

  • On-policy regeneration. Interleaves EAGLE-3 training with online target regeneration through a recipe_args.regen block that interleaves EAGLE-3 training with online target regeneration on a reserved GPU, hot-swapping the dataloader across all ranks in lockstep. (#3042)

  • Serving, benchmarking, and caches. Covers sglang and vllm target backends for training supervision, serve_vllm for trained EAGLE-3, P-EAGLE, and DFlash-family drafts, bench_vllm/bench_sweep acceptance and speedup benchmarks, and distributed DSpark offline precompute for targets too large for one node. (#2798, #2841, #2912, #2953)

  • Metrics and resume. Reports simulated accept length, per-position accept_rate@k, and periodic real accept length through decode_eval, and rejects a mismatched mask_token_id, a stale draft-vocab mapping, and mixed precompute shards on resume. (#2959, #3037, #3237, #3238)

Training, Parallelism, and Performance

  • Block-diagonal variable-length context parallelism. Introduces a blockdiag_cp implementation that shards packed multi-document sequences while keeping masking block-causal per document, driving varlen kernels from precomputed cu_seqlens with a fused K/V all-gather. (#2989, #3223)

  • Unified context-parallel input preparation. Returns a ContextParallelSharder from a single prepare_model_inputs_for_cp hook and replaces cp_utils with components.distributed.context_parallel. (#2937)

  • Frame-level vision-tower sharding. Partitions frame units across the CP group through distributed.multimodal.vision.frame_sharding and reassembles embeddings with a differentiable variable-length all-gather. (docs, #2990, #3186)

  • Attention backends. Selects flash_attention_3 and flash_attention_4 with a fa4 → fa3 → fa2 → sdpa → eager fallback ladder, MagiAttention as both a kernel and a context-parallel backend with CP1/CP2 Qwen3-MoE-30B packed recipes, and a CuTeDSL FFPA backend for Gemma 4’s head_dim=512 layers (8K, packed CP8 16K). (#2384, #2929, #3070)

  • QuACK kernels and partial CUDA graphs. Exposes quack options for backend.linear, backend.rms_norm, and backend.rope, and backend.cuda_graph.modules capture of attn, moe_router, and moe_preprocess while dispatch, expert compute, and combine stay eager. (#2917, #3115)

  • Activation checkpointing. Accepts activation_checkpointing: selective for DDP and an activation_checkpointing_scope selecting which layer groups are wrapped, and saves the MoE router and its top-k output so recompute preserves expert assignments. (#2786, #2840, #3140, #3247)

  • Cross-entropy memory. Supports FusedLinearCrossEntropy under pipeline parallelism and bounds ChunkedCrossEntropy memory with a kernel that saves only original-dtype logits and recomputes each softmax chunk in backward. (#2927, #2996)

  • FSDP2 Wraps HF transformer layers with PyTorch’s non-reentrant checkpoint_wrapper to remove duplicate prefetch all-gathers, and adds distributed.multimodal.frozen_sharding for fully frozen towers. (#3328, #3411, #3513)

  • Gradient correctness. Normalizes the MoE auxiliary loss by the gradient-accumulation and pipeline-microbatch count, all-reduces grad-norm accumulators on the mesh device, and preserves HSDP replica gradient synchronization for MoE. (#3135, #3359, #3461)

  • Damaged-embedding repair. Replaces, through an optional embedding_row_repair: section, input-embedding rows whose L2 norm is non-finite or below min_norm with the scaled output-embedding direction; it is rejected under pipeline parallelism. (docs, #3136)

  • Setup-time prewarms and mesh timeouts. Initializes lazily created resources through an opt-in prewarm: section covering cublas_backward, fla_gdn_autotune, mamba_ssd_autotune, and comm_groups, and applies dist_env.timeout_minutes to flattened and expert-parallel meshes. (#2846, #2992, #3296)

  • Rollout Routing Replay. Reuses the rollout’s discrete top-k expert selection in the training forward through MoEConfig.enable_routing_replay, which reuses the rollout’s discrete top-k expert selection in the training forward to remove the rollout/training routing mismatch in on-policy RL. (#2797)

  • Optimizer parameter groups and throughput. Adds optimizer.param_group_overrides with lr_mult and wd_mult multipliers, removes Python overhead from Transformer Engine attention and MoE expert hot paths, and shapes TPLinear/LinearLoRA graphs so async-TP fusion fires. (#2987, #3046, #3374)

  • PEFT under pipeline and expert parallelism. Gathers adapter weights across pipeline stages before writing and saves optimizer state for every PEFT model part namespaced by stage. (#3096, #3250, #3316)

  • PEFT v5 expert adapters. Exports MoE expert LoRA through an opt-in PEFT v0.18+ ParamWrapper export for MoE expert LoRA with fused target_parameters, validated on Qwen3-MoE, MiniMax-M2, and Nemotron v3. (#3439, #3458)

  • LoRA correctness. Makes use_memory_efficient_lora: false also suppress the fused SwiGLU/ReLU² LoRA MLP, restores adapters through DDP wrappers, and reaches the absorbed GLM MLA KV weight through a new materialize_effective_weight(). (#3126, #3150, #3176, #3470)

Checkpointing

  • Preemption checkpointing. Introduces step_scheduler.preemption_signal, which watches one or more signals (default SIGTERM), gathers the flag across ranks, and saves at the next step boundary before exiting cleanly. (docs, #3007)

  • Cloud checkpoint directories. Accepts msc:// paths for sharded DCP state; cloud roots require save_consolidated: false, reject max_recent_checkpoints, and keep RNG, dataloader, and pointer files local. (#1709)

  • Retention and interrupted saves. Prunes older directories through max_recent_checkpoints, and marks in-progress directories until every component is published so resume falls back instead of hanging all ranks. (#2416, #3261)

  • Async consolidation and new knobs. Runs consolidated safetensors export through the same all-rank consolidation as synchronous mode on a background thread, and adds cpu_offload, wait_for_staging, and consolidation_timeout_minutes. (#3108, #3125, #3130, #3131)

  • torchsave export to Hugging Face. Adds scripts/export_llm_dcp_to_hf.py, which rebuilds the original topology from the checkpoint’s recorded config.yaml and dispatches the LLM or VLM recipe named there, so a VLM checkpoint keeps its vision tower. (#2487)

  • Export fidelity and safety. Keeps fp32 parameters fp32 even under an explicit cast_dtype, writes safetensors without append, retains later-stage keys in pipeline-parallel consolidation, and loads auxiliary torch.save state with weights_only=True. (#2627, #3193, #3240, #3248)

  • Per-global-rank RNG state. Saves and restores RNG state per global rank rather than per data-parallel rank, so tensor- and pipeline-parallel peers keep distinct streams across a resume. (#3437)

  • Tied-embedding guards. Declares a TieSupport policy per model class and rejects an unsupported tie_word_embeddings value. (#2805, #2896, #2998)

Data

  • Typed dataset configs. Gives every shipped dataset a typed *Config with a build method and resolves dataset: and dataloader: into a single DataloaderConfig; legacy _target_ values still resolve through a compatibility registry. (#2390)

  • Prefix-tree attention for rollouts. Folds, through prefix_tree_collate_fn, one shared prompt and its completions into a single deduplicated sequence with a block-sparse mask so the prompt is encoded once. (#2564)

  • Long-context agent-SFT data pipeline. Tokenizes CoderForge trajectories once in a dedicated stage and drops over-length ones rather than truncating, with a paired Gemma-4-31B recipe at seq_length: 65536 and cp_size: 8. (#3151, #3446)

  • Chat-template masking and pre-tokenization. Validates prefix-built answer-only masks against the full-conversation render, keeps system turns in the ShareGPT converter, and adds packed_sequence.num_proc for parallel pre-tokenization before packing. (#3024, #3045, #3144)

Embedding and Re-ranker

  • Normalized Arrow retrieval datasets. Reads, through NormalizedRetrievalDatasetConfig, a portable Arrow bundle that stores each referenced document or image once, with a CPU-side preparation script and Slurm wrappers. (docs, #2596)

  • NVIDIA: Nemotron VL 1B. Adds a native llama_nemotron_vl model and processor for nvidia/llama-nemotron-embed-vl-1b-v2, trained as a vision bi-encoder with a bi-encoder recipe and a ColPali conversion notebook. (docs, #2354)

  • Cross-encoder re-ranking. use_text_in_document applies to the cross-encoder transform as well as the bi-encoder, so a re-ranker can score an image document together with its text. (#3189)

  • Embedding distillation. Distills a teacher embedding model into a student with EmbeddingDistillRecipe and an 8-GPU example, mixing cosine, MSE, and listwise InfoNCE terms with support for intermediate-layer distillation and cross-tokenizer cached teachers. (#3058)

  • Ministral 3 recipes. Loads the stock bidirectional Ministral backbone instead of a bespoke extraction, repairs the bi-encoder recipe, and guards the optional Weights & Biases import so the retrieval recipes import without it. (#3103, #3380, #3441)

  • Sentence Transformers metadata. Writes modules.json, 1_Pooling/config.json, and config_sentence_transformers.json alongside consolidated bi-encoder exports, which the hard-negative miner reads back for pooling and normalization. (#3464)

Distillation

  • Distillation on separate meshes. Places the teacher on its own ranks through separate_meshes: true and a teacher_distributed: block, with Llama-3.2 1B/3B and Qwen3.5 4B/9B examples under TP2, CP2, PP2, or DP2 teacher meshes. (#2954, #3280)

  • Distillation correctness. Resolves fp32 master weights for the student, runs gradient clipping on the recipe’s own mesh with the expert TP replication factor, and passes tensor-valued hidden states through unchanged. (#3019, #3302, #3324)

Packaging and Dependencies

  • Framework pins. Moves transformers from 5.8.1 to 5.12.1 and megatron-fsdp from 0.2.3 to a pinned 0.5.0 resolved from a Megatron-LM fork commit; both force an environment rebuild. (#2873)

  • QuACK is a default dependency. Promotes quack-kernels==0.6.1 to the base dependencies on Linux, which pulls nvidia-cutlass-dsl, apache-tvm-ffi, and torch-c-dlpack-ext into every Linux install. (#3115)

  • New optional dependency sets. Adds an ffpa extra with ffpa-attn and nvidia-cutlass-dsl, and ships MagiAttention as a uv dependency group installed with uv sync --group magi; neither is part of [all]. (#2436, #3070, #3225)

  • Container. Builds FlashAttention 3 from source on x86 by default and leaves FlashAttention 4 opt-in behind INSTALL_FA4=true, moves the optional PyTorch base stage to pytorch:26.06-py3, and moves the deploy image to vllm:26.04-py3. (#2929, #2983, #3061, #3329)

  • Security constraints. Advances aiohttp, cryptography, gitpython, pillow, starlette, mlflow, and thrift, and adds pyarrow and ray constraints. (#3346, #3398, #3482, #3523)

Media dependencies

Media packages remain opt-in. The vlm-media extra adds albumentations for the Nemotron-Parse processor and diffusion-media adds PyAV for LTX-2 audio decoding; the diffusion extra now requires diffusers>=0.39.0:

uv pip install 'nemo-automodel[media]==0.6.0'

Use nemo-automodel[vlm-media] or nemo-automodel[diffusion-media] when only one of those media stacks is required.

Breaking Changes

  • tie_word_embeddings is validated per model family and raises instead of leaving a randomly initialized lm_head. (#2805, #2896, #2998)

  • Tokenizer BOS/EOS insertion is opt-in; set add_bos_token / add_eos_token to keep the previous tokenization. (#3337)

  • Multi-turn masking raises on chat templates that rewrite earlier turns; affected templates must carry {% generation %} blocks. (#3024, #3051)

  • Unrecognized keys under dataloader: or on a typed dataset config now raise instead of being silently forwarded. (#2390)

  • DFlash requires an explicit mask_token_id, and loss_decay_gamma defaults to 7.0. (#2461, #2908)

  • BaichuanForCausalLM, Qwen2ForCausalLM, and KimiVLForConditionalGeneration are deprecated and scheduled for removal in 26.10. (#2807, #2884)

External Contributors

This release includes work from the following contributors outside the NVIDIA-NeMo organization. Thank you.

  • @khazic — most of the speculative-decoding stack in this release: the DSpark, Domino, and JetSpec drafts, ViSpec VLM draft training, sequence packing and target tensor/context parallelism across every draft family, the vLLM and SGLang target backends, serving and acceptance benchmarks, and Gemma 4 ring context parallelism (94 PRs). (#2449, #2810, #2819, #2867, #3173)

  • @Achyuthan-S — the TieSupport policy and tie_word_embeddings guards across the registered model families. (#2732, #2805, #2896, #2998)

  • @kashif — the DSpark draft model and training objective, context parallelism for draft training, and DSpark acceptance and confidence metrics. (#2810, #2918, #2957)

  • @edjson — MSC cloud-storage support for DCP checkpoints and preemption checkpointing. (#1709, #3007)

  • @Butterfingrz — the FFPA headdim=512 attention backend for Gemma 4. (#2436)

  • @beccohov — FusedLinearCrossEntropy support under pipeline parallelism. (#2927)

  • @fkuner — the DSpark offline target-cache path. (#2924)

  • @GITsologun — pre-extracted video frame sequences in the VLM data path. (#3211)

  • @hyfine — gathering PEFT adapters across pipeline stages before save. (#3096)

  • @wangzhxg — preserving HSDP replica gradient synchronization for MoE models. (#3135)

  • @shahafwa — applying LoRA to the TileLang MLA KV projection. (#3176)

  • @amolkhanna — activation checkpointing for Qwen3-Next linear-attention layers. (#3192)

  • @aminehd — running Gemma 4 SDPA in fp32 to avoid NaNs on Hopper. (#3141)

  • @huahuajhu — writing consolidated safetensors without append. (#2627)

  • @zhiqi-li — omitting the unset FSDP reshard argument. (#2926)

  • @DOGEUNNKIM — casting Gemma 4 dense parameters without casting buffers. (#2359)

  • @grgkovac — per-validation-dataset logging in Weights & Biases. (#2526)

Known Issues

  • Transformer Engine fused RoPE is force-disabled globally and overrides an explicit rope_fusion: true. (#3028)

  • Inter-node DeepEP dispatch still faults and deepep remains the default whenever DeepEP is importable; use HybridEP for inter-node EP.

  • FlashAttention 4 and the TileLang backend cannot coexist in one image; an INSTALL_FA4=true build loses the TileLang backend used by DeepSeek-V4 and GLM MoE DSA. (#2929)

  • P-EAGLE rejects a remote target backend, a cached target path, and packed sequences. (#2466)

  • Qwen-Image-Edit-2511 training is limited to cached latents on a single node, and diffusion context parallelism accepts only pure Ulysses sharding. (#3157, #3217)

0.5.0 · 26.06 · PyPI · GH · NGC Docker

Highlights

  • Agent SFT. Train Qwen2.5-3B on multi-turn, function-calling ChatML or ShareGPT data with full-SFT and LoRA recipes. The data adapter preserves tool calls, tool responses, and optional reasoning_content.
  • Speculative decoding. Train EAGLE-1/2/3/3.1, P-EAGLE, and DFlash drafts for Llama, Phi, Qwen3, Qwen3-MoE, GPT-OSS, and Nemotron Nano targets.
  • Discrete-diffusion LLMs (dLLMs). Train DFlash drafts and fine-tune LLaDA2 or Nemotron-Labs-Diffusion with the new discrete-diffusion workflows.
  • Diffusion. Fine-tune and generate with FLUX.2-dev, Qwen-Image, and Wan 2.2 T2V-A14B; updated FLUX, Hunyuan, and Wan 2.1 recipes add LoRA and throughput improvements.
  • Large-scale MoE. Run DeepSeek-V4 with TileLang kernels, train MoE models with MXFP8 on GB200, and use context parallelism for Qwen3.5/3.6 MoE VLMs.

New and Expanded Model Support

LLM and MoE

  • DeepSeek-V4 Flash: A native model and checkpoint adapter; HellaSwag full-SFT, LoRA, and packed-sequence recipes; and large-scale TileLang PP/EP recipes. The implementation covers hash routing, hyper-connections, sparse/compressed attention, FP4/FP8 checkpoint loading, and multi-token prediction.
  • ERNIE 4.5: HellaSwag recipes for the 0.3B dense and 21B-A3B MoE checkpoints.
  • MiMo-V2-Flash: A HellaSwag fine-tuning recipe and native model implementation with checkpoint conversion.
  • Ling 2.0: Mini, Flash, and 1T model paths with SFT, LoRA, HellaSwag, SQuAD, and pipeline-parallel recipes.
  • Hy3-preview: DeepEP full-SFT and LoRA recipes for the 295B MoE model, with model-specific expert-bias handling.
  • HY-MT2-30B-A3B: Native model support and a supervised fine-tuning recipe.
  • MiniMax-M2.7: Pipeline-parallel full-SFT and LoRA recipes.
  • Falcon H1: Full-SFT recipes for 0.5B, 1.5B, and 7B checkpoints, plus 7B and 34B PEFT recipes.
  • MiniCPM5-1B: Full-SFT and PEFT recipes.
  • Nemotron-3-Ultra-550B-A55B: A 16-node/128-GPU throughput benchmark and a 32-node/256-GPU full-SFT recipe using a real router, THD sequence packing, and a repeated MTP head.
  • Qwen3.5/3.6 dense and MoE: MTP support and a native dense Qwen3.5 backbone that keeps its SSM-gate parameters in fp32.

Vision-Language Models

  • Qwen3.6 VLM: 27B dense and 35B-A3B SFT/LoRA recipes, including a 4K MedPix EP8/CP2 recipe for the MoE model.
  • Mistral Medium 3.5: A 128B MedPix full fine-tuning recipe and matching LoRA recipe.
  • LLaVA-OneVision 1.5: A 4B full fine-tuning recipe and an 8B LoRA recipe.
  • VLM QLoRA: The existing quantization: configuration block now works with VLM fine-tuning, including BitsAndBytes 4-bit NF4 base weights with LoRA adapters.
  • VLM knowledge distillation: A Qwen3.5-VL 9B-to-4B KD recipe with a chunked KD loss, frozen vision/audio towers, and separate CE/KD metrics.
  • Step-3.7-Flash: Native support and MedPix SFT/LoRA recipes using EP and PP. The full-SFT recipe targets 16 nodes/128 GPUs; the LoRA recipe targets 8 nodes/64 GPUs.

Retrieval and Embeddings

  • Late-interaction retrieval: Multi-vector/MaxSim training for ColBERT-style encoders, including distributed in-batch negatives.
  • Bi-encoder positives: Cycling over every positive passage instead of choosing only one positive document per query.
  • Embedding and reranker recipes: A Ministral3 3B bi-encoder recipe and model-coverage pages for bidirectional Ministral and Llama models.

Agent SFT and Speculative Decoding

Agent SFT

  • The agent-chat data adapter accepts ChatML messages and ShareGPT conversations, merges consecutive tool calls, pairs tool responses to call IDs, and supervises assistant content and tool-call arguments.
  • train_on_last_turn_only masks every earlier assistant response and trains only on the final assistant turn.
  • reasoning_content is preserved during data conversion; mask_reasoning_content can include the trace in the prompt while excluding it from loss.
  • Generation-based tool-call evaluation reports call presence, tool-name accuracy, JSON validity, argument precision/recall, and exact argument match.

Speculative Decoding

  • EAGLE-1/2 recipes support Llama, Phi, Qwen3, and Qwen3-MoE targets. EAGLE-3 adds GPT-OSS, and EAGLE-3.1 recipes cover Llama targets.
  • A remote target server/client path can train a draft model while target supervision runs on a separate GPU.
  • P-EAGLE sequence_partitions splits a long-context draft step into loss-equivalent segments to reduce activation memory.
  • Packed variable-length training uses block-causal attention and controls to keep, save, or mask target-model reasoning traces.
  • Performance tools include SGLang acceptance/speedup benchmarking, offline target-output caching, and a fused Triton soft-target cross-entropy kernel that avoids allocating a full fp32 log-probability tensor.

Discrete-Diffusion LLMs

  • DFlash: A draft-training strategy that distills frozen target hidden states with a position-decay loss, with Nemotron Nano 30B and Qwen3-4B example recipes.
  • LLaDA2: A discrete-diffusion SFT recipe alongside the existing LLaDA workflow.
  • Nemotron-Labs-Diffusion: A hybrid diffusion-LLM SFT recipe.

Diffusion

  • FLUX.2-dev: A model adapter, dataset processor, full fine-tuning recipe, LoRA recipe, and generation configuration for its 4D positional-ID and Mistral3 text-embedding architecture.
  • Qwen-Image: Model and processor support plus full fine-tuning, pretraining, generation, and LoRA recipes. The LoRA recipe targets attention and image/text MLP layers; the data collator pads variable-length cached prompt embeddings.
  • Wan 2.2 T2V-A14B: Preprocessing, high-noise and low-noise stage fine-tuning, and generation that loads independently trained checkpoints into the matching denoisers.
  • FLUX, Hunyuan, and Wan 2.1: LoRA recipes for each workflow, a multi-node Wan 2.1 recipe, and tuned full-SFT/LoRA settings including compile and FSDP reduction-dtype controls.

Omni and Multimodal

  • Bagel: Model, dataset, pretraining and fine-tuning recipes, EMA support, checkpoint adapter, and distributed initialization. The one-node examples use token budgets validated for parity runs.
  • Qwen2.5-Omni ASR: AMI and multi-language SFT recipes for the 3B and 7B checkpoints.
  • Qwen3-Omni ASR: AMI and multi-language audio SFT recipes.
  • Nemotron-Omni vision: A dynamic-resolution image path aligned with vLLM image preprocessing, preventing training/rollout vision-embedding divergence.
  • Nemotron-Omni parallelism: Activation checkpointing for the nested language model and an EP8/CP2 VLM recipe that prepares and shards multimodal embeddings across the context mesh.

Gemma 4

  • TP/PP VLM support: TP4/PP2 and TP4/PP4 recipes for Gemma 4 31B, including pipeline-stage handling for image position IDs.
  • LoRA coverage: PEFT recipes for the 2B, 4B, 31B, and 26B-A4B MoE Gemma 4 VLM models.
  • Joint drafter: A Gemma 4 joint-drafter model, MedPix recipe, and MTP inference benchmark.

Training, Parallelism, and Performance

  • Selective FSDP2 activation checkpointing: distributed.activation_checkpointing: selective saves attention and communication operations while alternating save/recompute for matrix multiplies and grouped MoE GEMMs. It supports dense models, expert-parallel MoE, single-GPU runs, and torch.compile.
  • Memory-efficient LoRA: use_memory_efficient_lora reduces adapter training memory requirements.
  • Dion-family optimizers: Typed recipe configurations for Dion, Dion2, Muon, and NorMuon, plus a Qwen2.5-7B Muon fine-tuning recipe and corrected FSDP mesh alignment.
  • Qwen3.5/3.6 VLM context parallelism: Image/video embeddings and mRoPE positions are prepared before sequence sharding; the dense sequence index is passed to linear attention, with a CP-aware validation denominator.
  • Nemotron 3 parallelism: Context and pipeline parallelism plus MTP-related train-loop support.
  • GB200 MXFP8 MoE: Transformer Engine MXFP8 grouped-expert training for Qwen3-MoE-30B, GPT-OSS-120B, and Qwen3-MoE-235B; the bf16 expert backend now uses Transformer Engine grouped linear layers.
  • Cross-entropy kernels: Fused linear cross-entropy for custom models and a CUDA Triton implementation of EAGLE soft-target cross-entropy.
  • DeepSeek-V4 kernels: TileLang implementations for sparse MLA, the lightning indexer, and MHC Sinkhorn. The CUDA environment installs TileLang and TileKernels for the DeepSeek-V4 recipes.
  • Large-checkpoint loading: Memory-mapped local HF DCP reads let each rank map only its requested tensor slice instead of copying full tensors into host RAM.

Data, Recipes, and Operations

  • Added S3 and multi-storage-client object sources to the Megatron pretraining dataset path.
  • data_dir_list accepts [num_samples, path] entries for deterministic per-source sampling before concatenation while retaining plain-path entries.
  • Added flat-list blend JSON files and lazy dataset preprocessing.
  • Retrieval runtime tuning exposes DDP bucket size, static graph, buffer broadcast, unused-parameter, and gradient-bucket-view controls; FSDP2 reshard_after_forward; and averaged retrieval-loss logging.
  • Added runnable examples for agent SFT, EAGLE/DFlash, dLLM, diffusion, ASR, retrieval, DeepSeek-V4, Gemma 4, Qwen3.6, Nemotron Ultra, and Bagel.
  • Added NeMo-Run as a managed job launcher alongside local, Slurm, SkyPilot, and Kubernetes workflows.

Packaging and Media Dependencies

  • The package is classified as Production/Stable, pins Transformers 5.8.1, requires Transformer Engine 2.14.1 or newer, and resolves Linux PyTorch packages from the CUDA 13.0 index.

  • Video and media packages are no longer installed by default through the base, VLM, diffusion, or all extras. Install the media union when a workflow needs OpenCV, decord, Qwen media utilities, or FFmpeg:

    uv venv
    source .venv/bin/activate
    uv pip install "nemo-automodel[media]"

    Use nemo-automodel[vlm-media] or nemo-automodel[diffusion-media] when only one media stack is required.

Breaking Changes

FSDP2 now defaults gradient reduction to float32 while forward and backward compute remain in bfloat16. See Breaking Changes for the compatibility override and migration guidance.

0.4.0 · 26.04 (2026-04-28) · PyPI · GH · NGC Docker

Highlights

  • Discrete-diffusion LLMs (dLLM). SFT and generation support for dLLM models, including Llada.
  • Embedding and retrieval training. Reranker training, biencoder datasets loaded directly from the Hugging Face Hub, in-batch negative sampling, and ONNX export for biencoder models.
  • SkyPilot launcher. Native multi-node launch on cloud (SkyPilot, including Kubernetes), in addition to local interactive runs. SkyPilot and NeMo Run launchers are selected with YAML sections in the config; SLURM jobs use the sbatch slurm.sub workflow.
  • CLI install profile. The nemo-automodel[cli] extra declares pyyaml beyond the package’s base dependencies for job-submission configs.
  • Refreshed CLI. automodel <config.yaml> (alias am) replaces the older automodel <command> <domain> -c <config> form.

New Models

  • LLM: GLM-5, MiniMax-M2.5, Nemotron Super v3, Nemotron Nano 4B/8B.
  • MoE / VLM: Qwen3.5-MoE (397B-A17B, 35B-A3B).
  • VLM: Gemma 4, Mistral Small 4, Qwen3.5 small dense models.
  • Diffusion: FLUX.1-dev, Wan 2.1 T2V, HunyuanVideo 1.5; Wan multi-resolution and LoRA recipes for diffusion.

Distributed Training

  • Context parallelism for Qwen3.5-MoE and Nemotron v3.
  • Pipeline parallelism for knowledge distillation.
  • HybridEP and UCCL-EP as alternative expert-parallel dispatchers.
  • FSDP2 weight prefetching and async TP optimization.
  • TP > 1 in knowledge distillation.

Performance and Kernels

  • TE Linear layers enabled for PEFT/LoRA.
  • torch._grouped_mm expert backend.
  • fp32 RMSNorm backend and cast_model_to_dtype controls.
  • TP-aware KD loss with distributed softmax and T² scaling.
  • FlashOptim optimizer integration.
  • Sequence-packing updates: Qwen3.5-MoE VLM neat-packing recipe with EP+PP; Generic THD collation for chat datasets; CP/BSHD padding fixes.

PEFT

  • MoE LoRA: rank scaling, torch_mm integration, expert-LoRA init using config.expert_dim.
  • merge_lora tool for materializing adapters into the base model.
  • QLoRA PEFT checkpoints saved with the HF adapter prefix.

Recipes and Workflow

  • New recipes for Gemma 4 (LoRA), Nemotron Nano 4B SQuAD, Mistral Small 4, Tulu-3 E2E convergence, GPT-OSS 20B / Moonlight 16B convergence, and reranker / biencoder training.
  • MFU logging for LLM and dLLM train recipes.
  • Native Comet ML experiment tracking.
  • NEFTune noisy embeddings for instruction fine-tuning.
  • Scheduler-driven manual garbage collection.
  • Common inference utility and .generate() with KV cache for Nemotron v3.

Checkpointing

  • v4_compatible checkpoint format.
  • Diffusion full fine-tuning and pretraining examples use safetensors checkpoint format; diffusion LoRA examples use torch_save.
  • QLoRA / LoRA loading robustness; tied-weight handling moved out of _init_model.

Fixes

  • FSDP2 meta-device crash for Qwen3.5 GatedDeltaNet fp32 params.
  • Activation checkpointing silently skipped on registered VLMs (ModuleList flattening).
  • Gradient checkpointing for MoE models on single GPU (ep_size=1).
  • Gradient clipping with torch_mm + EP (GPT-OSS 120B recipe).
  • Rotary embeddings for v4 models; inputs_embeds passthrough for Nano v3.

Breaking Changes

A migration guide for the new CLI, the recipe YAML section, the SLURM sbatch-script workflow, and the nemo-automodel[cli] install profile is in Breaking Changes.

0.3.0 · 26.02 (2026-02-26) · PyPI · GH · NGC Docker

Highlights

  • Transformers v4 / v5 alignment. New transformers v4 API support and a v5 refactor for device-mesh-only model init.
  • Streaming safetensors writer for faster checkpoint export.
  • Faster fp8 dequant kernels with DTensor dequantization fixes for DSv3.

New Models

  • LLM: DeepSeek V3.2, Step-3.5-Flash, MiniMax-M2.1, Nemotron-3-Nano-30B-A3B, Nemotron Flash 1B, GLM-4.7, Devstral-Small-2-24B.
  • MoE / VLM / Omni: Qwen3-VL (4B/8B), Qwen3-VL-MoE (30B/235B), Kimi-VL, Kimi-K2.5 VL, Nemotron-Parse VLM, InternVL3.5-4B, Ministral3 (3B/8B/14B), Phi-4-multimodal.

Distributed Training

  • v5 refactor: device-mesh-only model init.
  • TP plan for Ministral; Ministral3 ported to transformers v4.
  • Pipeline-parallelism validation support.
  • Parallel diffusers generate().

Performance and Kernels

  • TE fp8 for models that support it.
  • GroupedExpertsTE backend (prerequisite for MoE fp8).
  • TE RoPE fusion for custom MoE models; norm fusion and RoPE cache for dense models.
  • Improved import time.

PEFT

  • DoRA implementation.
  • LoRA support for custom MoEs.
  • LoRA support in Biencoder.

Datasets and Workflow

  • Databricks Delta Lake dataset support; consolidation for Databricks.
  • Parquet file support; inline text dataset format.
  • ColumnMapped: configurable special tokens, chat-template flags, and answer-only masking.
  • Hard negative mining and biencoder + inline-dataset tests.
  • nsys benchmark support and model-layer name scoping in the CLI.
  • Updated checkpoint auto-loading with explicit restore_from.
  • Dion optimizer.
  • Functiongemma + xlam tool-calling recipes.

Fixes

  • inputs_embeds passthrough for Nano v3.
  • from_pretrained / from_config simplification with model-id pass-through.
  • Tied-embedding detection improvements.

0.2.0 · 25.11 (2025-12-04) · PyPI · GH · NGC Docker

Highlights

  • Async checkpointing. Checkpoint refactor with async DCP and HF safetensors backport / consolidation.
  • Custom MoE optimizations. FSDP optimizations, packed-sequence + context parallel through TE, configurable router precision, fp32 lm_head and fp32 apply_rope.
  • Performance documentation. New performance-summary doc and benchmarking recipe with configs.
  • Multinode + cluster guidance. Multinode configs and updated launcher docs.

New Models

  • MoE: Qwen3 MoE custom implementation, Qwen3 Next, GPT-OSS (custom implementation, dequantization, DGX Spark recipe), GLM 4 / 4.5 / 4.6 MoE, GLM 4.5 Air, Moonlight 2L test, Phi 4 (TP plan).
  • Omni / VLM: Qwen3-Omni OOTB recipe and custom implementation.
  • DeepSeek v3 with fp8 base checkpoint loading.
  • Sequence classification: Qwen3ForSequenceClassification registered; generic SFT sequence-classification recipe.

Distributed Training

  • VLM expert-parallel recipe support.
  • PP for VLM; PEFT with PP.
  • Sharding optimization for SP / LoRA.
  • clip_grad_norm across all parallelism modes.
  • fully_shard_by_dtype option.
  • Out-of-tree (OOT) parallelism decorator.

Performance and Kernels

  • Mask creation moved into the data pipeline for better performance.
  • TE attention for GPT-OSS.
  • Faster fp8 dequant; auto-detect base-weights dequant.

PEFT

  • LoRA-aware ColwiseParallel / RowwiseParallel.
  • LoRA + TE.
  • MFU estimation for LoRA.
  • Additional PEFT LoRA recipes.

Datasets and Recipes

  • Multiturn chat dataset; VLM multiturn chat support.
  • Tool-calling dataset and recipe.
  • Streaming dataset.
  • Multiple validation datasets with per-dataset logging.
  • ColumnMapped: surface truncating + padding options.
  • Configurable max-clip-grad; configurable remote-logging frequency using step_scheduler.
  • Validation-loss checkpoint, run-val-at-ckpt, best-ckpt symlink.
  • InternVL recipe; Qwen3-VL 30B recipe; Llama-Embed-Nemotron-8B training.

Logging and Observability

  • MLflow integration.
  • Metric logger with JSONL output.
  • YAML logging-to-stdout improvements.

Workflow

  • Knowledge-distillation custom validation step; ScopedModuleOffloading to reduce memory.
  • Model Registry component.
  • SIGTERM handling.
  • NEMO_ENABLE_USER_MODULES for user-extension modules.
  • Rank-0 download for custom models.
  • Dereference env vars in YAML.

0.1.2 (2025-10-23) · PyPI · GH

Patch release.

  • Fix: max_steps now set inside the constructor (#650).
  • Fix: step scheduler switched to zero-based indexing (#627).
  • Fix: sample-limit handling for ColumnMapped datasets (#521).

0.1.0 (2025-10-08) · PyPI · GH

Initial public release of NeMo AutoModel.

Highlights

  • PyTorch-native training framework for LLMs and VLMs with Hugging Face Transformers integration via NeMoAuto* wrapper classes.
  • YAML-driven recipes for SFT and PEFT.
  • FSDP2 / HSDP / DDP distributed training with DTensor sharding.
  • Megatron-FSDP available as the default heavy-duty sharding option (replaces the earlier nvFSDP path).
  • Knowledge distillation recipe.
  • MoE component with DeepSeek v3 model implementation.
  • ColumnMappedTextInstructionDataset for instruction tuning.
  • Gradient checkpointing.
  • SLURM launcher.

For the list of newly supported models per release, see the Model Support Log.