Important

You are viewing the NeMo 2.0 documentation. This release introduces significant changes to the API and a new library, NeMo Run. We are currently porting all features from NeMo 1.0 to 2.0. For documentation on previous versions or features not yet available in 2.0, please refer to the NeMo 24.07 documentation.

Changelog#

This section identifies the major changes in each version of the NVIDIA NeMo™ framework released to date.

NeMo Framework 24.12#

Training#

  • Fault Tolerance

    • Straggler Detection

    • Auto Relaunch

LLM & MM#

  • MM models

    • Llava-next

    • Llama 3.2

  • Sequence Model Parallel for NeVa

  • Enable Energon

  • SigLIP (NeMo 1.0 only)

  • LLM 2.0 migration

    • Starcoder2

    • Gemma 2

    • T5

    • Baichuan

    • BERT

    • Mamba

    • ChatGLM

  • DoRA support

Aligner#

  • NeMo 2.0 Model Support

  • Sequence Packing for DPO

  • Reinforce/RLOO Support

  • SFT Knowledge Distillation

  • Context Parallelism for SFT

Export#

  • Nemo 2.0 base model export path for NIM

  • PTQ in Nemo 2.0

Curator#

  • Synthetic Data Generation for Text Retrieval

    • LLM-based Filters

      • Easiness

      • Answerability

    • Q&A Retrieval Generation Pipeline

  • Parallel Dataset Curation for Machine Translation

    • Load/Write Bitext Files

    • Heuristic filtering (Histogram, Length Ratio)

    • Classifier filtering (Comet, Cometoid)

ASR#

  • Timestamps with TDT decoder

  • Timestamps option with .transcribe()

NeMo Framework 24.09#

LLM & MM#

  • Training

    • Long context recipe

    • PyTorch Native FSDP 1

    • Distributed Checkpoint

      • Torch native format

      • Async

      • Parallel r/w

    • NeMo Run OSS

  • Models

    • Llama 3

    • Mixtral

    • Nemotron

    • E2E BF16 Training of Llama3 with RedPajama2 data (~2.4T tokens)

      • NeMo2.0 llama3 8b trained on 2.4T RP2 tokens

      • NeMo2.0 llama3 70b trained on <1T RP2 tokens

  • NeMo 1.0

    • SDXL (text-2-image)

    • Model Opt

      • Depth Pruning [docs]

      • Logit based Knowledge Distillation [docs]

Aligner#

  • Rejection Sampling (SFT)

  • Packed sequence training in aligner for SFT

Export#

  • TensorRT-LLM v0.12 integration

  • LoRA support for vLLM

  • FP8 checkpoint

Curator#

  • Image Semantic Deduplication

  • NSFW Classifier

  • Aesthetic Classifier

  • CLIP Embedding Creation

  • AEGIS Classifier

  • Quality Classifier

ASR#

  • Parakeet large (ASR with PnC model)

  • Added Uzbek offline and Gregorian streaming models

  • Optimization feature for efficient bucketing to improve bs consumption on GPUs

NeMo Framework 24.07#

Training#

  • Features and Model architectures

    • PEFT: QLoRA support, LoRA/QLora for Mixture-of-Experts (MoE) dense layer

    • State Space Models & Hybrid Architecture support (Mamba2 and NV-Mamba2-hybrid)

    • Support Nemotron, Minitron, Gemma2, Qwen, RAG

  • Multimodal

    • NeVA: Add SOTA LLM backbone support (Mixtral/LLaMA3) and suite of model parallelism support (PP/EP)

    • Support Language Instructed Temporal-Localization Assistant (LITA) on top of video NeVA

  • Custom Tokenizer training in NeMo

  • Update the Auto-Configurator for EP, CP and FSDP

ASR#

  • SpeechLM and SALM

  • Adapters for Canary Customization

  • Pytorch allocator in PyTorch 2.2 improves training speed up to 30% for all ASR models

  • Cuda Graphs for Transducer Inference

  • Replaced webdataset with Lhotse - gives up to 2x speedup

  • Transcription Improvements - Speedup and QoL Changes

  • ASR Prompt Formatter for multimodal Canary

Aligner#

  • Speed up Aligner RLHF by 7x with TRT-LLM

  • Reward Preference Optimization (RPO)

  • Identity Preference Optimization (IPO)

  • SteerLM2

  • Llama 3 performance and convergence example

  • Constitutional AI algorithm (RLAIF)

Curator#

  • Semantic Deduplication

  • Resiliparse for Text Extraction

  • Improve Distributed Data Classification - Domain classifier is 1.55x faster through intelligent batching

  • Synthetic data generation for fine-tuning

Export & Deploy#

  • In framework PyTriton deployment with backends:

    • PyTorch

    • vLLM

    • TRT-LLM update to 0.10

  • TRT-LLM C++ runtime

NeMo Framework 24.05#

NeMo Framework now supports Large Language Models (LLM), Multimodal (MM), Automatic Speech Recognition (ASR), and Text-to-Speech (TTS) in a single consolidated container.

LLM and MM#

Models#

  • Megatron Core RETRO

    • Pre-training

    • Zero-shot Evaluation

  • Pretraining, conversion, evaluation, SFT, and PEFT for:

    • Mixtral 8X22B

    • Llama 3

    • SpaceGemma

  • Embedding Models Fine Tuning

    • Mistral

    • BERT

  • BERT models

    • Distributed checkpoint

  • Video capabilities with NeVa

Performance#

  • Distributed Checkpointing

    • Torch native backend

    • Parallel read/write

    • Async write

  • Multimodal LLM (LLAVA/NeVA)

    • Pipeline Parallelism support

    • Sequence packing support

Export#

  • Integration of Export & Deploy Modules into NeMo Framework container

    • Upgrade to TRT-LLM 0.9

Curator#

  • SFT/PEFT (LoRA, and p-tuning) Data Curation Pipeline and Example

  • Dataset Blending Tool

  • Domain Classifier

Aligner#

  • LoRA techniques with:

    • PPO Actor

    • PDO

    • SFT/SteerLM

  • Stable Diffusion models

Speech (ASR & TTS)#

Models#

  • AED Multi Task Models (Canary) - Multi-Task Multi-Lingual Speech Recognition / Speech Translation model

  • Multimodal Domain - Speech LLM supporting SALM Model

  • Parakeet-tdt_ctc-1.1b Model - RTFx of > 1500 (can transcribe 1500 seconds of audio in 1 second)

  • Audio Codec 16kHz Small - NeMo Neural Audio Codec for discretizing speech for use in LLMs

    • mel_codec_22khz_medium

    • mel_codec_44khz_medium

Perf Improvements#

  • Transcribe() upgrade - Enables one line transcribe with files, tensors, data loaders

  • Frame looping algorithm for RNNT faster decoding - Improves Real Time Factor (RTF) by 2-3x

  • Cuda Graphs + Label-Looping algorithm for RNN-T and TDT Decoding - Transducer Greedy decoding at over 1500x RTFx, on par with CTC Non-Autoregressive models

  • Semi Sorted Batching support - External User contribution that speeds up training by 15-30%.

Customization#

  • Context biasing for CTC word stamping - Improve accuracy for custom vocabulary and pronunciation

Longform Inference#

  • Longform inference support for AED models

  • Transcription of multi-channel audio for AED models

Misc#

  • Upgraded webdataset - Speech and LLM / Multimodal unified container

NeMo Framework 24.03.01#

Issues Fixed

  • GPT Memory Leak at Loss Function

  • Eval script issue for Mixtral PEFT

  • Llama 7B Out-of-memory issue when using 1TB system memory

  • Enable Pipeline Parallelism support for LoRA merge

  • Multi-node Llama training on Kubernetes while saving checkpoint

NeMo Framework 24.03#

  • Fully Sharded Data Parallel (FSDP) support for GPT

  • Post-Training Quantization (PTQ) with AMMO library (0.7.4) for Llama

  • Support Expert Parallelism on all MoE models e.g. Mixtral

  • Pipeline parallel for p-tuning

  • Updated PEFT metrics for all popular community models. (Support matrix Temp Internal Link)

  • Upgraded PyTorch Lightning to 2.2

  • Upgraded base container to PyTorch 24.02

  • Consolidation of the StarCoder2 and Gemma specific containers, with the previous Framework GA container

  • Customizable distributed data classification tool in Curator

  • GPU-accelerated quality classification model code in Curator

  • GPU-accelerated domain classification model code in Curator

NeMo Framework 24.01.01#

  • Added Mixture of Experts parameter passing for MCore

  • PP/TP support for Mixture of Experts

  • SFT / PEFT support for Gemma model

  • Training / SFT / PEFT / Evaluation support for - Baichuan model - CodeLlama model

  • Fixed SFT/PEFT support nemo-launcher configs for Mistral and Mixtral - Edited configs with correct values

  • Documentation refactor and landing page added

  • NeMo Framework developer docs added

NeMo Framework 24.01#

  • New end-to-end support (pretraining, conversion, evaluation, SFT, PEFT) for community models, featuring:

    • Support for community model Falcon

    • Support for community model Mixtral (expert parallelism coming in future release)

    • Support for community model Mistral

    • Support for community model Code Llama

  • General availability release of NeMo Multimodal, featuring:

    • Support for vision-language foundation models: CLIP

    • Support for text-2-image foundation models: Stable Diffusion and Imagen

    • Support for text-2-image customization: SD-LoRA, SD-ControlNet, SD-instruct pix2pix

    • Support for multimodal LLM: NeVA and LLAVA

    • Support for text-2-NeRF: DreamFusion++

    • Support for NSFW

  • New performance features and key optimization:

    • Support PyTorch Fully Sharded Data Parallel training (FSDP) with tensor-parallelism

    • Support CPU offloading and prefetch of activations and weights

    • Support Context Parallelism for performant long-sequence-length LLM training

    • Support framework-level FP8 precision that reduces memory usage and training step time

    • Transformer layer granularity re-computation with FP8 LLM training

    • Support pipelined tensor-parallel communication overlap with GEMM for all LLMs

    • Support LLM fine-tuning with packed sequences

    • Support fused RoPE and Swiglu for LLAMA2 like models

    • Device memory bug fix; removed FP8 cast/transpose duplicates in FP8 training

  • New features for NeMo Aligner:

    • Support for MultiEpoch

    • Added PPO: custom end strings + memory optimizations

    • Added SFT: LoRa and custom validation metrics

  • New features for NeMo Curator:

    • Multi-node multi-GPU fuzzy document-level deduplication supported within the launcher.

    • Added new Personal Identifiable Information (PII) Removal module

    • Task decontamination for SFT and PEFT (e.g., LoRA, p-tuning, adapters, etc.) datasets supported within the launcher

    • Code data filtering heuristics from StarCoder

NeMo Framework 23.11#

  • Open source release of NeMo-Aligner. NeMo-Aligner is a one stop shop for efficient model alignment algorithms, featuring:

    • Support for the full Reinforcement Learning from Human Feedback(RLHF) pipeline including SFT, Reward Model Training and Reinforcement Learning

    • Support for the SteerLM technique

    • Support for Direct Preference Optimization

    • Support for all Megatron Core GPT models such as LLAMA2 70B

    • Improved user experience

NeMo Framework 23.10#

  • General announcement of the NeMo Framework Inference container, featuring:

    • Deployment support for distributed checkpoints (Megatron Core) for NeMotron 8B and Llama 2 (BF16 only)

    • Deployment support for fine-tuned (SFT, RLHF, SteerLM) NeMotron 8B (BF16 only)

    • Deployment support for P-tuned Llama 2 on a single GPU (BF16 only)

    • Support for serving GPT and Llama 2 models using PyTriton on Triton Inference Server

    • Support for serving GPT and Llama 2 models using TensorRT-LLM C++ backend on Triton Inference Server

    • Support in-flight batching for TensorRT-LLM C++ backend on Triton Inference Server

NeMo Framework 23.08.03#

  • Enabled PEFT to work with Llama-2 models

  • Addressed an issue that occurred when resuming Supervised Fine-Tuning with constant learning rate scheduler

  • Fixed model parallelism bug in SFT and PEFT

  • Included P-tuning state dictionary handling for distributed checkpoints

  • Fixed bug that occurred when using the save_best_model flag

  • Fixed bug where progress bar would show the wrong number of steps

NeMo Framework 23.08.02#

  • Fixed container paths in Hydra configurations

NeMo Framework 23.08.01#

  • Fixed checkpoint search for distributed checkpoints

NeMo Framework 23.08#

  • Added the Distributed Checkpoint Format to NeMo and Megatron Core for GPT

  • New GPT transformer from Megatron Core which enables training of improved LLM configs

  • When training 175B GPT with FP8, use tensor parallelism TP=8 and micro batch size MBS = 2 to ensure the model-parallel partitioning fits GPU memory

  • New GPT transformer from Megatron Core which enables Group and Multi Query Attention for models like LLAMA2

  • Support Llama 1 and Llama 2 pre-training with Megatron Core

  • Customize LLMs for Llama 1 and Llama 2 models with techniques like SFT, PEFT (p-tuning, adapters, IA3)

  • Added examples and documentation for Kubernetes training

  • NeMo Data Curator: added downstream task decontamination support

NeMo Framework 23.07#

  • Added Low-Rank Adaptation (LoRA) Support for T5 and mT5

  • Added Batch Size Ramp-up Support for GPT

NeMo Framework 23.05#

  • Low-Rank Adaptation (LoRA) Support for GPT

  • LDDL (Language Datasets and Data Loaders) for BERT on 100B model resulting in a 30% performance speedup

  • Unify dataset and model classes for all PEFT (p-tuning, adapters, IA3) with SFT model class as parent for GPT

  • Converter from Interleaved PP to non-Interleaved PP

  • Dialog dataset guidance for SFT to help create better chat models

  • Support Dynamic Sequence Length Batches with GPT SFT

  • Data parallelism enabled for RLHF servers, providing a 2x end-to-end speedup in most jobs

NeMo Framework 23.04.1 ——————–@@

  • Addressed issue in RLHF which prevented some jobs from running in Slurm clusters

  • Corrections related to the renaming of NeMo Megatron to NeMo Framework

  • Modified run.name in the *_improved configuration files to match the correct parameter count

NeMo Framework 23.04#

  • Supports NeMo Data Curator, a scalable Python library for curating the large-scale datasets required for training large language foundation models

  • Enables continued training for P-tuning

  • Switches to Megatron Core for Model Parallelism

  • Extends the Data Validation Tool to provide P-tuning GPU runtime estimates

  • Supports tensor and pipeline parallelism conversion for GPT and T5 models

  • Supports supervised fine-tuning for GPT

  • Adds Reinforcement Learning from Human Feedback (RLHF) for GPT models

  • Adds four GPT model sizes based on new and improved model configurations:

    • 400M_improved

    • 1B_improved

    • 7B_improved

    • 40B_improved

Following is a list of GPT model configuration changes:

Configuration

Previous

New

Activation

GeLU

Fast-SwiGLU

Position Embedding

Learned Absolute

RoPE

Dropout

0.1

0

Embeddings and Output Layer

Tied

Untied

Bias terms

Yes

No

Normalization

LayerNorm

LayerNorm1p

NeMo Framework 23.03#

  • Adds a per microbatch data loader for GPT and BERT models

  • Supports SquaredReLU and SwiGLU activation functions for GPT and T5 models

  • Supports Rotary Position Embedding (RoPE) for GPT and RETRO

  • Supports early stopping when P‑tuning or prompt tuning GPT, T5, and mT5 models

  • Implements refactored adapter learning to mimic the parameter-efficient transfer learning of the NLP approach

  • Adds flash attention for GPT models in Transformer Engine

NeMo Framework 23.01#

  • Supports BERT models with tensor parallelism (training only)

  • Supports BERT models with pipeline parallelism (training only)

  • Supports sequence parallelism and selective activation checkpointing for BERT (training only)

  • Supports interleaved pipeline scheduling for BERT models

  • Adds Distributed Adam Optimizer for BERT models

  • Supports AutoConfigurator for BERT models

  • Adds 110M, 4B, 20B, and 100B BERT training configurations

  • Supports mixture-of-experts for T5 models (no expert parallelism, training only)

  • Improves performance for GPT P‑tuning (20%−25% speed-up)

  • Adds ALiBi position embeddings for T5 and mT5 (training only)

  • Logs total model size (across modal parallel ranks) for GPT, T5, mT5, and BERT models

NeMo Framework 22.11#

  • Adds interleaved pipeline scheduling for GPT models (training only)

  • Supports FP8 using Transformer Engine (training only)

  • Adds Distributed Adam Optimizer for T5 and mT5 models

  • Supports P‑tuning and prompt tuning for GPT models with sequence parallelism

  • Improves training configurations throughput by 7.9% (5B GPT), 9.6% (3B T5), 4.3% (11B T5), 52.4% (23B T5), and 26.6% (41B T5)

NeMo Framework 22.09#

  • Supports NeMo Framework training and inference containers on OCI; for details on orchestration scripts, reach out to oci_nm@nvidia.com

  • Supports P‑tuning and prompt tuning for T5 and mT5 models with pipeline parallelism (training only)

  • Supports adapter learning for GPT and T5 with tensor parallelism and pipeline parallelism (training only)

  • Supports IA3 learning for GPT and T5 with tensor parallelism and pipeline parallelism (training only)

  • Adds AutoConfigurator to find the highest-throughput configurations for training on Base Command Platform

  • Adds AutoConfigurator for parallel inference hyperparameter search for GPT on Base Command Manager

NeMo Framework 22.08.01#

  • Supports Amazon Web Services as a cloud service provider (performance validated up to 20 p4d.24xlarge instances)

  • Adds switched orchestration for cloud service providers from Azure CycleCloud to NVIDIA Nephele for Microsoft Azure

NeMo Framework 22.08#

  • Adds distributed Adam Optimizer for GPT models

  • Adds asymmetric encoder-decoder configuration for T5 and mT5 models

  • Supports untying embeddings from the classifier layer for T5 and mT5 models

  • Supports relative position embeddings for T5 and mT5 models (pipeline parallelism ≥3)

  • Supports P‑tuning and prompt tuning for T5 and mT5 models with tensor parallelism (training only)

  • Refactors code to yield improved consistency and readability of configurations and logs

  • Supports SQuAD fine-tuning and evaluation for T5 models with pipeline parallelism ≤2

  • Supports XQuAD fine tuning-and evaluation for mT5 models with pipeline parallelism ≤2

NeMo Framework 22.06-hotfix.01 —————————-@@

  • Fixes AutoConfigurator for T5 and mT5 models

  • Fixes Evaluation harness in GPT models

  • Fixes Prompt learning in GPT models

  • Fixes “out of memory” condition when pretraining GPT models with sequence parallelism

NeMo Framework 22.06#

  • Supports sequence parallelism and selective activation checkpointing for GPT

  • Supports relative position embeddings for T5

    NVIDIA used the mC4 dataset (24 Languages) for pretraining the mT5 models, and verified the results on KNLI, KorQuAD, KLUE-STS, and XNLI tasks.

  • Updates AutoConfigurator with sequence parallelism and selective activation checkpointing for GPT models

  • Adds AutoConfigurator support for DGX A100 40GB configurations for GPT, T5, and mT5 models

  • Supports P‑tuning and prompt tuning for GPT with pipeline parallelism (training only)

  • Supports operation fusions for higher training throughput (2%-7% speed-up)

  • Changes default GPT configurations to include sequence parallelism and selective activation checkpointing: 20B (speed-up: 14%), 40B (speed-up: 9%), and 175B (speed-up: 15%)

NeMo Framework 22.05.01#

  • Adds cloud service provider support for Microsoft Azure (performance validated up to 36 Standard_ND96amsr_A100_v4 instances)

  • Adds cluster validation tools (DGMI, NCCL)

  • Improves performance of 20B GPT training configuration by 2.7%

NeMo Framework 22.05#

  • Supports asynchronous gradient all-reduce for GPT, T5, mT5 models with pipeline parallel size equal to 1

  • Supports P‑tuning and prompt tuning for GPT with tensor parallelism (training only)

  • Adds AutoConfigurator to find the highest-throughput configurations for training and inference on Base Command Manager

  • Supports custom tokenizers (training only)

  • Supports GPT models with pipeline parallelism on Base Command Manager (inference)

  • Supports new hyperparameters for text generation: top-p, top-k, and temperature

NeMo Framework 22.04#

  • Supports T5 models with pipeline parallelism (training only)

  • Switches from GeLU to GeGLU as activation function for T5

  • Supports mT5 with tensor parallelism and pipeline parallelism (training only)

  • Adds 11B, 23B, and 41B T5 model training configurations

  • Adds 170M, 390M, and 3B mT5 model training configurations

  • Adds automatic and configurable Non-Uniform Memory Access (NUMA) mapping

NeMo Framework 22.03#

  • Adds tensor parallelism support for T5 models (optimized for <20B parameters, training only)

  • Adds 220M and 3B T5 model training configurations

  • Supports GLUE fine-tuning and evaluation for T5 models

NeMo Framework 22.02#

  • Supports GPT models with pipeline parallelism (training only)

  • Adds 40B and 175B GPT model training configurations

NeMo Framework 22.01#

  • Supports GPT with tensor parallelism on Base Command Platform

  • Supports O2-style AMP (accelerated training of larger models)

  • Includes a chatbot sample application using your trained GPT model

  • Supports training metric monitoring and visualization with Weights & Biases

On this page