Changelog

This section identifies the major changes in each version of the NVIDIA NeMo™ framework released to date.

• Open source release of NeMo-Aligner. NeMo-Aligner is a one stop shop for efficient model alignment algorithms, featuring:

  • Support for the full Reinforcement Learning from Human Feedback(RLHF) pipeline including SFT, Reward Model Training and Reinforcement Learning

  • Support for the SteerLM technique

  • Support for Direct Preference Optimization

  • Support for all Megatron Core GPT models such as LLAMA2 70B

  • Improved user experience

• General announcement of the NeMo Framework Inference container, featuring:

  • Deployment support for distributed checkpoints (Megatron Core) for NeMotron 8B and Llama 2 (BF16 only)

  • Deployment support for fine-tuned (SFT, RLHF, SteerLM) NeMotron 8B (BF16 only)

  • Deployment support for P-tuned Llama 2 on a single GPU (BF16 only)

  • Support for serving GPT and Llama 2 models using PyTriton on Triton Inference Server

  • Support for serving GPT and Llama 2 models using TensorRT-LLM C++ backend on Triton Inference Server

  • Support in-flight batching for TensorRT-LLM C++ backend on Triton Inference Server

• Enabled PEFT to work with Llama-2 models

• Addressed an issue that occurred when resuming Supervised Fine-Tuning with constant learning rate scheduler

• Fixed model parallelism bug in SFT and PEFT

• Included P-tuning state dictionary handling for distributed checkpoints

• Fixed bug that occurred when using the save_best_model flag

• Fixed bug where progress bar would show the wrong number of steps

• Fixed container paths in Hydra configurations

• Fixed checkpoint search for distributed checkpoints

• Added the Distributed Checkpoint Format to NeMo and Megatron Core for GPT

• New GPT transformer from Megatron Core which enables training of improved LLM configs

• When training 175B GPT with FP8, use tensor parallelism TP=8 and micro batch size MBS = 2 to ensure the model-parallel partitioning fits GPU memory

• New GPT transformer from Megatron Core which enables Group and Multi Query Attention for models like LLAMA2

• Support Llama 1 and Llama 2 pre-training with Megatron Core

• Customize LLMs for Llama 1 and Llama 2 models with techniques like SFT, PEFT (p-tuning, adapters, IA3)

• Added examples and documentation for Kubernetes training

• NeMo Data Curator: added downstream task decontamination support

• Added Low-Rank Adaptation (LoRA) Support for T5 and mT5

• Added Batch Size Ramp-up Support for GPT

• Low-Rank Adaptation (LoRA) Support for GPT

• LDDL (Language Datasets and Data Loaders) for BERT on 100B model resulting in a 30% performance speedup

• Unify dataset and model classes for all PEFT (p-tuning, adapters, IA3) with SFT model class as parent for GPT

• Converter from Interleaved PP to non-Interleaved PP

• Dialog dataset guidance for SFT to help create better chat models

• Support Dynamic Sequence Length Batches with GPT SFT

• Data parallelism enabled for RLHF servers, providing a 2x end-to-end speedup in most jobs

• Addressed issue in RLHF which prevented some jobs from running in Slurm clusters

• Corrections related to the renaming of NeMo Megatron to NeMo Framework

• Modified run.name in the *_improved configuration files to match the correct parameter count

• Supports NeMo Data Curator, a scalable Python library for curating the large-scale datasets required for training large language foundation models

• Enables continued training for P-tuning

• Switches to Megatron Core for Model Parallelism

• Extends the Data Validation Tool to provide P-tuning GPU runtime estimates

• Supports tensor and pipeline parallelism conversion for GPT and T5 models

• Supports supervised fine-tuning for GPT

• Adds Reinforcement Learning from Human Feedback (RLHF) for GPT models

• Adds four GPT model sizes based on new and improved model configurations:

  • 400M_improved

  • 1B_improved

  • 7B_improved

  • 40B_improved

Following is a list of GPT model configuration changes:

• Adds a per microbatch data loader for GPT and BERT models

• Supports SquaredReLU and SwiGLU activation functions for GPT and T5 models

• Supports Rotary Position Embedding (RoPE) for GPT and RETRO

• Supports early stopping when P‑tuning or prompt tuning GPT, T5, and mT5 models

• Implements refactored adapter learning to mimic the parameter-efficient transfer learning of the NLP approach

• Adds flash attention for GPT models in Transformer Engine

• Supports BERT models with tensor parallelism (training only)

• Supports BERT models with pipeline parallelism (training only)

• Supports sequence parallelism and selective activation checkpointing for BERT (training only)

• Supports interleaved pipeline scheduling for BERT models

• Adds Distributed Adam Optimizer for BERT models

• Supports AutoConfigurator for BERT models

• Adds 110M, 4B, 20B, and 100B BERT training configurations

• Supports mixture-of-experts for T5 models (no expert parallelism, training only)

• Improves performance for GPT P‑tuning (20%−25% speed-up)

• Adds ALiBi position embeddings for T5 and mT5 (training only)

• Logs total model size (across modal parallel ranks) for GPT, T5, mT5, and BERT models

• Adds interleaved pipeline scheduling for GPT models (training only)

• Supports FP8 using Transformer Engine (training only)

• Adds Distributed Adam Optimizer for T5 and mT5 models

• Supports P‑tuning and prompt tuning for GPT models with sequence parallelism

• Improves training configurations throughput by 7.9% (5B GPT), 9.6% (3B T5), 4.3% (11B T5), 52.4% (23B T5), and 26.6% (41B T5)

• Supports NeMo framework training and inference containers on OCI; for details on orchestration scripts, reach out to oci_nm@nvidia.com

• Supports P‑tuning and prompt tuning for T5 and mT5 models with pipeline parallelism (training only)

• Supports adapter learning for GPT and T5 with tensor parallelism and pipeline parallelism (training only)

• Supports IA3 learning for GPT and T5 with tensor parallelism and pipeline parallelism (training only)

• Adds AutoConfigurator to find the highest-throughput configurations for training on Base Command Platform

• Adds AutoConfigurator for parallel inference hyperparameter search for GPT on Base Command Manager

• Supports Amazon Web Services as a cloud service provider (performance validated up to 20 p4d.24xlarge instances)

• Adds switched orchestration for cloud service providers from Azure CycleCloud to NVIDIA Nephele for Microsoft Azure

• Adds distributed Adam Optimizer for GPT models

• Adds asymmetric encoder-decoder configuration for T5 and mT5 models

• Supports untying embeddings from the classifier layer for T5 and mT5 models

• Supports relative position embeddings for T5 and mT5 models (pipeline parallelism ≥3)

• Supports P‑tuning and prompt tuning for T5 and mT5 models with tensor parallelism (training only)

• Refactors code to yield improved consistency and readability of configurations and logs

• Supports SQuAD fine-tuning and evaluation for T5 models with pipeline parallelism ≤2

• Supports XQuAD fine tuning-and evaluation for mT5 models with pipeline parallelism ≤2

• Fixes AutoConfigurator for T5 and mT5 models

• Fixes Evaluation harness in GPT models

• Fixes Prompt learning in GPT models

• Fixes “out of memory” condition when pretraining GPT models with sequence parallelism

• Supports sequence parallelism and selective activation checkpointing for GPT

• Supports relative position embeddings for T5

  NVIDIA used the mC4 dataset (24 Languages) for pretraining the mT5 models, and verified the results on KNLI, KorQuAD, KLUE-STS, and XNLI tasks.

• Updates AutoConfigurator with sequence parallelism and selective activation checkpointing for GPT models

• Adds AutoConfigurator support for DGX A100 40GB configurations for GPT, T5, and mT5 models

• Supports P‑tuning and prompt tuning for GPT with pipeline parallelism (training only)

• Supports operation fusions for higher training throughput (2%-7% speed-up)

• Changes default GPT configurations to include sequence parallelism and selective activation checkpointing: 20B (speed-up: 14%), 40B (speed-up: 9%), and 175B (speed-up: 15%)

• Adds cloud service provider support for Microsoft Azure (performance validated up to 36 Standard_ND96amsr_A100_v4 instances)

• Adds cluster validation tools (DGMI, NCCL)

• Improves performance of 20B GPT training configuration by 2.7%

• Supports asynchronous gradient all-reduce for GPT, T5, mT5 models with pipeline parallel size equal to 1

• Supports P‑tuning and prompt tuning for GPT with tensor parallelism (training only)

• Adds AutoConfigurator to find the highest-throughput configurations for training and inference on Base Command Manager

• Supports custom tokenizers (training only)

• Supports GPT models with pipeline parallelism on Base Command Manager (inference)

• Supports new hyperparameters for text generation: top-p, top-k, and temperature

• Supports T5 models with pipeline parallelism (training only)

• Switches from GeLU to GeGLU as activation function for T5

• Supports mT5 with tensor parallelism and pipeline parallelism (training only)

• Adds 11B, 23B, and 41B T5 model training configurations

• Adds 170M, 390M, and 3B mT5 model training configurations

• Adds automatic and configurable Non-Uniform Memory Access (NUMA) mapping

• Adds tensor parallelism support for T5 models (optimized for <20B parameters, training only)

• Adds 220M and 3B T5 model training configurations

• Supports GLUE fine-tuning and evaluation for T5 models

• Supports GPT models with pipeline parallelism (training only)

• Adds 40B and 175B GPT model training configurations

• Supports GPT with tensor parallelism on Base Command Platform

• Supports O2-style AMP (accelerated training of larger models)

• Includes a chatbot sample application using your trained GPT model

• Supports training metric monitoring and visualization with Weights & Biases