Important

NeMo 2.0 is an experimental feature and currently released in the dev container only: nvcr.io/nvidia/nemo:dev. Please refer to NeMo 2.0 overview for information on getting started.

NVIDIA NeMo Framework User Guide

NeMo Framework

• Overview
  • Large Language Models and Multimodal Models
  • Speech AI
  • Programming Languages and Frameworks
  • Resources
    • GitHub Repos
    • Where to Get Help
    • Licenses
• Performance
  • Performance Summary for Large Language Models
    • Pretraining
    • Fine-Tuning
  • Example Scripts for Pretraining and Fine-tuning
    • Set Up Example Scripts
    • Collect Results
• Why NeMo Framework?
  • Large Language Models & Multimodal (LLM & MM)
  • Speech AI
    • Data Augmentation
    • Speech Data Explorer
    • Using Kaldi Formatted Data
    • Speech Command Recognition
  • General Optimizations
    • Mixed Precision Training
    • Multi-GPU Training
    • NeMo, PyTorch Lightning, and Hydra
    • Optimized Pretrained Models
  • Resources

Quickstart

• Getting Started
  • Get Access to NeMo Framework
  • Large Language Models
  • Multimodal Models
  • Speech AI
• Playbooks
  • Overview
    • Infrastructure Setup
    • Model Alignment and Customization
  • Pre-training
    • Model Optimization
      • Pruning and Distillation
    • Data Curation
    • Deployment
  • Playbooks
    • Run NeMo Framework on DGX Cloud
      • Prerequisites
      • Steps
      • Next Steps
    • Run NeMo Framework on Kubernetes
      • Prerequisites
      • Storage Options
      • Set up PVC Storage
      • Set up Docker Secrets
      • NeMo Stages
      • Advanced Use Cases
      • Debugging Tips
    • NeMo Framework SFT with Llama 2
      • Project Description
      • Convert Llama 2 from Hugging Face format to NeMo format
      • Prepare data
    • NeMo Framework SFT with Mistral-7B
      • Project Description
      • Convert Mistral-7B from Hugging Face format to NeMo Format
      • Prepare Data
    • NeMo Framework SFT with Mixtral-8x7B and Nemotron 4 340B
      • Project Description
      • Convert Mixtral-8x7B from Hugging Face Format to NeMo Format
      • Prepare Data
    • NeMo Framework PEFT with Mistral-7B
      • Project Description
      • Convert Mistral-7B from Hugging Face format to NeMo format
      • Prepare data
    • NeMo Framework PEFT with Llama2, Mixtral-8x7B, and Nemotron 4 340B
      • Playbook Description
      • Convert Hugging Face format to NeMo format
      • Convert Mixtral-8x7B from Hugging Face format to NeMo format
      • Prepare data
    • NeMo Framework Foundation Model Pre-training
      • Project Description
      • Project Instructions and Milestones
      • Project Deliverables
      • Project Evaluation
    • NeMo Framework AutoConfigurator
      • Project Description
      • Project Instructions and Milestones
      • [1] Model Size Recommendation
      • [2] Getting the optimal hyperparameters for training and inference for a given model size and compute budget
      • Project Deliverables
      • Project Evaluation
    • NeMo Framework Single Node Pre-training
      • Quick Start
      • Set up the NeMo Framework Container
      • Train a GPT-style Model
    • NeMo Framework Post-Training Quantization (PTQ) with Nemotron4 and Llama3
      • Project Description
      • Prepare the NeMo Checkpoint for Nemotron4-340B and Llama3-70B
      • Convert NeMo Checkpoint to qnemo Format
      • Export to TensorRT-LLM
      • Deploy Nemotron/Llama TensorRT-LLM to Triton
    • NeMo Framework Quantization Aware Training (QAT) for Llama2 SFT Model
      • Project Description
      • Run Quantization Aware Training (QAT)

NeMo 2.0

• Overview
  • Install NeMo 2.0
  • Quickstart
  • Where to Find NeMo 2.0
  • Additional Resources
  • Known Issues
• Quickstart with NeMo-Run
  • Pretraining
    • Set up the Prerequisites
    • Configure the Recipe
      • Override attributes
      • Swap Recipes
    • Execute Locally
      • Change the number of GPUs
    • Execute on a Slurm Cluster
• Migration Guide
  • Pre-Training
    • NeMo 1.0 (Previous Release)
    • NeMo 2.0 (New Release)
    • Migration Steps
  • SFT Training and Inference
    • NeMo 1.0 (Previous Release)
    • NeMo 2.0 (New Release)
    • Migration Steps
  • PEFT Training and Inference
    • NeMo 1.0 (Previous Release)
    • NeMo 2.0 (New Release)
    • Migration Steps
    • Some Notes on Migration
  • Trainer Configuration
    • NeMo 1.0 (Previous Release)
    • NeMo 2.0 (New Release)
    • Migration Steps
  • Experiment Manager
    • NeMo 1.0 (Previous Release)
    • NeMo 2.0 (New Release)
    • Migration Steps
  • Checkpointing Configurations
    • NeMo 1.0 (Previous Release)
    • NeMo 2.0 (New Release)
    • Migration Steps
  • Optimizer Configuration
    • NeMo 1.0 (Previous Release)
    • NeMo 2.0 (New Release)
    • Migration Steps
  • Data Configuration
    • NeMo 1.0 (Previous Release)
    • NeMo 2.0 (New Release)
    • Migration Steps
    • Some Notes on Migration
  • Nsys Profiling
    • NeMo 1.0 (Previous Release)
    • NeMo 2.0 (New Release)
    • Migration Steps
  • Tokenizers
    • NeMo 1.0 (Previous Release)
    • NeMo 2.0 (New Release)
• Feature Guide
  • The Bridge Between Lightning and Megatron Core
    • MegatronStrategy
    • MegatronParallel
    • MegatronMixedPrecision
  • Logging and Checkpointing
    • ModelCheckpoint
    • NeMoLogger
    • AutoResume
    • Putting it All Together
  • Serialization
    • IOMixin
  • Parameter Efficient Fine-Tuning (PEFT)
    • ModelTransform Class
    • PEFT Class
    • LoRA Class (or Any Other PEFT Method)
    • Set Up a LoRA Training Run
  • Hugging Face Integration
    • Fine-Tune a Model using a Hugging Face Checkpoint
    • Create a Model Importer
• Large Language Models
  • Llama
    • NeMo 2.0 Pretraining Recipes
  • Mixtral
    • NeMo 2.0 Pretraining Recipes
  • Nemotron
    • NeMo 2.0 Pretraining Recipes
• Long Context Recipe
  • Context Parallelism
    • Enable Context Parallelism
    • Implement Context Parallelism
  • Access Long Context Recipe
    • Llama 3
    • Mixtral
    • Nemotron 4

Training and Customization

• SFT and PEFT
  • Developer Quick Start
    • Terminology: PEFT vs Adapter
    • How PEFT work in NeMo models
    • PEFT config classes
    • Base model classes
    • Full fine-tuning vs PEFT
  • Supported PEFT Methods
  • SFT and PEFT Examples
    • Nemotron
    • Gemma and CodeGemma
    • Starcoder2
    • Mistral
    • Mixtral
    • Llama
    • Falcon
    • Baichuan2
    • T5
  • NeMo QLoRA Guide
    • What is QLoRA?
    • How to run QLoRA in NeMo?
      • QLoRA Feature Matrix
    • QLoRA Evaluation
      • 1. Online Quantization for Framework Inference
      • 2. Offline Quantization for QLoRA Deployment
• Continual Learning with Pretrained Checkpoints
  • Configure Continual Learning
  • Adjust Training Configurations
  • Important Notes on Continual Learning
• RAG
  • RAG Pipeline Overview
  • Index Corpus Data for RAG
    • Run Indexing on a Base Model
      • Run Indexing on a Slurm Cluster
  • Generate Text with RAG
    • Run Text Generation on a Base Model
      • Run Text Generation on a Slurm Cluster
• Optimizing Models with Pruning
  • Drop Model Layers (Depth Pruning)
  • Drop Model Width (Width Pruning)
  • Drop Model Depth and Width
  • Validate Trimmed Model
  • Validate Depth Pruning of Original Model on the Fly
• Optimizing Models with Knowledge Distillation
  • Example
  • Limitations

Models

• Large Language Models
  • Common
    • Prepare Environment
      • Slurm
      • Base Command Platform
      • Kubernetes
      • General Configuration
    • Data Preparation
      • Predefined Datasets
      • Bring Your Own Dataset
    • Monitoring and Logging
    • Auto Configurator
      • Auto Configurator Capabilities
    • Resuming Training with a Different Number of Nodes
    • Continual Learning with Pretrained Checkpoints
      • Configure Continual Learning
      • Adjust Training Configurations
      • Important Notes on Continual Learning
    • Deploy NeMo Large Language Models
      • Deploy NeMo Models by Exporting to Inference Optimized Libraries
      • Deploy NeMo Models in the Framework
      • Send Queries to the NVIDIA Triton Server for NeMo LLMs
      • Supported GPUs
      • Supported NeMo Checkpoint Formats
      • Nemotron LLMs
  • Llama and CodeLlama
    • New: Llama 3.1 Support
      • Data Preparation
      • Train with Predefined Configurations
      • Checkpoint Conversion
      • Model Evaluation
      • Parameter Efficient Fine-Tuning (PEFT)
  • Gemma and CodeGemma
    • Data Preparation for SFT and PEFT
    • Step 1: Download the Dataset
    • Step 2: Data Preprocessing
    • Step 3: Split the Data Into Train, Validation, and Test
    • Step 4: Prepare Packed Dataset
    • Checkpoint Conversion
      • Gemma and CodeGemma
      • Gemma 2
    • Supervised Fine-tuning (SFT)
      • Run SFT inside NeMo Container
      • Run SFT with NeMo Launcher
    • Parameter Efficient Fine-Tuning (PEFT)
      • Run PEFT inside NeMo Container
      • Run PEFT with NeMo Launcher
  • Griffin (Recurrent Gemma)
    • Data Preparation for SFT and PEFT
    • Checkpoint Conversion
    • Supervised Fine-tuning (SFT)
      • Run SFT inside NeMo Container
      • Run SFT with NeMo Launcher
    • Parameter Efficient Fine-Tuning (PEFT)
      • Run PEFT inside NeMo Container
      • Run PEFT with NeMo Launcher
  • Baichuan 2
    • Data Preparation
      • Run Data Preparation
    • Training with Predefined Configurations
      • Run Training
    • Checkpoint Conversion
      • Run Conversion
    • Model Evaluation
      • Run Evaluation
      • Run Evaluation on PEFT Baichuan2 models
    • Parameter Efficient Fine-Tuning (PEFT)
      • Run PEFT with NeMo Launcher
  • Falcon
    • Falcon
    • Data Preparation
      • Run Data Preparation
    • Training with Predefined Configurations
      • Run Training
    • Model Evaluation
      • Run Evaluation
      • Run Evaluation on PEFT Falcon models
    • Parameter Efficient Fine-Tuning (PEFT)
      • Run PEFT with NeMo Launcher
  • Mamba2 and Hybrid Models
    • Data Preparation for SFT
    • Checkpoint Conversion
      • Obtain the Checkpoints from Hugging Face
      • Convert the PyTorch Checkpoint to a NeMo Checkpoint
      • Run Tensor Parallelism (TP) for 8b Models
    • Supervised Fine-Tuning (SFT)
      • Run Fine-Tuning
      • Evaluate the Fine-Tuned Model
  • Mistral
    • Data Preparation
      • Run Data Preparation
    • Training with Predefined Configurations
      • Run Training
    • Checkpoint Conversion
      • Run Conversion
    • Model Evaluation
      • Run Evaluation
      • Run Evaluation on PEFT Mistral models
    • Parameter Efficient Fine-Tuning (PEFT)
      • Run PEFT with NeMo Launcher
  • Mixtral
    • Data Preparation
      • Run Data Preparation
    • Training with Predefined Configurations
      • Run Training
      • Configure the Model
      • Request Gated Model Assets
    • Checkpoint Conversion
      • Run Conversion
    • Model Evaluation
      • Run Evaluation
      • Run Evaluation on PEFT Mixtral Models
    • Parameter Efficient Fine-Tuning (PEFT)
      • Run PEFT with NeMo Launcher
  • Nemotron
    • Nemotron
    • Data Preparation
      • Run Data Preparation
    • Training with Predefined Configurations
      • Run Training
    • Model Evaluation
      • Run Evaluation
      • Run Evaluation on PEFT Nemotron Models
    • Parameter Efficient Fine-Tuning (PEFT)
      • Run PEFT Training with NeMo Framework Launcher
      • Run P-Tuning on a Common Cluster
      • Run P-Tuning on a Slurm Cluster
      • Run P-Tuning on the Base Command Platform
    • Supervised Fine-Tuning (SFT)
      • Run SFT inside NeMo Container
      • Run SFT with NeMo Launcher
  • StarCoder2
    • StarCoder2
    • Data Preparation for SFT and PEFT
      • Step 1: Download the Dataset
      • Step 2: Data Preprocessing
      • Step 3: Split the Data Into Train, Validation, and Test
      • Step 4: Prepare Packed Dataset
    • Checkpoint Conversion
    • Model Evaluation
      • Run Evaluation
    • Supervised Fine-tuning (SFT)
      • Run SFT inside NeMo Container
      • Run SFT with NeMo Launcher
    • Parameter Efficient Fine-Tuning (PEFT)
      • Run PEFT inside NeMo Container
      • Run PEFT with NeMo Launcher
  • T5
    • Data Preparation
      • Slurm
      • Base Command Platform
      • Common
    • Training
      • Slurm
      • Base Command Platform
    • Training with Predefined Configurations
      • 3B configuration
      • 11B configuration
      • 23B configuration
      • 41B configuration
    • Changing Embedding Type
    • Checkpoint Conversion
      • Common
      • Slurm
      • Base Command Platform
    • Model Evaluation
      • Base Model Evaluation
      • Prompt-Learned T5 and mT5 Evaluation
      • Adapter-Learned and IA3-Learned T5 Evaluation
    • Parameter-Efficient Fine-Tuning (PEFT)
      • Fine-Tune with PEFT
      • Evaluate with PEFT
    • Model Fine-Tuning
      • SQuAD Fine-Tuning
      • Fine-tuning on Custom Tasks
    • T5 Results
      • Training Accuracy Results
      • Training Performance Results
      • Inference Performance
  • mT5
    • Data Preparation
      • Slurm
      • Base Command Platform
      • Common
    • Training
      • Slurm
      • Base Command Platform
    • Training with Predefined Configurations
      • 170M configuration
      • 390M configuration
      • 3B configuration
    • Checkpoint Conversion
      • Common
      • Slurm
      • Base Command Platform
    • Model Evaluation
      • Common
      • Slurm
      • Base Command Platform
    • PEFT Training and Inference
    • Model Fine-Tuning
      • XQuAD Fine-Tuning
      • Fine-tuning on Custom Tasks
    • MT5 Results
      • Training Accuracy Results
      • Training Performance Results
      • Inference Performance
  • GPT
    • Data Preparation
      • Slurm
      • Base Command Platform
      • Kubernetes
      • Common
    • Model Training
      • Slurm
      • Base Command Platform
      • Kubernetes
    • Training with Predefined Configurations
      • 126M configuration
      • 400M_improved configuration
      • 1B_improved configuration
      • 5B configuration
      • 7B_improved configuration
      • 20B configuration
      • 40B configuration
      • 40B_improved configuration
      • 175B configuration
      • FP8 with Transformer Engine
    • Checkpoint Conversion
      • Common
      • Slurm
      • Base Command Platform
      • Kubernetes
    • Model Evaluation
      • Base Model Evaluation
      • Prompt-Learned GPT Evaluation
      • Adapter-Learned and IA3-Learned GPT Evaluation
    • Parameter Efficient Fine-Tuning (PEFT)
      • PEFT Training and Inference
      • PEFT Training with NeMo Megatron Launcher
      • Common
      • Slurm
      • Base Command Platform
    • Fully Sharded Data Parallel (FSDP)
      • Overview
      • Usage
    • Torch Distributed Checkpoint (TDC)
      • Overview
      • Usage
    • GPT Results
      • Training Accuracy Results
      • Training Performance Results
      • Inference Performance
  • BERT
    • Data Preparation
      • Slurm
      • Base Command Platform
      • Common
    • Model Training
      • Slurm
      • Base Command Platform
    • Training with Predefined Configurations
      • 110M configuration
      • 4B configuration
      • 20B configuration
      • 100B configuration
    • BERT Results
      • Training Accuracy Results
      • Training Performance Results
  • ChatGLM
    • Data Preparation
      • Run Data Preparation
    • Training with Predefined Configurations
      • Run Training
    • Checkpoint Conversion
      • Run Conversion
    • Model Evaluation
      • Run Evaluation
      • Run Evaluation on PEFT ChatGLM models
    • Parameter Efficient Fine-Tuning (PEFT)
      • Run PEFT with NeMo Launcher
  • RETRO
    • Data Preparation
      • Summary of Main Stages
    • Training
      • Slurm
    • Training with Predefined Configurations
      • 300M Model Configuration
    • Model Inferencing
      • Base Model Inferencing
    • Model Evaluation
      • Base Model Evaluation
  • Tokenizer
    • SentencePiece Tokenizer
      • Installation
      • Training
      • NeMo Usage
• Multimodal Models
  • Multimodal Language Models
    • NeVA (LLaVA)
      • New in NeVA
    • VideoNeVA
      • Configure VideoNeVA
  • Vision-Language Foundation Models
    • CLIP
      • Data Preparation
      • Training with Predefined Configurations
      • Evaluation
      • Framework Inference
      • Performance
    • Vision Transformer
      • Data Preparation
      • Vision Transformer Training
      • Fine-tuning
      • Evaluation
      • Framework Inference
      • Performance
    • NSFW Content Filter
      • Data Preparation
      • Fine-tuning
      • Framework Inference
      • Performance
  • Text-to-Image Models
    • Stable Diffusion and SDXL
      • Data Preparation
      • Training with Predefined Configurations
      • Run NeMo Framework Inference
      • Performance
    • Dreambooth
      • Data Preparation
      • Training with Predefined Configurations
      • Framework Inference
      • Performance
    • ControlNet
      • Data Preparation
      • Training with Predefined Configurations
      • Framework Inference
      • Performance
    • InstructPix2Pix
      • Data Preparation
      • Training with Predefined Configurations
      • Framework Inference
      • Performance
    • Imagen
      • Data Preparation
      • Training with Predefined Configurations
      • Framework Inference
      • Performance
  • Beyond 2D generation using NeRF
    • DreamFusion
      • DreamFusion-DMTet
      • Data Preparation
      • Training with Predefined Configurations
      • Performance
• Embedding Models
  • BERT Embedding Models
    • Prepare the Data
    • Convert the Checkpoint
    • Fine-tune the Model
      • Configure the Slurm Cluster
      • Configure the Base Command Platform
  • GPT Embedding Models
    • Training a GPT Embedding Model
    • Inference using a GPT Embedding Model
• Speech AI Models
  • Automatic Speech Recognition (ASR)
  • Speaker Diarization
  • Text-To-Speech (TTS)
  • Speech AI Tools

Deploy Models

• Overview
  • Deploy NeMo Models by Exporting Inference Optimized libraries
  • NVIDIA NIM for LLMs
  • Deploy PyTorch-level Models (In-Framework)
• NeMo Large Language Models
  • Deploy NeMo Models by Exporting to Inference Optimized Libraries
    • Supported LLMs
      • Deploy NeMo Models by Exporting TensorRT-LLM
      • Deploy NeMo Models by Exporting vLLM
  • Deploy NeMo Models in the Framework
    • Quick Example
    • Use a Script to Deploy NeMo LLMs on a Triton Server
      • Deploy a NeMo LLM Model
      • Use NeMo Deploy Module APIs to Run Inference
  • Send Queries to the NVIDIA Triton Server for NeMo LLMs
    • Send a Query using the Script
    • Send a Query using the NeMo APIs
  • Supported GPUs
  • Supported NeMo Checkpoint Formats
  • Nemotron LLMs
• NeMo Multimodal Models
  • Optimized Inference for Multimodal Models using TensorRT
  • Supported GPUs
  • Supported Models
    • Access the Models with a Hugging Face Token
    • Export and Deploy a NeMo Multimodal Checkpoint to TensorRT
    • Use a Script to Run Inference on a Triton Server
      • Export and Deploy a Multimodal Model to TensorRT
      • Send a Query
    • Use NeMo Export and Deploy Module APIs to Run Inference
      • Export a Multimodal Model to TensorRT
      • Deploy a Multimodal Model to TensorRT
      • Send a Query
      • Other Examples

Library Documentation

• Overview
• NeMo
  • Introduction
    • Prerequisites
    • Installation
    • Quick Start Guide
    • Discussion Board
    • Contribute to NeMo
    • License
  • NeMo Fundamentals
    • NeMo Models
    • Configuring and Training NeMo Models
    • Example Training Script
    • Overriding Configs
    • Running NeMo Scripts
      • NeMo Inference Scripts
    • Specifying training data
    • Model Checkpoints
    • Fine-Tuning
    • Where To Go Next?
  • Tutorials
    • Running Tutorials on Colab
    • Tutorial Overview
  • Mixed Precision Training
    • Half-precision Training
    • FP8 Training
      • Overview
      • Resources
  • Parallelisms
    • Data Parallelism
      • Distributed Data Parallelism
      • Distributed Optimizer
      • Fully-Shared Data Parallelism (FSDP)
    • Model Parallelism
      • Tensor Parallelism
      • Pipeline Parallelism
      • Expert Parallelism
    • Activation Partitioning
      • Sequence Parallelism
      • Context Parallelism
      • Parallelism Nomenclature
  • Mixture of Experts
    • Overview
    • Use MoE
    • Configure MoE-specific Loss Functions
  • Optimizations
    • Attention Optimizations
      • Flash Attention
      • Multi-query Attention (MQA) and Grouped-query Attention (GQA)
    • Sequence Packing
      • Sequence Packing for SFT/PEFT
      • Sequence Packing for NeVA
    • Activation Recomputation
      • Transformer Layer Recomputation
      • Self-attention Recomputation
    • Communication Overlap
      • Data-parallel Communication Overlap
      • Tensor-parallel Communication Overlap
      • Pipeline-parallel Communication Overlap
      • Context-parallel Communication Overlap
    • CPU Offloading
      • Overview
      • Features
      • Usage
  • Checkpoints
    • Understanding Checkpoint Formats
    • Sharded Model Weights
    • Quantized Checkpoints
    • Community Checkpoint Converter
      • Distributed Checkpoints
      • Community Model Converter User Guide
      • Community Model Converter Development Guide
      • Converting from Megatron-LM
  • NeMo APIs
    • NeMo Models
      • Basics
      • Pretrained
      • Training
      • Configuration
      • Optimization
      • Save and Restore
      • Register Artifacts
      • Push to Hugging Face Hub
      • Nested NeMo Models
      • Profiling
    • Neural Modules
      • Dynamic Layer Freezing
    • Experiment Manager
      • Resume Training
      • Experiment Loggers
      • Exponential Moving Average
      • Hydra Multi-Run with NeMo
      • ExpManagerConfig
    • Neural Types
      • Motivation
      • NeuralType class
      • Type Comparison Results
      • Examples
    • Exporting NeMo Models
      • Exporting Models
      • How to Use Model Export
      • How to Make Model Exportable
    • Adapters
      • What are Adapters?
      • torch.nn.Module with Adapters
      • AdapterModuleMixin
      • Using the Adapter Module
      • Adapter Compatible Models
      • References
    • NeMo Core APIs
      • Base class for all NeMo models
      • Base Neural Module class
      • Base Mixin classes
      • Base Connector classes
      • Base Mixin Classes
      • Neural Type checking
      • Neural Type classes
      • Experiment manager
      • Exportable
    • NeMo Common Collection API
      • Callbacks
      • Losses
      • Metrics
      • Tokenizers
      • Data
      • S3 Checkpointing
    • NeMo Large language Model API
      • Pretraining Model Classes
      • Customization Model Classes
      • Modules
      • Datasets
      • Adapter Mixin Class
      • Exportable Model Classes
    • NeMo Multimodal API
      • Model Classes
      • Modules
      • Datasets
    • NeMo ASR API
      • Model Classes
      • Modules
      • Parts
      • Mixins
      • Datasets
      • Audio Preprocessors
      • Audio Augmentors
      • Miscellaneous Classes
    • NeMo TTS API
      • Model Classes
      • Base Classes
      • Dataset Processing Classes
  • NeMo Collections
    • Large Language Models
      • GPT model training
      • Batching
      • RETRO Model
      • Hiddens Module
      • Parameter-Efficient Fine-Tuning (PEFT)
      • Positional embeddings
      • Positional interpolation
      • References
      • Megatron Core Customization
      • References
    • Machine Translation Models
      • Quick Start Guide
      • Data Format
      • Data Cleaning, Normalization & Tokenization
      • Training a BPE Tokenization
      • Applying BPE Tokenization, Batching, Bucketing and Padding
      • Tarred Datasets for Large Corpora
      • Model Configuration and Training
      • Multi-Validation
      • Bottleneck Models and Latent Variable Models (VAE, MIM)
      • Model Inference
      • Inference Improvements
      • Pretrained Encoders
      • References
    • ONNX Export of Megatron Models
      • Requirements
      • Export to ONNX
    • Quantization
      • Post-Training Quantization (PTQ)
      • Quantization-Aware Training (QAT)
      • References
    • Multimodal Language Models
      • Multimodal Language Model Datasets
      • Common Configuration Files
      • NeVA
      • Video NeVA
      • Inference with Video NeVA
      • Evaluation with Mixtral as a judge
      • Sequence Packing for NeVA
      • Speech-agumented Large Language Models (SpeechLLM)
    • Vision-Language Foundation
      • Datasets
      • Common Configuration Files
      • Dataset Configuration
      • Checkpoints
      • CLIP
    • Text to Image Models
      • Datasets
      • Common Configuration Files
      • Checkpoints
      • Stable Diffusion
      • Imagen
      • DreamBooth
      • ControlNet
      • InstructPix2Pix
      • Stable Diffusion XL Int8 Quantization
    • NeRF
      • Datasets
      • Common Configuration Files
      • DreamFusion
    • Vision Models
      • Supported Models
      • Spotlight Models
      • References
    • Automatic Speech Recognition (ASR)
      • Transcribe speech with 3 lines of code
      • Transcribe speech via command line
      • Incorporate a language model (LM) to improve ASR transcriptions
      • Use real-time transcription
      • Try different ASR models
      • Try out NeMo ASR transcription in your browser
      • ASR tutorial notebooks
      • ASR model configuration
      • Preparing ASR datasets
      • NeMo ASR Documentation
    • Speech Classification
      • Models
      • Datasets
      • Checkpoints
      • NeMo Speech Classification Configuration Files
      • Resource and Documentation Guide
      • Resource and Documentation Guide
    • Speaker Recognition (SR)
      • Models
      • NeMo Speaker Recognition Configuration Files
      • Datasets
      • Checkpoints
      • NeMo Speaker Recognition API
      • Resource and Documentation Guide
      • Resource and Documentation Guide
    • Speaker Diarization
      • Models
      • Datasets
      • Checkpoints
      • NeMo Speaker Diarization Configuration Files
      • Hydra Configurations for Diarization Training
      • Hydra Configurations for Diarization Inference
      • NeMo Speaker Diarization API
      • Resource and Documentation Guide
      • Resource and Documentation Guide
    • Speech Self-Supervised Learning
      • Models
      • Datasets
      • Checkpoints
      • NeMo SSL Configuration Files
      • NeMo SSL collection API
      • Resources and Documentation
      • Resources and Documentation
    • Speech Intent Classification and Slot Filling
      • Models
      • Datasets
      • Checkpoints
      • NeMo Speech Intent Classification and Slot Filling Configuration Files
      • NeMo Speech Intent Classification and Slot Filling collection API
      • Resources and Documentation
      • Resources and Documentation
    • Text-to-Speech (TTS)
      • Models
      • Data Preprocessing
      • Checkpoints
      • NeMo TTS Configuration Files
      • Grapheme-to-Phoneme Models
    • Resources and Documentation
  • Speech AI Tools
    • NeMo Forced Aligner (NFA)
      • Demos & Tutorials
      • Quickstart
      • How do I use NeMo Forced Aligner?
      • Input manifest file format
      • Output CTM file format
      • Output ASS file format
      • Output JSON manifest file format
      • How do I evaluate the alignment accuracy?
    • Dataset Creation Tool Based on CTC-Segmentation
      • References
    • Speech Data Explorer
      • SDE Demo Instance
      • Getting Started
      • User Interface
      • Analysis of Speech Datasets
    • Comparison tool for ASR Models
      • Getting Started
      • User Interface
      • Analysis of Speech Datasets
    • ASR Evaluator
    • Speech Data Processor
    • (Inverse) Text Normalization
      • WFST-based (Inverse) Text Normalization
      • Neural Models for (Inverse) Text Normalization
• NeMo Framework Launcher
  • Launcher Introduction
    • Key Features
    • How the Launcher Operates
  • Quickstart Guide for NeMo Launcher
    • Installation Steps
    • Starting the NeMo Framework Container
      • Example: GPT3 5B Pretraining
  • NeMo Launcher Tutorial
    • NeMo Launcher Tutorial
    • Understand the Configurations
      • Overall Hierarchy:
      • Pipeline Configurations
      • Cluster Configurations
      • Environment Variables Configurations
      • NUMA Mapping Configurations
    • Launch Experiment with NeMo Launcher
      • Configure and Run a Launcher Pipeline
      • Example: Pre-train Stable Diffusion 860M Model for 10 Epochs with Resolution 256
    • Logs and Results
    • Launcher Multirun
    • Debugging
• NeMo Aligner
  • Obtain a Pretrained Model
  • Model Alignment by Supervised Fine-Tuning (SFT)
    • Fine-Tune with a Prompt-Response Dataset
      • Step 1: Format the data
      • Step 2: Run SFT training
      • Step 3: Run inference or further fine-tuning
    • Fine-Tune with a Chat Dataset
      • Step 1: Format the data
      • Step 2: Run SFT training
      • Step 3: Run inference or further fine-tuning
  • Model Alignment by RLHF
    • Data Processing for RLHF
    • Reward Model Training
      • Data Preprocessing
    • PPO Training
      • Launching the Reward Model and Critic Server
      • Launching the Initial Policy and PPO Actor Training
      • Launching Both Servers for RLHF Training
      • Speeding up PPO with TensorRT-LLM
      • PPO Results with TensorRT-LLM
      • PPO Performance Results with TensorRT-LLM
      • PPO Results
    • Scaling the Tutorial to Bigger Models
  • Model Alignment by SteerLM Method
    • SteerLM
    • SteerLM vs RLHF
    • Train a SteerLM model
    • Step 1: Download Llama 2 LLM model
    • Step 2: Download and Preprocess data for Attribute Prediction Modelling
    • Step 3: Train the regression reward model on OASST+HelpSteer data
    • Step 4: Generate annotations
    • Step 5: Train the Attribute-Conditioned SFT model
    • Step 6: Inference
    • SteerLM: Novel Technique for Simple and Controllable Model Alignment
  • SteerLM 2.0: Iterative Training for Attribute-Conditioned Language Model Alignment
    • Overview
    • Method Details
    • Train a SteerLM 2.0 Model
      • Preparing the Training Dataset
      • Training Example
      • Inference
      • References
  • Model Alignment by Direct Preference Optimization (DPO)
    • Obtaining a Pretrained Model
    • Instruction Following Taught by Supervised Fine-Tuning (SFT)
    • DPO Model Training
  • Model Alignment by Self-Play Fine-Tuning (SPIN)
    • Obtaining a pretrained model
    • SFT vs Foundational (base) model for SPIN Training
    • SPIN Model Training
  • Fine-tuning Stable Diffusion with DRaFT+
    • Data Input for running DRaFT+
    • Reward Model
    • DRaFT+ Training
      • DRaFT+ Results
  • Constitutional AI: Harmlessness from AI Feedback
    • CAI
    • Motivation
    • Train a CAI model
    • Step 1: Download models and datasets
    • Step 2: Generate and revise responses to harmful prompts creating the SL-CAI dataset
    • Step 3: Fine-tune Mistral-7B on the revised responses to create a Mistral-7B-SL-CAI model
    • Step 4: Generate the RL-CAI (preference) dataset for RM and PPO training
    • Step 5: Train the RM
    • Step 6: Fine-tune Mistral-7B-SL-CAI with PPO and the RM to train a Mistral-7B-RL-CAI model
    • Step 7: Inference
• NeMo Curator
  • Download and Extract Text
    • Background
    • Usage
    • Related Scripts
      • Common Crawl Example
  • Working with DocumentDataset
    • Background
    • Usage
      • Reading and Writing
      • Resuming from Interruptions
      • Blending and Shuffling
  • CPU and GPU Modules with Dask
    • Initializing the Dask Cluster
    • CPU Modules
    • GPU Modules
    • Dask with SLURM
    • Dask with K8s
  • Classifier and Heuristic Quality Filtering
    • Background
    • Usage
      • Batched Filtering
    • Classifier Filtering
    • Heuristic Filtering
  • Language Identification and Unicode Fixing
    • Background
    • Usage
    • Related Scripts
  • GPU Accelerated Exact and Fuzzy Deduplication
    • Background
    • Usage
  • Semantic Deduplication
    • Background
    • How It Works
    • Configure Semantic Deduplication
    • Change Embedding Models
    • Deduplication Thresholds
    • Usage
      • Use Individual Components
      • Use the SemDedup Class
    • Parameters
    • Output
    • Performance Considerations
  • Synthetic Data Generation
    • Connect to an LLM Service
      • Deploy an LLM Inference Service
      • Query a Reward Model
    • Customize the Nemotron-4 340B Pipeline
      • Generate Synthetic Prompts
      • Generate Open Q&A Prompts
      • Generate Writing Prompts
      • Generate Closed Q&A Prompts
      • Generate Math Prompts
      • Generate Coding Prompts
      • Change Prompt Templates
      • Generate Dialogue
      • Generate Synthetic Two-Turn Prompts
      • Classify Entities
      • Generate Asynchronously
    • Combine Synthetic Data Generation with other NeMo Curator Modules
  • Downstream Task Decontamination/Deduplication
    • Background
    • Usage
  • PII Identification and Removal
    • Background
    • Usage
      • Reading documents and de-identifying
      • Resuming from Interruptions
  • Distributed Data Classification
    • Background
    • Usage
      • AEGIS Safety Model
      • CrossFit Integration
  • Running NeMo Curator on Kubernetes
    • Prerequisuites
    • Storage
      • Setup PVC Busybox Helper Pod
    • Setup Docker Secrets
    • Setup Python Environment
    • Upload Data to PVC
    • Create a Dask Cluster
    • (Opt #1) Running Existing Module
    • (Opt #2) Running Custom Module
    • Deleting Cluster
    • Download Data from PVC
  • Best Practices
    • Choosing the Right Quality Model Type
  • Next Steps
    • Training a Tokenizer
    • Training a Large Language Model
    • Aligning a Large Language Model
  • API Reference
    • Dask Cluster Functions
    • Datasets
      • DocumentDataset
    • Download and Extract
      • Base Classes
      • Common Crawl
      • Wikipedia
      • Arxiv
    • Filters
      • Base Class
      • Modules
      • FastText Filters
      • Heuristic Filters
      • Code Filters
    • Classifiers
    • Modifiers
      • Base Class
      • Module
      • Modifiers
    • Deduplication
      • Exact
      • Fuzzy
      • Semantic
    • Task Decontamination
      • Base Class
      • Module
      • Tasks
    • LLM Services
    • Synthetic Data
    • Miscellaneous

Cloud Service Providers

• Overview
  • Cluster Bring-Up
    • Common
    • OCI
    • GCP
    • AWS
  • Cluster Validation
    • Validation Script Usage
    • Running tests manually
  • Configuration Changes
    • Set NCCL Topology
    • Environment Variables
      • Azure Variables
      • AWS Variables

Releases

• Software Component Versions
  • NeMo Framework 24.07
  • NeMo Framework 24.05
  • NeMo Framework 24.01
  • NeMo Framework 23.11
• Changelog
  • NeMo Framework 24.07
    • Training
    • ASR
    • Aligner
    • Curator
    • Export & Deploy
  • NeMo Framework 24.05
    • LLM and MM
      • Models
    • Performance
    • Export
    • Curator
    • Aligner
    • Speech (ASR & TTS)
      • Models
    • Perf Improvements
    • Customization
    • Longform Inference
    • Misc
  • NeMo Framework 24.03.01
  • NeMo Framework 24.03
  • NeMo Framework 24.01.01
  • NeMo Framework 24.01
  • NeMo Framework 23.11
  • NeMo Framework 23.10
  • NeMo Framework 23.08.03
  • NeMo Framework 23.08.02
  • NeMo Framework 23.08.01
  • NeMo Framework 23.08
  • NeMo Framework 23.07
  • NeMo Framework 23.05
  • NeMo Framework 23.04
  • NeMo Framework 23.03
  • NeMo Framework 23.01
  • NeMo Framework 22.11
  • NeMo Framework 22.09
  • NeMo Framework 22.08.01
  • NeMo Framework 22.08
  • NeMo Framework 22.06
  • NeMo Framework 22.05.01
  • NeMo Framework 22.05
  • NeMo Framework 22.04
  • NeMo Framework 22.03
  • NeMo Framework 22.02
  • NeMo Framework 22.01
• Known Issues