Important
You are viewing the NeMo 2.0 documentation. This release introduces significant changes to the API and a new library, NeMo Run. We are currently porting all features from NeMo 1.0 to 2.0. For documentation on previous versions or features not yet available in 2.0, please refer to the NeMo 24.07 documentation.
NVIDIA NeMo Framework User Guide#
NeMo Framework
Getting Started
Developer Guides
- Migration Guide
- Feature Guide
- Best Practices
Training and Customization
Models
Deploy Models
Library Documentation
- Overview
- NeMo
- Introduction
- NeMo Fundamentals
- Tutorials
- Mixed Precision Training
- Parallelisms
- Mixture of Experts
- Optimizations
- Checkpoints
- NeMo APIs
- NeMo Collections
- Large Language Models
- GPT Model Training
- Batching
- RETRO Model
- Hiddens Module
- Parameter-Efficient Fine-Tuning (PEFT)
- Positional embeddings
- Positional interpolation
- References
- Megatron Core Customization
- Drop Model Laeyrs
- Validate Trimmed Model
- Validate Original Model
- Reset Learning Rate
- Parameters
- Use Cases
- Ramp Up Batch Size
- Usage
- Ramp Up Stages and Training Interruption
- Automatic Node Scheduling
- Example
- References
- Machine Translation Models
- Quick Start Guide
- Data Format
- Data Cleaning, Normalization & Tokenization
- Training a BPE Tokenization
- Applying BPE Tokenization, Batching, Bucketing and Padding
- Tarred Datasets for Large Corpora
- Model Configuration and Training
- Multi-Validation
- Bottleneck Models and Latent Variable Models (VAE, MIM)
- Model Inference
- Inference Improvements
- Pretrained Encoders
- References
- ONNX Export of Megatron Models
- Quantization
- Multimodal Language Models
- Vision-Language Foundation
- Text to Image Models
- NeRF
- Speech-agumented Large Language Models (SpeechLLM)
- Vision Models
- Automatic Speech Recognition (ASR)
- Transcribe speech with 3 lines of code
- Transcribe speech via command line
- Incorporate a language model (LM) to improve ASR transcriptions
- Use real-time transcription
- Try different ASR models
- Try out NeMo ASR transcription in your browser
- ASR tutorial notebooks
- ASR model configuration
- Preparing ASR datasets
- NeMo ASR Documentation
- Speech Classification
- Speaker Recognition (SR)
- Speaker Diarization
- Speech Self-Supervised Learning
- Speech Intent Classification and Slot Filling
- Text-to-Speech (TTS)
- Resources and Documentation
- Large Language Models
- Speech AI Tools
- NeMo Aligner
- Obtain a Pretrained Model
- Model Alignment by Supervised Fine-Tuning (SFT)
- Model Alignment by RLHF
- Model Alignment by SteerLM Method
- SteerLM
- SteerLM vs RLHF
- Train a SteerLM model
- Step 1: Download Llama 2 LLM model
- Step 2: Download and Preprocess data for Attribute Prediction Modelling
- Step 3: Train the regression reward model on OASST+HelpSteer data
- Step 4: Generate annotations
- Step 5: Train the Attribute-Conditioned SFT model
- Step 6: Inference
- SteerLM: Novel Technique for Simple and Controllable Model Alignment
- SteerLM 2.0: Iterative Training for Attribute-Conditioned Language Model Alignment
- Model Alignment by Direct Preference Optimization (DPO)
- Model Alignment by Self-Play Fine-Tuning (SPIN)
- Fine-tuning Stable Diffusion with DRaFT+
- Constitutional AI: Harmlessness from AI Feedback
- CAI
- Motivation
- Train a CAI model
- Step 1: Download models and datasets
- Step 2: Generate and revise responses to harmful prompts creating the SL-CAI dataset
- Step 3: Fine-tune Mistral-7B on the revised responses to create a Mistral-7B-SL-CAI model
- Step 4: Generate the RL-CAI (preference) dataset for RM and PPO training
- Step 5: Train the RM
- Step 6: Fine-tune Mistral-7B-SL-CAI with PPO and the RM to train a Mistral-7B-RL-CAI model
- Step 7: Inference
- NeMo Curator
- Text Curation
- Download and Extract Text
- Working with DocumentDataset
- CPU and GPU Modules with Dask
- Classifier and Heuristic Quality Filtering
- Language Identification and Unicode Fixing
- GPU Accelerated Exact and Fuzzy Deduplication
- Semantic Deduplication
- Synthetic Data Generation
- Downstream Task Decontamination/Deduplication
- PII Identification and Removal
- Distributed Data Classification
- Image Curation
- Reference
- Text Curation
- NeMo Run
- API Reference
- Frequently Asked Questions
- Configuration
- Q: UnserializableValueError when using
run.Partialorrun.Config - Q: Deserialization error when using
run.Partialorrun.Config - Q: I made a change locally in my git repo and tested it using the local executor. However, the change is not reflected in the remote job.
- Q: I made a change locally outside my git repo and tested it using the local executor. However, the change is not reflected in the remote job.
- Q: UnserializableValueError when using
- Execution
- Management
- Configuration
- Installation
- Tutorials
Releases
- Software Component Versions
- Changelog
- NeMo Framework 24.09
- NeMo Framework 24.07
- NeMo Framework 24.05
- NeMo Framework 24.03.01
- NeMo Framework 24.03
- NeMo Framework 24.01.01
- NeMo Framework 24.01
- NeMo Framework 23.11
- NeMo Framework 23.10
- NeMo Framework 23.08.03
- NeMo Framework 23.08.02
- NeMo Framework 23.08.01
- NeMo Framework 23.08
- NeMo Framework 23.07
- NeMo Framework 23.05
- NeMo Framework 23.04
- NeMo Framework 23.03
- NeMo Framework 23.01
- NeMo Framework 22.11
- NeMo Framework 22.09
- NeMo Framework 22.08.01
- NeMo Framework 22.08
- NeMo Framework 22.06
- NeMo Framework 22.05.01
- NeMo Framework 22.05
- NeMo Framework 22.04
- NeMo Framework 22.03
- NeMo Framework 22.02
- NeMo Framework 22.01
- Known Issues