NeMo Framework User Guide

NeMo Framework contains scripts and code to provide end-to-end data preparation and training for large transformer based models.

Get access to the training and inference containers automatically at https://developer.nvidia.com/nemo-framework.

• NeMo Framework User Guide
• Playbooks
  
  • Playbooks
  • Running NeMo Framework on DGX Cloud
    
    • Pre-Requisites
    • Steps
      
      • Setup
      • Data Preparation
      • Model Training
    • Next Steps
  • Running NeMo Framework on Kubernetes
    
    • Pre-Requisites
    • Steps
      
      • Setup
      • Data Preparation
      • Model Training
    • Next Steps
    • Troubleshooting
  • Foundation Model Pre-training using NeMo Framework
    
    • Project Description
      
      • Learning Goals
      • NeMo Tools and Resources
      • Data
      • Requirements
    • Project Instructions and Milestones
    • Project Deliverables
    • Project Evaluation
  • NeMo Framework AutoConfigurator
    
    • Project Description
      
      • Learning Goals
      • NeMo Tools and Resources
      • Data
      • Requirements
    • Project Instructions and Milestones
    • [1] Model Size Recommendation
    • [2] Getting the optimal hyperparameters for training and inference for a given model size and compute budget
    • Project Deliverables
    • Project Evaluation
  • NeMo Framework Supervised fine-tuning (SFT) with Llama2
    
    • Project Description
      
      • Learning Goals
      • NeMo Tools and Resources
      • Software Requirements
      • Hardware Requirements
      • Data
    • [Optional] Convert Llama2 from Huggingface format to NeMo format
      
      • Optional Step 1: Download llama2 in huggingface format
      • Optional Step 2: Convert to .nemo
    • Prepare data
      
      • Step 1: Download dataset
      • Step 2: Data Preprocessing
      • Step 3: Split the data into train, validation and test.
    • Step 4: Run the SFT finetuning script
      
      • Step 6: Run evaluation
  • NeMo Framework PEFT with Llama2
    
    • Project Description
      
      • Learning Goals
      • NeMo Tools and Resources
      • Educational Resources
      • Software Requirements
      • Hardware Requirements
    • [Optional] Convert Llama2 from Huggingface format to NeMo format
      
      • Step 1: Download llama2 in huggingface format
      • Step 2: Convert to .nemo
    • Prepare data
      
      • Step 1: Download the PubMedQA dataset and run the split_dataset.py script in the cloned directory.
      • Step 2: Data Preprocessing
    • Run the PEFT finetuning script
      
      • Step 1: Set the experiment configs
      • Step 2: Run PEFT training
      • Step 3: Run inference
• Model Overview
• Feature Matrix
  
  • GPT Models
  • T5 and mT5 Models
  • BERT Models
• Software Component Versions
  
  • NeMo Framework Training Container
  • NeMo Framework Inference Container
• Cloud Service Providers
  
  • Cluster Bring-Up
    
    • Common
    • OCI
    • AWS
  • Cluster Validation
    
    • Validation Script Usage
    • Running tests manually
  • Configuration Changes
    
    • Set NCCL Topology
    • Environment Variables
      
      • Azure Variables
      • AWS Variables
• Model Guide
  
  • Model Guide
  • Training NeMo Framework Models
    
    • Prepare Environment
      
      • Slurm
      • Base Command Platform
      • Kubernetes
      • General Configuration
    • Data Preparation
      
      • Data Preparation for GPT Models
      • Data Preparation for T5 Models
      • Data Preparation for mT5 Models
      • Data Preparation for BERT Models
  • Training with Predefined Configurations
    
    • Predefined Configurations of GPT Models
      
      • 126M configuration
      • 400M_improved configuration
      • 1B_improved configuration
      • 5B configuration
      • 7B_improved configuration
      • 20B configuration
      • 40B configuration
      • 40B_improved configuration
      • 175B configuration
      • FP8 with Transformer Engine
    • Predefined Configurations of T5 Models
      
      • 3B configuration
      • 11B configuration
      • 23B configuration
      • 41B configuration
    • Predefined Configurations of mT5 Models
      
      • 170M configuration
      • 390M configuration
      • 3B configuration
    • Training Logs with TensorBoard and Weights and Biases
    • Predefined Configurations of BERT Models
      
      • 110M configuration
      • 4B configuration
      • 20B configuration
      • 100B configuration
  • Using AutoConfigurator to Find the Optimal Configuration
    
    • AutoConfigurator Capabilities
      
      • Model Size Recommendation
      • Base Configuration Generation
      • Training AutoConfigurator HP Search
      • Inference AutoConfigurator HP Search
    • Usage
      
      • General Configuration
      • Predefined Configurations
      • Running Custom Model Size Configurations
      • Interpreting the Results
      • Logging Runs with Weights and Biases
  • Training with Custom Configurations
    
    • Example: Changing Embedding Type for T5 Models
  • Bring Your Own Dataset
    
    • Slurm
    • Base Command Platform
    • Kubernetes
    • Common
  • Model Training
    
    • GPT Training
      
      • Slurm
      • Base Command Platform
      • Kubernetes
    • T5 Training
      
      • Slurm
      • Base Command Platform
    • mT5 Training
      
      • Slurm
      • Base Command Platform
    • BERT Training
      
      • Slurm
      • Base Command Platform
  • Resuming Training with a Different Number of Nodes
  • Checkpoint Conversion
    
    • GPT Conversion
      
      • Common
      • Slurm
      • Base Command Platform
      • Kubernetes
    • T5 Conversion
      
      • Common
      • Slurm
      • Base Command Platform
    • mT5 Conversion
      
      • Common
      • Slurm
      • Base Command Platform
  • Generalized PEFT Framework
    
    • PEFT Training and Inference for GPT-style Models
    • PEFT Training and Inference
    • PEFT Training with NeMo Megatron Launcher
    • Common
    • Slurm
    • Base Command Platform
    • PEFT Training and Inference for mT5/T5-style Models
    • PEFT Training and Inference
  • Model Fine-Tuning
    
    • T5 Fine-Tuning
      
      • Common
      • Slurm
      • Base Command Platform
    • mT5 Fine-Tuning
      
      • Common
      • Slurm
      • Base Command Platform
    • Fine-tuning on Custom Tasks
  • Model Prompt Learning
    
    • GPT Prompt Learning
      
      • Common
      • Slurm
      • Base Command Platform
    • T5 and mT5 Prompt Learning
      
      • Common
      • Slurm
      • Base Command Platform
  • Model Adapter Learning and IA3 Learning
    
    • GPT Adapter Learning and IA3 Learning
      
      • Common
      • Slurm
      • Base Command Platform
    • T5 Adapter Learning and IA3 Learning
      
      • Common
      • Slurm
      • Base Command Platform
  • Model Evaluation
    
    • GPT Evaluation
      
      • Common
      • Slurm
      • Base Command Platform
      • Kubernetes
    • T5 Evaluation
      
      • Common
      • Slurm
      • Base Command Platform
    • mT5 Evaluation
      
      • Common
      • Slurm
      • Base Command Platform
    • Prompt-Learned GPT Evaluation
      
      • Common
      • Slurm
      • Base Command Platform
    • Prompt-Learned T5 and mT5 Evaluation
      
      • Common
      • Slurm
      • Base Command Platform
    • Adapter-Learned and IA3-Learned GPT Evaluation
      
      • Common
      • Slurm
      • Base Command Platform
    • Adapter-Learned and IA3-Learned T5 Evaluation
      
      • Common
      • Slurm
      • Base Command Platform
  • Exporting the NeMo Models to TensorRT-LLM
    
    • Supported Model and GPUs
    • User Guide
    • NeMo Export Module and APIs
      
      • PyTest for Export Module
  • NeMo Data Curator
    
    • Coverage
    • General Usage
    • Module-specific documentation
      
      • NeMo Data Curator
      • Downloading and extracting text
      • Document filtering
      • Text cleaning and language separation
      • Exact and fuzzy deduplication
      • Classifier and heuristic-based quality filtering
      • 1 Downstream task decontamination (task deduplication)
      • prepare_task_data
      • 3 Find the matching task N-grams within the training documents
      • 4 Remove matching N-grams above a user-defined threshold.
• Model Alignment
  
  • Model Alignment by RLHF
    
    • Obtaining a pretrained model
    • Instruction Following Taught by Supervised Fine-Tuning (SFT)
      
      • SFT Data Formatting
    • Data processing for RLHF
    • Reward Model Training
      
      • Data Preprocessing
    • PPO Training
      
      • Launching the Reward Model and Critic Server
      • Launching the Initial Policy and PPO Actor Training
      • Launching Both Servers for RLHF training
      • PPO Results
    • Scaling the tutorial to bigger models
  • Model Alignment by SteerLM Method
    
    • SteerLM
    • SteerLM vs RLHF
    • Train a SteerLM model
    • Step 1: Download Llama 2 LLM model
    • Step 2: Download and Preprocess data for Attribute Prediction Modelling
    • Step 3: Train the regression reward model on OASST+HelpSteer data
    • Step 4: Generate annotations
    • Step 5: Train the Attribute-Conditioned SFT model
    • Step 6: Inference
    • SteerLM: Novel Technique for Simple and Controllable Model Alignment
  • Model Alignment by Direct Preference Optimisation (DPO)
    
    • Obtaining a pretrained model
    • Instruction Following Taught by Supervised Fine-Tuning (SFT)
    • DPO Model Training
• Llama
  
  • Llama
  • Data Preparation
    
    • Run Data Preparation
      
      • Configuration
  • Training with Predefined Configurations
    
    • Run Training
      
      • Configuration
  • Checkpoint Conversion
    
    • Run Conversion
      
      • Configuration
  • Model Evaluation
    
    • Run Evaluation
      
      • Configuration
    • Run Evaluation on PEFT Llama models
      
      • Configuration
  • Parameter Efficient Fine-Tuning (PEFT)
    
    • Run PEFT
      
      • Configuration
  • Model Export to TensorRT-LLM
    
    • Run Export
  • Model Deployment
    
    • Serve in-framework or TensorRT-LLM model on Triton
      
      • Sending a Query using NeMo APIs
      • Sending a Query using PyTriton
    • Deploy a LLM model with NeMo APIs
• Deploying the NeMo Models in the NeMo Framework Inference Container
  
  • Supported Model and GPUs
  • Quick User Guide
  • User Guide
    
    • Serve in-framework or TensorRT-LLM model on Triton
    • Sending a Query using NeMo APIs
    • Sending a Query using PyTriton
  • NeMo Export and Deploy Modules and APIs
    
    • Export a LLM model to TensorRT-LLM with NeMo APIs
    • Deploy a LLM model with NeMo APIs
    • Performance Measurements
    • Measuring Model Accuracy using PyTests for Deploy and Export Modules
• Performance
  
  • Performance
  • GPT Results
    
    • Training Accuracy Results
    • Training Performance Results
    • Inference Performance
      
      • Configuration 1: Chatbot Conversation use case
      • Configuration 2: Translation / Style Transfer use case
  • T5 Results
    
    • Training Accuracy Results
    • Training Performance Results
    • Inference Performance
  • MT5 Results
    
    • Training Accuracy Results
    • Training Performance Results
    • Inference Performance
  • BERT Results
    
    • Training Accuracy Results
    • Training Performance Results
  • Llama-2 Results
    
    • Inference Performance
      
      • Configuration 1: Chatbot Conversation use case
      • Configuration 2: Translation / Style Transfer use case
• Changelog
  
  • NeMo Framework 23.11
  • NeMo Framework 23.10
  • NeMo Framework 23.08.03
  • NeMo Framework 23.08.02
  • NeMo Framework 23.08.01
  • NeMo Framework 23.08
  • NeMo Framework 23.07
  • NeMo Framework 23.05
  • NeMo Framework 23.04.1
  • NeMo Framework 23.04
  • NeMo Framework 23.03
  • NeMo Framework 23.01
  • NeMo Framework 22.11
  • NeMo Framework 22.09
  • NeMo Framework 22.08.01
  • NeMo Framework 22.08
  • NeMo Framework 22.06-hotfix.01
  • NeMo Framework 22.06
  • NeMo Framework 22.05.01
  • NeMo Framework 22.05
  • NeMo Framework 22.04
  • NeMo Framework 22.03
  • NeMo Framework 22.02
  • NeMo Framework 22.01
• Known Issues