Important
NeMo 2.0 is an experimental feature and currently released in the dev container only: nvcr.io/nvidia/nemo:dev. Please refer to NeMo 2.0 overview for information on getting started.
Model Alignment
- Obtain a Pretrained Model
- Model Alignment by Supervised Fine-Tuning (SFT)
- Model Alignment by RLHF
- Model Alignment by SteerLM Method
- SteerLM 2.0: Iterative Training for Attribute-Conditioned Language Model Alignment
- Model Alignment by Direct Preference Optimization (DPO)
- Model Alignment by Self-Play Fine-Tuning (SPIN)
- Fine-tuning Stable Diffusion with DRaFT+
- Constitutional AI: Harmlessness from AI Feedback
- Prerequisite Obtaining a Pre-Trained Model
This section provides instructions on how to download pre-trained LLMs in .nemo format. The following section will use these base LLMs for further fine-tuning and alignment.
- Model Alignment by Supervised Fine-Tuning (SFT)
In this section, we walk you through the most straightforward alignment method. We use a supervised dataset in the prompt-response pairs format to fine-tune the base model according to the desired behavior.
- Model Alignment by RLHF
RLHF is the next step up in alignment and is still responsible for most state-of-the-art chat models. In this section, we walk you through the process of RLHF alignment, including training a reward model and RLHF training with the PPO algorithm.
- Model Alignment by SteerLM Method
SteerLM is a novel approach developed by NVIDIA. SteerLM simplifies alignment compared to RLHF. It is based on SFT, but allows user-steerable AI by enabling you to adjust attributes at inference time.
- Model Alignment by SteerLM 2.0 Method
SteerLM 2.0 is an extenstion to SteerLM method that introduces an iterative training procedure to explicitly enforce the generated responses to follow the desired attribute distribution.
- Model Alignment by Direct Preference Optimization (DPO)
DPO is a simpler alignment method compared to RLHF. DPO introduces a novel parameterization of the reward model in RLHF. This parameterization allows us to extract the corresponding optimal policy.
- Fine-tuning Stable Diffusion with DRaFT+
DRaFT+ is an algorithm for fine-tuning text-to-image generative diffusion models. It achieves this by directly backpropagating through a reward model. This approach addresses the mode collapse issues from the original DRaFT algorithm and improves diversity through regularization.
- Constitutional AI: Harmlessness from AI Feedback
CAI, an alignment method developed by Anthropic, enables the incorporation of AI feedback for aligning LLMs. This feedback is grounded in a small set of principles (referred to as the ‘Constitution’) that guide the model toward desired behaviors, emphasizing helpfulness, honesty, and harmlessness.