Supervised Fine-Tuning (SFT) and Parameter-Efficient Fine-Tuning (PEFT) with NeMo AutoModel
Supervised Fine-Tuning (SFT) and Parameter-Efficient Fine-Tuning (PEFT) with NeMo AutoModel
Introduction
Pretrained language models are general-purpose: they know a lot about language but nothing about your particular domain, terminology, or task. Fine-tuning bridges that gap. You fine-tune the model on your own examples so it produces answers that are accurate and relevant for your use case, without the cost of training a model from scratch. The result is a model optimized for your data that you can evaluate, publish, and deploy. This guide walks you through that process end-to-end with NeMo AutoModel (from installation through training, evaluation, and deployment) using Meta Llama 3.2 1B and the SQuAD v1.1 dataset as a running example.
NeMo AutoModel supports two fine-tuning modes:
- Supervised Fine-Tuning (SFT) updates all model parameters. Use SFT when you need maximum accuracy and have sufficient compute.
- Parameter-Efficient Fine-Tuning (PEFT) using LoRA freezes the base model and trains small low-rank adapters. PEFT reduces trainable parameters to less than 1% of the original model, lowering memory and storage costs.
Workflow Overview
The following table outlines each step in the fine-tuning workflow for SFT and PEFT modes.
Install NeMo AutoModel
Alternatively, if you run into dependency or driver issues, use the pre-built Docker container:
Docker containers are ephemeral, and files written inside the container are lost when it stops. The -v flag in the docker run command above bind-mounts a local checkpoints/ directory into the container so that saved checkpoints persist across runs. For more details, see Save Checkpoints When Using Docker.
For the full set of installation methods, see the Installation Guide.
Configure Your Training Recipe
Training is configured through a YAML config file. For the automodel CLI, the file must name a recipe. The fine-tuning runtime also requires model, dataset, dataloader, loss_fn, optimizer, and distributed sections. If you configure a validation_dataset, also configure a validation_dataloader. The step_scheduler section is optional because it has typed default values, but you should set its global and local batch sizes explicitly for a predictable effective batch. Add peft only for LoRA. For the complete copy-pastable file, see Full Config YAML.
Under the hood, both SFT and PEFT are executed by a recipe, which is a self-contained Python class that integrates model loading, dataset preparation, training, checkpointing, and logging. The fine-tuning recipe is TrainFinetuneRecipeForNextTokenPrediction. The config file tells the recipe what to build; the recipe decides how to build it.
How the Config System Works
NeMo AutoModel configs use a convention borrowed from Hydra: the special _target_ key tells the framework which Python class or function to call, and every other key in the same YAML block is passed as a keyword argument to that call. For example:
At the YAML boundary, RecipeConfig normalizes this target and its fields into an OptimizerConfig wrapper. The recipe then supplies the runtime model parameters when it builds the optimizer:
The _target_ value is a dotted Python import path: the same string you would use in an import statement. The framework resolves it at runtime by importing the module and looking up the attribute. This means you can point _target_ at any class constructor or factory function, and the remaining keys become its arguments.
For raw model, dataset, dataloader, and PEFT sections, inspect the Python signature of _target_. For typed sections such as optimizer, loss, step scheduler, learning rate (LR) scheduler, and checkpoint, inspect the corresponding config dataclass fields because RecipeConfig validates and owns those settings.
From YAML to running code, here is the path a config takes through the framework:
Raw ConfigNode sections call .instantiate() inside the relevant recipe builder. .instantiate() does two things:
- Resolves
_target_: Imports the Python path and obtains the callable (class or function). - Calls it: Passes every other key in the section as a keyword argument.
Nested _target_ blocks (such as a configured collate_fn inside dataloader) are recursively instantiated the same way. In contrast, RecipeConfig converts recognized sections to typed configs that own construction through .build(...): optimizer receives the model and device mesh, step scheduler receives the dataloader, data-parallel (DP) size, and local batch size, and checkpoint receives runtime ranks and meshes.
The recipe key. Every config file includes a top-level recipe key that tells the CLI which recipe class to run. You can write it as a short name or as a fully qualified Python path, and both resolve to the same class:
The short name form is a convenience. The CLI scans all recipe modules under nemo_automodel.recipes and matches the bare class name. If you invoke the recipe script directly with torchrun instead of the automodel CLI, the recipe key is not required because the script itself is the recipe.
Not every section uses _target_. step_scheduler and checkpoint are converted into StepSchedulerConfig and CheckpointingConfig, whose .build(...) methods receive runtime-only values from the recipe. The distributed mapping is parsed into typed strategy, parallelism, pipeline, and Mixture-of-Experts (MoE) configs before DistributedSetup.build(...) creates the meshes.
Model
This guide uses Meta Llama 3.2 1B as a running example. Replace pretrained_model_name_or_path with any supported Hugging Face model ID.
About Llama 3.2 1B
Llama is a family of decoder-only transformer models developed by Meta. The 1B variant is a compact model suitable for research and edge deployment, featuring rotary position embeddings (RoPE), grouped-query attention (GQA), and SwiGLU activations.
Accessing Gated Models
Some Hugging Face models are gated. If the model page shows a Request access button, complete the following steps:
- Log in with your Hugging Face account and accept the license.
- Ensure the token you use (from
huggingface-cli loginorHF_TOKEN) belongs to the approved account.
Pulling a gated model without an authorized token triggers a 403 error.
Dataset
This guide uses SQuAD v1.1 as a running example. Swap the dataset by changing _target_ and the dataset arguments. See Integrate Your Own Text Dataset and Dataset Overview: LLM, VLM, and Retrieval Datasets.
About SQuAD v1.1
The Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset where each example consists of a Wikipedia passage, a question, and an answer span. SQuAD v1.1 guarantees all questions are answerable from the context, making it suitable for straightforward fine-tuning.
Example:
Optional PEFT
Including a peft: section enables LoRA fine-tuning. Remove it entirely to run SFT instead. See Switch Between SFT and PEFT.
QLoRA for Quantized Low-Rank Adaptation
If GPU memory is a constraint, QLoRA combines LoRA with 4-bit NormalFloat (NF4) quantization to reduce memory usage by up to 75% compared to full-parameter SFT in 16-bit precision, while maintaining comparable quality to standard LoRA.
QLoRA requires bitsandbytes, which is provided by NeMo AutoModel’s cuda_source
install profile. Add it to the active environment created above before running a
QLoRA recipe:
To enable QLoRA, add a quantization: section alongside the peft: section in your config. The example keeps the "*_proj" projection-layer pattern and increases dim from 8 to 16 for additional adapter capacity.
Training Schedule
step_scheduler has no _target_. RecipeConfig converts it to StepSchedulerConfig, and the recipe builds the runtime scheduler with the training dataloader, data-parallel group size, and local_batch_size. Gradient accumulation is derived rather than configured directly:
With the values above, a single DP rank accumulates 8 micro-batches; 8 DP ranks require no accumulation. global_batch_size must be divisible by local_batch_size × DP size.
Most example recipes use bf16 training by default for memory and throughput. If you are running long fine-tuning, especially full-parameter SFT, and you require a higher-precision optimizer state, set model.dtype: float32 explicitly when using torch.optim.Adam or AdamW. The deprecated model.torch_dtype key remains supported. The mixed-precision compute policy does not set optimizer-state precision. See the Mixed-Precision Training Guide for the recommended Transformer Engine (TE) and torch AdamW patterns.
Full Config YAML
finetune_config.yaml (Click to Expand)
Save as finetune_config.yaml. This config runs PEFT (LoRA). To run SFT instead, remove the peft: section. Use this complete config rather than the current checked-in Llama SFT YAML. That file still nests the seed under rng:, which this recipe ignores, and its commented W&B block uses unsupported save_dir instead of dir.
Fine-Tune the Model
You can run the recipe using the automodel CLI or directly with torchrun (advanced).
The --nproc-per-node=8 flag specifies the number of GPUs per node. Adjust as needed. For a single GPU, omit the --nproc-per-node option.
Invoke the Recipe Script Directly (Advanced)
Alternatively, you can invoke the train_ft.py recipe script directly by using torchrun:
Sample Output
Running the recipe with the automodel application or by invoking the recipe script directly produces the following log output:
Each log line reports the current loss, gradient norm, peak GPU memory, and tokens per second (TPS). Small fluctuations between steps (for example, 0.2057 to 0.2557) are normal. Look at the overall downward trend rather than individual values.
Checkpoint Contents
Checkpoints are saved as Hugging Face-compatible safetensors. For SFT, save_consolidated: final is the default. Intermediate checkpoints contain sharded model weights and a generated model/consolidate.sh helper. The final checkpoint also contains model/consolidated/ for Transformers, vLLM, lm-eval-harness, and other Hugging Face ecosystem tools. Set save_consolidated: false to keep every checkpoint sharded, and then run the helper script for whichever checkpoint you want to export. Use save_consolidated: every (or the legacy value true) only when you intentionally want inline Hugging Face export at every checkpoint save.
PEFT checkpoints contain only the adapter weights (megabytes instead of gigabytes) and are saved directly under model/. They do not use model/consolidate.sh. At inference time, you can load the original base model and apply the adapter on top. To produce a standalone checkpoint instead, run python tools/merge_lora.py --base-model <base-model> --adapter-path <checkpoint>/model --output-dir <output>. For AutoModel PEFT-v5 checkpoints with grouped-expert adapters, add --automodel-native-export to restore the adapter through the model family’s state-dict adapter and write canonical sharded Hugging Face weights. This distinction affects every downstream step, including inference, publishing, and deployment.
Checkpoint Directory Structure
SFT checkpoint:
PEFT checkpoint:
Run Inference
Inference uses the Hugging Face generate API. Because exported SFT checkpoints are self-contained while PEFT checkpoints store only adapter weights (see Checkpoint Contents), the loading procedure differs between the two modes.
SFT Inference
For an intermediate checkpoint, or any run with save_consolidated: false, first run the generated helper for the checkpoint you want to load:
The exported SFT checkpoint at model/consolidated/ is a complete Hugging Face model and can be loaded directly:
PEFT Inference
PEFT adapters must be loaded on top of the base model. Because the base NeMo AutoModel installation does not include the Hugging Face PEFT library, install a tested version in your active environment before running the inference code:
LoRA adapters for models with fused MoE expert weights (for example, Qwen3-MoE, MiniMax-M2, Nemotron 3) are exported in the ParamWrapper layout that peft has used since 0.19.1, when peft changed it (huggingface/peft#3165); that is why the step above installs peft>=0.19.1. If such an adapter has to be loaded by a consumer pinned to an older peft version, set checkpoint.legacy_paramwrapper_layout: true in the training config to export the pre-0.19.1 layout instead. An adapter exported that way does not load with peft 0.19.1 or newer.
Evaluate the Fine-Tuned Model
Validation Loss During Training
The recipe automatically computes validation loss at the interval set by val_every_steps. Look for [val] lines in the training log:
A decreasing validation loss across checkpoints indicates the model is learning. If validation loss plateaus or increases while training loss continues to drop, the model might be overfitting, so consider stopping earlier or reducing the learning rate.
Post-Training Evaluation Using lm-eval-harness
For task-specific benchmarks (for example, MMLU, GSM8K, and HellaSwag accuracy), use lm-eval-harness with the fine-tuned checkpoint. For an intermediate SFT checkpoint, or any SFT run with save_consolidated: false, run bash checkpoints/epoch_0_step_20/model/consolidate.sh before pointing evaluation at model/consolidated/. Create and activate a clean evaluation environment, and then install both model backends before running either command:
The quoted lm_eval[hf,vllm] requirement installs both backends and their
dependencies. The SFT example uses the vllm backend for faster evaluation.
The PEFT example uses the hf backend with lm-eval’s built-in PEFT support to
load the adapter on top of the base model.
Run lm_eval on the base model before fine-tuning to establish a baseline, and then compare it against the fine-tuned checkpoint.
Publish to the Hugging Face Hub
Fine-tuned checkpoints and PEFT adapters are stored in Hugging Face-native format and can be uploaded directly to the Hub. For an intermediate SFT checkpoint, or any SFT run with save_consolidated: false, upload model/consolidated/ after running the generated consolidation helper.
- Install the Hugging Face Hub library (if not already installed):
- Log in to Hugging Face:
- Upload the folder:
SFT checkpoint:
PEFT adapter:
After uploading, load the checkpoint or adapter directly from the Hugging Face Hub:
SFT:
PEFT:
Deploy with vLLM
vLLM is an efficient inference engine for production deployment of LLMs.
Use a vLLM-compatible environment. For NVIDIA GPUs, create and activate a virtual environment before installing vLLM:
You can skip these commands if your active environment already includes vLLM.
SFT Checkpoint with vLLM
For an intermediate checkpoint, or any run with save_consolidated: false, run the generated model/consolidate.sh helper before serving from model/consolidated/:
PEFT Adapter with vLLM
vLLM can load the NeMo AutoModel LoRA adapter directly on top of the original base model. Enable LoRA when constructing LLM, and then select the adapter for the request with LoRARequest:
Full Configuration Reference
This section documents all available config fields for the fine-tuning recipe. For the Quickstart config, see Configure Your Training Recipe.
Switch Between SFT and PEFT
The peft: section controls which mode runs, as described in the following table.
All other config sections remain the same for both modes.
Full Configuration
Full Config
Config Field Reference
The following table describes the config fields for the fine-tuning recipe.
For the fine-tuning recipe itself, see train_ft.py.
Per-Parameter-Group Learning Rate
To give a subset of parameters a different learning rate or weight decay (for example, a smaller learning rate for MoE router parameters), add param_group_overrides to the optimizer block. Each entry matches parameters by name and applies a multiplier that the LR scheduler folds into that group every step:
Each parameter joins the group of the first override whose pattern matches its name; parameters matching no pattern keep the base learning rate. A pattern that matches nothing is skipped with a warning. This works for the standard torch.optim optimizers; Dion-family optimizers do their own parameter grouping and ignore param_group_overrides.
Distributed Training with TP, PP, CP, and EP
The distributed: section controls how the model and data are split across GPUs. NeMo AutoModel supports five parallelism dimensions, each of which slices the workload differently, as described in the following table:
These dimensions compose with each other. The relationship between them and total GPU count is:
When dp_size is set to null (the default), it is inferred automatically:
EP does not appear in this formula. Experts are distributed across the DP×CP rank groups, with the constraint that (dp_size × cp_size) must be divisible by ep_size.
Data Parallelism as the Default
Data parallelism is the default. With strategy: fsdp2, FSDP2 shards both model parameters and optimizer states across the DP group, so memory usage shrinks as you add GPUs:
Tensor Parallelism
TP splits weight matrices across GPUs within a single node. Set tp_size to the number of GPUs you want to shard over. The value is typically 2, 4, or 8 and should divide evenly into the number of attention heads:
sequence_parallel: true extends TP to also shard activation memory along the sequence dimension, further reducing per-GPU memory at the cost of additional communication.
Pipeline Parallelism
PP assigns groups of layers to different GPUs and streams micro-batches through the stages. It requires an additional nested pipeline: section:
PP is available only when the model declares pipeline-parallel support and defines a _pp_plan that tells the framework how to split layers into stages. Do not assume every built-in model supports PP; check the model’s declared capabilities before setting pp_size > 1.
Context Parallelism
CP splits the sequence across GPUs, which is useful for very long contexts that exceed single-GPU memory. Set cp_size to the target split factor:
When cp_size > 1, fused RoPE is automatically disabled. Some models also require the Transformer Engine (TE) attention backend for CP with packed sequences. The framework will raise an error with instructions if this applies.
Expert Parallelism for MoE Models
EP distributes MoE experts across GPUs. Set ep_size to the number of GPUs that share the full set of experts:
EP only applies to Mixture-of-Experts models (such as Qwen3-MoE, Mixtral, and DeepSeek-V3). For dense models, leave ep_size at 1 or omit it.
Combine Multiple Dimensions
You can combine TP, PP, CP, and EP in a single config. For example, a large MoE model on a multi-node cluster might use:
When choosing a combination, keep these rules in mind:
world_sizemust be evenly divisible bypp_size × tp_size × cp_sizeso that the quotient is an integerdp_size.(dp_size × cp_size) % ep_size == 0, where EP shares the DP×CP groups.- TP within a node, PP across nodes is the typical layout. TP requires fast NVLink bandwidth, whereas PP tolerates higher latency.
- Start simple. Use DP-only first. Add TP if the model does not fit on one GPU. Add PP for very large models. Add CP for long sequences. Add EP only for MoE architectures.
Next Steps
- Integrate Your Own Text Dataset to swap the SQuAD example for your own data.
- Recipes and End-to-End Examples to browse the validated recipe guides available in NeMo AutoModel.
- Dataset Overview: LLM, VLM, and Retrieval Datasets to see all supported dataset types across LLM, VLM, and retrieval tasks.
- Knowledge Distillation to distill a fine-tuned model into a smaller one.