Sequence Classification (SFT/PEFT) with NeMo AutoModel

Introduction

Sequence classification tasks (for example, sentiment analysis, topic classification, and GLUE tasks) map input text to a discrete label. NeMo AutoModel provides a lightweight recipe specialized for this setting that integrates with popular pretrained model formats and dataset sources. Integration with Hugging Face is supported.

This guide shows how to train a sequence classification model using the TrainFinetuneRecipeForSequenceClassification recipe, including optional Parameter-Efficient Fine-Tuning (PEFT) with LoRA.

Quickstart

Use the example config for GLUE MRPC with RoBERTa-large and LoRA:

$
uv run automodel examples/llm_seq_cls/glue/mrpc_roberta_lora.yaml

• Loads roberta-large with num_labels: 2
• Builds GLUE MRPC datasets (train/validation)
• Optionally, enables LoRA via the peft block
• Trains and validates per step_scheduler

What is the Sequence Classification Recipe?

TrainFinetuneRecipeForSequenceClassification is a config-driven trainer that orchestrates:

• Model and optimizer construction
• Dataset/Dataloader setup
• Training and validation loops
• Checkpointing and logging

It follows the same design as the SFT recipe in the fine-tune guide, but uses a standard cross-entropy classification loss and a simplified batching pipeline.

Minimal Config Anatomy

1
# GLUE MRPC with RoBERTa-large + LoRA
2
step_scheduler:
3
  global_batch_size: 32
4
  local_batch_size: 32
5
  ckpt_every_steps: 200
6
  val_every_steps: 100
7
  num_epochs: 2
8
  max_steps: 10
9
10
dist_env:
11
  backend: nccl
12
  timeout_minutes: 1
13
14
model:
15
  _target_: nemo_automodel.NeMoAutoModelForSequenceClassification.from_pretrained
16
  pretrained_model_name_or_path: roberta-large
17
  num_labels: 2
18
19
checkpoint:
20
  enabled: true
21
  checkpoint_dir: checkpoints/
22
  model_save_format: safetensors
23
  save_consolidated: final
24
25
distributed:
26
  strategy: fsdp2
27
  dp_size: null
28
  dp_replicate_size: null
29
  tp_size: 1
30
  cp_size: 1
31
  sequence_parallel: false
32
  autocast_dtype: bfloat16
33
34
peft:
35
  _target_: nemo_automodel.components._peft.lora.PeftConfig
36
  target_modules:
37
  - "*.query"
38
  - "*.value"
39
  dim: 8
40
  alpha: 16
41
  dropout: 0.1
42
43
freeze_config:
44
  unfreeze_modules:
45
  - glob: "*classifier"
46
47
dataset:
48
  _target_: nemo_automodel.components.datasets.llm.seq_cls.GLUE_MRPC
49
  split: train
50
51
dataloader:
52
  _target_: torchdata.stateful_dataloader.StatefulDataLoader
53
  collate_fn: nemo_automodel.components.datasets.utils.default_collater
54
55
validation_dataset:
56
  _target_: nemo_automodel.components.datasets.llm.seq_cls.GLUE_MRPC
57
  split: validation
58
59
validation_dataloader:
60
  _target_: torchdata.stateful_dataloader.StatefulDataLoader
61
  collate_fn: nemo_automodel.components.datasets.utils.default_collater
62
63
optimizer:
64
  _target_: torch.optim.AdamW
65
  betas: [0.9, 0.999]
66
  eps: 1e-8
67
  lr: 3.0e-4
68
  weight_decay: 0

Dataset Notes

• nemo_automodel.components.datasets.llm.seq_cls provides GLUE_MRPC, the implemented sentence-pair adapter. It tokenizes (sentence1, sentence2) with truncation; default_collater pads each batch.
• For other datasets, including single-sentence datasets such as yelp_review_full or imdb, provide a custom dataset class or factory through _target_. Accept a tokenizer argument so the recipe can inject the model tokenizer, and return input_ids, attention_mask, a one-element labels list, and the ___PAD_TOKEN_IDS___ mapping used by default_collater for each sample. Use GLUE_MRPC as the adapter pattern.

LoRA (PEFT) Settings

• target_modules: Glob to select linear layers (for example, "*.proj").
• dim (rank), alpha, dropout: Tune per model and compute budget. Values dim=8, alpha=16, dropout=0.1 are a good starting point for RoBERTa.
• freeze_config.unfreeze_modules: Keeps the classification head fully trainable while PEFT freezes other non-LoRA parameters.
• distributed.autocast_dtype: Runs the forward pass in the selected compute dtype while trainable parameters retain their configured storage dtype, including when single-rank FSDP is skipped.
• The recipe automatically applies the adapters; no additional code changes are required.

Running on Multiple GPUs

$
uv run automodel examples/llm_seq_cls/glue/mrpc_roberta_lora.yaml --nproc-per-node 2

You can adjust the number of GPUs as necessary using the --nproc-per-node knob.