Breaking Changes

0.5.0

FSDP2 Default `reduce_dtype` Is Now `float32`

The default MixedPrecisionPolicy built by FSDP2Config now uses reduce_dtype=torch.float32 instead of torch.bfloat16. Forward/backward compute still uses param_dtype=torch.bfloat16, but gradient reduction now accumulates in fp32 to reduce communication-rounding error at larger data-parallel world sizes.

To restore the previous behavior, override the policy explicitly:

1 distributed:
2   _target_: nemo_automodel.components.distributed.config.FSDP2Config
3   mp_policy:
4     _target_: torch.distributed.fsdp.MixedPrecisionPolicy
5     param_dtype: bfloat16
6     reduce_dtype: bfloat16
7     output_dtype: bfloat16

See the mixed-precision training guide for how FSDP2 reduction dtype interacts with model storage dtype and optimizer state.

0.4.0 · 26.04

CLI Signature Change

Before:

automodel <command> <domain> -c <config.yaml>

After:

automodel <config.yaml> [--nproc-per-node N] [--overrides ...]

A short alias am is also available:

am <config.yaml> [--nproc-per-node N] [--overrides ...]

The positional <command> and <domain> arguments have been removed. The recipe class is now specified inside the YAML config through the recipe._target_ key.

YAML Config: New Required `recipe` Section

All YAML configs now require a top-level recipe: key:

1 recipe:
2   _target_: nemo_automodel.recipes.llm.train_ft.TrainFinetuneRecipeForNextTokenPrediction

Configs without this key will produce an error with guidance on which target to add.

Available Recipe Targets

Use Case	`_target_`
LLM fine-tuning / pre-training	`nemo_automodel.recipes.llm.train_ft.TrainFinetuneRecipeForNextTokenPrediction`
VLM fine-tuning	`nemo_automodel.recipes.vlm.finetune.FinetuneRecipeForVLM`
Knowledge distillation	`nemo_automodel.recipes.llm.kd.KnowledgeDistillationRecipeForNextTokenPrediction`
Benchmarking	`nemo_automodel.recipes.llm.benchmark.BenchmarkingRecipeForNextTokenPrediction`
Sequence classification	`nemo_automodel.recipes.llm.train_seq_cls.TrainFinetuneRecipeForSequenceClassification`
Biencoder training	`nemo_automodel.recipes.biencoder.train_biencoder.TrainBiencoderRecipe`

Launcher Configuration Moved to YAML

Multi-node launch settings (Kubernetes, NeMo Run) are now configured entirely within the YAML config file rather than through CLI arguments.

Launcher	YAML section
Kubernetes	`k8s:`
NeMo Run	`nemo_run:`

If none of these sections are present, the job runs locally (interactive mode).

SLURM: Script-Based Submission

The slurm: YAML section and all related fields have been removed. SLURM jobs are now submitted with sbatch directly, using a self-contained sbatch script. Copy the reference template and adapt it to your cluster:

$ cp slurm.sub my_cluster.sub
$ # Edit CONFIG, #SBATCH directives, container, mounts, etc.
$ sbatch my_cluster.sub

The script runs torchrun -m nemo_automodel.cli.app on each node, which detects the distributed environment and executes the recipe in-process. All cluster-specific configuration lives in the sbatch script where you can see and edit it directly.

Lightweight CLI-Only Install

A new automodel[cli] install extra is available for login nodes or environments where you only need to submit jobs (SLURM, k8s, NeMo-Run) without running training locally:

pip install nemo-automodel[cli]

This installs only pyyaml — no PyTorch, no CUDA dependencies. It is enough to submit jobs through SLURM or Kubernetes. If you also need NeMo Run, install it separately (pip install nemo-run). If you try to run a local/interactive job with the CLI-only install, you will get a clear error message with instructions to install the full package.

Media Dependencies Are Opt-In

The FFmpeg-bearing media dependencies (OpenCV, decord, the Qwen vision utils, and imageio-ffmpeg) are now opt-in extras, installed neither by default nor in the Docker container.

$ pip install "nemo-automodel[vlm,vlm-media]"              # VLM video / Qwen / Mistral
$ pip install "nemo-automodel[diffusion,diffusion-media]"  # diffusion preprocessing / export

Two consequences when upgrading:

[vlm] alone no longer trains Qwen2.5-VL, Qwen3-Omni, or Mistral VLMs — add vlm-media.
[all] no longer includes the media extras — add them with pip install nemo-automodel[media].

See the Installation Guide for details.

CLI Module Lives Inside the Package

The CLI entry-point lives at nemo_automodel/cli/app.py and is registered as the automodel / am console entry-points. A thin convenience wrapper (app.py) at the repository root is available for running from a source checkout but is not installed as part of the package.

Example Wrapper Scripts Deprecated

The Python wrapper scripts in examples/ (for example, examples/llm_finetune/finetune.py) are deprecated. They now print a deprecation warning and delegate to the recipe directly. Use automodel <config.yaml> instead.