Breaking Changes

0.5.0

FSDP2 Default reduce_dtype Is Now float32

The default MixedPrecisionPolicy built by FSDP2Config now uses reduce_dtype=torch.float32 instead of torch.bfloat16. Forward/backward compute still uses param_dtype=torch.bfloat16, but gradient reduction now accumulates in fp32 to reduce communication-rounding error at larger data-parallel world sizes.

To restore the previous behavior, override the policy explicitly:

1
distributed:
2
  _target_: nemo_automodel.components.distributed.config.FSDP2Config
3
  mp_policy:
4
    _target_: torch.distributed.fsdp.MixedPrecisionPolicy
5
    param_dtype: bfloat16
6
    reduce_dtype: bfloat16
7
    output_dtype: bfloat16

See the mixed-precision training guide for how FSDP2 reduction dtype interacts with model storage dtype and optimizer state.

0.4.0 · 26.04

CLI Signature Change

Before:

automodel <command> <domain> -c <config.yaml>

After:

automodel <config.yaml> [--nproc-per-node N] [--overrides ...]

A short alias am is also available:

am <config.yaml> [--nproc-per-node N] [--overrides ...]

The positional <command> and <domain> arguments have been removed. The recipe class is now specified inside the YAML config through the recipe._target_ key.

YAML Config: New Required recipe Section

All YAML configs now require a top-level recipe: key:

1
recipe:
2
  _target_: nemo_automodel.recipes.llm.train_ft.TrainFinetuneRecipeForNextTokenPrediction

Configs without this key will produce an error with guidance on which target to add.

Available Recipe Targets

Launcher Configuration Moved to YAML

Multi-node launch settings (Kubernetes, NeMo Run) are now configured entirely within the YAML config file rather than through CLI arguments.

If neither section is present, the job runs locally (interactive mode).

SLURM: Script-Based Submission

The slurm: YAML section and all related fields have been removed. SLURM jobs are now submitted with sbatch directly, using a self-contained sbatch script. Copy the reference template and adapt it to your cluster:

$
cp slurm.sub my_cluster.sub
$
# Edit CONFIG, #SBATCH directives, container, mounts, etc.
$
sbatch my_cluster.sub

The script runs torchrun -m nemo_automodel.cli.app on each node, which detects the distributed environment and executes the recipe in-process. All cluster-specific configuration lives in the sbatch script where you can see and edit it directly.

CLI Install Extra

The cli optional dependency extra adds NeMo Run. PyYAML is declared in both the package’s base dependencies and the cli extra:

$
uv venv
$
source .venv/bin/activate
$
uv pip install "nemo-automodel[cli]"

Optional extras are additive, so this still installs all base dependencies, including PyTorch. NeMo Run is already included in the extra and does not need to be installed separately.

Media Dependencies Are Opt-In

The FFmpeg-bearing media dependencies (OpenCV, decord, the Qwen vision utils, and imageio-ffmpeg) are now opt-in extras, installed neither by default nor in the Docker container.

$
uv venv
$
source .venv/bin/activate
$
uv pip install "nemo-automodel[vlm,vlm-media]"              # VLM video / Qwen / Mistral
$
uv pip install "nemo-automodel[diffusion,diffusion-media]"  # diffusion preprocessing / export

Two consequences when upgrading:

• [vlm] alone no longer trains Qwen2.5-VL, Qwen3-Omni, or Mistral VLMs — add vlm-media.
• [all] no longer includes the media extras — add them with uv pip install "nemo-automodel[media]" in the activated environment.

See the Installation Guide for details.

CLI Module Lives Inside the Package

The CLI entry-point lives at nemo_automodel/cli/app.py and is registered as the automodel / am console entry-points. A thin convenience wrapper (app.py) at the repository root is available for running from a source checkout but is not installed as part of the package.

Example Wrapper Scripts Deprecated

The Python wrapper scripts in examples/ (for example, examples/llm_finetune/finetune.py) are deprecated. They now print a deprecation warning and delegate to the recipe directly. Use automodel <config.yaml> instead.