Breaking Changes

0.5.0

Frozen Multimodal FSDP2 Modules Default to Root Sharding

Fully frozen vision and audio towers and multimodal projectors now default to the root FSDP2 policy. Previously, dense frozen vision towers commonly used per-layer FSDP units. Root ownership keeps the collective sequence aligned when some ranks execute a modality branch and other ranks skip it, but it can increase peak unshard memory.

To retain the previous dense vision topology, opt in explicitly:

1
distributed:
2
  strategy: fsdp2
3
  multimodal:
4
    frozen_sharding: per_layer

per_layer requires every rank in the FSDP group to execute or skip the frozen multimodal units the same number of times and in the same order on every microbatch. Use replicate instead to avoid those collectives while keeping a full copy of the frozen parameters on every rank. See YAML Configuration for all three policies and the wrap_outer_model constraint.

FSDP2 Default reduce_dtype Is Now float32

The default MixedPrecisionPolicy built by FSDP2Config now uses reduce_dtype=torch.float32 instead of torch.bfloat16. Forward/backward compute still uses param_dtype=torch.bfloat16, but gradient reduction now accumulates in FP32 to reduce communication-rounding error at larger data-parallel world sizes.

To restore the previous behavior, override the policy explicitly:

1
distributed:
2
  _target_: nemo_automodel.components.distributed.config.FSDP2Config
3
  mp_policy:
4
    _target_: torch.distributed.fsdp.MixedPrecisionPolicy
5
    param_dtype: bfloat16
6
    reduce_dtype: bfloat16
7
    output_dtype: bfloat16

See the mixed-precision training guide for how FSDP2 reduction dtype interacts with model storage dtype and optimizer state.

0.4.0 · 26.04

CLI Signature Change

Before:

automodel <command> <domain> -c <config.yaml>

After:

automodel <config.yaml> [--nproc-per-node N] [--overrides ...]

A short alias am is also available:

am <config.yaml> [--nproc-per-node N] [--overrides ...]

The positional <command> and <domain> arguments have been removed. The recipe class is now specified inside the YAML config through the recipe._target_ key.

YAML Config: New Required recipe Section

All YAML configs now require a top-level recipe: key:

1
recipe:
2
  _target_: nemo_automodel.recipes.llm.train_ft.TrainFinetuneRecipeForNextTokenPrediction

Configs without this key will produce an error with guidance on which target to add.

Available Recipe Targets

Launcher Configuration Moved to YAML

Multi-node launch settings (Kubernetes, NeMo Run) are now configured entirely within the YAML config file rather than through CLI arguments.

If neither section is present, the job runs locally (interactive mode).

Slurm: Script-Based Submission

The slurm: YAML section and all related fields have been removed. Slurm jobs are now submitted with sbatch directly, using a self-contained sbatch script. Copy the reference template and adapt it to your cluster:

$
cp slurm.sub my_cluster.sub
$
# Edit CONFIG, #SBATCH directives, container, mounts, etc.
$
sbatch my_cluster.sub

The script runs torchrun -m nemo_automodel.cli.app on each node, which detects the distributed environment and executes the recipe in-process. All cluster-specific configuration lives in the sbatch script where you can edit it directly.

CLI Install Extra

The cli optional dependency extra adds NeMo Run. PyYAML is declared in both the package’s base dependencies and the cli extra:

$
uv venv
$
source .venv/bin/activate
$
uv pip install "nemo-automodel[cli]"

Optional extras are additive, so this still installs all base dependencies, including PyTorch. NeMo Run is already included in the extra and does not need to be installed separately.

Media Dependencies Are Opt-In

The FFmpeg-bearing media dependencies (OpenCV, decord, the Qwen vision utilities, and imageio-ffmpeg) are now opt-in extras, installed neither by default nor in the Docker container.

$
uv venv
$
source .venv/bin/activate
$
uv pip install "nemo-automodel[vlm,vlm-media]"              # VLM video / Qwen / Mistral
$
uv pip install "nemo-automodel[diffusion,diffusion-media]"  # diffusion preprocessing / export

Two consequences when upgrading:

• [vlm] alone no longer trains Qwen2.5-VL, Qwen3-Omni, or Mistral VLMs. Add vlm-media.
• [all] no longer includes the media extras. Add them with uv pip install "nemo-automodel[media]" in the activated environment.

See the Installation Guide for details.

CLI Module Lives Inside the Package

The CLI entry point lives at nemo_automodel/cli/app.py and is registered as the automodel or am console entry points. A thin convenience wrapper (app.py) at the repository root is available for running from a source checkout but is not installed as part of the package.

Example Wrapper Scripts Deprecated

The Python wrapper scripts in examples/ (for example, examples/llm_finetune/finetune.py) are deprecated. They now print a deprecation warning and delegate to the recipe directly. Use automodel <config.yaml> instead.