> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# Breaking Changes

## 0.5.0

### FSDP2 Default `reduce_dtype` Is Now `float32`

The default [`MixedPrecisionPolicy`](https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_shard.html) built by `FSDP2Config` now uses `reduce_dtype=torch.float32` instead of `torch.bfloat16`. Forward/backward compute still uses `param_dtype=torch.bfloat16`, but gradient reduction now accumulates in fp32 to reduce communication-rounding error at larger data-parallel world sizes.

To restore the previous behavior, override the policy explicitly:

```yaml
distributed:
  _target_: nemo_automodel.components.distributed.config.FSDP2Config
  mp_policy:
    _target_: torch.distributed.fsdp.MixedPrecisionPolicy
    param_dtype: bfloat16
    reduce_dtype: bfloat16
    output_dtype: bfloat16
```

See the [mixed-precision training guide](/development/mixed-precision-training) for how FSDP2 reduction dtype interacts with model storage dtype and optimizer state.

## 0.4.0 · 26.04

### CLI Signature Change

**Before:**

```
automodel <command> <domain> -c <config.yaml>
```

**After:**

```
automodel <config.yaml> [--nproc-per-node N] [--overrides ...]
```

A short alias `am` is also available:

```
am <config.yaml> [--nproc-per-node N] [--overrides ...]
```

The positional `<command>` and `<domain>` arguments have been removed. The recipe
class is now specified inside the YAML config through the `recipe._target_` key.

### YAML Config: New Required `recipe` Section

All YAML configs now require a top-level `recipe:` key:

```yaml
recipe:
  _target_: nemo_automodel.recipes.llm.train_ft.TrainFinetuneRecipeForNextTokenPrediction
```

Configs without this key will produce an error with guidance on which target to add.

#### Available Recipe Targets

| Use Case                       | `_target_`                                                                              |
| ------------------------------ | --------------------------------------------------------------------------------------- |
| LLM fine-tuning / pre-training | `nemo_automodel.recipes.llm.train_ft.TrainFinetuneRecipeForNextTokenPrediction`         |
| VLM fine-tuning                | `nemo_automodel.recipes.vlm.finetune.FinetuneRecipeForVLM`                              |
| Knowledge distillation         | `nemo_automodel.recipes.llm.kd.KnowledgeDistillationRecipeForNextTokenPrediction`       |
| Benchmarking                   | `nemo_automodel.recipes.llm.benchmark.BenchmarkingRecipeForNextTokenPrediction`         |
| Sequence classification        | `nemo_automodel.recipes.llm.train_seq_cls.TrainFinetuneRecipeForSequenceClassification` |
| Bi-encoder training            | `nemo_automodel.recipes.retrieval.train_bi_encoder.TrainBiEncoderRecipe`                |

### Launcher Configuration Moved to YAML

Multi-node launch settings (Kubernetes, NeMo Run) are now configured
entirely within the YAML config file rather than through CLI arguments.

| Launcher                  | YAML section                         |
| ------------------------- | ------------------------------------ |
| Kubernetes (via SkyPilot) | `skypilot:` with `cloud: kubernetes` |
| NeMo Run                  | `nemo_run:`                          |

If neither section is present, the job runs locally (interactive mode).

### SLURM: Script-Based Submission

The `slurm:` YAML section and all related fields have been removed. SLURM
jobs are now submitted with `sbatch` directly, using a self-contained sbatch
script. Copy the reference template and adapt it to your cluster:

```bash
cp slurm.sub my_cluster.sub
# Edit CONFIG, #SBATCH directives, container, mounts, etc.
sbatch my_cluster.sub
```

The script runs `torchrun -m nemo_automodel.cli.app` on each node, which
detects the distributed environment and executes the recipe in-process.
All cluster-specific configuration lives in the sbatch script where you can
see and edit it directly.

### CLI Install Extra

The `cli` optional dependency extra adds NeMo Run. PyYAML is declared in both
the package's base dependencies and the `cli` extra:

```bash
uv venv
source .venv/bin/activate
uv pip install "nemo-automodel[cli]"
```

Optional extras are additive, so this still installs all base dependencies,
including PyTorch. NeMo Run is already included in the extra and does not need
to be installed separately.

### Media Dependencies Are Opt-In

The FFmpeg-bearing media dependencies (OpenCV, decord, the Qwen vision utils, and
imageio-ffmpeg) are now opt-in extras, installed neither by default nor in the
Docker container.

```bash
uv venv
source .venv/bin/activate
uv pip install "nemo-automodel[vlm,vlm-media]"              # VLM video / Qwen / Mistral
uv pip install "nemo-automodel[diffusion,diffusion-media]"  # diffusion preprocessing / export
```

Two consequences when upgrading:

* `[vlm]` alone no longer trains Qwen2.5-VL, Qwen3-Omni, or Mistral VLMs — add `vlm-media`.
* `[all]` no longer includes the media extras — add them with `uv pip install "nemo-automodel[media]"` in the activated environment.

See the [Installation Guide](/get-started/installation#media-extras-video--image-decode) for details.

### CLI Module Lives Inside the Package

The CLI entry-point lives at `nemo_automodel/cli/app.py` and is registered as
the `automodel` / `am` console entry-points. A thin convenience wrapper
(`app.py`) at the repository root is available for running from a source
checkout but is **not** installed as part of the package.

### Example Wrapper Scripts Deprecated

The Python wrapper scripts in `examples/` (for example, `examples/llm_finetune/finetune.py`)
are deprecated. They now print a deprecation warning and delegate to the recipe
directly. Use `automodel <config.yaml>` instead.