> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/step5/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.models.step5

Step-5 Preview text and vision forward/training support.

The released primary decoder has 92 layers; the perception encoder has 47.
Checkpoint FP8 expert tensors are dequantized into the selected training dtype.
MTP auxiliary layers, incremental KV-cache generation, and TP are excluded.

The default CUDA attention backend is FA4 on SM90, with native 192-dimensional
heads. It requires flash-attention revision 3c2c6b31e730fd13e7d6f0d384af22684b24c532
or its SM90 GQA backward fix (#2888). CSA uses route-derived forward/reverse
block lists and compact region membership. It skips empty tiles; independent
query selections can still make almost every physical tile active.

CP uses contiguous query shards. SWA exchanges only the preceding window through
point-to-point transfers, with local masked zeros preserving global tile
alignment. CSA gathers original K/V and indexer K/Z; attention backward
reduce-scatters K/V gradients. Packed documents use resetting position IDs.

Validation entry points live in tests/functional_tests/models/step5 and
tests/functional_tests/context_parallel/run_step5_cp.py.

## Submodules

* **[`nemo_automodel.components.models.step5.configuration`](/nemo-automodel/nemo_automodel/components/models/step5/configuration)**
* **[`nemo_automodel.components.models.step5.cp`](/nemo-automodel/nemo_automodel/components/models/step5/cp)**
* **[`nemo_automodel.components.models.step5.fa4`](/nemo-automodel/nemo_automodel/components/models/step5/fa4)**
* **[`nemo_automodel.components.models.step5.flops`](/nemo-automodel/nemo_automodel/components/models/step5/flops)**
* **[`nemo_automodel.components.models.step5.layers`](/nemo-automodel/nemo_automodel/components/models/step5/layers)**
* **[`nemo_automodel.components.models.step5.model`](/nemo-automodel/nemo_automodel/components/models/step5/model)**
* **[`nemo_automodel.components.models.step5.parallelization`](/nemo-automodel/nemo_automodel/components/models/step5/parallelization)**
* **[`nemo_automodel.components.models.step5.rope`](/nemo-automodel/nemo_automodel/components/models/step5/rope)**
* **[`nemo_automodel.components.models.step5.sparse`](/nemo-automodel/nemo_automodel/components/models/step5/sparse)**
* **[`nemo_automodel.components.models.step5.state_dict_adapter`](/nemo-automodel/nemo_automodel/components/models/step5/state_dict_adapter)**
* **[`nemo_automodel.components.models.step5.vision`](/nemo-automodel/nemo_automodel/components/models/step5/vision)**

## Docs

- [nemo_automodel.components.models.step5.configuration](https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/step5/configuration.md)
- [nemo_automodel.components.models.step5.cp](https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/step5/cp.md)
- [nemo_automodel.components.models.step5.fa4](https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/step5/fa4.md)
- [nemo_automodel.components.models.step5.flops](https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/step5/flops.md)
- [nemo_automodel.components.models.step5.layers](https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/step5/layers.md)
- [nemo_automodel.components.models.step5.model](https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/step5/model.md)
- [nemo_automodel.components.models.step5.parallelization](https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/step5/parallelization.md)
- [nemo_automodel.components.models.step5.rope](https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/step5/rope.md)
- [nemo_automodel.components.models.step5.sparse](https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/step5/sparse.md)
- [nemo_automodel.components.models.step5.state_dict_adapter](https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/step5/state_dict_adapter.md)
- [nemo_automodel.components.models.step5.vision](https://docs.nvidia.com/nemo/automodel/v0.5/nemo-automodel/nemo_automodel/components/models/step5/vision.md)