> This page is for version 0.4.0 · 26.04.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.models.step5

Step-5 Preview text and vision forward/training support.

The released primary decoder has 92 layers; the perception encoder has 47.
Checkpoint FP8 expert tensors are dequantized into the selected training dtype.
MTP auxiliary layers, incremental KV-cache generation, and TP are excluded.

The default CUDA attention backend is FA4 on SM90, with native 192-dimensional
heads. It requires flash-attention revision 3c2c6b31e730fd13e7d6f0d384af22684b24c532
or its SM90 GQA backward fix (#2888). CSA uses route-derived forward/reverse
block lists and compact region membership. It skips empty tiles; independent
query selections can still make almost every physical tile active.

CP uses contiguous query shards. SWA exchanges only the preceding window through
point-to-point transfers, with local masked zeros preserving global tile
alignment. CSA gathers original K/V and indexer K/Z; attention backward
reduce-scatters K/V gradients. Packed documents use resetting position IDs.

Validation entry points live in tests/functional_tests/models/step5 and
tests/functional_tests/context_parallel/run_step5_cp.py.

## Submodules

* **[`nemo_automodel.components.models.step5.configuration`](/nemo-automodel/nemo_automodel/components/models/step5/configuration)**
* **[`nemo_automodel.components.models.step5.cp`](/nemo-automodel/nemo_automodel/components/models/step5/cp)**
* **[`nemo_automodel.components.models.step5.fa4`](/nemo-automodel/nemo_automodel/components/models/step5/fa4)**
* **[`nemo_automodel.components.models.step5.flops`](/nemo-automodel/nemo_automodel/components/models/step5/flops)**
* **[`nemo_automodel.components.models.step5.layers`](/nemo-automodel/nemo_automodel/components/models/step5/layers)**
* **[`nemo_automodel.components.models.step5.model`](/nemo-automodel/nemo_automodel/components/models/step5/model)**
* **[`nemo_automodel.components.models.step5.parallelization`](/nemo-automodel/nemo_automodel/components/models/step5/parallelization)**
* **[`nemo_automodel.components.models.step5.rope`](/nemo-automodel/nemo_automodel/components/models/step5/rope)**
* **[`nemo_automodel.components.models.step5.sparse`](/nemo-automodel/nemo_automodel/components/models/step5/sparse)**
* **[`nemo_automodel.components.models.step5.state_dict_adapter`](/nemo-automodel/nemo_automodel/components/models/step5/state_dict_adapter)**
* **[`nemo_automodel.components.models.step5.vision`](/nemo-automodel/nemo_automodel/components/models/step5/vision)**