> This page is for version Nightly (default).
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# Step-5-Preview

> Step-5-Preview vision-language architecture, packed context-parallel inputs, and current preprocessing scope in NeMo AutoModel.

The [Step 5 Preview](https://huggingface.co/stepfun-ai/Step-5-Preview-BF16)
multimodal checkpoint declares `Step5ForConditionalGeneration` with model type
`step5_vl`. It combines the
[Step 5 language backbone](/model-coverage/large-language-models/stepfun-ai/Step-5-Preview)
with a perception encoder and an image-token projector.

## Model Reference

### Model Architecture

| Property                      | Value                                                                                     |
| ----------------------------- | ----------------------------------------------------------------------------------------- |
| **Architecture**              | `Step5ForConditionalGeneration`                                                           |
| Checkpoint                    | [`stepfun-ai/Step-5-Preview-BF16`](https://huggingface.co/stepfun-ai/Step-5-Preview-BF16) |
| Language backbone             | 92-layer MoE decoder with SWA and CSA                                                     |
| Vision encoder                | 47 layers, width 1,536, 16 attention heads                                                |
| Configured image / patch size | 728 / 14 pixels                                                                           |
| Projection                    | Two stride-2 convolutions, followed by a 6,144-to-4,096 linear projection                 |
| Image inputs                  | Overview images and optional image crops                                                  |
| Context / expert parallelism  | Supported by the model implementation                                                     |
| Pipeline / tensor parallelism | Not supported for this multimodal class                                                   |

The model accepts already-preprocessed `pixel_values`, optional
`patch_pixel_values`, and per-image `num_patches`, together with the matching
image placeholders in the text. Projected image tokens replace those
placeholders before the language backbone executes.

## Packed Sequences and Context Parallelism

The packed path preserves each document's position reset, loss mask, image
placeholder span, and crop ownership. CP shards the resulting language
sequence into contiguous query ranges. The document mask applies to image
tokens as well as text tokens, including when an image span or document crosses
a CP boundary.

## Current Scope

The checkpoint does not include a Hugging Face image processor, so raw-image
datasets need a compatible processor before training. This page does not
provide a ready-to-run image SFT recipe.

The full-depth H100 benchmark recipe on the
[language model page](/model-coverage/large-language-models/stepfun-ai/Step-5-Preview#full-depth-h100-benchmark)
covers the language backbone with mock text and excludes vision compute.

## Related Resources

* [Step-5-Preview](https://huggingface.co/stepfun-ai/Step-5-Preview)
* [Hugging Face checkpoint](https://huggingface.co/stepfun-ai/Step-5-Preview-BF16)
* [Language architecture, attention, and benchmark](/model-coverage/large-language-models/stepfun-ai/Step-5-Preview)
* [Model implementation](https://github.com/NVIDIA-NeMo/Automodel/blob/main/nemo_automodel/components/models/step5/model.py)
* [Multimodal dataset guide](/datasets/multi-modal-dataset)