Skip to navigation
Model CoverageVision Language ModelsStepFun AIStep-5-Preview

Step-5-Preview

View as Markdown

The Step 5 Preview multimodal checkpoint declares Step5ForConditionalGeneration with model type step5_vl. It combines the Step 5 language backbone with a perception encoder and an image-token projector.

Model Reference

Model Architecture

PropertyValue
ArchitectureStep5ForConditionalGeneration
Checkpointstepfun-ai/Step-5-Preview-BF16
Language backbone92-layer MoE decoder with SWA and CSA
Vision encoder47 layers, width 1,536, 16 attention heads
Configured image / patch size728 / 14 pixels
ProjectionTwo stride-2 convolutions, followed by a 6,144-to-4,096 linear projection
Image inputsOverview images and optional image crops
Context / expert parallelismSupported by the model implementation
Pipeline / tensor parallelismNot supported for this multimodal class

The model accepts already-preprocessed pixel_values, optional patch_pixel_values, and per-image num_patches, together with the matching image placeholders in the text. Projected image tokens replace those placeholders before the language backbone executes.

Packed Sequences and Context Parallelism

The packed path preserves each document’s position reset, loss mask, image placeholder span, and crop ownership. CP shards the resulting language sequence into contiguous query ranges. The document mask applies to image tokens as well as text tokens, including when an image span or document crosses a CP boundary.

Current Scope

The checkpoint does not include a Hugging Face image processor, so raw-image datasets need a compatible processor before training. This page does not provide a ready-to-run image SFT recipe.

The full-depth H100 benchmark recipe on the language model page covers the language backbone with mock text and excludes vision compute.