Model CoverageVision Language ModelsLMMS LabLLaVA-OneVision-1.5-4B-Instruct

LLaVA-OneVision-1.5-4B-Instruct

View as Markdown

LLaVA-OneVision 1.5 is a vision-language model combining a Rice ViT encoder with a Qwen3 language backbone, capable of handling both image and video understanding. NeMo AutoModel ships a custom NVIDIA implementation (LlavaOneVisionForConditionalGeneration) with FSDP2/HSDP support, LoRA fine-tuning and distributed training.

Set up NeMo AutoModel with the latest container or follow the installation instructions.

Fine-Tune LLaVA-OneVision-1.5-4B-Instruct

From the repository root, run:

uv run automodel --nproc-per-node=8 examples/vlm_finetune/llava_onevision/llava_ov_1_5_4b_finetune.yaml

Choose a Workflow

GoalStart Here
Supervised fine-tuning (SFT) - LLaVA-OneVision-1.5 4B on LLaVA-Instruct-150KUse llava_ov_1_5_4b_finetune.yaml.

Model Reference

Model Architecture

PropertyValue
TaskImage-Text-to-Text
ArchitectureLlavaOneVisionForConditionalGeneration
Parameters4B
Hugging Face Organizationlmms-lab
  • LlavaOneVisionForConditionalGeneration

Vision tower is the Rice Transformer: 14x14 patch embed with 2D RoPE, standard Transformer blocks (LayerNorm + Attention + MLP), and a 2x2 spatial Patch Merger that projects to the language-model hidden size.

Available Models

ModelHF ID
LLaVA-OneVision-1.5 4B Instructlmms-lab/LLaVA-OneVision-1.5-4B-Instruct