Vision Language Models (VLMs)

Introduction

Vision Language Models (VLMs) integrate vision and language processing capabilities, enabling models to understand images and generate text descriptions, answer visual questions, and perform multimodal reasoning.

NeMo AutoModel LLM APIs can be easily extended to support VLM tasks. While most of the training setup is the same as for LLMs, some additional steps are required to prepare the data and model for VLM training.

Run VLMs with NeMo AutoModel

To run VLMs with NeMo AutoModel, use NeMo container version 26.06.00 or later. If the model you want to fine-tune requires newer source than the container provides, use a current checkout so the package and its example recipes stay on one revision:

$ git clone https://github.com/NVIDIA-NeMo/Automodel.git
$ cd Automodel
$ uv sync --locked --all-groups --all-extras --extra vlm-media

For other installation options, see our Installation Guide.

VLMs that decode video or use Qwen / Mistral vision preprocessing also need the vlm-media extra included in the source-sync command above; it is not installed in the Docker container by default. See Media Extras.

Supported Models

NeMo AutoModel supports AutoModelForImageTextToText in the Image-Text-to-Text category.

Owner	Model	Architectures
Moonshot AI	Kimi-VL	`KimiVLForConditionalGeneration`, `KimiK25ForConditionalGeneration`, `KimiK25VLForConditionalGeneration`
Moonshot AI	Kimi-K3	`KimiK3ForConditionalGeneration`, `KimiK3ForCausalLM`
Thinking Machines Lab	Inkling	`InklingForConditionalGeneration`
Google	Gemma 3 / Gemma 3n	`Gemma3ForCausalLM`, `Gemma3ForConditionalGeneration`
Google	Gemma 4	`Gemma4UnifiedForConditionalGeneration`, `Gemma4ForConditionalGeneration`, `Gemma4AssistantForCausalLM`
Qwen / Alibaba Cloud	Qwen2.5-VL	`Qwen2VLForConditionalGeneration`, `Qwen2_5VLForConditionalGeneration`
Qwen / Alibaba Cloud	Qwen3-VL / Qwen3-VL-MoE	`Qwen3VLForConditionalGeneration`, `Qwen3VLMoeForConditionalGeneration`
Qwen / Alibaba Cloud	Qwen3.5	`Qwen3_5ForConditionalGeneration`, `Qwen3_5MoeForConditionalGeneration`
Qwen / Alibaba Cloud	Qwen3.6	`Qwen3_5ForConditionalGeneration`, `Qwen3_5MoeForConditionalGeneration`
NVIDIA	Nemotron-Parse	`NemotronParseForConditionalGeneration`
Mistral AI	Ministral3 VL	`Mistral3ForConditionalGeneration`
Mistral AI	Mistral-Small-4	`MistralForConditionalGeneration`, `Mistral4ForCausalLM`
Mistral AI	Mistral Medium 3.5	`Mistral3ForConditionalGeneration`, `Mistral3FP8VLMForConditionalGeneration`
InternLM / Shanghai AI Lab	InternVL	`InternVLForConditionalGeneration`
Meta	Llama 4	`Llama4ForConditionalGeneration`
HuggingFace	SmolVLM	`SmolVLMForConditionalGeneration`
LLaVA	LLaVA	`LlavaForConditionalGeneration`, `LlavaNextForConditionalGeneration`, `LlavaNextVideoForConditionalGeneration`, `LlavaOnevisionForConditionalGeneration`
lmms-lab	LLaVA-OneVision 1.5	`LLaVAOneVision1_5_ForConditionalGeneration`, `LlavaOneVisionForConditionalGeneration`
Stepfun AI	Step-3.7-Flash	`Step3p6ForConditionalGeneration`, `Step3p7ForConditionalGeneration`
MiniMaxAI	MiniMax-M3	`MiniMaxM3SparseForConditionalGeneration`

Fine-Tuning

All supported models can be fine-tuned using either full SFT or PEFT (LoRA) approaches. See the Gemma 3 Fine-Tuning Guide for a complete walkthrough covering dataset preparation, configuration, and multi-GPU training.

In these guides, we use the quintend/rdr-items and naver-clova-ix/cord-v2 datasets for demonstration purposes. Update the recipe YAML dataset section to use your own data. See VLM datasets and dataset overview.