Integrate Your Own Multi-Modal Dataset
This guide shows you how to integrate your own dataset into NeMo AutoModel for training. You will learn about multi-modal datasets that combine text with images or other modalities. The guide covers how to create custom datasets by implementing the required methods and preprocessing functions. It also shows how to define your own data logic in YAML configuration with file paths, without modifying the main codebase.
Quick Start Summary
Multi-Modal Datasets
Multi-modal datasets combine text with other input types (for example, images, audio, or video) and are essential for training Vision-Language Models (VLMs). These datasets introduce specific challenges such as aligning modalities, batching diverse data types, and formatting prompts for multi-turn, multi-modal dialogue.
NeMo AutoModel supports multi-modal dataset integration through flexible preprocessing, custom formatting, and YAML-based configuration.
Typical Types in Multi-Modal Datasets
A multi-modal dataset typically contains:
- Images, videos, audio, or other non-text modalities.
- Textual inputs such as questions, instructions, or captions.
- Answers or expected outputs from the model.
These are formatted into structured conversations or instruction-response pairs for use with VLMs such as BLIP, Llava, or Flamingo.
MedPix-VQA Dataset Example
The MedPix-VQA dataset is a comprehensive medical Visual Question Answering dataset designed for training and evaluating VQA models in the medical domain. It contains radiological images (from MedPix, a well-known medical image dataset) and associated QA pairs used for medical image interpretation.
Structure:
- 20,500 total examples
- Columns:
image_id,mode,case_id,question,answer
The example dataset preprocessing performs the following steps:
- Loads the dataset using Hugging Face’s
datasetslibrary. - Extracts the
questionandanswer. - Transforms the data into a chat-like format that is compatible with Hugging Face’s
AutoProcessorapply_chat_templatefunction. For example:
For more detailed examples of how to process multi-modal datasets for VLMs, see the examples in datasets.py.
Audio Datasets
Audio datasets combine speech input with text transcriptions and are essential for training models capable of speech recognition and transcription tasks. NeMo AutoModel supports audio dataset integration through specialized preprocessing functions and custom collate functions for multi-modal models such as Phi-4.
Common Voice 17 Dataset Example
The Common Voice 17 dataset is a comprehensive speech recognition dataset containing audio clips and corresponding transcriptions. This particular version focuses on Turkish speech data and has been preprocessed and fixed for compatibility with modern training frameworks.
Structure:
- Audio: Speech recordings in various formats
- Transcription: Text transcriptions of the spoken content
- Use case: Speech-to-text transcription for multi-modal models
The example dataset preprocessing performs the following steps:
- Loads the dataset using Hugging Face’s
datasetslibrary. - Extracts the
audioandtranscriptionfields.
For more detailed examples of how to process multi-modal datasets, see the examples in datasets.py.
Collate Functions
NeMo AutoModel provides specialized collate functions for different VLM processors. The collate function is responsible for batching examples and preparing them for model input.
Multi-modal models require custom collate functions to batch and process each sample correctly. If your model uses a Hugging Face AutoProcessor, you can use it directly. Otherwise, you can define your own collate logic and point to it in your YAML config. NeMo AutoModel provides example custom collate functions that you can use as references for your implementation. After you implement your own collate function, you can specify it in your YAML config.
YAML-Based Custom Dataset Configuration
NeMo AutoModel supports YAML-based dataset specification using the _target_ key. This enables you to reference dataset-building classes or functions using either a Python dotted path or a file path and function name.
To use a Python dotted path, specify the full import path:
To use a file path and function name, specify the following format:
The pattern has the following parts:
<file-path>: The absolute path to a Python file containing your dataset function<function-name>: The name of the function to call from that file
This calls build_my_dataset() from the specified file with the other keys (for example, num_blocks) as arguments. This approach enables you to integrate custom datasets through config alone, with no need to alter the codebase or package structure.
Loading the file runs its code, so NeMo AutoModel loads a file from outside the installed package only after you opt in. Launch with NEMO_ENABLE_USER_MODULES=1 automodel my_config.yaml, and only load files that you trust.
DataLoader Workers
For VLM training and validation, num_workers: 0 automatically selects one worker
per CPU available to the training process and logs a warning with the resolved
count. Zero is also the default when omitted. This enables background preprocessing and batch prefetching. The count
respects CPU affinity when available, falls back to the system CPU count, and
uses one worker if neither count is available.
The selection does not inspect container CPU-time quotas.
The count applies to each dataloader independently. When several training ranks share CPUs, set an explicit positive worker count to limit CPU and host-memory use. For example:
Custom datasets and collators run in worker processes and must prepare data on
the CPU. Move batches to the GPU in the training process. All VLM loaders use
worker processes. Setting num_workers: 1 selects one worker. Prefetched tensor batches
use shared memory, so size container shared memory for your batches and worker count.
Automatic and explicit positive worker counts can resume checkpoints saved by older VLM recipes with zero workers. The loader migrates the map-style dataset state and sampler position to worker processes. Sampler position and order are preserved. Preprocessing that depends on worker-local state, including random transforms, can differ when moving to workers or changing their count. Worker checkpoints can change count when no per-worker dataset or iterator state was saved. Otherwise, restore with the saved worker count. Changing it raises an error during checkpoint loading.
Custom Chat Template
By default, VLM fine-tuning uses the chat template built into the model’s AutoProcessor. To use a custom template, add chat_template under dataset: in your YAML config:
The chat_template field accepts the following values:
- A Jinja template string (for example,
"{% for msg in messages %}...") - A path to a
.jinjatemplate file - A path to a JSON file containing a
chat_templatekey (for example,tokenizer_config.json)
The override is applied to both processor.chat_template and processor.tokenizer.chat_template before dataset loading.
Troubleshooting Tips
Use the following checks when dataset training does not behave as expected:
- Tokenization mismatch: Ensure your tokenizer aligns with the model’s expected inputs.
- Dataset too large: Use
limit_dataset_samplesin your YAML config to load a subset, which is useful for quick debugging. - Loss not decreasing: Verify that your loss mask correctly ignores prompt tokens.