Skip to navigation
DataIntegrate Your Own Multi-Modal Dataset

Integrate Your Own Multi-Modal Dataset

View as MarkdownOpen in Claude

This guide shows you how to integrate your own dataset into NeMo AutoModel for training. You will learn about multi-modal datasets that combine text with images or other modalities. The guide covers how to create custom datasets by implementing the required methods and preprocessing functions. It also shows how to define your own data logic in YAML configuration with file paths, without modifying the main codebase.

Quick Start Summary

TypeUse CaseExamplePreprocessorSection
🖼️ Multi-modalVision and LanguageMedPix-VQAapply_chat_template, collate functionMulti-Modal Datasets
🎤 AudioSpeech and LanguageCommon Voice 17apply_chat_template, collate functionAudio Datasets

Multi-Modal Datasets

Multi-modal datasets combine text with other input types (for example, images, audio, or video) and are essential for training Vision-Language Models (VLMs). These datasets introduce specific challenges such as aligning modalities, batching diverse data types, and formatting prompts for multi-turn, multi-modal dialogue.

NeMo AutoModel supports multi-modal dataset integration through flexible preprocessing, custom formatting, and YAML-based configuration.

Typical Types in Multi-Modal Datasets

A multi-modal dataset typically contains:

  • Images, videos, audio, or other non-text modalities.
  • Textual inputs such as questions, instructions, or captions.
  • Answers or expected outputs from the model.

These are formatted into structured conversations or instruction-response pairs for use with VLMs such as BLIP, Llava, or Flamingo.

MedPix-VQA Dataset Example

The MedPix-VQA dataset is a comprehensive medical Visual Question Answering dataset designed for training and evaluating VQA models in the medical domain. It contains radiological images (from MedPix, a well-known medical image dataset) and associated QA pairs used for medical image interpretation.

Structure:

  • 20,500 total examples
  • Columns: image_id, mode, case_id, question, answer
{
"image_id": "medpix_0143.jpg",
"mode": "CT",
"case_id": "case_101",
"question": "What abnormality is visible in the left hemisphere?",
"answer": "Subdural hematoma"
}

The example dataset preprocessing performs the following steps:

  1. Loads the dataset using Hugging Face’s datasets library.
  2. Extracts the question and answer.
  3. Transforms the data into a chat-like format that is compatible with Hugging Face’s AutoProcessor apply_chat_template function. For example:
conversation = [
{
"role": "user",
"content": [
{"type": "image", "image": example["image_id"]},
{"type": "text", "text": example["question"]},
],
},
{
"role": "assistant",
"content": [{"type": "text", "text": example["answer"]}]
},
]

For more detailed examples of how to process multi-modal datasets for VLMs, see the examples in datasets.py.

Audio Datasets

Audio datasets combine speech input with text transcriptions and are essential for training models capable of speech recognition and transcription tasks. NeMo AutoModel supports audio dataset integration through specialized preprocessing functions and custom collate functions for multi-modal models such as Phi-4.

Common Voice 17 Dataset Example

The Common Voice 17 dataset is a comprehensive speech recognition dataset containing audio clips and corresponding transcriptions. This particular version focuses on Turkish speech data and has been preprocessed and fixed for compatibility with modern training frameworks.

Structure:

  • Audio: Speech recordings in various formats
  • Transcription: Text transcriptions of the spoken content
  • Use case: Speech-to-text transcription for multi-modal models
{
"audio": {
"path": "common_voice_tr_17528071.mp3",
"array": [-0.1600779, -0.13843077],
"sampling_rate": 16000
},
"transcription": "Kosova başkentinswki yolcu sayısı arttı."
}

The example dataset preprocessing performs the following steps:

  1. Loads the dataset using Hugging Face’s datasets library.
  2. Extracts the audio and transcription fields.

For more detailed examples of how to process multi-modal datasets, see the examples in datasets.py.

Collate Functions

NeMo AutoModel provides specialized collate functions for different VLM processors. The collate function is responsible for batching examples and preparing them for model input.

Multi-modal models require custom collate functions to batch and process each sample correctly. If your model uses a Hugging Face AutoProcessor, you can use it directly. Otherwise, you can define your own collate logic and point to it in your YAML config. NeMo AutoModel provides example custom collate functions that you can use as references for your implementation. After you implement your own collate function, you can specify it in your YAML config.

YAML-Based Custom Dataset Configuration

NeMo AutoModel supports YAML-based dataset specification using the _target_ key. This enables you to reference dataset-building classes or functions using either a Python dotted path or a file path and function name.

To use a Python dotted path, specify the full import path:

dataset:
_target_: nemo_automodel.components.datasets.llm.hellaswag.HellaSwag
path_or_dataset: rowan/hellaswag
split: train

To use a file path and function name, specify the following format:

<file-path>:<function-name>

The pattern has the following parts:

  • <file-path>: The absolute path to a Python file containing your dataset function
  • <function-name>: The name of the function to call from that file
dataset:
_target_: /path/to/your/custom_dataset.py:build_my_dataset
num_blocks: 111

This calls build_my_dataset() from the specified file with the other keys (for example, num_blocks) as arguments. This approach enables you to integrate custom datasets through config alone, with no need to alter the codebase or package structure.

Loading the file runs its code, so NeMo AutoModel loads a file from outside the installed package only after you opt in. Launch with NEMO_ENABLE_USER_MODULES=1 automodel my_config.yaml, and only load files that you trust.

DataLoader Workers

For VLM training and validation, num_workers: 0 automatically selects one worker per CPU available to the training process and logs a warning with the resolved count. Zero is also the default when omitted. This enables background preprocessing and batch prefetching. The count respects CPU affinity when available, falls back to the system CPU count, and uses one worker if neither count is available. The selection does not inspect container CPU-time quotas.

The count applies to each dataloader independently. When several training ranks share CPUs, set an explicit positive worker count to limit CPU and host-memory use. For example:

dataloader:
num_workers: 4
persistent_workers: true
prefetch_factor: 2
validation_dataloader:
num_workers: 2

Custom datasets and collators run in worker processes and must prepare data on the CPU. Move batches to the GPU in the training process. All VLM loaders use worker processes. Setting num_workers: 1 selects one worker. Prefetched tensor batches use shared memory, so size container shared memory for your batches and worker count.

Automatic and explicit positive worker counts can resume checkpoints saved by older VLM recipes with zero workers. The loader migrates the map-style dataset state and sampler position to worker processes. Sampler position and order are preserved. Preprocessing that depends on worker-local state, including random transforms, can differ when moving to workers or changing their count. Worker checkpoints can change count when no per-worker dataset or iterator state was saved. Otherwise, restore with the saved worker count. Changing it raises an error during checkpoint loading.

Custom Chat Template

By default, VLM fine-tuning uses the chat template built into the model’s AutoProcessor. To use a custom template, add chat_template under dataset: in your YAML config:

dataset:
_target_: nemo_automodel.components.datasets.vlm.datasets.make_medpix_dataset
split: train
chat_template: /path/to/template.jinja

The chat_template field accepts the following values:

  • A Jinja template string (for example, "{% for msg in messages %}...")
  • A path to a .jinja template file
  • A path to a JSON file containing a chat_template key (for example, tokenizer_config.json)

The override is applied to both processor.chat_template and processor.tokenizer.chat_template before dataset loading.

Troubleshooting Tips

Use the following checks when dataset training does not behave as expected:

  • Tokenization mismatch: Ensure your tokenizer aligns with the model’s expected inputs.
  • Dataset too large: Use limit_dataset_samples in your YAML config to load a subset, which is useful for quick debugging.
  • Loss not decreasing: Verify that your loss mask correctly ignores prompt tokens.