> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.datasets.vlm.datasets

## Module Contents

### Classes

| Name                                                                                                                        | Description                                                                                     |
| --------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| [`CordV2DatasetConfig`](#nemo_automodel-components-datasets-vlm-datasets-CordV2DatasetConfig)                               | Construction-time configuration for the CORD-V2 dataset.                                        |
| [`LlavaOnevisionDatasetConfig`](#nemo_automodel-components-datasets-vlm-datasets-LlavaOnevisionDatasetConfig)               | Construction-time configuration for the LLaVA-OneVision dataset.                                |
| [`MedPixDatasetConfig`](#nemo_automodel-components-datasets-vlm-datasets-MedPixDatasetConfig)                               | Construction-time configuration for the MedPix dataset.                                         |
| [`MetaDatasetConfig`](#nemo_automodel-components-datasets-vlm-datasets-MetaDatasetConfig)                                   | Construction-time configuration for the meta (multi-source) VLM dataset.                        |
| [`PreTokenizedDatasetWrapper`](#nemo_automodel-components-datasets-vlm-datasets-PreTokenizedDatasetWrapper)                 | Dataset wrapper that tokenizes samples in `__getitem__`.                                        |
| [`PreTokenizedDatasetWrapperConfig`](#nemo_automodel-components-datasets-vlm-datasets-PreTokenizedDatasetWrapperConfig)     | Construction-time configuration for :class:`PreTokenizedDatasetWrapper`.                        |
| [`RdrDatasetConfig`](#nemo_automodel-components-datasets-vlm-datasets-RdrDatasetConfig)                                     | Construction-time configuration for the RDR dataset.                                            |
| [`RobustDatasetWrapper`](#nemo_automodel-components-datasets-vlm-datasets-RobustDatasetWrapper)                             | Wrapper that catches `__getitem__` and collate errors, substituting random replacement samples. |
| [`Tulu3MagicoderTextMixDatasetConfig`](#nemo_automodel-components-datasets-vlm-datasets-Tulu3MagicoderTextMixDatasetConfig) | Construction-time configuration for the Tulu-3/Magicoder text mixture.                          |
| [`UnimmChatDatasetConfig`](#nemo_automodel-components-datasets-vlm-datasets-UnimmChatDatasetConfig)                         | Construction-time configuration for the UniMM-Chat dataset.                                     |
| [`_ExamplesWithStats`](#nemo_automodel-components-datasets-vlm-datasets-_ExamplesWithStats)                                 | list subclass that carries pre-computed dataset statistics.                                     |

### Functions

| Name                                                                                                                              | Description                                                                                |
| --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| [`_collect_sample_stats`](#nemo_automodel-components-datasets-vlm-datasets-_collect_sample_stats)                                 | Count images, videos, text-only samples and estimate token counts.                         |
| [`_convert_sharegpt_to_conversation`](#nemo_automodel-components-datasets-vlm-datasets-_convert_sharegpt_to_conversation)         | Convert a single sharegpt-format example to Automodel conversation format.                 |
| [`_grid_media_token_count`](#nemo_automodel-components-datasets-vlm-datasets-_grid_media_token_count)                             | -                                                                                          |
| [`_load_json_or_jsonl`](#nemo_automodel-components-datasets-vlm-datasets-_load_json_or_jsonl)                                     | Load data from a JSON or JSONL file.                                                       |
| [`_load_jsonl_for_rank`](#nemo_automodel-components-datasets-vlm-datasets-_load_jsonl_for_rank)                                   | Load only the JSONL lines needed for this rank, avoiding full json.loads on skipped lines. |
| [`_log_dataset_loading_summary`](#nemo_automodel-components-datasets-vlm-datasets-_log_dataset_loading_summary)                   | Print a visual summary of per-dataset loading times and data statistics.                   |
| [`_media_token_mismatch`](#nemo_automodel-components-datasets-vlm-datasets-_media_token_mismatch)                                 | Return a mismatch description if media grids survived without tokens.                      |
| [`_resolve_processor_token_id`](#nemo_automodel-components-datasets-vlm-datasets-_resolve_processor_token_id)                     | Resolve a model-specific media token id from processor/config/tokenizer.                   |
| [`make_cord_v2_dataset`](#nemo_automodel-components-datasets-vlm-datasets-make_cord_v2_dataset)                                   | Load and preprocess the CORD-V2 dataset for image-to-text fine-tuning.                     |
| [`make_llava_onevision_dataset`](#nemo_automodel-components-datasets-vlm-datasets-make_llava_onevision_dataset)                   | Load and preprocess the LLaVA-Instruct-150K dataset for LLaVA-OneVision-1.5.               |
| [`make_medpix_dataset`](#nemo_automodel-components-datasets-vlm-datasets-make_medpix_dataset)                                     | Load and preprocess the MedPix dataset for image-to-text fine-tuning.                      |
| [`make_meta_dataset`](#nemo_automodel-components-datasets-vlm-datasets-make_meta_dataset)                                         | Load datasets defined in a meta JSON file and convert to Automodel conversation format.    |
| [`make_rdr_dataset`](#nemo_automodel-components-datasets-vlm-datasets-make_rdr_dataset)                                           | Load and preprocess the RDR dataset for image-to-text fine-tuning.                         |
| [`make_tulu3_dataset`](#nemo_automodel-components-datasets-vlm-datasets-make_tulu3_dataset)                                       | Load `allenai/tulu-3-sft-mixture` directly from the HF Hub as text-only conversations.     |
| [`make_tulu3_magicoder_text_mix_dataset`](#nemo_automodel-components-datasets-vlm-datasets-make_tulu3_magicoder_text_mix_dataset) | Build a text-only 80/20 mix of Tulu-3-SFT-mixture and Magicoder-OSS-Instruct-75K.          |
| [`make_unimm_chat_dataset`](#nemo_automodel-components-datasets-vlm-datasets-make_unimm_chat_dataset)                             | Load and preprocess the UniMM-Chat dataset for image-to-text fine-tuning.                  |

### Data

[`logger`](#nemo_automodel-components-datasets-vlm-datasets-logger)

### API

```python
class nemo_automodel.components.datasets.vlm.datasets.CordV2DatasetConfig(
    path_or_dataset: str = 'naver-clova-ix/cord-v2',
    split: str = 'train',
    limit_dataset_samples: int | None = None
)
```

Dataclass

Construction-time configuration for the CORD-V2 dataset.

Optional maximum number of samples to load.

HuggingFace dataset id or local path for the CORD-V2 dataset.

Dataset split to load (e.g. `"train"`, `"test"`).

```python
nemo_automodel.components.datasets.vlm.datasets.CordV2DatasetConfig.build() -> list[dict[str, object]]
```

Build the CORD-V2 dataset from this config.

```python
class nemo_automodel.components.datasets.vlm.datasets.LlavaOnevisionDatasetConfig(
    path_or_dataset: str = 'liuhaotian/LLaVA-Instruct-...,
    split: str = 'train'
)
```

Dataclass

Construction-time configuration for the LLaVA-OneVision dataset.

HuggingFace dataset id or local path.

Dataset split to load.

```python
nemo_automodel.components.datasets.vlm.datasets.LlavaOnevisionDatasetConfig.build() -> list[dict[str, object]]
```

Build the LLaVA-OneVision conversation dataset.

```python
class nemo_automodel.components.datasets.vlm.datasets.MedPixDatasetConfig(
    path_or_dataset: str = 'medpix-dataset/medpix-data...,
    split: str = 'train'
)
```

Dataclass

Construction-time configuration for the MedPix dataset.

HuggingFace dataset id or local path for MedPix.

Dataset split to load.

```python
nemo_automodel.components.datasets.vlm.datasets.MedPixDatasetConfig.build() -> object
```

Build the lazily decoded MedPix dataset.

```python
class nemo_automodel.components.datasets.vlm.datasets.MetaDatasetConfig(
    path_or_dataset: str = '',
    dataset_names: list[str] | None = None,
    split: str = 'train',
    shard_data: bool = False
)
```

Dataclass

Construction-time configuration for the meta (multi-source) VLM dataset.

Names of datasets to load from the meta file. `None` loads all.

Path to the meta JSON file that defines the datasets to load.

If `True`, each rank loads only its `1/world_size` slice.

Dataset split to load (passed through for API consistency).

```python
nemo_automodel.components.datasets.vlm.datasets.MetaDatasetConfig.build(
    rank: int | None = None,
    world_size: int | None = None
) -> list[dict[str, object]]
```

Build the meta VLM dataset from this config.

**Parameters:**

Runtime data-parallel rank. Inferred from `torch.distributed` when `None`.

Runtime data-parallel world size. Inferred from `torch.distributed` when `None`.

**Returns:** `list[dict[str, object]]`

Materialized multimodal examples from the selected source datasets.

```python
class nemo_automodel.components.datasets.vlm.datasets.PreTokenizedDatasetWrapper(
    dataset,
    processor,
    max_length = None,
    max_retries = 10,
    truncate = False,
    post_tokenize_hook = None,
    inject_fake_images = True
)
```

**Bases:** `Dataset`

Dataset wrapper that tokenizes samples in `__getitem__`.

Instead of deferring `apply_chat_template` to the collate function, this
wrapper performs tokenization per-sample so that:

* The collate function only needs to pad and stack.
* Overlong samples are detected **after** precise tokenization (including
  media-token expansion) and replaced with a different random sample.
* Tokenization work is distributed across DataLoader workers.

Each `__getitem__` call returns a dict with at least::

\{
"input\_ids":      (seq\_len,),
"attention\_mask": (seq\_len,),
"labels":         (seq\_len,),
}

Plus optional media tensors (`pixel_values`, `image_grid_thw`,
`pixel_values_videos`, `video_grid_thw`).

```python
nemo_automodel.components.datasets.vlm.datasets.PreTokenizedDatasetWrapper.__getitem__(
    idx
)
```

```python
nemo_automodel.components.datasets.vlm.datasets.PreTokenizedDatasetWrapper.__len__()
```

```python
nemo_automodel.components.datasets.vlm.datasets.PreTokenizedDatasetWrapper.robust_collate(
    collate_fn
)
```

Wrap *collate\_fn* so that on failure the entire batch is re-sampled.

```python
class nemo_automodel.components.datasets.vlm.datasets.PreTokenizedDatasetWrapperConfig(
    max_length: int | None = None,
    max_retries: int = 10,
    truncate: bool = False,
    inject_fake_images: bool = True,
    post_tokenize_hook: collections.abc.Callable[[dict[str, object]], dict[str, object]] | None = None
)
```

Dataclass

Construction-time configuration for :class:`PreTokenizedDatasetWrapper`.

If `True`, inject a fake image into text-only samples for sharded training.

Maximum token sequence length. Overlong samples are replaced or truncated.

Number of retry attempts when a sample fails to tokenize.

Optional declarative callback applied to each tokenizer result.

If `True`, truncate overlong samples instead of replacing them.

```python
nemo_automodel.components.datasets.vlm.datasets.PreTokenizedDatasetWrapperConfig.build(
    dataset: torch.utils.data.Dataset | collections.abc.Sequence[dict[str, object]],
    processor: 'ProcessorMixin'
) -> 'PreTokenizedDatasetWrapper'
```

Build a :class:`PreTokenizedDatasetWrapper` from this config.

**Parameters:**

The raw dataset (conversations) to wrap.

HuggingFace processor for tokenization.

```python
class nemo_automodel.components.datasets.vlm.datasets.RdrDatasetConfig(
    path_or_dataset: str = 'quintend/rdr-items',
    split: str = 'train'
)
```

Dataclass

Construction-time configuration for the RDR dataset.

HuggingFace dataset id or local path for the RDR dataset.

Dataset split to load (e.g. `"train"`, `"test"`).

```python
nemo_automodel.components.datasets.vlm.datasets.RdrDatasetConfig.build() -> list[dict[str, object]]
```

Build the RDR dataset from this config.

```python
class nemo_automodel.components.datasets.vlm.datasets.RobustDatasetWrapper(
    dataset,
    max_retries: int = 10
)
```

**Bases:** `Dataset`

Wrapper that catches `__getitem__` and collate errors, substituting random replacement samples.

This handles failures such as corrupted files, missing media, bad data,
or processor errors (e.g. multimodal token mismatch from truncation)
without crashing the entire training run.

```python
nemo_automodel.components.datasets.vlm.datasets.RobustDatasetWrapper.__getitem__(
    idx
)
```

```python
nemo_automodel.components.datasets.vlm.datasets.RobustDatasetWrapper.__len__()
```

```python
nemo_automodel.components.datasets.vlm.datasets.RobustDatasetWrapper.robust_collate(
    collate_fn
)
```

Wrap a collate\_fn so that on failure the entire batch is re-sampled and retried.

```python
class nemo_automodel.components.datasets.vlm.datasets.Tulu3MagicoderTextMixDatasetConfig(
    tulu_split: str = 'train',
    magicoder_split: str = 'train',
    seed: int = 42,
    max_turns: int = 16,
    limit_total: int | None = None
)
```

Dataclass

Construction-time configuration for the Tulu-3/Magicoder text mixture.

Optional cap on the merged row count.

Split expression for the Magicoder source.

Maximum number of turns retained from a Tulu-3 conversation.

Sampling seed for interleaving the two sources.

Split expression for the Tulu-3 source.

```python
nemo_automodel.components.datasets.vlm.datasets.Tulu3MagicoderTextMixDatasetConfig.build() -> list[dict[str, object]]
```

Build the text-only Tulu-3/Magicoder mixture.

```python
class nemo_automodel.components.datasets.vlm.datasets.UnimmChatDatasetConfig(
    path_or_dataset: str = 'Yirany/UniMM-Chat',
    split: str = 'train'
)
```

Dataclass

Construction-time configuration for the UniMM-Chat dataset.

HuggingFace dataset id or local path for the UniMM-Chat dataset.

Dataset split to load (e.g. `"train"`, `"test"`).

```python
nemo_automodel.components.datasets.vlm.datasets.UnimmChatDatasetConfig.build() -> list[dict[str, object]]
```

Build the UniMM-Chat dataset from this config.

```python
class nemo_automodel.components.datasets.vlm.datasets._ExamplesWithStats()
```

**Bases:** `list`

list subclass that carries pre-computed dataset statistics.

Attached by :func:`make_meta_dataset` so downstream code (e.g.
`_log_global_dataset_stats`) can read aggregated stats without
re-scanning all examples.

```python
nemo_automodel.components.datasets.vlm.datasets._collect_sample_stats(
    examples
)
```

Count images, videos, text-only samples and estimate token counts.

Token estimation mirrors the logic in LengthGroupedSampler.\_estimate\_tokens:

* Text tokens: uses pre-computed `_text_tokens` when present (written by
  `scripts/precompute_tokens.py`), otherwise falls back to `chars // 3`.
* Media tokens: uses `mm_inputs_meta` image/video dimensions when present
  (populated by the precompute script), otherwise `500` per media item.

**Returns:**

dict with keys n\_images, n\_videos, n\_text\_only, n\_text\_tokens,

```python
nemo_automodel.components.datasets.vlm.datasets._convert_sharegpt_to_conversation(
    example,
    columns = None,
    tags = None,
    media_dir = None
)
```

Convert a single sharegpt-format example to Automodel conversation format.

**Parameters:**

A single data example in sharegpt format.

Column name mapping with keys 'messages', 'images', 'videos'.

Tag mapping with keys 'role\_tag', 'content\_tag', 'user\_tag', 'assistant\_tag'.

Directory prefix for resolving relative media paths.

**Returns:**

Example in Automodel conversation format.

```python
nemo_automodel.components.datasets.vlm.datasets._grid_media_token_count(
    grid,
    merge_size: int
) -> int
```

```python
nemo_automodel.components.datasets.vlm.datasets._load_json_or_jsonl(
    file_path
)
```

Load data from a JSON or JSONL file.

**Parameters:**

Path to the JSON or JSONL file.

**Returns:**

list\[dict]: List of data examples.

```python
nemo_automodel.components.datasets.vlm.datasets._load_jsonl_for_rank(
    file_path,
    sample_ratio,
    rank,
    world_size
)
```

Load only the JSONL lines needed for this rank, avoiding full json.loads on skipped lines.

Handles sample\_ratio and sharding so that each rank only parses and stores
its own subset.  The semantics match the original load-all-then-slice approach:

1. Apply `sample_ratio` (deterministic `Random(42).sample`) on the full
   index range.
2. Shard the resulting list with `[rank::world_size]`.

**Returns:**

tuple\[list\[dict], int]: (parsed examples for this rank, total line count).

```python
nemo_automodel.components.datasets.vlm.datasets._log_dataset_loading_summary(
    timings,
    wall_time,
    total_samples,
    rank = None
)
```

Print a visual summary of per-dataset loading times and data statistics.

```python
nemo_automodel.components.datasets.vlm.datasets._media_token_mismatch(
    input_ids,
    result,
    processor
) -> str | None
```

Return a mismatch description if media grids survived without tokens.

```python
nemo_automodel.components.datasets.vlm.datasets._resolve_processor_token_id(
    processor,
    attr_names,
    token_names
)
```

Resolve a model-specific media token id from processor/config/tokenizer.

```python
nemo_automodel.components.datasets.vlm.datasets.make_cord_v2_dataset(
    path_or_dataset = 'naver-clova-ix/cord-v2',
    split = 'train',
    limit_dataset_samples: int | None = None,
    kwargs = {}
)
```

Load and preprocess the CORD-V2 dataset for image-to-text fine-tuning.

```python
nemo_automodel.components.datasets.vlm.datasets.make_llava_onevision_dataset(
    path_or_dataset = 'liuhaotian/LLaVA-Instruct-...,
    split = 'train',
    kwargs = {}
)
```

Load and preprocess the LLaVA-Instruct-150K dataset for LLaVA-OneVision-1.5.

This function loads conversation-format data with images and returns it in
the standard NeMo VLM format expected by the collate function.

**Parameters:**

Path to the dataset on HuggingFace Hub or local path.

Dataset split to load (e.g., "train", "train\[:1000]").

Additional arguments passed to load\_dataset.

**Returns:**

List of dicts with "conversation" and "image" keys.

```python
nemo_automodel.components.datasets.vlm.datasets.make_medpix_dataset(
    path_or_dataset = 'medpix-dataset/medpix-data...,
    split = 'train',
    kwargs = {}
)
```

Load and preprocess the MedPix dataset for image-to-text fine-tuning.

Formatting is deferred to `__getitem__` via `with_transform` so the
dataset stays Arrow-backed (no Python-side copy of every image). Images are
loaded undecoded (`Image(decode=False)`) and wrapped as lazy `PIL`
handles (`Image.open` reads only the header); the actual pixel decode then
happens on demand in the DataLoader workers, rather than eagerly decoding the
whole split up front.

```python
nemo_automodel.components.datasets.vlm.datasets.make_meta_dataset(
    path_or_dataset,
    dataset_names = None,
    split = 'train',
    shard_data = False,
    rank = None,
    world_size = None,
    kwargs = {}
)
```

Load datasets defined in a meta JSON file and convert to Automodel conversation format.

The meta JSON file maps dataset names to their configurations. Each configuration can have:

* file\_name (str): Path to the data file (JSON/JSONL). Relative paths are resolved
  against the meta file's directory.
* columns (dict): Column name mapping (messages, images, videos).
* tags (dict): Tag mapping (role\_tag, content\_tag, user\_tag, assistant\_tag).
* media\_dir (str): Directory prefix for media files.
* sample\_ratio (float): Sampling ratio (0.0 to 1.0, default 1.0).

When `shard_data=True`, each rank loads only its `1/world_size` slice of
every dataset file (interleaved: `raw_data[rank::world_size]`).  This
reduces per-rank memory and I/O.  The caller should use a local sampler
(e.g. `RandomSampler`) instead of `DistributedSampler` since data is
already partitioned.

Video frame sampling (fps, min\_frames, max\_frames) should be configured on
the **processor** rather than here.  For example in YAML::

processor:
*target*: transformers.AutoProcessor.from\_pretrained
pretrained\_model\_name\_or\_path: ...
fps: 1
min\_frames: 4
max\_frames: 128

Example meta JSON::

\{
"my\_dataset": \{
"file\_name": "data/train.jsonl",
"columns": \{"messages": "conversations"},
"media\_dir": "/data/media"
}
}

**Parameters:**

Path to the meta JSON file.

Which datasets to load. None means all.

Unused, kept for API consistency.

If True, each rank loads only its 1/world\_size slice.

Data-parallel rank. Inferred from torch.distributed if None.

Data-parallel world size. Inferred from torch.distributed if None.

Additional arguments (unused).

**Returns:**

list\[dict]: Combined list of examples in Automodel conversation format.

```python
nemo_automodel.components.datasets.vlm.datasets.make_rdr_dataset(
    path_or_dataset = 'quintend/rdr-items',
    split = 'train',
    kwargs = {}
)
```

Load and preprocess the RDR dataset for image-to-text fine-tuning.

**Parameters:**

Path or identifier for the RDR dataset.

Dataset split to load.

Additional arguments.

**Returns:**

The processed dataset.

```python
nemo_automodel.components.datasets.vlm.datasets.make_tulu3_dataset(
    path_or_dataset: str = 'allenai/tulu-3-sft-mixture',
    split: str = 'train',
    kwargs = {}
)
```

Load `allenai/tulu-3-sft-mixture` directly from the HF Hub as text-only conversations.

This avoids the meta-JSON + JSONL data-prep step required by
:func:`make_meta_dataset`: point the recipe's `dataset._target_` at this
function and the Tulu-3 split is pulled straight from the Hub. Each row's
`messages` field is converted with the same helper the meta-JSON path uses
(:func:`_convert_sharegpt_to_conversation`), so the resulting data composition is
**identical** to dumping the split to JSONL and loading it via
:func:`make_meta_dataset`: no turn cap, `system` turns dropped, every row kept
in the original split order. Conversations are text-only (no `image` entries),
so batches carry no `pixel_values` / vision tensors.

The returned dataset stays Arrow-backed (`map`) so the full \~939k-row split is
not copied into a Python list.

**Parameters:**

HF Hub id (or local path) of the Tulu-3 SFT mixture.

HF split expression (e.g. `"train"` or `"train[:50000]"`).

Ignored. Accepted so recipe-level dataset keys (e.g. `truncate`)
that are forwarded to the dataset target do not raise.

**Returns:**

datasets.Dataset: Rows with a single `conversation` column, each a list of

```python
nemo_automodel.components.datasets.vlm.datasets.make_tulu3_magicoder_text_mix_dataset(
    tulu_split: str = 'train',
    magicoder_split: str = 'train',
    seed: int = 42,
    max_turns: int = 16,
    limit_total: int | None = None,
    kwargs = {}
) -> list
```

Build a text-only 80/20 mix of Tulu-3-SFT-mixture and Magicoder-OSS-Instruct-75K.

Both datasets are converted into the NeMo VLM `{"conversation": [...]}` shape
consumed by :func:`nemo_automodel.components.datasets.vlm.collate_fns.default_collate_fn`.
Because `default_collate_fn` is image-aware only when a conversation turn
contains an `{"type": "image", ...}` entry, returning text-only conversations
here yields batches with no `pixel_values` / vision tensors -- which is what
the Gemma 4 base+drafter composite expects for text-only training.

Mixing uses `datasets.interleave_datasets` with probabilities `[0.8, 0.2]`
and `stopping_strategy="all_exhausted"` so both datasets are sampled until
every example has been drawn at least once.

**Parameters:**

HF split expression for the Tulu-3 source (e.g. `"train"`
or `"train[:50000]"`).

HF split expression for the Magicoder source.

Seed forwarded to `interleave_datasets` for reproducibility.

Drop Tulu-3 conversations with more than this many turns to
keep memory bounded. Magicoder samples are always 2 turns.

If set, cap the merged dataset to this many rows.

Additional arguments forwarded to `load_dataset` for both
sources.

**Returns:** `list`

List of `{"conversation": [...]}` dicts. Each conversation is a list of

```python
nemo_automodel.components.datasets.vlm.datasets.make_unimm_chat_dataset(
    path_or_dataset = 'Yirany/UniMM-Chat',
    split = 'train',
    kwargs = {}
)
```

Load and preprocess the UniMM-Chat dataset for image-to-text fine-tuning.

```python
nemo_automodel.components.datasets.vlm.datasets.logger = logging.getLogger(__name__)
```