> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.datasets.llm.column_mapped_text_instruction_iterable_dataset

## Module Contents

### Classes

| Name                                                                                                                                                                                           | Description                                                                              |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| [`ColumnMappedTextInstructionIterableDataset`](#nemo_automodel-components-datasets-llm-column_mapped_text_instruction_iterable_dataset-ColumnMappedTextInstructionIterableDataset)             | Streaming iterable variant that reuses the column-mapping/tokenization logic.            |
| [`ColumnMappedTextInstructionIterableDatasetConfig`](#nemo_automodel-components-datasets-llm-column_mapped_text_instruction_iterable_dataset-ColumnMappedTextInstructionIterableDatasetConfig) | Construction-time configuration for :class:`ColumnMappedTextInstructionIterableDataset`. |

### Functions

| Name                                                                                                                                         | Description                                                                        |
| -------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| [`_load_streaming_dataset`](#nemo_automodel-components-datasets-llm-column_mapped_text_instruction_iterable_dataset-_load_streaming_dataset) | Load a dataset from HuggingFace Hub, local JSON/JSONL files, or Delta Lake tables. |

### Data

[`logger`](#nemo_automodel-components-datasets-llm-column_mapped_text_instruction_iterable_dataset-logger)

### API

```python
class nemo_automodel.components.datasets.llm.column_mapped_text_instruction_iterable_dataset.ColumnMappedTextInstructionIterableDataset(
    path_or_dataset_id: typing.Union[str, typing.List[str]],
    column_mapping: typing.Dict[str, str],
    tokenizer,
    split: str | None = None,
    name: str | None = None,
    answer_only_loss_mask: bool = True,
    seq_length: int | None = None,
    padding: typing.Union[str, bool] = 'do_not_pad',
    truncation: typing.Union[str, bool] = 'do_not_truncate',
    start_of_turn_token: str | None = None,
    limit_dataset_samples: int | None = None,
    repeat_on_exhaustion: bool = True,
    use_hf_chat_template: bool = False,
    delta_storage_options: typing.Dict[str, str] | None = None,
    delta_version: int | None = None,
    delta_sql_query: str | None = None
)
```

**Bases:** `IterableDataset`, [ColumnMappedTextInstructionDataset](/nemo-automodel/nemo_automodel/components/datasets/llm/column_mapped_text_instruction_dataset#nemo_automodel-components-datasets-llm-column_mapped_text_instruction_dataset-ColumnMappedTextInstructionDataset)

Streaming iterable variant that reuses the column-mapping/tokenization logic.

This wraps a Hugging Face streaming dataset (IterableDataset from `datasets`)
or Delta Lake table and yields tokenized samples compatible with the non-streaming
variant, while supporting sharding and epoch-setting for deterministic shuffles upstream.

Supports the following data sources:

* HuggingFace Hub datasets
* Local JSON/JSONL files
* Delta Lake tables (via delta://, dbfs\:/, or local directories with \_delta\_log)

**`_current_epoch_for_repeat`** `= 0`

---

**`num_shards`**

---

```python
nemo_automodel.components.datasets.llm.column_mapped_text_instruction_iterable_dataset.ColumnMappedTextInstructionIterableDataset.__getitem__(
    idx: int
) -> typing.Dict[str, typing.List[int]]
```

```python
nemo_automodel.components.datasets.llm.column_mapped_text_instruction_iterable_dataset.ColumnMappedTextInstructionIterableDataset.__iter__() -> typing.Iterator[typing.Dict[str, typing.List[int]]]
```

```python
nemo_automodel.components.datasets.llm.column_mapped_text_instruction_iterable_dataset.ColumnMappedTextInstructionIterableDataset.__len__() -> int
```

```python
nemo_automodel.components.datasets.llm.column_mapped_text_instruction_iterable_dataset.ColumnMappedTextInstructionIterableDataset.set_epoch(
    epoch: int
) -> None
```

```python
nemo_automodel.components.datasets.llm.column_mapped_text_instruction_iterable_dataset.ColumnMappedTextInstructionIterableDataset.shard(
    num_shards: int,
    index: int
)
```

```python
nemo_automodel.components.datasets.llm.column_mapped_text_instruction_iterable_dataset.ColumnMappedTextInstructionIterableDataset.shuffle(
    buffer_size: int = 1000,
    seed: int | None = None
)
```

```python
class nemo_automodel.components.datasets.llm.column_mapped_text_instruction_iterable_dataset.ColumnMappedTextInstructionIterableDatasetConfig(
    path_or_dataset_id: str | list[str],
    column_mapping: dict[str, str],
    split: str | None = None,
    name: str | None = None,
    answer_only_loss_mask: bool = True,
    seq_length: int | None = None,
    padding: str | bool = 'do_not_pad',
    truncation: str | bool = 'do_not_truncate',
    start_of_turn_token: str | None = None,
    limit_dataset_samples: int | None = None,
    repeat_on_exhaustion: bool = True,
    use_hf_chat_template: bool = False,
    delta_storage_options: dict[str, str] | None = None,
    delta_version: int | None = None,
    delta_sql_query: str | None = None
)
```

Dataclass

Construction-time configuration for :class:`ColumnMappedTextInstructionIterableDataset`.

**`accepts_tokenizer`** `bool = True`

---

**`answer_only_loss_mask`** `bool = True`

Whether to compute the loss mask only on the answer tokens.

---

**`column_mapping`** `dict[str, str]`

Mapping of logical column roles (context/question/answer) to raw column names.

---

**`delta_sql_query`** `str | None = None`

Optional SQL query applied to the Delta Lake table.

---

**`delta_storage_options`** `dict[str, str] | None = None`

Storage options forwarded to the Delta Lake reader.

---

**`delta_version`** `int | None = None`

Delta Lake table version to read.

---

**`limit_dataset_samples`** `int | None = None`

The number of samples to take from the (streamed) dataset.

---

**`name`** `str | None = None`

The name of the dataset configuration/subset to load.

---

**`padding`** `str | bool = 'do_not_pad'`

Padding mode for formatting.

---

**`path_or_dataset_id`** `str | list[str]`

The path or dataset id of the dataset.

---

**`repeat_on_exhaustion`** `bool = True`

Whether to restart iteration when the stream is exhausted.

---

**`seq_length`** `int | None = None`

The sequence length to use for padding.

---

**`split`** `str | None = None`

The split of the dataset to load.

---

**`start_of_turn_token`** `str | None = None`

Optional token marking assistant turns for answer-only loss.

---

**`truncation`** `str | bool = 'do_not_truncate'`

Truncation mode for formatting.

---

**`use_hf_chat_template`** `bool = False`

Whether to format samples using the tokenizer's chat template.

---

```python
nemo_automodel.components.datasets.llm.column_mapped_text_instruction_iterable_dataset.ColumnMappedTextInstructionIterableDatasetConfig.build(
    tokenizer: 'PreTrainedTokenizerBase | None'
) -> 'ColumnMappedTextInstructionIterableDataset'
```

Build a :class:`ColumnMappedTextInstructionIterableDataset` from this :class:`ColumnMappedTextInstructionIterableDatasetConfig` and tokenizer.

```python
nemo_automodel.components.datasets.llm.column_mapped_text_instruction_iterable_dataset._load_streaming_dataset(
    path_or_dataset_id: typing.Union[str, typing.List[str]],
    split: str | None = None,
    streaming: bool = False,
    name: str | None = None,
    delta_storage_options: typing.Dict[str, str] | None = None,
    delta_version: int | None = None,
    delta_sql_query: str | None = None
)
```

Load a dataset from HuggingFace Hub, local JSON/JSONL files, or Delta Lake tables.

If *path\_or\_dataset\_id* resembles a HF repo ID (i.e. of the form
`org/dataset` and the path does **not** exist on the local filesystem),
we defer to `datasets.load_dataset` directly. If the path is a Delta Lake
table (prefixed with `delta://`, `dbfs:/`, or a directory containing
`_delta_log`), we load using the Delta Lake reader. Otherwise, we assume
the argument points to one or more local JSON/JSONL files and let
`datasets.load_dataset` with the *"json"* script handle the parsing.

**Parameters:**

**`path_or_dataset_id`** `Union[str, List[str]]`

Either a HF dataset identifier (`org/name`),
a Delta Lake table path (`delta://path/to/table`), or
a path / list of paths to local `.json` / `.jsonl` files.

---

**`split`** `str | None` — default: None

Optional split to load when retrieving a remote dataset. This
parameter is ignored for local files and Delta Lake tables.

---

**`streaming`** `bool` — default: False

Whether to stream the dataset.

---

**`name`** `str | None` — default: None

Optional name of the dataset configuration/subset to load

---

**`delta_storage_options`** `Dict[str, str] | None` — default: None

Optional dict of storage options for Delta Lake
cloud authentication (e.g., `{"DATABRICKS_TOKEN": "dapi..."}`)

---

**`delta_version`** `int | None` — default: None

Optional specific version of the Delta table to read.

---

**`delta_sql_query`** `str | None` — default: None

Optional SQL query to execute against the Delta Lake source.
This is supported when running with a SparkSession (Databricks / pyspark)
or when using the Databricks SQL Connector. The query must return the
columns expected by `column_mapping`.

---

**Returns:**

datasets.Dataset: The loaded dataset.

**Examples:**

```python
>>> # Load from HuggingFace Hub
>>> ds = _load_dataset("org/dataset", split="train")
```

```python
>>> # Load from local Delta Lake table
>>> ds = _load_dataset("delta:///path/to/delta_table", streaming=True)
```

```python
>>> # Load from Databricks with authentication
>>> ds = _load_dataset(
...     "delta://catalog.schema.table",
...     delta_storage_options={"DATABRICKS_TOKEN": "dapi..."},
...     streaming=True,
... )
```

```python
nemo_automodel.components.datasets.llm.column_mapped_text_instruction_iterable_dataset.logger = logging.getLogger(__name__)
```