> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.stages.interleaved.io.readers.webdataset

## Module Contents

### Classes

| Name                                                                                                                          | Description                                                          |
| ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| [`InterleavedWebdatasetReaderStage`](#nemo_curator-stages-interleaved-io-readers-webdataset-InterleavedWebdatasetReaderStage) | Read MINT1T-style WebDataset shards into a row-wise multimodal task. |
| [`_ReadContext`](#nemo_curator-stages-interleaved-io-readers-webdataset-_ReadContext)                                         | Per-tar state shared across all members in a single tar archive.     |
| [`_SampleContext`](#nemo_curator-stages-interleaved-io-readers-webdataset-_SampleContext)                                     | Per-sample state passed to row builder methods.                      |

### API

```python
class nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage(
    read_kwargs: dict[str, typing.Any] = dict(),
    schema: pyarrow.Schema | None = None,
    schema_overrides: dict[str, pyarrow.DataType] | None = None,
    name: str = 'webdataset_reader',
    materialize_on_read: bool = False,
    max_batch_bytes: int | None = None,
    json_extensions: tuple[str, ...] = DEFAULT_JSON_EXTENSIONS,
    image_extensions: tuple[str, ...] = (lambda: DEFAULT_IMAGE_EXTE...,
    sample_id_field: str | None = None,
    texts_field: str = 'texts',
    images_field: str = 'images',
    image_member_field: str | None = None,
    fields: tuple[str, ...] | None = None,
    per_image_fields: tuple[str, ...] = (),
    per_text_fields: tuple[str, ...] = ()
)
```

Dataclass

**Bases:** [BaseInterleavedReader](/nemo-curator/nemo_curator/stages/interleaved/io/readers/base#nemo_curator-stages-interleaved-io-readers-base-BaseInterleavedReader)

Read MINT1T-style WebDataset shards into a row-wise multimodal task.

**`fields`** `tuple[str, ...] | None = None`

---

**`image_extensions`** `tuple[str, ...]`

---

**`image_member_field`** `str | None = None`

---

**`images_field`** `str = 'images'`

---

**`json_extensions`** `tuple[str, ...] = DEFAULT_JSON_EXTENSIONS`

---

**`materialize_on_read`** `bool = False`

---

**`max_batch_bytes`** `int | None = None`

---

**`name`** `str = 'webdataset_reader'`

---

**`per_image_fields`** `tuple[str, ...] = ()`

---

**`per_text_fields`** `tuple[str, ...] = ()`

---

**`sample_id_field`** `str | None = None`

---

**`texts_field`** `str = 'texts'`

---

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage.__post_init__() -> None
```

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._apply_per_modality_fields(
    row: dict[str, typing.Any],
    passthrough: dict[str, list[typing.Any]],
    index: int
) -> None
```

staticmethod

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._build_passthrough_row(
    sample: dict[str, typing.Any]
) -> dict[str, typing.Any]
```

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._build_row(
    ctx: nemo_curator.stages.interleaved.io.readers.webdataset._SampleContext,
    row_fields: dict[str, typing.Any]
) -> dict[str, typing.Any]
```

staticmethod

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._build_source_ref(
    ctx: nemo_curator.stages.interleaved.io.readers.webdataset._SampleContext,
    content_key: str | None,
    frame_index: int | None = None
) -> str
```

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._empty_output_schema() -> pyarrow.Schema
```

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._extract_per_modality_fields(
    sample: dict[str, typing.Any],
    field_names: tuple[str, ...]
) -> dict[str, list[typing.Any]]
```

staticmethod

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._extract_tar_member(
    tf: tarfile.TarFile,
    member_name: str,
    cache: dict[str, bytes | None]
) -> bytes | None
```

staticmethod

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._image_rows(
    ctx: nemo_curator.stages.interleaved.io.readers.webdataset._SampleContext
) -> list[dict[str, typing.Any]]
```

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._metadata_row(
    ctx: nemo_curator.stages.interleaved.io.readers.webdataset._SampleContext
) -> dict[str, typing.Any]
```

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._resolve_default_image_member_name(
    sample_id: str,
    sample: dict[str, typing.Any],
    images: list[object] | None,
    member_names: set[str]
) -> str | None
```

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._resolve_image_content_key(
    image_token: object,
    default_image_member_name: str | None,
    member_names: set[str]
) -> str | None
```

staticmethod

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._rows_from_member(
    tf: tarfile.TarFile,
    member: tarfile.TarInfo,
    read_ctx: nemo_curator.stages.interleaved.io.readers.webdataset._ReadContext
) -> list[dict[str, typing.Any]]
```

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._rows_from_sample(
    ctx: nemo_curator.stages.interleaved.io.readers.webdataset._SampleContext
) -> list[dict[str, typing.Any]]
```

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._source_files_for_split(
    split: pyarrow.Table,
    idx: int,
    sample_id_to_tar: dict[str, str],
    all_tars: list[str]
) -> list[str]
```

staticmethod

Return source\_files for one split, listing only the contributing tars.

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._text_rows(
    ctx: nemo_curator.stages.interleaved.io.readers.webdataset._SampleContext
) -> list[dict[str, typing.Any]]
```

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage._warn_per_modality_length_mismatch(
    sample_id: str,
    passthrough: dict[str, list[typing.Any]],
    actual_count: int,
    modality: str
) -> None
```

staticmethod

```python
nemo_curator.stages.interleaved.io.readers.webdataset.InterleavedWebdatasetReaderStage.process(
    task: nemo_curator.tasks.FileGroupTask
) -> nemo_curator.tasks.InterleavedBatch | list[nemo_curator.tasks.InterleavedBatch]
```

```python
class nemo_curator.stages.interleaved.io.readers.webdataset._ReadContext(
    tar_path: str,
    member_names: set[str],
    member_info: dict[str, tarfile.TarInfo],
    storage_options: dict[str, object],
    byte_cache: dict[str, bytes | None]
)
```

Dataclass

Per-tar state shared across all members in a single tar archive.

**`byte_cache`** `dict[str, bytes | None]`

---

**`member_info`** `dict[str, TarInfo]`

---

**`member_names`** `set[str]`

---

**`storage_options`** `dict[str, object]`

---

**`tar_path`** `str`

---

```python
class nemo_curator.stages.interleaved.io.readers.webdataset._SampleContext(
    sample_id: str,
    sample: dict[str, typing.Any],
    tar_path: str,
    json_member_name: str,
    member_names: set[str],
    member_info: dict[str, tarfile.TarInfo] | None,
    passthrough: dict[str, typing.Any],
    per_image_passthrough: dict[str, list[typing.Any]],
    per_text_passthrough: dict[str, list[typing.Any]]
)
```

Dataclass

Per-sample state passed to row builder methods.

**`json_member_name`** `str`

---

**`member_info`** `dict[str, TarInfo] | None`

---

**`member_names`** `set[str]`

---

**`passthrough`** `dict[str, Any]`

---

**`per_image_passthrough`** `dict[str, list[Any]]`

---

**`per_text_passthrough`** `dict[str, list[Any]]`

---

**`sample`** `dict[str, Any]`

---

**`sample_id`** `str`

---

**`tar_path`** `str`

---