> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.stages.interleaved.io.writers.webdataset

## Module Contents

### Classes

| Name                                                                                                                          | Description                                                          |
| ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| [`InterleavedWebdatasetWriterStage`](#nemo_curator-stages-interleaved-io-writers-webdataset-InterleavedWebdatasetWriterStage) | Write an `InterleavedBatch` as a MINT-1T-style WebDataset tar shard. |

### Functions

| Name                                                                                                            | Description                                                                    |
| --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| [`_escape_key`](#nemo_curator-stages-interleaved-io-writers-webdataset-_escape_key)                             | Percent-encode a sample\_id so it is safe as a tar member name stem.           |
| [`_ext_from_content_type`](#nemo_curator-stages-interleaved-io-writers-webdataset-_ext_from_content_type)       | Return a file extension for *content\_type*, falling back to `"bin"`.          |
| [`_is_null`](#nemo_curator-stages-interleaved-io-writers-webdataset-_is_null)                                   | Return True for Python/pandas null-ish scalars.                                |
| [`_per_modality_passthrough`](#nemo_curator-stages-interleaved-io-writers-webdataset-_per_modality_passthrough) | Build per-modality passthrough lists from content rows.                        |
| [`_series_to_nullable_list`](#nemo_curator-stages-interleaved-io-writers-webdataset-_series_to_nullable_list)   | Return a Python list from *series*, replacing any null-like value with `None`. |
| [`_write_sample`](#nemo_curator-stages-interleaved-io-writers-webdataset-_write_sample)                         | Write one sample (JSON + image binaries) into *tf*.                            |

### Data

[`_CONTENT_TYPE_TO_EXT`](#nemo_curator-stages-interleaved-io-writers-webdataset-_CONTENT_TYPE_TO_EXT)

### API

```python
class nemo_curator.stages.interleaved.io.writers.webdataset.InterleavedWebdatasetWriterStage(
    path: str,
    file_extension: str = 'tar',
    write_kwargs: dict[str, typing.Any] = dict(),
    materialize_on_write: bool = True,
    name: str = 'interleaved_webdataset_wri...,
    mode: typing.Literal['ignore', 'overwrite', 'append', 'error'] = 'ignore',
    append_mode_implemented: bool = False,
    on_materialize_error: typing.Literal['error', 'warn', 'drop_row', 'drop_sample'] = 'error',
    schema: pyarrow.Schema | None = None,
    schema_overrides: dict[str, pyarrow.DataType] | None = None
)
```

Dataclass

**Bases:** [BaseInterleavedWriter](/nemo-curator/nemo_curator/stages/interleaved/io/writers/base#nemo_curator-stages-interleaved-io-writers-base-BaseInterleavedWriter)

Write an `InterleavedBatch` as a MINT-1T-style WebDataset tar shard.

Each sample is reconstructed from its row-based representation:

* `metadata` rows supply passthrough fields embedded in the JSON.
* `text` rows are assembled into the `"texts"` list (`None` at gaps).
* `image` rows are assembled into the `"images"` list and written as
  individual tar members; `binary_content` must be populated (either by
  the upstream pipeline or via `materialize_on_write=True`).

The JSON member key is `urllib.parse.quote(sample_id, safe="")` so that
roundtripping via :class:`InterleavedWebdatasetReaderStage` with
`sample_id_field="sample_id"` recovers the original sample\_id.

Only `"metadata"`, `"text"`, and `"image"` modalities are supported.
Any other modality raises `ValueError` at write time.

```python
nemo_curator.stages.interleaved.io.writers.webdataset.InterleavedWebdatasetWriterStage._write_dataframe(
    df: pandas.DataFrame,
    file_path: str,
    _write_kwargs: dict[str, typing.Any]
) -> None
```

```python
nemo_curator.stages.interleaved.io.writers.webdataset._escape_key(
    sample_id: str
) -> str
```

Percent-encode a sample\_id so it is safe as a tar member name stem.

```python
nemo_curator.stages.interleaved.io.writers.webdataset._ext_from_content_type(
    content_type: str | None
) -> str
```

Return a file extension for *content\_type*, falling back to `"bin"`.

```python
nemo_curator.stages.interleaved.io.writers.webdataset._is_null(
    value: object
) -> bool
```

Return True for Python/pandas null-ish scalars.

```python
nemo_curator.stages.interleaved.io.writers.webdataset._per_modality_passthrough(
    passthrough_cols: list[str],
    meta_keys: set[str],
    image_rows: pandas.DataFrame,
    text_rows: pandas.DataFrame,
    content_rows: pandas.DataFrame
) -> dict[str, typing.Any]
```

Build per-modality passthrough lists from content rows.

For each passthrough column not already in *meta\_keys*:

* **Pure per-image** (non-null only in image rows): emitted as a list with
  one entry per image row (None preserved for sparse nulls).
* **Pure per-text** (non-null only in text rows): emitted as a list with
  one entry per text row (None preserved for sparse nulls).
* **Mixed** (non-null in both image and text rows): emitted as a
  position-aligned list with one entry per content row in position order
  (None where the value is absent).  When read back without declaring the
  field in `per_image_fields` / `per_text_fields`, the reader treats
  the entire list as a sample-level passthrough on the metadata row.

```python
nemo_curator.stages.interleaved.io.writers.webdataset._series_to_nullable_list(
    series: pandas.Series
) -> list
```

Return a Python list from *series*, replacing any null-like value with `None`.

```python
nemo_curator.stages.interleaved.io.writers.webdataset._write_sample(
    tf: tarfile.TarFile,
    sample_df: pandas.DataFrame,
    sample_id: str,
    passthrough_cols: list[str]
) -> None
```

Write one sample (JSON + image binaries) into *tf*.

```python
nemo_curator.stages.interleaved.io.writers.webdataset._CONTENT_TYPE_TO_EXT: dict[str, str] = {'image/jpeg': 'jpg', 'image/png': 'png', 'image/tiff': 'tiff', 'image/webp': 'w...
```