> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.stages.text.io.reader.base

## Module Contents

### Classes

| Name                                                                        | Description                                                     |
| --------------------------------------------------------------------------- | --------------------------------------------------------------- |
| [`BaseFileReader`](#nemo_curator-stages-text-io-reader-base-BaseFileReader) | Base reader for file-group readers that consume lists of paths. |
| [`BaseReader`](#nemo_curator-stages-text-io-reader-base-BaseReader)         | Common base for tabular readers.                                |
| [`ReaderOutput`](#nemo_curator-stages-text-io-reader-base-ReaderOutput)     | -                                                               |

### Data

[`ReaderData`](#nemo_curator-stages-text-io-reader-base-ReaderData)

[`ReaderTask`](#nemo_curator-stages-text-io-reader-base-ReaderTask)

### API

```python
class nemo_curator.stages.text.io.reader.base.BaseFileReader(
    fields: list[str] | None = None,
    read_kwargs: dict[str, typing.Any] = dict(),
    name: str = '',
    _generate_ids: bool = False,
    _assign_ids: bool = False,
    allow_empty: bool = False
)
```

Dataclass

**Bases:** [BaseReader](#nemo_curator-stages-text-io-reader-base-BaseReader)

Base reader for file-group readers that consume lists of paths.

```python
nemo_curator.stages.text.io.reader.base.BaseFileReader.read_data(
    file_paths: list[str],
    read_kwargs: dict[str, typing.Any] | None,
    fields: list[str] | None
) -> nemo_curator.stages.text.io.reader.base.ReaderData
```

```python
nemo_curator.stages.text.io.reader.base.BaseFileReader.read_task(
    task: nemo_curator.tasks.FileGroupTask,
    read_kwargs: dict[str, typing.Any] | None,
    fields: list[str] | None
) -> nemo_curator.stages.text.io.reader.base.ReaderOutput
```

```python
class nemo_curator.stages.text.io.reader.base.BaseReader(
    fields: list[str] | None = None,
    read_kwargs: dict[str, typing.Any] = dict(),
    name: str = '',
    _generate_ids: bool = False,
    _assign_ids: bool = False,
    allow_empty: bool = False
)
```

Dataclass

**Bases:** [ProcessingStage\[ReaderTask, DocumentBatch\]](/nemo-curator/nemo_curator/stages/base#nemo_curator-stages-base-ProcessingStage)

Common base for tabular readers.

Subclasses must implement read\_task for their input task type.

**`_assign_ids`** `bool = False`

---

**`_generate_ids`** `bool = False`

---

**`allow_empty`** `bool = False`

---

**`fields`** `list[str] | None = None`

---

**`name`** `str = ''`

---

**`read_kwargs`** `dict[str, Any] = field(default_factory=dict)`

---

```python
nemo_curator.stages.text.io.reader.base.BaseReader.__post_init__() -> None
```

```python
nemo_curator.stages.text.io.reader.base.BaseReader._assign_ids_func(
    filepath: str | list[str],
    df: pandas.DataFrame
) -> pandas.DataFrame
```

```python
nemo_curator.stages.text.io.reader.base.BaseReader._document_batch(
    task: nemo_curator.stages.text.io.reader.base.ReaderTask,
    output: nemo_curator.stages.text.io.reader.base.ReaderOutput
) -> nemo_curator.tasks.DocumentBatch
```

```python
nemo_curator.stages.text.io.reader.base.BaseReader._generate_ids_func(
    filepath: str | list[str],
    df: pandas.DataFrame
) -> pandas.DataFrame
```

```python
nemo_curator.stages.text.io.reader.base.BaseReader._validate_result(
    task: nemo_curator.stages.text.io.reader.base.ReaderTask,
    result: nemo_curator.stages.text.io.reader.base.ReaderData
) -> None
```

```python
nemo_curator.stages.text.io.reader.base.BaseReader.inputs() -> tuple[list[str], list[str]]
```

```python
nemo_curator.stages.text.io.reader.base.BaseReader.outputs() -> tuple[list[str], list[str]]
```

```python
nemo_curator.stages.text.io.reader.base.BaseReader.process(
    task: nemo_curator.stages.text.io.reader.base.ReaderTask
) -> nemo_curator.tasks.DocumentBatch
```

```python
nemo_curator.stages.text.io.reader.base.BaseReader.ray_stage_spec() -> dict[str, typing.Any]
```

```python
nemo_curator.stages.text.io.reader.base.BaseReader.read_task(
    task: nemo_curator.stages.text.io.reader.base.ReaderTask,
    read_kwargs: dict[str, typing.Any] | None,
    fields: list[str] | None
) -> nemo_curator.stages.text.io.reader.base.ReaderOutput
```

```python
nemo_curator.stages.text.io.reader.base.BaseReader.setup(
    _: nemo_curator.backends.base.WorkerMetadata | None = None
) -> None
```

```python
class nemo_curator.stages.text.io.reader.base.ReaderOutput(
    data: nemo_curator.stages.text.io.reader.base.ReaderData,
    metadata: dict[str, typing.Any] | None = None
)
```

Dataclass

**`data`** `ReaderData`

---

**`metadata`** `dict[str, Any] | None = None`

---

```python
nemo_curator.stages.text.io.reader.base.ReaderData: TypeAlias = pd.DataFrame | pa.Table
```

```python
nemo_curator.stages.text.io.reader.base.ReaderTask: TypeAlias = FileGroupTask | LanceReadTask
```