> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.stages.text.download.base.iterator

## Module Contents

### Classes

| Name                                                                                                          | Description                                                         |
| ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| [`DocumentIterateExtractStage`](#nemo_curator-stages-text-download-base-iterator-DocumentIterateExtractStage) | Stage that iterates through downloaded files with DocumentIterator, |
| [`DocumentIterator`](#nemo_curator-stages-text-download-base-iterator-DocumentIterator)                       | Abstract base class for document iterators.                         |

### API

```python
class nemo_curator.stages.text.download.base.iterator.DocumentIterateExtractStage(
    iterator: nemo_curator.stages.text.download.base.iterator.DocumentIterator,
    extractor: nemo_curator.stages.text.download.base.extract.DocumentExtractor | None = None,
    record_limit: int | None = None,
    add_filename_column: bool | str = True,
    max_calls_per_worker: int | None = None
)
```

Dataclass

**Bases:** [ProcessingStage\[FileGroupTask, DocumentBatch\]](/nemo-curator/nemo_curator/stages/base#nemo_curator-stages-base-ProcessingStage)

Stage that iterates through downloaded files with DocumentIterator,
then extracts structured content from raw records with DocumentExtractor.

Takes local file paths and produces a DocumentBatch with extracted content.
If DocumentIterator produces the final format, then DocumentExtractor is not needed.

**`add_filename_column`** `bool | str = True`

---

**`extractor`** `DocumentExtractor | None = None`

---

**`iterator`** `DocumentIterator`

---

**`max_calls_per_worker`** `int | None = None`

---

**`record_limit`** `int | None = None`

---

```python
nemo_curator.stages.text.download.base.iterator.DocumentIterateExtractStage.__post_init__()
```

Initialize the stage.

```python
nemo_curator.stages.text.download.base.iterator.DocumentIterateExtractStage.inputs() -> tuple[list[str], list[str]]
```

Define input requirements - expects FileGroupTask with local file paths.

```python
nemo_curator.stages.text.download.base.iterator.DocumentIterateExtractStage.outputs() -> tuple[list[str], list[str]]
```

Define output - produces DocumentBatch with processed records.

```python
nemo_curator.stages.text.download.base.iterator.DocumentIterateExtractStage.process(
    task: nemo_curator.tasks.FileGroupTask
) -> nemo_curator.tasks.DocumentBatch
```

Iterate through files and extract structured content.

**Parameters:**

**`task`** `FileGroupTask`

Task containing local file paths

---

**Returns:** `DocumentBatch`

Batch containing extracted records

```python
nemo_curator.stages.text.download.base.iterator.DocumentIterateExtractStage.ray_stage_spec() -> dict[str, typing.Any]
```

Get Ray configuration for this stage.

```python
class nemo_curator.stages.text.download.base.iterator.DocumentIterator()
```

Abstract

Abstract base class for document iterators.

Always yields dict\[str, str] records. For raw content that needs extraction,
the iterator can put it in any field (e.g., "raw\_content", "html", "content", etc.)

```python
nemo_curator.stages.text.download.base.iterator.DocumentIterator.iterate(
    file_path: str
) -> collections.abc.Iterator[dict[str, typing.Any]]
```

abstract

Iterate over records in a file, yielding dict records.

```python
nemo_curator.stages.text.download.base.iterator.DocumentIterator.output_columns() -> list[str]
```

abstract

Define output columns - produces DocumentBatch with records.