> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.stages.text.deduplication.removal

Removal stage for distributed deduplication pipeline.

This stage implements the removal phase of the distributed deduplication approach:

1. Takes a DocumentBatch and determines the min/max ID range
2. Filters the parquet files for IDs to remove within this range
3. Filters out documents based on the removal list
4. Returns the filtered DocumentBatch

## Module Contents

### Classes

| Name                                                                                                       | Description                                                                 |
| ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| [`TextDuplicatesRemovalStage`](#nemo_curator-stages-text-deduplication-removal-TextDuplicatesRemovalStage) | Stage for removing duplicate documents based on pre-computed removal lists. |

### API

```python
class nemo_curator.stages.text.deduplication.removal.TextDuplicatesRemovalStage(
    ids_to_remove_path: str,
    id_field: str = CURATOR_DEDUP_ID_STR,
    duplicate_id_field: str = 'id',
    read_kwargs: dict[str, typing.Any] | None = None,
    drop_id_field: bool = False
)
```

Dataclass

**Bases:** [ProcessingStage\[DocumentBatch, DocumentBatch\]](/nemo-curator/nemo_curator/stages/base#nemo_curator-stages-base-ProcessingStage)

Stage for removing duplicate documents based on pre-computed removal lists.

**Parameters:**

**`ids_to_remove_path`** `str`

Path to parquet files containing IDs to remove

---

**`id_field`** `str` — default: CURATOR\_DEDUP\_ID\_STR

Field to use for deduplication within the input dataframe. Defaults to CURATOR\_DEDUP\_ID\_STR.

---

**`duplicate_id_field`** `str` — default: 'id'

Field to use for deduplication within the removal dataframe. Defaults to "id".

---

**`read_kwargs`** `dict[str, Any] | None` — default: None

Additional arguments for reading parquet files

---

**`drop_id_field`** `bool` — default: False

Whether to drop the deduplication ID field from the output batch.

---

**`drop_id_field`** `bool = False`

---

**`duplicate_id_field`** `str = 'id'`

---

**`id_field`** `str = CURATOR_DEDUP_ID_STR`

---

**`ids_to_remove_path`** `str`

---

**`read_kwargs`** `dict[str, Any] | None = None`

---

```python
nemo_curator.stages.text.deduplication.removal.TextDuplicatesRemovalStage.__post_init__()
```

Initialize parent class after dataclass initialization.

```python
nemo_curator.stages.text.deduplication.removal.TextDuplicatesRemovalStage.inputs() -> tuple[list[str], list[str]]
```

```python
nemo_curator.stages.text.deduplication.removal.TextDuplicatesRemovalStage.process(
    task: nemo_curator.tasks.DocumentBatch
) -> nemo_curator.tasks.DocumentBatch
```

Our deduplicator should've written out a parquet file with the IDs to remove.
We read that file, filter the input dataframe to only include the IDs to remove,
and return the filtered dataframe.
We optimize by not loading the whole ids to remove into memory, but only loading the ids that are in the range of the input dataframe.