> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.stages.deduplication.fuzzy.identify_duplicates

## Module Contents

### Classes

| Name                                                                                                              | Description                                               |
| ----------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| [`IdentifyDuplicatesStage`](#nemo_curator-stages-deduplication-fuzzy-identify_duplicates-IdentifyDuplicatesStage) | Stage that generates removal IDs for fuzzy deduplication. |

### Data

[`DUPLICATE_IDS_SUBDIR`](#nemo_curator-stages-deduplication-fuzzy-identify_duplicates-DUPLICATE_IDS_SUBDIR)

### API

```python
class nemo_curator.stages.deduplication.fuzzy.identify_duplicates.IdentifyDuplicatesStage(
    duplicate_group_field: str = CURATOR_FUZZY_DUPLICATE_GRO...,
    document_id_field: str = CURATOR_DEDUP_ID_STR,
    total_nparts: int | None = None,
    output_path: str = './',
    read_kwargs: dict[str, typing.Any] | None = None,
    write_kwargs: dict[str, typing.Any] | None = None,
    rmm_pool_size: int | typing.Literal['auto'] | None = 'auto',
    spill_memory_limit: int | typing.Literal['auto'] | None = 'auto',
    enable_statistics: bool = False
)
```

**Bases:** [ShuffleStage](/nemo-curator/nemo_curator/stages/deduplication/shuffle_utils/stage#nemo_curator-stages-deduplication-shuffle_utils-stage-ShuffleStage)

Stage that generates removal IDs for fuzzy deduplication.
The approach involves shuffling the data based on the duplicate group field similar to grouping by the group field.
followed by selecting one document per group.
Currently the removal strategy is to randomly keep one document per group.

## Parameters

duplicate\_group\_field
Column name representing the group id for a document.
total\_nparts
Total number of output partitions. If None, will be set automatically by the executor.
output\_path
Path to write output files.
read\_kwargs
Keyword arguments for cudf.read\_parquet method.
write\_kwargs
Keyword arguments for cudf.to\_parquet method.
rmm\_pool\_size
Size of the RMM GPU memory pool in bytes.
If "auto", the memory pool is set to 90% of the free GPU memory.
If None, the memory pool is set to 50% of the free GPU memory that can expand if needed.
spill\_memory\_limit
Device memory limit in bytes for spilling to host.
If "auto", the limit is set to 80% of the RMM pool size.
If None spilling is disabled.
enable\_statistics
Whether the underlying rapidsmpf shuffler should collect shuffle statistics.

**`name`** `= 'IdentifyDuplicates'`

---

**`output_fs`**

---

**`output_path`**

---

```python
nemo_curator.stages.deduplication.fuzzy.identify_duplicates.IdentifyDuplicatesStage._get_removal_ids(
    df: cudf.DataFrame
) -> cudf.DataFrame
```

Get the removal ids for the given dataframe.

```python
nemo_curator.stages.deduplication.fuzzy.identify_duplicates.IdentifyDuplicatesStage.extract_and_write() -> list[nemo_curator.tasks.FileGroupTask]
```

```python
nemo_curator.stages.deduplication.fuzzy.identify_duplicates.IdentifyDuplicatesStage.insert_finished() -> None
```

```python
nemo_curator.stages.deduplication.fuzzy.identify_duplicates.IdentifyDuplicatesStage.process(
    task: nemo_curator.tasks.FileGroupTask
) -> nemo_curator.tasks.FileGroupTask
```

```python
nemo_curator.stages.deduplication.fuzzy.identify_duplicates.IdentifyDuplicatesStage.ray_stage_spec() -> dict[str, typing.Any]
```

```python
nemo_curator.stages.deduplication.fuzzy.identify_duplicates.IdentifyDuplicatesStage.read_and_insert(
    task: nemo_curator.tasks.FileGroupTask
) -> nemo_curator.tasks.FileGroupTask
```

```python
nemo_curator.stages.deduplication.fuzzy.identify_duplicates.DUPLICATE_IDS_SUBDIR = 'FuzzyDuplicateIds'
```