> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.stages.deduplication.exact.workflow

## Module Contents

### Classes

| Name                                                                                                         | Description                                                |
| ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------- |
| [`ExactDeduplicationWorkflow`](#nemo_curator-stages-deduplication-exact-workflow-ExactDeduplicationWorkflow) | A pipeline that performs exact deduplication of a dataset. |

### Data

[`ID_GENERATOR_OUTPUT_FILENAME`](#nemo_curator-stages-deduplication-exact-workflow-ID_GENERATOR_OUTPUT_FILENAME)

### API

```python
class nemo_curator.stages.deduplication.exact.workflow.ExactDeduplicationWorkflow(
    output_path: str,
    input_path: str | list[str] | None = None,
    input_filetype: typing.Literal['jsonl', 'parquet'] = 'parquet',
    input_blocksize: str | int = '2GiB',
    identification_batchsize: int = 1,
    input_file_extensions: list[str] | None = None,
    read_kwargs: dict[str, typing.Any] | None = None,
    write_kwargs: dict[str, typing.Any] | None = None,
    assign_id: bool = True,
    id_field: str | None = None,
    text_field: str = 'text',
    perform_removal: bool = False,
    total_nparts: int | None = None,
    rmm_pool_size: int | typing.Literal['auto'] | None = 'auto',
    spill_memory_limit: int | typing.Literal['auto'] | None = 'auto',
    env_vars: dict[str, typing.Any] | None = None
)
```

**Bases:** [WorkflowBase](/nemo-curator/nemo_curator/pipeline/workflow#nemo_curator-pipeline-workflow-WorkflowBase)

A pipeline that performs exact deduplication of a dataset.
It consists of the following stages:

* FilePartitioningStage
  Groups input files into smaller groups that can be processed in parallel.
* ExactDuplicateIdentification
  Finds exact duplicates in a given column by hashing the column.
* Removal (Optional)
  Currently not implemented.

**`executor_config`**

---

```python
nemo_curator.stages.deduplication.exact.workflow.ExactDeduplicationWorkflow._create_identification_pipeline(
    num_input_tasks: int
) -> nemo_curator.pipeline.Pipeline
```

```python
nemo_curator.stages.deduplication.exact.workflow.ExactDeduplicationWorkflow._create_input_filegroups() -> nemo_curator.pipeline.Pipeline
```

```python
nemo_curator.stages.deduplication.exact.workflow.ExactDeduplicationWorkflow._validate_initial_tasks(
    initial_tasks: list[nemo_curator.tasks.FileGroupTask] | None
) -> None
```

```python
nemo_curator.stages.deduplication.exact.workflow.ExactDeduplicationWorkflow._validate_inputs() -> None
```

```python
nemo_curator.stages.deduplication.exact.workflow.ExactDeduplicationWorkflow.run(
    initial_tasks: list[nemo_curator.tasks.FileGroupTask] | None = None,
    executor: nemo_curator.backends.ray_actor_pool.RayActorPoolExecutor | None = None
) -> nemo_curator.pipeline.workflow.WorkflowRunResult
```

Run the deduplication pipeline.

executor: RayActorPoolExecutor | None
Executor to use for the pipeline.
If not provided, the default RayActorPoolExecutor will be used.

**Parameters:**

**`initial_tasks`** `list[FileGroupTask] | None` — default: None

---

**Returns:** `WorkflowRunResult`

WorkflowRunResult object containing the results and timing information

```python
nemo_curator.stages.deduplication.exact.workflow.ID_GENERATOR_OUTPUT_FILENAME = 'exact_id_generator.json'
```