> This page is for version 26.09 · v1.4.0. > For other versions, use one of these documentation indexes: > - Latest · v1.4.0 (26.09) (default): https://docs.nvidia.com/nemo/curator/latest/llms.txt > - Main · preview: https://docs.nvidia.com/nemo/curator/main/llms.txt > - 26.09 · v1.4.0: https://docs.nvidia.com/nemo/curator/v26.09/llms.txt > - 26.07 · v1.3.0: https://docs.nvidia.com/nemo/curator/v26.07/llms.txt > - 26.04 · v1.2.0: https://docs.nvidia.com/nemo/curator/v26.04/llms.txt > - 26.02 · v1.1.0: https://docs.nvidia.com/nemo/curator/v26.02/llms.txt > - 25.09 · v1.0.0: https://docs.nvidia.com/nemo/curator/v25.09/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server. > Identify and remove exact duplicates using MD5 hashing in a Ray-based workflow # Exact Duplicate Removal Remove character-for-character duplicate documents using NeMo Curator's exact duplicate removal workflow. This method computes MD5 hashes for each document's text and identifies documents with identical hashes as duplicates. For an overview of all duplicate removal options, refer to [Deduplication ](/curate-text/process-data/deduplication). ## How It Works Exact deduplication uses MD5 hashing to identify identical documents: 1. Computes MD5 hash for each document's text content 2. Groups documents by identical hash values 3. Identifies duplicates and saves IDs for removal This method targets character-for-character duplicates and is recommended for removing exact copies of documents. ## Before You Start **Prerequisites**: * Ray cluster with GPU support (required for distributed processing) * Stable document identifiers for removal (either existing IDs or IDs assigned by the workflow) ## Quick Start Get started with exact deduplication using the following example of identifying duplicates, then remove them: ```python from nemo_curator.core.client import RayClient from nemo_curator.stages.deduplication.exact.workflow import ExactDeduplicationWorkflow from nemo_curator.stages.text.deduplication.removal_workflow import TextDuplicatesRemovalWorkflow ray_client = RayClient() ray_client.start() # Step 1: Identify duplicates exact_workflow = ExactDeduplicationWorkflow( input_path="input_data/", output_path="./results", text_field="text", assign_id=True, perform_removal=False, input_filetype="parquet" ) result = exact_workflow.run() # result.metadata contains: total_time, num_duplicates, identification_time, id_generator_path # Step 2: Remove duplicates removal_workflow = TextDuplicatesRemovalWorkflow( input_path="input_data/", ids_to_remove_path="./results/ExactDuplicateIds", output_path="./deduplicated", input_filetype="parquet", id_field="_curator_dedup_id", duplicate_id_field="_curator_dedup_id", id_generator_path="./results/exact_id_generator.json" ) result = removal_workflow.run() # result.metadata contains: total_time, num_duplicates_removed ``` ## Configuration Configure exact deduplication using these key parameters: | Parameter | Type | Default | Description | | -------------------------- | --------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `input_path` | str \| list\[str] | None | Path(s) to input files or directories | | `output_path` | str | Required | Directory to write duplicate IDs and ID generator | | `text_field` | str | "text" | Name of the text field in input data | | `normalize_text` | bool | False | Whether to lowercase text, collapse whitespace runs, and trim leading and trailing whitespace for matching. The workflow preserves the original text. | | `assign_id` | bool | True | Whether to automatically assign unique IDs | | `id_field` | str \| None | None | Existing ID field name (if assign\_id=False) | | `input_filetype` | str | "parquet" | Input file format ("parquet" or "jsonl") | | `input_file_extensions` | list\[str] \| None | None | Extensions to discover. `None` or `[]` uses `[".parquet"]` for Parquet and `[".jsonl", ".json"]` for JSONL. A non-empty list replaces those defaults. | | `input_blocksize` | str \| int | "2GiB" | Size of input blocks for processing | | `identification_batchsize` | int | 1 | Number of input blocks to concatenate and insert into the shuffler per call. Higher values increase GPU throughput at the cost of memory. For example, `input_blocksize="256MiB"` with `identification_batchsize=4` processes \~1 GB per call. | | `perform_removal` | bool | False | Reserved; must remain `False`. Exact removal is performed with `TextDuplicatesRemovalWorkflow`. | | `total_nparts` | int \| None | None | Total number of output partitions. If `None`, defaults to one-third of the number of input tasks. | | `rmm_pool_size` | int \| "auto" \| None | "auto" | RMM GPU memory pool size in bytes. `"auto"` uses 90% of free GPU memory; `None` uses 50% with dynamic expansion. | | `use_async_memory` | bool | True | Use CUDA asynchronous memory allocation for the shuffle. Set to `False` to use the legacy pool-wrapped CUDA memory resource. | | `spill_memory_limit` | int \| "auto" \| None | "auto" | Device memory spill-to-host limit in bytes. `"auto"` sets the limit to 80% of the RMM pool size; `None` disables spilling. When `rmm_pool_size` is `None`, `"auto"` also resolves to no spilling. | For recursive discovery behavior, custom suffix examples, and supported formats, see [Input File Discovery](/curate-text/process-data/deduplication#input-file-discovery). ## Removing Duplicates After identifying duplicates, use `TextDuplicatesRemovalWorkflow` to remove them: ```python from nemo_curator.core.client import RayClient from nemo_curator.stages.text.deduplication.removal_workflow import TextDuplicatesRemovalWorkflow ray_client = RayClient() ray_client.start() removal_workflow = TextDuplicatesRemovalWorkflow( input_path="/path/to/input/data", ids_to_remove_path="/path/to/output/ExactDuplicateIds", output_path="/path/to/deduplicated", input_filetype="parquet", # Set input_file_extensions here too if identification used a custom suffix. id_field="_curator_dedup_id", duplicate_id_field="_curator_dedup_id", id_generator_path="/path/to/output/exact_id_generator.json" # Required if assign_id=True ) result = removal_workflow.run() ``` #### ID Field Configuration **When `assign_id=True`**: * Duplicate IDs file contains `_curator_dedup_id` column * Set `duplicate_id_field="_curator_dedup_id"` * `id_generator_path` is required **When `assign_id=False`**: * Duplicate IDs file contains the column specified by `id_field` (e.g., `"id"`) * Set `duplicate_id_field` to match the ID column written by the identification workflow * `id_generator_path` not required To control the fields written by `TextDuplicatesRemovalWorkflow`, use `output_fields`. Keep `drop_id_field=False` (the default) and include `id_field` when specifying `output_fields` to retain the ID. Set it to `True` to omit the ID. The workflow rejects configurations that include the ID in `output_fields` while dropping it. ## Output Format The exact deduplication process produces the following directory structure: ```s output_path/ ├── ExactDuplicateIds/ # Duplicate identification results │ └── *.parquet # Parquet files with document IDs to remove └── exact_id_generator.json # ID generator mapping (if assign_id=True) ``` ### File Formats The workflow produces these output files: 1. **Duplicate IDs** (`ExactDuplicateIds/*.parquet`): * Contains document IDs to remove * Format: Parquet files with a single ID column * Column name depends on `assign_id`: * When `assign_id=True`: Column is `"_curator_dedup_id"` * When `assign_id=False`: Column matches the `id_field` parameter (e.g., `"id"`) * **Important**: Contains only the IDs of documents to remove, not the full document content 2. **ID Generator** (`exact_id_generator.json`): * JSON file containing ID generator state * Required for removal workflow when `assign_id=True` * Ensures consistent ID mapping across workflow stages #### Performance Considerations **Performance characteristics**: * Uses MD5 hashing over the configured text field to derive duplicate groups * Runs as a Ray-based workflow and writes duplicate IDs to the `ExactDuplicateIds/` directory * Stores only document IDs to remove in the output files, not full document content **Performance tuning**: * **Batch sizing**: Use smaller `input_blocksize` values (`256MiB` to `512MiB`) with a larger `identification_batchsize` to target 2-6 GB of overall batches as memory allows. For example, `input_blocksize="256MiB"` with `identification_batchsize=8` processes \~2 GB per insertion call. This improves both shuffle throughput and removal performance compared to the `2GiB` default. * **Shuffle partitions**: Setting `total_nparts` to smaller values (`256` or `512`) often leads to better shuffle performance compared to the defaults for really large runs. * **GPU memory**: For large cluster runs, tune `rmm_pool_size` and `spill_memory_limit` explicitly to match your hardware and dataset size. * **GPU writes**: NeMo Curator defaults `KVIKIO_AUTO_DIRECT_IO_WRITE=0` because KvikIO direct writes can reduce throughput on distributed filesystems such as Lustre. Setting it to `1` before importing NeMo Curator or starting the job may improve writes to a local SSD. Ensure every distributed worker inherits the same value, and benchmark both settings against the target filesystem. This affects GPU-backed cuDF writes from the exact deduplication workflow, not general CPU-based writes. See [Tune GPU Writes for the Target Filesystem](/reference/infra/gpu-processing#tune-gpu-writes-for-the-target-filesystem) for details. **Best practices**: * Clear the output directory between runs. * Use `assign_id=True` for consistent ID tracking. For comparison with other deduplication methods and guidance on when to use exact deduplication, refer to the [Deduplication overview ](/curate-text/process-data/deduplication).