Fuzzy Duplicate Removal

Find and remove near-duplicate documents with small edits or reformatting using MinHash and Locality Sensitive Hashing (LSH). This approach identifies candidate pairs with a similarity threshold efficiently at scale on GPU.

For other approaches, refer to Deduplication .

How It Works

Fuzzy deduplication uses MinHash and LSH to find near-duplicate content:

1. Computes MinHash signatures over character n-grams
2. Uses Locality Sensitive Hashing (LSH) to find candidate matches
3. Builds a graph of duplicate relationships
4. Identifies groups of near-duplicate documents

Ideal for detecting documents with minor differences such as formatting changes, typos, or small edits, where documents share a high degree of overlapping content.

Before You Start

Prerequisites:

• Ray cluster with GPU support (required for distributed processing)
• Stable document identifiers for removal (either existing IDs or IDs generated by the workflow and removal stages)

Quick Start

Get started with fuzzy deduplication using the following example of identifying duplicates, then remove them:

from nemo_curator.core.client import RayClient
from nemo_curator.stages.deduplication.fuzzy.workflow import FuzzyDeduplicationWorkflow
from nemo_curator.stages.text.deduplication.removal_workflow import TextDuplicatesRemovalWorkflow
ray_client = RayClient()
ray_client.start()
# Keep this value identical for identification and removal when using auto-generated IDs.
input_blocksize = "1GiB"
# Step 1: Identify duplicates
fuzzy_workflow = FuzzyDeduplicationWorkflow(
    input_path="input_data/",
    cache_path="./cache",
    output_path="./results",
    text_field="text",
    perform_removal=False,
    input_filetype="parquet",
    input_blocksize=input_blocksize,
    char_ngrams=24,
    num_bands=20,
    minhashes_per_band=13
)
result = fuzzy_workflow.run()
# result.metadata contains: total_time, num_duplicates, minhash_time, lsh_time, connected_components_pipeline_time, id_generator_path
# Step 2: Remove duplicates
removal_workflow = TextDuplicatesRemovalWorkflow(
    input_path="input_data/",
    ids_to_remove_path="./results/FuzzyDuplicateIds",
    output_path="./deduplicated",
    input_filetype="parquet",
    input_blocksize=input_blocksize,
    id_field="_curator_dedup_id",
    duplicate_id_field="_curator_dedup_id",
    id_generator_path="./results/fuzzy_id_generator.json"
)
result = removal_workflow.run()
# result.metadata contains: total_time, num_duplicates_removed

Configuration

Configure fuzzy deduplication using these key parameters:

For recursive discovery behavior, custom suffix examples, and supported formats, see Input File Discovery.

When you pass a DocumentBatch to MinHashStage, its data must contain the configured text_field and the _curator_dedup_id column. The stage assigns IDs at read time only for FileGroupTask inputs. read_format and read_kwargs apply only to those file inputs.

Disable Weighted Connected-Components Assignment

By default, BucketsToEdgesStage records each task’s edge count and the Ray Actor Pool executor uses it to balance connected-components work across RAFT actors. When composing the connected-components pipeline directly, set USE_TASK_WEIGHTS=False on ConnectedComponentsStage to restore equal task-count assignment:

from nemo_curator.backends.ray_actor_pool import RayActorPoolExecutor
from nemo_curator.backends.utils import RayStageSpecKeys
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.deduplication.fuzzy.buckets_to_edges import BucketsToEdgesStage
from nemo_curator.stages.deduplication.fuzzy.connected_components import ConnectedComponentsStage
from nemo_curator.stages.deduplication.fuzzy.identify_duplicates import IdentifyDuplicatesStage
connected_components = ConnectedComponentsStage(
    output_path="cache/",
).with_(
    ray_stage_spec={
        RayStageSpecKeys.USE_TASK_WEIGHTS: False,
    }
)
pipeline = Pipeline(
    name="connected_components_pipeline",
    stages=[
        BucketsToEdgesStage(output_path="cache/"),
        connected_components,
        IdentifyDuplicatesStage(output_path="results/"),
    ],
)
pipeline.run(
    executor=RayActorPoolExecutor(),
    initial_tasks=lsh_tasks,
)

Here, lsh_tasks is the output of the preceding fuzzy LSH pipeline. FuzzyDeduplicationWorkflow constructs this pipeline internally and keeps weighted assignment enabled. Disabling weights is mainly useful for comparisons or troubleshooting; skewed edge counts can otherwise leave some actors with substantially more work.

Similarity Threshold

Control matching strictness with num_bands and minhashes_per_band:

• More strict matching: Decrease num_bands or increase minhashes_per_band
• Less strict matching: Increase num_bands or decrease minhashes_per_band

Default (num_bands=20, minhashes_per_band=13) provides a balanced trade-off between recall and precision for many datasets. As a rule of thumb, the similarity threshold is approximately (1 / num_bands) ** (1 / minhashes_per_band). The exact similarity at which pairs are detected depends on your data distribution.

# Example: stricter matching (fewer pairs detected, higher required similarity)
fuzzy_workflow = FuzzyDeduplicationWorkflow(
    num_bands=15,           # Fewer bands = stricter matching
    minhashes_per_band=15  # More hashes per band = stricter matching
)
# Example: less strict matching (more pairs detected, lower required similarity)
fuzzy_workflow = FuzzyDeduplicationWorkflow(
    num_bands=25,           # More bands = less strict matching
    minhashes_per_band=10  # Fewer hashes per band = less strict matching
)

Removing Duplicates

After identifying duplicates, use TextDuplicatesRemovalWorkflow to remove them:

from nemo_curator.stages.text.deduplication.removal_workflow import TextDuplicatesRemovalWorkflow
removal_workflow = TextDuplicatesRemovalWorkflow(
    input_path="/path/to/input/data",
    ids_to_remove_path="/path/to/output/FuzzyDuplicateIds",
    output_path="/path/to/deduplicated",
    input_filetype="parquet",
    # Set input_file_extensions here too if identification used a custom suffix.
    input_blocksize="1GiB",  # Must match the identification workflow when IDs were auto-assigned
    id_field="_curator_dedup_id",
    duplicate_id_field="_curator_dedup_id",
    id_generator_path="/path/to/output/fuzzy_id_generator.json"  # Required if IDs were auto-assigned
)
result = removal_workflow.run()

Set drop_id_field=True to omit the deduplication ID from removal output. The default is False. If you set it to True, do not include the ID in output_fields.

Output Format

The fuzzy deduplication process produces the following directory structure:

cache_path/
├── MinHashStage/                    # MinHash signatures
│   └── *.parquet
├── LSHStage/                        # LSH buckets
│   └── *.parquet
├── BucketsToEdges/                  # Graph edges
│   └── *.parquet
└── ConnectedComponents/             # Connected components
    └── *.parquet
output_path/
├── FuzzyDuplicateIds/               # Duplicate identification results
│   └── *.parquet                    # Parquet files with document IDs to remove
└── fuzzy_id_generator.json          # ID generator mapping (if IDs were auto-assigned)

File Formats

The workflow produces these output files:

1. Duplicate IDs (FuzzyDuplicateIds/*.parquet):

   • Contains document IDs to remove
   • Format: Parquet files with column: ["_curator_dedup_id"]
   • Important: Contains only the IDs of documents to remove, not the full document content

2. ID Generator (fuzzy_id_generator.json):

   • JSON file containing ID generator state
   • Required for removal workflow when IDs were auto-assigned
   • Ensures consistent ID mapping across workflow stages

3. Cache Files (cache_path/):

   • Intermediate results for debugging and analysis
   • Can be reused if re-running with different parameters
   • Clear cache between runs if parameters change significantly

For comparison with other deduplication methods and guidance on when to use fuzzy deduplication, refer to the Deduplication overview .