Deduplication

Remove duplicate and near-duplicate documents from text datasets using NeMo Curator’s GPU-accelerated deduplication workflows. Removing duplicates prevents overrepresentation of repeated content in language model training.

NeMo Curator provides three deduplication approaches: exact matching (MD5 hashing), fuzzy matching (MinHash + LSH), and semantic matching (embeddings). All methods are GPU-accelerated and integrate with the data processing pipeline .

How It Works

NeMo Curator provides three deduplication approaches, each optimized for different duplicate types:

1
from nemo_curator.core.client import RayClient
2
from nemo_curator.stages.deduplication.exact.workflow import ExactDeduplicationWorkflow
3
4
ray_client = RayClient()
5
ray_client.start()
6
7
exact_workflow = ExactDeduplicationWorkflow(
8
    input_path="/path/to/input/data",
9
    output_path="/path/to/output",
10
    text_field="text",
11
    perform_removal=False,  # Identification only
12
    assign_id=True,
13
    input_filetype="parquet"
14
)
15
result = exact_workflow.run()
16
# result.metadata contains: total_time, num_duplicates, identification_time, id_generator_path

1
from nemo_curator.core.client import RayClient
2
from nemo_curator.stages.deduplication.fuzzy.workflow import FuzzyDeduplicationWorkflow
3
4
ray_client = RayClient()
5
ray_client.start()
6
7
fuzzy_workflow = FuzzyDeduplicationWorkflow(
8
    input_path="/path/to/input/data",
9
    cache_path="/path/to/cache",
10
    output_path="/path/to/output",
11
    text_field="text",
12
    perform_removal=False,  # Identification only
13
    input_blocksize="1GiB",
14
    seed=42,
15
    char_ngrams=24,
16
    num_bands=20,
17
    minhashes_per_band=13
18
)
19
result = fuzzy_workflow.run()
20
# result.metadata contains: total_time, num_duplicates, minhash_time, lsh_time, connected_components_pipeline_time, id_generator_path

1
from nemo_curator.stages.text.deduplication.semantic import TextSemanticDeduplicationWorkflow
2
3
text_workflow = TextSemanticDeduplicationWorkflow(
4
    input_path="/path/to/input/data",
5
    output_path="/path/to/output",
6
    cache_path="/path/to/cache",
7
    text_field="text",
8
    n_clusters=100,
9
    eps=0.01,  # Similarity threshold
10
    perform_removal=True  # Complete deduplication
11
)
12
result = text_workflow.run()
13
# result.metadata contains: total_time, num_duplicates, num_duplicates_removed

1
from nemo_curator.stages.deduplication.id_generator import create_id_generator_actor
2
from nemo_curator.stages.text.embedders.vllm import VLLMEmbeddingModelStage
3
from nemo_curator.stages.deduplication.semantic import SemanticDeduplicationWorkflow
4
5
# 1. Create ID generator
6
create_id_generator_actor()
7
8
# 2. Generate embeddings separately (using vLLM)
9
embedding_pipeline = Pipeline(
10
    stages=[
11
        ParquetReader(file_paths=input_path, _generate_ids=True),
12
        VLLMEmbeddingModelStage(
13
            model_identifier="google/embeddinggemma-300m",
14
            text_field="text",
15
            embedding_field="embeddings",
16
        ),
17
        ParquetWriter(path=embedding_output_path, fields=["_curator_dedup_id", "embeddings"])
18
    ]
19
)
20
embedding_out = embedding_pipeline.run()
21
22
# 3. Run clustering and pairwise similarity
23
semantic_workflow = SemanticDeduplicationWorkflow(
24
    input_path=embedding_output_path,
25
    output_path=semantic_workflow_path,
26
    n_clusters=100,
27
    id_field="_curator_dedup_id",
28
    embedding_field="embeddings",
29
    eps=None  # Skip duplicate identification for analysis
30
)
31
result = semantic_workflow.run()
32
33
# 4. Analyze results and choose eps parameter
34
# 5. Identify and remove duplicates

Deduplication Methods

Choose a deduplication method based on your needs:

Common Operations

Input File Discovery

The exact, fuzzy, semantic, and duplicate-removal workflows use input_filetype to select both the reader and the default file extensions. When input_file_extensions is omitted, the workflows discover these files:

Only "parquet" and "jsonl" are supported by these deduplication workflows. Other file types are rejected rather than falling back to the generic file partitioner defaults.

input_file_extensions follows these rules:

• None uses the defaults for input_filetype.
• A non-empty list replaces the defaults. Extensions can include or omit the leading dot and matching is case-insensitive.
• An empty list behaves like None; it does not disable file discovery.
• The extension selects files, but input_filetype still selects the reader. For example, input_file_extensions=[".pq"] with input_filetype="parquet" reads .pq files as Parquet.

For a single directory string, file discovery includes all nested subdirectories. For workflows that accept a list of input paths, each listed directory is scanned only at its top level; list individual files or directories when you do not want recursive discovery. Extension filtering also applies when an input path names a file directly.

1
from nemo_curator.stages.deduplication.exact.workflow import ExactDeduplicationWorkflow
2
3
workflow = ExactDeduplicationWorkflow(
4
    input_path="input_data/",
5
    output_path="results/",
6
    input_filetype="jsonl",
7
)
8
# Discovers input_data/**/*.jsonl and input_data/**/*.json.

Use the same input_filetype, input_file_extensions, input paths, and partitioning settings for identification and TextDuplicatesRemovalWorkflow when IDs are generated from input partitions. Changing file discovery or partition boundaries can change generated IDs and prevent removal from resolving them.

When you supply prebuilt initial_tasks, the workflow skips file discovery. In that mode, input-path, extension, and partitioning options do not change the supplied tasks.

Document IDs

Duplicate removal workflows require stable document identifiers. Choose one approach:

• Use AddId to add IDs at the start of your pipeline
• Use reader-based ID generation (_generate_ids, _assign_ids) backed by the ID Generator actor for stable integer IDs
• Use existing IDs if your documents already have unique identifiers

Some workflows write an ID generator state file (*_id_generator.json) for later removal when IDs are auto-assigned.

Removing Duplicates

Use TextDuplicatesRemovalWorkflow to apply duplicate IDs to your original dataset. Works with IDs from exact, fuzzy, or semantic deduplication.

1
from nemo_curator.stages.text.deduplication.removal_workflow import TextDuplicatesRemovalWorkflow
2
3
removal_workflow = TextDuplicatesRemovalWorkflow(
4
    input_path="/path/to/input",
5
    ids_to_remove_path="/path/to/duplicates",  # ExactDuplicateIds/, FuzzyDuplicateIds/, or duplicates/
6
    output_path="/path/to/clean",
7
    input_filetype="parquet",
8
    id_field="_curator_dedup_id",
9
    duplicate_id_field="_curator_dedup_id",
10
    id_generator_path="/path/to/id_generator.json"  # Required when IDs were auto-assigned
11
)
12
result = removal_workflow.run()
13
# result.metadata contains: total_time, num_duplicates_removed

1
removal_workflow = TextDuplicatesRemovalWorkflow(
2
    input_path="/path/to/input",
3
    ids_to_remove_path="/path/to/duplicates",
4
    output_path="/path/to/clean",
5
    input_filetype="parquet",
6
    input_file_extensions=[".pq"],
7
)

Workflow Results

All deduplication workflows return a WorkflowRunResult object with timing and duplicate count metadata:

1
from nemo_curator.pipeline.workflow import WorkflowRunResult
2
3
result = exact_workflow.run()
4
print(result.metadata)  # {"total_time": 42.1, "num_duplicates": 1500, ...}

Available metadata varies by workflow. Common keys include total_time and num_duplicates.

Outputs and Artifacts

Each deduplication method produces specific output files and directories:

Column names:

• _curator_dedup_id when assign_id=True or IDs are auto-assigned
• Matches id_field parameter when assign_id=False

Choosing a Deduplication Method

Compare deduplication methods to select the best approach for your dataset:

Quick Decision Guide

Use this guide to quickly select the right method:

• Start with Exact if you have numerous identical documents or need the fastest speed
• Use Fuzzy if you need to catch near-duplicates with minor formatting differences
• Use Semantic for meaning-based deduplication on large, diverse datasets

For detailed implementation guides, see:

• Exact Duplicate Removal
• Fuzzy Duplicate Removal
• Semantic Deduplication

1
from nemo_curator.core.client import RayClient
2
3
client = RayClient(
4
    num_cpus=64,    # Adjust based on available cores
5
    num_gpus=4      # Should be roughly 2x the memory of embeddings
6
)
7
client.start()
8
9
try:
10
    result = workflow.run()
11
finally:
12
    client.stop()

1
from nemo_curator.stages.deduplication.id_generator import (
2
    create_id_generator_actor,
3
    write_id_generator_to_disk,
4
    kill_id_generator_actor
5
)
6
7
create_id_generator_actor()
8
id_generator_path = "semantic_id_generator.json"
9
write_id_generator_to_disk(id_generator_path)
10
kill_id_generator_actor()
11
12
# Use saved ID generator in removal workflow
13
removal_workflow = TextDuplicatesRemovalWorkflow(
14
    input_path=input_path,
15
    ids_to_remove_path=duplicates_path,
16
    output_path=output_path,
17
    id_generator_path=id_generator_path,
18
    # ... other parameters
19
)

Next Steps

Ready to use deduplication?

• New to deduplication: Start with Exact Duplicate Removal for the fastest approach
• Need near-duplicate detection: See Fuzzy Duplicate Removal for MinHash-based matching
• Require semantic matching: Explore Semantic Deduplication for meaning-based deduplication

For hands-on guidance: See Text Curation Tutorials for step-by-step examples.