Deduplication

For fine-grained control, break semantic deduplication into separate stages:

1
from nemo_curator.stages.deduplication.id_generator import create_id_generator_actor
2
from nemo_curator.stages.text.embedders.vllm import VLLMEmbeddingModelStage
3
from nemo_curator.stages.deduplication.semantic import SemanticDeduplicationWorkflow
4
5
# 1. Create ID generator
6
create_id_generator_actor()
7
8
# 2. Generate embeddings separately (using vLLM)
9
embedding_pipeline = Pipeline(
10
    stages=[
11
        ParquetReader(file_paths=input_path, _generate_ids=True),
12
        VLLMEmbeddingModelStage(
13
            model_identifier="google/embeddinggemma-300m",
14
            text_field="text",
15
            embedding_field="embeddings",
16
        ),
17
        ParquetWriter(path=embedding_output_path, fields=["_curator_dedup_id", "embeddings"])
18
    ]
19
)
20
embedding_out = embedding_pipeline.run()
21
22
# 3. Run clustering and pairwise similarity
23
semantic_workflow = SemanticDeduplicationWorkflow(
24
    input_path=embedding_output_path,
25
    output_path=semantic_workflow_path,
26
    n_clusters=100,
27
    id_field="_curator_dedup_id",
28
    embedding_field="embeddings",
29
    eps=None  # Skip duplicate identification for analysis
30
)
31
result = semantic_workflow.run()
32
33
# 4. Analyze results and choose eps parameter
34
# 5. Identify and remove duplicates

This approach enables analysis of intermediate results and fine-grained control.

When assign_id=True (IDs auto-assigned):

• Duplicate IDs file contains _curator_dedup_id column
• Set ids_to_remove_duplicate_id_field="_curator_dedup_id"
• id_generator_path is required

When assign_id=False (using existing IDs):

• Duplicate IDs file contains the column specified by id_field (e.g., "id")
• Set ids_to_remove_duplicate_id_field to match your id_field value
• id_generator_path not required

Exact Deduplication:

• Removing identical copies of documents
• Fast initial deduplication pass
• Datasets with numerous exact duplicates
• When speed is more important than detecting near-duplicates

Fuzzy Deduplication:

• Removing near-duplicate documents with minor formatting differences
• Detecting documents with small edits or typos
• Fast deduplication when exact matching misses numerous duplicates
• When speed is important but some near-duplicate detection is needed

Semantic Deduplication:

• Removing semantically similar content (paraphrases, translations)
• Large, diverse web-scale datasets
• When meaning-based deduplication is more important than speed
• Advanced use cases requiring embedding-based similarity detection

You can combine deduplication methods for comprehensive duplicate removal:

1. Exact → Fuzzy → Semantic: Start with fastest methods, then apply more sophisticated methods
2. Exact → Semantic: Use exact for quick wins, then semantic for meaning-based duplicates
3. Fuzzy → Semantic: Use fuzzy for near-duplicates, then semantic for paraphrases

Run each method independently, then combine duplicate IDs before removal.

GPU Acceleration

All deduplication workflows require GPU acceleration:

• Exact: Ray backend with GPU support for MD5 hashing operations
• Fuzzy: Ray backend with GPU support for MinHash computation and LSH operations
• Semantic: GPU required for embedding generation (transformer models), K-means clustering, and pairwise similarity computation

GPU acceleration provides significant speedup for large datasets through parallel processing.

Hardware Requirements

• GPU: Required for all workflows (Ray with GPU support for exact/fuzzy, GPU for semantic)
• Memory: GPU memory requirements scale with dataset size, batch sizes, and embedding dimensions
• Executors: Can use various executors (XennaExecutor, RayDataExecutor) with GPU support

Backend Setup

For optimal performance with large datasets, configure Ray backend:

1
from nemo_curator.core.client import RayClient
2
3
client = RayClient(
4
    num_cpus=64,    # Adjust based on available cores
5
    num_gpus=4      # Should be roughly 2x the memory of embeddings
6
)
7
client.start()
8
9
try:
10
    result = workflow.run()
11
finally:
12
    client.stop()

For TB-scale datasets, consider distributed GPU clusters with Ray.

ID Generator for Large-Scale Operations

For large-scale duplicate removal, persist the ID Generator for consistent document tracking:

1
from nemo_curator.stages.deduplication.id_generator import (
2
    create_id_generator_actor,
3
    write_id_generator_to_disk,
4
    kill_id_generator_actor
5
)
6
7
create_id_generator_actor()
8
id_generator_path = "semantic_id_generator.json"
9
write_id_generator_to_disk(id_generator_path)
10
kill_id_generator_actor()
11
12
# Use saved ID generator in removal workflow
13
removal_workflow = TextDuplicatesRemovalWorkflow(
14
    input_path=input_path,
15
    ids_to_remove_path=duplicates_path,
16
    output_path=output_path,
17
    id_generator_path=id_generator_path,
18
    # ... other parameters
19
)

The ID Generator ensures consistent IDs across workflow stages.