Deduplication

When assign_id=True (IDs auto-assigned):

• Duplicate IDs file contains _curator_dedup_id column
• Set ids_to_remove_duplicate_id_field="_curator_dedup_id"
• id_generator_path is required

When assign_id=False (using existing IDs):

• Duplicate IDs file contains the column specified by id_field (e.g., "id")
• Set ids_to_remove_duplicate_id_field to match your id_field value
• id_generator_path not required

Exact Deduplication:

• Removing identical copies of documents
• Fast initial deduplication pass
• Datasets with numerous exact duplicates
• When speed is more important than detecting near-duplicates

Fuzzy Deduplication:

• Removing near-duplicate documents with minor formatting differences
• Detecting documents with small edits or typos
• Fast deduplication when exact matching misses numerous duplicates
• When speed is important but some near-duplicate detection is needed

Semantic Deduplication:

• Removing semantically similar content (paraphrases, translations)
• Large, diverse web-scale datasets
• When meaning-based deduplication is more important than speed
• Advanced use cases requiring embedding-based similarity detection

You can combine deduplication methods for comprehensive duplicate removal:

1. Exact → Fuzzy → Semantic: Start with fastest methods, then apply more sophisticated methods
2. Exact → Semantic: Use exact for quick wins, then semantic for meaning-based duplicates
3. Fuzzy → Semantic: Use fuzzy for near-duplicates, then semantic for paraphrases

Run each method independently, then combine duplicate IDs before removal.

GPU Acceleration

All deduplication workflows require GPU acceleration:

• Exact: Ray backend with GPU support for MD5 hashing operations
• Fuzzy: Ray backend with GPU support for MinHash computation and LSH operations
• Semantic: GPU required for embedding generation (transformer models), K-means clustering, and pairwise similarity computation

GPU acceleration provides significant speedup for large datasets through parallel processing.

Hardware Requirements

• GPU: Required for all workflows (Ray with GPU support for exact/fuzzy, GPU for semantic)
• Memory: GPU memory requirements scale with dataset size, batch sizes, and embedding dimensions
• Executors: Can use various executors (XennaExecutor, RayDataExecutor) with GPU support

Backend Setup

For optimal performance with large datasets, configure Ray backend:

1
from nemo_curator.core.client import RayClient
2
3
client = RayClient(
4
    num_cpus=64,    # Adjust based on available cores
5
    num_gpus=4      # Should be roughly 2x the memory of embeddings
6
)
7
client.start()
8
9
try:
10
    workflow.run()
11
finally:
12
    client.stop()

For TB-scale datasets, consider distributed GPU clusters with Ray.

ID Generator for Large-Scale Operations

For large-scale duplicate removal, persist the ID Generator for consistent document tracking:

1
from nemo_curator.stages.deduplication.id_generator import (
2
    create_id_generator_actor,
3
    write_id_generator_to_disk,
4
    kill_id_generator_actor
5
)
6
7
create_id_generator_actor()
8
id_generator_path = "semantic_id_generator.json"
9
write_id_generator_to_disk(id_generator_path)
10
kill_id_generator_actor()
11
12
# Use saved ID generator in removal workflow
13
removal_workflow = TextDuplicatesRemovalWorkflow(
14
    input_path=input_path,
15
    ids_to_remove_path=duplicates_path,
16
    output_path=output_path,
17
    id_generator_path=id_generator_path,
18
    # ... other parameters
19
)

The ID Generator ensures consistent IDs across workflow stages.