Duplicate Removal

Use clip-level embeddings to identify and remove near-duplicate video clips so your dataset remains compact, diverse, and efficient to train on.

How it Works

Duplicate removal operates on clip-level embeddings produced during processing:

1. Inputs

   • Parquet batches from ClipWriterStage under iv2_embd_parquet/ or ce1_embd_parquet/
   • Columns: id, embedding

2. Outputs

   • Cluster: KMeansStage partitions embeddings and writes centroid distances (for example, cosine_dist_to_cent).
   • Pairwise: PairwiseStage computes within-cluster similarity on GPU and, for each clip, emits max_id and cosine_sim_score. Ranking controls whether to prefer outliers (“hard”) or representatives (“easy”).
   • Identify: IdentifyDuplicatesStage filters pairs with cosine_sim_score >= 1.0 - eps and writes Parquet files of duplicate ids for removal during export.

Before You Start

• Verify local paths or configure S3-compatible credentials. Provide storage_options in read/write keyword arguments when reading or writing cloud paths.
• Create output directories for KMeansStage, PairwiseStage, and IdentifyDuplicatesStage.

Quickstart

Use the generic semantic duplicate-removal stages with clip embeddings written to Parquet.

1
from nemo_curator.stages.deduplication.semantic.kmeans import KMeansStage
2
from nemo_curator.stages.deduplication.semantic.pairwise import PairwiseStage
3
from nemo_curator.stages.deduplication.semantic.ranking import RankingStrategy
4
from nemo_curator.stages.deduplication.semantic.identify_duplicates import IdentifyDuplicatesStage
5
6
kmeans = KMeansStage(
7
    n_clusters=1000,
8
    id_field="id",
9
    embedding_field="embedding",
10
    input_path="/path/to/embeddings/",
11
    output_path="/path/to/kmeans_out/",
12
    input_filetype="parquet",
13
)
14
15
pairwise = PairwiseStage(
16
    id_field="id",
17
    embedding_field="embedding",
18
    input_path="/path/to/kmeans_out/",
19
    output_path="/path/to/pairwise_out/",
20
    ranking_strategy=RankingStrategy.metadata_based(
21
        metadata_cols=["cosine_dist_to_cent", "id"],
22
        ascending=[True, True],
23
    ),
24
)
25
26
identify = IdentifyDuplicatesStage(
27
    output_path="/path/to/duplicates/",
28
    eps=0.1,
29
)

Input format: Parquet with columns id and embedding (produced by the video pipeline’s embedding stages and writer). Duplicate removal operates at the clip level using these embeddings. The IdentifyDuplicatesStage writes Parquet files containing duplicate ids; perform removal by filtering out rows whose id appears in those files during export.

Parameters