Data Mining#

The Data Mining service of TAO Data Services helps you curate large, unlabeled or loosely labeled image pools by finding the samples that are most similar to a set of examples you care about. Instead of manually sifting through millions of images, you describe what you are looking for with a small set of target images, and the service mines the closest matches from a larger source pool.

Data mining is a two-stage pipeline:

  1. Embedding generation (embedding command) — encode every image in your source and target pools into a fixed-length feature vector (an embedding) using a CLIP, SigLIP, or TAO-trained vision-language model. Embeddings place visually and semantically similar images close together in vector space.

  2. Targeted mode matching (tmm command) — run a GPU-accelerated k-nearest-neighbor (k-NN) search that, for each target (query) image, retrieves the top-N most similar source images. Optional label and distance filters keep only the matches you want.

Stage 1 is optional. If you already have embeddings — for example, from Cosmos Embed or your own pipeline — you can skip embedding generation and run tmm directly on them. See Data Input for Targeted Mode Matching for the expected parquet schema.

Note

TMM stands for Targeted Mode Matching: you provide a target set that describes the concept or class you want more of, and the service matches source images against those targets. It is well suited for expanding a dataset with more examples of a specific class, mining hard or rare cases, and sourcing candidates for data augmentation.

Typical Workflow#

The two-stage TAO data mining pipeline. Source images and target images are each embedded with the same model to produce source and target embedding parquets, which feed the TMM targeted-mode-matching step. TMM writes a mined parquet and a mining summary.

The two-stage data mining pipeline: both image pools are embedded with the same model, then TMM matches source images to each target and writes out the mined set.#

The source pool is the large collection you want to mine from; the target set is a small collection of examples that describe what you are looking for. Both pools are embedded with the same model, then TMM matches source images to each target and writes out the unique list of mined source images.

When to Use Data Mining#

  • Dataset expansion — grow the number of examples for an under-represented class by mining visually similar images from a large unlabeled pool.

  • Data augmentation sourcing — find real images similar to a small curated set to use as augmentation candidates.

  • Hard-example and rare-case mining — surface samples that resemble known difficult or rare cases so they can be labeled and added to training.

  • Deduplication and curation — identify near-duplicate or highly similar images.

Prerequisites#

  • A source parquet and a target parquet, each containing a filepath column that points to the images to mine from and to match against, respectively. An optional label column enables same-class filtering during matching.

  • A supported embedding model: a Hugging Face CLIP or SigLIP checkpoint, or a TAO-trained CLIP checkpoint with its experiment spec.

  • The tmm matching step is GPU-accelerated and requires a CUDA-capable GPU (it uses RAPIDS cuDF/cuML for the k-NN search).

Both commands run inside the TAO Data Services (tao_ds) container.