Process Data for Text Curation#

Process text data you’ve loaded through NeMo Curator’s pipeline architecture.

NeMo Curator provides a comprehensive suite of tools for processing text data as part of the AI training pipeline. These tools help you analyze, transform, and filter your text datasets to ensure high-quality input for language model training.

How it Works#

NeMo Curator’s text processing capabilities are organized into five main categories:

  1. Quality Assessment & Filtering: Score and remove low-quality content using heuristics and ML classifiers

  2. Deduplication: Remove duplicate and near-duplicate documents efficiently

  3. Content Processing & Cleaning: Clean, normalize, and transform text content

  4. Language Management: Handle multilingual content and language-specific processing

  5. Specialized Processing: Domain-specific processing for code and advanced curation tasks

Each category provides specific implementations optimized for different curation needs. The result is a cleaned and filtered dataset ready for model training.

Quality Assessment & Filtering#

Score and remove low-quality content using heuristics and ML classifiers.

Heuristic Filtering

Filter text using configurable rules and metrics

rules metrics fast

Heuristic Filtering
Classifier Filtering

Filter text using trained quality classifiers

ml-models quality scoring

Classifier-Based Filtering
Distributed Classification

GPU-accelerated classification with pre-trained models

gpu distributed scalable

Distributed Data Classification

Deduplication#

Remove duplicate and near-duplicate documents efficiently from your text datasets.

Exact Duplicate Removal

Identify character-for-character duplicates using hashing

hashing fast

Exact Duplicate Removal
Fuzzy Duplicate Removal

Identify near-duplicates using MinHash and LSH

minhash lsh gpu-accelerated

Fuzzy Duplicate Removal
Semantic Deduplication

Remove semantically similar documents using embeddings

embeddings meaning-based gpu-accelerated

Semantic Deduplication

Content Processing & Cleaning#

Clean, normalize, and transform text content for high-quality training data.

Text Cleaning

Fix Unicode issues, standardize spacing, and remove URLs

unicode normalization preprocessing

Text Cleaning

Language Management#

Handle multilingual content and language-specific processing requirements.

Language Identification

Identify document languages and separate multilingual datasets

fasttext 176-languages detection

Language Identification
Stop Words

Manage high-frequency words to enhance text extraction and content detection

preprocessing filtering language-specific

Stop Words in Text Processing

Specialized Processing#

Domain-specific processing for code and advanced curation tasks.

Code Processing

Specialized filters for programming content and source code

programming syntax comments

Code Filtering