Process Data for Text Curation#

Process text data you’ve loaded through NeMo Curator’s pipeline architecture.

NeMo Curator provides a comprehensive suite of tools for processing text data as part of the AI training pipeline. These tools help you analyze, transform, and filter your text datasets to ensure high-quality input for language model training.

How it Works#

NeMo Curator’s text processing capabilities are organized into five main categories:

  1. Language Management: Handle multilingual content and language-specific processing

  2. Content Processing & Cleaning: Clean, normalize, and transform text content

  3. Deduplication: Remove duplicate and near-duplicate documents efficiently

  4. Quality Assessment & Filtering: Score and remove low-quality content using heuristics and ML classifiers

  5. Specialized Processing: Domain-specific processing for code and advanced curation tasks

Each category provides specific implementations optimized for different curation needs. The result is a cleaned and filtered dataset ready for model training.

Language Management#

Handle multilingual content and language-specific processing requirements.

Language Identification

Identify document languages and separate multilingual datasets

fasttext 176-languages detection

Language Identification
Stop Words

Manage high-frequency words to enhance text extraction and content detection

preprocessing filtering language-specific

Stop Words in Text Processing

Content Processing & Cleaning#

Clean, normalize, and transform text content for high-quality training data.

Text Cleaning

Fix Unicode issues, standardize spacing, and remove URLs

unicode normalization preprocessing

Text Cleaning

Deduplication#

Remove duplicate and near-duplicate documents efficiently from your text datasets. All deduplication methods support both identification (finding duplicates) and removal (filtering them out) workflows.

Exact Duplicate Removal

Identify and remove character-for-character duplicates using MD5 hashing

hashing fast gpu-accelerated

Exact Duplicate Removal
Fuzzy Duplicate Removal

Identify and remove near-duplicates using MinHash and LSH similarity

minhash lsh gpu-accelerated

Fuzzy Duplicate Removal
Semantic Deduplication

Identify and remove semantically similar documents using embeddings and clustering

embeddings meaning-based gpu-accelerated

Semantic Deduplication

Quality Assessment & Filtering#

Score and remove low-quality content using heuristics and ML classifiers.

Heuristic Filtering

Filter text using configurable rules and metrics

rules metrics fast

Heuristic Filtering
Classifier Filtering

Filter text using trained quality classifiers

ml-models quality scoring

Classifier-Based Filtering
Distributed Classification

GPU-accelerated classification with pre-trained models

gpu distributed scalable

Distributed Data Classification

Specialized Processing#

Domain-specific processing for code and advanced curation tasks.

Code Processing

Specialized filters for programming content and source code

programming syntax comments

Code Filtering