Curate TextProcess Data

Process Data for Text Curation

View as Markdown

Process text data you’ve loaded through NeMo Curator’s pipeline architecture .

NeMo Curator provides a comprehensive suite of tools for processing text data as part of the AI training pipeline. These tools help you analyze, transform, and filter your text datasets to ensure high-quality input for language model training.

How it Works

NeMo Curator’s text processing capabilities are organized into six main categories:

  1. Language Management: Handle multilingual content, translation, and language-specific processing
  2. Content Processing & Cleaning: Clean, normalize, and transform text content
  3. Deduplication: Remove duplicate and near-duplicate documents efficiently
  4. Quality Assessment & Filtering: Score and remove low-quality content using heuristics and ML classifiers
  5. Specialized Processing: Domain-specific processing for code and advanced curation tasks
  6. Interleaved Datasets: Read, write, and filter MINT-1T-style image-text datasets

Each category provides specific implementations optimized for different curation needs. The result is a cleaned and filtered dataset ready for model training.

Language Management

Handle multilingual content, translation, and language-specific processing requirements.

Language Identification

Identify document languages and separate multilingual datasets fasttext 176-languages detection

Stop Words

Manage high-frequency words to enhance text extraction and content detection preprocessing filtering language-specific

Translation

Translate flat or structured text fields with optional FAITH and round-trip evaluation translation experimental wildcard-fields faith

Content Processing & Cleaning

Clean, normalize, and transform text content for high-quality training data.

Text Cleaning

Fix Unicode issues, standardize spacing, and remove URLs unicode normalization preprocessing

Deduplication

Remove duplicate and near-duplicate documents efficiently from your text datasets. All deduplication methods support both identification (finding duplicates) and removal (filtering them out) workflows.

Exact Duplicate Removal

Identify and remove character-for-character duplicates using MD5 hashing hashing fast gpu-accelerated

Fuzzy Duplicate Removal

Identify and remove near-duplicates using MinHash and LSH similarity minhash lsh gpu-accelerated

Semantic Deduplication

Identify and remove semantically similar documents using embeddings and clustering embeddings meaning-based gpu-accelerated

Quality Assessment & Filtering

Score and remove low-quality content using heuristics and ML classifiers.

Heuristic Filtering

Filter text using configurable rules and metrics rules metrics fast

Classifier Filtering

Filter text using trained quality classifiers ml-models quality scoring

Distributed Classification

GPU-accelerated classification with pre-trained models gpu distributed scalable

Specialized Processing

Domain-specific processing for code and advanced curation tasks.

Code Processing

Specialized filters for programming content and source code programming syntax comments

Interleaved Datasets

Read, write, and filter MINT-1T-style image-text interleaved datasets across WebDataset and Parquet formats.

Interleaved IO

Round-trip readers and writers between WebDataset tar shards and Parquet parquet webdataset schema-utilities

Interleaved Filters

Sample-level filters for image quality, QR-code detection, CLIP alignment, and image-to-text ratio blur clip qr-detection