Curate TextProcess Data

Process Data for Text Curation

View as Markdown

Process text data that you loaded through NeMo Curator’s pipeline architecture.

NeMo Curator provides tools for processing text data as part of a model training pipeline. These tools help you analyze, transform, and filter datasets to create high-quality training input.

How It Works

NeMo Curator’s text processing capabilities are organized into six main categories:

  1. Language Management: Handle multilingual content, translation, and language-specific processing.
  2. Content Processing and Cleaning: Clean, normalize, and transform text content.
  3. Deduplication: Remove duplicate and near-duplicate documents efficiently.
  4. Quality Assessment and Filtering: Score and remove low-quality content using heuristics and machine learning classifiers.
  5. Specialized Processing: Apply domain-specific processing for code and advanced curation tasks.
  6. Interleaved Datasets: Read, write, and filter MINT-1T-style image-text datasets.

Each category provides specific implementations optimized for different curation needs. The result is a cleaned and filtered dataset ready for model training.

Language Management

Handle multilingual content, translation, and language-specific processing requirements.

Language Identification

Identify document languages and separate multilingual datasets fasttext 176-languages detection

Stop Words

Manage high-frequency words to enhance text extraction and content detection preprocessing filtering language-specific

Translation

Translate flat or structured text fields with optional FAITH and round-trip evaluation translation experimental wildcard-fields faith

Content Processing and Cleaning

Clean, normalize, and transform text content for high-quality training data.

Text Cleaning

Fix Unicode issues, standardize spacing, and remove URLs unicode normalization preprocessing

Deduplication

Remove duplicate and near-duplicate documents efficiently from your text datasets. All deduplication methods support both identification (finding duplicates) and removal (filtering them out) workflows.

Exact Duplicate Removal

Identify and remove character-for-character duplicates using MD5 hashing hashing fast gpu-accelerated

Fuzzy Duplicate Removal

Identify and remove near-duplicates using MinHash and LSH similarity minhash lsh gpu-accelerated

Semantic Deduplication

Identify and remove semantically similar documents using embeddings and clustering embeddings meaning-based gpu-accelerated

Quality Assessment and Filtering

Score and remove low-quality content using heuristics and machine learning classifiers.

Heuristic Filtering

Filter text using configurable rules and metrics rules metrics fast

Classifier Filtering

Filter text using trained quality classifiers ml-models quality scoring

Distributed Classification

GPU-accelerated classification with pretrained models gpu distributed scalable

LLM Judge Workflow

Evaluate and filter records with config-driven large language model (LLM) judges and structured rubrics llm evaluation structured-scoring

Specialized Processing

Domain-specific processing for code and advanced curation tasks.

Code Processing

Specialized filters for programming content and source code programming syntax comments

Interleaved Datasets

Read, write, and filter MINT-1T-style image-text interleaved datasets across WebDataset and Parquet formats.

Interleaved IO

Round-trip readers and writers between WebDataset tar shards and Parquet parquet webdataset schema-utilities

Interleaved Filters

Sample-level filters for image quality, QR-code detection, CLIP alignment, and image-to-text ratio blur clip qr-detection