NeMo CuratorProcess text data you’ve loaded through NeMo Curator’s pipeline architecture.
NeMo Curator provides a comprehensive suite of tools for processing text data as part of the AI training pipeline. These tools help you analyze, transform, and filter your text datasets to ensure high-quality input for language model training.
NeMo Curator’s text processing capabilities are organized into five main categories:
Each category provides specific implementations optimized for different curation needs. The result is a cleaned and filtered dataset ready for model training.
Handle multilingual content and language-specific processing requirements.
Identify document languages and separate multilingual datasets
Manage high-frequency words to enhance text extraction and content detection
Clean, normalize, and transform text content for high-quality training data.
Remove duplicate and near-duplicate documents efficiently from your text datasets. All deduplication methods support both identification (finding duplicates) and removal (filtering them out) workflows.
Identify and remove character-for-character duplicates using MD5 hashing
Identify and remove near-duplicates using MinHash and LSH similarity
Identify and remove semantically similar documents using embeddings and clustering
Score and remove low-quality content using heuristics and ML classifiers.
Filter text using configurable rules and metrics
Filter text using trained quality classifiers
GPU-accelerated classification with pre-trained models
Domain-specific processing for code and advanced curation tasks.
