> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server. # Process Data ## Docs - [Content Processing & Cleaning](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/content-processing.md): Clean, normalize, and transform text content to meet specific requirements including text cleaning and normalization - [Adding Document IDs](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/content-processing/add-id.md): Add unique identifiers to documents in your text dataset for tracking and deduplication workflows - [Text Cleaning](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/content-processing/text-cleaning.md): Remove undesirable text including improperly decoded Unicode characters, inconsistent spacing, and excessive URLs - [Deduplication](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/deduplication.md): Remove duplicate and near-duplicate documents efficiently using GPU-accelerated and semantic deduplication modules - [Exact Duplicate Removal](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/deduplication/exact.md): Identify and remove exact duplicates using MD5 hashing in a Ray-based workflow - [Fuzzy Duplicate Removal](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/deduplication/fuzzy.md): Identify and remove near-duplicate documents using MinHash and LSH with GPU acceleration - [Semantic Deduplication](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/deduplication/semdedup.md): Remove semantically redundant data using embeddings and clustering to identify meaning-based duplicates in large text datasets - [Language Management](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/language-management.md): Handle multilingual content and language-specific processing including language identification and stop word management - [Language Identification](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/language-management/language.md): Identify document languages accurately using FastText models supporting 176 languages for multilingual text processing - [Stop Words in Text Processing](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/language-management/stopwords.md): Manage high-frequency words to enhance text extraction and content detection with language-specific stop word lists - [Quality Assessment & Filtering](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/quality-assessment.md): Score and remove low-quality content using heuristics and ML classifiers with comprehensive filtering capabilities - [Classifier-Based Filtering](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/quality-assessment/classifier.md): Filter text using trained quality classifiers including FastText models and pre-trained language classification - [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/quality-assessment/distributed-classifier.md): Perform distributed data classification using GPU-accelerated models for domain, quality, safety, and content assessment - [Heuristic Filtering](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/quality-assessment/heuristic.md): Filter text using rule-based metrics to identify and remove low-quality documents with configurable thresholds - [Specialized Processing](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/specialized-processing.md): Domain-specific processing for code and advanced curation tasks with specialized modules - [Code Filtering](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/specialized-processing/code.md): Specialized filters for assessing and filtering code snippets and programming files with quality standards and pattern detection > **Note:** This page contains both a page directory (above) and the landing page content (below). The page directory is generated for agent use and does not appear on the landing page. > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server. # Process Data for Text Curation > Process text data using comprehensive filtering, deduplication, content processing, and specialized tools for high-quality datasets Process text data you've loaded through NeMo Curator's [pipeline architecture](/about/concepts/text/data/loading). NeMo Curator provides a comprehensive suite of tools for processing text data as part of the AI training pipeline. These tools help you analyze, transform, and filter your text datasets to ensure high-quality input for language model training. ## How it Works NeMo Curator's text processing capabilities are organized into five main categories: 1. **Language Management**: Handle multilingual content and language-specific processing 2. **Content Processing & Cleaning**: Clean, normalize, and transform text content 3. **Deduplication**: Remove duplicate and near-duplicate documents efficiently 4. **Quality Assessment & Filtering**: Score and remove low-quality content using heuristics and ML classifiers 5. **Specialized Processing**: Domain-specific processing for code and advanced curation tasks Each category provides specific implementations optimized for different curation needs. The result is a cleaned and filtered dataset ready for model training. --- ## Language Management Handle multilingual content and language-specific processing requirements. #### [Language Identification](/curate-text/process-data/language-management/language) Identify document languages and separate multilingual datasets #### [Stop Words](/curate-text/process-data/language-management/stopwords) Manage high-frequency words to enhance text extraction and content detection ## Content Processing & Cleaning Clean, normalize, and transform text content for high-quality training data. #### [Text Cleaning](/curate-text/process-data/content-processing/text-cleaning) Fix Unicode issues, standardize spacing, and remove URLs ## Deduplication Remove duplicate and near-duplicate documents efficiently from your text datasets. All deduplication methods support both identification (finding duplicates) and removal (filtering them out) workflows. #### [Exact Duplicate Removal](/curate-text/process-data/deduplication/exact) Identify and remove character-for-character duplicates using MD5 hashing #### [Fuzzy Duplicate Removal](/curate-text/process-data/deduplication/fuzzy) Identify and remove near-duplicates using MinHash and LSH similarity #### [Semantic Deduplication](/curate-text/process-data/deduplication/semdedup) Identify and remove semantically similar documents using embeddings and clustering ## Quality Assessment & Filtering Score and remove low-quality content using heuristics and ML classifiers. #### [Heuristic Filtering](/curate-text/process-data/quality-assessment/heuristic) Filter text using configurable rules and metrics #### [Classifier Filtering](/curate-text/process-data/quality-assessment/classifier) Filter text using trained quality classifiers #### [Distributed Classification](/curate-text/process-data/quality-assessment/distributed-classifier) GPU-accelerated classification with pre-trained models ## Specialized Processing Domain-specific processing for code and advanced curation tasks. #### [Code Processing](/curate-text/process-data/specialized-processing/code) Specialized filters for programming content and source code