> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server. # Process Data ## Docs - [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/content-processing.md): Clean, normalize, and transform text content to meet specific requirements including text cleaning and normalization - [Add IDs](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/content-processing/add-id.md): Add unique identifiers to documents in your text dataset for tracking and deduplication workflows - [Text Cleaning](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/content-processing/text-cleaning.md): Remove undesirable text including improperly decoded Unicode characters, inconsistent spacing, and excessive URLs - [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/embeddings.md): Generate text embeddings using vLLM, Sentence Transformers, or Hugging Face models for deduplication, similarity search, and downstream tasks - [vLLM Embedder](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/embeddings/vllm-embedder.md): Generate text embeddings using vLLM for high-throughput GPU-accelerated inference with large embedding models - [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/deduplication.md): Remove duplicate and near-duplicate documents efficiently using GPU-accelerated and semantic deduplication modules - [Exact Deduplication](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/deduplication/exact.md): Identify and remove exact duplicates using MD5 hashing in a Ray-based workflow - [Fuzzy Deduplication](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/deduplication/fuzzy.md): Identify and remove near-duplicate documents using MinHash and LSH with GPU acceleration - [Semantic Deduplication](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/deduplication/semdedup.md): Remove semantically redundant data using embeddings and clustering to identify meaning-based duplicates in large text datasets - [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/language-management.md): Handle multilingual content and language-specific processing including language identification and stop word management - [Language Detection](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/language-management/language.md): Identify document languages accurately using FastText models supporting 176 languages for multilingual text processing - [Stopwords](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/language-management/stopwords.md): Manage high-frequency words to enhance text extraction and content detection with language-specific stop word lists - [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/quality-assessment.md): Score and remove low-quality content using heuristics and ML classifiers with comprehensive filtering capabilities - [Classifier](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/quality-assessment/classifier.md): Filter text using trained quality classifiers including FastText models and pre-trained language classification - [Distributed Classifier](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/quality-assessment/distributed-classifier.md): Perform distributed data classification using GPU-accelerated models for domain, quality, safety, and content assessment - [Heuristic Filtering](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/quality-assessment/heuristic.md): Filter text using rule-based metrics to identify and remove low-quality documents with configurable thresholds - [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/specialized-processing.md): Domain-specific processing for code and advanced curation tasks with specialized modules - [Code Filtering](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/specialized-processing/code.md): - [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/interleaved.md): Read, write, and filter MINT-1T-style interleaved image-text datasets across WebDataset and Parquet formats - [Interleaved IO](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/interleaved/io.md): Read and write interleaved image-text datasets between WebDataset tar shards and Parquet using InterleavedParquetReader, InterleavedWebdatasetReader, and matching writers - [Interleaved Filters](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/interleaved/filters.md): Filter interleaved image-text samples by image sharpness, QR-code area, CLIP image-text alignment, and image-to-text ratio > **Note:** This page contains both a page directory (above) and the landing page content (below). The page directory is generated for agent use and does not appear on the landing page. > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server. > Process text data using comprehensive filtering, deduplication, content processing, and specialized tools for high-quality datasets # Process Data for Text Curation Process text data you've loaded through NeMo Curator's [pipeline architecture ](/about/concepts/text/data/loading). NeMo Curator provides a comprehensive suite of tools for processing text data as part of the AI training pipeline. These tools help you analyze, transform, and filter your text datasets to ensure high-quality input for language model training. ## How it Works NeMo Curator's text processing capabilities are organized into six main categories: 1. **Language Management**: Handle multilingual content and language-specific processing 2. **Content Processing & Cleaning**: Clean, normalize, and transform text content 3. **Deduplication**: Remove duplicate and near-duplicate documents efficiently 4. **Quality Assessment & Filtering**: Score and remove low-quality content using heuristics and ML classifiers 5. **Specialized Processing**: Domain-specific processing for code and advanced curation tasks 6. **Interleaved Datasets**: Read, write, and filter MINT-1T-style image-text datasets Each category provides specific implementations optimized for different curation needs. The result is a cleaned and filtered dataset ready for model training. --- ## Language Management Handle multilingual content and language-specific processing requirements. #### [Language Identification](/curate-text/process-data/language-management/language) Identify document languages and separate multilingual datasets fasttext 176-languages detection #### [Stop Words](/curate-text/process-data/language-management/stopwords) Manage high-frequency words to enhance text extraction and content detection preprocessing filtering language-specific ## Content Processing & Cleaning Clean, normalize, and transform text content for high-quality training data. #### [Text Cleaning](/curate-text/process-data/content-processing/text-cleaning) Fix Unicode issues, standardize spacing, and remove URLs unicode normalization preprocessing ## Deduplication Remove duplicate and near-duplicate documents efficiently from your text datasets. All deduplication methods support both identification (finding duplicates) and removal (filtering them out) workflows. #### [Exact Duplicate Removal](/curate-text/process-data/deduplication/exact) Identify and remove character-for-character duplicates using MD5 hashing hashing fast gpu-accelerated #### [Fuzzy Duplicate Removal](/curate-text/process-data/deduplication/fuzzy) Identify and remove near-duplicates using MinHash and LSH similarity minhash lsh gpu-accelerated #### [Semantic Deduplication](/curate-text/process-data/deduplication/semdedup) Identify and remove semantically similar documents using embeddings and clustering embeddings meaning-based gpu-accelerated ## Quality Assessment & Filtering Score and remove low-quality content using heuristics and ML classifiers. #### [Heuristic Filtering](/curate-text/process-data/quality-assessment/heuristic) Filter text using configurable rules and metrics rules metrics fast #### [Classifier Filtering](/curate-text/process-data/quality-assessment/classifier) Filter text using trained quality classifiers ml-models quality scoring #### [Distributed Classification](/curate-text/process-data/quality-assessment/distributed-classifier) GPU-accelerated classification with pre-trained models gpu distributed scalable ## Specialized Processing Domain-specific processing for code and advanced curation tasks. #### [Code Processing](/curate-text/process-data/specialized-processing/code) Specialized filters for programming content and source code programming syntax comments ## Interleaved Datasets Read, write, and filter MINT-1T-style image-text interleaved datasets across WebDataset and Parquet formats. #### [Interleaved IO](/curate-text/process-data/interleaved/io) Round-trip readers and writers between WebDataset tar shards and Parquet parquet webdataset schema-utilities #### [Interleaved Filters](/curate-text/process-data/interleaved/filters) Sample-level filters for image quality, QR-code detection, CLIP alignment, and image-to-text ratio blur clip qr-detection