> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server. # Process Data ## Docs - [Overview](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/content-processing.md): Clean, normalize, and transform text content to meet specific requirements including text cleaning and normalization - [Add IDs](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/content-processing/add-id.md): Add unique identifiers to documents in your text dataset for tracking and deduplication workflows - [Text Cleaning](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/content-processing/text-cleaning.md): Remove undesirable text including improperly decoded Unicode characters, inconsistent spacing, and excessive URLs - [Overview](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/embeddings.md): Generate text embeddings using vLLM, Sentence Transformers, or Hugging Face models for deduplication, similarity search, and downstream tasks - [vLLM Embedder](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/embeddings/vllm-embedder.md): Generate text embeddings using vLLM for high-throughput GPU-accelerated inference with large embedding models - [Overview](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/deduplication.md): Remove duplicate and near-duplicate documents efficiently using GPU-accelerated and semantic deduplication modules - [Exact Deduplication](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/deduplication/exact.md): Identify and remove exact duplicates using MD5 hashing in a Ray-based workflow - [Fuzzy Deduplication](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/deduplication/fuzzy.md): Identify and remove near-duplicate documents using MinHash and LSH with GPU acceleration - [Semantic Deduplication](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/deduplication/semdedup.md): Remove semantically redundant data using embeddings and clustering to identify meaning-based duplicates in large text datasets - [Overview](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/language-management.md): Handle multilingual content and language-specific processing including language identification, stop word management, and translation - [Language Detection](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/language-management/language.md): Identify document languages and scripts using FastText and GlotLID models for multilingual text processing - [Stopwords](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/language-management/stopwords.md): Manage high-frequency words to enhance text extraction and content detection with language-specific stop word lists - [Translation](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/language-management/translation.md): Translate flat and structured text fields with Curator's experimental translation pipeline, quality scoring, and backend integrations - [Overview](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/quality-assessment.md): Score and remove low-quality content using heuristics and machine learning classifiers - [Classifier](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/quality-assessment/classifier.md): Filter text using trained quality classifiers including FastText models and pre-trained language classification - [Distributed Classifier](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/quality-assessment/distributed-classifier.md): Perform distributed data classification using GPU-accelerated models for domain, quality, safety, and content assessment - [Heuristic Filtering](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/quality-assessment/heuristic.md): Filter text using rule-based metrics to identify and remove low-quality documents with configurable thresholds - [LLM Judge Workflow](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/quality-assessment/llm-judge.md): Evaluate and filter JSONL or Parquet records with config-driven LLM judges, Jinja prompts, structured rubrics, and local Dynamo/vLLM serving - [Overview](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/specialized-processing.md): Domain-specific processing for code and advanced curation tasks with specialized modules - [Code Filtering](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/specialized-processing/code.md): - [Overview](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/interleaved.md): Read, write, and filter MINT-1T-style interleaved image-text datasets across WebDataset and Parquet formats - [Interleaved IO](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/interleaved/io.md): Read and write interleaved image-text datasets between WebDataset tar shards and Parquet, and read Lance datasets - [Interleaved Filters](https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/interleaved/filters.md): Filter interleaved image-text samples by image sharpness, QR-code area, CLIP image-text alignment, and image-to-text ratio > **Note:** This page contains both a page directory (above) and the landing page content (below). The page directory is generated for agent use and does not appear on the landing page. > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v26.09/curate-text/process-data/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server. > Process text data using language management, translation, filtering, deduplication, content processing, and specialized tools for high-quality datasets # Process Data for Text Curation Process text data that you loaded through NeMo Curator's [pipeline architecture](/about/concepts/text/data/loading). NeMo Curator provides tools for processing text data as part of a model training pipeline. These tools help you analyze, transform, and filter datasets to create high-quality training input. ## How It Works NeMo Curator's text processing capabilities are organized into six main categories: 1. **Language Management**: Handle multilingual content, translation, and language-specific processing. 2. **Content Processing and Cleaning**: Clean, normalize, and transform text content. 3. **Deduplication**: Remove duplicate and near-duplicate documents efficiently. 4. **Quality Assessment and Filtering**: Score and remove low-quality content using heuristics and machine learning classifiers. 5. **Specialized Processing**: Apply domain-specific processing for code and advanced curation tasks. 6. **Interleaved Datasets**: Read, write, and filter MINT-1T-style image-text datasets. Each category provides specific implementations optimized for different curation needs. The result is a cleaned and filtered dataset ready for model training. --- ## Language Management Handle multilingual content, translation, and language-specific processing requirements. #### [Language Identification](/curate-text/process-data/language-management/language) Identify document languages and separate multilingual datasets fasttext 176-languages detection #### [Stop Words](/curate-text/process-data/language-management/stopwords) Manage high-frequency words to enhance text extraction and content detection preprocessing filtering language-specific #### [Translation](/curate-text/process-data/language-management/translation) Translate flat or structured text fields with optional FAITH and round-trip evaluation translation experimental wildcard-fields faith ## Content Processing and Cleaning Clean, normalize, and transform text content for high-quality training data. #### [Text Cleaning](/curate-text/process-data/content-processing/text-cleaning) Fix Unicode issues, standardize spacing, and remove URLs unicode normalization preprocessing ## Deduplication Remove duplicate and near-duplicate documents efficiently from your text datasets. All deduplication methods support both identification (finding duplicates) and removal (filtering them out) workflows. #### [Exact Duplicate Removal](/curate-text/process-data/deduplication/exact) Identify and remove character-for-character duplicates using MD5 hashing hashing fast gpu-accelerated #### [Fuzzy Duplicate Removal](/curate-text/process-data/deduplication/fuzzy) Identify and remove near-duplicates using MinHash and LSH similarity minhash lsh gpu-accelerated #### [Semantic Deduplication](/curate-text/process-data/deduplication/semdedup) Identify and remove semantically similar documents using embeddings and clustering embeddings meaning-based gpu-accelerated ## Quality Assessment and Filtering Score and remove low-quality content using heuristics and machine learning classifiers. #### [Heuristic Filtering](/curate-text/process-data/quality-assessment/heuristic) Filter text using configurable rules and metrics rules metrics fast #### [Classifier Filtering](/curate-text/process-data/quality-assessment/classifier) Filter text using trained quality classifiers ml-models quality scoring #### [Distributed Classification](/curate-text/process-data/quality-assessment/distributed-classifier) GPU-accelerated classification with pretrained models gpu distributed scalable #### [LLM Judge Workflow](/curate-text/process-data/quality-assessment/llm-judge) Evaluate and filter records with config-driven large language model (LLM) judges and structured rubrics llm evaluation structured-scoring ## Specialized Processing Domain-specific processing for code and advanced curation tasks. #### [Code Processing](/curate-text/process-data/specialized-processing/code) Specialized filters for programming content and source code programming syntax comments ## Interleaved Datasets Read, write, and filter MINT-1T-style image-text interleaved datasets across WebDataset and Parquet formats. #### [Interleaved IO](/curate-text/process-data/interleaved/io) Round-trip readers and writers between WebDataset tar shards and Parquet parquet webdataset schema-utilities #### [Interleaved Filters](/curate-text/process-data/interleaved/filters) Sample-level filters for image quality, QR-code detection, CLIP alignment, and image-to-text ratio blur clip qr-detection