> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v25.09/curate-text/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.
# Curate Text
## Docs
- [Text Curation Tutorials](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/tutorials.md): Hands-on tutorials for text curation workflows including quality assessment with NeMo Curator
- [Download Data](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/load-data.md): Load text data from Common Crawl, Wikipedia, and custom datasets using Curator.
- [ArXiv](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/load-data/arxiv.md): Download and extract text from arXiv using Curator.
- [Common Crawl](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/load-data/common-crawl.md): Download and extract text from Common Crawl web archives using Curator.
- [Custom Data Loading](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/load-data/custom.md): Create custom data loading pipelines using Curator.
- [Read Existing Data](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/load-data/read-existing.md): Read existing JSONL and Parquet datasets using Curator's reader stages.
- [Wikipedia](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/load-data/wikipedia.md): Download and extract text from Wikipedia dumps using Curator.
- [Process Data for Text Curation](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data.md): Process text data using comprehensive filtering, deduplication, content processing, and specialized tools for high-quality datasets
- [Content Processing & Cleaning](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/content-processing.md): Clean, normalize, and transform text content to meet specific requirements including text cleaning and normalization
- [Adding Document IDs](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/content-processing/add-id.md): Add unique identifiers to documents in your text dataset for tracking and deduplication workflows
- [Text Cleaning](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/content-processing/text-cleaning.md): Remove undesirable text including improperly decoded Unicode characters, inconsistent spacing, and excessive URLs
- [Deduplication](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/deduplication.md): Remove duplicate and near-duplicate documents efficiently using GPU-accelerated and semantic deduplication modules
- [Exact Duplicate Removal](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/deduplication/exact.md): Identify and remove exact duplicates using MD5 hashing in a Ray-based workflow
- [Fuzzy Duplicate Removal](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/deduplication/fuzzy.md): Identify and remove near-duplicate documents using MinHash and LSH with GPU acceleration
- [Semantic Deduplication](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/deduplication/semdedup.md): Remove semantically redundant data using embeddings and clustering to identify meaning-based duplicates in large text datasets
- [Language Management](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/language-management.md): Handle multilingual content and language-specific processing including language identification and stop word management
- [Language Identification](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/language-management/language.md): Identify document languages accurately using FastText models supporting 176 languages for multilingual text processing
- [Stop Words in Text Processing](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/language-management/stopwords.md): Manage high-frequency words to enhance text extraction and content detection with language-specific stop word lists
- [Quality Assessment & Filtering](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/quality-assessment.md): Score and remove low-quality content using heuristics and ML classifiers with comprehensive filtering capabilities
- [Classifier-Based Filtering](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/quality-assessment/classifier.md): Filter text using trained quality classifiers including FastText models and pre-trained language classification
- [Distributed Data Classification](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/quality-assessment/distributed-classifier.md): Perform distributed data classification using GPU-accelerated models for domain, quality, safety, and content assessment
- [Heuristic Filtering](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/quality-assessment/heuristic.md): Filter text using rule-based metrics to identify and remove low-quality documents with configurable thresholds
- [Specialized Processing](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/specialized-processing.md): Domain-specific processing for code and advanced curation tasks with specialized modules
- [Code Filtering](https://docs.nvidia.com/nemo/curator/v25.09/curate-text/process-data/specialized-processing/code.md): Specialized filters for assessing and filtering code snippets and programming files with quality standards and pattern detection
> **Note:** This page contains both a page directory (above) and the landing page content (below). The page directory is generated for agent use and does not appear on the landing page.
> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v25.09/curate-text/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.
# About Text Curation
> Comprehensive text curation capabilities for preparing high-quality data for large language model training with loading, filtering, and quality assessment
NeMo Curator provides comprehensive text curation capabilities to prepare high-quality data for large language model (LLM) training. The toolkit includes a collection of processors for loading, filtering, formatting, and analyzing text data from various sources using a [pipeline-based architecture](/about/concepts/text/data/data-curation-pipeline).
## Use Cases
* Clean and prepare web-scraped data from sources like Common Crawl, Wikipedia, and arXiv
* Create custom text curation pipelines for specific domain needs
* Scale text processing across CPU and GPU clusters efficiently
## Architecture
The following diagram provides a high-level outline of NeMo Curator's text curation architecture.
```mermaid
flowchart LR
A["Data Sources
(Cloud, Local,
Common Crawl, arXiv,
Wikipedia)"] --> B["Data Acquisition
& Loading"]
B --> C["Content Processing
& Cleaning"]
C --> D["Quality Assessment
& Filtering"]
D --> E["Deduplication
(Exact, Fuzzy,
Semantic)"]
E --> F["Curated Dataset
(JSONL/Parquet)"]
G["Ray + RAPIDS
(GPU-accelerated)"] -.->|"Distributed Execution"| B
G -.->|"Distributed Execution"| C
G -.->|"GPU Acceleration"| D
G -.->|"GPU Acceleration"| E
classDef stage fill:#e3f2fd,stroke:#1976d2,stroke-width:2px,color:#000
classDef infra fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px,color:#000
classDef output fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px,color:#000
class A,B,C,D,E stage
class F output
class G infra
```
---
## Introduction
Master the fundamentals of NeMo Curator and set up your text processing environment.
#### [Concepts](/about/concepts/text)
Learn about pipeline architecture and core processing stages for efficient text curation
#### [Get Started](/get-started/text)
Learn prerequisites, setup instructions, and initial configuration for text curation
## Curation Tasks
### Download Data
Download text data from remote sources and import existing datasets into NeMo Curator's processing pipeline.
#### [Read Existing Data](/curate-text/load-data/read-existing)
Read existing JSONL and Parquet datasets using Curator's reader stages
#### [arXiv](/curate-text/load-data/arxiv)
Download and extract scientific papers from arXiv
#### [Common Crawl](/curate-text/load-data/common-crawl)
Download and extract web archive data from Common Crawl
#### [Wikipedia](/curate-text/load-data/wikipedia)
Download and extract Wikipedia articles from Wikipedia dumps
#### [Custom Data Sources](/curate-text/load-data/custom)
Implement a download and extract pipeline for a custom data source
### Process Data
Transform and enhance your text data through comprehensive processing and curation steps.
#### [Language Management](/curate-text/process-data/language-management)
Handle multilingual content and language-specific processing
#### [Content Processing & Cleaning](/curate-text/process-data/content-processing)
Clean, normalize, and transform text content
#### [Deduplication](/curate-text/process-data/deduplication)
Remove duplicate and near-duplicate documents efficiently
#### [Quality Assessment & Filtering](/curate-text/process-data/quality-assessment)
Score and remove low-quality content
#### [Specialized Processing](/curate-text/process-data/specialized-processing)
Domain-specific processing for code and advanced curation tasks