> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v26.04/curate-text/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.
# Curate Text
## Docs
- [Tutorials](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/tutorials.md): Hands-on tutorials for text curation workflows including quality assessment with NeMo Curator
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/load-data.md): Load text data from Common Crawl, Wikipedia, and custom datasets using Curator.
- [ArXiv](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/load-data/arxiv.md): Download and extract text from arXiv using Curator.
- [Common Crawl](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/load-data/common-crawl.md): Download and extract text from Common Crawl web archives using Curator.
- [Custom Sources](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/load-data/custom.md): Create custom data loading pipelines using Curator.
- [Nemotron-Parse PDF Pipeline](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/load-data/nemotron-parse-pdf.md): Convert PDF datasets into interleaved Parquet output using NVIDIA's Nemotron-Parse VLM with the four-stage NemotronParsePDFReader composite pipeline
- [Read Existing Data](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/load-data/read-existing.md): Read existing JSONL and Parquet datasets using Curator's reader stages.
- [Wikipedia](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/load-data/wikipedia.md): Download and extract text from Wikipedia dumps using Curator.
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data.md): Process text data using comprehensive filtering, deduplication, content processing, and specialized tools for high-quality datasets
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/content-processing.md): Clean, normalize, and transform text content to meet specific requirements including text cleaning and normalization
- [Add IDs](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/content-processing/add-id.md): Add unique identifiers to documents in your text dataset for tracking and deduplication workflows
- [Text Cleaning](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/content-processing/text-cleaning.md): Remove undesirable text including improperly decoded Unicode characters, inconsistent spacing, and excessive URLs
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/embeddings.md): Generate text embeddings using vLLM, Sentence Transformers, or Hugging Face models for deduplication, similarity search, and downstream tasks
- [vLLM Embedder](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/embeddings/vllm-embedder.md): Generate text embeddings using vLLM for high-throughput GPU-accelerated inference with large embedding models
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/deduplication.md): Remove duplicate and near-duplicate documents efficiently using GPU-accelerated and semantic deduplication modules
- [Exact Deduplication](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/deduplication/exact.md): Identify and remove exact duplicates using MD5 hashing in a Ray-based workflow
- [Fuzzy Deduplication](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/deduplication/fuzzy.md): Identify and remove near-duplicate documents using MinHash and LSH with GPU acceleration
- [Semantic Deduplication](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/deduplication/semdedup.md): Remove semantically redundant data using embeddings and clustering to identify meaning-based duplicates in large text datasets
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/language-management.md): Handle multilingual content and language-specific processing including language identification and stop word management
- [Language Detection](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/language-management/language.md): Identify document languages accurately using FastText models supporting 176 languages for multilingual text processing
- [Stopwords](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/language-management/stopwords.md): Manage high-frequency words to enhance text extraction and content detection with language-specific stop word lists
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/quality-assessment.md): Score and remove low-quality content using heuristics and ML classifiers with comprehensive filtering capabilities
- [Classifier](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/quality-assessment/classifier.md): Filter text using trained quality classifiers including FastText models and pre-trained language classification
- [Distributed Classifier](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/quality-assessment/distributed-classifier.md): Perform distributed data classification using GPU-accelerated models for domain, quality, safety, and content assessment
- [Heuristic Filtering](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/quality-assessment/heuristic.md): Filter text using rule-based metrics to identify and remove low-quality documents with configurable thresholds
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/specialized-processing.md): Domain-specific processing for code and advanced curation tasks with specialized modules
- [Code Filtering](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/specialized-processing/code.md):
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/interleaved.md): Read, write, and filter MINT-1T-style interleaved image-text datasets across WebDataset and Parquet formats
- [Interleaved IO](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/interleaved/io.md): Read and write interleaved image-text datasets between WebDataset tar shards and Parquet using InterleavedParquetReader, InterleavedWebdatasetReader, and matching writers
- [Interleaved Filters](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/process-data/interleaved/filters.md): Filter interleaved image-text samples by image sharpness, QR-code area, CLIP image-text alignment, and image-to-text ratio
- [Save and Export](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/save-export.md): Tokenize and export text datasets to Megatron-LM binary format for large-scale language model pretraining
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/synthetic.md): Generate and augment training data using LLMs with NeMo Curator's synthetic data generation pipeline
- [LLM Client Setup](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/synthetic/llm-client.md): Configure LLM clients for synthetic data generation with NVIDIA APIs or custom endpoints
- [Inference Server](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/synthetic/inference-server.md): Serve LLMs locally via Ray Serve and vLLM alongside NeMo Curator pipelines using InferenceServer
- [NeMo Data Designer](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/synthetic/nemo-data-designer.md): Scale synthetic data generation using NeMo Data Designer (NDD) with declarative configuration and structured column generation
- [Multilingual Q&A](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/synthetic/multilingual-qa.md): Generate multilingual Q&A pairs using LLMs with NeMo Curator's synthetic data pipeline
- [Overview](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/synthetic/nemotron-cc.md): Advanced synthetic data generation using Nemotron-CC pipelines for text transformation and knowledge extraction
- [Task Reference](https://docs.nvidia.com/nemo/curator/v26.04/curate-text/synthetic/nemotron-cc/tasks.md): Reference documentation for Nemotron-CC synthetic data generation tasks and stages
> **Note:** This page contains both a page directory (above) and the landing page content (below). The page directory is generated for agent use and does not appear on the landing page.
> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v26.04/curate-text/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.
> Comprehensive text curation capabilities for preparing high-quality data for large language model training with loading, filtering, and quality assessment
# About Text Curation
NeMo Curator provides comprehensive text curation capabilities to prepare high-quality data for large language model (LLM) training. The toolkit includes a collection of processors for loading, filtering, formatting, and analyzing text data from various sources using a [pipeline-based architecture ](/about/concepts/text/data/data-curation-pipeline).
## Use Cases
* Clean and prepare web-scraped data from sources like Common Crawl, Wikipedia, and arXiv
* Create custom text curation pipelines for specific domain needs
* Scale text processing across CPU and GPU clusters efficiently
## Architecture
The following diagram provides a high-level outline of NeMo Curator's text curation architecture.
```mermaid
flowchart LR
A["Data Sources
(Cloud, Local,
Common Crawl, arXiv,
Wikipedia)"] --> B["Data Acquisition
& Loading"]
B --> C["Content Processing
& Cleaning"]
C --> D["Quality Assessment
& Filtering"]
D --> E["Deduplication
(Exact, Fuzzy,
Semantic)"]
E --> F["Curated Dataset
(JSONL/Parquet)"]
G["Ray + RAPIDS
(GPU-accelerated)"] -.->|"Distributed Execution"| B
G -.->|"Distributed Execution"| C
G -.->|"GPU Acceleration"| D
G -.->|"GPU Acceleration"| E
classDef stage fill:#e3f2fd,stroke:#1976d2,stroke-width:2px,color:#000
classDef infra fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px,color:#000
classDef output fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px,color:#000
class A,B,C,D,E stage
class F output
class G infra
```
---
## Introduction
Master the fundamentals of NeMo Curator and set up your text processing environment.
#### [Concepts](/about/concepts/text)
Learn about pipeline architecture and core processing stages for efficient text curation
data-structures
distributed
architecture
#### [Get Started](/get-started/text)
Learn prerequisites, setup instructions, and initial configuration for text curation
setup
configuration
quickstart
## Curation Tasks
### Download Data
Download text data from remote sources and import existing datasets into NeMo Curator's processing pipeline.
#### [Read Existing Data](/curate-text/load-data/read-existing)
Read existing JSONL and Parquet datasets using Curator's reader stages
jsonl
parquet
#### [arXiv](/curate-text/load-data/arxiv)
Download and extract scientific papers from arXiv
academic
pdf
latex
#### [Common Crawl](/curate-text/load-data/common-crawl)
Download and extract web archive data from Common Crawl
web-data
warc
distributed
#### [Wikipedia](/curate-text/load-data/wikipedia)
Download and extract Wikipedia articles from Wikipedia dumps
articles
multilingual
dumps
#### [Custom Data Sources](/curate-text/load-data/custom)
Implement a download and extract pipeline for a custom data source
jsonl
parquet
custom-formats
### Process Data
Transform and enhance your text data through comprehensive processing and curation steps.
#### [Language Management](/curate-text/process-data/language-management)
Handle multilingual content and language-specific processing
language-detection
stopwords
multilingual
#### [Content Processing & Cleaning](/curate-text/process-data/content-processing)
Clean, normalize, and transform text content
cleaning
normalization
formatting
#### [Deduplication](/curate-text/process-data/deduplication)
Remove duplicate and near-duplicate documents efficiently
fuzzy-dedup
semantic-dedup
exact-dedup
#### [Quality Assessment & Filtering](/curate-text/process-data/quality-assessment)
Score and remove low-quality content
heuristics
classifiers
quality-scoring
#### [Specialized Processing](/curate-text/process-data/specialized-processing)
Domain-specific processing for code and advanced curation tasks
code-processing
#### [Synthetic Data Generation](/curate-text/synthetic)
Generate and augment training data using LLMs
llm
augmentation
multilingual
nemotron-cc