> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/latest/curate-text/process-data/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

> Process text data using language management, translation, filtering, deduplication, content processing, and specialized tools for high-quality datasets

# Process Data for Text Curation

Process text data that you loaded through NeMo Curator's [pipeline architecture](/about/concepts/text/data/loading).

NeMo Curator provides tools for processing text data as part of a model training pipeline. These tools help you analyze, transform, and filter datasets to create high-quality training input.

## How It Works

NeMo Curator's text processing capabilities are organized into six main categories:

1. **Language Management**: Handle multilingual content, translation, and language-specific processing.
2. **Content Processing and Cleaning**: Clean, normalize, and transform text content.
3. **Deduplication**: Remove duplicate and near-duplicate documents efficiently.
4. **Quality Assessment and Filtering**: Score and remove low-quality content using heuristics and machine learning classifiers.
5. **Specialized Processing**: Apply domain-specific processing for code and advanced curation tasks.
6. **Interleaved Datasets**: Read, write, and filter MINT-1T-style image-text datasets.

Each category provides specific implementations optimized for different curation needs. The result is a cleaned and filtered dataset ready for model training.

---

## Language Management

Handle multilingual content, translation, and language-specific processing requirements.

#### [Language Identification](/curate-text/process-data/language-management/language)

Identify document languages and separate multilingual datasets
fasttext
176-languages
detection

#### [Stop Words](/curate-text/process-data/language-management/stopwords)

Manage high-frequency words to enhance text extraction and content detection
preprocessing
filtering
language-specific

#### [Translation](/curate-text/process-data/language-management/translation)

Translate flat or structured text fields with optional FAITH and round-trip evaluation
translation
experimental
wildcard-fields
faith

## Content Processing and Cleaning

Clean, normalize, and transform text content for high-quality training data.

#### [Text Cleaning](/curate-text/process-data/content-processing/text-cleaning)

Fix Unicode issues, standardize spacing, and remove URLs
unicode
normalization
preprocessing

## Deduplication

Remove duplicate and near-duplicate documents efficiently from your text datasets. All deduplication methods support both identification (finding duplicates) and removal (filtering them out) workflows.

#### [Exact Duplicate Removal](/curate-text/process-data/deduplication/exact)

Identify and remove character-for-character duplicates using MD5 hashing
hashing
fast
gpu-accelerated

#### [Fuzzy Duplicate Removal](/curate-text/process-data/deduplication/fuzzy)

Identify and remove near-duplicates using MinHash and LSH similarity
minhash
lsh
gpu-accelerated

#### [Semantic Deduplication](/curate-text/process-data/deduplication/semdedup)

Identify and remove semantically similar documents using embeddings and clustering
embeddings
meaning-based
gpu-accelerated

## Quality Assessment and Filtering

Score and remove low-quality content using heuristics and machine learning classifiers.

#### [Heuristic Filtering](/curate-text/process-data/quality-assessment/heuristic)

Filter text using configurable rules and metrics
rules
metrics
fast

#### [Classifier Filtering](/curate-text/process-data/quality-assessment/classifier)

Filter text using trained quality classifiers
ml-models
quality
scoring

#### [Distributed Classification](/curate-text/process-data/quality-assessment/distributed-classifier)

GPU-accelerated classification with pretrained models
gpu
distributed
scalable

#### [LLM Judge Workflow](/curate-text/process-data/quality-assessment/llm-judge)

Evaluate and filter records with config-driven large language model (LLM) judges and structured rubrics
llm
evaluation
structured-scoring

## Specialized Processing

Domain-specific processing for code and advanced curation tasks.

#### [Code Processing](/curate-text/process-data/specialized-processing/code)

Specialized filters for programming content and source code
programming
syntax
comments

## Interleaved Datasets

Read, write, and filter MINT-1T-style image-text interleaved datasets across WebDataset and Parquet formats.

#### [Interleaved IO](/curate-text/process-data/interleaved/io)

Round-trip readers and writers between WebDataset tar shards and Parquet
parquet
webdataset
schema-utilities

#### [Interleaved Filters](/curate-text/process-data/interleaved/filters)

Sample-level filters for image quality, QR-code detection, CLIP alignment, and image-to-text ratio
blur
clip
qr-detection

> Process text data using language management, translation, filtering, deduplication, content processing, and specialized tools for high-quality datasets