> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/text/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# Text Concepts

## Docs

- [Loading](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/text/data/loading.md): Core concepts for loading and managing text datasets using pipeline-based readers and DocumentBatch tasks
- [Acquisition](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/text/data/acquisition.md): Core concepts for acquiring text data from remote sources including DocumentDownloader, DocumentIterator, and DocumentExtractor components
- [Processing](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/text/data/processing.md): Text processing workflows including quality filtering, fuzzy deduplication, content cleaning, and pipeline design
- [Curation Pipeline](https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/text/data/data-curation-pipeline.md): Comprehensive overview of NeMo Curator's text curation pipeline architecture including data acquisition and processing


> **Note:** This page contains both a page directory (above) and the landing page content (below). The page directory is generated for agent use and does not appear on the landing page.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/v26.09/about/concepts/text/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

> Essential concepts for text data curation including loading and processing.

# Text Curation Concepts

This document covers the essential concepts for text data curation in NVIDIA NeMo Curator. These concepts assume basic familiarity with data science and machine learning principles.

## Core Concept Areas

Text curation in NeMo Curator focuses on these key areas:

#### [Text Curation Pipeline](/about/concepts/text/data/data-curation-pipeline)

Comprehensive overview of the end-to-end text curation architecture and workflow
overview architecture

#### [Data Loading](/about/concepts/text/data/loading)

Core concepts for loading and managing text datasets from local files
local-files formats

#### [Data Acquisition](/about/concepts/text/data/acquisition)

Components for downloading and extracting data from remote sources
remote-sources download

#### [Data Processing](/about/concepts/text/data/processing)

Concepts for filtering, deduplication, and classification
filtering quality

## Infrastructure Components

The text curation concepts build on NVIDIA NeMo Curator's core infrastructure components, which are shared across all modalities. These components include:

#### [Memory Management](/reference/infra/memory-management)

Optimize memory usage when processing large datasets
partitioning
batching
monitoring

#### [GPU Acceleration](/reference/infra/gpu-processing)

Leverage NVIDIA GPUs for faster data processing
cuda
rmm
performance

#### [Resumable Processing](/reference/infra/resumable-processing)

Continue interrupted operations across large datasets
checkpoints
recovery
batching