Text Curation Concepts#

This document covers the essential concepts for text data curation in NVIDIA NeMo Curator. These concepts assume basic familiarity with data science and machine learning principles.

Core Concept Areas#

Text curation in NVIDIA NeMo Curator focuses on these key areas:

Text Curation Pipeline

Comprehensive overview of the end-to-end text curation architecture and workflow

overview architecture

Text Data Curation Pipeline

Data Loading

Core concepts for loading and managing text datasets from local files

local-files formats

Data Loading Concepts

Data Acquisition

Components for downloading and extracting data from remote sources

remote-sources download

Data Acquisition Concepts

Data Processing

Concepts for filtering, deduplication, and classification

filtering quality

Text Processing Concepts

Data Generation

Concepts for generating high-quality synthetic text

synthetic llm

Data Generation Concepts

Infrastructure Components#

The text curation concepts build on NVIDIA NeMo Curator’s core infrastructure components, which are shared across all modalities. These components include:

Distributed Computing

Configure and manage distributed processing across multiple machines

dask clusters scaling

Distributed Computing Reference

Memory Management

Optimize memory usage when processing large datasets

partitioning batching monitoring

Memory Management Guide

GPU Acceleration

Leverage NVIDIA GPUs for faster data processing

cuda rmm performance

GPU Processing Guide