Text Curation Concepts | NeMo Curator

This document covers the essential concepts for text data curation in NVIDIA NeMo Curator. These concepts assume basic familiarity with data science and machine learning principles.

Core Concept Areas

Text curation in NeMo Curator focuses on these key areas:

Text Curation Pipeline

Comprehensive overview of the end-to-end text curation architecture and workflow

Data Loading

Core concepts for loading and managing text datasets from local files

Data Acquisition

Components for downloading and extracting data from remote sources

Data Processing

Concepts for filtering, deduplication, and classification

Infrastructure Components

The text curation concepts build on NVIDIA NeMo Curator’s core infrastructure components, which are shared across all modalities. These components include:

Memory Management

Optimize memory usage when processing large datasets

GPU Acceleration

Leverage NVIDIA GPUs for faster data processing

Resumable Processing

Continue interrupted operations across large datasets