Curate TextTutorials

Text Curation Tutorials

View as Markdown

Hands-on tutorials for text curation workflows are available in the tutorials/text directory of the NeMo Curator GitHub repository.

End-to-End Recipes

Nemotron-CLIMB Data Curation

Discover a pretraining data mixture by embedding, clustering, pruning, tokenizing, training proxy models, and fitting a performance predictor.

Key Concepts for Tutorial Success

Before diving into the tutorials, familiarize yourself with these essential NeMo Curator concepts:

Pipeline Architecture

Core processing stages and pipeline concepts for text curation workflows data-structures distributed

Quality Assessment

Scoring and filtering techniques used in tutorials heuristics classifiers

Data Loading

Loading data from various sources common-crawl custom-data

Distributed Classification

GPU-accelerated classification concepts gpu scalable