Data Curation

Downloading and Extracting Text

Downloading a massive public dataset is usually the first step in data curation, and it can be cumbersome due to the dataset’s massive size and hosting method. This section describes how to download and extract large corpora efficiently.

Working with DocumentDataset

DocumentDataset is the standard format for datasets in NeMo Curator. This section describes how to get datasets in and out of this format, as well as how DocumentDataset interacts with the modules.

CPU and GPU Modules with Dask

NeMo Curator provides both CPU based modules and GPU based modules and supports methods for creating compatible Dask clusters and managing the dataset transfer between CPU and GPU.

Document Filtering

This section describes how to use the 30+ filters available within the NeMo Curator and implement custom filters to apply to the documents within the corpora.

Language Identification and Unicode Fixing

Large, unlabeled text corpora often contain a variety of languages. The NeMo Curator provides utilities to identify languages and fix improperly decoded Unicode characters.

GPU Accelerated Exact and Fuzzy Deduplication

Both exact and fuzzy deduplication functionalities are supported in NeMo Curator and accelerated using RAPIDS cuDF.

Classifier and Heuristic Quality Filtering

Classifier-based filtering involves training a small text classifer to label a document as either high quality or low quality. Heuristic-based filtering uses simple rules (e.g. Are there too many punctuation marks?) to score a document. NeMo Curator offers both classifier and heuristic-based quality filtering of documents.

Downstream Task Decontamination/Deduplication

After training, large language models are usually evaluated by their performance on downstream tasks consisting of unseen test data. When dealing with large datasets, there is a potential for leakage of this test data into the model’s training dataset. NeMo Curator allows you to remove sections of documents in your dataset that are present in downstream tasks.

Personally Identifiable Information Identification and Removal

The purpose of the personally identifiable information (PII) redaction tool is to help scrub sensitive data out of training datasets


Demonstration of how to run the NeMo Curator on a Dask Cluster deployed on top of Kubernetes

Previous Fine-tuning Stable Diffusion with DRaFT+
Next Downloading and Extracting Text
© Copyright 2023-2024, NVIDIA. Last updated on May 17, 2024.