Curate TextProcess DataContent Processing

Content Processing & Cleaning

View as Markdown

Clean, normalize, and transform text content to meet specific requirements for training language models using NeMo Curator’s tools and utilities.

Content processing involves transforming your text data while preserving essential information. This includes fixing encoding issues and standardizing text format to ensure high-quality input for model training.

How it Works

Content processing transformations typically modify documents in place or create new versions with specific changes. Most processing tools follow this pattern:

  1. Load your dataset using pipeline readers (JsonlReader, ParquetReader)
  2. Configure and apply the appropriate processor
  3. Save the transformed dataset for further processing

You can combine processing tools in sequence or use them alongside other curation steps like filtering and language management.

Available Processing Tools

Document IDs

Add unique identifiers to documents for tracking and deduplication identifiers tracking preprocessing deduplication

Text Cleaning

Fix Unicode issues, standardize spacing, and remove URLs unicode normalization preprocessing urls

Usage

Here’s an example of a typical content processing pipeline:

1from nemo_curator.core.client import RayClient
2from nemo_curator.pipeline import Pipeline
3from nemo_curator.stages.text.io.reader import JsonlReader
4from nemo_curator.stages.text.io.writer import JsonlWriter
5from nemo_curator.stages.text.modifiers import UnicodeReformatter, UrlRemover, NewlineNormalizer
6from nemo_curator.stages.text.modules import Modify
7
8# Initialize Ray client
9ray_client = RayClient()
10ray_client.start()
11
12# Create a comprehensive cleaning pipeline
13processing_pipeline = Pipeline(
14    name="content_processing_pipeline",
15    description="Comprehensive text cleaning and processing"
16)
17
18# Load dataset
19reader = JsonlReader(file_paths="input_data/")
20processing_pipeline.add_stage(reader)
21
22# Fix Unicode encoding issues
23processing_pipeline.add_stage(
24    Modify(modifier_fn=UnicodeReformatter(), input_fields="text")
25)
26
27# Standardize newlines
28processing_pipeline.add_stage(
29    Modify(modifier_fn=NewlineNormalizer(), input_fields="text")
30)
31
32# Remove URLs
33processing_pipeline.add_stage(
34    Modify(modifier_fn=UrlRemover(), input_fields="text")
35)
36
37# Save the processed dataset
38writer = JsonlWriter(path="processed_output/")
39processing_pipeline.add_stage(writer)
40
41# Execute pipeline
42results = processing_pipeline.run()
43
44# Stop Ray client
45ray_client.stop()

Common Processing Tasks

Text Normalization

  • Fix broken Unicode characters (mojibake)
  • Standardize whitespace and newlines
  • Remove or normalize special characters

Content Sanitization

  • Strip unwanted URLs or links
  • Remove boilerplate text or headers

Format Standardization

  • Ensure consistent text encoding
  • Normalize punctuation and spacing
  • Standardize document structure