Data Loading Concepts

This guide covers the core concepts for loading and managing text data from local files in NVIDIA NeMo Curator.

Pipeline-Based Data Loading

NeMo Curator uses a pipeline-based architecture for handling large-scale text data processing. Data flows through processing stages that transform tasks, enabling distributed processing of local files.

The system provides two primary readers for text data:

• JsonlReader - For JSON Lines format files (most common)
• ParquetReader - For columnar Parquet files (better performance for large datasets with PyArrow optimization)

Both readers support optimization through:

• Field selection - Reading specified columns to reduce memory usage
• Partitioning control - Using blocksize or files_per_partition to optimize DocumentBatch sizes during distributed processing
• Recommended block size - Use ~128MB for optimal object store performance with smaller data chunks

1
from nemo_curator.core.client import RayClient
2
from nemo_curator.pipeline import Pipeline
3
from nemo_curator.stages.text.io.reader import JsonlReader, ParquetReader
4
5
# Initialize Ray client
6
ray_client = RayClient()
7
ray_client.start()
8
9
# Basic usage with optimization
10
pipeline = Pipeline(name="data_processing")
11
12
# Define file type (example)
13
file_type = "jsonl"  # or "parquet" based on your data
14
15
if file_type == "jsonl":
16
    # JSONL reader with field selection and partitioning
17
    jsonl_reader = JsonlReader(
18
        file_paths="/path/to/jsonl_directory",
19
        blocksize="128MB",  # Recommended for object store optimization
20
        fields=["text", "id"]  # Column selection for efficiency
21
    )
22
    pipeline.add_stage(jsonl_reader)
23
else:
24
    # Parquet reader with performance optimization
25
    parquet_reader = ParquetReader(
26
        file_paths="/path/to/parquet_directory",
27
        files_per_partition=4,  # Alternative to blocksize
28
        fields=["text", "metadata"]
29
    )
30
    pipeline.add_stage(parquet_reader)
31
32
# Execute pipeline
33
results = pipeline.run()
34
35
# Stop Ray client
36
ray_client.stop()

Optimization Strategies

Partitioning Control

1
# Option 1: Size-based partitioning (recommended)
2
reader = JsonlReader(
3
    file_paths="/path/to/data",
4
    blocksize="128MB"  # Optimal for object store performance
5
)
6
7
# Option 2: File count-based partitioning  
8
reader = JsonlReader(
9
    file_paths="/path/to/data",
10
    files_per_partition=16  # Match your cluster size
11
)

Performance Recommendations

• Block size and files per partition: Use ~128MB for optimal performance. Very large batches lead to memory overheads when passing data between stages through the object store, while very small batches induce overhead from processing many more tasks. We recommend ~128MB as a good balance. Try to avoid going below 32MB or above 1GiB partition sizes.
• Field selection: Specify fields parameter to read required columns only
• Engine choice: ParquetReader defaults to PyArrow with dtype_backend="pyarrow" for optimal performance and memory efficiency. If you encounter compatibility issues with certain data types or schemas, you can override these defaults through read_kwargs:

  1
  # Remove PyArrow dtype backend if compatibility issues arise
  2
  reader = ParquetReader(
  3
      file_paths="data.parquet",
  4
      read_kwargs={"dtype_backend": "numpy_nullable"}  # Falls back to Pandas default behavior
  5
  )

Data Export Options

NeMo Curator provides flexible export options for processed datasets:

1
from nemo_curator.stages.text.io.writer import JsonlWriter, ParquetWriter
2
3
# Add writers to pipeline after processing stages
4
pipeline.add_stage(JsonlWriter(path="output_directory/"))
5
# or
6
pipeline.add_stage(ParquetWriter(path="output_directory/"))

Common Loading Patterns

Multi-Source Data

1
# Combine multiple directories with same reader type
2
reader = JsonlReader(file_paths=[
3
    "dataset_v1/",
4
    "dataset_v2/", 
5
    "additional_data/"
6
])

Remote Data Sources

This page focuses on loading text data from local files using JsonlReader and ParquetReader. Both readers support remote storage locations (Amazon S3, Azure) when you provide remote file paths.

For downloading and processing data from remote sources like ArXiv, Common Crawl, and Wikipedia, refer to the Data Acquisition Concepts page which covers:

• URLGenerator, DocumentDownloader, DocumentIterator, DocumentExtractor components
• Built-in support for Common Crawl, ArXiv, Wikipedia, and custom sources
• Integration patterns with pipeline-based processing
• Configuration and scaling strategies

The data acquisition process produces standardized output that integrates seamlessly with the pipeline-based loading concepts described on this page.