Specialized Processing

Domain-specific processing for code and advanced curation tasks using NeMo Curator’s specialized modules.

This section covers advanced processing techniques for specific data types and use cases that require specialized handling beyond general text processing. These tools are designed for specific domains like programming content.

How It Works

Specialized processing modules in NeMo Curator are designed for specific data types and use cases:

• Code Processing: Handles programming languages with syntax-aware filtering

Each specialized processor understands the unique characteristics of its target domain and applies appropriate metrics and thresholds within the broader data processing framework .

Available Specialized Tools

Usage

Quick Examples

1
from nemo_curator.pipeline import Pipeline
2
from nemo_curator.stages.text.filters import ScoreFilter
3
from nemo_curator.stages.text.filters.heuristic.code import PythonCommentToCodeFilter, NumberOfLinesOfCodeFilter
4
from nemo_curator.stages.text.io.reader import JsonlReader
5
6
# Filter Python code based on quality metrics
7
code_pipeline = Pipeline(
8
    name="code_processing_pipeline",
9
    stages=[
10
    JsonlReader(
11
        file_paths="code_data/*.jsonl",
12
        fields=["content"]
13
    ),
14
    ScoreFilter(
15
        filter_obj=PythonCommentToCodeFilter(
16
            min_comment_to_code_ratio=0.01,
17
            max_comment_to_code_ratio=0.8
18
        ),
19
        text_field="content",
20
        score_field="comment_ratio"
21
    ),
22
    ScoreFilter(
23
        filter_obj=NumberOfLinesOfCodeFilter(min_lines=5, max_lines=1000),
24
        text_field="content", 
25
        score_field="line_count"
26
    )
27
])
28
29
results = code_pipeline.run()

When to Use Specialized Processing

• Code datasets: When working with programming content that needs syntax-aware filtering

Performance Considerations

• Code processing: Fast heuristic-based filtering, suitable for large code repositories