> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For full documentation content, see https://docs.nvidia.com/nemo/curator/llms-full.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

> Domain-specific processing for code and advanced curation tasks with specialized modules

# Specialized Processing

Domain-specific processing for code and advanced curation tasks using NeMo Curator's specialized modules.

This section covers advanced processing techniques for specific data types and use cases that require specialized handling beyond general text processing. These tools are designed for specific domains like programming content.

## How It Works

Specialized processing modules in NeMo Curator are designed for specific data types and use cases:

* **Code Processing**: Handles programming languages with syntax-aware filtering

Each specialized processor understands the unique characteristics of its target domain and applies appropriate metrics and thresholds within the broader [data processing framework ](/about/concepts/text/data/processing).

***

## Available Specialized Tools

Specialized filters for programming content and source code
programming
syntax
comments
languages

## Usage

### Quick Examples

```python
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.filters import ScoreFilter
from nemo_curator.stages.text.filters.heuristic.code import PythonCommentToCodeFilter, NumberOfLinesOfCodeFilter
from nemo_curator.stages.text.io.reader import JsonlReader

# Filter Python code based on quality metrics
code_pipeline = Pipeline(
    name="code_processing_pipeline",
    stages=[
    JsonlReader(
        file_paths="code_data/*.jsonl",
        fields=["content"]
    ),
    ScoreFilter(
        filter_obj=PythonCommentToCodeFilter(
            min_comment_to_code_ratio=0.01,
            max_comment_to_code_ratio=0.8
        ),
        text_field="content",
        score_field="comment_ratio"
    ),
    ScoreFilter(
        filter_obj=NumberOfLinesOfCodeFilter(min_lines=5, max_lines=1000),
        text_field="content", 
        score_field="line_count"
    )
])

results = code_pipeline.run()
```

## When to Use Specialized Processing

* **Code datasets**: When working with programming content that needs syntax-aware filtering

## Performance Considerations

* **Code processing**: Fast heuristic-based filtering, suitable for large code repositories