> This page is for version 26.09 · v1.4.0.
> For other versions, use one of these documentation indexes:
> - Latest · v1.4.0 (26.09) (default): https://docs.nvidia.com/nemo/curator/latest/llms.txt
> - Main · preview: https://docs.nvidia.com/nemo/curator/main/llms.txt
> - 26.09 · v1.4.0: https://docs.nvidia.com/nemo/curator/v26.09/llms.txt
> - 26.07 · v1.3.0: https://docs.nvidia.com/nemo/curator/v26.07/llms.txt
> - 26.04 · v1.2.0: https://docs.nvidia.com/nemo/curator/v26.04/llms.txt
> - 26.02 · v1.1.0: https://docs.nvidia.com/nemo/curator/v26.02/llms.txt
> - 25.09 · v1.0.0: https://docs.nvidia.com/nemo/curator/v25.09/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

> Score and remove low-quality content using heuristics and machine learning classifiers

# Quality Assessment and Filtering

Score and remove low-quality content using heuristics and machine learning classifiers to prepare your data for model training.

Large datasets often contain many documents considered "low quality." In this context, low-quality data is unsuitable for downstream model training. High-quality data contains the characteristics that you want downstream models to learn. The metrics that define quality vary widely.

## How It Works

NeMo Curator's filtering framework includes several components that work within the [data processing architecture](/about/concepts/text/data/processing):

#### ScoreFilter

The `ScoreFilter` is at the center of filtering in NeMo Curator. It applies a filter to a document and optionally saves the score as metadata:

```python
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.io.reader import JsonlReader
from nemo_curator.stages.text.io.writer import JsonlWriter
from nemo_curator.stages.text.filters import ScoreFilter
from nemo_curator.stages.text.filters.heuristic import WordCountFilter

# Create pipeline
pipeline = Pipeline(name="quality_filtering")

# Load dataset
reader = JsonlReader(
    file_paths="books_dataset/*.jsonl",
    fields=["text", "id"]
)
pipeline.add_stage(reader)

# Create and apply filter
filter_stage = ScoreFilter(
    filter_obj=WordCountFilter(min_words=80),
    text_field="text",
    score_field="word_count",
)
pipeline.add_stage(filter_stage)

# Save filtered dataset
writer = JsonlWriter(path="long_books/")
pipeline.add_stage(writer)

# Execute pipeline (uses XennaExecutor by default)
results = pipeline.run()
```

> **Note**
>
> **Default Executor**: When you call `pipeline.run()` without specifying an executor, NeMo Curator automatically uses `XennaExecutor()` as the default. You can optionally specify a different executor by passing it as a parameter: `pipeline.run(executor=my_executor)`.

The filter object implements two key methods:

* `score_document`: Computes a quality score for a document.
* `keep_document`: Determines whether to keep a document based on its score.

#### Filter and Score Modules

For more specific use cases, NeMo Curator provides two specialized modules:

* `Score`: A module that only adds metadata scores to records without filtering.
  * Takes a scoring function that evaluates text and returns a score.
  * Adds the score to a specified metadata field.
  * Supports analysis or multistage filtering pipelines.

```python
# Example: Score documents without filtering
from nemo_curator.stages.text.filters import Score

scoring_step = Score(
    WordCountFilter().score_document,  # Use just the scoring part
    text_field="text",
    score_field="word_count"
)
scored_dataset = scoring_step.process(dataset)
```

Use `Filter` to apply conditions to metadata that already exists:

* `Filter`: A module that filters based on precomputed metadata.
  * Takes a filter function that evaluates metadata and returns `True` or `False`.
  * Uses existing metadata fields and does not compute new scores.
  * Efficiently filters on precomputed metrics.

```python
# Example: Filter using precomputed scores
from nemo_curator.stages.text.filters import Filter

filter_step = Filter(
    lambda score: score >= 100,  # Keep documents with score >= 100
    filter_field="word_count"
)
filtered_dataset = filter_step.process(scored_dataset)
```

You can combine these modules in pipelines:

```python
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.filters import Score, Filter
# Assume `word_counter` and `symbol_counter` are callables that return numeric scores
pipeline = Pipeline(name="multi_stage_filtering")
pipeline.add_stage(Score(word_counter, score_field="word_count"))
pipeline.add_stage(Score(symbol_counter, score_field="symbol_ratio"))
pipeline.add_stage(Filter(lambda x: x >= 100, filter_field="word_count"))
pipeline.add_stage(Filter(lambda x: x <= 0.3, filter_field="symbol_ratio"))
```

---

## Filtering Approaches

#### [Heuristic Filtering](/curate-text/process-data/quality-assessment/heuristic)

Filter text using configurable rules and metrics
rules
metrics
fast

#### [Classifier Filtering](/curate-text/process-data/quality-assessment/classifier)

Filter text using trained quality classifiers
ml-models
quality
scoring

#### [Distributed Classification](/curate-text/process-data/quality-assessment/distributed-classifier)

GPU-accelerated classification with pretrained models
gpu
distributed
scalable

#### [LLM Judge Workflow](/curate-text/process-data/quality-assessment/llm-judge)

Evaluate and filter records with config-driven large language model (LLM) judges and structured rubrics
llm
evaluation
structured-scoring

## Usage

NeMo Curator provides programmatic interfaces for document filtering through the Pipeline framework:

```python
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.io.reader import JsonlReader
from nemo_curator.stages.text.io.writer import JsonlWriter
from nemo_curator.stages.text.filters import ScoreFilter
from nemo_curator.stages.text.filters.heuristic import WordCountFilter

# Create and configure pipeline
pipeline = Pipeline(name="document_filtering")

# Add data loading
reader = JsonlReader(
    file_paths="/path/to/input/data/*.jsonl",
    fields=["text", "id"]
)
pipeline.add_stage(reader)

# Add filtering stage
filter_stage = ScoreFilter(
    filter_obj=WordCountFilter(min_words=80),
    text_field="text",
    score_field="word_count"
)
pipeline.add_stage(filter_stage)

# Add output stage
writer = JsonlWriter(path="/path/to/output/filtered/")
pipeline.add_stage(writer)

# Execute pipeline (uses XennaExecutor by default)
results = pipeline.run()
```

## Best Practices

When filtering large datasets, consider these performance tips:

1. **Order matters**: Place computationally inexpensive filters early in your pipeline.
2. **Batch size tuning**: Adjust batch sizes based on your hardware capabilities.
3. **Use vectorization**: Implement batched methods for compute-intensive filters.
4. **Disk I/O**: Consider compression and chunking strategies for large datasets.
5. **Distributed processing**: For TB-scale datasets, use distributed filtering with the XennaExecutor.

## Related Topics

Refer to the following related documentation:

* [Get Started with Text Curation](/get-started/text)
* [Process Data for Text Curation](/curate-text/process-data)
* [Deduplication](/curate-text/process-data/deduplication)
* [Language Management](/curate-text/process-data/language-management)