Heuristic Filtering | NeMo Curator

Heuristic filtering uses simple, rule-based metrics to identify and remove low-quality documents from your dataset. NVIDIA NeMo Curator provides prebuilt heuristic filters that you can configure and combine for your requirements.

How It Works

Heuristic filters examine specific attributes of text documents and apply predefined thresholds to determine document quality. Unlike classifier-based filtering, heuristic filters do not require training data. They rely on configurable thresholds and rules.

These filters assess quality using measurable document characteristics such as:

Document length (word or character count)
Punctuation ratios and patterns
Repetitive content detection
Language-specific patterns
Text completeness and coherence

For details on filter structure and the filtering process, refer to Data Processing Concepts .

Usage

Python

Configuration

from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.io.reader import JsonlReader
from nemo_curator.stages.text.io.writer import JsonlWriter
from nemo_curator.stages.text.filters import ScoreFilter
from nemo_curator.stages.text.filters.heuristic import WordCountFilter, PunctuationFilter
from nemo_curator.stages.text.filters.heuristic.repetition import RepeatingTopNGramsFilter
# Create pipeline
pipeline = Pipeline(name="heuristic_filtering")
# Load your dataset
reader = JsonlReader(
    file_paths="input_data/",
    fields=["text", "id"]
)
pipeline.add_stage(reader)
# Add filter stages
pipeline.add_stage(ScoreFilter(
    filter_obj=WordCountFilter(min_words=80),
    text_field="text",
    score_field="word_count"
))
pipeline.add_stage(ScoreFilter(
    filter_obj=PunctuationFilter(max_num_sentences_without_endmark_ratio=0.85),
    text_field="text"
))
pipeline.add_stage(ScoreFilter(
    filter_obj=RepeatingTopNGramsFilter(n=2, max_repeating_ngram_ratio=0.2),
    text_field="text"
))
pipeline.add_stage(ScoreFilter(
    filter_obj=RepeatingTopNGramsFilter(n=3, max_repeating_ngram_ratio=0.18),
    text_field="text"
))
pipeline.add_stage(ScoreFilter(
    filter_obj=RepeatingTopNGramsFilter(n=4, max_repeating_ngram_ratio=0.16),
    text_field="text"
))
# Add output stage
writer = JsonlWriter(path="high_quality_output/")
pipeline.add_stage(writer)
# Execute pipeline
results = pipeline.run()

Available Filters

NeMo Curator includes more than 30 heuristic filters for assessing document quality. Below are the most commonly used filters with their parameters:

Text Length Filters

The following filters measure document and word length:

Filter	Description	Key Parameters	Default Values
`WordCountFilter`	Filters by word count	`min_words`, `max_words`	`min=50`, `max=100000`
`TokenCountFilter`	Filters by token count	`min_tokens`, `max_tokens`	`min=0`, `max=∞`
`MeanWordLengthFilter`	Filters by average word length	`min_mean_word_length`, `max_mean_word_length`	`min=3`, `max=10`
`LongWordFilter`	Filters by presence of extremely long words	`max_word_length`	`1000`

Repetition Detection Filters

The following filters detect repeated lines, paragraphs, or n-grams:

Filter	Description	Key Parameters	Default Values
`RepeatedLinesFilter`	Detects repeated lines	`max_repeated_line_fraction`	`0.7`
`RepeatedParagraphsFilter`	Detects repeated paragraphs	`max_repeated_paragraphs_ratio`	`0.7`
`RepeatedLinesByCharFilter`	Detects repeated lines by character count	`max_repeated_lines_char_ratio`	`0.8`
`RepeatedParagraphsByCharFilter`	Detects repeated paragraphs by character count	`max_repeated_paragraphs_char_ratio`	`0.8`
`RepeatingTopNGramsFilter`	Detects excessive repetition of n-grams	`n`, `max_repeating_ngram_ratio`	`n=2`, `ratio=0.2`
`RepeatingDuplicateNGramsFilter`	Detects duplicate n-grams	`n`, `max_repeating_duplicate_ngram_ratio`	`n=2`, `ratio=0.2`

Character and Symbol Filters

The following filters measure character, symbol, numeric, URL, punctuation, and whitespace patterns:

Filter	Description	Key Parameters	Default Values
`NonAlphaNumericFilter`	Limits non-alphanumeric content	`max_non_alpha_numeric_to_text_ratio`	`0.25`
`SymbolsToWordsFilter`	Limits symbols in text	`max_symbol_to_word_ratio`	`0.1`
`NumbersFilter`	Limits numeric content	`max_number_to_text_ratio`	`0.15`
`UrlsFilter`	Limits URL content	`max_url_to_text_ratio`	`0.2`
`PunctuationFilter`	Limits sentences without proper punctuation	`max_num_sentences_without_endmark_ratio`	`0.85`
`WhiteSpaceFilter`	Limits excessive whitespace	`max_white_space_ratio`	`0.25`

Content-Specific Filters

The following filters identify content-specific quality signals:

Filter	Description	Key Parameters	Default Values
`CommonEnglishWordsFilter`	Ensures text contains common words	`min_num_common_words`	`2`
`WordsWithoutAlphabetsFilter`	Limits words without alphabetic characters	`min_words_with_alphabets`	`0.8`
`BulletsFilter`	Limits content with many bullet points	`max_bullet_lines_ratio`	`0.9`
`BoilerPlateStringFilter`	Detects boilerplate text	`max_boilerplate_string_ratio`, `remove_if_at_top_or_bottom`	`0.4`, `True`
`ParenthesesFilter`	Limits content in parentheses	`max_parentheses_ratio`	`0.1`

Special Purpose Filters

The following filters address specialized quality conditions:

Filter	Description	Key Parameters	Default Values
`PornographicUrlsFilter`	Detects URLs containing the “porn” substring	`None`	Not applicable
`EllipsisFilter`	Limits excessive ellipses	`max_num_lines_ending_with_ellipsis_ratio`	`0.3`
`HistogramFilter`	Filters based on character distribution	`threshold`	`0.8`
`SubstringFilter`	Filters based on the presence of a specific substring in a position	`substring`, `position`	Required

Configuration

NeMo Curator pipelines can be configured using YAML files with Hydra. The configuration uses _target_ to specify class paths:

Hydra Configuration

# Hydra-based pipeline configuration
input_path: /path/to/input
output_path: /path/to/output
text_field: text
stages:
  - _target_: nemo_curator.stages.text.io.reader.JsonlReader
    file_paths: ${input_path}
    fields: null
  - _target_: nemo_curator.stages.text.filters.score_filter.ScoreFilter
    filter_obj:
      _target_: nemo_curator.stages.text.filters.heuristic.string.WordCountFilter
      min_words: 50
      max_words: 100000
    text_field: ${text_field}
    score_field: word_count
  - _target_: nemo_curator.stages.text.filters.score_filter.ScoreFilter
    filter_obj:
      _target_: nemo_curator.stages.text.filters.heuristic.string.PunctuationFilter
      max_num_sentences_without_endmark_ratio: 0.85
    text_field: ${text_field}
    score_field: null
  - _target_: nemo_curator.stages.text.io.writer.JsonlWriter
    path: ${output_path}

Refer to nemo_curator/config/text/ for complete pipeline examples.

For non-English text, you can adjust the filter parameters based on the characteristics of your target language.

Best Practices

When building filter chains, follow these best practices:

Order for Efficiency

Performance Tuning

Precision vs. Recall

Language Considerations

Multiple Filters

# Efficient ordering - place fast filters first
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.filters import ScoreFilter
from nemo_curator.stages.text.filters.heuristic import WordCountFilter, UrlsFilter
from nemo_curator.stages.text.filters.heuristic.repetition import RepeatingTopNGramsFilter
pipeline = Pipeline(name="efficient_filtering")
# Fast filters first
pipeline.add_stage(ScoreFilter(filter_obj=WordCountFilter(min_words=50), text_field="text"))
# Medium complexity filters
pipeline.add_stage(ScoreFilter(filter_obj=UrlsFilter(), text_field="text"))
# Slow filters last
pipeline.add_stage(ScoreFilter(filter_obj=RepeatingTopNGramsFilter(), text_field="text"))

Analyzing Filter Results

When tuning filter thresholds, analyze score distributions before applying filters. NeMo Curator provides two modules for this workflow:

Score: Computes scores and adds them as columns without removing documents.
ScoreFilter: Computes scores, filters based on thresholds, and optionally retains scores in output.

Use Score first to understand your data distribution, then apply ScoreFilter with tuned thresholds.

By default, Filter and ScoreFilter log when an input batch is already empty or when a stage removes every row, but they do not report partial retention. Set verbose=True to also log <stage> batch <task_id> retained <kept>/<input> rows for each non-empty batch. A stage containing multiple filters reports one count after the complete chain. These messages contain aggregate counts, not the identities or rejection reasons of individual records.

Score Without Filtering

Analyze Score Distribution

Apply Tuned Filters

Use Score to add score columns to your data without removing any documents:

from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.io.reader import JsonlReader
from nemo_curator.stages.text.io.writer import JsonlWriter
from nemo_curator.stages.text.filters import Score
from nemo_curator.stages.text.filters.heuristic import WordCountFilter
from nemo_curator.stages.text.filters.heuristic.repetition import RepeatingTopNGramsFilter
# Create scoring pipeline (no filtering)
pipeline = Pipeline(name="score_analysis")
# Load data
pipeline.add_stage(JsonlReader(file_paths="input_data/", fields=["text", "id"]))
# Add scores without filtering
pipeline.add_stage(Score(
    score_fn=WordCountFilter(min_words=80),
    text_field="text",
    score_field="word_count"
))
pipeline.add_stage(Score(
    score_fn=RepeatingTopNGramsFilter(n=3, max_repeating_ngram_ratio=0.18),
    text_field="text",
    score_field="ngram_ratio"
))
# Write scored data (all documents preserved)
pipeline.add_stage(JsonlWriter(path="scored_output/"))
pipeline.run()

Output files are written to the scored_output/ directory with one file per input partition.

Performance Tuning

For large datasets, consider these performance optimizations:

XennaExecutor (Default)

RayDataExecutor

Batch Size Optimization

XennaExecutor is the default executor, optimized for streaming workloads. You can customize its configuration or use the defaults:

from nemo_curator.backends.xenna import XennaExecutor
# Custom configuration for streaming processing
executor = XennaExecutor(config={
    "execution_mode": "streaming",
    "cpu_allocation_percentage": 0.95,
    "logging_interval": 60
})
results = pipeline.run(executor)

If no executor is specified, pipeline.run() uses XennaExecutor with default settings.

Pipeline Metrics

When you run a filtering pipeline, each stage tracks the number of documents it processes. You can use these metrics to understand how each filter affects your dataset and to tune thresholds.

After calling pipeline.run(), the returned task objects contain per-stage performance statistics through _stage_perf. Each entry is a StagePerfStats object with a num_items_processed field that records how many documents passed through that stage.

# Run the pipeline and inspect filter metrics
output_tasks = pipeline.run()
for task in output_tasks:
    # _stage_perf[0] is file partitioning, _stage_perf[1] is the reader
    num_input = task._stage_perf[1].num_items_processed
    # The last stage is the writer — its count reflects documents that survived all filters
    num_output = task._stage_perf[-1].num_items_processed
    if num_input > 0:
        print(f"Task {task.task_id}: {num_input} input → {num_output} kept ({num_output / num_input:.1%})")
    else:
        print(f"Task {task.task_id}: 0 input → 0 kept")

These same metrics power the nightly benchmarks, which track num_documents_processed, num_kept_documents, and throughput_docs_per_sec for every pipeline run.

Remember that the goal of filtering is to improve the quality of your training data, not necessarily to remove as many documents as possible. Monitor your filtering results and adjust thresholds based on your specific data characteristics and downstream tasks.