Heuristic Filtering | NeMo Curator

Heuristic filtering uses simple, rule-based metrics to identify and filter out low-quality documents from your dataset. NVIDIA NeMo Curator provides a variety of pre-built heuristic filters that can be configured and combined to meet your specific needs.

How It Works

Heuristic filters examine specific attributes of text documents and apply predefined thresholds to determine document quality. Unlike classifier-based filtering, heuristic filters don’t require training data but rely on configurable thresholds and rules.

These filters assess quality using measurable document characteristics such as:

Document length (word or character count)
Punctuation ratios and patterns
Repetitive content detection
Language-specific patterns
Text completeness and coherence

For details on filter structure and the filtering process, refer to Data Processing Concepts.

Usage

Python

Configuration

1 from nemo_curator.pipeline import Pipeline
2 from nemo_curator.stages.text.io.reader import JsonlReader
3 from nemo_curator.stages.text.io.writer import JsonlWriter
4 from nemo_curator.stages.text.modules import ScoreFilter
5 from nemo_curator.stages.text.filters import (
6     WordCountFilter,
7     RepeatingTopNGramsFilter,
8     PunctuationFilter
9 )
10 
11 # Create pipeline
12 pipeline = Pipeline(name="heuristic_filtering")
13 
14 # Load your dataset
15 reader = JsonlReader(
16     file_paths="input_data/",
17     fields=["text", "id"]
18 )
19 pipeline.add_stage(reader)
20 
21 # Add filter stages
22 pipeline.add_stage(ScoreFilter(
23     filter_obj=WordCountFilter(min_words=80),
24     text_field="text",
25     score_field="word_count"
26 ))
27 pipeline.add_stage(ScoreFilter(
28     filter_obj=PunctuationFilter(max_num_sentences_without_endmark_ratio=0.85),
29     text_field="text"
30 ))
31 pipeline.add_stage(ScoreFilter(
32     filter_obj=RepeatingTopNGramsFilter(n=2, max_repeating_ngram_ratio=0.2),
33     text_field="text"
34 ))
35 pipeline.add_stage(ScoreFilter(
36     filter_obj=RepeatingTopNGramsFilter(n=3, max_repeating_ngram_ratio=0.18),
37     text_field="text"
38 ))
39 pipeline.add_stage(ScoreFilter(
40     filter_obj=RepeatingTopNGramsFilter(n=4, max_repeating_ngram_ratio=0.16),
41     text_field="text"
42 ))
43 
44 # Add output stage
45 writer = JsonlWriter(path="high_quality_output/")
46 pipeline.add_stage(writer)
47 
48 # Execute pipeline
49 results = pipeline.run()

Available Filters

NeMo Curator includes more than 30 heuristic filters for assessing document quality. Below are the most commonly used filters with their parameters:

Text Length Filters

Filter	Description	Key Parameters	Default Values
WordCountFilter	Filters by word count	`min_words`, `max_words`	min=50, max=100000
TokenCountFilter	Filters by token count	`min_tokens`, `max_tokens`	min=0, max=∞
MeanWordLengthFilter	Filters by average word length	`min_mean_word_length`, `max_mean_word_length`	min=3, max=10
LongWordFilter	Filters by presence of extremely long words	`max_word_length`	1000

Repetition Detection Filters

Filter	Description	Key Parameters	Default Values
RepeatedLinesFilter	Detects repeated lines	`max_repeated_line_fraction`	0.7
RepeatedParagraphsFilter	Detects repeated paragraphs	`max_repeated_paragraphs_ratio`	0.7
RepeatingTopNGramsFilter	Detects excessive repetition of n-grams	`n`, `max_repeating_ngram_ratio`	n=2, ratio=0.2
RepeatingDuplicateNGramsFilter	Detects duplicate n-grams	`n`, `max_repeating_duplicate_ngram_ratio`	n=2, ratio=0.2

Character and Symbol Filters

Filter	Description	Key Parameters	Default Values
NonAlphaNumericFilter	Limits non-alphanumeric content	`max_non_alpha_numeric_to_text_ratio`	0.25
SymbolsToWordsFilter	Limits symbols in text	`max_symbol_to_word_ratio`	0.1
NumbersFilter	Limits numeric content	`max_number_to_text_ratio`	0.15
UrlsFilter	Limits URL content	`max_url_to_text_ratio`	0.2
PunctuationFilter	Limits sentences without proper punctuation	`max_num_sentences_without_endmark_ratio`	0.85
WhiteSpaceFilter	Limits excessive whitespace	`max_white_space_ratio`	0.25

Content-specific Filters

Filter	Description	Key Parameters	Default Values
CommonEnglishWordsFilter	Ensures text contains common words	`min_num_common_words`	2
WordsWithoutAlphabetsFilter	Limits words without alphabetic chars	`min_words_with_alphabets`	0.8
BulletsFilter	Limits bullet-point heavy content	`max_bullet_lines_ratio`	0.9
BoilerPlateStringFilter	Detects boilerplate text	`max_boilerplate_string_ratio`, `remove_if_at_top_or_bottom`	0.4, True
ParenthesesFilter	Limits parentheses content	`max_parentheses_ratio`	0.1

Special Purpose Filters

Filter	Description	Key Parameters	Default Values
PornographicUrlsFilter	Detects URLs containing “porn” substring	None	N/A
EllipsisFilter	Limits excessive ellipses	`max_num_lines_ending_with_ellipsis_ratio`	0.3
HistogramFilter	Filters based on character distribution	`threshold`	0.8
SubstringFilter	Filters based on presence of specific substring in a position	`substring`, `position`	"", “any”

Configuration

Example Configuration

1 # Sample filter configuration (simplified)
2 filters:
3   - name: ScoreFilter
4     filter:
5       name: WordCountFilter
6       min_words: 50
7       max_words: 100000
8     text_field: text
9     score_field: word_count
10 
11   - name: ScoreFilter
12     filter:
13       name: PunctuationFilter
14       max_num_sentences_without_endmark_ratio: 0.85
15     text_field: text
16     score_field: punctuation_ratio
17 
18   - name: ScoreFilter
19     filter:
20       name: RepeatingTopNGramsFilter
21       n: 2
22       max_repeating_ngram_ratio: 0.18
23     text_field: text
24     score_field: ngram_repetition

For non-English texts, you may need to adjust the filter parameters based on the specific characteristics of your target language.

Best Practices

When building filter chains, follow these best practices:

Order for Efficiency

Performance Tuning

Precision vs. Recall

Language Considerations

Multiple Filters

1 # Efficient ordering - place fast filters first
2 from nemo_curator.pipeline import Pipeline
3 from nemo_curator.stages.text.modules import ScoreFilter
4 from nemo_curator.stages.text.filters import WordCountFilter, UrlsFilter, RepeatingTopNGramsFilter
5 
6 pipeline = Pipeline(name="efficient_filtering")
7 # Fast filters first
8 pipeline.add_stage(ScoreFilter(filter_obj=WordCountFilter(min_words=50), text_field="text"))
9 # Medium complexity filters
10 pipeline.add_stage(ScoreFilter(filter_obj=UrlsFilter(), text_field="text"))
11 # Slow filters last
12 pipeline.add_stage(ScoreFilter(filter_obj=RepeatingTopNGramsFilter(), text_field="text"))

Analyzing Filter Results

When working with non-English data or tuning your filtering pipeline, it’s valuable to examine which filters are removing documents:

Filter Analysis

1 import pandas as pd
2 
3 # Load scores from filter run
4 scores = pd.read_json("output/scores/scores.jsonl", lines=True)
5 
6 # Analyze rejection reasons
7 rejection_counts = scores[scores["rejected"] == True].groupby("rejected_by").size()
8 print(f"Documents rejected by filter:\n{rejection_counts}")
9 
10 # Analyze score distributions
11 import matplotlib.pyplot as plt
12 scores.hist(column="word_count", bins=50)
13 plt.title("Word Count Distribution")
14 plt.savefig("word_count_hist.png")

Performance Tuning

For large datasets, consider these performance optimizations:

Memory Efficient Processing

Distributed Processing

Batch Size Optimization

1 # Process large datasets efficiently using pipeline streaming
2 
3 # Configure for streaming processing
4 executor_config = {
5     "execution_mode": "streaming",
6     "cpu_allocation_percentage": 0.8,
7     "logging_interval": 60
8 }
9 
10 # Use custom configuration for large datasets
11 executor = XennaExecutor(config=executor_config)
12 results = pipeline.run(executor)
13 
14 # Default configuration works for most cases
15 # results = pipeline.run()

Remember that the goal of filtering is to improve the quality of your training data, not necessarily to remove as many documents as possible. Monitor your filtering results and adjust thresholds based on your specific data characteristics and downstream tasks.