> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.stages.text.filters.heuristic.repetition.repetition

## Module Contents

### Classes

| Name                                                                                                                                 | Description                                                            |
| ------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------- |
| [`RepeatedLinesByCharFilter`](#nemo_curator-stages-text-filters-heuristic-repetition-repetition-RepeatedLinesByCharFilter)           | If the document shrinks by > 20% in terms of number of lines           |
| [`RepeatedLinesFilter`](#nemo_curator-stages-text-filters-heuristic-repetition-repetition-RepeatedLinesFilter)                       | If the document shrinks by > 30% in terms of number of lines after     |
| [`RepeatedParagraphsByCharFilter`](#nemo_curator-stages-text-filters-heuristic-repetition-repetition-RepeatedParagraphsByCharFilter) | If the document shrinks by > 10% in terms of number of lines after     |
| [`RepeatedParagraphsFilter`](#nemo_curator-stages-text-filters-heuristic-repetition-repetition-RepeatedParagraphsFilter)             | If the document shrinks by > 30% in terms of number of lines after     |
| [`RepeatingDuplicateNGramsFilter`](#nemo_curator-stages-text-filters-heuristic-repetition-repetition-RepeatingDuplicateNGramsFilter) | If the document shrinks by > x% in terms of number of characters       |
| [`RepeatingTopNGramsFilter`](#nemo_curator-stages-text-filters-heuristic-repetition-repetition-RepeatingTopNGramsFilter)             | If the document shrinks by > x% in terms of number of characters after |

### API

```python
class nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatedLinesByCharFilter(
    max_repeated_lines_char_ratio: float = 0.8
)
```

**Bases:** [DocumentFilter](/nemo-curator/nemo_curator/stages/text/filters/doc_filter#nemo_curator-stages-text-filters-doc_filter-DocumentFilter)

If the document shrinks by > 20% in terms of number of lines
after removing duplicate lines, then discard.
Source: Gopher (Rae et al., 2021)

**`_name`** `= 'repeated_lines_char'`

---

```python
nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatedLinesByCharFilter.keep_document(
    score: float
) -> bool
```

```python
nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatedLinesByCharFilter.score_document(
    text: str
) -> float
```

```python
class nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatedLinesFilter(
    max_repeated_line_fraction: float = 0.7
)
```

**Bases:** [DocumentFilter](/nemo-curator/nemo_curator/stages/text/filters/doc_filter#nemo_curator-stages-text-filters-doc_filter-DocumentFilter)

If the document shrinks by > 30% in terms of number of lines after
removing duplicate lines, then discard.
Source: Gopher (Rae et al., 2021)

**`_name`** `= 'repeated_lines'`

---

```python
nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatedLinesFilter.keep_document(
    score: float
) -> bool
```

```python
nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatedLinesFilter.score_document(
    text: str
) -> float
```

```python
class nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatedParagraphsByCharFilter(
    max_repeated_paragraphs_char_ratio: float = 0.8
)
```

**Bases:** [DocumentFilter](/nemo-curator/nemo_curator/stages/text/filters/doc_filter#nemo_curator-stages-text-filters-doc_filter-DocumentFilter)

If the document shrinks by > 10% in terms of number of lines after
removing duplicate paragraphs, then discard.
Source: Gopher (Rae et al., 2021)

**`_name`** `= 'repeated_paragraphs_char'`

---

```python
nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatedParagraphsByCharFilter.keep_document(
    score: float
) -> bool
```

```python
nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatedParagraphsByCharFilter.score_document(
    text: str
) -> float
```

```python
class nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatedParagraphsFilter(
    max_repeated_paragraphs_ratio: float = 0.7
)
```

**Bases:** [DocumentFilter](/nemo-curator/nemo_curator/stages/text/filters/doc_filter#nemo_curator-stages-text-filters-doc_filter-DocumentFilter)

If the document shrinks by > 30% in terms of number of lines after
removing duplicate paragraphs, then discard.
Source: Gopher (Rae et al., 2021)

**`_name`** `= 'repeated_paragraphs'`

---

```python
nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatedParagraphsFilter.keep_document(
    score: float
) -> bool
```

```python
nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatedParagraphsFilter.score_document(
    text: str
) -> float
```

```python
class nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatingDuplicateNGramsFilter(
    n: int = 2,
    max_repeating_duplicate_ngram_ratio: float = 0.2,
    lang: str = 'en'
)
```

**Bases:** [DocumentFilter](/nemo-curator/nemo_curator/stages/text/filters/doc_filter#nemo_curator-stages-text-filters-doc_filter-DocumentFilter)

If the document shrinks by > x% in terms of number of characters
after removing all duplicate n-grams, then discard.
Source: Gopher (Rae et al., 2021)

For Chinese and Japanese text, we use external libraries to split the text
because these languages are not separated by spaces. For all other languages,
such as English, we assume words are separated by spaces.

**`_max_ratio`** `= 1.0`

---

**`_name`** `= f'repeating_dup_{n}gram'`

---

**`_word_splitter`** `= get_word_splitter(lang)`

---

```python
nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatingDuplicateNGramsFilter.keep_document(
    score: float
) -> bool
```

```python
nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatingDuplicateNGramsFilter.score_document(
    text: str
) -> float
```

```python
class nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatingTopNGramsFilter(
    n: int = 2,
    max_repeating_ngram_ratio: float = 0.2,
    lang: str = 'en'
)
```

**Bases:** [DocumentFilter](/nemo-curator/nemo_curator/stages/text/filters/doc_filter#nemo_curator-stages-text-filters-doc_filter-DocumentFilter)

If the document shrinks by > x% in terms of number of characters after
removing the top n-grams, then discard.
Source: Gopher (Rae et al., 2021)

For Chinese and Japanese text, we use external libraries to split the text
because these languages are not separated by spaces. For all other languages,
such as English, we assume words are separated by spaces.

**`_max_ratio`** `= 1.0`

---

**`_name`** `= f'repeating_top_{n}grams'`

---

**`_word_splitter`** `= get_word_splitter(lang)`

---

```python
nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatingTopNGramsFilter.keep_document(
    score: float
) -> bool
```

```python
nemo_curator.stages.text.filters.heuristic.repetition.repetition.RepeatingTopNGramsFilter.score_document(
    text: str
) -> float
```