> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.stages.text.filters.score_filter

## Module Contents

### Classes

| Name                                                                        | Description                                                                                       |
| --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| [`Filter`](#nemo_curator-stages-text-filters-score_filter-Filter)           | The module responsible for filtering records based on a metadata field.                           |
| [`Score`](#nemo_curator-stages-text-filters-score_filter-Score)             | The module responsible for adding metadata to records based on statistics about the text.         |
| [`ScoreFilter`](#nemo_curator-stages-text-filters-score_filter-ScoreFilter) | The module responsible for applying a filter (or chain of filters) to all documents in a dataset. |

### Functions

| Name                                                                                                                | Description                                                                                  |
| ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| [`_filter_name`](#nemo_curator-stages-text-filters-score_filter-_filter_name)                                       | -                                                                                            |
| [`_format_field_list`](#nemo_curator-stages-text-filters-score_filter-_format_field_list)                           | In the case of a list of DocumentFilters or Callables, format the relevant field             |
| [`_format_single_field_list`](#nemo_curator-stages-text-filters-score_filter-_format_single_field_list)             | In the case of a single DocumentFilter or Callable, format the relevant field                |
| [`_get_filter_stage_name`](#nemo_curator-stages-text-filters-score_filter-_get_filter_stage_name)                   | Derive the stage name from the provided score/filter functions.                              |
| [`_validate_and_normalize_filters`](#nemo_curator-stages-text-filters-score_filter-_validate_and_normalize_filters) | Validate and normalize all parameters needed for the Score, Filter, and ScoreFilter modules. |

### API

```python
class nemo_curator.stages.text.filters.score_filter.Filter(
    filter_fn: collections.abc.Callable | nemo_curator.stages.text.filters.doc_filter.DocumentFilter | list[nemo_curator.stages.text.filters.doc_filter.DocumentFilter],
    filter_field: str | list[str],
    invert: bool | list[bool] = False,
    name: str = 'filter_fn'
)
```

Dataclass

**Bases:** [ProcessingStage\[DocumentBatch, DocumentBatch\]](/nemo-curator/nemo_curator/stages/base#nemo_curator-stages-base-ProcessingStage)

The module responsible for filtering records based on a metadata field.
It accepts an arbitrary filter function that accepts a metadata field and returns True if the field should be kept.
It also accepts a DocumentFilter object, in which case the filter\_fn will be the keep\_document method of the DocumentFilter.
Unlike ScoreFilter, it does not compute the metadata based on a document.
It only filters using existing metadata.

If a list of DocumentFilters is provided, the filters are applied in order.
In this case, the filter\_field parameter should be a list of strings corresponding to the filters.
If some filters should be inverted and others not, then invert should be a list of booleans corresponding to the filters.

**Parameters:**

**`filter_fn`** `Callable | DocumentFilter | list[DocumentFilter]`

A function (or list of functions) that returns True if the document is to be kept or a DocumentFilter object,
in which case the filter\_fn will be the keep\_document method of the DocumentFilter.

---

**`filter_field`** `str | list[str]`

The field (or list of fields) to be passed into the filter function.

---

**`invert`** `bool | list[bool]` — default: False

Whether to invert the filter condition.

---

**`filter_field`** `str | list[str]`

---

**`filter_fn`** `Callable | DocumentFilter | list[DocumentFilter]`

---

**`invert`** `bool | list[bool] = False`

---

**`name`** `str = 'filter_fn'`

---

```python
nemo_curator.stages.text.filters.score_filter.Filter.__post_init__()
```

```python
nemo_curator.stages.text.filters.score_filter.Filter.compute_filter_mask(
    df: pandas.DataFrame,
    filter_fn: collections.abc.Callable | nemo_curator.stages.text.filters.doc_filter.DocumentFilter,
    filter_field: str,
    invert: bool
) -> pandas.Series
```

Compute the bool mask to filter the dataset.

**Parameters:**

**`df`** `pd.DataFrame`

The dataset to compute filter mask on.

---

**`filter_fn`** `Callable | DocumentFilter`

The filter function to use.

---

**`filter_field`** `str`

The field to read the filter from.

---

**`invert`** `bool`

Whether to invert the filter condition.

---

**Returns:** `pd.Series`

A mask corresponding to each data instance indicating whether it will be retained.

```python
nemo_curator.stages.text.filters.score_filter.Filter.inputs() -> tuple[list[str], list[str]]
```

```python
nemo_curator.stages.text.filters.score_filter.Filter.outputs() -> tuple[list[str], list[str]]
```

```python
nemo_curator.stages.text.filters.score_filter.Filter.process(
    batch: nemo_curator.tasks.DocumentBatch
) -> nemo_curator.tasks.DocumentBatch | None
```

Applies the filtering to a dataset

**Parameters:**

**`batch`** `DocumentBatch`

The batch to apply the module to

---

**Returns:** `DocumentBatch | None`

A batch with entries removed according to the filter

```python
class nemo_curator.stages.text.filters.score_filter.Score(
    score_fn: collections.abc.Callable[[str], float | str] | nemo_curator.stages.text.filters.doc_filter.DocumentFilter | list[nemo_curator.stages.text.filters.doc_filter.DocumentFilter],
    score_field: str | list[str],
    text_field: str | list[str] = 'text',
    name: str = 'score_fn'
)
```

Dataclass

**Bases:** [ProcessingStage\[DocumentBatch, DocumentBatch\]](/nemo-curator/nemo_curator/stages/base#nemo_curator-stages-base-ProcessingStage)

The module responsible for adding metadata to records based on statistics about the text.
It accepts an arbitrary scoring function that accepts a text field and returns a score.
It also accepts a DocumentFilter object, in which case the score\_fn will be the score\_document method of the DocumentFilter.

Unlike ScoreFilter, it does not filter based on the computed score.
It only adds metadata to the record.

If a list of DocumentFilters is provided, the filters are applied in order.
In this case, the score\_field parameter should be a list of strings corresponding to the filters.
If different filters should be applied to different text fields, then text\_field should be a list of strings corresponding to the filters.

**Parameters:**

**`score_fn`** `Callable | DocumentFilter | list[DocumentFilter]`

The score function or the DocumentFilter object (or list of DocumentFilters). If it is a DocumentFilter object, the score\_fn will be the score\_document method of the DocumentFilter.

---

**`score_field`** `str | list[str]`

The field (or list of fields) the score will be stored in.

---

**`text_field`** `str | list[str]` — default: 'text'

The field (or list of fields) the documents will be read from.

---

**`name`** `str = 'score_fn'`

---

**`score_field`** `str | list[str]`

---

**`score_fn`** `Callable[[str], float | str] | DocumentFilter | list[DocumentFilter]`

---

**`text_field`** `str | list[str] = 'text'`

---

```python
nemo_curator.stages.text.filters.score_filter.Score.__post_init__()
```

```python
nemo_curator.stages.text.filters.score_filter.Score.inputs() -> tuple[list[str], list[str]]
```

```python
nemo_curator.stages.text.filters.score_filter.Score.outputs() -> tuple[list[str], list[str]]
```

```python
nemo_curator.stages.text.filters.score_filter.Score.process(
    batch: nemo_curator.tasks.DocumentBatch
) -> nemo_curator.tasks.DocumentBatch | None
```

Applies the scoring to a dataset

**Parameters:**

**`batch`** `DocumentBatch`

The batch to apply the module to

---

**Returns:** `DocumentBatch | None`

A batch with the new score

```python
nemo_curator.stages.text.filters.score_filter.Score.ray_stage_spec() -> dict[str, typing.Any]
```

```python
nemo_curator.stages.text.filters.score_filter.Score.setup(
    _: nemo_curator.backends.base.WorkerMetadata | None = None
) -> None
```

```python
nemo_curator.stages.text.filters.score_filter.Score.setup_on_node(
    _node_info: nemo_curator.backends.base.NodeInfo | None = None,
    _worker_metadata: nemo_curator.backends.base.WorkerMetadata | None = None
) -> None
```

```python
class nemo_curator.stages.text.filters.score_filter.ScoreFilter(
    filter_obj: nemo_curator.stages.text.filters.doc_filter.DocumentFilter | list[nemo_curator.stages.text.filters.doc_filter.DocumentFilter],
    text_field: str | list[str] = 'text',
    score_field: str | list[str] | None = None,
    invert: bool | list[bool] = False,
    name: str = 'score_filter'
)
```

Dataclass

**Bases:** [ProcessingStage\[DocumentBatch, DocumentBatch\]](/nemo-curator/nemo_curator/stages/base#nemo_curator-stages-base-ProcessingStage)

The module responsible for applying a filter (or chain of filters) to all documents in a dataset.
It accepts an arbitrary DocumentFilter and first computes the score for a document.
Then, determines whether to keep the document based on the criteria in the DocumentFilter.

The filter can be applied to any field in the dataset, and the score can be logged for later.
Also, the filter can be inverted such that "rejected" documents are kept.

If a list of DocumentFilters is provided, the filters are applied in order.
If different filters should be applied to different text fields, then text\_field should be a list of strings corresponding to the filters.
If different score fields should be created for each filter, then score\_field should be a list of strings corresponding to the filters.
If some filters should be inverted and others not, then invert should be a list of booleans corresponding to the filters.

**Parameters:**

**`filter_obj`** `DocumentFilter | list[DocumentFilter]`

The score function (or list of score functions) that takes in a document string and outputs a score for the document.

---

**`text_field`** `str | list[str]` — default: 'text'

The field (or list of fields) the documents will be read from.

---

**`score_field`** `str | list[str] | None` — default: None

The field (or list of fields) to which the scores will be written. If None, scores will be immediately discarded after use.

---

**`invert`** `bool | list[bool]` — default: False

If True, will keep all documents that are normally discarded.

---

**`filter_obj`** `DocumentFilter | list[DocumentFilter]`

---

**`invert`** `bool | list[bool] = False`

---

**`name`** `str = 'score_filter'`

---

**`score_field`** `str | list[str] | None = None`

---

**`text_field`** `str | list[str] = 'text'`

---

```python
nemo_curator.stages.text.filters.score_filter.ScoreFilter.__post_init__()
```

```python
nemo_curator.stages.text.filters.score_filter.ScoreFilter.compute_filter_mask(
    df: pandas.DataFrame,
    filter_obj: nemo_curator.stages.text.filters.doc_filter.DocumentFilter,
    text_field: str,
    score_field: str | None,
    invert: bool
) -> pandas.Series
```

Compute the bool mask to filter the dataset.

**Parameters:**

**`df`** `pd.DataFrame`

The dataset to compute filter mask on.

---

**`filter_obj`** `DocumentFilter`

The filter object to use.

---

**`text_field`** `str`

The field to read the text from.

---

**`score_field`** `str | None`

The field to write the scores to.

---

**`invert`** `bool`

Whether to invert the filter condition.

---

**Returns:** `pd.Series`

A mask corresponding to each data instance indicating whether it will be retained.

```python
nemo_curator.stages.text.filters.score_filter.ScoreFilter.inputs() -> tuple[list[str], list[str]]
```

```python
nemo_curator.stages.text.filters.score_filter.ScoreFilter.outputs() -> tuple[list[str], list[str]]
```

```python
nemo_curator.stages.text.filters.score_filter.ScoreFilter.process(
    batch: nemo_curator.tasks.DocumentBatch
) -> nemo_curator.tasks.DocumentBatch | None
```

Scores and filters all records in the dataset

**Parameters:**

**`batch`** `DocumentBatch`

The batch to apply the module to

---

**Returns:** `DocumentBatch | None`

A batch with the score and filter applied

```python
nemo_curator.stages.text.filters.score_filter.ScoreFilter.ray_stage_spec() -> dict[str, typing.Any]
```

```python
nemo_curator.stages.text.filters.score_filter.ScoreFilter.setup(
    _: nemo_curator.backends.base.WorkerMetadata | None = None
) -> None
```

```python
nemo_curator.stages.text.filters.score_filter.ScoreFilter.setup_on_node(
    _node_info: nemo_curator.backends.base.NodeInfo | None = None,
    _worker_metadata: nemo_curator.backends.base.WorkerMetadata | None = None
) -> None
```

```python
nemo_curator.stages.text.filters.score_filter._filter_name(
    x: nemo_curator.stages.text.filters.doc_filter.DocumentFilter | collections.abc.Callable
) -> str
```

```python
nemo_curator.stages.text.filters.score_filter._format_field_list(
    _field: str | list[str] | None,
    filter_count: int,
    field_name: str,
    field_type: type = str
) -> list[str] | list[bool]
```

In the case of a list of DocumentFilters or Callables, format the relevant field
(filter\_field, score\_field, text\_field, invert) to a list of length equal to the number of filters.

**Parameters:**

**`_field`** `str | list[str] | None`

The field to check and format.

---

**`filter_count`** `int`

The number of filters. This will be the length of the output list.

---

**`field_name`** `str`

The name of the field, which is used in error messages.

---

**`field_type`** `type` — default: str

The type of the field, which is used in an isinstance check.

---

**Returns:** `list[str] | list[bool]`

list\[str] | list\[bool]: The reformatted field.

```python
nemo_curator.stages.text.filters.score_filter._format_single_field_list(
    _field: str | list[str] | None,
    field_name: str,
    field_type: type = str
) -> list[str] | list[bool]
```

In the case of a single DocumentFilter or Callable, format the relevant field
(filter\_field, score\_field, text\_field, invert) to a list of length 1.

**Parameters:**

**`_field`** `str | list[str] | None`

The field to check and format.

---

**`field_name`** `str`

The name of the field, which is used in error messages.

---

**`field_type`** `type` — default: str

The type of the field, which is used in an isinstance check.

---

**Returns:** `list[str] | list[bool]`

list\[str] | list\[bool]: The reformatted field.

```python
nemo_curator.stages.text.filters.score_filter._get_filter_stage_name(
    filters: list[nemo_curator.stages.text.filters.doc_filter.DocumentFilter | collections.abc.Callable],
    prefix: str
) -> str
```

Derive the stage name from the provided score/filter functions.

```python
nemo_curator.stages.text.filters.score_filter._validate_and_normalize_filters(
    _filter: nemo_curator.stages.text.filters.doc_filter.DocumentFilter | collections.abc.Callable | list[nemo_curator.stages.text.filters.doc_filter.DocumentFilter | collections.abc.Callable],
    input_field: str | list[str] | None,
    invert: bool | list[bool] | None,
    output_field: str | list[str] | None,
    fn_type: typing.Literal['score', 'filter', 'score_filter']
) -> tuple[str, list[nemo_curator.stages.text.filters.doc_filter.DocumentFilter | collections.abc.Callable], list[str] | None, list[bool] | None, list[str] | None]
```

Validate and normalize all parameters needed for the Score, Filter, and ScoreFilter modules.
"Normalize" means to reformat all parameters to a list of length equal to the number of filters.

**Parameters:**

**`_filter`** `DocumentFilter | Callable | list[DocumentFilter | Callable]`

The filter object or list of filter objects.

---

**`input_field`** `str | list[str] | None`

The input field. For Score and ScoreFilter, this is the text field. For Filter, this is the filter field.

---

**`invert`** `bool | list[bool] | None`

The invert flag. This is used for Filter and ScoreFilter.

---

**`output_field`** `str | list[str] | None`

The output field. For Score and ScoreFilter, this is the score field. For Filter, this is not used.

---

**`fn_type`** `Literal['score', 'filter', 'score_filter']`

The type of the module.

---

**Returns:** `tuple[str, list[DocumentFilter | Callable], list[str] | None, list[bool] | None, list[str] | None]`

tuple\[str, list\[DocumentFilter | Callable], list\[str] | None, list\[bool] | None, list\[str] | None]:
The first string returned corresponds to the name given to the DocumentFilter or Callable.
The normalized filters, input fields, invert flags, and output fields make up the rest of the tuple.