`stages.text.modifiers.line_remover`#

Module Contents#

Removes lines from a document if the content of the line matches a given string.

class stages.text.modifiers.line_remover.LineRemover(patterns: list[str])#

Bases: nemo_curator.stages.text.modifiers.doc_modifier.DocumentModifier

Removes lines from a document if the content of the line matches a given string.

Initialization

Args: patterns (List[str]): The patterns to check

modify_document(text: str) → str#: Transform the provided value(s) and return the result.