cudf.core.character_normalizer.CharacterNormalizer

class cudf.core.character_normalizer.CharacterNormalizer(

do_lower: bool,

special_tokens: Series | None = None,

)

A normalizer object used to normalize input text.

Parameters:

do_lowerbool

If True, the normalizer should also lower-case while normalizing.

special_tokenscudf.Series

Series of special tokens. These are expected to be all upper case and include the bracket [] characters.

Methods

normalize(text)