Is this page helpful?

normalize#

class pylibcudf.nvtext.normalize.CharacterNormalizer#

The normalizer object to be used with normalize_characters.

For details, see cudf::nvtext::character_normalizer.

pylibcudf.nvtext.normalize.NormalizeFlags#

See also normalize_flags.

Enum members

  • NONE

  • STRIP_ACCENTS

  • PAD_PUNCTUATION

pylibcudf.nvtext.normalize.normalize_characters(
Column input,
CharacterNormalizer normalizer,
underlying_type_t_normalize_flags flags=...,
stream: CudaStreamLike | None = None,
DeviceMemoryResource mr=None,
) → Column#

Normalizes strings characters for tokenizing.

For details, see normalize_characters()

Parameters:
inputColumn

Input strings

normalizerCharacterNormalizer

Normalizer object used for modifying the input column text

flagsNormalizeFlags, optional

Bitmask of NormalizeFlags values controlling normalization behavior. Default is NormalizeFlags.PAD_PUNCTUATION.

streamStream | None

CUDA stream on which to perform the operation.

Returns:
Column

Normalized strings column

pylibcudf.nvtext.normalize.normalize_spaces(
Column input,
stream: CudaStreamLike | None = None,
DeviceMemoryResource mr=None,
) → Column#

Returns a new strings column by normalizing the whitespace in each string in the input column.

For details, see normalize_spaces()

Parameters:
inputColumn

Input strings

streamStream | None

CUDA stream on which to perform the operation.

Returns:
Column

New strings columns of normalized strings.