core.tokenizers.utils.utils

Module Contents

Functions

has_gigatoken_support	Check if gigatoken library is installed.
init_gigatoken_from_hf	Initialize gigatoken tokenizer from Hugging Face.

Data

logger

API

core.tokenizers.utils.utils.logger

‘getLogger(…)’

core.tokenizers.utils.utils.has_gigatoken_support() → bool

Check if gigatoken library is installed.

core.tokenizers.utils.utils.init_gigatoken_from_hf(

tokenizer: AutoTokenizer,

tokenizer_path: str,

) → HFCompat

Initialize gigatoken tokenizer from Hugging Face.