core.tokenizers.text.models.t5_tokenizer#

Module Contents#

Classes#

T5Tokenizer

Base class for Megatron T5 tokenizer.

API#

class core.tokenizers.text.models.t5_tokenizer.T5Tokenizer(path: str = None, config: dict = None, **kwargs)#

Bases: megatron.core.tokenizers.text.text_tokenizer.MegatronTokenizerText

Base class for Megatron T5 tokenizer.

Initialization

Parameters:

  • path (str) – path to the tokenizer model.

  • config (dict) – tokenizer parameters. library (str): tokenizer library. class_name (str): name of tokenizer class. class_path (str): path to tokenizer class. model_type (str): type of the model to be used with tokenizer. chat_template (str): tokenizer chat template.