core.tokenizers.text.models.mamba_tokenizer#

Module Contents#

Classes#

MambaTokenizer

Base class for Megatron Mamba tokenizer.

API#

class core.tokenizers.text.models.mamba_tokenizer.MambaTokenizer(path: str = None, config: dict = None, **kwargs)#

Bases: megatron.core.tokenizers.text.text_tokenizer.MegatronTokenizerText

Base class for Megatron Mamba tokenizer.

Initialization

Parameters:

  • path (str) – path to the tokenizer model.

  • config (dict) – tokenizer parameters. library (str): tokenizer library. class_name (str): name of tokenizer class. class_path (str): path to tokenizer class. model_type (str): type of the model to be used with tokenizer. chat_template (str): tokenizer chat template.