core.tokenizers.vision.models.default_tokenizer#

Module Contents#

Classes#

DefaultTokenizerVision

Base class for Megatron default vision tokenizer.

API#

class core.tokenizers.vision.models.default_tokenizer.DefaultTokenizerVision(
path: str = None,
config: dict = None,
**kwargs,
)#

Bases: megatron.core.tokenizers.vision.vision_tokenizer.MegatronTokenizerVision

Base class for Megatron default vision tokenizer.

Initialization

Parameters:

  • path (str) – path to the tokenizer model.

  • config (dict) – tokenizer parameters. library (str): tokenizer library. class_name (str): name of tokenizer class. class_path (str): path to tokenizer class. model_type (str): type of the model to be used with tokenizer.