bridge.training.tokenizers.utils

Module Contents

Functions

build_new_tokenizer

Initialize tokenizer from megatron.core.tokenizers based on the provided configuration.

Data

MEGATRON_TOKENIZERS
SP_TOKENIZERS

API

bridge.training.tokenizers.utils.MEGATRON_TOKENIZERS

[‘BertWordPieceLowerCase’, ‘BertWordPieceCase’, ‘GPT2BPETokenizer’]

bridge.training.tokenizers.utils.SP_TOKENIZERS

[‘SentencePieceTokenizer’, ‘GPTSentencePieceTokenizer’, ‘Llama2Tokenizer’]

bridge.training.tokenizers.utils.build_new_tokenizer(

config: megatron.bridge.training.tokenizers.config.TokenizerConfig,

) → megatron.core.tokenizers.MegatronTokenizer

Initialize tokenizer from megatron.core.tokenizers based on the provided configuration.

Parameters:

config (TokenizerConfig) – Configuration object specifying the tokenizer type, paths to vocab/model files, and other tokenizer-specific settings.

Returns:

An instance of the initialized tokenizer.

Return type:

MegatronTokenizer