Is this page helpful?

`core.tokenizers.utils.build_tokenizer`#

Module Contents#

Functions#

`build_tokenizer`	Initialize tokenizer.
`vocab_size_with_padding`	Pad vocab size so it is divisible by model parallel size and still having GPU friendly size.

Data#

`MEGATRON_TOKENIZERS`
`SP_TOKENIZERS`
`SUPPORTED_TOKENIZERS`
`logger`

API#

core.tokenizers.utils.build_tokenizer.MEGATRON_TOKENIZERS#: [‘BertWordPieceLowerCase’, ‘BertWordPieceCase’, ‘GPT2BPETokenizer’]

core.tokenizers.utils.build_tokenizer.SP_TOKENIZERS#: [‘SentencePieceTokenizer’, ‘GPTSentencePieceTokenizer’, ‘Llama2Tokenizer’]

core.tokenizers.utils.build_tokenizer.SUPPORTED_TOKENIZERS#: None

core.tokenizers.utils.build_tokenizer.logger#: ‘getLogger(…)’

core.tokenizers.utils.build_tokenizer.build_tokenizer(args, **kwargs)#: Initialize tokenizer.

core.tokenizers.utils.build_tokenizer.vocab_size_with_padding( orig_vocab_size: int, make_vocab_size_divisible_by: int, tensor_model_parallel_size: int, logging_enabled=True, )#: Pad vocab size so it is divisible by model parallel size and still having GPU friendly size.