Is this page helpful?

`core.tokenizers.utils.build_tokenizer`#

Module Contents#

Functions#

`build_tokenizer`	Initialize tokenizer.
`vocab_size_with_padding`	Pad vocab size so it is divisible by model parallel size and still having GPU friendly size.
`_set_padded_vocab_size`	Sets padded vocab size if None.

Data#

`MEGATRON_TOKENIZERS`
`SP_TOKENIZERS`
`SUPPORTED_TOKENIZERS`
`logger`

API#

core.tokenizers.utils.build_tokenizer.MEGATRON_TOKENIZERS#: [‘BertWordPieceLowerCase’, ‘BertWordPieceCase’, ‘GPT2BPETokenizer’]

core.tokenizers.utils.build_tokenizer.SP_TOKENIZERS#: [‘SentencePieceTokenizer’, ‘GPTSentencePieceTokenizer’, ‘Llama2Tokenizer’]

core.tokenizers.utils.build_tokenizer.SUPPORTED_TOKENIZERS#: None

core.tokenizers.utils.build_tokenizer.logger#: ‘getLogger(…)’

core.tokenizers.utils.build_tokenizer.build_tokenizer(args, **kwargs)#: Initialize tokenizer.

core.tokenizers.utils.build_tokenizer.vocab_size_with_padding(orig_vocab_size, args, logging_enabled=True)#: Pad vocab size so it is divisible by model parallel size and still having GPU friendly size.

core.tokenizers.utils.build_tokenizer._set_padded_vocab_size(args, tokenizer)#: Sets padded vocab size if None.