`core.tokenizers.text.libraries.tiktoken_tokenizer`#

Module Contents#

Classes#

TikTokenTokenizer

TikTokenTokenizer https://github.com/openai/tiktoken.

Functions#

reload_mergeable_ranks

Reload the tokenizer JSON file and convert it to Tiktoken format.

Data#

`PATTERN_TIKTOKEN_V1`
`PATTERN_TIKTOKEN_V2`
`DEFAULT_TIKTOKEN_MAX_VOCAB`
`SPECIAL_TOKENS`
`SPECIAL_TOKEN_TEMPLATE`

API#

core.tokenizers.text.libraries.tiktoken_tokenizer.PATTERN_TIKTOKEN_V1#: ‘[^\\r\\n\\p{L}\\p{N}]?+\p{L}+|\p{N}| ?[^\\s\\p{L}\\p{N}]++[\r\n]|\s[\r\n]|\s+(?!\S)|\s+’

core.tokenizers.text.libraries.tiktoken_tokenizer.PATTERN_TIKTOKEN_V2#: ‘[^\\r\\n\\p{L}\\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|[^\\r\\n\\p{L}\\p{N}]?[\p{L…’

core.tokenizers.text.libraries.tiktoken_tokenizer.DEFAULT_TIKTOKEN_MAX_VOCAB#: None

core.tokenizers.text.libraries.tiktoken_tokenizer.SPECIAL_TOKENS#: [’’, ‘~~’, ‘~~’, ‘’, ‘’, ‘’, ‘’]

core.tokenizers.text.libraries.tiktoken_tokenizer.SPECIAL_TOKEN_TEMPLATE#: ‘<SPECIAL_{id}>’

core.tokenizers.text.libraries.tiktoken_tokenizer.reload_mergeable_ranks( path: str, max_vocab: Optional[int] = None, num_special_tokens: Optional[int] = None, ) → Dict[bytes, int]#

Reload the tokenizer JSON file and convert it to Tiktoken format.

Parameters:

path (str) – path to the tokenizer.
max_vocab (Optional[int]) – maximum size of vocabulary.
num_special_tokens (Optional[int]) – number of added special tokens.

Returns:

reloaded tokenizer vocab.

Return type:

Dict[bytes, int]

class core.tokenizers.text.libraries.tiktoken_tokenizer.TikTokenTokenizer( tokenizer_path: str, special_tokens: Optional[List[str]] = None, num_special_tokens: Optional[int] = 1000, chat_template: Optional[str] = None, pattern: Optional[str] = 'v2', vocab_size: Optional[int] = DEFAULT_TIKTOKEN_MAX_VOCAB, )#

Bases: core.tokenizers.text.libraries.abstract_tokenizer.MegatronTokenizerTextAbstract, core.tokenizers.text.libraries.chat_template.MegatronTokenizerChatTemplate