nemo_automodel.components.models.kimi_k3.tokenization

Module Contents

Classes

Name	Description
`TikTokenTokenizer`	Tokenizing and encoding/decoding text using the Tiktoken tokenizer. See megatron/tokenizer/tiktoken_tokenizer.py.

Functions

Name	Description
`_build_kimi_k3_pat_str`	Build the Kimi K3 tiktoken regex without exposing raw set intersections to autodoc.

Data

VOCAB_FILES_NAMES

logger

API

class nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer(
    vocab_file,
    bos_token: typing.Union[str, tokenizers.AddedToken] = '[BOS]',
    eos_token: typing.Union[str, tokenizers.AddedToken] = '[EOS]',
    unk_token: typing.Union[str, tokenizers.AddedToken, None] = None,
    pad_token: typing.Union[str, tokenizers.AddedToken, None] = None,
    additional_special_tokens: typing.List[str] = None,
    added_tokens_decoder: typing.Optional[dict] = None,
    kwargs = {}
)

Bases: PreTrainedTokenizer

Tokenizing and encoding/decoding text using the Tiktoken tokenizer. See megatron/tokenizer/tiktoken_tokenizer.py.

This tokenizer inherits from [PreTrainedTokenizer] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

Parameters:

vocab_file

`str`

The path to the Tiktoken model file.

bos_token

`str` or `tokenizers.AddedToken`, *optional*Defaults to '[BOS]'

The beginning of sequence token that was used during pretraining. Defaults to [BOS]. Can be used as a sequence classifier token.

eos_token

`str` or `tokenizers.AddedToken`, *optional*Defaults to '[EOS]'

The end of sequence token. Defaults to [EOS].

unk_token

`str` or `tokenizers.AddedToken`, *optional*Defaults to None

The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. Defaults to None.

pad_token

`str` or `tokenizers.AddedToken`, *optional*Defaults to None

The token used for padding, for example when batching sequences of different lengths. Defaults to None.

additional_special_tokens

list of `str`, *optional*Defaults to None

A tuple or a list of additional tokens, which will be marked as special, meaning that they will be skipped when decoding if skip_special_tokens is set to True.

all_special_ids_set

= set(self.all_special_ids)

bos_id

int = self.special_tokens[str(bos_token)]

byte_decoder

= {v: k for k, v in (self.byte_encoder.items())}

byte_encoder

= bytes_to_unicode()

decoder

= {}

encoder

= {}

eos_id

int = self.special_tokens[str(eos_token)]

model

model_input_names

= ['input_ids', 'attention_mask']

n_words

int = self.model.n_vocab

num_reserved_special_tokens

= 256

pad_id

int = self.special_tokens[str(pad_token)]

pat_str

= _build_kimi_k3_pat_str()

special_tokens

Dict[str, int]

unk_id

int = self.special_tokens[str(unk_token)]

vocab_size

int

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._convert_id_to_token(
    index: int
) -> str

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._convert_token_to_id(
    token: str
) -> int

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._encode_chat_segments(
    segments,
    return_assistant_tokens_mask: bool = False
) -> typing.List[int] | tuple[typing.List[int], typing.List[int]]

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._encode_text_piece(
    text: str,
    allow_special_tokens: bool = True
) -> typing.List[int]

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._format_chat_token_output(
    encoded_inputs: typing.List[typing.List[int]],
    is_batched: bool,
    padding = False,
    truncation: bool = False,
    max_length: typing.Optional[int] = None,
    return_tensors = None,
    return_dict: bool = False,
    assistant_masks: typing.Optional[typing.List[typing.List[int]]] = None
)

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._split_whitespaces_or_nonwhitespaces(
    s: str,
    max_consecutive_slice_len: int
) -> typing.Iterator[str]

staticmethod

Splits the string s so that each substring contains no more than max_consecutive_slice_len consecutive whitespaces or consecutive non-whitespaces.

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._tokenize(
    text: str,
    kwargs = {}
) -> typing.List[str]

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._truncate(
    ids: typing.List[int],
    truncation: bool = False,
    max_length: typing.Optional[int] = None
) -> typing.List[int]

staticmethod

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer.apply_chat_template(
    conversation,
    tools: typing.Optional[list[dict]] = None,
    tokenize: bool = False,
    add_generation_prompt: bool = False,
    thinking: bool = True,
    padding = False,
    truncation: bool = False,
    max_length: typing.Optional[int] = None,
    return_tensors = None,
    return_dict: bool = False,
    return_assistant_tokens_mask: bool = False,
    kwargs = {}
)

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer.clean_up_tokenization(
    out_string: str
) -> str

staticmethod

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer.convert_tokens_to_string(
    tokens: typing.List[str]
) -> str

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer.decode(
    token_ids: typing.Union[int, typing.List[int]],
    kwargs = {}
) -> str

Decodes a list of token IDs into a string.

Parameters:

token_ids

List[int]

The list of token IDs to be decoded.

Returns: str

The decoded string.

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer.encode(
    text: str,
    allow_special_tokens: bool = True,
    kwargs = {}
) -> typing.List[int]

Encodes a string into a list of token IDs.

Parameters:

text

str

The input string to be encoded.

Returns: List[int]

list[int]: A list of token IDs.

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer.get_vocab() -> typing.Dict[str, int]

nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer.save_vocabulary(
    save_directory: str,
    filename_prefix: typing.Optional[str] = None
) -> typing.Tuple[str]

nemo_automodel.components.models.kimi_k3.tokenization._build_kimi_k3_pat_str() -> str

Build the Kimi K3 tiktoken regex without exposing raw set intersections to autodoc.

nemo_automodel.components.models.kimi_k3.tokenization.VOCAB_FILES_NAMES = {'vocab_file': 'tiktoken.model'}

nemo_automodel.components.models.kimi_k3.tokenization.logger = getLogger(__name__)