> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.models.kimi_k3.tokenization

## Module Contents

### Classes

| Name                                                                                            | Description                                                                                                        |
| ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| [`TikTokenTokenizer`](#nemo_automodel-components-models-kimi_k3-tokenization-TikTokenTokenizer) | Tokenizing and encoding/decoding text using the Tiktoken tokenizer. See megatron/tokenizer/tiktoken\_tokenizer.py. |

### Functions

| Name                                                                                                      | Description                                                                         |
| --------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| [`_build_kimi_k3_pat_str`](#nemo_automodel-components-models-kimi_k3-tokenization-_build_kimi_k3_pat_str) | Build the Kimi K3 tiktoken regex without exposing raw set intersections to autodoc. |

### Data

[`VOCAB_FILES_NAMES`](#nemo_automodel-components-models-kimi_k3-tokenization-VOCAB_FILES_NAMES)

[`logger`](#nemo_automodel-components-models-kimi_k3-tokenization-logger)

### API

```python
class nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer(
    vocab_file,
    bos_token: typing.Union[str, tokenizers.AddedToken] = '[BOS]',
    eos_token: typing.Union[str, tokenizers.AddedToken] = '[EOS]',
    unk_token: typing.Union[str, tokenizers.AddedToken, None] = None,
    pad_token: typing.Union[str, tokenizers.AddedToken, None] = None,
    additional_special_tokens: typing.List[str] = None,
    added_tokens_decoder: typing.Optional[dict] = None,
    kwargs = {}
)
```

**Bases:** `PreTrainedTokenizer`

Tokenizing and encoding/decoding text using the Tiktoken tokenizer. See megatron/tokenizer/tiktoken\_tokenizer.py.

This tokenizer inherits from \[`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.

**Parameters:**

**`vocab_file`** `` `str` ``

The path to the Tiktoken model file.

---

**`bos_token`** `` `str` or `tokenizers.AddedToken`, *optional* `` — default: '\[BOS]'

The beginning of sequence token that was used during pretraining. Defaults to `[BOS]`. Can be used as a
sequence classifier token.

---

**`eos_token`** `` `str` or `tokenizers.AddedToken`, *optional* `` — default: '\[EOS]'

The end of sequence token. Defaults to `[EOS]`.

---

**`unk_token`** `` `str` or `tokenizers.AddedToken`, *optional* `` — default: None

The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead. Defaults to `None`.

---

**`pad_token`** `` `str` or `tokenizers.AddedToken`, *optional* `` — default: None

The token used for padding, for example when batching sequences of different lengths. Defaults to `None`.

---

**`additional_special_tokens`** ``list of `str`, *optional*`` — default: None

A tuple or a list of additional tokens, which will be marked as `special`, meaning that they will be
skipped when decoding if `skip_special_tokens` is set to `True`.

---

**`all_special_ids_set`** `= set(self.all_special_ids)`

---

**`bos_id`** `int = self.special_tokens[str(bos_token)]`

---

**`byte_decoder`** `= {v: k for k, v in (self.byte_encoder.items())}`

---

**`byte_encoder`** `= bytes_to_unicode()`

---

**`decoder`** `= {}`

---

**`encoder`** `= {}`

---

**`eos_id`** `int = self.special_tokens[str(eos_token)]`

---

**`model`**

---

**`model_input_names`** `= ['input_ids', 'attention_mask']`

---

**`n_words`** `int = self.model.n_vocab`

---

**`num_reserved_special_tokens`** `= 256`

---

**`pad_id`** `int = self.special_tokens[str(pad_token)]`

---

**`pat_str`** `= _build_kimi_k3_pat_str()`

---

**`special_tokens`** `Dict[str, int]`

---

**`unk_id`** `int = self.special_tokens[str(unk_token)]`

---

**`vocab_size`** `int`

---

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._convert_id_to_token(
    index: int
) -> str
```

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._convert_token_to_id(
    token: str
) -> int
```

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._encode_chat_segments(
    segments,
    return_assistant_tokens_mask: bool = False
) -> typing.List[int] | tuple[typing.List[int], typing.List[int]]
```

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._encode_text_piece(
    text: str,
    allow_special_tokens: bool = True
) -> typing.List[int]
```

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._format_chat_token_output(
    encoded_inputs: typing.List[typing.List[int]],
    is_batched: bool,
    padding = False,
    truncation: bool = False,
    max_length: typing.Optional[int] = None,
    return_tensors = None,
    return_dict: bool = False,
    assistant_masks: typing.Optional[typing.List[typing.List[int]]] = None
)
```

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._split_whitespaces_or_nonwhitespaces(
    s: str,
    max_consecutive_slice_len: int
) -> typing.Iterator[str]
```

staticmethod

Splits the string `s` so that each substring contains no more than `max_consecutive_slice_len`
consecutive whitespaces or consecutive non-whitespaces.

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._tokenize(
    text: str,
    kwargs = {}
) -> typing.List[str]
```

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer._truncate(
    ids: typing.List[int],
    truncation: bool = False,
    max_length: typing.Optional[int] = None
) -> typing.List[int]
```

staticmethod

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer.apply_chat_template(
    conversation,
    tools: typing.Optional[list[dict]] = None,
    tokenize: bool = False,
    add_generation_prompt: bool = False,
    thinking: bool = True,
    padding = False,
    truncation: bool = False,
    max_length: typing.Optional[int] = None,
    return_tensors = None,
    return_dict: bool = False,
    return_assistant_tokens_mask: bool = False,
    kwargs = {}
)
```

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer.clean_up_tokenization(
    out_string: str
) -> str
```

staticmethod

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer.convert_tokens_to_string(
    tokens: typing.List[str]
) -> str
```

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer.decode(
    token_ids: typing.Union[int, typing.List[int]],
    kwargs = {}
) -> str
```

Decodes a list of token IDs into a string.

**Parameters:**

**`token_ids`** `List[int]`

The list of token IDs to be decoded.

---

**Returns:** `str`

The decoded string.

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer.encode(
    text: str,
    allow_special_tokens: bool = True,
    kwargs = {}
) -> typing.List[int]
```

Encodes a string into a list of token IDs.

**Parameters:**

**`text`** `str`

The input string to be encoded.

---

**Returns:** `List[int]`

list\[int]: A list of token IDs.

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer.get_vocab() -> typing.Dict[str, int]
```

```python
nemo_automodel.components.models.kimi_k3.tokenization.TikTokenTokenizer.save_vocabulary(
    save_directory: str,
    filename_prefix: typing.Optional[str] = None
) -> typing.Tuple[str]
```

```python
nemo_automodel.components.models.kimi_k3.tokenization._build_kimi_k3_pat_str() -> str
```

Build the Kimi K3 tiktoken regex without exposing raw set intersections to autodoc.

```python
nemo_automodel.components.models.kimi_k3.tokenization.VOCAB_FILES_NAMES = {'vocab_file': 'tiktoken.model'}
```

```python
nemo_automodel.components.models.kimi_k3.tokenization.logger = getLogger(__name__)
```