`core.tokenizers.text.libraries.sft_tokenizer`#

Module Contents#

Classes#

`PromptConfig`	Config options for different prompt formats.
`SFTTokenizer`	SFT Tokenizer.

Data#

`nemotron_h_aligned_custom_template`
`nemotron_nano_v2_custom_template`
`identity_template`
`IGNORE_INDEX`

API#

core.tokenizers.text.libraries.sft_tokenizer.nemotron_h_aligned_custom_template = <Multiline-String>#

core.tokenizers.text.libraries.sft_tokenizer.nemotron_nano_v2_custom_template = <Multiline-String>#

core.tokenizers.text.libraries.sft_tokenizer.identity_template#: “{% for message in messages %}{{ message[‘content’] }}{% endfor %}”

core.tokenizers.text.libraries.sft_tokenizer.IGNORE_INDEX#: None

class core.tokenizers.text.libraries.sft_tokenizer.PromptConfig#

Config options for different prompt formats.

assistant_prefix_len: int#: None

pad_token_id: int#: None

custom_chat_template: str#: None

has_bos: bool#: None

has_system_role: bool#: None

force_system_message: bool#: False

system_default: dict#: None

class core.tokenizers.text.libraries.sft_tokenizer.SFTTokenizer(tokenizer_path: str, prompt_format: str)#

SFT Tokenizer.

Initialization

Note: Currently, only HuggingFaceTokenizer is supported as the underlying text tokenizer.

Parameters:

tokenizer_path (str) – Underlying tokenizer path.
prompt_format (str) – Prompt format for the tokenizer.

tokenize_conversation( conversation: List[Dict], return_target: bool, add_generation_prompt: bool, )#

Convert a conversation to tokens.