bridge.models.inkling.data.tokenizer#
Megatron-Core tokenizer for Inkling2 text training.
The thinkingmachines/Inkling-2-Small tokenizer configuration assigns no BOS or EOS token, which pretraining
datasets and SFT padding need, and its chat template marks no assistant span for the SFT loss.
Inkling2MegatronTokenizer loads the release’s Hugging Face tokenizer with the reference tokenizer’s roles and
declares the chat template’s assistant-turn boundaries. Recipes select it through Megatron-Core’s tokenizer metadata:
cfg.tokenizer.metadata_path = inkling2_tokenizer_metadata().
Module Contents#
Classes#
The Inkling2 Hugging Face tokenizer with BOS and EOS roles and the chat template’s assistant-turn boundaries. |
Functions#
Return the Megatron-Core tokenizer metadata that loads a Hugging Face tokenizer as |
Data#
API#
- bridge.models.inkling.data.tokenizer.INKLING2_BOS_TOKEN#
‘<|begin_of_text|>’
- bridge.models.inkling.data.tokenizer.INKLING2_EOS_TOKEN#
‘<|endoftext|>’
- class bridge.models.inkling.data.tokenizer.Inkling2MegatronTokenizer#
Bases:
megatron.core.tokenizers.text.text_tokenizer.MegatronTokenizerTextThe Inkling2 Hugging Face tokenizer with BOS and EOS roles and the chat template’s assistant-turn boundaries.
- _restore_model(
- **kwargs: Any,
Load the Hugging Face tokenizer with the BOS and EOS roles; their tokens must already be in its vocabulary.
- assistant_mask_boundary_config() megatron.bridge.data.conversation_processing.AssistantMaskBoundaryConfig#
Return the chat template’s assistant-turn boundaries for the SFT loss mask.
The template renders every message as role-tagged blocks such as
<|message_model|><|content_text|>...<|end_message|>and ends an assistant turn with<|content_model_end_sampling|>. The model writes its own<|message_model|>headers, so the loss covers the whole turn, from its first header through the end-of-sampling token.
- bridge.models.inkling.data.tokenizer.inkling2_tokenizer_metadata() dict[str, str]#
Return the Megatron-Core tokenizer metadata that loads a Hugging Face tokenizer as
Inkling2MegatronTokenizer.TokenizerConfig.metadata_pathtakes this metadata in place of a path to a metadata file.
- bridge.models.inkling.data.tokenizer.__all__#
[‘INKLING2_BOS_TOKEN’, ‘INKLING2_EOS_TOKEN’, ‘Inkling2MegatronTokenizer’, ‘inkling2_tokenizer_metada…