Concepts

These pages explain how the tokenizer_extension steps work and why they make the choices they do. Read them when you need to decide between methods rather than run a fixed procedure.

Page	Topic
Pipeline Overview	The four steps, the artifacts they exchange, and the hand-off to continued pretraining
Add, Replace, and Expand	Add, Replace, and Expand, and why the splice into the merge table is constructive
Embedding Initialization	The baseline, subword, and FOCUS engines, norm correction, and Add versus Replace surgery
Evaluation Metrics	Fertility and bits per byte, and why per-token perplexity does not compare across vocabularies

For measured results in Hindi, Malayalam, and Vietnamese, read the Tokenizer Extension Guidebook in the repository.