How-To Guides#
This section has task-focused procedures for the tokenizer_extension steps.
Start with Getting Started With Tokenizer Extension if you have not run the steps yet.
Build and Initialize#
Extend a tokenizer
Select method: add, replace, or expand, point corpus at your data, and verify tokens_spliced.
Initialize embeddings
Match arm to the extension method, choose an engine, and produce the checkpoint for continued pretraining.
Measure#
Evaluate a tokenizer
Compare fertility across tokenizers and BPB across resized checkpoints on identical text.
Adapt and Scale#
Add a language
Register a LanguageProfile and, if needed, Unicode ranges so that one language key configures every step.
Run on a cluster
Use the generated *_tokenizer_* environment profiles and install the optional dependencies in the container.