Tokenizer Extension With Nemotron#

The tokenizer_extension step category extends a base-model tokenizer with subwords for a target language, initializes the model embedding rows for the new vocabulary, and measures the result. The output is a resized Hugging Face checkpoint that you pass to the pretrain/megatron_bridge step as hf_model_path for continued pretraining (CPT).

Tip

New here? Start with Getting Started With Tokenizer Extension, then use this page as the map to deeper topics.

When to Use#

Use the tokenizer_extension steps when you need:

Lower token fertility, the number of tokens per word, for a language that the base tokenizer represents inefficiently. Fewer tokens per word reduce context consumption and serving cost.
A resized checkpoint whose new embedding rows start from an informed initialization rather than random values, so that CPT converges from a better starting point.
Comparable measurements across tokenizers with different vocabularies, by using bits per byte (BPB) rather than per-token perplexity.

Tokenizer extension is primarily an efficiency intervention. The Tokenizer Extension Guidebook reports that an independent contribution to downstream model quality has not been established.

Pipeline Summary#

        flowchart LR
    A[Base checkpoint_hf] --> B[extend]
    C[Target-language corpus] --> B
    B --> D[tokenizer]
    D --> E[init_embeddings]
    A --> E
    E --> F[Resized checkpoint_hf]
    F --> G[pretrain/megatron_bridge]
    D --> H[evaluate: fertility]
    F --> I[eval_init: BPB]

The category contains four steps. Each step reads what the previous one wrote.

Step	Compute	Consumes	Produces
`tokenizer_extension/extend`	CPU	Base tokenizer and a text corpus	Extended tokenizer directory and `summary.json`
`tokenizer_extension/init_embeddings`	GPU	Extended tokenizer and base checkpoint	Resized Hugging Face checkpoint
`tokenizer_extension/evaluate`	CPU	Any tokenizer	Fertility report JSON
`tokenizer_extension/eval_init`	GPU	One or more resized checkpoints	BPB and perplexity report JSON

Documentation Series#

Tutorial

Build an extended tokenizer, initialize embeddings, and measure fertility with the shipped default.yaml configurations.

hands-on

Getting Started With Tokenizer Extension

How-to guides

Choose an extension method, initialize embeddings, evaluate, register a new language, and run on a cluster.

task-based

How-To Guides

Concepts

Pipeline architecture, Add versus Replace versus Expand, embedding initialization, and evaluation metrics.

learn

Concepts

Reference

YAML parameters for each step, CLI syntax, output files, and troubleshooting.

lookup

Reference

All Documentation#

Tutorial

Guide	What you do
Prerequisites	Software, hardware, data, and access requirements
Getting Started With Tokenizer Extension	Run `extend`, `init_embeddings`, and `evaluate` for one language

How-to guides

Guide	Focus
Extend a Tokenizer	`method: add`, `replace`, or `expand`
Initialize Embeddings	`method: baseline`, `subword`, or `focus` and the `arm` setting
Evaluate a Tokenizer	Fertility and BPB comparisons
Add a Language	`languages.py` and `script_ranges.py` entries
Run on a Cluster	Environment profiles and optional dependencies

Concepts

Guide	Topic
Pipeline Overview	Steps, artifacts, and the hand-off to CPT
Add, Replace, and Expand	Add, Replace, and Expand
Embedding Initialization	Baseline, subword, and FOCUS engines
Evaluation Metrics	Fertility and bits per byte

Reference

Guide	Content
Extend YAML Reference	`extend/config/default.yaml` fields
Init Embeddings YAML Reference	`init_embeddings/config/default.yaml` fields
Evaluate YAML Reference	`evaluate/config/default.yaml` fields
Eval Init YAML Reference	`eval_init/config/default.yaml` fields
CLI Reference	`nemotron steps run tokenizer_extension/<step>` syntax
Output Files	Files each step writes
Troubleshooting	Symptoms, causes, and corrections

Limitations and Considerations#

Scope: the evaluate step measures tokenizer-level fertility only. Model and downstream evaluation after CPT belongs to the eval step catalog; see About Model Evaluation.
Optional dependencies: indic-nlp-library, fasttext-wheel, and accelerate are declared as the tokenizer-extension extra in pyproject.toml. The base install does not include these packages. A run that needs one of them and cannot import it fails with the install command rather than producing a quietly different tokenizer.
Learning-rate policy: the guidebook reports differential learning-rate results that were measured outside this repository. The pretrain/megatron_bridge step does not ship a differential learning-rate option.
Remote execution: use --run <profile> or --batch <profile> with the *_tokenizer_extend, *_tokenizer_init_embeddings, *_tokenizer_evaluate, and *_tokenizer_eval_init profiles that env/env_toml generates.

Quick Paths#

First run: Getting Started With Tokenizer Extension
Choose a method: Add, Replace, and Expand
Register a language: Add a Language
Look up a field: Reference