Getting Started With Tokenizer Extension#

You will build a tokenizer that adds 30,000 target-language subwords to the Nemotron 3 Nano tokenizer, initialize the matching embedding rows in the base model, and measure the fertility of the result.

This tutorial runs three of the four tokenizer_extension steps in order. Each step uses its config/default.yaml and CLI dotlist overrides.

What You Will Produce

An extended tokenizer under ./output/tokenizer_extension/add/, a resized Hugging Face checkpoint under ./output/resized_checkpoint/, and a fertility report JSON.

Overview

Install the tokenizer-extension extra.
Run tokenizer_extension/extend with method=add.
Run tokenizer_extension/init_embeddings with arm=add.
Run tokenizer_extension/evaluate on the extended tokenizer.

The extend and evaluate steps run on CPU and stream their corpora. The init_embeddings step loads the base model and needs a GPU node.

Prerequisites#

See Prerequisites for the full requirements. For this tutorial you need:

A Hugging Face dataset, or a local Parquet or JSON Lines corpus, in the target language. The extend default configuration points at ai4bharat/sangraha with hf_name: verified and hf_split: hin.
A GPU node for init_embeddings. The base model, nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16 by default, is loaded host-resident for embedding surgery; the GPU serves the auxiliary encoders, and the generated environment profiles request one GPU for this step.
A language value from the registry in languages.py. The registered values are hindi, marathi, nepali, sanskrit, bengali, punjabi, gujarati, odia, tamil, telugu, kannada, malayalam, urdu, and vietnamese.

Procedure#

Clone the repository, if you have not already:

$ git clone https://github.com/NVIDIA-NeMo/Nemotron && cd Nemotron

Install the optional dependencies.

indic-nlp-library is required when language selects the Devanagari normalizer (hindi, marathi, nepali, or sanskrit). fasttext-wheel is required only for method: focus in init_embeddings.
```
$ uv pip install -e '.[tokenizer-extension]'
```
Set the values that the following commands share.
```
$ L=hindi
$ OUT=./output/tokenizer_extension
```
Substitute a registered language value and a corpus that matches it. The commands below keep the corpus from extend/config/default.yaml; to use a different Hugging Face dataset, add corpus.hf_dataset=<id> corpus.hf_name=<config> corpus.hf_split=<split> overrides.
Build the extended tokenizer.
```
$ uv run nemotron steps run tokenizer_extension/extend -c default \
    language=$L method=add extension_size=30000 output_dir=$OUT
```
The step writes $OUT/add/, which contains the tokenizer files and summary.json. Confirm that tokens_spliced in summary.json equals extension_size. The step logs a warning when the two values differ, because fertility and BPB comparisons between arms are valid only at a matched budget.
Initialize the new embedding rows and write the resized checkpoint.
```
$ uv run nemotron steps run tokenizer_extension/init_embeddings -c default \
    language=$L arm=add extended_tokenizer=$OUT/add \
    output_dir=./output/resized_checkpoint
```
The default method is subword with input_averaging: uniform, which initializes each new row from the mean of its base-vocabulary subword rows. The arm value must match how extend built the tokenizer: add for method=add or method=expand, and replace for method=replace.

The output directory is a Hugging Face checkpoint with weights and tokenizer. Pass it to pretrain/megatron_bridge as hf_model_path when you start continued pretraining.
Measure fertility.
```
$ uv run nemotron steps run tokenizer_extension/evaluate -c default \
    tokenizer=$OUT/add output=./output/eval/fertility_add.json
```
The default evaluation corpus is ai4bharat/samanantar with hf_config: hi and text_field: tgt. Change the corpus block to match your target language. The report records fertility, chars_per_token, unique_tokens_used, and vocab_coverage. Lower fertility is better. To compare tokenizers, run the step once per tokenizer with the same corpus block.
Optional: Print the merged configuration without running a step.

Pass --dry-run or -d to any of the commands above.
```
$ uv run nemotron steps run tokenizer_extension/extend -d -c default language=$L method=add
```

Next Steps#

Compare the three extension methods: Add, Replace, and Expand
Choose an initialization engine: Initialize Embeddings
Score the resized checkpoint with bits per byte: Evaluate a Tokenizer
Run the steps on Lepton, DGX Cloud, or Slurm: Run on a Cluster
Field meanings: Extend YAML Reference