Extend a Tokenizer#

Use tokenizer_extension/extend to train a BPE on a target-language corpus and splice the new tokens into the base tokenizer. The step runs on CPU.

Before You Begin#

  • Choose a registered language value; see Add a Language if yours is not registered.

  • Install indic-nlp-library when the language uses the Devanagari normalizer (hindi, marathi, nepali, sanskrit):

    $ uv pip install -e '.[tokenizer-extension]'

  • Decide the method; see Add, Replace, and Expand.

Select the Corpus#

Provide either a Hugging Face dataset or a local path, plus the text column. The keys live under corpus in extend/config/default.yaml.

Hugging Face dataset:

$ uv run nemotron steps run tokenizer_extension/extend -c default \
    language=vietnamese method=add extension_size=30000 \
    corpus.hf_dataset=<org/dataset> corpus.hf_name=<config> corpus.hf_split=<split> \
    corpus.text_field=text

corpus.hf_name is the dataset configuration or subset passed as name to load_dataset; hf_data_dir, hf_data_files, and hf_revision are optional. Set corpus.streaming=true to stream instead of downloading and caching the dataset.

Local Parquet or JSON Lines:

$ uv run nemotron steps run tokenizer_extension/extend -c default \
    language=vietnamese method=add extension_size=30000 \
    corpus.hf_dataset=null corpus.path=<dir-or-glob> corpus.glob="'*.parquet'" \
    corpus.text_field=text

Leave corpus.hf_dataset null when corpus.path is set. Use inner quotes for a glob override: corpus.glob="'*.parquet'". The default is *.parquet; omit the key when it is unchanged. For a local directory, keep corpus.diversify: true so that sampling covers every shard.

Choose the Method#

$ uv run nemotron steps run tokenizer_extension/extend -c default language=<lang> method=add
$ uv run nemotron steps run tokenizer_extension/extend -c default language=<lang> method=replace
$ uv run nemotron steps run tokenizer_extension/extend -c default language=<lang> method=expand

Each command writes a separate subdirectory of output_dir: add/, replace/, or expand/. Run the arms you want to compare as separate jobs with the same extension_size.

Do not set remove_script or script_normalizer unless you intend to override the language profile. An explicit value takes precedence over language; for example, language=vietnamese remove_script=devanagari prunes Devanagari rows from a Vietnamese run.

Bound Memory on a Small Node#

The BPE trainer is CPU- and RAM-bound. The step.toml strategies recommend:

$ uv run nemotron steps run tokenizer_extension/extend -c default language=<lang> method=add \
    corpus.samples=200000 corpus.max_doc_chars=600 corpus.min_frequency=3

corpus.min_frequency ignores merges below that word frequency; 0 keeps every merge.

Verify the Result#

Open output_dir/<method>/summary.json and compare tokens_spliced with extension_size.

  • Equal values mean the requested budget was reached.

  • A smaller tokens_spliced means the corpus was too small for the budget; widen the corpus or lower extension_size. The step logs a warning because fertility and BPB comparisons between arms are valid only at a matched tokens_spliced.

  • A tokens_spliced value of 0 never reaches this file. The step raises an error and writes no tokenizer. A corpus.text_field that does not name a corpus column also raises an error: Hugging Face and JSON Lines inputs report the missing column, and local Parquet reports IndexError: index out of bounds. Setting both corpus.hf_dataset and corpus.path is also an error. Refer to Troubleshooting.

The replace/ directory also contains id_remap.json, which the init_embeddings step needs for arm=replace.

Preview the Configuration#

$ uv run nemotron steps run tokenizer_extension/extend -d -c default language=<lang> method=add

Next Steps#