Extend YAML Reference#

The tokenizer_extension/extend step ships src/nemotron/steps/tokenizer_extension/extend/config/default.yaml as its starter configuration. This page lists the keys you can override with nemotron steps run tokenizer_extension/extend key=value dotlists, with the full file inlined below.

Default Configuration File#

# Tokenizer extension — ONE arm per job (add OR replace). CPU-only. Production starter.

model_id: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16
trust_remote_code: true

method: replace                  # add | replace  (run the two as separate jobs)
extension_size: 30000

# `language:` selects the corpus normalizer AND the Replace prune script from
# the profile table in ../../languages.py (hindi, vietnamese, malayalam, ...).
# Set it to match your corpus. Left unset, the legacy Devanagari defaults apply.
language: hindi
# Both are optional overrides; leave unset (or null) to use the profile above.
# Setting remove_script here OVERRIDES `language:` -- that is how a Vietnamese
# run can silently end up pruning Devanagari rows.
# remove_script: devanagari      # for Replace: script(s) whose residual tokens are pruned
# script_normalizer: devanagari

# Corpus for TRAINING the BPE. Provide EITHER an HF dataset OR a local path.
# HF subset model: hf_dataset (path) -> hf_name (config/subset) -> hf_split,
# plus optional hf_data_dir / hf_data_files / hf_revision.
corpus:
  hf_dataset: ai4bharat/sangraha
  hf_name: verified              # e.g. 'verified' (sangraha) or 'hi' (samanantar)
  hf_split: hin                  # e.g. 'hin' (sangraha) or 'train' (samanantar)
  hf_data_dir: null
  hf_data_files: null
  hf_revision: null
  streaming: false               # false = download+cache (reused by later jobs)
  path: null                     # OR a local parquet dir/glob or jsonl (leave hf_dataset null)
  glob: "*.parquet"
  text_field: text               # the text column
  samples: 1000000
  max_doc_chars: 20000
  min_frequency: 0               # 0 = keep all merges (splice is rank-dead-proof)
  diversify: true

batch_size: 1000
output_dir: ./output/tokenizer_extension

Keys Grouped by Concern#

Base Tokenizer and Method#

Key	Default	Description
`model_id`	`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16`	Hugging Face identifier or local path whose tokenizer is extended. Only the tokenizer is read.
`trust_remote_code`	`true`	Passed to `AutoTokenizer.from_pretrained`. Set `false` for repositories you do not trust.
`method`	`replace`	`add`, `replace`, or `expand`. One arm per job. See Add, Replace, and Expand.
`extension_size`	`30000`	Number of new tokens to splice into the vocabulary.
`batch_size`	`1000`	Documents per batch fed to the BPE trainer.
`output_dir`	`./output/tokenizer_extension`	Parent directory; the step writes to `output_dir/<method>/`.

Language and Overrides#

Key	Default	Description
`language`	`hindi`	Registered language profile that selects the corpus normalizer and the Replace prune script. Absent, the Devanagari defaults apply.
`remove_script`	unset	Comma-separated `SCRIPT_UNICODE_RANGES` keys whose residual tokens `replace` prunes. Overrides the profile when set.
`script_normalizer`	unset	`none`, `nfkc`, or `devanagari`. Overrides the profile when set.

Corpus#

The corpus block accepts either a Hugging Face dataset or a local path; the corpus_source_unset error is raised when neither is set.

Key	Default	Description
`corpus.hf_dataset`	`ai4bharat/sangraha`	Dataset identifier passed as `path` to `load_dataset`. Set `null` to use `corpus.path`. Setting both keys is an error.
`corpus.hf_name`	`verified`	Dataset configuration or subset passed as `name`.
`corpus.hf_split`	`hin`	Split to read.
`corpus.hf_data_dir`	`null`	Optional `data_dir` argument.
`corpus.hf_data_files`	`null`	Optional `data_files` argument.
`corpus.hf_revision`	`null`	Optional dataset revision.
`corpus.streaming`	`false`	`false` downloads and caches the dataset so later jobs reuse it; `true` streams.
`corpus.path`	`null`	Local Parquet directory or glob, or a JSON Lines path. `corpus.hf_dataset` must be `null` when this is set.
`corpus.glob`	`"*.parquet"`	File pattern applied when `corpus.path` is a directory.
`corpus.text_field`	`text`	Column that holds the document text. For a Hugging Face dataset or JSON Lines input, a missing column raises an error listing the available columns. For local Parquet, it raises `IndexError: index out of bounds`. The step never reads a different column.
`corpus.samples`	`1000000`	Maximum documents used for training.
`corpus.max_doc_chars`	`20000`	Documents are truncated to this many characters. Lower it on a memory-limited node.
`corpus.min_frequency`	`0`	Minimum word frequency for a merge; `0` keeps every merge.
`corpus.diversify`	`true`	For a local directory, sample evenly across shards.