Reference#

Lookup material for the four tokenizer_extension steps. Every field on these pages traces to the step’s config/default.yaml, step.toml, or executor source under src/nemotron/steps/tokenizer_extension/.

Page

Content

Extend YAML Reference

extend/config/default.yaml: method, language, corpus, and output fields

Init Embeddings YAML Reference

init_embeddings/config/default.yaml: arm, engine, and per-engine fields

Evaluate YAML Reference

evaluate/config/default.yaml: tokenizer and evaluation corpus fields

Eval Init YAML Reference

eval_init/config/default.yaml: models, corpus, and budget fields

CLI Reference

nemotron steps run tokenizer_extension/<step> syntax and global options

Output Files

Directories and JSON files each step writes

Troubleshooting

Named errors from the step.toml files with their recoveries

Where to Read More#

Document

Content

src/nemotron/steps/tokenizer_extension/guide.md

Category overview and quickstart

src/nemotron/steps/tokenizer_extension/LANGUAGES.md

Language profiles and registration

src/nemotron/steps/tokenizer_extension/extend/README.md

Extension methods and the constructive splice

src/nemotron/steps/tokenizer_extension/init_embeddings/README.md

Initialization engines and the Add versus Replace support table

src/nemotron/steps/tokenizer_extension/evaluate/README.md

Fertility evaluation

src/nemotron/steps/tokenizer_extension/eval_init/README.md

BPB evaluation

src/nemotron/steps/tokenizer_extension/guidebook/README.md

Measured results and decision rules
On this page