Output Files

Each step writes to a path you set in its configuration. Field names below trace to the writer in each step’s executor.

extend

Writes output_dir/<method>/, where <method> is add, replace, or expand.

File	Written by	Content
Tokenizer files (tokenizer.json, tokenizer_config.json, and related files)	save_pretrained	The extended tokenizer, loadable with AutoTokenizer.
summary.json	every method	Run record; fields below.
id_remap.json	replace only	Map from old token ID (string key) to new token ID for surviving tokens. Consumed by init_embeddings with arm=replace.
removed_tokens.txt	replace only	One pruned token per line.

summary.json fields:

Field	Meaning
model_id	Base tokenizer identifier.
method	add, replace, or expand.
language	The language key as configured, lower-cased, or null when it was not set.
language_source	config when language was set. legacy-devanagari-default when the run used the Devanagari defaults.
script_normalizer, remove_script	The values the run acted on, after any override.
language_overrides	Whether script_normalizer and remove_script were set explicitly rather than taken from the language profile.
language_profile_defaults	The script_normalizer, remove_script, fasttext, and encoder values from the language profile. Compare these values with an override to see what the override changed. The value is null when no profile matched.
extension_size	Requested budget.
corpus	The corpus block as configured.
base_vocab_size	Token count of the base tokenizer.
final_vocab_size	Token count of the written tokenizer.
new_candidates	Novel tokens produced by BPE training before the splice.
tokens_requested	Same value as extension_size.
tokens_spliced	Tokens actually added. Constructive merging may add intermediate tokens, so this value can differ from tokens_requested. Compare arms only at equal tokens_spliced. A value of 0 is an error. The step writes no tokenizer, though an empty output_dir/<method>/ may remain.
removed	replace only: number of pruned tokens.
output	Output directory.
timings_sec	train, build, and total wall-clock seconds.

init_embeddings

Writes output_dir/ as a resized Hugging Face checkpoint: model weights saved with save_pretrained in the configured dtype, plus a copy of the extended tokenizer. The directory is the hf_model_path input for pretrain/megatron_bridge. Per-token initialization samples are printed to the log rather than written to a file.

evaluate

Writes the JSON report at output.

Field	Meaning
label	The configured label, or the tokenizer value.
tokenizer	Tokenizer identifier or path.
vocab_size	Token count.
fix_mistral_regex	Whether the Mistral pre-tokenizer regex correction was applied.
eval_corpus	The corpus block as configured.
fertility_definition	sum(tokens)/sum(words).
totals	docs, words, tokens, chars.
metrics.fertility	Tokens per whitespace-delimited word; lower is better.
metrics.chars_per_token	Characters per token; higher is better.
metrics.unique_tokens_used	Distinct token IDs observed.
metrics.vocab_coverage	unique_tokens_used / vocab_size.
elapsed_sec	Wall-clock seconds.

eval_init

Writes the JSON report at output_json.

Field	Meaning
data_file, max_docs, max_tokens, max_length, stride, dtype	The effective evaluation settings.
results.<label>.loss	Mean negative log-likelihood per token.
results.<label>.perplexity	exp(loss); not comparable across tokenizers.
results.<label>.bpb	Bits per byte; comparable across tokenizers on identical text.
results.<label>.num_tokens, num_bytes, num_docs	Scored volume.

The base model appears under its own label; the console output also prints each model’s BPB delta against it.