Init Embeddings YAML Reference#

The tokenizer_extension/init_embeddings step ships src/nemotron/steps/tokenizer_extension/init_embeddings/config/default.yaml as its starter configuration. This page lists the keys you can override with nemotron steps run tokenizer_extension/init_embeddings key=value dotlists, with the full file inlined below.

Default Configuration File#

# Embedding init -> resized HF checkpoint for CPT. Needs a GPU node (loads the model).
#
# Dispatches to an init engine (embeddings.py):
#   method: baseline | subword | focus
# The default is subword/uniform (a.k.a. mean-of-constituents): every new token
# is initialized from the uniform mean of its base-vocabulary subword rows, with
# input norm correction to the median Devanagari norm.

# Nemotron-H and similar architectures ship their modeling code in the model
# repo, so loading them executes that code. The step defaults to false; this
# config opts in because the base model below requires it. Set false for any
# repo you do not trust, and prefer a pinned revision.
trust_remote_code: true

base_model: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16
dtype: bfloat16

# Target language. Selects the script used to find the base model's existing
# target rows, and the auxiliary encoder / fastText vectors (MuRIL for Indic,
# XLM-R otherwise). Set this instead of hardcoding bert_model / candidate_pool.
language: hindi
arm: add                            # add | replace (replace needs extend's replace/ output + its id_remap.json)

# Extended (append-style) tokenizer from the extend step.
extended_tokenizer: ./output/tokenizer_extension/add

# Resized HF checkpoint (weights + tokenizer) -> becomes CPT's hf_model_path.
output_dir: ./output/resized_checkpoint

method: subword

baseline:
  mode: mean_target                 # hf_default | mean_all | mean_target
  norm_correction: true
  num_samples: 10

subword:
  input_averaging: uniform          # uniform | char_weighted | max_char | bert_weighted | gemma_weighted
  output_averaging: uniform
  input_norm_correction: true
  output_norm_correction: false     # ON risks LM-head over-confidence -> loss explosion
  input_target_norm: true           # normalize to the median target-script norm
  output_target_norm: false
  temperature: 0.1                  # only used by bert_weighted / gemma_weighted
  # Leave bert_model unset so `language:` picks the right encoder. Setting it
  # here OVERRIDES the profile -- that is how a Vietnamese run silently ended up
  # weighting with MuRIL, which has no Vietnamese.
  # bert_model: google/muril-base-cased
  gemma_model: google/gemma-2-27b
  num_samples: 10

focus:
  fasttext_model: null              # required for method: focus (a cc.hi.300.bin)
  candidate_pool: target            # target | all
  sparsemax_temperature: 0.05
  num_samples: 10
  top_contributors: 5

Keys Grouped by Concern#

Model, Tokenizer, and Arm#

Key	Default	Description
`base_model`	`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16`	Base checkpoint whose embeddings are resized. Its embedding row count must equal the base tokenizer size.
`trust_remote_code`	`true`	Passed when loading the model. The step default is `false`; `default.yaml` opts in because the base model requires it.
`dtype`	`bfloat16`	Weight dtype for loading and saving.
`language`	`hindi`	Registered language profile that selects the target script, the auxiliary encoder, and the fastText vectors.
`arm`	`add`	`add` for an `add/` or `expand/` tokenizer; `replace` for a `replace/` tokenizer with `id_remap.json`.
`extended_tokenizer`	`./output/tokenizer_extension/add`	Tokenizer directory written by `extend`.
`output_dir`	`./output/resized_checkpoint`	Resized checkpoint directory; the value for `hf_model_path` in `pretrain/megatron_bridge`.
`method`	`subword`	`baseline`, `subword`, or `focus`. See Embedding Initialization.

`baseline`#

Key	Default	Description
`baseline.mode`	`mean_target`	`hf_default`, `mean_all`, or `mean_target`. `mean_hindi` is accepted as an alias for `mean_target`.
`baseline.norm_correction`	`true`	Rescale new rows to the median norm of the existing rows.
`baseline.num_samples`	`10`	Number of new tokens whose initialization is reported in the log.

`subword`#

Key	Default	Description
`subword.input_averaging`	`uniform`	`uniform`, `char_weighted`, `max_char`, `bert_weighted`, or `gemma_weighted` for input embedding rows.
`subword.output_averaging`	`uniform`	Same choices for language-model-head rows.
`subword.input_norm_correction`	`true`	Rescale new input rows to the median norm.
`subword.output_norm_correction`	`false`	Rescale new output rows. Enabling it risks language-model-head over-confidence and loss divergence.
`subword.input_target_norm`	`true`	Use the median norm of the target-script rows rather than of all rows. `input_hindi_norm` is accepted as an alias.
`subword.output_target_norm`	`false`	Same for output rows. `output_hindi_norm` is accepted as an alias.
`subword.temperature`	`0.1`	Softmax temperature; used only by `bert_weighted` and `gemma_weighted`.
`subword.bert_model`	unset	Encoder for `bert_weighted`. Leave unset so that `language` selects one that covers the language.
`subword.gemma_model`	`google/gemma-2-27b`	Decoder whose input embedding table `gemma_weighted` uses.
`subword.num_samples`	`10`	Number of new tokens reported in the log.

`focus`#

Key	Default	Description
`focus.fasttext_model`	`null`	Path to a fastText `.bin` file for the target language. Optional. When the path is unset, the step downloads the `cc.<code>.300.bin` vectors for the resolved `language` into `FASTTEXT_CACHE_DIR` on first use. Later runs reuse the downloaded vectors.
`focus.candidate_pool`	`target`	`target` restricts neighbors to the base model’s target-script rows; `all` uses every base row. `hindi` is accepted as an alias for `target`.
`focus.sparsemax_temperature`	`0.05`	Divisor applied to cosine similarities before Sparsemax.
`focus.num_samples`	`10`	Number of new tokens reported in the log.
`focus.top_contributors`	`5`	Number of contributing base tokens listed per reported token.