Adapting Nemotron to a New Language#

This page maps the Nemotron step categories onto a single workflow: adapting an English-centric Nemotron base model to a target language, region, or domain, from tokenizer through evaluation. Each stage is documented in its own section; this page explains how the stages connect and which artifact each one hands to the next.

Workflow at a Glance#

        flowchart LR
    tok["tokenizer_extension/*<br/>checkpoint_hf"] --> cpt
    cur["curate/nemo_curator<br/>filtered_jsonl"] --> prep["data_prep/pretrain_prep<br/>binidx"]
    prep --> cpt["pretrain/megatron_bridge<br/>checkpoint_megatron"]
    cpt --> sft_mb["sft/megatron_bridge<br/>checkpoint_megatron"]
    sdg["sdg/persona_mcq<br/>training_jsonl"] --> pack["data_prep/sft_packing<br/>packed_parquet"]
    pack --> sft_mb
    sdg --> sft_auto["sft/automodel<br/>checkpoint_hf"]
    sft_auto --> convert["convert/hf_to_megatron<br/>checkpoint_megatron"]
    sft_mb --> ev["eval/model_eval<br/>eval_results"]
    convert --> ev

Steps communicate through typed artifacts declared in each step.toml; the label under each node is the artifact the step produces. A stage can be skipped when its input already exists: for example, start at pretrain/megatron_bridge when the base tokenizer already covers the target script, or start at sft/* when a continued-pretraining checkpoint is already available.

Stages#

Stage	Step category	Consumes	Produces	Documentation
Extend the tokenizer	`tokenizer_extension/extend`, `init_embeddings`, `evaluate`, `eval_init`	Base tokenizer and Hugging Face checkpoint, target-language corpus	Extended tokenizer, resized Hugging Face checkpoint, fertility and bits-per-byte reports	Tokenizer Extension With Nemotron
Curate the corpus	`curate/nemo_curator`	JSONL text or a Hugging Face dataset snapshot	`filtered_jsonl`	About Data Curation
Prepare pretraining data	`data_prep/pretrain_prep`	`filtered_jsonl`	`binidx`	Data Preparation Module
Continue pretraining	`pretrain/megatron_bridge`	`binidx`, resized checkpoint as `hf_model_path`	`checkpoint_megatron`	Model Training with Nemotron Steps
Generate SFT data	`sdg/persona_mcq`, `sdg/data_designer`	Persona locales and teacher-model endpoints	`training_jsonl` and blend manifests	Generate Persona-Grounded MCQ Data for SFT, About Synthetic Data Generation
Pack SFT data	`data_prep/sft_packing`	`training_jsonl`	`packed_parquet`	Data Preparation Module
Fine-tune with Megatron Bridge	`sft/megatron_bridge`	`packed_parquet`, `checkpoint_megatron`	`checkpoint_megatron`	Choose an SFT Backend
Fine-tune with AutoModel	`sft/automodel`	`training_jsonl`	`checkpoint_hf`	Run SFT with AutoModel on Custom Data
Convert an AutoModel checkpoint	`convert/hf_to_megatron`	`checkpoint_hf`	`checkpoint_megatron`	Convert Checkpoints Between Training Steps
Build benchmarks	`byob/mcq`	Source corpus or an existing benchmark	`mcq_benchmark_parquet`	About Building Multiple-Choice Question Benchmarks
Translate a corpus	`translate/nemo_curator`	`filtered_jsonl`	`translated_jsonl`	Translation With Nemotron
Evaluate	`eval/model_eval`	`checkpoint_megatron` or a hosted endpoint, benchmark tasks	`eval_results`	About Model Evaluation

How the Stages Connect#

Tokenizer extension precedes continued pretraining. tokenizer_extension/extend adds target-script subwords to the base tokenizer and init_embeddings writes a resized Hugging Face checkpoint whose new embedding rows are initialized from existing subwords. pretrain/megatron_bridge loads that checkpoint through hf_model_path and trains the new rows on the curated corpus. tokenizer_extension/evaluate reports fertility (tokens per word) before any GPU is used, and eval_init reports bits per byte on the resized checkpoint, which is the metric that remains comparable across vocabularies.

Curation and data preparation feed continued pretraining. curate/nemo_curator applies language identification, word-count, and domain filters to JSONL shards and writes filtered_jsonl. data_prep/pretrain_prep tokenizes that output into Megatron binidx files with the extended tokenizer, so the pretraining corpus is tokenized with the same vocabulary the model will train on.

Synthetic data supplies the target-language SFT set. sdg/persona_mcq authors multiple-choice questions grounded in regional personas, deduplicates them lexically and semantically, and keeps only questions on which a panel of three or more teacher models agrees. Agreement is a consistency filter, not factual verification; validate a sample before training. The step writes one aligned train.jsonl and blend.json per teacher, which sft/automodel reads directly and data_prep/sft_packing packs for sft/megatron_bridge. sft/megatron_bridge produces checkpoint_megatron directly; convert the checkpoint_hf produced by sft/automodel with convert/hf_to_megatron before checkpoint-based evaluation.

Benchmarks are built independently of training. byob/mcq generates multiple-choice benchmarks from a source corpus and can translate existing benchmarks; translate/nemo_curator translates JSONL or Parquet corpora with NeMo Curator backends. Keep benchmark sources disjoint from the SFT data so that benchmark results measure generalization rather than memorization.

Choosing an Entry Point#

Situation	Start at
The base tokenizer produces high fertility on the target language	Getting Started With Tokenizer Extension
The tokenizer is adequate but the model lacks target-language knowledge	About Data Curation, then Model Training with Nemotron Steps
A continued-pretraining checkpoint exists and needs instruction data	Generate Persona-Grounded MCQ Data for SFT
A target-language benchmark needs to be generated	About Building Multiple-Choice Question Benchmarks
A fine-tuned model needs target-language evaluation	About Model Evaluation

Limitations#

The workflow is assembled from independently documented steps; there is no single command that runs all stages.
sdg/persona_mcq depends on managed Data Designer persona locales; languages without a managed locale reuse a related locale, as the shipped Malayalam configuration reuses en_IN.
Cluster environment profiles (env.toml) are per step; refer to each section’s cluster guide for the profile names.