Run on a Cluster#

The bundled environment templates for Lepton, Slurm, and DGX Cloud each define four profiles for the tokenizer_extension steps, because the steps have different resource shapes. Pass a profile with --run for attached execution or --batch for detached execution.

Before You Begin#

Generate and edit an environment profile file as described in Nemotron Steps Basics:

$ uv run nemotron steps run env/env_toml -c slurm
$ export NEMOTRON_ENV_FILE=env.slurm.toml

Substitute -c lepton or -c dgxcloud for the other targets. Replace the site-specific values in the generated file before use. On Lepton, the profiles read credentials from platform secrets. Create the HF_TOKEN and WANDB_API_KEY secrets before the first submission. Refer to Prerequisites.

Profiles#

The generated profiles follow the pattern <target>_tokenizer_<step>.

Profile suffix

Step

Shape

Optional dependencies installed at start-up

_tokenizer_extend

extend

CPU only, 1 node

indic-nlp-library

_tokenizer_init_embeddings

init_embeddings

1 GPU, 1 node

fasttext-wheel, indic-nlp-library

_tokenizer_evaluate

evaluate

CPU only, 1 node

none

_tokenizer_eval_init

eval_init

1 GPU, 1 node

accelerate

The CPU profiles select a CPU partition or CPU resource shape; the extend trainer holds the whole word-count table in memory, so choose a large-memory shape for a large corpus. The init_embeddings profile and the eval_init profile each request one GPU. One GPU is enough for every engine except subword.gemma_weighted on a large Gemma model. On Lepton, LEPTON_INIT_SHAPE sets the GPU shape for init_embeddings and LEPTON_BPB_SHAPE sets the GPU shape for eval_init. A profile’s pip_extras list replaces the base profile’s list. It does not extend the base list. When you edit a pip_extras list, restate the base packages.

The extend and init_embeddings profiles set TOKEXT_OUTPUT_DIR to a shared workspace path on all three targets, and lepton_tokenizer_eval_init sets it as well. The evaluate profiles do not set it. Pass the tokenizer path explicitly with those profiles. The init_embeddings profile sets FASTTEXT_CACHE_DIR, the directory where focus caches downloaded cc.<code>.300.bin vectors so that a second run reuses them.

Run the Pipeline Detached#

The following commands use the Slurm profile names; replace the slurm_ prefix with lepton_ or dgxcloud_ for the other targets.

$ L=hindi
$ OUT=/lustre/$USER/output/tokenizer_extension   # shared Slurm output path

$ uv run nemotron steps run tokenizer_extension/extend \
    -b slurm_tokenizer_extend -c default \
    language=$L method=add extension_size=30000 output_dir=$OUT

$ uv run nemotron steps run tokenizer_extension/init_embeddings \
    -b slurm_tokenizer_init_embeddings -c default \
    language=$L arm=add extended_tokenizer=$OUT/add \
    output_dir=$OUT/resized_checkpoint

$ uv run nemotron steps run tokenizer_extension/evaluate \
    -b slurm_tokenizer_evaluate -c default tokenizer=$OUT/add

$ uv run nemotron steps run tokenizer_extension/eval_init \
    -b slurm_tokenizer_eval_init -c default \
    models=[$OUT/resized_checkpoint] max_docs=2000

Paths passed as overrides must be visible to the compute node. Use the shared workspace that the profile’s TOKEXT_OUTPUT_DIR points at rather than a path on the submitting host. In the submitting shell, set OUT to the same shared path as the profile’s TOKEXT_OUTPUT_DIR.

Confirm the Merged Configuration First#

Add --dry-run to compile the configuration without submitting a job:

$ uv run nemotron steps run tokenizer_extension/extend -d -c default language=$L method=add