bridge.training.tokenizers.config

Module Contents

Classes

TokenizerConfig

Configuration settings for tokenizers.

API

class bridge.training.tokenizers.config.TokenizerConfig

Bases: megatron.training.config.TokenizerConfig

Configuration settings for tokenizers.

make_vocab_size_divisible_by: int

1

Keep MCore tokenizer padding neutral; model providers apply vocab padding.

tensor_model_parallel_size: int

1

Tensor parallel size used by MCore tokenizer padded vocab-size calculation.

rank: int

0

Distributed rank used by MCore tokenizer helper logging.

use_tokenizer_vocab_size: bool

False

Use the runtime tokenizer vocabulary size for the model.

Enable this for from-scratch pretraining, where the tokenizer selected for the dataset defines the embedding and output vocabulary. Keep it disabled when model or checkpoint compatibility requires an explicitly configured model vocabulary size. This policy also applies during checkpoint loading; disable it and configure the checkpoint’s original model vocabulary when resuming a run created with a different vocabulary policy.

hf_tokenizer_kwargs: dict[str, Any] | None

‘field(…)’

Additional keyword arguments to pass to HuggingFace AutoTokenizer.from_pretrained.

Common options include: - use_fast (bool): Whether to use fast tokenizer implementation - trust_remote_code (bool): Whether to trust remote code when loading tokenizer - include_special_tokens (bool): Whether to include special tokens when converting text to ids - revision (str): Hugging Face Hub revision used to resolve an immutable tokenizer snapshot

.. rubric:: Example

hf_tokenizer_kwargs = { “use_fast”: True, “trust_remote_code”: True, “include_special_tokens”: True }

sp_tokenizer_kwargs: dict[str, Any] | None

‘field(…)’

Additional keyword arguments to pass to SentencePiece tokenizer.

Common options include: - legacy (bool): Whether to use legacy format of sentencepiece tokenizer

.. rubric:: Example

sp_tokenizer_kwargs = { “legacy”: True, }

chat_template_path: Optional[str]

None

Path to a jinja chat template file, loaded at build time as chat_template. Supports local paths and msc:// URLs. Mutually exclusive with chat_template. Useful for supplying a template from an external/process caller (e.g. CLI overrides) where inlining the jinja is impractical.

tokenizer_prompt_format: Optional[str]

None

Prompt format for the tokenizer.

image_tag_type: Optional[str]

None

Image tag to apply, if any. For example .

force_system_message: Optional[bool]

False

__post_init__() → None

Sync with MCore values