bridge.training.tokenizers.tokenizer

Megatron tokenizers.

Module Contents

Functions

_resolve_chat_template	Resolve the effective chat template from config.
_is_local_tokenizer_path	Return whether a tokenizer reference is explicitly a local path.
_resolve_hf_tokenizer_revision	Resolve an immutable Hugging Face tokenizer revision without mutating persisted config.
build_tokenizer	Initialize tokenizer from megatron.core.tokenizers based on the provided configuration.

Data

_HF_TOKENIZER_SNAPSHOT_ALLOW_PATTERNS
_HF_MODEL_WEIGHT_IGNORE_PATTERNS

API

bridge.training.tokenizers.tokenizer._resolve_chat_template(

config: megatron.bridge.training.tokenizers.config.TokenizerConfig,

) → Optional[str]

Resolve the effective chat template from config.

Returns chat_template verbatim when set, or the contents of the file at chat_template_path (local path or msc:// URL) otherwise.

Parameters:

config – Tokenizer configuration.

Returns:

The chat template string, or None when neither field is set.

Raises:

ValueError – If both chat_template and chat_template_path are set.

bridge.training.tokenizers.tokenizer._HF_TOKENIZER_SNAPSHOT_ALLOW_PATTERNS

(‘config.json’, ‘tokenizer*’, ‘special_tokens_map.json’, ‘added_tokens.json’, ‘vocab*’, ‘merges*’, ‘…

bridge.training.tokenizers.tokenizer._HF_MODEL_WEIGHT_IGNORE_PATTERNS

(’.safetensors’, ‘.bin’, ‘.pt’, ‘.pth’, ‘.ckpt’, ‘.h5’, ‘.msgpack’, ‘.gguf’, ‘.onnx’, ‘.np…

bridge.training.tokenizers.tokenizer._is_local_tokenizer_path(tokenizer_model: object) → bool

Return whether a tokenizer reference is explicitly a local path.

bridge.training.tokenizers.tokenizer._resolve_hf_tokenizer_revision(

config: megatron.bridge.training.tokenizers.config.TokenizerConfig,

) → megatron.bridge.training.tokenizers.config.TokenizerConfig

Resolve an immutable Hugging Face tokenizer revision without mutating persisted config.

bridge.training.tokenizers.tokenizer.build_tokenizer(

config: megatron.bridge.training.tokenizers.config.TokenizerConfig,

**kwargs,

) → megatron.core.tokenizers.MegatronTokenizer

Initialize tokenizer from megatron.core.tokenizers based on the provided configuration.

Parameters:

config (TokenizerConfig) – Configuration object specifying the tokenizer type, paths to vocab/model files, and other tokenizer-specific settings.

Returns:

An instance of the initialized tokenizer.

Return type:

MegatronTokenizer