nemo_automodel.shared.tokenizer_serialization

Serialization cleanup shared by portable tokenizer exporters.

Module Contents

Functions

Name	Description
`_write_json`	Write tokenizer metadata as formatted JSON.
`restore_source_tokenizer_serialization_state`	Remove training-time tokenizer state while retaining tokenizer edits.

API

nemo_automodel.shared.tokenizer_serialization._write_json(
    path: str,
    contents: dict
) -> None

Write tokenizer metadata as formatted JSON.

nemo_automodel.shared.tokenizer_serialization.restore_source_tokenizer_serialization_state(
    original_model_path: str | None,
    hf_metadata_dir: str,
    tokenizer: transformers.PreTrainedTokenizerBase
) -> None

Remove training-time tokenizer state while retaining tokenizer edits.

Parameters:

original_model_path

str | None

Local source snapshot, or None to clear transient state.

hf_metadata_dir

str

Export directory containing the serialized tokenizer.

tokenizer

PreTrainedTokenizerBase

Tokenizer supplying the current padding token.