> This page is for version Nightly (default).
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.shared.tokenizer_serialization

Serialization cleanup shared by portable tokenizer exporters.

## Module Contents

### Functions

| Name                                                                                                                                          | Description                                                           |
| --------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| [`_write_json`](#nemo_automodel-shared-tokenizer_serialization-_write_json)                                                                   | Write tokenizer metadata as formatted JSON.                           |
| [`restore_source_tokenizer_serialization_state`](#nemo_automodel-shared-tokenizer_serialization-restore_source_tokenizer_serialization_state) | Remove training-time tokenizer state while retaining tokenizer edits. |

### API

```python
nemo_automodel.shared.tokenizer_serialization._write_json(
    path: str,
    contents: dict
) -> None
```

Write tokenizer metadata as formatted JSON.

```python
nemo_automodel.shared.tokenizer_serialization.restore_source_tokenizer_serialization_state(
    original_model_path: str | None,
    hf_metadata_dir: str,
    tokenizer: transformers.PreTrainedTokenizerBase
) -> None
```

Remove training-time tokenizer state while retaining tokenizer edits.

**Parameters:**

**`original_model_path`** `str | None`

Local source snapshot, or None to clear transient state.

---

**`hf_metadata_dir`** `str`

Export directory containing the serialized tokenizer.

---

**`tokenizer`** `PreTrainedTokenizerBase`

Tokenizer supplying the current padding token.

---