> This page is for version Nightly (default).
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.models.beam.tokenization

Beam tokenizer: render JSON-string tool-call arguments as the dicts Beam's chat template requires.

Beam's chat template renders every tool-call argument as its own `<rfl:parameter>` block, so it needs
`function.arguments` as a mapping and raises on strings. Automodel's chat datasets store arguments as JSON
strings (the OpenAI wire format, and the only form Arrow-backed datasets keep intact). `BeamTokenizer` loads
the regular tokenizer and wraps that instance's `apply_chat_template` to parse such strings back into dicts
first, which is also what the serving stacks pass to the same template. Other models are unaffected: the
wrapper is registered for `model_type == "beam"` only, and the tokenizer class (and hence `save_pretrained`
output) is unchanged.

## Module Contents

### Classes

| Name                                                                                 | Description                                                                     |
| ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- |
| [`BeamTokenizer`](#nemo_automodel-components-models-beam-tokenization-BeamTokenizer) | Tokenizer factory registered for `model_type == "beam"` in `TokenizerRegistry`. |

### Functions

| Name                                                                                                                                       | Description                                                                                        |
| ------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------- |
| [`_apply_chat_template_with_dict_arguments`](#nemo_automodel-components-models-beam-tokenization-_apply_chat_template_with_dict_arguments) | -                                                                                                  |
| [`_message_with_dict_arguments`](#nemo_automodel-components-models-beam-tokenization-_message_with_dict_arguments)                         | Return `message` with JSON-object string arguments parsed; the input is never mutated.             |
| [`tool_call_arguments_as_dicts`](#nemo_automodel-components-models-beam-tokenization-tool_call_arguments_as_dicts)                         | Parse JSON-string tool-call arguments into dicts for one conversation or a batch of conversations. |
| [`wrap_tokenizer`](#nemo_automodel-components-models-beam-tokenization-wrap_tokenizer)                                                     | Make `tokenizer.apply_chat_template` accept JSON-string tool-call arguments.                       |

### API

```python
class nemo_automodel.components.models.beam.tokenization.BeamTokenizer()
```

Tokenizer factory registered for `model_type == "beam"` in `TokenizerRegistry`.

```python
nemo_automodel.components.models.beam.tokenization.BeamTokenizer.from_pretrained(
    pretrained_model_name_or_path: str | os.PathLike[str],
    args: object = (),
    kwargs: object = {}
) -> transformers.PreTrainedTokenizerBase
```

classmethod

Load the tokenizer exactly as the default path does, then wrap `apply_chat_template`.

```python
nemo_automodel.components.models.beam.tokenization._apply_chat_template_with_dict_arguments(
    tokenizer,
    conversation,
    args = (),
    kwargs = {}
)
```

```python
nemo_automodel.components.models.beam.tokenization._message_with_dict_arguments(
    message: typing.Any
) -> typing.Any
```

Return `message` with JSON-object string arguments parsed; the input is never mutated.

```python
nemo_automodel.components.models.beam.tokenization.tool_call_arguments_as_dicts(
    conversation: typing.Any
) -> typing.Any
```

Parse JSON-string tool-call arguments into dicts for one conversation or a batch of conversations.

```python
nemo_automodel.components.models.beam.tokenization.wrap_tokenizer(
    tokenizer: transformers.PreTrainedTokenizerBase
) -> transformers.PreTrainedTokenizerBase
```

Make `tokenizer.apply_chat_template` accept JSON-string tool-call arguments.

The wrapper is an instance attribute, so the tokenizer class (and its `save_pretrained` output and
pickling) stays the one the default loader picked. A `functools.partial` over a module-level function,
not a closure, keeps the tokenizer picklable and deep-copyable.