> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.stages.math.modifiers.chunking

## Module Contents

### Classes

| Name                                                                                    | Description                                                                |
| --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| [`TokenSplitterStage`](#nemo_curator-stages-math-modifiers-chunking-TokenSplitterStage) | Token-based text chunking stage that splits long texts into smaller chunks |

### API

```python
class nemo_curator.stages.math.modifiers.chunking.TokenSplitterStage(
    model_name: str,
    max_length_tokens: int = 8000,
    separator: str = '\n\n',
    text_field: str = 'text',
    chunk_id_field: str = 'chunk_id',
    n_tokens_field: str = 'n_tokens'
)
```

**Bases:** [ProcessingStage\[DocumentBatch, DocumentBatch\]](/nemo-curator/nemo_curator/stages/base#nemo_curator-stages-base-ProcessingStage)

Token-based text chunking stage that splits long texts into smaller chunks
while preserving paragraph boundaries.

**`name`**

---

```python
nemo_curator.stages.math.modifiers.chunking.TokenSplitterStage.inputs() -> tuple[list[str], list[str]]
```

```python
nemo_curator.stages.math.modifiers.chunking.TokenSplitterStage.outputs() -> tuple[list[str], list[str]]
```

```python
nemo_curator.stages.math.modifiers.chunking.TokenSplitterStage.process(
    batch: nemo_curator.tasks.DocumentBatch
) -> nemo_curator.tasks.DocumentBatch
```

Process a batch of documents and split them into token-based chunks.

```python
nemo_curator.stages.math.modifiers.chunking.TokenSplitterStage.setup(
    _worker_metadata: nemo_curator.backends.base.WorkerMetadata | None = None
) -> None
```

Load tokenizer from local cache per worker.

```python
nemo_curator.stages.math.modifiers.chunking.TokenSplitterStage.setup_on_node(
    _node_info: nemo_curator.backends.base.NodeInfo | None = None,
    _worker_metadata: nemo_curator.backends.base.WorkerMetadata | None = None
) -> None
```

Download model weights to local cache once per physical node.