> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For full documentation content, see https://docs.nvidia.com/nemo/curator/llms-full.txt.
# nemo_curator.stages.text.io.writer.megatron_tokenizer
## Module Contents
### Classes
| Name | Description |
| ----------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| [`MegatronTokenizerWriter`](#nemo_curator-stages-text-io-writer-megatron_tokenizer-MegatronTokenizerWriter) | Writer that writes a DocumentBatch to Megatron ready tokenized files. |
### Data
[`_INDEX_HEADER`](#nemo_curator-stages-text-io-writer-megatron_tokenizer-_INDEX_HEADER)
### API
```python
class nemo_curator.stages.text.io.writer.megatron_tokenizer.MegatronTokenizerWriter(
path: str,
file_extension: list[str] = (lambda: FILETYPE_TO_DEFAUL...,
write_kwargs: dict[str, typing.Any] = dict(),
fields: list[str] | None = None,
name: str = 'megatron_tokenizer_writer',
mode: typing.Literal['ignore', 'overwrite', 'append', 'error'] = 'ignore',
append_mode_implemented: bool = False,
model_identifier: str | None = None,
cache_dir: str | None = None,
hf_token: str | None = None,
text_field: str = 'text',
tokenization_batch_size: int = 1000,
append_eod: bool = False,
transformers_init_kwargs: dict[str, typing.Any] = dict()
)
```
Dataclass
**Bases:** [BaseWriter](/nemo-curator/nemo_curator/stages/text/io/writer/base#nemo_curator-stages-text-io-writer-base-BaseWriter)
Writer that writes a DocumentBatch to Megatron ready tokenized files.
```python
nemo_curator.stages.text.io.writer.megatron_tokenizer.MegatronTokenizerWriter.__post_init__()
```
```python
nemo_curator.stages.text.io.writer.megatron_tokenizer.MegatronTokenizerWriter._sequence_pointers(
sequence_lengths: list[int],
token_size: int
) -> list[int]
```
staticmethod
Build the sequence pointers per the sequence lengths and dtype size
Returns:
list\[int]: The pointer to the beginning of each sequence
**Parameters:**
The length of each sequence
The size of each token in bytes
```python
nemo_curator.stages.text.io.writer.megatron_tokenizer.MegatronTokenizerWriter.process(
task: nemo_curator.tasks.DocumentBatch
) -> nemo_curator.tasks.FileGroupTask
```
```python
nemo_curator.stages.text.io.writer.megatron_tokenizer.MegatronTokenizerWriter.setup(
_worker_metadata: nemo_curator.backends.base.WorkerMetadata | None = None
) -> None
```
```python
nemo_curator.stages.text.io.writer.megatron_tokenizer.MegatronTokenizerWriter.setup_on_node(
_node_info: nemo_curator.backends.base.NodeInfo | None = None,
_worker_metadata: nemo_curator.backends.base.WorkerMetadata = None
) -> None
```
```python
nemo_curator.stages.text.io.writer.megatron_tokenizer.MegatronTokenizerWriter.write_data(
bin_file: typing.BinaryIO,
token_dtype: numpy.dtype,
eod_token_id: int,
tokens_batch: list[list[int]],
sequence_lengths: list[int]
) -> None
```
Write tokens to the .bin file
Args:
tokens\_batch (list\[list\[int]]): The batch of tokens to write
```python
nemo_curator.stages.text.io.writer.megatron_tokenizer.MegatronTokenizerWriter.write_idx_data(
file_prefix: str,
token_size: int,
token_dtype_code: int,
sequence_lengths: list[int]
) -> None
```
Write the .idx file data