nemo_automodel.components.datasets.llm.mock_packed

Module Contents

Classes

Name	Description
`MockPackedDatasetConfig`	Construction-time configuration for the mock packed dataset (tokenizer is a build arg).

Functions

Name	Description
`build_packed_dataset`	Dataset builder.
`flush_block`	Flush helper (build position_ids that reset after <eos>).
`gen_sentence_ids`	Sentence generator with Gaussian length control.
`make_vocab`	Build a trivial vocab; index 0=<pad>, 1=<eos>, rest = word_i.

Data

ds

API

class nemo_automodel.components.datasets.llm.mock_packed.MockPackedDatasetConfig(
    num_blocks: int = 10,
    block_size: int = 128,
    mean_len: float = 20.0,
    std_len: float = 6.0,
    vocab_size: int = 100,
    max_sentence_len: int = 64,
    seed: int = 0
)

Dataclass

Construction-time configuration for the mock packed dataset (tokenizer is a build arg).

block_size

int = 128

Token length of each packed block.

max_sentence_len

int = 64

Maximum sentence length.

mean_len

float = 20.0

Mean sentence length (Gaussian).

num_blocks

int = 10

Number of packed blocks to generate.

seed

int = 0

Seed for the random generator.

std_len

float = 6.0

Standard deviation of sentence length (Gaussian).

vocab_size

int = 100

Vocabulary size for the synthetic tokens.

nemo_automodel.components.datasets.llm.mock_packed.MockPackedDatasetConfig.build(
    tokenizer: 'PreTrainedTokenizerBase | None' = None
) -> datasets.Dataset

Build the mock packed :class:~datasets.Dataset from this :class:MockPackedDatasetConfig.

nemo_automodel.components.datasets.llm.mock_packed.build_packed_dataset(
    num_blocks: int = 10,
    block_size: int = 128,
    mean_len: float = 20.0,
    std_len: float = 6.0,
    vocab_size: int = 100,
    max_sentence_len: int = 64,
    seed: int = 0,
    tokenizer = None
)

Dataset builder.

nemo_automodel.components.datasets.llm.mock_packed.flush_block(
    block,
    block_size
)

Flush helper (build position_ids that reset after <eos>).

nemo_automodel.components.datasets.llm.mock_packed.gen_sentence_ids(
    vocab,
    mean_len: float,
    std_len: float,
    max_len: int
)

Sentence generator with Gaussian length control.

nemo_automodel.components.datasets.llm.mock_packed.make_vocab(
    vocab_size: int = 100
)

Build a trivial vocab; index 0=<pad>, 1=<eos>, rest = word_i.

nemo_automodel.components.datasets.llm.mock_packed.ds = build_packed_dataset(num_blocks=3, block_size=32, mean_len=10, std_len=3, vocab_...