`bridge.data.packing.config`#

Declarative configuration for offline GPT SFT sequence packing.

Module Contents#

Classes#

PackedSequenceSpecs

Settings and optional artifact paths for offline sequence packing.

API#

class bridge.data.packing.config.PackedSequenceSpecs#

Settings and optional artifact paths for offline sequence packing.

packed_sequence_size: int#: None

tokenizer_model_name: str | None#: None

num_tokenizer_workers: int#: None

packed_train_data_path: str | pathlib.Path | None#: None

packed_val_data_path: str | pathlib.Path | None#: None

packed_metadata_path: str | pathlib.Path | None#: None

pad_cu_seqlens: bool#: False

pad_seq_to_mult: int | None#: 1

__post_init__() → None#: Validate alignment settings and any explicitly supplied artifacts.

_validate_packed_path( attr_name: str, path_value: str | pathlib.Path, ) → None#: Validate an explicitly supplied packed artifact path.