`bridge.data.builders.gpt_sft`#

Serializable config and runtime builder for text-only GPT SFT datasets.

Module Contents#

Classes#

`GPTSFTDatasetConfig`	Serializable configuration for text-only `GPTSFTDataset` construction.
`FinetuningDatasetConfig`	Deprecated compatibility name for :class:`GPTSFTDatasetConfig`.
`GPTSFTDatasetBuilder`	Runtime builder for :class:`GPTSFTDatasetConfig`.
`FinetuningDatasetBuilder`	Deprecated constructor-compatible adapter for :class:`GPTSFTDatasetBuilder`.

Functions#

`_default_gpt_sft_preprocessing`	Return preprocessing compatible with the established local JSONL schema.
`_packing_fingerprint`	Fingerprint every setting that can change builder-managed packed rows.
`resolve_gpt_sft_preprocessing`	Resolve explicit preprocessing or source-compatible legacy defaults.
`resolve_gpt_sft_dataset_root`	Resolve the local JSONL root for the configured source.
`normalize_gpt_sft_dataset_kwargs`	Return runtime dataset kwargs for the selected preprocessing variant.
`_load_hf_examples`
`_write_hf_examples`
`_hf_jsonl_index_paths`	Return memmap index sidecars associated with one materialized JSONL split.
`_remove_hf_materialized_split`	Remove one materialized split and any stale memmap index sidecars.
`_needs_hf_write`
`_materialize_hf_split`
`materialize_hf_dataset`	Materialize and normalize a Hugging Face source as JSONL splits.
`build_gpt_sft_split`	Build one GPT SFT split from a local JSONL or packed-data path.
`gpt_sft_train_valid_test_datasets_provider`	Build text-only SFT datasets through the canonical runtime builder.

Data#

`logger`
`_SEMANTIC_DATASET_KWARGS`

API#

bridge.data.builders.gpt_sft.logger#: ‘getLogger(…)’

bridge.data.builders.gpt_sft._SEMANTIC_DATASET_KWARGS#: None

bridge.data.builders.gpt_sft._default_gpt_sft_preprocessing() → megatron.bridge.data.sft_processing.PromptCompletionSFTPreprocessingConfig#: Return preprocessing compatible with the established local JSONL schema.

bridge.data.builders.gpt_sft._packing_fingerprint( config: GPTSFTDatasetConfig, dataset_kwargs: dict[str, Any] | None, ) → str#: Fingerprint every setting that can change builder-managed packed rows.

class bridge.data.builders.gpt_sft.GPTSFTDatasetConfig#

Bases: megatron.bridge.data.base.DataloaderConfig

Serializable configuration for text-only GPTSFTDataset construction.

Exactly one source is required: dataset_root selects existing local JSONL/packed artifacts, while hf_dataset selects a declarative Hugging Face source that is materialized before construction. New callers should set preprocessing explicitly. None preserves the established local prompt-completion and Hugging Face chat defaults for compatibility.

seq_length: int#: None

dataset_root: str | pathlib.Path | None#: None

hf_dataset: megatron.bridge.data.sources.hf.HFDatasetSourceConfig | None#: None

hf_validation_dataset: megatron.bridge.data.sources.hf.HFDatasetSourceConfig | None#: None

hf_test_dataset: megatron.bridge.data.sources.hf.HFDatasetSourceConfig | None#: None

hf_output_root: str | pathlib.Path | None#: None

hf_validation_proportion: float | None#: None

hf_rewrite: bool#: False

seed: int#: 1234

memmap_workers: int#: 1

max_train_samples: int | None#: None

preprocessing: megatron.bridge.data.sft_processing.SFTPreprocessingConfig | None#: None

enable_offline_packing: bool#: False

offline_packing_specs: megatron.bridge.data.packing.PackedSequenceSpecs | None#: None

dataset_kwargs: dict[str, Any] | None#: None

do_validation: bool#: True

do_test: bool#: True

dataloader_type: Literal[single, cyclic, batch, external] | None#: ‘batch’

validate() → None#: Validate source selection and text-only SFT settings.

finalize() → None#: Finalize dataloader settings and validate this config.

class bridge.data.builders.gpt_sft.FinetuningDatasetConfig#

Bases: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig

Deprecated compatibility name for :class:GPTSFTDatasetConfig.

__post_init__() → None#

bridge.data.builders.gpt_sft.resolve_gpt_sft_preprocessing( config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, ) → megatron.bridge.data.sft_processing.SFTPreprocessingConfig#: Resolve explicit preprocessing or source-compatible legacy defaults.

bridge.data.builders.gpt_sft.resolve_gpt_sft_dataset_root( config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, ) → str | pathlib.Path#: Resolve the local JSONL root for the configured source.

bridge.data.builders.gpt_sft.normalize_gpt_sft_dataset_kwargs( config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, ) → dict[str, Any]#: Return runtime dataset kwargs for the selected preprocessing variant.

bridge.data.builders.gpt_sft._load_hf_examples( source: megatron.bridge.data.sources.hf.HFDatasetSourceConfig, preprocessing: megatron.bridge.data.sft_processing.SFTPreprocessingConfig, ) → list[dict[str, Any]]#

bridge.data.builders.gpt_sft._write_hf_examples( root: pathlib.Path, output_name: str, examples: list[dict[str, Any]], ) → None#

bridge.data.builders.gpt_sft._hf_jsonl_index_paths( output_path: pathlib.Path, ) → tuple[pathlib.Path, pathlib.Path]#: Return memmap index sidecars associated with one materialized JSONL split.

bridge.data.builders.gpt_sft._remove_hf_materialized_split( root: pathlib.Path, output_name: str, ) → None#: Remove one materialized split and any stale memmap index sidecars.

bridge.data.builders.gpt_sft._needs_hf_write( config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, root: pathlib.Path, output_name: str, ) → bool#

bridge.data.builders.gpt_sft._materialize_hf_split( config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, source: megatron.bridge.data.sources.hf.HFDatasetSourceConfig, root: pathlib.Path, *, output_name: str, ) → None#

bridge.data.builders.gpt_sft.materialize_hf_dataset( config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, root: pathlib.Path, ) → None#: Materialize and normalize a Hugging Face source as JSONL splits.

bridge.data.builders.gpt_sft.build_gpt_sft_split( path: str | pathlib.Path, *, tokenizer: megatron.bridge.training.tokenizers.tokenizer.MegatronTokenizer, seq_length: int, memmap_workers: int, seed: int, packed_sequence_size: int, pack_metadata_path: str | pathlib.Path | None = None, pad_cu_seqlens: bool = False, pad_seq_to_mult: int | None = None, is_test: bool = False, dataset_kwargs: dict[str, Any] | None = None, ) → Any | None#: Build one GPT SFT split from a local JSONL or packed-data path.

class bridge.data.builders.gpt_sft.GPTSFTDatasetBuilder( config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, tokenizer: megatron.bridge.training.tokenizers.tokenizer.MegatronTokenizer, )#

Runtime builder for :class:GPTSFTDatasetConfig.

The config remains serializable and declarative. This builder resolves the selected source, performs any Hugging Face materialization or offline packing, and constructs the runtime GPT SFT datasets.

Parameters:

config – Serializable GPT SFT dataset configuration.
tokenizer – Tokenizer used to preprocess text.

Initialization

prepare_data() → None#

Materialize the selected source and prepare packed data if needed.

Call this entry point on one rank before dataset construction. It is also used by the standalone pre-packing script.

prepare_packed_data() → None#

Prepare packed sequence data files if configured.

Skips preparation if:

packed_sequence_size <= 0 (packing disabled)
packed data files already exist (parquet or legacy .npy), unless hf_rewrite requested regeneration

_prepare_packed_split( split_name: str, packed_path: Union[str, pathlib.Path], input_path: pathlib.Path, ) → None#

Prepare a single packed data split if it doesn’t already exist.

Parameters:

split_name – Name of the split (for logging).
packed_path – Output path for the packed data.
input_path – Input path to the raw dataset.

_packed_path_exists(path: Union[str, pathlib.Path]) → bool#

Check if a packed data path exists.

For .npy files: check file exists For packed parquet specs: check if resolution returns non-empty

Parameters:: path – The path to check
Returns:: True if the packed data exists

_remove_packed_path(path: Union[str, pathlib.Path]) → None#: Remove one builder-managed packed artifact if it exists.

build() → list[Optional[Any]]#

Build train, validation, and test datasets.

This method creates the necessary datasets based on the configuration. It first ensures data preparation (e.g., packing) is done (on rank 0), then builds the datasets potentially using the prepared files.

Returns:: A list containing the train, validation, and test datasets. Elements can be None if the corresponding data file doesn’t exist or if dataset building is skipped for the split.

_build_datasets() → list[Optional[Any]]#

Internal method to build all datasets.

Returns:: The train, validation, and test datasets.
Return type:: list[Optional[Any]]

property train_path: pathlib.Path#: Path to the training dataset file (training.jsonl).

property default_pack_path: pathlib.Path#

The default directory path for storing packed sequence files.

Constructed based on the dataset root and tokenizer model name. Creates the directory if it doesn’t exist.

Returns:: The Path object for the default packing directory.

property pack_metadata: pathlib.Path#

Path to the metadata file for packed sequences.

Determined by offline_packing_specs or defaults based on the default_pack_path and packed_sequence_size.

Returns:: The Path object for the packed sequence metadata file.
Raises:: ValueError – If packed sequences are not configured.

property train_path_packed: pathlib.Path#

Path to the packed training dataset file.

Determined by offline_packing_specs or defaults based on the default_pack_path and packed_sequence_size.

Returns:: The Path object for the packed training data file.
Raises:: ValueError – If packed sequences are not configured.

property validation_path_packed: pathlib.Path#

Path to the packed validation dataset file.

Determined by offline_packing_specs or defaults based on the default_pack_path and packed_sequence_size.

Returns:: The Path object for the packed validation data file.
Raises:: ValueError – If packed sequences are not configured.

property validation_path: pathlib.Path#: Path to the validation dataset file (validation.jsonl).

property test_path: pathlib.Path#: Path to the test dataset file (test.jsonl).

_extract_tokenizer_model_name() → str#: Automatically get the model name from model path.

bridge.data.builders.gpt_sft.gpt_sft_train_valid_test_datasets_provider( train_val_test_num_samples: list[int], dataset_config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, tokenizer: megatron.bridge.training.tokenizers.tokenizer.MegatronTokenizer | None = None, pg_collection: megatron.core.process_groups_config.ProcessGroupCollection | None = None, ) → tuple[Any | None, Any | None, Any | None]#: Build text-only SFT datasets through the canonical runtime builder.

class bridge.data.builders.gpt_sft.FinetuningDatasetBuilder( dataset_root: str | pathlib.Path, tokenizer: megatron.bridge.training.tokenizers.tokenizer.MegatronTokenizer, seq_length: int = 2048, seed: int = 1234, memmap_workers: int = 1, max_train_samples: int | None = None, enable_offline_packing: bool = False, offline_packing_specs: megatron.bridge.data.packing.PackedSequenceSpecs | None = None, dataset_kwargs: dict[str, Any] | None = None, do_validation: bool = True, do_test: bool = True, )#

Bases: bridge.data.builders.gpt_sft.GPTSFTDatasetBuilder

Deprecated constructor-compatible adapter for :class:GPTSFTDatasetBuilder.

Initialization

bridge.data.builders.gpt_sft#

Module Contents#

Classes#

Functions#

Data#

API#

`bridge.data.builders.gpt_sft`#