`bridge.data.builders.gpt_sft`#

Serializable config and runtime builder for text-only GPT SFT datasets.

Module Contents#

Classes#

`_GPTSFTDataBlend`	Resolved JSONL sources and optional relative weights for one split.
`GPTSFTDatasetConfig`	Serializable configuration for text-only `GPTSFTDataset` construction.
`FinetuningDatasetConfig`	Deprecated compatibility name for :class:`GPTSFTDatasetConfig`.
`GPTSFTDatasetBuilder`	Runtime builder for :class:`GPTSFTDatasetConfig`.
`FinetuningDatasetBuilder`	Deprecated constructor-compatible adapter for :class:`GPTSFTDatasetBuilder`.

Functions#

`_default_gpt_sft_preprocessing`	Return preprocessing compatible with the established local JSONL schema.
`_parse_gpt_sft_data_blend`	Parse paths; an even-length list with numeric even-index entries is weight/path pairs.
`_load_gpt_sft_data_blends`	Load and validate an MLM-compatible per-split SFT data source manifest.
`_gpt_sft_blend_identity`	Return cache identity from the per-split JSONL paths, sizes, and weights.
`_packing_fingerprint`	Fingerprint every setting that can change builder-managed packed rows.
`resolve_gpt_sft_preprocessing`	Resolve explicit preprocessing or source-compatible legacy defaults.
`resolve_gpt_sft_dataset_root`	Resolve the local artifact root for the configured source.
`normalize_gpt_sft_dataset_kwargs`	Return runtime dataset kwargs for the selected preprocessing variant.
`_load_hf_examples`
`_ensure_hf_output_root`	Create the HF output root despite transient shared-filesystem metadata races.
`_write_hf_examples`
`_hf_jsonl_index_paths`	Return memmap index sidecars associated with one materialized JSONL split.
`_remove_hf_materialized_split`	Remove one materialized split and any stale memmap index sidecars.
`_needs_hf_write`
`_materialize_hf_split`
`materialize_hf_dataset`	Materialize and normalize a Hugging Face source as JSONL splits.
`build_gpt_sft_split`	Build one GPT SFT split from a local JSONL or packed-data path.
`gpt_sft_train_valid_test_datasets_provider`	Build text-only SFT datasets through the canonical runtime builder.

Data#

`logger`
`_SHARED_FS_DIRECTORY_RESTAT_ATTEMPTS`
`_SHARED_FS_DIRECTORY_RESTAT_DELAY_S`
`_SEMANTIC_DATASET_KWARGS`

API#

bridge.data.builders.gpt_sft.logger#: ‘getLogger(…)’

bridge.data.builders.gpt_sft._SHARED_FS_DIRECTORY_RESTAT_ATTEMPTS#: 3

bridge.data.builders.gpt_sft._SHARED_FS_DIRECTORY_RESTAT_DELAY_S#: 0.1

bridge.data.builders.gpt_sft._SEMANTIC_DATASET_KWARGS#: None

bridge.data.builders.gpt_sft._default_gpt_sft_preprocessing() → megatron.bridge.data.sft_processing.PromptCompletionSFTPreprocessingConfig#: Return preprocessing compatible with the established local JSONL schema.

class bridge.data.builders.gpt_sft._GPTSFTDataBlend#

Resolved JSONL sources and optional relative weights for one split.

paths: tuple[str, ...]#: None

weights: tuple[float, ...] | None#: None

bridge.data.builders.gpt_sft._parse_gpt_sft_data_blend( value: Any, *, split_name: str, ) → bridge.data.builders.gpt_sft._GPTSFTDataBlend#: Parse paths; an even-length list with numeric even-index entries is weight/path pairs.

bridge.data.builders.gpt_sft._load_gpt_sft_data_blends( config: GPTSFTDatasetConfig, ) → dict[str, bridge.data.builders.gpt_sft._GPTSFTDataBlend]#: Load and validate an MLM-compatible per-split SFT data source manifest.

bridge.data.builders.gpt_sft._gpt_sft_blend_identity( config: GPTSFTDatasetConfig, ) → dict[str, Any] | None#: Return cache identity from the per-split JSONL paths, sizes, and weights.

bridge.data.builders.gpt_sft._packing_fingerprint( config: GPTSFTDatasetConfig, dataset_kwargs: dict[str, Any] | None, ) → str#: Fingerprint every setting that can change builder-managed packed rows.

class bridge.data.builders.gpt_sft.GPTSFTDatasetConfig#

Bases: megatron.bridge.data.base.DataloaderConfig

Serializable configuration for text-only GPTSFTDataset construction.

Exactly one source is required: dataset_root selects existing local JSONL/packed artifacts, per_split_data_source_manifest_path selects MLM-style JSONL blends, and hf_dataset selects a declarative Hugging Face source that is materialized before construction. New callers should set preprocessing explicitly. None preserves the established local prompt-completion and Hugging Face chat defaults for compatibility.

seq_length: int#: None

dataset_root: str | pathlib.Path | None#: None

per_split_data_source_manifest_path: str | pathlib.Path | None#: None

hf_dataset: megatron.bridge.data.sources.hf.HFDatasetSourceConfig | None#: None

hf_validation_dataset: megatron.bridge.data.sources.hf.HFDatasetSourceConfig | None#: None

hf_test_dataset: megatron.bridge.data.sources.hf.HFDatasetSourceConfig | None#: None

hf_output_root: str | pathlib.Path | None#: None

hf_validation_proportion: float | None#: None

hf_rewrite: bool#: False

seed: int#: 1234

memmap_workers: int#: 1

max_train_samples: int | None#: None

preprocessing: megatron.bridge.data.sft_processing.SFTPreprocessingConfig | None#: None

enable_in_batch_packing: bool#: False

in_batch_packing_pad_to_multiple_of: int#: 1

enable_offline_packing: bool#: False

offline_packing_specs: megatron.bridge.data.packing.PackedSequenceSpecs | None#: None

enable_global_batch_packing: bool#

False

Yield unpacked per-sample rows (identity collate) for Megatron-Core’s online packing scheduler, which packs the global batch per step across DP x CP ranks.

global_batch_packing_pad_to_multiple_of: int#

Per-sequence alignment multiple for CP THD slicing; ConfigContainer.validate sets it.

fold_alignment_padding: bool#

False

Report the alignment padding as part of each sequence (loss-masked) so packed bins carry no padding between sequences. Set it when attention runs on FlashAttention under context parallelism, e.g. head dimension 256 on Blackwell; see megatron.bridge.data.packing.global_batch.fold_alignment_padding.

dataset_kwargs: dict[str, Any] | None#: None

do_validation: bool#: True

do_test: bool#: True

dataloader_type: Literal[single, cyclic, batch, external] | None#: ‘batch’

property yields_unpacked_samples: bool#: Whether the built datasets yield the per-sample dicts global-batch packing consumes.

validate() → None#: Validate source selection and text-only SFT settings.

finalize() → None#: Finalize dataloader settings and validate this config.

class bridge.data.builders.gpt_sft.FinetuningDatasetConfig#

Bases: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig

Deprecated compatibility name for :class:GPTSFTDatasetConfig.

__post_init__() → None#

bridge.data.builders.gpt_sft.resolve_gpt_sft_preprocessing( config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, ) → megatron.bridge.data.sft_processing.SFTPreprocessingConfig#: Resolve explicit preprocessing or source-compatible legacy defaults.

bridge.data.builders.gpt_sft.resolve_gpt_sft_dataset_root( config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, ) → str | pathlib.Path#: Resolve the local artifact root for the configured source.

bridge.data.builders.gpt_sft.normalize_gpt_sft_dataset_kwargs( config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, ) → dict[str, Any]#: Return runtime dataset kwargs for the selected preprocessing variant.

bridge.data.builders.gpt_sft._load_hf_examples( source: megatron.bridge.data.sources.hf.HFDatasetSourceConfig, preprocessing: megatron.bridge.data.sft_processing.SFTPreprocessingConfig, ) → list[dict[str, Any]]#

bridge.data.builders.gpt_sft._ensure_hf_output_root(root: pathlib.Path) → None#: Create the HF output root despite transient shared-filesystem metadata races.

bridge.data.builders.gpt_sft._write_hf_examples( root: pathlib.Path, output_name: str, examples: list[dict[str, Any]], ) → None#

bridge.data.builders.gpt_sft._hf_jsonl_index_paths( output_path: pathlib.Path, ) → tuple[pathlib.Path, pathlib.Path]#: Return memmap index sidecars associated with one materialized JSONL split.

bridge.data.builders.gpt_sft._remove_hf_materialized_split( root: pathlib.Path, output_name: str, ) → None#: Remove one materialized split and any stale memmap index sidecars.

bridge.data.builders.gpt_sft._needs_hf_write( config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, root: pathlib.Path, output_name: str, ) → bool#

bridge.data.builders.gpt_sft._materialize_hf_split( config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, source: megatron.bridge.data.sources.hf.HFDatasetSourceConfig, root: pathlib.Path, *, output_name: str, ) → None#

bridge.data.builders.gpt_sft.materialize_hf_dataset( config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, root: pathlib.Path, ) → None#: Materialize and normalize a Hugging Face source as JSONL splits.

bridge.data.builders.gpt_sft.build_gpt_sft_split( path: str | pathlib.Path | bridge.data.builders.gpt_sft._GPTSFTDataBlend, *, tokenizer: megatron.bridge.training.tokenizers.tokenizer.MegatronTokenizer, seq_length: int, memmap_workers: int, seed: int, packed_sequence_size: int, pack_metadata_path: str | pathlib.Path | None = None, pad_cu_seqlens: bool = False, pad_seq_to_mult: int | None = None, enable_in_batch_packing: bool = False, in_batch_packing_pad_to_multiple_of: int = 1, enable_global_batch_packing: bool = False, global_batch_packing_pad_to_multiple_of: int = 1, fold_alignment_padding: bool = False, is_test: bool = False, dataset_kwargs: dict[str, Any] | None = None, ) → Any | None#: Build one GPT SFT split from a local JSONL or packed-data path.

class bridge.data.builders.gpt_sft.GPTSFTDatasetBuilder( config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, tokenizer: megatron.bridge.training.tokenizers.tokenizer.MegatronTokenizer, )#

Runtime builder for :class:GPTSFTDatasetConfig.

The config remains serializable and declarative. This builder resolves the selected source, performs any Hugging Face materialization or offline packing, and constructs the runtime GPT SFT datasets.

Parameters:

config – Serializable GPT SFT dataset configuration.
tokenizer – Tokenizer used to preprocess text.

Initialization

prepare_data() → None#

Materialize the selected source and prepare packed data if needed.

Call this entry point on one rank before dataset construction. It is also used by the standalone pre-packing script.

prepare_packed_data() → None#

Prepare packed sequence data files if configured.

Skips preparation if:

packed_sequence_size <= 0 (packing disabled)
packed data files already exist (parquet or legacy .npy), unless hf_rewrite requested regeneration

_prepare_packed_split( split_name: str, packed_path: Union[str, pathlib.Path], input_path: pathlib.Path | bridge.data.builders.gpt_sft._GPTSFTDataBlend, ) → None#

Prepare a single packed data split if it doesn’t already exist.

Parameters:

split_name – Name of the split (for logging).
packed_path – Output path for the packed data.
input_path – Input path to the raw dataset.

_packed_path_exists(path: Union[str, pathlib.Path]) → bool#

Check if a packed data path exists.

For .npy files: check file exists For packed parquet specs: check if resolution returns non-empty

Parameters:: path – The path to check
Returns:: True if the packed data exists

_remove_packed_path(path: Union[str, pathlib.Path]) → None#: Remove one builder-managed packed artifact if it exists.

build() → list[Optional[Any]]#

Build train, validation, and test datasets.

This method creates the necessary datasets based on the configuration. It first ensures data preparation (e.g., packing) is done (on rank 0), then builds the datasets potentially using the prepared files.

Returns:: A list containing the train, validation, and test datasets. Elements can be None if the corresponding data file doesn’t exist or if dataset building is skipped for the split.

_build_datasets() → list[Optional[Any]]#

Internal method to build all datasets.

Returns:: The train, validation, and test datasets.
Return type:: list[Optional[Any]]

property train_path: pathlib.Path | bridge.data.builders.gpt_sft._GPTSFTDataBlend#: Training JSONL path or resolved multi-source blend.

property default_pack_path: pathlib.Path#

The default directory path for storing packed sequence files.

Constructed based on the dataset root and tokenizer model name. Creates the directory if it doesn’t exist.

Returns:: The Path object for the default packing directory.

property pack_metadata: pathlib.Path#

Path to the metadata file for packed sequences.

Determined by offline_packing_specs or defaults based on the default_pack_path and packed_sequence_size.

Returns:: The Path object for the packed sequence metadata file.
Raises:: ValueError – If packed sequences are not configured.

property train_path_packed: pathlib.Path#

Path to the packed training dataset file.

Determined by offline_packing_specs or defaults based on the default_pack_path and packed_sequence_size.

Returns:: The Path object for the packed training data file.
Raises:: ValueError – If packed sequences are not configured.

property validation_path_packed: pathlib.Path#

Path to the packed validation dataset file.

Determined by offline_packing_specs or defaults based on the default_pack_path and packed_sequence_size.

Returns:: The Path object for the packed validation data file.
Raises:: ValueError – If packed sequences are not configured.

property validation_path: pathlib.Path | bridge.data.builders.gpt_sft._GPTSFTDataBlend#: Validation JSONL path or resolved multi-source blend.

property test_path: pathlib.Path | bridge.data.builders.gpt_sft._GPTSFTDataBlend#: Test JSONL path or resolved multi-source blend.

_extract_tokenizer_model_name() → str#: Automatically get the model name from model path.

bridge.data.builders.gpt_sft.gpt_sft_train_valid_test_datasets_provider( train_val_test_num_samples: list[int], dataset_config: bridge.data.builders.gpt_sft.GPTSFTDatasetConfig, tokenizer: megatron.bridge.training.tokenizers.tokenizer.MegatronTokenizer | None = None, pg_collection: megatron.core.process_groups_config.ProcessGroupCollection | None = None, ) → tuple[Any | None, Any | None, Any | None]#: Build text-only SFT datasets through the canonical runtime builder.

class bridge.data.builders.gpt_sft.FinetuningDatasetBuilder( dataset_root: str | pathlib.Path, tokenizer: megatron.bridge.training.tokenizers.tokenizer.MegatronTokenizer, seq_length: int = 2048, seed: int = 1234, memmap_workers: int = 1, max_train_samples: int | None = None, enable_offline_packing: bool = False, offline_packing_specs: megatron.bridge.data.packing.PackedSequenceSpecs | None = None, dataset_kwargs: dict[str, Any] | None = None, do_validation: bool = True, do_test: bool = True, )#

Bases: bridge.data.builders.gpt_sft.GPTSFTDatasetBuilder

Deprecated constructor-compatible adapter for :class:GPTSFTDatasetBuilder.

Initialization

bridge.data.builders.gpt_sft#

Module Contents#

Classes#

Functions#

Data#

API#

`bridge.data.builders.gpt_sft`#