`bridge.data.builders.direct_hf_sft`#

Serializable config and runtime builder for direct Hugging Face SFT.

Module Contents#

Classes#

`DirectHFSFTDatasetConfig`	Serializable configuration for direct Hugging Face SFT datasets.
`DirectHFSFTDatasetBuilder`	Build runtime SFT datasets from declarative Hugging Face sources.

Functions#

`normalize_direct_hf_sft_processor`	Ensure the runtime tokenizer can pad batched conversation text.
`load_direct_hf_sft_processor`	Load the configured HF processor or adapt the training tokenizer.
`load_direct_hf_sft_examples`	Load and normalize one declarative Hugging Face source.
`select_direct_hf_sft_collate`	Select a shared text collator for the explicit preprocessing variant.
`build_direct_hf_sft_split`	Build one requested direct-HF SFT split.
`direct_hf_sft_train_valid_test_datasets_provider`	Build direct-HF SFT datasets through the canonical runtime builder.

Data#

`logger`
`CollateFunction`

API#

bridge.data.builders.direct_hf_sft.logger#: ‘getLogger(…)’

bridge.data.builders.direct_hf_sft.CollateFunction#: None

class bridge.data.builders.direct_hf_sft.DirectHFSFTDatasetConfig#

Bases: megatron.bridge.data.base.DataloaderConfig

Serializable configuration for direct Hugging Face SFT datasets.

Chat preprocessing is the compatibility default for multimodal and conversation sources. New text recipes should select chat or paired-text preprocessing explicitly.

seq_length: int#: None

source: megatron.bridge.data.sources.hf.HFDatasetSourceConfig#: None

validation_source: megatron.bridge.data.sources.hf.HFDatasetSourceConfig | None#: None

test_source: megatron.bridge.data.sources.hf.HFDatasetSourceConfig | None#: None

preprocessing: megatron.bridge.data.sft_processing.SFTPreprocessingConfig#: ‘field(…)’

hf_processor_path: str | None#: None

do_validation: bool#: True

do_test: bool#: True

skip_getting_attention_mask_from_dataset: bool#: True

dataloader_type: Literal[single, cyclic, batch, external] | None#: ‘single’

enable_in_batch_packing: bool#: False

defer_in_batch_packing_to_step: bool#: False

pad_to_max_length: bool#: False

pad_to_multiple_of: int#: 128

in_batch_packing_pad_to_multiple_of: int#: 1

validate() → None#: Validate declarative source and dataset settings.

_inherit_source_adapter_kwargs( split_source: megatron.bridge.data.sources.hf.HFDatasetSourceConfig, ) → None#: Fill unset adapter arguments on another split of the training source.

finalize() → None#: Finalize dataloader settings and validate this config.

bridge.data.builders.direct_hf_sft.normalize_direct_hf_sft_processor(processor: Any) → Any#: Ensure the runtime tokenizer can pad batched conversation text.

bridge.data.builders.direct_hf_sft.load_direct_hf_sft_processor( config: bridge.data.builders.direct_hf_sft.DirectHFSFTDatasetConfig, tokenizer: Any | None, ) → Any#: Load the configured HF processor or adapt the training tokenizer.

bridge.data.builders.direct_hf_sft.load_direct_hf_sft_examples( source: megatron.bridge.data.sources.hf.HFDatasetSourceConfig, preprocessing: megatron.bridge.data.sft_processing.SFTPreprocessingConfig, ) → list[dict[str, Any]]#: Load and normalize one declarative Hugging Face source.

bridge.data.builders.direct_hf_sft.select_direct_hf_sft_collate( examples: list[dict[str, Any]], preprocessing: megatron.bridge.data.sft_processing.SFTPreprocessingConfig | None = None, collate_impl: bridge.data.builders.direct_hf_sft.CollateFunction | None = None, ) → bridge.data.builders.direct_hf_sft.CollateFunction | None#: Select a shared text collator for the explicit preprocessing variant.

bridge.data.builders.direct_hf_sft.build_direct_hf_sft_split( config: bridge.data.builders.direct_hf_sft.DirectHFSFTDatasetConfig, source: megatron.bridge.data.sources.hf.HFDatasetSourceConfig, target_length: int, processor: Any, *, collate_impl: bridge.data.builders.direct_hf_sft.CollateFunction | None = None, ) → megatron.bridge.data.datasets.direct_sft.DirectSFTDataset | None#: Build one requested direct-HF SFT split.

class bridge.data.builders.direct_hf_sft.DirectHFSFTDatasetBuilder( config: bridge.data.builders.direct_hf_sft.DirectHFSFTDatasetConfig, *, collate_impl: bridge.data.builders.direct_hf_sft.CollateFunction | None = None, )#

Build runtime SFT datasets from declarative Hugging Face sources.

Initialization

build( context: megatron.bridge.data.base.DatasetBuildContext, ) → tuple[megatron.bridge.data.datasets.direct_sft.DirectSFTDataset | None, megatron.bridge.data.datasets.direct_sft.DirectSFTDataset | None, megatron.bridge.data.datasets.direct_sft.DirectSFTDataset | None]#: Build train, validation, and test datasets for requested sample counts.

bridge.data.builders.direct_hf_sft.direct_hf_sft_train_valid_test_datasets_provider( train_val_test_num_samples: list[int], dataset_config: bridge.data.builders.direct_hf_sft.DirectHFSFTDatasetConfig, tokenizer: megatron.bridge.training.tokenizers.tokenizer.MegatronTokenizer | None = None, pg_collection: megatron.core.process_groups_config.ProcessGroupCollection | None = None, ) → tuple[megatron.bridge.data.datasets.direct_sft.DirectSFTDataset | None, megatron.bridge.data.datasets.direct_sft.DirectSFTDataset | None, megatron.bridge.data.datasets.direct_sft.DirectSFTDataset | None]#: Build direct-HF SFT datasets through the canonical runtime builder.

bridge.data.builders.direct_hf_sft#