`bridge.data.sources.hf`#

Declarative Hugging Face sources, named presets, and shared normalization.

Module Contents#

Classes#

`HFDatasetSourceConfig`	Serializable source selection for one Hugging Face dataset split.
`_HFDatasetPreset`	Resolved physical source metadata for a built-in dataset.

Functions#

`_has_unset_data_file`	Return whether a declarative `data_files` value contains no usable path.
`_resolve_preset_split`
`hf_dataset_supports_split`	Return whether a source declares support for a logical split.
`resolve_hf_dataset_source`	Resolve a named preset or custom source to complete physical metadata.
`load_hf_dataset_source`	Load one declarative Hugging Face source without adapting its rows.
`prepare_hf_dataset_sources`	Materialize Hugging Face caches once before distributed readers start.
`load_and_adapt_hf_dataset`	Load a Hugging Face split and normalize it to canonical SFT rows.

Data#

_HF_DATASET_PRESETS

API#

bridge.data.sources.hf._has_unset_data_file(value: Any) → bool#: Return whether a declarative data_files value contains no usable path.

class bridge.data.sources.hf.HFDatasetSourceConfig#

Serializable source selection for one Hugging Face dataset split.

Exactly one source mode is required. dataset_name selects a built-in dataset preset that owns its Hub path, subset, and schema adapter. path_or_dataset selects a custom source; schema_adapter is optional when its rows already match the selected chat or prompt-completion preprocessing schema.

dataset_name: str | None#: None

path_or_dataset: str | None#: None

split: str | None#: None

subset: str | list[str] | None#: None

load_kwargs: dict[str, Any] | None#: None

schema_adapter: str | None#: None

adapter_kwargs: dict[str, Any] | None#: None

validate() → None#: Validate declarative source and adapter settings.

with_split(split: str) → bridge.data.sources.hf.HFDatasetSourceConfig#: Return a copy selecting another split expression.

class bridge.data.sources.hf._HFDatasetPreset#

Resolved physical source metadata for a built-in dataset.

path_or_dataset: str#: None

schema_adapter: str | None#: None

split: str#: ‘train’

subset: str | list[str] | None#: None

load_kwargs: dict[str, Any] | None#: None

adapter_kwargs: dict[str, Any] | None#: None

required_adapter_kwargs: tuple[str, ...]#: ()

split_aliases: dict[str, str] | None#: None

supported_splits: tuple[str, ...] | None#: None

bridge.data.sources.hf._HF_DATASET_PRESETS: dict[str, bridge.data.sources.hf._HFDatasetPreset]#: None

bridge.data.sources.hf._resolve_preset_split( dataset_name: str, preset: bridge.data.sources.hf._HFDatasetPreset, split: str, ) → str#

bridge.data.sources.hf.hf_dataset_supports_split( source: bridge.data.sources.hf.HFDatasetSourceConfig, split: str, ) → bool#: Return whether a source declares support for a logical split.

bridge.data.sources.hf.resolve_hf_dataset_source( source: bridge.data.sources.hf.HFDatasetSourceConfig, ) → bridge.data.sources.hf.HFDatasetSourceConfig#: Resolve a named preset or custom source to complete physical metadata.

bridge.data.sources.hf.load_hf_dataset_source( source: bridge.data.sources.hf.HFDatasetSourceConfig, ) → Any#: Load one declarative Hugging Face source without adapting its rows.

bridge.data.sources.hf.prepare_hf_dataset_sources( sources: collections.abc.Sequence[bridge.data.sources.hf.HFDatasetSourceConfig], ) → None#

Materialize Hugging Face caches once before distributed readers start.

Hugging Face cache creation is not reliable when multiple distributed ranks concurrently build the same source on a shared filesystem. Rank zero loads each requested source first, then broadcasts completion so every rank reads an already-stable cache. Single-process callers need no preparation because their normal load has no competing writer.

bridge.data.sources.hf.load_and_adapt_hf_dataset( source: bridge.data.sources.hf.HFDatasetSourceConfig, ) → list[dict[str, Any]]#: Load a Hugging Face split and normalize it to canonical SFT rows.

bridge.data.sources.hf#