bridge.data.sources.hf#

Declarative Hugging Face sources, named presets, and shared normalization.

Module Contents#

Classes#

HFDatasetSourceConfig

Serializable source selection for one Hugging Face dataset split.

_HFDatasetPreset

Resolved physical source metadata for a built-in dataset.

Functions#

_has_unset_data_file

Return whether a declarative data_files value contains no usable path.

_resolve_preset_split

hf_dataset_supports_split

Return whether a source declares support for a logical split.

resolve_hf_dataset_source

Resolve a named preset or custom source to complete physical metadata.

load_hf_dataset_source

Load one declarative Hugging Face source without adapting its rows.

prepare_hf_dataset_sources

Materialize Hugging Face caches once before distributed readers start.

load_and_adapt_hf_dataset

Load a Hugging Face split and normalize it to canonical SFT rows.

resolve_blend_weights

Validate blend weights against their sources and fill in the uniform default.

blend_sft_rows

Draw rows from several sources under per-source epoch counts.

load_and_blend_hf_datasets

Load several Hugging Face splits and mix them under per-source epoch counts.

Data#

_HF_DATASET_PRESETS

API#

bridge.data.sources.hf._has_unset_data_file(value: Any) → bool#

Return whether a declarative data_files value contains no usable path.

class bridge.data.sources.hf.HFDatasetSourceConfig#

Serializable source selection for one Hugging Face dataset split.

Exactly one source mode is required. dataset_name selects a built-in dataset preset that owns its Hub path, subset, and schema adapter. path_or_dataset selects a custom source; schema_adapter is optional when its rows already match the selected chat or prompt-completion preprocessing schema.

dataset_name: str | None#

None

path_or_dataset: str | None#

None

split: str | None#

None

subset: str | list[str] | None#

None

load_kwargs: dict[str, Any] | None#

None

schema_adapter: str | None#

None

adapter_kwargs: dict[str, Any] | None#

None

validate() → None#

Validate declarative source and adapter settings.

with_split(split: str) → bridge.data.sources.hf.HFDatasetSourceConfig#

Return a copy selecting another split expression.

class bridge.data.sources.hf._HFDatasetPreset#

Resolved physical source metadata for a built-in dataset.

path_or_dataset: str#

None

schema_adapter: str | None#

None

split: str#

‘train’

subset: str | list[str] | None#

None

load_kwargs: dict[str, Any] | None#

None

adapter_kwargs: dict[str, Any] | None#

None

required_adapter_kwargs: tuple[str, ...]#

()

split_aliases: dict[str, str] | None#

None

supported_splits: tuple[str, ...] | None#

None

bridge.data.sources.hf._HF_DATASET_PRESETS: dict[str, bridge.data.sources.hf._HFDatasetPreset]#

None

bridge.data.sources.hf._resolve_preset_split(
dataset_name: str,
preset: bridge.data.sources.hf._HFDatasetPreset,
split: str,
) → str#
bridge.data.sources.hf.hf_dataset_supports_split(
source: bridge.data.sources.hf.HFDatasetSourceConfig,
split: str,
) → bool#

Return whether a source declares support for a logical split.

bridge.data.sources.hf.resolve_hf_dataset_source(
source: bridge.data.sources.hf.HFDatasetSourceConfig,
) → bridge.data.sources.hf.HFDatasetSourceConfig#

Resolve a named preset or custom source to complete physical metadata.

bridge.data.sources.hf.load_hf_dataset_source(
source: bridge.data.sources.hf.HFDatasetSourceConfig,
) → Any#

Load one declarative Hugging Face source without adapting its rows.

bridge.data.sources.hf.prepare_hf_dataset_sources(
sources: collections.abc.Sequence[bridge.data.sources.hf.HFDatasetSourceConfig],
) → None#

Materialize Hugging Face caches once before distributed readers start.

Hugging Face cache creation is not reliable when multiple distributed ranks concurrently build the same source on a shared filesystem. Rank zero loads each requested source first, then broadcasts completion so every rank reads an already-stable cache. Single-process callers need no preparation because their normal load has no competing writer.

bridge.data.sources.hf.load_and_adapt_hf_dataset(
source: bridge.data.sources.hf.HFDatasetSourceConfig,
) → list[dict[str, Any]]#

Load a Hugging Face split and normalize it to canonical SFT rows.

bridge.data.sources.hf.resolve_blend_weights(
sources: collections.abc.Sequence[bridge.data.sources.hf.HFDatasetSourceConfig],
weights: collections.abc.Sequence[float] | None,
) → list[float]#

Validate blend weights against their sources and fill in the uniform default.

Parameters:

  • sources – The sources taking part in the blend.

  • weights – One positive weight per source, or None for equal weights.

Returns:

One weight per source.

Raises:

ValueError – If no source is given, if the counts disagree, or if a weight is not a positive finite number.

bridge.data.sources.hf.blend_sft_rows(
rows_per_source: collections.abc.Sequence[collections.abc.Sequence[dict[str, Any]]],
weights: collections.abc.Sequence[float],
*,
seed: int,
) → list[dict[str, Any]]#

Draw rows from several sources under per-source epoch counts.

A weight is how many passes the blend takes over a source: 1.0 keeps every row once, 2.5 keeps every row twice plus half of them again, and 0.5 keeps half of them. Fractional passes draw without replacement and round up to a whole row, so a positive weight always keeps at least one.

Weights count rows, not tokens. SFT rows vary in length, so equal row counts do not make equal token counts.

Parameters:

  • rows_per_source – Canonical SFT rows for each source of the blend.

  • weights – One positive weight per source.

  • seed – Seeds the fractional draws and the shuffle that interleaves sources.

Returns:

The blended rows, holding references to the rows that were passed in.

Raises:

ValueError – If a source contributed no rows.

bridge.data.sources.hf.load_and_blend_hf_datasets(
sources: collections.abc.Sequence[bridge.data.sources.hf.HFDatasetSourceConfig],
weights: collections.abc.Sequence[float] | None,
*,
seed: int,
) → list[dict[str, Any]]#

Load several Hugging Face splits and mix them under per-source epoch counts.

Each source is adapted to canonical SFT rows first, so sources with unrelated column layouts can be blended as long as they share a schema adapter target.

Parameters:

  • sources – The sources taking part in the blend.

  • weights – One positive weight per source, or None for one pass each.

  • seed – Seeds the fractional draws and the shuffle that interleaves sources.

Returns:

The blended canonical SFT rows.