`bridge.data.sft_processing`#

Shared declarative preprocessing and canonical examples for SFT backends.

Module Contents#

Classes#

`ChatSFTExample`	Canonical structured chat example.
`PromptCompletionSFTExample`	Canonical unformatted prompt-completion example.
`ChatSFTPreprocessingConfig`	Declarative preprocessing for structured conversation examples.
`PromptCompletionSFTPreprocessingConfig`	Declarative preprocessing for unformatted prompt-completion examples.
`TokenizedPromptCompletion`	Tokenized prompt-completion row with an unshifted loss mask.

Functions#

`validate_sft_preprocessing_config`	Validate one supported SFT preprocessing variant.
`_canonical_prompt_completion_values`
`normalize_sft_example`	Normalize one adapted row for the selected preprocessing semantics.
`normalize_sft_examples`	Normalize adapted rows for one explicit SFT preprocessing variant.
`is_text_only_prompt_completion_example`	Return whether a row is a text-only prompt-completion example.
`sft_example_metadata`	Return non-training columns retained as example metadata.
`_tokenize_text`
`_is_space_sensitive`	Return whether a leading word-boundary space changes tokenization.
`_token_id`
`_truncate`
`tokenize_prompt_completion_example`	Tokenize a prompt-completion row without applying a chat template.

Data#

`CanonicalSFTExample`
`SFTPreprocessingConfig`
`_CONVERSATION_KEYS`
`_CANONICAL_PROMPT_KEY`
`_CANONICAL_COMPLETION_KEY`
`_MEDIA_KEYS`

API#

class bridge.data.sft_processing.ChatSFTExample#

Bases: typing.TypedDict

Canonical structured chat example.

Initialization

Initialize self. See help(type(self)) for accurate signature.

conversation: list[dict[str, Any]]#: None

class bridge.data.sft_processing.PromptCompletionSFTExample#

Bases: typing.TypedDict

Canonical unformatted prompt-completion example.

Initialization

Initialize self. See help(type(self)) for accurate signature.

prompt: str#: None

completion: str#: None

bridge.data.sft_processing.CanonicalSFTExample#: None

class bridge.data.sft_processing.ChatSFTPreprocessingConfig#

Declarative preprocessing for structured conversation examples.

Parameters:: loss_mode – Tokens that contribute to the loss. assistant trains all assistant turns, last_turn trains only the final assistant span, and full trains the complete rendered conversation.

loss_mode: Literal[assistant, last_turn, full]#: ‘assistant’

validate() → None#: Validate chat preprocessing settings.

class bridge.data.sft_processing.PromptCompletionSFTPreprocessingConfig#

Declarative preprocessing for unformatted prompt-completion examples.

The prompt and completion are tokenized separately and concatenated without calling apply_chat_template. prompt_template may contain only the {prompt} placeholder; raw column selection remains explicit through prompt_column and completion_column.

Parameters:

prompt_column – Source column containing the prompt text.
completion_column – Source column containing the completion text.
prompt_template – Formatting applied to the prompt before tokenization.
separator – Text inserted between the rendered prompt and completion.
loss_mode – completion masks the prompt; full trains all tokens.
strip_whitespace – Strip leading and trailing ASCII spaces from both fields.
add_bos – Add the tokenizer BOS token before the prompt.
add_sep – Add the tokenizer SEP token between prompt and completion.
add_eos – Add the tokenizer EOS token after the completion.
truncation_method – Side retained when a prompt or completion must be truncated.

prompt_column: str#: ‘prompt’

completion_column: str#: ‘completion’

prompt_template: str#: ‘{prompt}’

separator: str = <Multiline-String>#

loss_mode: Literal[completion, full]#: ‘completion’

strip_whitespace: bool#: True

add_bos: bool#: False

add_sep: bool#: False

add_eos: bool#: True

truncation_method: Literal[left, right]#: ‘right’

validate() → None#: Validate prompt-completion preprocessing settings.

bridge.data.sft_processing.SFTPreprocessingConfig#: None

class bridge.data.sft_processing.TokenizedPromptCompletion#

Tokenized prompt-completion row with an unshifted loss mask.

input_ids: torch.Tensor#: None

loss_mask: torch.Tensor#: None

prompt_ids: torch.Tensor#: None

completion_ids: torch.Tensor#: None

bridge.data.sft_processing._CONVERSATION_KEYS#: (‘messages’, ‘conversation’, ‘conversations’)

bridge.data.sft_processing._CANONICAL_PROMPT_KEY#: ‘prompt’

bridge.data.sft_processing._CANONICAL_COMPLETION_KEY#: ‘completion’

bridge.data.sft_processing._MEDIA_KEYS#: (‘image’, ‘images’, ‘image_path’, ‘image_paths’, ‘video’, ‘videos’, ‘video_path’, ‘video_paths’, ‘au…

bridge.data.sft_processing.validate_sft_preprocessing_config( config: bridge.data.sft_processing.SFTPreprocessingConfig, ) → None#: Validate one supported SFT preprocessing variant.

bridge.data.sft_processing._canonical_prompt_completion_values( example: collections.abc.Mapping[str, Any], ) → tuple[str, str] | None#

bridge.data.sft_processing.normalize_sft_example( example: collections.abc.Mapping[str, Any], preprocessing: bridge.data.sft_processing.SFTPreprocessingConfig, ) → dict[str, Any]#

Normalize one adapted row for the selected preprocessing semantics.

Prompt-completion rows may be promoted to a two-turn chat. Structured conversations are deliberately not flattened into prompt-completion text, because doing so without a chat template would silently change their model semantics.

bridge.data.sft_processing.normalize_sft_examples( examples: collections.abc.Sequence[collections.abc.Mapping[str, Any]], preprocessing: bridge.data.sft_processing.SFTPreprocessingConfig, ) → list[dict[str, Any]]#: Normalize adapted rows for one explicit SFT preprocessing variant.

bridge.data.sft_processing.is_text_only_prompt_completion_example( example: collections.abc.Mapping[str, Any], preprocessing: bridge.data.sft_processing.PromptCompletionSFTPreprocessingConfig, ) → bool#: Return whether a row is a text-only prompt-completion example.

bridge.data.sft_processing.sft_example_metadata( example: collections.abc.Mapping[str, Any], preprocessing: bridge.data.sft_processing.SFTPreprocessingConfig, ) → dict[str, Any]#: Return non-training columns retained as example metadata.

bridge.data.sft_processing._tokenize_text( tokenizer_or_processor: Any, text: str, ) → list[int]#

bridge.data.sft_processing._is_space_sensitive(tokenizer_or_processor: Any) → bool#: Return whether a leading word-boundary space changes tokenization.

bridge.data.sft_processing._token_id( tokenizer_or_processor: Any, *names: str, ) → int | None#

bridge.data.sft_processing._truncate( ids: list[int], target_length: int, method: Literal[left, right], ) → list[int]#

bridge.data.sft_processing.tokenize_prompt_completion_example( example: collections.abc.Mapping[str, Any], tokenizer_or_processor: Any, preprocessing: bridge.data.sft_processing.PromptCompletionSFTPreprocessingConfig, *, max_length: int | None = None, skipped_tokens: torch.Tensor | None = None, allow_missing_completion: bool = False, prefix_token_ids: collections.abc.Sequence[int] = (), minimum_completion_length: int = 0, sep_token_id: int | None = None, ) → bridge.data.sft_processing.TokenizedPromptCompletion#: Tokenize a prompt-completion row without applying a chat template.

bridge.data.sft_processing#