`bridge.models.deepseek.data.collate_fn`#

DeepSeek text-chat collators.

Module Contents#

Functions#

`_deepseek_v4_options`
`_normalize_deepseek_v4_conversation`	Add the omitted content field used by OpenAI tool-call-only turns.
`_attach_tools`
`tokenize_deepseek_v4_example`	Render and tokenize one DeepSeek-V4 chat row with the official encoder.
`deepseek_v4_collate_fn`	Collate DeepSeek-V4 chats without synthesizing a Jinja template.

API#

bridge.models.deepseek.data.collate_fn._deepseek_v4_options( example: collections.abc.Mapping[str, Any], ) → tuple[Literal[chat, thinking], bool, Literal[high, max] | None]#

bridge.models.deepseek.data.collate_fn._normalize_deepseek_v4_conversation( example_or_conversation: collections.abc.Mapping[str, Any] | collections.abc.Sequence[collections.abc.Mapping[str, Any]], ) → list[dict[str, Any]]#: Add the omitted content field used by OpenAI tool-call-only turns.

bridge.models.deepseek.data.collate_fn._attach_tools( conversation: list[dict[str, Any]], tools: collections.abc.Sequence[collections.abc.Mapping[str, Any]] | None, ) → list[dict[str, Any]]#

bridge.models.deepseek.data.collate_fn.tokenize_deepseek_v4_example(

example_or_conversation: collections.abc.Mapping[str, Any] | collections.abc.Sequence[collections.abc.Mapping[str, Any]],

processor: Any,

*,

max_length: int | None = None,

skipped_tokens: torch.Tensor | None = None,

boundary_config: megatron.bridge.data.conversation_processing.AssistantMaskBoundaryConfig | None = None,

warn_on_all_masked: bool = True,

loss_mode: Literal[assistant, last_turn, full] = 'assistant',

**_: Any,

) → megatron.bridge.data.conversation_processing.TokenizedConversation#: Render and tokenize one DeepSeek-V4 chat row with the official encoder.

bridge.models.deepseek.data.collate_fn.deepseek_v4_collate_fn(

examples: list[collections.abc.Mapping[str, Any]],

processor: Any,

*,

sequence_length: int | None = None,

max_length: int | None = None,

pad_to_max_length: bool = False,

pad_to_multiple_of: int = 1,

warn_on_all_masked: bool = True,

loss_mode: Literal[assistant, last_turn, full] = 'assistant',

enable_in_batch_packing: bool = False,

in_batch_packing_pad_to_multiple_of: int = 1,

**kwargs: Any,

) → dict[str, Any]#: Collate DeepSeek-V4 chats without synthesizing a Jinja template.