bridge.models.inkling.data.processor#
Inkling2 processor for image-text SFT.
The thinkingmachines/Inkling-2-Small release names an Inkling2Processor in its
processor_config.json but ships no implementation. This one renders a conversation straight to token
IDs exactly as the release’s chat_template.jinja does, then gives each image its patches: every
content part is its own block, <|message_ROLE|>[name]<|content_KIND|>payload<|end_message|>, the
conversation’s tools are declared in a system block before every message, a Thinking effort level system
line precedes the first non-system message, and every assistant turn ends with
<|content_model_end_sampling|>. Where the template writes one <|unused_200054|> image
placeholder, the processor writes one per 40x40-pixel patch, which the model replaces with the image’s
features. The whole assistant turn, headers and end-of-sampling token included, is the supervised span.
Two inputs go beyond the template: an assistant message may carry its reasoning in thinking (the field
Bridge’s dataset adapters write) or reasoning instead of reasoning_content, the first one set being used,
as Transformers does from version 5.22; and an image part carries its pixels in image.
DirectHFSFTDatasetConfig.hf_processor_class names it, and hf_processor_path the checkpoint directory or Hub
repository it loads. AutoProcessor resolves Transformers’ processor of the same name; Bridge’s image-text
recipes explicitly select this processor for the model-owned batch format.
Module Contents#
Classes#
Turns conversations with images into Inkling2 token IDs, assistant masks and vision patches. |
Functions#
Return the chat template’s |
Data#
API#
- bridge.models.inkling.data.processor._MESSAGE_TOKENS#
None
- bridge.models.inkling.data.processor._FRAMING_TOKENS#
()
- bridge.models.inkling.data.processor._TEMPLATE_KWARGS#
‘frozenset(…)’
- bridge.models.inkling.data.processor.REASONING_EFFORTS#
None
- bridge.models.inkling.data.processor.DEFAULT_REASONING_EFFORT#
0.9
- bridge.models.inkling.data.processor._TEXT_PARTS#
(‘text’, ‘input_text’)
- bridge.models.inkling.data.processor._REASONING_FIELDS#
(‘reasoning_content’, ‘thinking’, ‘reasoning’)
- bridge.models.inkling.data.processor._IMAGE_PARTS#
(‘image’, ‘input_image’, ‘image_url’)
- bridge.models.inkling.data.processor._TOKENIZER_KWARGS#
(‘revision’, ‘trust_remote_code’, ‘cache_dir’, ‘local_files_only’, ‘token’)
- bridge.models.inkling.data.processor.reasoning_effort_level(effort: str | float | None) str#
Return the chat template’s
Thinking effort leveltext for a named or numeric effort.- Parameters:
effort – A named level such as
"none"or"high", a number in[0, 0.99], orNonefor the template’s default.- Returns:
The level as the template prints it.
- Raises:
ValueError – For an unknown name or a number outside
[0, 0.99].
- class bridge.models.inkling.data.processor.Inkling2Processor(
- tokenizer: Any,
- image_processor: megatron.bridge.models.inkling.data.image_processing.Inkling2ImageProcessor,
- *,
- image_token_id: int,
- reasoning_effort: str | float | None = None,
Turns conversations with images into Inkling2 token IDs, assistant masks and vision patches.
- Parameters:
tokenizer – The checkpoint’s Hugging Face tokenizer.
image_processor – Image preprocessing for the checkpoint.
image_token_id – Placeholder token the model replaces with image features.
reasoning_effort – Thinking effort written before the first non-system message, as the chat template’s
reasoning_effortargument;Nonewrites the template’s default. A conversation’s ownreasoning_effortreplaces it.
Initialization
- classmethod from_pretrained(
- path: str | os.PathLike,
- *,
- max_image_tokens: int | None = None,
- reasoning_effort: str | float | None = None,
- **kwargs: Any,
Load the tokenizer, image processing and image placeholder of a checkpoint.
- Parameters:
path – Checkpoint directory or Hugging Face repository ID.
max_image_tokens – Token budget per image; larger images are encoded at the largest size that fits.
Nonekeeps native sizes.reasoning_effort – Thinking effort for every rendered conversation;
Nonewrites the template’s default.**kwargs –
AutoProcessorarguments; the loading-related ones are passed on.
- Returns:
The processor.
- Raises:
ValueError – If the checkpoint is not an Inkling2 checkpoint with the release’s processor settings, or its image token is not the one the model replaces.
- _text(text: str) list[int]#
- _block(header: list[int], kind: str, payload: list[int]) list[int]#
- static _tool_header_name(
- message: collections.abc.Mapping[str, Any],
- conversation: collections.abc.Sequence[collections.abc.Mapping[str, Any]],
Name a tool result by its
name, or by every assistant tool call itstool_call_idmatches.
- _tool_calls(
- message: collections.abc.Mapping[str, Any],
- _tool_declaration(
- tools: collections.abc.Sequence[collections.abc.Mapping[str, Any]],
Declare the tools as the chat template does: one system block of their sorted JSON specifications.
- render(
- conversation: collections.abc.Sequence[collections.abc.Mapping[str, Any]],
- *,
- tools: collections.abc.Sequence[collections.abc.Mapping[str, Any]] | None = None,
- reasoning_effort: str | float | None = None,
Render one conversation.
- Parameters:
conversation – Messages with
roleandcontent; content is a string or a list of strings,{"type": "text", "text": ...}and{"type": "image", "image": ...}parts. An assistant message may carry its reasoning inreasoning_content,thinkingorreasoningand its calls intool_calls.tools – Tool specifications, as the chat template’s
toolsargument, declared before every message.reasoning_effort – This conversation’s thinking effort, in place of the processor’s.
- Returns:
Token IDs, a 0/1 mask of the supervised (assistant) tokens, and the images in order. The tool declaration and the thinking-effort line are not supervised.
- Raises:
ValueError – On an unsupported role or content part, an image outside a user or tool message, malformed tool calls, a tool without a function name, or an unknown thinking effort.
- property chat_template: str | None#
The checkpoint’s chat template, which Hugging Face-style callers render prompts with.
- apply_chat_template(
- conversation: collections.abc.Sequence[collections.abc.Mapping[str, Any]],
- *,
- tokenize: bool = False,
- add_generation_prompt: bool = False,
- **kwargs: Any,
Render a conversation with the checkpoint’s chat template, as a Transformers processor does.
The template writes one image placeholder per image;
processor(text=..., images=...)expands each into one placeholder per patch. Inkling 2’s template adds no generation prompt, since the model writes its own message headers.- Parameters:
conversation – The messages.
tokenize – Return token IDs instead of text.
add_generation_prompt – Passed to the template, which ignores it.
**kwargs – Template arguments such as
reasoning_effortortools.
- Returns:
The rendered text, or token IDs with
tokenize=True.
- _encode_text(
- text: str | collections.abc.Sequence[str],
- images: Any,
Tokenize rendered prompts and give each image placeholder one position per patch.
- __call__(
- conversations: collections.abc.Sequence[collections.abc.Sequence[collections.abc.Mapping[str, Any]]] | None = None,
- *,
- max_length: int | None = None,
- chat_template_kwargs: collections.abc.Sequence[collections.abc.Mapping[str, Any]] | None = None,
- text: str | collections.abc.Sequence[str] | None = None,
- images: Any = None,
- **kwargs: Any,
Render and right-pad a batch of conversations, or encode rendered text as a Transformers processor.
- Parameters:
conversations – One message list per sample.
max_length – Longest allowed sample. Text after the last image is cut to fit; a sample whose image placeholders would be cut is rejected, since its image features could not be placed.
chat_template_kwargs – One mapping of chat template arguments per conversation:
toolsandreasoning_effort, the arguments the template reads; see :meth:render.text – Prompts already rendered, for example by :meth:
apply_chat_template, with one image placeholder per image. Given instead ofconversations.images – The images for
text’s placeholders, in order: one image or a list of them, possibly nested per prompt. An image is a PIL image, a local path, or encoded bytes.**kwargs – Transformers processor arguments such as
return_tensorsandpadding; the outputs are always right-padded PyTorch tensors.
- Returns:
input_ids,attention_maskand, for conversations,assistant_masks[B, L];pixel_values[patches, 2, 40, 40, 3]BF16 andimage_grid_thw[images, 3]for the batch’s images in order, orNonewhen the batch has none.- Raises:
ValueError – If an image does not fit in
max_length, the images do not match the text’s image placeholders, or a conversation’s template arguments are ones the chat template does not read.
- bridge.models.inkling.data.processor.__all__#
[‘DEFAULT_REASONING_EFFORT’, ‘REASONING_EFFORTS’, ‘Inkling2Processor’, ‘reasoning_effort_level’]