bridge.models.inkling.data.processor#

Inkling2 processor for image-text SFT.

The thinkingmachines/Inkling-2-Small release names an Inkling2Processor in its processor_config.json but ships no implementation. This one renders a conversation straight to token IDs exactly as the release’s chat_template.jinja does, then gives each image its patches: every content part is its own block, <|message_ROLE|>[name]<|content_KIND|>payload<|end_message|>, the conversation’s tools are declared in a system block before every message, a Thinking effort level system line precedes the first non-system message, and every assistant turn ends with <|content_model_end_sampling|>. Where the template writes one <|unused_200054|> image placeholder, the processor writes one per 40x40-pixel patch, which the model replaces with the image’s features. The whole assistant turn, headers and end-of-sampling token included, is the supervised span.

Two inputs go beyond the template: an assistant message may carry its reasoning in thinking (the field Bridge’s dataset adapters write) or reasoning instead of reasoning_content, the first one set being used, as Transformers does from version 5.22; and an image part carries its pixels in image.

DirectHFSFTDatasetConfig.hf_processor_class names it, and hf_processor_path the checkpoint directory or Hub repository it loads. AutoProcessor resolves Transformers’ processor of the same name; Bridge’s image-text recipes explicitly select this processor for the model-owned batch format.

Module Contents#

Classes#

Inkling2Processor

Turns conversations with images into Inkling2 token IDs, assistant masks and vision patches.

Functions#

reasoning_effort_level

Return the chat template’s Thinking effort level text for a named or numeric effort.

Data#

API#

bridge.models.inkling.data.processor._MESSAGE_TOKENS#

None

bridge.models.inkling.data.processor._FRAMING_TOKENS#

()

bridge.models.inkling.data.processor._TEMPLATE_KWARGS#

‘frozenset(…)’

bridge.models.inkling.data.processor.REASONING_EFFORTS#

None

bridge.models.inkling.data.processor.DEFAULT_REASONING_EFFORT#

0.9

bridge.models.inkling.data.processor._TEXT_PARTS#

(‘text’, ‘input_text’)

bridge.models.inkling.data.processor._REASONING_FIELDS#

(‘reasoning_content’, ‘thinking’, ‘reasoning’)

bridge.models.inkling.data.processor._IMAGE_PARTS#

(‘image’, ‘input_image’, ‘image_url’)

bridge.models.inkling.data.processor._TOKENIZER_KWARGS#

(‘revision’, ‘trust_remote_code’, ‘cache_dir’, ‘local_files_only’, ‘token’)

bridge.models.inkling.data.processor.reasoning_effort_level(effort: str | float | None) → str#

Return the chat template’s Thinking effort level text for a named or numeric effort.

Parameters:

effort – A named level such as "none" or "high", a number in [0, 0.99], or None for the template’s default.

Returns:

The level as the template prints it.

Raises:

ValueError – For an unknown name or a number outside [0, 0.99].

class bridge.models.inkling.data.processor.Inkling2Processor(
tokenizer: Any,
image_processor: megatron.bridge.models.inkling.data.image_processing.Inkling2ImageProcessor,
*,
image_token_id: int,
reasoning_effort: str | float | None = None,
)#

Turns conversations with images into Inkling2 token IDs, assistant masks and vision patches.

Parameters:
  • tokenizer – The checkpoint’s Hugging Face tokenizer.

  • image_processor – Image preprocessing for the checkpoint.

  • image_token_id – Placeholder token the model replaces with image features.

  • reasoning_effort – Thinking effort written before the first non-system message, as the chat template’s reasoning_effort argument; None writes the template’s default. A conversation’s own reasoning_effort replaces it.

Initialization

classmethod from_pretrained(
path: str | os.PathLike,
*,
max_image_tokens: int | None = None,
reasoning_effort: str | float | None = None,
**kwargs: Any,
) → bridge.models.inkling.data.processor.Inkling2Processor#

Load the tokenizer, image processing and image placeholder of a checkpoint.

Parameters:
  • path – Checkpoint directory or Hugging Face repository ID.

  • max_image_tokens – Token budget per image; larger images are encoded at the largest size that fits. None keeps native sizes.

  • reasoning_effort – Thinking effort for every rendered conversation; None writes the template’s default.

  • **kwargs – AutoProcessor arguments; the loading-related ones are passed on.

Returns:

The processor.

Raises:

ValueError – If the checkpoint is not an Inkling2 checkpoint with the release’s processor settings, or its image token is not the one the model replaces.

_text(text: str) → list[int]#
_block(header: list[int], kind: str, payload: list[int]) → list[int]#
static _tool_header_name(
message: collections.abc.Mapping[str, Any],
conversation: collections.abc.Sequence[collections.abc.Mapping[str, Any]],
) → str#

Name a tool result by its name, or by every assistant tool call its tool_call_id matches.

_tool_calls(
message: collections.abc.Mapping[str, Any],
) → list[int]#
_tool_declaration(
tools: collections.abc.Sequence[collections.abc.Mapping[str, Any]],
) → list[int]#

Declare the tools as the chat template does: one system block of their sorted JSON specifications.

render(
conversation: collections.abc.Sequence[collections.abc.Mapping[str, Any]],
*,
tools: collections.abc.Sequence[collections.abc.Mapping[str, Any]] | None = None,
reasoning_effort: str | float | None = None,
) → tuple[list[int], list[int], list[megatron.bridge.models.inkling.data.image_processing.Inkling2EncodedImage]]#

Render one conversation.

Parameters:
  • conversation – Messages with role and content; content is a string or a list of strings, {"type": "text", "text": ...} and {"type": "image", "image": ...} parts. An assistant message may carry its reasoning in reasoning_content, thinking or reasoning and its calls in tool_calls.

  • tools – Tool specifications, as the chat template’s tools argument, declared before every message.

  • reasoning_effort – This conversation’s thinking effort, in place of the processor’s.

Returns:

Token IDs, a 0/1 mask of the supervised (assistant) tokens, and the images in order. The tool declaration and the thinking-effort line are not supervised.

Raises:

ValueError – On an unsupported role or content part, an image outside a user or tool message, malformed tool calls, a tool without a function name, or an unknown thinking effort.

property chat_template: str | None#

The checkpoint’s chat template, which Hugging Face-style callers render prompts with.

apply_chat_template(
conversation: collections.abc.Sequence[collections.abc.Mapping[str, Any]],
*,
tokenize: bool = False,
add_generation_prompt: bool = False,
**kwargs: Any,
) → Any#

Render a conversation with the checkpoint’s chat template, as a Transformers processor does.

The template writes one image placeholder per image; processor(text=..., images=...) expands each into one placeholder per patch. Inkling 2’s template adds no generation prompt, since the model writes its own message headers.

Parameters:
  • conversation – The messages.

  • tokenize – Return token IDs instead of text.

  • add_generation_prompt – Passed to the template, which ignores it.

  • **kwargs – Template arguments such as reasoning_effort or tools.

Returns:

The rendered text, or token IDs with tokenize=True.

_encode_text(
text: str | collections.abc.Sequence[str],
images: Any,
) → transformers.BatchFeature#

Tokenize rendered prompts and give each image placeholder one position per patch.

__call__(
conversations: collections.abc.Sequence[collections.abc.Sequence[collections.abc.Mapping[str, Any]]] | None = None,
*,
max_length: int | None = None,
chat_template_kwargs: collections.abc.Sequence[collections.abc.Mapping[str, Any]] | None = None,
text: str | collections.abc.Sequence[str] | None = None,
images: Any = None,
**kwargs: Any,
) → dict[str, torch.Tensor | None] | transformers.BatchFeature#

Render and right-pad a batch of conversations, or encode rendered text as a Transformers processor.

Parameters:
  • conversations – One message list per sample.

  • max_length – Longest allowed sample. Text after the last image is cut to fit; a sample whose image placeholders would be cut is rejected, since its image features could not be placed.

  • chat_template_kwargs – One mapping of chat template arguments per conversation: tools and reasoning_effort, the arguments the template reads; see :meth:render.

  • text – Prompts already rendered, for example by :meth:apply_chat_template, with one image placeholder per image. Given instead of conversations.

  • images – The images for text’s placeholders, in order: one image or a list of them, possibly nested per prompt. An image is a PIL image, a local path, or encoded bytes.

  • **kwargs – Transformers processor arguments such as return_tensors and padding; the outputs are always right-padded PyTorch tensors.

Returns:

input_ids, attention_mask and, for conversations, assistant_masks [B, L]; pixel_values [patches, 2, 40, 40, 3] BF16 and image_grid_thw [images, 3] for the batch’s images in order, or None when the batch has none.

Raises:

ValueError – If an image does not fit in max_length, the images do not match the text’s image placeholders, or a conversation’s template arguments are ones the chat template does not read.

bridge.models.inkling.data.processor.__all__#

[‘DEFAULT_REASONING_EFFORT’, ‘REASONING_EFFORTS’, ‘Inkling2Processor’, ‘reasoning_effort_level’]