bridge.models.inkling.data.image_processing#

Inkling2 image preprocessing for training.

The thinkingmachines/Inkling-2-Small release names an Inkling2ImageProcessor in its processor_config.json but ships no implementation, and Transformers adds its own only in v5.23.0, so this implements the release’s settings: an image keeps its native size, is padded on the right and bottom to a multiple of 40 pixels with a value just below black, rescaled by 1/255 and normalized with mean and standard deviation 0.5 (to [-1, 1]), and cut into 40x40-pixel patches in row-major order. Each patch repeats twice in time, and each one becomes one image token for the language model. The vision encoder splits a patch into four 20-pixel patches and merges their outputs back into one embedding.

With max_image_tokens, a larger image is encoded at the largest size within that token budget that keeps its aspect ratio, resized with a Lanczos3 kernel in FP32 arithmetic that takes its sine from the C library’s single-precision sinf. On smooth images it stays within one level of Pillow’s LANCZOS filter.

Module Contents#

Classes#

Inkling2EncodedImage

One preprocessed image.

Inkling2ImageProcessor

Turns images into Inkling2 vision patches and image-token counts.

Functions#

_sinf

Return the C library’s single-precision sine.

_sinc

_lanczos_weights

First source index and normalized FP32 Lanczos3 weights for one output index.

resize_lanczos3

Resize [H, W, C] uint8 pixels with a Lanczos3 kernel: rows first, then columns.

_image_bytes

Return encoded image bytes; a PIL image is saved as PNG, as the reference does.

_decode_rgb

Decode to [H, W, 3] uint8, compositing transparent images onto an 8-pixel checkerboard.

load_processor_config

Read a checkpoint’s processor_config.json.

Data#

API#

bridge.models.inkling.data.image_processing.PATCH_SIZE#

40

bridge.models.inkling.data.image_processing.TEMPORAL_PATCH_SIZE#

2

bridge.models.inkling.data.image_processing.PROCESSOR_CONFIG_FILE#

‘processor_config.json’

bridge.models.inkling.data.image_processing.IMAGE_PROCESSOR_CONFIG#

None

bridge.models.inkling.data.image_processing._MEAN#

‘array(…)’

bridge.models.inkling.data.image_processing._STD#

‘array(…)’

bridge.models.inkling.data.image_processing._LANCZOS_LOBES#

‘float32(…)’

bridge.models.inkling.data.image_processing._sinf() → collections.abc.Callable[[float], float]#

Return the C library’s single-precision sine.

The Lanczos3 kernel evaluates its sine in single precision. A double-precision sine rounded to FP32 differs for about 1% of the kernel’s arguments, enough to move some resized pixels by one level.

bridge.models.inkling.data.image_processing._sinc(value: numpy.float32) → numpy.float32#
bridge.models.inkling.data.image_processing._lanczos_weights(
source_size: int,
target_size: int,
output_index: int,
) → tuple[int, numpy.ndarray]#

First source index and normalized FP32 Lanczos3 weights for one output index.

bridge.models.inkling.data.image_processing.resize_lanczos3(
pixels: numpy.ndarray,
width: int,
height: int,
) → numpy.ndarray#

Resize [H, W, C] uint8 pixels with a Lanczos3 kernel: rows first, then columns.

Each output value sums its taps in order in FP32, then rounds half up and clips to uint8.

class bridge.models.inkling.data.image_processing.Inkling2EncodedImage#

One preprocessed image.

.. attribute:: patches

[rows * columns, 2, 40, 40, 3] BF16 patches, the model’s pixel_values rows.

.. attribute:: grid_thw

[1, 3] grid (1, rows, columns), the model’s image_grid_thw row.

.. attribute:: num_tokens

Image tokens the language model reserves for this image, rows * columns.

patches: torch.Tensor#

None

grid_thw: torch.Tensor#

None

num_tokens: int#

None

bridge.models.inkling.data.image_processing._image_bytes(
image: bytes | str | os.PathLike | PIL.Image.Image,
) → bytes#

Return encoded image bytes; a PIL image is saved as PNG, as the reference does.

bridge.models.inkling.data.image_processing._decode_rgb(image_bytes: bytes) → numpy.ndarray#

Decode to [H, W, 3] uint8, compositing transparent images onto an 8-pixel checkerboard.

bridge.models.inkling.data.image_processing.load_processor_config(
path: str | os.PathLike,
**download_kwargs: Any,
) → dict[str, Any]#

Read a checkpoint’s processor_config.json.

Parameters:
  • path – Checkpoint directory or Hugging Face repository ID.

  • **download_kwargs – hf_hub_download arguments, such as revision, for a repository ID.

Returns:

The parsed configuration.

class bridge.models.inkling.data.image_processing.Inkling2ImageProcessor(*, max_image_tokens: int | None = None)#

Turns images into Inkling2 vision patches and image-token counts.

Parameters:

max_image_tokens – Token budget per image; a larger image is encoded at the largest size that fits, keeping its aspect ratio. None keeps every image at its native size, as the reference does.

Initialization

patch_size#

None

temporal_patch_size#

None

classmethod from_pretrained(
path: str | os.PathLike,
*,
max_image_tokens: int | None = None,
**download_kwargs: Any,
) → bridge.models.inkling.data.image_processing.Inkling2ImageProcessor#

Load from a checkpoint, checking the image_processor section of its processor_config.json.

Parameters:
  • path – Checkpoint directory or Hugging Face repository ID.

  • max_image_tokens – Token budget per image, or None for native sizes.

  • **download_kwargs – hf_hub_download arguments, such as revision, for a repository ID.

Returns:

The image processor.

Raises:

ValueError – If the checkpoint asks for image processing this port does not implement.

classmethod from_dict(
processor_config: dict[str, Any],
*,
max_image_tokens: int | None = None,
) → bridge.models.inkling.data.image_processing.Inkling2ImageProcessor#

Build from a parsed processor_config.json.

Parameters:
  • processor_config – The checkpoint’s processor configuration.

  • max_image_tokens – Token budget per image, or None for native sizes.

Returns:

The image processor.

Raises:

ValueError – If the configuration asks for image processing this port does not implement.

static count_tokens(width: int, height: int) → int#

Return the image tokens for an image of width x height pixels.

fitted_size(width: int, height: int) → tuple[int, int]#

Return the size an image is encoded at: native, or the largest same-aspect size within the budget.

encode(
image: bytes | str | os.PathLike | PIL.Image.Image,
) → bridge.models.inkling.data.image_processing.Inkling2EncodedImage#

Preprocess one image.

Parameters:

image – Encoded image bytes, a PIL image, or a local path.

Returns:

The image’s patches, grid, and token count.

bridge.models.inkling.data.image_processing.__all__#

[‘IMAGE_PROCESSOR_CONFIG’, ‘PROCESSOR_CONFIG_FILE’, ‘Inkling2EncodedImage’, ‘Inkling2ImageProcessor’…