bridge.models.inkling.data.image_processing#
Inkling2 image preprocessing for training.
The thinkingmachines/Inkling-2-Small release names an Inkling2ImageProcessor in its
processor_config.json but ships no implementation, and Transformers adds its own only in v5.23.0, so this
implements the release’s settings: an image keeps its native size, is padded on the
right and bottom to a multiple of 40 pixels with a value just below black, rescaled by 1/255 and normalized
with mean and standard deviation 0.5 (to [-1, 1]), and cut into 40x40-pixel patches in row-major order.
Each patch repeats twice in time, and each one becomes one image token for the language model. The vision
encoder splits a patch into four 20-pixel patches and merges their outputs back into one embedding.
With max_image_tokens, a larger image is encoded at the largest size within that token budget that keeps
its aspect ratio, resized with a Lanczos3 kernel in FP32 arithmetic that takes its sine from the C library’s
single-precision sinf. On smooth images it stays within one level of Pillow’s LANCZOS filter.
Module Contents#
Classes#
One preprocessed image. |
|
Turns images into Inkling2 vision patches and image-token counts. |
Functions#
Return the C library’s single-precision sine. |
|
First source index and normalized FP32 Lanczos3 weights for one output index. |
|
Resize |
|
Return encoded image bytes; a PIL image is saved as PNG, as the reference does. |
|
Decode to |
|
Read a checkpoint’s |
Data#
API#
- bridge.models.inkling.data.image_processing.PATCH_SIZE#
40
- bridge.models.inkling.data.image_processing.TEMPORAL_PATCH_SIZE#
2
- bridge.models.inkling.data.image_processing.PROCESSOR_CONFIG_FILE#
‘processor_config.json’
- bridge.models.inkling.data.image_processing.IMAGE_PROCESSOR_CONFIG#
None
- bridge.models.inkling.data.image_processing._MEAN#
‘array(…)’
- bridge.models.inkling.data.image_processing._STD#
‘array(…)’
- bridge.models.inkling.data.image_processing._LANCZOS_LOBES#
‘float32(…)’
- bridge.models.inkling.data.image_processing._sinf() collections.abc.Callable[[float], float]#
Return the C library’s single-precision sine.
The Lanczos3 kernel evaluates its sine in single precision. A double-precision sine rounded to FP32 differs for about 1% of the kernel’s arguments, enough to move some resized pixels by one level.
- bridge.models.inkling.data.image_processing._sinc(value: numpy.float32) numpy.float32#
- bridge.models.inkling.data.image_processing._lanczos_weights(
- source_size: int,
- target_size: int,
- output_index: int,
First source index and normalized FP32 Lanczos3 weights for one output index.
- bridge.models.inkling.data.image_processing.resize_lanczos3(
- pixels: numpy.ndarray,
- width: int,
- height: int,
Resize
[H, W, C]uint8 pixels with a Lanczos3 kernel: rows first, then columns.Each output value sums its taps in order in FP32, then rounds half up and clips to uint8.
- class bridge.models.inkling.data.image_processing.Inkling2EncodedImage#
One preprocessed image.
.. attribute:: patches
[rows * columns, 2, 40, 40, 3]BF16 patches, the model’spixel_valuesrows... attribute:: grid_thw
[1, 3]grid(1, rows, columns), the model’simage_grid_thwrow... attribute:: num_tokens
Image tokens the language model reserves for this image,
rows * columns.- patches: torch.Tensor#
None
- grid_thw: torch.Tensor#
None
- num_tokens: int#
None
- bridge.models.inkling.data.image_processing._image_bytes(
- image: bytes | str | os.PathLike | PIL.Image.Image,
Return encoded image bytes; a PIL image is saved as PNG, as the reference does.
- bridge.models.inkling.data.image_processing._decode_rgb(image_bytes: bytes) numpy.ndarray#
Decode to
[H, W, 3]uint8, compositing transparent images onto an 8-pixel checkerboard.
- bridge.models.inkling.data.image_processing.load_processor_config(
- path: str | os.PathLike,
- **download_kwargs: Any,
Read a checkpoint’s
processor_config.json.- Parameters:
path – Checkpoint directory or Hugging Face repository ID.
**download_kwargs –
hf_hub_downloadarguments, such asrevision, for a repository ID.
- Returns:
The parsed configuration.
- class bridge.models.inkling.data.image_processing.Inkling2ImageProcessor(*, max_image_tokens: int | None = None)#
Turns images into Inkling2 vision patches and image-token counts.
- Parameters:
max_image_tokens – Token budget per image; a larger image is encoded at the largest size that fits, keeping its aspect ratio.
Nonekeeps every image at its native size, as the reference does.
Initialization
- patch_size#
None
- temporal_patch_size#
None
- classmethod from_pretrained(
- path: str | os.PathLike,
- *,
- max_image_tokens: int | None = None,
- **download_kwargs: Any,
Load from a checkpoint, checking the
image_processorsection of itsprocessor_config.json.- Parameters:
path – Checkpoint directory or Hugging Face repository ID.
max_image_tokens – Token budget per image, or
Nonefor native sizes.**download_kwargs –
hf_hub_downloadarguments, such asrevision, for a repository ID.
- Returns:
The image processor.
- Raises:
ValueError – If the checkpoint asks for image processing this port does not implement.
- classmethod from_dict(
- processor_config: dict[str, Any],
- *,
- max_image_tokens: int | None = None,
Build from a parsed
processor_config.json.- Parameters:
processor_config – The checkpoint’s processor configuration.
max_image_tokens – Token budget per image, or
Nonefor native sizes.
- Returns:
The image processor.
- Raises:
ValueError – If the configuration asks for image processing this port does not implement.
- static count_tokens(width: int, height: int) int#
Return the image tokens for an image of
widthxheightpixels.
- fitted_size(width: int, height: int) tuple[int, int]#
Return the size an image is encoded at: native, or the largest same-aspect size within the budget.
- encode(
- image: bytes | str | os.PathLike | PIL.Image.Image,
Preprocess one image.
- Parameters:
image – Encoded image bytes, a PIL image, or a local path.
- Returns:
The image’s patches, grid, and token count.
- bridge.models.inkling.data.image_processing.__all__#
[‘IMAGE_PROCESSOR_CONFIG’, ‘PROCESSOR_CONFIG_FILE’, ‘Inkling2EncodedImage’, ‘Inkling2ImageProcessor’…