bridge.models.inkling.modeling_inkling2.vision#

Inkling2 image encoder: a Perception Encoder-style vision transformer and its decoder adapter.

No Transformers implementation of this encoder exists, so it is defined here with the reference implementation’s parameter names and arithmetic. Inputs are merged 40x40 patches with two identical temporal slices; each is split into four native 20x20 patches. A vision transformer with a CLS token, interpolated learned position embeddings, 2D rotary embeddings, and LayerScale encodes each image bidirectionally; neighboring 2x2 features are then concatenated and projected to the decoder width.

Module Contents#

Classes#

Inkling2VisionAttention

Bidirectional multi-head attention within each image.

Inkling2VisionMLP

GELU feed-forward block.

Inkling2VisionLayerScale

Learned per-channel residual-branch scale.

Inkling2VisionLayer

Pre-LayerNorm transformer layer with LayerScale.

Inkling2VisionEmbeddings

Native-patch embedding, CLS token, and learned position grid.

Inkling2VisionBackbone

Embeddings, pre-LayerNorm, and transformer layers.

Inkling2VisionEncoder

Encode preprocessed image patches into decoder-width embeddings, one per merged patch.

Functions#

interpolate_position_embeddings

Bilinearly resample a learned [G, G, D] position grid at each token’s patch center.

apply_2d_rotary

Rotate interleaved channel pairs by column angles (first half) and row angles (second half).

API#

bridge.models.inkling.modeling_inkling2.vision.interpolate_position_embeddings(
grid: torch.Tensor,
rows: torch.Tensor,
cols: torch.Tensor,
heights: torch.Tensor,
widths: torch.Tensor,
) → torch.Tensor#

Bilinearly resample a learned [G, G, D] position grid at each token’s patch center.

bridge.models.inkling.modeling_inkling2.vision.apply_2d_rotary(
query: torch.Tensor,
key: torch.Tensor,
positions: torch.Tensor,
)#

Rotate interleaved channel pairs by column angles (first half) and row angles (second half).

class bridge.models.inkling.modeling_inkling2.vision.Inkling2VisionAttention(width: int, heads: int)#

Bases: torch.nn.Module

Bidirectional multi-head attention within each image.

Initialization

forward(
hidden: torch.Tensor,
positions: torch.Tensor,
lengths: list[int],
) → torch.Tensor#

Attend within each image span of the flattened token sequence.

class bridge.models.inkling.modeling_inkling2.vision.Inkling2VisionMLP(width: int, intermediate_size: int)#

Bases: torch.nn.Module

GELU feed-forward block.

Initialization

forward(hidden: torch.Tensor) → torch.Tensor#

Apply the feed-forward block.

class bridge.models.inkling.modeling_inkling2.vision.Inkling2VisionLayerScale(width: int)#

Bases: torch.nn.Module

Learned per-channel residual-branch scale.

Initialization

forward(hidden: torch.Tensor) → torch.Tensor#

Scale the residual branch.

class bridge.models.inkling.modeling_inkling2.vision.Inkling2VisionLayer(config: dict)#

Bases: torch.nn.Module

Pre-LayerNorm transformer layer with LayerScale.

Initialization

forward(
hidden: torch.Tensor,
positions: torch.Tensor,
lengths: list[int],
) → torch.Tensor#

Apply attention and MLP residual branches.

class bridge.models.inkling.modeling_inkling2.vision.Inkling2VisionEmbeddings(config: dict)#

Bases: torch.nn.Module

Native-patch embedding, CLS token, and learned position grid.

Initialization

forward(pixels: torch.Tensor) → torch.Tensor#

Embed flattened native patches.

class bridge.models.inkling.modeling_inkling2.vision.Inkling2VisionBackbone(config: dict)#

Bases: torch.nn.Module

Embeddings, pre-LayerNorm, and transformer layers.

Initialization

class bridge.models.inkling.modeling_inkling2.vision.Inkling2VisionEncoder(vision_config: dict, text_hidden_size: int)#

Bases: torch.nn.Module

Encode preprocessed image patches into decoder-width embeddings, one per merged patch.

Initialization

_preprocess(patches: torch.Tensor) → torch.Tensor#
_embed(patches: torch.Tensor, grid: list[list[int]])#
_pool(
hidden: torch.Tensor,
grid: list[list[int]],
lengths: list[int],
) → torch.Tensor#
forward(patches: torch.Tensor, grid_thw: torch.Tensor) → torch.Tensor#

Encode images.

Parameters:
  • patches – [N, T, P * M, P * M, 3] merged patches scaled to [-1, 1] (mean and standard deviation 0.5, as the release’s processor_config.json sets), one per decoder image token, where P is the native patch size and M the adapter merge size.

  • grid_thw – [num_images, 3] merged-token grids with temporal size 1.

Returns:

Image embeddings [N, text_hidden_size].