bridge.models.inkling.modeling_inkling2.vision#
Inkling2 image encoder: a Perception Encoder-style vision transformer and its decoder adapter.
No Transformers implementation of this encoder exists, so it is defined here with the reference implementation’s parameter names and arithmetic. Inputs are merged 40x40 patches with two identical temporal slices; each is split into four native 20x20 patches. A vision transformer with a CLS token, interpolated learned position embeddings, 2D rotary embeddings, and LayerScale encodes each image bidirectionally; neighboring 2x2 features are then concatenated and projected to the decoder width.
Module Contents#
Classes#
Bidirectional multi-head attention within each image. |
|
GELU feed-forward block. |
|
Learned per-channel residual-branch scale. |
|
Pre-LayerNorm transformer layer with LayerScale. |
|
Native-patch embedding, CLS token, and learned position grid. |
|
Embeddings, pre-LayerNorm, and transformer layers. |
|
Encode preprocessed image patches into decoder-width embeddings, one per merged patch. |
Functions#
Bilinearly resample a learned |
|
Rotate interleaved channel pairs by column angles (first half) and row angles (second half). |
API#
- bridge.models.inkling.modeling_inkling2.vision.interpolate_position_embeddings(
- grid: torch.Tensor,
- rows: torch.Tensor,
- cols: torch.Tensor,
- heights: torch.Tensor,
- widths: torch.Tensor,
Bilinearly resample a learned
[G, G, D]position grid at each token’s patch center.
- bridge.models.inkling.modeling_inkling2.vision.apply_2d_rotary(
- query: torch.Tensor,
- key: torch.Tensor,
- positions: torch.Tensor,
Rotate interleaved channel pairs by column angles (first half) and row angles (second half).
- class bridge.models.inkling.modeling_inkling2.vision.Inkling2VisionAttention(width: int, heads: int)#
Bases:
torch.nn.ModuleBidirectional multi-head attention within each image.
Initialization
- forward(
- hidden: torch.Tensor,
- positions: torch.Tensor,
- lengths: list[int],
Attend within each image span of the flattened token sequence.
- class bridge.models.inkling.modeling_inkling2.vision.Inkling2VisionMLP(width: int, intermediate_size: int)#
Bases:
torch.nn.ModuleGELU feed-forward block.
Initialization
- forward(hidden: torch.Tensor) torch.Tensor#
Apply the feed-forward block.
- class bridge.models.inkling.modeling_inkling2.vision.Inkling2VisionLayerScale(width: int)#
Bases:
torch.nn.ModuleLearned per-channel residual-branch scale.
Initialization
- forward(hidden: torch.Tensor) torch.Tensor#
Scale the residual branch.
- class bridge.models.inkling.modeling_inkling2.vision.Inkling2VisionLayer(config: dict)#
Bases:
torch.nn.ModulePre-LayerNorm transformer layer with LayerScale.
Initialization
- forward(
- hidden: torch.Tensor,
- positions: torch.Tensor,
- lengths: list[int],
Apply attention and MLP residual branches.
- class bridge.models.inkling.modeling_inkling2.vision.Inkling2VisionEmbeddings(config: dict)#
Bases:
torch.nn.ModuleNative-patch embedding, CLS token, and learned position grid.
Initialization
- forward(pixels: torch.Tensor) torch.Tensor#
Embed flattened native patches.
- class bridge.models.inkling.modeling_inkling2.vision.Inkling2VisionBackbone(config: dict)#
Bases:
torch.nn.ModuleEmbeddings, pre-LayerNorm, and transformer layers.
Initialization
- class bridge.models.inkling.modeling_inkling2.vision.Inkling2VisionEncoder(vision_config: dict, text_hidden_size: int)#
Bases:
torch.nn.ModuleEncode preprocessed image patches into decoder-width embeddings, one per merged patch.
Initialization
- _preprocess(patches: torch.Tensor) torch.Tensor#
- _embed(patches: torch.Tensor, grid: list[list[int]])#
- _pool(
- hidden: torch.Tensor,
- grid: list[list[int]],
- lengths: list[int],
- forward(patches: torch.Tensor, grid_thw: torch.Tensor) torch.Tensor#
Encode images.
- Parameters:
patches –
[N, T, P * M, P * M, 3]merged patches scaled to [-1, 1] (mean and standard deviation 0.5, as the release’sprocessor_config.jsonsets), one per decoder image token, wherePis the native patch size andMthe adapter merge size.grid_thw –
[num_images, 3]merged-token grids with temporal size 1.
- Returns:
Image embeddings
[N, text_hidden_size].