bridge.models.inkling.modeling_inkling2.model#

Inkling2 HybridModel: normalized token and image embeddings, AttnRes decoder blocks, and a soft-capped head.

Module Contents#

Classes#

Inkling2OutputLayerMixin

Soft-cap the logits and exclude the padded vocabulary rows from the softmax.

Inkling2Model

Inkling2 decoder as an MCore HybridModel with an embedding RMSNorm and an optional image encoder.

Functions#

reject_document_masks

Reject an attention mask that keeps documents apart, which Inkling2 would ignore.

API#

bridge.models.inkling.modeling_inkling2.model.reject_document_masks(
attention_mask: torch.Tensor | None,
packed_seq_params: Any,
cp_group: torch.distributed.ProcessGroup | None = None,
) → None#

Reject an attention mask that keeps documents apart, which Inkling2 would ignore.

Inkling2’s attention and short convolutions are causal and separate documents only through packed sequences (packed_seq_params); they never read attention_mask. That loses nothing for a causal mask or for padding at the end, but a mask that hides the first token from a later one, as reset_attention_mask and left padding produce, would be ignored. Context parallelism gives each rank different query rows, so the ranks agree on the verdict over cp_group.

Parameters:
  • attention_mask – Megatron-Core mask. In its query-by-key form, [b, 1, s_q, s_k], True hides a key; a [b, s] padding mask is not checked.

  • packed_seq_params – Document boundaries of a packed batch, or None.

  • cp_group – Context-parallel group, whose ranks hold different query rows.

Raises:

ValueError – If an unpacked batch’s mask hides the first token from a later one.

class bridge.models.inkling.modeling_inkling2.model.Inkling2OutputLayerMixin#

Bases: torch.nn.Module

Soft-cap the logits and exclude the padded vocabulary rows from the softmax.

forward(
*args: Any,
**kwargs: Any,
) → tuple[torch.Tensor, torch.Tensor | None]#
class bridge.models.inkling.modeling_inkling2.model.Inkling2Model(
config,
*,
vocab_size: int,
pg_collection: megatron.core.process_groups_config.ProcessGroupCollection,
pre_process: bool,
post_process: bool,
)#

Bases: megatron.core.models.hybrid.hybrid_model.HybridModel

Inkling2 decoder as an MCore HybridModel with an embedding RMSNorm and an optional image encoder.

The Hybrid pattern repeats *E: each decoder block is an attention layer followed by a MoE layer. The image encoder sits on the embedding stage.

Initialization

_freeze_requested_modules() → None#
embed(
input_ids: torch.Tensor,
pixel_values: torch.Tensor | None = None,
image_grid_thw: torch.Tensor | None = None,
) → torch.Tensor#

Return full-sequence decoder inputs [S, B, H]: normalized token embeddings with image features.

Image features replace the placeholder positions after the embedding norm, as in the reference implementation.

forward(
input_ids: torch.Tensor | None = None,
position_ids: torch.Tensor | None = None,
attention_mask: torch.Tensor | None = None,
*,
pixel_values: torch.Tensor | None = None,
image_grid_thw: torch.Tensor | None = None,
labels: torch.Tensor | None = None,
loss_mask: torch.Tensor | None = None,
packed_seq_params: Any = None,
decoder_input: torch.Tensor | None = None,
**kwargs: Any,
) → torch.Tensor | tuple[torch.Tensor, torch.Tensor]#

Embed tokens and images on the first stage, then run the decoder and output head.