bridge.models.inkling.modeling_inkling2.layers#

Inkling2 decoder blocks as Hybrid layers joined by Block Attention Residuals.

Each decoder block is an attention layer (Hybrid symbol *) followed by a MoE layer (E), so Hybrid layers 2 * i and 2 * i + 1 hold block i. Each sublayer reads a learned softmax mixture over the completed residual blocks and the current partial block instead of a single residual stream. Full-attention blocks mix separate Q, K, and V inputs. Every attention and MoE output passes through a residual short convolution before it joins the partial block.

HybridStack passes a single tensor from layer to layer, so every layer packs the AttnRes bank of completed blocks next to the partial block along the hidden dimension. The width grows by one hidden size per completed block, also across pipeline stages, whose communication exchanges tensor shapes.

Module Contents#

Classes#

Inkling2SiTU

Softcapped SwiGLU expert activation, installed through MCore’s custom-activation path.

_Inkling2HybridLayer

AttnRes bookkeeping shared by the attention and MoE layers of a decoder block.

Inkling2AttentionLayer

Attention layer of an Inkling2 decoder block (Hybrid symbol *).

Inkling2MoELayer

MoE layer of an Inkling2 decoder block (Hybrid symbol E).

Functions#

num_attn_res_blocks_before

Return the number of completed AttnRes blocks that decoder block block_index reads.

unpack_attn_res_state

Split a packed AttnRes state into its count hidden-size slices.

API#

class bridge.models.inkling.modeling_inkling2.layers.Inkling2SiTU(config)#

Bases: torch.nn.Module

Softcapped SwiGLU expert activation, installed through MCore’s custom-activation path.

Initialization

forward(inputs: torch.Tensor) → torch.Tensor#

Apply the activation to a fused [gate, up] projection.

bridge.models.inkling.modeling_inkling2.layers.num_attn_res_blocks_before(block_index: int, block_size: int) → int#

Return the number of completed AttnRes blocks that decoder block block_index reads.

bridge.models.inkling.modeling_inkling2.layers.unpack_attn_res_state(
state: torch.Tensor,
hidden_size: int,
count: int,
) → list[torch.Tensor]#

Split a packed AttnRes state into its count hidden-size slices.

class bridge.models.inkling.modeling_inkling2.layers._Inkling2HybridLayer(*args, **kwargs)#

Bases: megatron.core.transformer.transformer_layer.TransformerLayer

AttnRes bookkeeping shared by the attention and MoE layers of a decoder block.

Initialization

_add_float32_parameter(name: str, *shape: int) → None#

Register an AttnRes query, which stays FP32 under BF16 training.

_mix(
sources: list[torch.Tensor],
queries: torch.Tensor,
**kwargs,
) → torch.Tensor#
_normalize(
norm: torch.nn.Module,
inputs: torch.Tensor,
) → torch.Tensor#

Normalize an AttnRes mixture into the activation dtype.

_check_supported(inference_context) → None#
class bridge.models.inkling.modeling_inkling2.layers.Inkling2AttentionLayer(*args, **kwargs)#

Bases: bridge.models.inkling.modeling_inkling2.layers._Inkling2HybridLayer

Attention layer of an Inkling2 decoder block (Hybrid symbol *).

Initialization

forward(
hidden_states: torch.Tensor,
attention_mask: torch.Tensor | None = None,
inference_context=None,
packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams | None = None,
**kwargs,
) → tuple[torch.Tensor, None]#

Attend over the AttnRes mixtures and hand the block’s state to its MoE layer.

Parameters:
  • hidden_states – Packed [partial, *bank] state of the previous block.

  • packed_seq_params – Document boundaries of a packed batch.

Returns:

[attention, *bank, partial] when the block starts a new AttnRes block, otherwise [partial, attention, *bank]; and no context.

class bridge.models.inkling.modeling_inkling2.layers.Inkling2MoELayer(*args, **kwargs)#

Bases: bridge.models.inkling.modeling_inkling2.layers._Inkling2HybridLayer

MoE layer of an Inkling2 decoder block (Hybrid symbol E).

Initialization

_mlp_conv(
output: torch.Tensor,
packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams | None,
) → torch.Tensor#
static _without_bias(
output_with_bias: tuple[torch.Tensor, torch.Tensor | None],
) → torch.Tensor#
forward(
hidden_states: torch.Tensor,
attention_mask: torch.Tensor | None = None,
inference_context=None,
packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams | None = None,
padding_mask: torch.Tensor | None = None,
**kwargs,
) → tuple[torch.Tensor, None]#

Run the MoE on its AttnRes mixture and update the partial block.

Parameters:
  • hidden_states – Packed state from the block’s attention layer.

  • packed_seq_params – Document boundaries of a packed batch.

  • padding_mask – Optional MoE router padding mask.

Returns:

The packed [partial, *bank] state, or the final AttnRes output after the last block; and no context.